2024-04-23T18:00:15.241 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-23T18:00:15.242 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} --archive /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101 --name leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi --verbose -- /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101/orig.config.yaml 2024-04-23T18:00:15.311 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T18:00:15.416 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7670101' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 57e0caf76fdb7ab8b1358588f292d08519163844 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5182 sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f sleep_before_teardown: 0 subset: 200/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph suite_sha1: 57e0caf76fdb7ab8b1358588f292d08519163844 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGaaK12rhtBlQ2ZraE3HzZOX8bInYJtCLxd2m4a+aIEOq0tEQ/DmRtHB2ZsZKFwF1h2qtohYm3R7ducpnFjcA8= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC9QRn+TObUz6O8/tWtEBg6n7xqlLin8R3q7SRJd0CArp42c1jR8FMV4CCgahXIxX7VFx1po6Vf6OAtGjwZHYmg= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL+gYn/+XUNQld8rFQ0rlBXLVFItgMBa/qRJwUNTE+WQ4StgyyRIyyLBbTFawnvfN7pwqXw8GLNCut9Nzodlks4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-23_17:44:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-23T18:00:15.416 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa; will attempt to use it 2024-04-23T18:00:15.417 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks 2024-04-23T18:00:15.417 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-23T18:00:15.419 INFO:teuthology.task.internal:Checking packages... 2024-04-23T18:00:15.448 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd69f13d827952bb07bb9b7973bca14f97d764d6f' 2024-04-23T18:00:15.448 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T18:00:15.448 INFO:teuthology.packaging:ref: None 2024-04-23T18:00:15.448 INFO:teuthology.packaging:tag: None 2024-04-23T18:00:15.448 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-23T18:00:15.448 INFO:teuthology.packaging:sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:00:15.448 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-23T18:00:15.611 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.gd69f13d8 2024-04-23T18:00:15.613 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-23T18:00:15.619 INFO:teuthology.task.internal:no buildpackages task found 2024-04-23T18:00:15.620 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-23T18:00:15.625 INFO:teuthology.task.internal:Saving configuration 2024-04-23T18:00:15.649 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-23T18:00:15.655 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T18:00:15.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:55:22.059170', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGaaK12rhtBlQ2ZraE3HzZOX8bInYJtCLxd2m4a+aIEOq0tEQ/DmRtHB2ZsZKFwF1h2qtohYm3R7ducpnFjcA8='} 2024-04-23T18:00:15.706 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:55:22.057853', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC9QRn+TObUz6O8/tWtEBg6n7xqlLin8R3q7SRJd0CArp42c1jR8FMV4CCgahXIxX7VFx1po6Vf6OAtGjwZHYmg='} 2024-04-23T18:00:15.729 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:55:22.056185', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL+gYn/+XUNQld8rFQ0rlBXLVFItgMBa/qRJwUNTE+WQ4StgyyRIyyLBbTFawnvfN7pwqXw8GLNCut9Nzodlks4='} 2024-04-23T18:00:15.729 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-23T18:00:15.735 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-23T18:00:15.736 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-23T18:00:15.736 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-23T18:00:15.736 INFO:teuthology.run_tasks:Running task console_log... 2024-04-23T18:00:15.842 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f96f04fa0d0>, signals=[15]) 2024-04-23T18:00:15.842 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-23T18:00:15.848 INFO:teuthology.task.internal:Opening connections... 2024-04-23T18:00:15.848 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-04-23T18:00:15.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T18:00:15.935 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-04-23T18:00:15.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T18:00:16.012 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-04-23T18:00:16.013 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T18:00:16.089 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-23T18:00:16.096 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-04-23T18:00:16.115 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-04-23T18:00:16.116 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-04-23T18:00:16.173 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2024-04-23T18:00:16.174 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="9" 2024-04-23T18:00:16.174 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2024-04-23T18:00:16.174 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2024-04-23T18:00:16.174 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="9" 2024-04-23T18:00:16.174 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el9" 2024-04-23T18:00:16.174 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T18:00:16.174 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2024-04-23T18:00:16.174 INFO:teuthology.orchestra.run.smithi050.stdout:LOGO="fedora-logo-icon" 2024-04-23T18:00:16.174 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T18:00:16.174 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2024-04-23T18:00:16.174 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T18:00:16.175 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T18:00:16.175 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T18:00:16.175 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-04-23T18:00:16.195 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-04-23T18:00:16.213 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-04-23T18:00:16.213 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-04-23T18:00:16.271 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-04-23T18:00:16.271 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-04-23T18:00:16.271 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-04-23T18:00:16.271 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-04-23T18:00:16.271 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-04-23T18:00:16.271 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-04-23T18:00:16.271 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T18:00:16.272 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-04-23T18:00:16.272 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-04-23T18:00:16.272 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T18:00:16.272 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-04-23T18:00:16.272 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T18:00:16.272 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T18:00:16.272 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T18:00:16.272 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-04-23T18:00:16.294 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-04-23T18:00:16.312 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-04-23T18:00:16.312 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-04-23T18:00:16.369 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2024-04-23T18:00:16.369 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="9" 2024-04-23T18:00:16.369 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2024-04-23T18:00:16.369 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2024-04-23T18:00:16.369 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="9" 2024-04-23T18:00:16.369 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el9" 2024-04-23T18:00:16.369 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T18:00:16.369 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-04-23T18:00:16.369 INFO:teuthology.orchestra.run.smithi188.stdout:LOGO="fedora-logo-icon" 2024-04-23T18:00:16.369 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T18:00:16.370 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2024-04-23T18:00:16.370 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T18:00:16.370 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T18:00:16.370 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T18:00:16.370 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-04-23T18:00:16.394 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-23T18:00:16.404 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-23T18:00:16.409 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-23T18:00:16.409 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-04-23T18:00:16.412 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-04-23T18:00:16.414 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-04-23T18:00:16.431 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-23T18:00:16.437 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-23T18:00:16.437 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-04-23T18:00:16.470 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-04-23T18:00:16.473 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-04-23T18:00:16.575 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-23T18:00:16.652 INFO:teuthology.run_tasks:Running task kernel... 2024-04-23T18:00:16.667 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-23T18:00:16.667 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-23T18:00:16.667 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-23T18:00:16.668 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T18:00:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:00:16.685 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-04-23T18:00:16.744 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-437.el9.x86_64 2024-04-23T18:00:16.744 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.14.0-437.el9.x86_64 2024-04-23T18:00:16.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2024-04-23T18:00:19.110 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 9.9 MB/s | 8.0 MB 00:00 2024-04-23T18:00:36.459 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-04-23T18:00:42.360 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.2 MB 00:02 2024-04-23T18:00:43.944 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-04-23T18:00:46.303 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 21 MB 00:02 2024-04-23T18:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-23T18:00:53.451 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-23T18:00:53.452 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-23T18:00:53.546 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-04-23T18:00:53.547 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-04-23T18:00:53.548 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-04-23T18:00:53.688 DEBUG:teuthology.orchestra.run.smithi050:> echo no | sudo yum reinstall kernel || true 2024-04-23T18:00:54.212 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:03 ago on Tue 23 Apr 2024 06:00:51 PM UTC. 2024-04-23T18:00:54.335 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-04-23T18:00:54.348 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-04-23T18:00:54.348 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-04-23T18:00:54.348 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-04-23T18:00:54.348 INFO:teuthology.orchestra.run.smithi050.stdout:Reinstalling: 2024-04-23T18:00:54.348 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T18:00:54.348 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:00:54.349 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-04-23T18:00:54.349 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-04-23T18:00:54.349 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:00:54.349 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 255 k 2024-04-23T18:00:54.349 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 0 2024-04-23T18:00:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:Operation aborted. 2024-04-23T18:00:54.406 INFO:teuthology.orchestra.run.smithi050.stdout:Is this ok [y/N]: 2024-04-23T18:00:54.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum reinstall -y kernel || true 2024-04-23T18:00:54.921 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:03 ago on Tue 23 Apr 2024 06:00:51 PM UTC. 2024-04-23T18:00:55.042 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-04-23T18:00:55.044 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-04-23T18:00:55.044 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-04-23T18:00:55.044 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-04-23T18:00:55.044 INFO:teuthology.orchestra.run.smithi050.stdout:Reinstalling: 2024-04-23T18:00:55.044 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T18:00:55.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:00:55.045 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-04-23T18:00:55.045 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-04-23T18:00:55.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:00:55.045 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 255 k 2024-04-23T18:00:55.045 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 0 2024-04-23T18:00:55.046 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-04-23T18:01:02.948 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.4 MB/s | 255 kB 00:00 2024-04-23T18:01:02.948 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:01:02.948 INFO:teuthology.orchestra.run.smithi050.stdout:Total 32 kB/s | 255 kB 00:07 2024-04-23T18:01:02.969 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-04-23T18:01:03.158 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-04-23T18:01:03.158 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-04-23T18:01:03.163 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-04-23T18:01:03.164 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-04-23T18:01:03.318 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-04-23T18:01:03.406 INFO:teuthology.orchestra.run.smithi050.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T18:01:03.407 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T18:01:04.085 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T18:01:04.086 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T18:01:04.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T18:01:04.794 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:01:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:Reinstalled: 2024-04-23T18:01:04.794 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-23T18:01:04.794 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:01:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-04-23T18:01:04.997 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T18:01:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-23T18:01:05.045 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-23T18:01:05.045 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-23T18:01:05.046 DEBUG:teuthology.parallel:result is None 2024-04-23T18:01:05.046 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-23T18:01:05.103 INFO:teuthology.task.internal:Creating test directory... 2024-04-23T18:01:05.103 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T18:01:05.106 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T18:01:05.108 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T18:01:05.126 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-23T18:01:05.133 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-23T18:01:05.141 INFO:teuthology.task.internal:Creating archive directory... 2024-04-23T18:01:05.141 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T18:01:05.164 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T18:01:05.166 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T18:01:05.220 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-23T18:01:05.229 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-23T18:01:05.229 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T18:01:05.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:01:05.246 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T18:01:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:01:05.263 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T18:01:05.265 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T18:01:05.289 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T18:01:05.294 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T18:01:05.306 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T18:01:05.335 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T18:01:05.366 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T18:01:05.369 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T18:01:05.375 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T18:01:05.377 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-23T18:01:05.384 INFO:teuthology.task.internal:Configuring sudo... 2024-04-23T18:01:05.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T18:01:05.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T18:01:05.389 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T18:01:05.415 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-23T18:01:05.424 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-23T18:01:05.424 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T18:01:05.452 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T18:01:05.454 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T18:01:05.475 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T18:01:05.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T18:01:05.599 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T18:01:05.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T18:01:05.721 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:01:05.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T18:01:05.785 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T18:01:05.829 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T18:01:05.884 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T18:01:05.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T18:01:06.010 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-23T18:01:06.010 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T18:01:06.076 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T18:01:06.118 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T18:01:06.176 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T18:01:06.255 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T18:01:06.301 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T18:01:06.301 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T18:01:06.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-04-23T18:01:06.369 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-04-23T18:01:06.372 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-04-23T18:01:06.407 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T18:01:06.408 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T18:01:06.450 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T18:01:06.864 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-23T18:01:06.872 INFO:teuthology.task.internal:Starting timer... 2024-04-23T18:01:06.872 INFO:teuthology.run_tasks:Running task pcp... 2024-04-23T18:01:06.882 INFO:teuthology.run_tasks:Running task selinux... 2024-04-23T18:01:06.891 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-23T18:01:06.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2024-04-23T18:01:06.975 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2024-04-23T18:01:06.977 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-04-23T18:01:07.020 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-04-23T18:01:07.021 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2024-04-23T18:01:07.066 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2024-04-23T18:01:07.069 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-23T18:01:07.069 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2024-04-23T18:01:07.092 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2024-04-23T18:01:07.093 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-04-23T18:01:07.118 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-04-23T18:01:07.119 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2024-04-23T18:01:07.140 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2024-04-23T18:01:07.140 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2024-04-23T18:01:07.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T18:01:07.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:01:07.165 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T18:01:07.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:01:07.193 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T18:01:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:01:07.219 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-23T18:01:07.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2024-04-23T18:01:07.254 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-04-23T18:01:07.281 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2024-04-23T18:01:07.311 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-23T18:01:07.322 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-23T18:01:07.328 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-23T18:01:07.481 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-23T18:01:07.497 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-23T18:01:07.498 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com,smithi177.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-23T18:05:09.725 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-04-23T18:05:09.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-23T18:05:09.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T18:05:09.809 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-04-23T18:05:09.890 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-23T18:05:09.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-23T18:05:09.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T18:05:09.968 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-04-23T18:05:10.051 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-23T18:05:10.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-23T18:05:10.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T18:05:10.131 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-04-23T18:05:10.211 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-23T18:05:10.212 INFO:teuthology.run_tasks:Running task clock... 2024-04-23T18:05:10.224 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-23T18:05:10.224 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T18:05:10.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:05:10.227 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T18:05:10.227 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:05:10.230 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T18:05:10.230 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T18:05:10.256 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T18:05:10.258 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T18:05:10.271 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T18:05:10.272 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T18:05:10.282 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T18:05:10.296 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2024-04-23T18:05:10.296 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-04-23T18:05:10.296 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T18:05:10.307 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2024-04-23T18:05:10.307 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-04-23T18:05:10.321 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T18:05:10.321 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T18:05:10.321 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2024-04-23T18:05:10.332 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2024-04-23T18:05:10.333 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T18:05:10.335 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T18:05:10.345 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T18:05:10.359 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T18:05:10.385 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:05:10.388 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:05:10.389 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-04-23T18:05:10.389 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:05:10.389 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:05:10.389 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:05:10.389 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:05:10.397 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:05:10.401 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:05:10.401 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-04-23T18:05:10.401 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:05:10.401 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:05:10.401 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:05:10.401 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:05:10.416 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-04-23T18:05:10.420 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T18:05:10.420 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-23T18:05:10.420 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:05:10.420 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:05:10.420 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:05:10.420 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T18:05:10.421 INFO:teuthology.run_tasks:Running task install... 2024-04-23T18:05:10.429 DEBUG:teuthology.task.install:project ceph 2024-04-23T18:05:10.429 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'}} 2024-04-23T18:05:10.429 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'} 2024-04-23T18:05:10.429 INFO:teuthology.task.install:Using flavor: default 2024-04-23T18:05:10.435 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-23T18:05:10.435 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-23T18:05:10.435 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-23T18:05:10.436 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:05:10.438 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-23T18:05:10.438 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:05:10.439 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-23T18:05:10.439 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:05:10.587 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-23T18:05:10.588 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-23T18:05:10.596 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-23T18:05:10.596 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-23T18:05:10.659 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-23T18:05:10.659 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-23T18:05:10.708 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T18:05:10.708 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-23T18:05:10.708 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T18:05:10.733 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T18:05:10.734 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T18:05:10.758 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T18:05:10.759 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:05:10.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T18:05:10.785 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T18:05:10.785 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T18:05:10.797 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T18:05:10.830 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T18:05:10.830 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T18:05:10.831 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T18:05:10.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T18:05:10.856 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T18:05:10.856 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T18:05:10.871 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T18:05:10.921 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T18:05:10.927 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T18:05:10.933 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-04-23T18:05:10.935 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-04-23T18:05:10.991 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2024-04-23T18:05:10.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2024-04-23T18:05:10.994 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T18:05:11.057 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2024-04-23T18:05:11.059 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2024-04-23T18:05:11.159 INFO:teuthology.orchestra.run.smithi177.stdout:48 files removed 2024-04-23T18:05:11.186 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T18:05:11.226 INFO:teuthology.orchestra.run.smithi050.stdout:48 files removed 2024-04-23T18:05:11.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T18:05:11.283 INFO:teuthology.orchestra.run.smithi188.stdout:48 files removed 2024-04-23T18:05:11.311 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T18:05:11.767 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 262 kB/s | 83 kB 00:00 2024-04-23T18:05:11.850 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 259 kB/s | 83 kB 00:00 2024-04-23T18:05:12.040 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 178 kB/s | 83 kB 00:00 2024-04-23T18:05:12.134 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-04-23T18:05:12.276 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-23T18:05:12.476 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-04-23T18:05:12.482 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-23T18:05:12.651 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-04-23T18:05:12.925 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-04-23T18:05:13.285 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-23T18:05:14.101 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-04-23T18:05:14.200 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.0 MB 00:01 2024-04-23T18:05:15.144 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-23T18:05:16.811 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-23T18:05:16.926 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-23T18:05:19.703 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 6.2 MB 00:00 2024-04-23T18:05:21.371 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-23T18:05:21.468 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - CRB 6.6 MB/s | 6.2 MB 00:00 2024-04-23T18:05:21.728 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 6.2 MB 00:00 2024-04-23T18:05:23.156 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-23T18:05:23.412 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-23T18:05:23.537 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 16 kB 00:00 2024-04-23T18:05:24.424 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-04-23T18:05:24.931 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-23T18:05:28.214 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-04-23T18:05:29.625 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-04-23T18:05:30.053 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-23T18:05:30.315 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:05:30.316 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:05:30.326 INFO:teuthology.orchestra.run.smithi177.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T18:05:30.327 INFO:teuthology.orchestra.run.smithi177.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T18:05:30.328 INFO:teuthology.orchestra.run.smithi177.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T18:05:30.329 INFO:teuthology.orchestra.run.smithi177.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T18:05:30.331 INFO:teuthology.orchestra.run.smithi177.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T18:05:30.333 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T18:05:30.334 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T18:05:30.395 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-04-23T18:05:30.415 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-04-23T18:05:30.415 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-04-23T18:05:30.415 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-23T18:05:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T18:05:30.417 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T18:05:30.418 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T18:05:30.418 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T18:05:30.418 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T18:05:30.418 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T18:05:30.418 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-23T18:05:30.418 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-23T18:05:30.418 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-23T18:05:30.418 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-23T18:05:30.418 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-23T18:05:30.418 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:05:30.418 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-23T18:05:30.418 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-23T18:05:30.419 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-23T18:05:30.420 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T18:05:30.421 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T18:05:30.422 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T18:05:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T18:05:30.424 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T18:05:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T18:05:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout:Install 148 Packages 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 4 Packages 2024-04-23T18:05:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:05:30.428 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 222 M 2024-04-23T18:05:30.428 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-04-23T18:05:31.616 INFO:teuthology.orchestra.run.smithi177.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 39 kB/s | 6.6 kB 00:00 2024-04-23T18:05:31.818 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:05:31.818 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:05:31.829 INFO:teuthology.orchestra.run.smithi050.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T18:05:31.830 INFO:teuthology.orchestra.run.smithi050.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T18:05:31.831 INFO:teuthology.orchestra.run.smithi050.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T18:05:31.832 INFO:teuthology.orchestra.run.smithi050.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T18:05:31.833 INFO:teuthology.orchestra.run.smithi050.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T18:05:31.836 INFO:teuthology.orchestra.run.smithi050.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T18:05:31.836 INFO:teuthology.orchestra.run.smithi050.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T18:05:31.882 INFO:teuthology.orchestra.run.smithi177.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 3.3 MB/s | 890 kB 00:00 2024-04-23T18:05:31.900 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-04-23T18:05:31.910 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-04-23T18:05:31.910 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-04-23T18:05:31.910 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-04-23T18:05:31.910 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-04-23T18:05:31.910 INFO:teuthology.orchestra.run.smithi050.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T18:05:31.910 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-23T18:05:31.910 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:05:31.910 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-23T18:05:31.911 INFO:teuthology.orchestra.run.smithi050.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-23T18:05:31.912 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-23T18:05:31.913 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-23T18:05:31.913 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-23T18:05:31.913 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:05:31.913 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-23T18:05:31.913 INFO:teuthology.orchestra.run.smithi050.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T18:05:31.913 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2024-04-23T18:05:31.913 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-23T18:05:31.913 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:05:31.913 INFO:teuthology.orchestra.run.smithi050.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T18:05:31.913 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T18:05:31.913 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-04-23T18:05:31.913 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T18:05:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T18:05:31.915 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T18:05:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T18:05:31.917 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T18:05:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T18:05:31.919 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T18:05:31.920 INFO:teuthology.orchestra.run.smithi050.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T18:05:31.921 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:05:31.922 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-04-23T18:05:31.922 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-04-23T18:05:31.922 INFO:teuthology.orchestra.run.smithi050.stdout:Install 148 Packages 2024-04-23T18:05:31.922 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 4 Packages 2024-04-23T18:05:31.922 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:05:31.922 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 222 M 2024-04-23T18:05:31.922 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-04-23T18:05:31.949 INFO:teuthology.orchestra.run.smithi177.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 2.1 MB/s | 145 kB 00:00 2024-04-23T18:05:32.166 INFO:teuthology.orchestra.run.smithi177.stdout:(4/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 7.5 MB/s | 5.4 MB 00:00 2024-04-23T18:05:32.171 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:05:32.172 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T18:05:32.186 INFO:teuthology.orchestra.run.smithi188.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T18:05:32.186 INFO:teuthology.orchestra.run.smithi188.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T18:05:32.186 INFO:teuthology.orchestra.run.smithi188.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T18:05:32.186 INFO:teuthology.orchestra.run.smithi188.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T18:05:32.187 INFO:teuthology.orchestra.run.smithi188.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T18:05:32.189 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T18:05:32.190 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T18:05:32.251 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-23T18:05:32.278 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-04-23T18:05:32.278 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-23T18:05:32.278 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-04-23T18:05:32.278 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-23T18:05:32.278 INFO:teuthology.orchestra.run.smithi188.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T18:05:32.278 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-23T18:05:32.278 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:05:32.278 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-23T18:05:32.278 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-23T18:05:32.278 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-23T18:05:32.278 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T18:05:32.279 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-23T18:05:32.280 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-23T18:05:32.281 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:05:32.281 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-23T18:05:32.281 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T18:05:32.281 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-04-23T18:05:32.281 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-23T18:05:32.281 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T18:05:32.281 INFO:teuthology.orchestra.run.smithi188.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T18:05:32.281 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T18:05:32.281 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T18:05:32.282 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T18:05:32.283 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T18:05:32.284 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T18:05:32.285 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T18:05:32.286 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T18:05:32.287 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T18:05:32.288 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout:Installing weak dependencies: 2024-04-23T18:05:32.289 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T18:05:32.290 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:05:32.290 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-23T18:05:32.290 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-04-23T18:05:32.290 INFO:teuthology.orchestra.run.smithi188.stdout:Install 148 Packages 2024-04-23T18:05:32.290 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 4 Packages 2024-04-23T18:05:32.290 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:05:32.290 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 222 M 2024-04-23T18:05:32.290 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-23T18:05:32.350 INFO:teuthology.orchestra.run.smithi177.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 8.2 MB/s | 1.5 MB 00:00 2024-04-23T18:05:32.433 INFO:teuthology.orchestra.run.smithi177.stdout:(6/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 4.9 MB/s | 2.4 MB 00:00 2024-04-23T18:05:32.867 INFO:teuthology.orchestra.run.smithi177.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 9.1 MB/s | 4.7 MB 00:00 2024-04-23T18:05:33.051 INFO:teuthology.orchestra.run.smithi050.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 13 kB/s | 6.6 kB 00:00 2024-04-23T18:05:33.434 INFO:teuthology.orchestra.run.smithi177.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 11 MB/s | 21 MB 00:01 2024-04-23T18:05:33.509 INFO:teuthology.orchestra.run.smithi177.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 336 kB/s | 25 kB 00:00 2024-04-23T18:05:33.676 INFO:teuthology.orchestra.run.smithi050.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.4 MB/s | 890 kB 00:00 2024-04-23T18:05:33.705 INFO:teuthology.orchestra.run.smithi188.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 20 kB/s | 6.6 kB 00:00 2024-04-23T18:05:33.818 INFO:teuthology.orchestra.run.smithi050.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 1.0 MB/s | 145 kB 00:00 2024-04-23T18:05:34.159 INFO:teuthology.orchestra.run.smithi177.stdout:(10/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 8.2 MB/s | 11 MB 00:01 2024-04-23T18:05:34.243 INFO:teuthology.orchestra.run.smithi177.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.6 MB/s | 225 kB 00:00 2024-04-23T18:05:34.318 INFO:teuthology.orchestra.run.smithi177.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 362 kB/s | 27 kB 00:00 2024-04-23T18:05:34.402 INFO:teuthology.orchestra.run.smithi177.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 380 kB/s | 32 kB 00:00 2024-04-23T18:05:34.577 INFO:teuthology.orchestra.run.smithi177.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 4.2 MB/s | 747 kB 00:00 2024-04-23T18:05:34.677 INFO:teuthology.orchestra.run.smithi177.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.6 MB/s | 163 kB 00:00 2024-04-23T18:05:34.754 INFO:teuthology.orchestra.run.smithi188.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 848 kB/s | 890 kB 00:01 2024-04-23T18:05:34.761 INFO:teuthology.orchestra.run.smithi177.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.5 MB/s | 126 kB 00:00 2024-04-23T18:05:34.768 INFO:teuthology.orchestra.run.smithi050.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 2.5 MB/s | 2.4 MB 00:00 2024-04-23T18:05:34.844 INFO:teuthology.orchestra.run.smithi050.stdout:(5/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 2.3 MB/s | 5.4 MB 00:02 2024-04-23T18:05:34.880 INFO:teuthology.orchestra.run.smithi188.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 1.1 MB/s | 145 kB 00:00 2024-04-23T18:05:34.911 INFO:teuthology.orchestra.run.smithi177.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 3.3 MB/s | 500 kB 00:00 2024-04-23T18:05:35.436 INFO:teuthology.orchestra.run.smithi050.stdout:(6/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 2.2 MB/s | 1.5 MB 00:00 2024-04-23T18:05:35.953 INFO:teuthology.orchestra.run.smithi177.stdout:(18/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 4.8 MB/s | 17 MB 00:03 2024-04-23T18:05:36.029 INFO:teuthology.orchestra.run.smithi177.stdout:(19/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 4.8 MB/s | 5.4 MB 00:01 2024-04-23T18:05:36.046 INFO:teuthology.orchestra.run.smithi177.stdout:(20/152): python3-ceph-argparse-19.0.0-3156.gd6 488 kB/s | 45 kB 00:00 2024-04-23T18:05:36.079 INFO:teuthology.orchestra.run.smithi177.stdout:(21/152): python3-ceph-common-19.0.0-3156.gd69f 2.6 MB/s | 135 kB 00:00 2024-04-23T18:05:36.113 INFO:teuthology.orchestra.run.smithi177.stdout:(22/152): python3-cephfs-19.0.0-3156.gd69f13d8. 2.4 MB/s | 161 kB 00:00 2024-04-23T18:05:36.155 INFO:teuthology.orchestra.run.smithi177.stdout:(23/152): python3-rados-19.0.0-3156.gd69f13d8.e 4.2 MB/s | 320 kB 00:00 2024-04-23T18:05:36.189 INFO:teuthology.orchestra.run.smithi177.stdout:(24/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 3.9 MB/s | 298 kB 00:00 2024-04-23T18:05:36.245 INFO:teuthology.orchestra.run.smithi177.stdout:(25/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.1 MB/s | 99 kB 00:00 2024-04-23T18:05:36.246 INFO:teuthology.orchestra.run.smithi188.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 1.7 MB/s | 2.4 MB 00:01 2024-04-23T18:05:36.264 INFO:teuthology.orchestra.run.smithi177.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.1 MB/s | 85 kB 00:00 2024-04-23T18:05:36.322 INFO:teuthology.orchestra.run.smithi188.stdout:(5/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 1.8 MB/s | 5.4 MB 00:02 2024-04-23T18:05:36.339 INFO:teuthology.orchestra.run.smithi177.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 2.2 MB/s | 170 kB 00:00 2024-04-23T18:05:36.415 INFO:teuthology.orchestra.run.smithi177.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 456 kB/s | 34 kB 00:00 2024-04-23T18:05:36.507 INFO:teuthology.orchestra.run.smithi177.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.5 MB/s | 142 kB 00:00 2024-04-23T18:05:36.786 INFO:teuthology.orchestra.run.smithi050.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 2.4 MB/s | 4.7 MB 00:01 2024-04-23T18:05:36.874 INFO:teuthology.orchestra.run.smithi177.stdout:(30/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 4.9 MB/s | 3.1 MB 00:00 2024-04-23T18:05:36.905 INFO:teuthology.orchestra.run.smithi188.stdout:(6/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 2.3 MB/s | 1.5 MB 00:00 2024-04-23T18:05:37.516 INFO:teuthology.orchestra.run.smithi177.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 3.5 MB/s | 3.5 MB 00:01 2024-04-23T18:05:37.691 INFO:teuthology.orchestra.run.smithi177.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.4 MB/s | 249 kB 00:00 2024-04-23T18:05:37.791 INFO:teuthology.orchestra.run.smithi177.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 493 kB/s | 49 kB 00:00 2024-04-23T18:05:37.883 INFO:teuthology.orchestra.run.smithi177.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 179 kB/s | 16 kB 00:00 2024-04-23T18:05:38.133 INFO:teuthology.orchestra.run.smithi177.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 1.1 MB/s | 273 kB 00:00 2024-04-23T18:05:38.492 INFO:teuthology.orchestra.run.smithi177.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 2.1 MB/s | 762 kB 00:00 2024-04-23T18:05:38.576 INFO:teuthology.orchestra.run.smithi177.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 219 kB/s | 18 kB 00:00 2024-04-23T18:05:38.722 INFO:teuthology.orchestra.run.smithi188.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 2.0 MB/s | 4.7 MB 00:02 2024-04-23T18:05:38.792 INFO:teuthology.orchestra.run.smithi177.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 371 kB/s | 80 kB 00:00 2024-04-23T18:05:38.976 INFO:teuthology.orchestra.run.smithi177.stdout:(39/152): ceph-mgr-diskprediction-local-19.0.0- 3.5 MB/s | 7.4 MB 00:02 2024-04-23T18:05:39.068 INFO:teuthology.orchestra.run.smithi177.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-04-23T18:05:39.527 INFO:teuthology.orchestra.run.smithi177.stdout:(41/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 8.2 MB/s | 49 MB 00:06 2024-04-23T18:05:39.552 INFO:teuthology.orchestra.run.smithi177.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 523 kB/s | 253 kB 00:00 2024-04-23T18:05:39.619 INFO:teuthology.orchestra.run.smithi177.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-23T18:05:39.644 INFO:teuthology.orchestra.run.smithi177.stdout:(44/152): libquadmath-11.4.1-3.el9.x86_64.rpm 292 kB/s | 195 kB 00:00 2024-04-23T18:05:39.686 INFO:teuthology.orchestra.run.smithi177.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-23T18:05:39.711 INFO:teuthology.orchestra.run.smithi177.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-23T18:05:39.738 INFO:teuthology.orchestra.run.smithi188.stdout:(8/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x86 5.9 MB/s | 17 MB 00:02 2024-04-23T18:05:39.753 INFO:teuthology.orchestra.run.smithi177.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-04-23T18:05:39.780 INFO:teuthology.orchestra.run.smithi188.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 611 kB/s | 25 kB 00:00 2024-04-23T18:05:39.829 INFO:teuthology.orchestra.run.smithi177.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 5.8 MB/s | 689 kB 00:00 2024-04-23T18:05:39.887 INFO:teuthology.orchestra.run.smithi177.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 6.9 MB/s | 943 kB 00:00 2024-04-23T18:05:39.904 INFO:teuthology.orchestra.run.smithi177.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-04-23T18:05:39.938 INFO:teuthology.orchestra.run.smithi177.stdout:(51/152): python3-cryptography-36.0.1-4.el9.x86 3.0 MB/s | 1.2 MB 00:00 2024-04-23T18:05:39.963 INFO:teuthology.orchestra.run.smithi177.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 285 kB/s | 21 kB 00:00 2024-04-23T18:05:39.989 INFO:teuthology.orchestra.run.smithi177.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-04-23T18:05:40.014 INFO:teuthology.orchestra.run.smithi177.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 434 kB/s | 33 kB 00:00 2024-04-23T18:05:40.080 INFO:teuthology.orchestra.run.smithi177.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 199 kB/s | 18 kB 00:00 2024-04-23T18:05:40.106 INFO:teuthology.orchestra.run.smithi177.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-04-23T18:05:40.156 INFO:teuthology.orchestra.run.smithi177.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-23T18:05:40.190 INFO:teuthology.orchestra.run.smithi177.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 541 kB/s | 45 kB 00:00 2024-04-23T18:05:40.240 INFO:teuthology.orchestra.run.smithi177.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-04-23T18:05:40.266 INFO:teuthology.orchestra.run.smithi177.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-04-23T18:05:40.316 INFO:teuthology.orchestra.run.smithi177.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 338 kB/s | 25 kB 00:00 2024-04-23T18:05:40.349 INFO:teuthology.orchestra.run.smithi177.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 3.6 MB/s | 582 kB 00:00 2024-04-23T18:05:40.366 INFO:teuthology.orchestra.run.smithi177.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 472 kB/s | 47 kB 00:00 2024-04-23T18:05:40.371 INFO:teuthology.orchestra.run.smithi050.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 2.7 MB/s | 21 MB 00:07 2024-04-23T18:05:40.392 INFO:teuthology.orchestra.run.smithi177.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-23T18:05:40.433 INFO:teuthology.orchestra.run.smithi177.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 530 kB/s | 35 kB 00:00 2024-04-23T18:05:40.459 INFO:teuthology.orchestra.run.smithi177.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-04-23T18:05:40.502 INFO:teuthology.orchestra.run.smithi050.stdout:(9/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 2.8 MB/s | 11 MB 00:03 2024-04-23T18:05:40.527 INFO:teuthology.orchestra.run.smithi050.stdout:(10/152): ceph-selinux-19.0.0-3156.gd69f13d8.el 161 kB/s | 25 kB 00:00 2024-04-23T18:05:40.617 INFO:teuthology.orchestra.run.smithi177.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-23T18:05:40.711 INFO:teuthology.orchestra.run.smithi050.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 1.2 MB/s | 225 kB 00:00 2024-04-23T18:05:40.734 INFO:teuthology.orchestra.run.smithi177.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-04-23T18:05:40.768 INFO:teuthology.orchestra.run.smithi177.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 5.9 MB/s | 911 kB 00:00 2024-04-23T18:05:40.801 INFO:teuthology.orchestra.run.smithi177.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-04-23T18:05:40.803 INFO:teuthology.orchestra.run.smithi050.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 295 kB/s | 27 kB 00:00 2024-04-23T18:05:40.885 INFO:teuthology.orchestra.run.smithi177.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-04-23T18:05:40.903 INFO:teuthology.orchestra.run.smithi050.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 317 kB/s | 32 kB 00:00 2024-04-23T18:05:40.944 INFO:teuthology.orchestra.run.smithi177.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-23T18:05:41.010 INFO:teuthology.orchestra.run.smithi177.stdout:(73/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.7 MB/s | 4.4 MB 00:00 2024-04-23T18:05:41.064 INFO:teuthology.orchestra.run.smithi188.stdout:(10/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 4.5 MB/s | 11 MB 00:02 2024-04-23T18:05:41.086 INFO:teuthology.orchestra.run.smithi177.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 3.2 MB/s | 244 kB 00:00 2024-04-23T18:05:41.129 INFO:teuthology.orchestra.run.smithi050.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 3.2 MB/s | 747 kB 00:00 2024-04-23T18:05:41.147 INFO:teuthology.orchestra.run.smithi188.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.6 MB/s | 225 kB 00:00 2024-04-23T18:05:41.161 INFO:teuthology.orchestra.run.smithi177.stdout:(75/152): postgresql-server-13.14-1.el9.x86_64. 15 MB/s | 5.8 MB 00:00 2024-04-23T18:05:41.187 INFO:teuthology.orchestra.run.smithi177.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-23T18:05:41.197 INFO:teuthology.orchestra.run.smithi188.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 541 kB/s | 27 kB 00:00 2024-04-23T18:05:41.205 INFO:teuthology.orchestra.run.smithi050.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 2.1 MB/s | 163 kB 00:00 2024-04-23T18:05:41.212 INFO:teuthology.orchestra.run.smithi177.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 510 kB/s | 26 kB 00:00 2024-04-23T18:05:41.237 INFO:teuthology.orchestra.run.smithi177.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-23T18:05:41.250 INFO:teuthology.orchestra.run.smithi188.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 609 kB/s | 32 kB 00:00 2024-04-23T18:05:41.262 INFO:teuthology.orchestra.run.smithi177.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-23T18:05:41.279 INFO:teuthology.orchestra.run.smithi177.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 829 kB/s | 35 kB 00:00 2024-04-23T18:05:41.288 INFO:teuthology.orchestra.run.smithi050.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.5 MB/s | 126 kB 00:00 2024-04-23T18:05:41.363 INFO:teuthology.orchestra.run.smithi177.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-04-23T18:05:41.389 INFO:teuthology.orchestra.run.smithi177.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.2 MB/s | 250 kB 00:00 2024-04-23T18:05:41.422 INFO:teuthology.orchestra.run.smithi177.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-23T18:05:41.425 INFO:teuthology.orchestra.run.smithi188.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 4.2 MB/s | 747 kB 00:00 2024-04-23T18:05:41.430 INFO:teuthology.orchestra.run.smithi050.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 3.5 MB/s | 500 kB 00:00 2024-04-23T18:05:41.439 INFO:teuthology.orchestra.run.smithi177.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-23T18:05:41.484 INFO:teuthology.orchestra.run.smithi188.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 2.7 MB/s | 163 kB 00:00 2024-04-23T18:05:41.514 INFO:teuthology.orchestra.run.smithi177.stdout:(85/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-04-23T18:05:41.531 INFO:teuthology.orchestra.run.smithi177.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 388 kB/s | 42 kB 00:00 2024-04-23T18:05:41.542 INFO:teuthology.orchestra.run.smithi188.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 2.1 MB/s | 126 kB 00:00 2024-04-23T18:05:41.556 INFO:teuthology.orchestra.run.smithi177.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 2.2 MB/s | 269 kB 00:00 2024-04-23T18:05:41.582 INFO:teuthology.orchestra.run.smithi177.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-04-23T18:05:41.598 INFO:teuthology.orchestra.run.smithi177.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-23T18:05:41.624 INFO:teuthology.orchestra.run.smithi177.stdout:(90/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.9 MB/s | 279 kB 00:00 2024-04-23T18:05:41.641 INFO:teuthology.orchestra.run.smithi177.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 912 kB/s | 54 kB 00:00 2024-04-23T18:05:41.666 INFO:teuthology.orchestra.run.smithi177.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 992 kB/s | 42 kB 00:00 2024-04-23T18:05:41.683 INFO:teuthology.orchestra.run.smithi177.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-04-23T18:05:41.693 INFO:teuthology.orchestra.run.smithi188.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 3.3 MB/s | 500 kB 00:00 2024-04-23T18:05:41.717 INFO:teuthology.orchestra.run.smithi177.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-23T18:05:41.742 INFO:teuthology.orchestra.run.smithi177.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 614 kB/s | 36 kB 00:00 2024-04-23T18:05:41.759 INFO:teuthology.orchestra.run.smithi177.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-23T18:05:41.793 INFO:teuthology.orchestra.run.smithi177.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-23T18:05:41.818 INFO:teuthology.orchestra.run.smithi177.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 991 kB/s | 58 kB 00:00 2024-04-23T18:05:41.843 INFO:teuthology.orchestra.run.smithi177.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-23T18:05:41.868 INFO:teuthology.orchestra.run.smithi177.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-23T18:05:41.894 INFO:teuthology.orchestra.run.smithi177.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 894 kB/s | 45 kB 00:00 2024-04-23T18:05:42.022 INFO:teuthology.orchestra.run.smithi177.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 148 kB/s | 22 kB 00:00 2024-04-23T18:05:42.194 INFO:teuthology.orchestra.run.smithi177.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.8 MB/s | 863 kB 00:00 2024-04-23T18:05:42.678 INFO:teuthology.orchestra.run.smithi177.stdout:(104/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-04-23T18:05:42.688 INFO:teuthology.orchestra.run.smithi050.stdout:(18/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 4.3 MB/s | 5.4 MB 00:01 2024-04-23T18:05:42.772 INFO:teuthology.orchestra.run.smithi050.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 542 kB/s | 45 kB 00:00 2024-04-23T18:05:42.856 INFO:teuthology.orchestra.run.smithi050.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 1.6 MB/s | 135 kB 00:00 2024-04-23T18:05:42.923 INFO:teuthology.orchestra.run.smithi050.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 2.4 MB/s | 161 kB 00:00 2024-04-23T18:05:43.023 INFO:teuthology.orchestra.run.smithi050.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 3.1 MB/s | 320 kB 00:00 2024-04-23T18:05:43.084 INFO:teuthology.orchestra.run.smithi188.stdout:(18/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 3.9 MB/s | 5.4 MB 00:01 2024-04-23T18:05:43.123 INFO:teuthology.orchestra.run.smithi050.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 2.9 MB/s | 298 kB 00:00 2024-04-23T18:05:43.126 INFO:teuthology.orchestra.run.smithi188.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 1.1 MB/s | 45 kB 00:00 2024-04-23T18:05:43.137 INFO:teuthology.orchestra.run.smithi177.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 42 kB/s | 19 kB 00:00 2024-04-23T18:05:43.162 INFO:teuthology.orchestra.run.smithi177.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 319 kB/s | 309 kB 00:00 2024-04-23T18:05:43.182 INFO:teuthology.orchestra.run.smithi050.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.7 MB/s | 99 kB 00:00 2024-04-23T18:05:43.227 INFO:teuthology.orchestra.run.smithi188.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 1.3 MB/s | 135 kB 00:00 2024-04-23T18:05:43.249 INFO:teuthology.orchestra.run.smithi050.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.3 MB/s | 85 kB 00:00 2024-04-23T18:05:43.279 INFO:teuthology.orchestra.run.smithi177.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 435 kB/s | 548 kB 00:01 2024-04-23T18:05:43.286 INFO:teuthology.orchestra.run.smithi188.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 2.7 MB/s | 161 kB 00:00 2024-04-23T18:05:43.304 INFO:teuthology.orchestra.run.smithi177.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-04-23T18:05:43.379 INFO:teuthology.orchestra.run.smithi177.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-04-23T18:05:43.403 INFO:teuthology.orchestra.run.smithi188.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 2.7 MB/s | 320 kB 00:00 2024-04-23T18:05:43.539 INFO:teuthology.orchestra.run.smithi188.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 2.1 MB/s | 298 kB 00:00 2024-04-23T18:05:43.588 INFO:teuthology.orchestra.run.smithi177.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 237 kB/s | 67 kB 00:00 2024-04-23T18:05:43.603 INFO:teuthology.orchestra.run.smithi188.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.5 MB/s | 99 kB 00:00 2024-04-23T18:05:43.655 INFO:teuthology.orchestra.run.smithi177.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 549 kB/s | 151 kB 00:00 2024-04-23T18:05:43.670 INFO:teuthology.orchestra.run.smithi188.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.2 MB/s | 85 kB 00:00 2024-04-23T18:05:43.954 INFO:teuthology.orchestra.run.smithi188.stdout:(26/152): ceph-common-19.0.0-3156.gd69f13d8.el7 2.0 MB/s | 21 MB 00:10 2024-04-23T18:05:44.032 INFO:teuthology.orchestra.run.smithi050.stdout:(26/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 2.0 MB/s | 17 MB 00:08 2024-04-23T18:05:44.099 INFO:teuthology.orchestra.run.smithi050.stdout:(27/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 3.6 MB/s | 3.1 MB 00:00 2024-04-23T18:05:44.158 INFO:teuthology.orchestra.run.smithi050.stdout:(28/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 1.3 MB/s | 170 kB 00:00 2024-04-23T18:05:44.175 INFO:teuthology.orchestra.run.smithi050.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3156.g 455 kB/s | 34 kB 00:00 2024-04-23T18:05:44.259 INFO:teuthology.orchestra.run.smithi050.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.4 MB/s | 142 kB 00:00 2024-04-23T18:05:44.397 INFO:teuthology.orchestra.run.smithi177.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 739 kB/s | 548 kB 00:00 2024-04-23T18:05:44.413 INFO:teuthology.orchestra.run.smithi188.stdout:(27/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 11 MB/s | 49 MB 00:04 2024-04-23T18:05:44.438 INFO:teuthology.orchestra.run.smithi188.stdout:(28/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 353 kB/s | 170 kB 00:00 2024-04-23T18:05:44.463 INFO:teuthology.orchestra.run.smithi188.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3156.g 684 kB/s | 34 kB 00:00 2024-04-23T18:05:44.505 INFO:teuthology.orchestra.run.smithi177.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-04-23T18:05:44.530 INFO:teuthology.orchestra.run.smithi188.stdout:(30/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 3.6 MB/s | 3.1 MB 00:00 2024-04-23T18:05:44.564 INFO:teuthology.orchestra.run.smithi188.stdout:(31/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.1 MB/s | 142 kB 00:00 2024-04-23T18:05:44.614 INFO:teuthology.orchestra.run.smithi177.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-04-23T18:05:44.689 INFO:teuthology.orchestra.run.smithi188.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.9 MB/s | 249 kB 00:00 2024-04-23T18:05:44.714 INFO:teuthology.orchestra.run.smithi177.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-04-23T18:05:44.748 INFO:teuthology.orchestra.run.smithi177.stdout:(116/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 723 kB/s | 838 kB 00:01 2024-04-23T18:05:44.756 INFO:teuthology.orchestra.run.smithi188.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 742 kB/s | 49 kB 00:00 2024-04-23T18:05:44.775 INFO:teuthology.orchestra.run.smithi050.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 5.9 MB/s | 3.5 MB 00:00 2024-04-23T18:05:44.806 INFO:teuthology.orchestra.run.smithi188.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 330 kB/s | 16 kB 00:00 2024-04-23T18:05:44.867 INFO:teuthology.orchestra.run.smithi050.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 2.7 MB/s | 249 kB 00:00 2024-04-23T18:05:44.890 INFO:teuthology.orchestra.run.smithi188.stdout:(35/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 8.3 MB/s | 3.5 MB 00:00 2024-04-23T18:05:44.932 INFO:teuthology.orchestra.run.smithi188.stdout:(36/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 2.1 MB/s | 273 kB 00:00 2024-04-23T18:05:44.934 INFO:teuthology.orchestra.run.smithi050.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 740 kB/s | 49 kB 00:00 2024-04-23T18:05:44.973 INFO:teuthology.orchestra.run.smithi177.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 664 kB/s | 172 kB 00:00 2024-04-23T18:05:44.990 INFO:teuthology.orchestra.run.smithi188.stdout:(37/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 7.4 MB/s | 762 kB 00:00 2024-04-23T18:05:45.009 INFO:teuthology.orchestra.run.smithi050.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 219 kB/s | 16 kB 00:00 2024-04-23T18:05:45.016 INFO:teuthology.orchestra.run.smithi188.stdout:(38/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 218 kB/s | 18 kB 00:00 2024-04-23T18:05:45.085 INFO:teuthology.orchestra.run.smithi050.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 3.6 MB/s | 273 kB 00:00 2024-04-23T18:05:45.252 INFO:teuthology.orchestra.run.smithi050.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 4.5 MB/s | 762 kB 00:00 2024-04-23T18:05:45.257 INFO:teuthology.orchestra.run.smithi177.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 705 kB/s | 358 kB 00:00 2024-04-23T18:05:45.319 INFO:teuthology.orchestra.run.smithi050.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 273 kB/s | 18 kB 00:00 2024-04-23T18:05:45.365 INFO:teuthology.orchestra.run.smithi177.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 326 kB/s | 35 kB 00:00 2024-04-23T18:05:45.391 INFO:teuthology.orchestra.run.smithi177.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 727 kB/s | 303 kB 00:00 2024-04-23T18:05:45.453 INFO:teuthology.orchestra.run.smithi050.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 603 kB/s | 80 kB 00:00 2024-04-23T18:05:45.574 INFO:teuthology.orchestra.run.smithi188.stdout:(39/152): ceph-mgr-diskprediction-local-19.0.0- 7.1 MB/s | 7.4 MB 00:01 2024-04-23T18:05:45.603 INFO:teuthology.orchestra.run.smithi050.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 5.2 MB/s | 803 kB 00:00 2024-04-23T18:05:45.661 INFO:teuthology.orchestra.run.smithi050.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2024-04-23T18:05:45.716 INFO:teuthology.orchestra.run.smithi177.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 587 kB/s | 206 kB 00:00 2024-04-23T18:05:45.758 INFO:teuthology.orchestra.run.smithi188.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 105 kB/s | 80 kB 00:00 2024-04-23T18:05:45.895 INFO:teuthology.orchestra.run.smithi050.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-04-23T18:05:45.999 INFO:teuthology.orchestra.run.smithi177.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 510 kB/s | 144 kB 00:00 2024-04-23T18:05:46.046 INFO:teuthology.orchestra.run.smithi050.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 8.2 MB/s | 1.2 MB 00:00 2024-04-23T18:05:46.096 INFO:teuthology.orchestra.run.smithi050.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-23T18:05:46.108 INFO:teuthology.orchestra.run.smithi177.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-04-23T18:05:46.154 INFO:teuthology.orchestra.run.smithi050.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-23T18:05:46.205 INFO:teuthology.orchestra.run.smithi050.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-23T18:05:46.216 INFO:teuthology.orchestra.run.smithi177.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-04-23T18:05:46.297 INFO:teuthology.orchestra.run.smithi050.stdout:(46/152): ceph-mgr-diskprediction-local-19.0.0- 3.6 MB/s | 7.4 MB 00:02 2024-04-23T18:05:46.323 INFO:teuthology.orchestra.run.smithi050.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-04-23T18:05:46.325 INFO:teuthology.orchestra.run.smithi177.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-04-23T18:05:46.425 INFO:teuthology.orchestra.run.smithi188.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 230 kB/s | 195 kB 00:00 2024-04-23T18:05:46.434 INFO:teuthology.orchestra.run.smithi177.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-04-23T18:05:46.534 INFO:teuthology.orchestra.run.smithi177.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-23T18:05:46.650 INFO:teuthology.orchestra.run.smithi188.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 491 kB/s | 803 kB 00:01 2024-04-23T18:05:46.709 INFO:teuthology.orchestra.run.smithi188.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 266 kB/s | 253 kB 00:00 2024-04-23T18:05:47.417 INFO:teuthology.orchestra.run.smithi188.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 144 kB/s | 106 kB 00:00 2024-04-23T18:05:47.468 INFO:teuthology.orchestra.run.smithi188.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 178 kB/s | 135 kB 00:00 2024-04-23T18:05:47.606 INFO:teuthology.orchestra.run.smithi050.stdout:(48/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 6.9 MB/s | 49 MB 00:07 2024-04-23T18:05:47.851 INFO:teuthology.orchestra.run.smithi177.stdout:(128/152): python3-grpcio-1.46.7-10.el9.x86_64. 848 kB/s | 2.0 MB 00:02 2024-04-23T18:05:47.885 INFO:teuthology.orchestra.run.smithi177.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 774 kB/s | 1.0 MB 00:01 2024-04-23T18:05:47.923 INFO:teuthology.orchestra.run.smithi050.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 424 kB/s | 689 kB 00:01 2024-04-23T18:05:47.944 INFO:teuthology.orchestra.run.smithi177.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 503 kB/s | 46 kB 00:00 2024-04-23T18:05:48.069 INFO:teuthology.orchestra.run.smithi177.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 429 kB/s | 79 kB 00:00 2024-04-23T18:05:48.128 INFO:teuthology.orchestra.run.smithi177.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 314 kB/s | 58 kB 00:00 2024-04-23T18:05:48.201 INFO:teuthology.orchestra.run.smithi188.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 159 kB/s | 125 kB 00:00 2024-04-23T18:05:48.236 INFO:teuthology.orchestra.run.smithi177.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-23T18:05:48.273 INFO:teuthology.orchestra.run.smithi050.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 483 kB/s | 943 kB 00:01 2024-04-23T18:05:48.348 INFO:teuthology.orchestra.run.smithi050.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 144 kB/s | 107 kB 00:00 2024-04-23T18:05:48.351 INFO:teuthology.orchestra.run.smithi188.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 243 kB/s | 215 kB 00:00 2024-04-23T18:05:48.411 INFO:teuthology.orchestra.run.smithi177.stdout:(134/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 514 kB/s | 90 kB 00:00 2024-04-23T18:05:48.520 INFO:teuthology.orchestra.run.smithi177.stdout:(135/152): python3-pyperclip-1.8.2-2.el9.noarch 261 kB/s | 28 kB 00:00 2024-04-23T18:05:48.545 INFO:teuthology.orchestra.run.smithi177.stdout:(136/152): python3-pecan-1.4.2-3.el9.noarch.rpm 572 kB/s | 272 kB 00:00 2024-04-23T18:05:48.577 INFO:teuthology.orchestra.run.smithi188.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 584 kB/s | 1.2 MB 00:02 2024-04-23T18:05:48.588 INFO:teuthology.orchestra.run.smithi050.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 32 kB/s | 21 kB 00:00 2024-04-23T18:05:48.620 INFO:teuthology.orchestra.run.smithi177.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 307 kB/s | 31 kB 00:00 2024-04-23T18:05:48.679 INFO:teuthology.orchestra.run.smithi177.stdout:(138/152): libarrow-9.0.0-10.el9.x86_64.rpm 816 kB/s | 4.4 MB 00:05 2024-04-23T18:05:48.796 INFO:teuthology.orchestra.run.smithi177.stdout:(139/152): python3-tempora-5.0.0-2.el9.noarch.r 307 kB/s | 36 kB 00:00 2024-04-23T18:05:48.821 INFO:teuthology.orchestra.run.smithi177.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2024-04-23T18:05:48.880 INFO:teuthology.orchestra.run.smithi177.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 564 kB/s | 188 kB 00:00 2024-04-23T18:05:48.922 INFO:teuthology.orchestra.run.smithi177.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 406 kB/s | 51 kB 00:00 2024-04-23T18:05:49.036 INFO:teuthology.orchestra.run.smithi188.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 827 kB/s | 689 kB 00:00 2024-04-23T18:05:49.040 INFO:teuthology.orchestra.run.smithi050.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 47 kB/s | 33 kB 00:00 2024-04-23T18:05:49.114 INFO:teuthology.orchestra.run.smithi177.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 384 kB/s | 90 kB 00:00 2024-04-23T18:05:49.144 INFO:teuthology.orchestra.run.smithi188.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 199 kB/s | 21 kB 00:00 2024-04-23T18:05:49.194 INFO:teuthology.orchestra.run.smithi188.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 173 kB/s | 107 kB 00:00 2024-04-23T18:05:49.239 INFO:teuthology.orchestra.run.smithi177.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 160 kB/s | 20 kB 00:00 2024-04-23T18:05:49.256 INFO:teuthology.orchestra.run.smithi177.stdout:(145/152): python3-webob-1.8.7-6.el9.noarch.rpm 529 kB/s | 230 kB 00:00 2024-04-23T18:05:49.261 INFO:teuthology.orchestra.run.smithi188.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-04-23T18:05:49.286 INFO:teuthology.orchestra.run.smithi188.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 356 kB/s | 33 kB 00:00 2024-04-23T18:05:49.378 INFO:teuthology.orchestra.run.smithi188.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 198 kB/s | 18 kB 00:00 2024-04-23T18:05:49.441 INFO:teuthology.orchestra.run.smithi050.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 269 kB/s | 313 kB 00:01 2024-04-23T18:05:49.470 INFO:teuthology.orchestra.run.smithi188.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-04-23T18:05:49.529 INFO:teuthology.orchestra.run.smithi188.stdout:(56/152): bison-3.7.4-5.el9.x86_64.rpm 801 kB/s | 943 kB 00:01 2024-04-23T18:05:49.539 INFO:teuthology.orchestra.run.smithi177.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 639 kB/s | 192 kB 00:00 2024-04-23T18:05:49.581 INFO:teuthology.orchestra.run.smithi177.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 648 kB/s | 427 kB 00:00 2024-04-23T18:05:49.607 INFO:teuthology.orchestra.run.smithi050.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 32 kB/s | 18 kB 00:00 2024-04-23T18:05:49.637 INFO:teuthology.orchestra.run.smithi188.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-04-23T18:05:49.729 INFO:teuthology.orchestra.run.smithi188.stdout:(58/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.4 MB/s | 3.0 MB 00:00 2024-04-23T18:05:49.755 INFO:teuthology.orchestra.run.smithi188.stdout:(59/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 564 kB/s | 160 kB 00:00 2024-04-23T18:05:49.838 INFO:teuthology.orchestra.run.smithi188.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 582 kB 00:00 2024-04-23T18:05:49.864 INFO:teuthology.orchestra.run.smithi188.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 234 kB/s | 25 kB 00:00 2024-04-23T18:05:49.947 INFO:teuthology.orchestra.run.smithi188.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 437 kB/s | 47 kB 00:00 2024-04-23T18:05:49.983 INFO:teuthology.orchestra.run.smithi050.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 21 kB/s | 11 kB 00:00 2024-04-23T18:05:50.007 INFO:teuthology.orchestra.run.smithi177.stdout:(148/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 7.4 MB/s | 3.1 MB 00:00 2024-04-23T18:05:50.065 INFO:teuthology.orchestra.run.smithi188.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-04-23T18:05:50.089 INFO:teuthology.orchestra.run.smithi188.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-04-23T18:05:50.115 INFO:teuthology.orchestra.run.smithi177.stdout:(149/152): librados2-19.0.0-3156.gd69f13d8.el7. 5.9 MB/s | 3.4 MB 00:00 2024-04-23T18:05:50.202 INFO:teuthology.orchestra.run.smithi188.stdout:(65/152): librdkafka-1.6.1-102.el9.x86_64.rpm 1.1 MB/s | 662 kB 00:00 2024-04-23T18:05:50.232 INFO:teuthology.orchestra.run.smithi188.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 210 kB/s | 35 kB 00:00 2024-04-23T18:05:50.332 INFO:teuthology.orchestra.run.smithi177.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 4.2 MB/s | 1.3 MB 00:00 2024-04-23T18:05:50.425 INFO:teuthology.orchestra.run.smithi050.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 196 kB/s | 160 kB 00:00 2024-04-23T18:05:50.532 INFO:teuthology.orchestra.run.smithi188.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 11 MB/s | 4.7 MB 00:00 2024-04-23T18:05:50.574 INFO:teuthology.orchestra.run.smithi188.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 4.6 MB/s | 1.6 MB 00:00 2024-04-23T18:05:50.658 INFO:teuthology.orchestra.run.smithi188.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 7.1 MB/s | 911 kB 00:00 2024-04-23T18:05:50.683 INFO:teuthology.orchestra.run.smithi188.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-23T18:05:50.750 INFO:teuthology.orchestra.run.smithi050.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 59 kB/s | 45 kB 00:00 2024-04-23T18:05:50.817 INFO:teuthology.orchestra.run.smithi188.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 7.6 MB/s | 1.0 MB 00:00 2024-04-23T18:05:50.891 INFO:teuthology.orchestra.run.smithi177.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 3.1 MB/s | 2.4 MB 00:00 2024-04-23T18:05:50.917 INFO:teuthology.orchestra.run.smithi188.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 874 kB/s | 87 kB 00:00 2024-04-23T18:05:51.332 INFO:teuthology.orchestra.run.smithi177.stdout:(152/152): thrift-0.15.0-2.el9.x86_64.rpm 780 kB/s | 1.6 MB 00:02 2024-04-23T18:05:51.339 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:05:51.339 INFO:teuthology.orchestra.run.smithi177.stdout:Total 11 MB/s | 222 MB 00:20 2024-04-23T18:05:51.983 INFO:teuthology.orchestra.run.smithi050.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 425 kB/s | 662 kB 00:01 2024-04-23T18:05:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 411 kB/s | 582 kB 00:01 2024-04-23T18:05:52.467 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-04-23T18:05:52.613 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-04-23T18:05:52.613 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-04-23T18:05:52.684 INFO:teuthology.orchestra.run.smithi050.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 36 kB/s | 25 kB 00:00 2024-04-23T18:05:52.825 INFO:teuthology.orchestra.run.smithi188.stdout:(73/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 1.7 MB/s | 4.4 MB 00:02 2024-04-23T18:05:52.859 INFO:teuthology.orchestra.run.smithi050.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 72 kB/s | 47 kB 00:00 2024-04-23T18:05:52.942 INFO:teuthology.orchestra.run.smithi188.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.0 MB/s | 244 kB 00:00 2024-04-23T18:05:53.051 INFO:teuthology.orchestra.run.smithi188.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-04-23T18:05:53.109 INFO:teuthology.orchestra.run.smithi050.stdout:(63/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 681 kB/s | 3.0 MB 00:04 2024-04-23T18:05:53.151 INFO:teuthology.orchestra.run.smithi188.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-04-23T18:05:53.251 INFO:teuthology.orchestra.run.smithi188.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 192 kB/s | 19 kB 00:00 2024-04-23T18:05:53.360 INFO:teuthology.orchestra.run.smithi188.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-04-23T18:05:53.468 INFO:teuthology.orchestra.run.smithi188.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 321 kB/s | 35 kB 00:00 2024-04-23T18:05:53.660 INFO:teuthology.orchestra.run.smithi050.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 300 kB/s | 292 kB 00:00 2024-04-23T18:05:53.718 INFO:teuthology.orchestra.run.smithi050.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 58 kB/s | 35 kB 00:00 2024-04-23T18:05:53.752 INFO:teuthology.orchestra.run.smithi050.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 211 kB/s | 188 kB 00:00 2024-04-23T18:05:54.379 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-04-23T18:05:54.379 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-04-23T18:05:55.127 INFO:teuthology.orchestra.run.smithi188.stdout:(80/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.1 MB/s | 5.1 MB 00:01 2024-04-23T18:05:55.235 INFO:teuthology.orchestra.run.smithi188.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-23T18:05:55.353 INFO:teuthology.orchestra.run.smithi188.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 222 kB 00:00 2024-04-23T18:05:55.453 INFO:teuthology.orchestra.run.smithi188.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 772 kB/s | 77 kB 00:00 2024-04-23T18:05:55.570 INFO:teuthology.orchestra.run.smithi188.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 362 kB/s | 42 kB 00:00 2024-04-23T18:05:55.679 INFO:teuthology.orchestra.run.smithi188.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 2.4 MB/s | 269 kB 00:00 2024-04-23T18:05:55.787 INFO:teuthology.orchestra.run.smithi188.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-04-23T18:05:55.904 INFO:teuthology.orchestra.run.smithi188.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-04-23T18:05:56.004 INFO:teuthology.orchestra.run.smithi188.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 510 kB/s | 51 kB 00:00 2024-04-23T18:05:56.104 INFO:teuthology.orchestra.run.smithi188.stdout:(89/152): python3-requests-oauthlib-1.3.0-12.el 538 kB/s | 54 kB 00:00 2024-04-23T18:05:56.352 INFO:teuthology.orchestra.run.smithi050.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 625 kB/s | 1.6 MB 00:02 2024-04-23T18:05:56.388 INFO:teuthology.orchestra.run.smithi188.stdout:(90/152): python3-babel-2.9.1-2.el9.noarch.rpm 1.1 MB/s | 6.0 MB 00:05 2024-04-23T18:05:56.480 INFO:teuthology.orchestra.run.smithi188.stdout:(91/152): postgresql-server-13.14-1.el9.x86_64. 1.0 MB/s | 5.8 MB 00:05 2024-04-23T18:05:56.506 INFO:teuthology.orchestra.run.smithi188.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 356 kB/s | 42 kB 00:00 2024-04-23T18:05:56.581 INFO:teuthology.orchestra.run.smithi188.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 441 kB/s | 44 kB 00:00 2024-04-23T18:05:56.706 INFO:teuthology.orchestra.run.smithi188.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 287 kB/s | 36 kB 00:00 2024-04-23T18:05:56.815 INFO:teuthology.orchestra.run.smithi188.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 438 kB/s | 47 kB 00:00 2024-04-23T18:05:56.890 INFO:teuthology.orchestra.run.smithi188.stdout:(96/152): qatlib-24.02.0-1.el9.x86_64.rpm 574 kB/s | 221 kB 00:00 2024-04-23T18:05:56.999 INFO:teuthology.orchestra.run.smithi188.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 534 kB/s | 58 kB 00:00 2024-04-23T18:05:57.182 INFO:teuthology.orchestra.run.smithi188.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 512 kB/s | 94 kB 00:00 2024-04-23T18:05:57.308 INFO:teuthology.orchestra.run.smithi188.stdout:(99/152): socat-1.7.4.1-5.el9.x86_64.rpm 619 kB/s | 305 kB 00:00 2024-04-23T18:05:57.366 INFO:teuthology.orchestra.run.smithi188.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 347 kB/s | 64 kB 00:00 2024-04-23T18:05:57.408 INFO:teuthology.orchestra.run.smithi188.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 447 kB/s | 45 kB 00:00 2024-04-23T18:05:57.500 INFO:teuthology.orchestra.run.smithi188.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-04-23T18:05:57.734 INFO:teuthology.orchestra.run.smithi188.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.3 MB/s | 548 kB 00:00 2024-04-23T18:05:57.784 INFO:teuthology.orchestra.run.smithi188.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 6.1 MB/s | 309 kB 00:00 2024-04-23T18:05:57.809 INFO:teuthology.orchestra.run.smithi188.stdout:(105/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-04-23T18:05:57.834 INFO:teuthology.orchestra.run.smithi188.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 387 kB/s | 19 kB 00:00 2024-04-23T18:05:57.902 INFO:teuthology.orchestra.run.smithi188.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 659 kB/s | 26 kB 00:00 2024-04-23T18:05:57.952 INFO:teuthology.orchestra.run.smithi188.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 991 kB/s | 49 kB 00:00 2024-04-23T18:05:57.994 INFO:teuthology.orchestra.run.smithi188.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-23T18:05:58.044 INFO:teuthology.orchestra.run.smithi188.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-04-23T18:05:58.086 INFO:teuthology.orchestra.run.smithi050.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 526 kB/s | 911 kB 00:01 2024-04-23T18:05:58.136 INFO:teuthology.orchestra.run.smithi188.stdout:(111/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-04-23T18:05:58.237 INFO:teuthology.orchestra.run.smithi188.stdout:(112/152): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-23T18:05:58.270 INFO:teuthology.orchestra.run.smithi188.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-04-23T18:05:58.295 INFO:teuthology.orchestra.run.smithi188.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 741 kB/s | 43 kB 00:00 2024-04-23T18:05:58.321 INFO:teuthology.orchestra.run.smithi188.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-04-23T18:05:58.346 INFO:teuthology.orchestra.run.smithi188.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-04-23T18:05:58.371 INFO:teuthology.orchestra.run.smithi188.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-04-23T18:05:58.397 INFO:teuthology.orchestra.run.smithi188.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 6.9 MB/s | 358 kB 00:00 2024-04-23T18:05:58.431 INFO:teuthology.orchestra.run.smithi188.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.0 MB/s | 303 kB 00:00 2024-04-23T18:05:58.456 INFO:teuthology.orchestra.run.smithi188.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 603 kB/s | 35 kB 00:00 2024-04-23T18:05:58.481 INFO:teuthology.orchestra.run.smithi188.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-04-23T18:05:58.548 INFO:teuthology.orchestra.run.smithi188.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-04-23T18:05:58.598 INFO:teuthology.orchestra.run.smithi188.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-23T18:05:58.624 INFO:teuthology.orchestra.run.smithi188.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-04-23T18:05:58.657 INFO:teuthology.orchestra.run.smithi188.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-04-23T18:05:58.683 INFO:teuthology.orchestra.run.smithi188.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 394 kB/s | 23 kB 00:00 2024-04-23T18:05:58.716 INFO:teuthology.orchestra.run.smithi188.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 331 kB/s | 19 kB 00:00 2024-04-23T18:05:58.741 INFO:teuthology.orchestra.run.smithi188.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 337 kB/s | 20 kB 00:00 2024-04-23T18:05:58.792 INFO:teuthology.orchestra.run.smithi188.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 14 MB/s | 1.0 MB 00:00 2024-04-23T18:05:58.817 INFO:teuthology.orchestra.run.smithi188.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 614 kB/s | 46 kB 00:00 2024-04-23T18:05:58.843 INFO:teuthology.orchestra.run.smithi188.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-23T18:05:58.868 INFO:teuthology.orchestra.run.smithi188.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-23T18:05:58.893 INFO:teuthology.orchestra.run.smithi188.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-23T18:05:58.918 INFO:teuthology.orchestra.run.smithi188.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-04-23T18:05:58.919 INFO:teuthology.orchestra.run.smithi050.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 160 kB/s | 133 kB 00:00 2024-04-23T18:05:58.944 INFO:teuthology.orchestra.run.smithi188.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-04-23T18:05:58.969 INFO:teuthology.orchestra.run.smithi188.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-04-23T18:05:58.994 INFO:teuthology.orchestra.run.smithi188.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-04-23T18:05:59.028 INFO:teuthology.orchestra.run.smithi188.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-04-23T18:05:59.053 INFO:teuthology.orchestra.run.smithi188.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-23T18:05:59.079 INFO:teuthology.orchestra.run.smithi188.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 715 kB/s | 36 kB 00:00 2024-04-23T18:05:59.104 INFO:teuthology.orchestra.run.smithi188.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-23T18:05:59.262 INFO:teuthology.orchestra.run.smithi188.stdout:(142/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.1 MB/s | 16 MB 00:03 2024-04-23T18:05:59.287 INFO:teuthology.orchestra.run.smithi188.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 488 kB/s | 90 kB 00:00 2024-04-23T18:05:59.321 INFO:teuthology.orchestra.run.smithi188.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 947 kB/s | 230 kB 00:00 2024-04-23T18:05:59.372 INFO:teuthology.orchestra.run.smithi188.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-04-23T18:05:59.406 INFO:teuthology.orchestra.run.smithi188.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.3 MB/s | 192 kB 00:00 2024-04-23T18:05:59.456 INFO:teuthology.orchestra.run.smithi188.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-04-23T18:05:59.598 INFO:teuthology.orchestra.run.smithi188.stdout:(148/152): python3-werkzeug-2.0.3-3.el9.1.noarc 1.2 MB/s | 427 kB 00:00 2024-04-23T18:05:59.748 INFO:teuthology.orchestra.run.smithi188.stdout:(149/152): librados2-19.0.0-3156.gd69f13d8.el7. 9.9 MB/s | 3.4 MB 00:00 2024-04-23T18:05:59.948 INFO:teuthology.orchestra.run.smithi188.stdout:(150/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 6.4 MB/s | 3.1 MB 00:00 2024-04-23T18:05:59.961 INFO:teuthology.orchestra.run.smithi050.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 726 kB/s | 4.4 MB 00:06 2024-04-23T18:06:00.286 INFO:teuthology.orchestra.run.smithi050.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 732 kB/s | 4.7 MB 00:06 2024-04-23T18:06:00.492 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-04-23T18:06:00.580 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T18:06:00.626 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T18:06:00.752 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T18:06:00.914 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T18:06:00.940 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T18:06:00.978 INFO:teuthology.orchestra.run.smithi050.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 127 kB/s | 87 kB 00:00 2024-04-23T18:06:00.983 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T18:06:01.310 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T18:06:01.424 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:06:01.545 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:06:01.575 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:06:01.665 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:06:01.696 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:06:01.844 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T18:06:01.893 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T18:06:01.903 INFO:teuthology.orchestra.run.smithi050.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 532 kB/s | 1.0 MB 00:01 2024-04-23T18:06:01.921 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:06:01.931 INFO:teuthology.orchestra.run.smithi188.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 592 kB/s | 1.3 MB 00:02 2024-04-23T18:06:02.025 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:06:02.047 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:06:02.109 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:06:02.131 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:06:02.155 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:06:02.348 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:06:02.381 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:06:02.491 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T18:06:02.589 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T18:06:02.622 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T18:06:02.657 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T18:06:02.731 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T18:06:02.773 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T18:06:02.817 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T18:06:02.837 INFO:teuthology.orchestra.run.smithi050.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 262 kB/s | 244 kB 00:00 2024-04-23T18:06:02.849 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T18:06:02.877 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T18:06:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T18:06:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T18:06:03.038 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T18:06:03.082 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T18:06:03.189 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T18:06:03.226 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-23T18:06:03.269 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-23T18:06:03.298 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-23T18:06:03.360 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T18:06:03.413 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T18:06:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T18:06:03.622 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T18:06:03.674 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T18:06:03.756 INFO:teuthology.orchestra.run.smithi188.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 611 kB/s | 2.4 MB 00:04 2024-04-23T18:06:03.763 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:06:03.763 INFO:teuthology.orchestra.run.smithi188.stdout:Total 7.0 MB/s | 222 MB 00:31 2024-04-23T18:06:04.004 INFO:teuthology.orchestra.run.smithi050.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 213 kB/s | 248 kB 00:01 2024-04-23T18:06:04.193 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T18:06:04.277 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T18:06:04.307 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T18:06:04.349 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T18:06:04.388 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-23T18:06:04.429 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T18:06:04.582 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T18:06:04.641 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T18:06:04.694 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T18:06:04.720 INFO:teuthology.orchestra.run.smithi050.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 36 kB/s | 26 kB 00:00 2024-04-23T18:06:04.767 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T18:06:04.799 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T18:06:04.830 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-23T18:06:04.835 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T18:06:04.882 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T18:06:04.918 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T18:06:04.961 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T18:06:04.973 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-23T18:06:04.973 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-23T18:06:05.004 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T18:06:05.139 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T18:06:05.193 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T18:06:05.223 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T18:06:05.264 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T18:06:05.358 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T18:06:05.389 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T18:06:05.429 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T18:06:05.437 INFO:teuthology.orchestra.run.smithi050.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 27 kB/s | 19 kB 00:00 2024-04-23T18:06:05.455 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T18:06:05.480 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T18:06:05.557 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T18:06:05.681 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T18:06:05.712 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T18:06:05.732 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T18:06:05.766 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T18:06:05.806 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T18:06:05.837 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T18:06:05.867 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T18:06:05.910 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T18:06:05.943 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T18:06:05.985 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T18:06:06.019 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T18:06:06.106 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T18:06:06.150 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T18:06:06.190 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T18:06:06.236 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T18:06:06.335 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T18:06:06.390 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T18:06:06.571 INFO:teuthology.orchestra.run.smithi050.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 152 kB/s | 172 kB 00:01 2024-04-23T18:06:06.757 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-23T18:06:06.758 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-23T18:06:07.107 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T18:06:07.155 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T18:06:07.189 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T18:06:07.287 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T18:06:07.321 INFO:teuthology.orchestra.run.smithi050.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 46 kB/s | 35 kB 00:00 2024-04-23T18:06:07.355 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T18:06:07.442 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T18:06:07.474 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T18:06:07.652 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T18:06:07.688 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T18:06:07.767 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T18:06:07.863 INFO:teuthology.orchestra.run.smithi050.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 662 kB/s | 5.8 MB 00:08 2024-04-23T18:06:08.405 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T18:06:08.520 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T18:06:08.921 INFO:teuthology.orchestra.run.smithi050.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 237 kB/s | 250 kB 00:01 2024-04-23T18:06:09.756 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T18:06:09.780 INFO:teuthology.orchestra.run.smithi050.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 259 kB/s | 222 kB 00:00 2024-04-23T18:06:09.827 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T18:06:09.854 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T18:06:09.887 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T18:06:10.139 INFO:teuthology.orchestra.run.smithi050.stdout:(83/152): python3-babel-2.9.1-2.el9.noarch.rpm 667 kB/s | 6.0 MB 00:09 2024-04-23T18:06:10.200 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T18:06:10.234 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:06:10.324 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:06:10.355 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-23T18:06:10.391 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T18:06:10.497 INFO:teuthology.orchestra.run.smithi050.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 108 kB/s | 77 kB 00:00 2024-04-23T18:06:10.798 INFO:teuthology.orchestra.run.smithi050.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 64 kB/s | 42 kB 00:00 2024-04-23T18:06:10.886 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T18:06:10.928 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:06:11.000 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:06:11.030 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-23T18:06:11.489 INFO:teuthology.orchestra.run.smithi050.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 272 kB/s | 269 kB 00:00 2024-04-23T18:06:11.707 INFO:teuthology.orchestra.run.smithi050.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 175 kB/s | 159 kB 00:00 2024-04-23T18:06:12.365 INFO:teuthology.orchestra.run.smithi050.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 77 kB/s | 51 kB 00:00 2024-04-23T18:06:12.473 INFO:teuthology.orchestra.run.smithi050.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 283 kB/s | 279 kB 00:00 2024-04-23T18:06:12.698 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-23T18:06:12.792 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T18:06:12.831 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T18:06:12.884 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T18:06:12.927 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T18:06:12.952 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T18:06:12.995 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T18:06:13.065 INFO:teuthology.orchestra.run.smithi050.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 77 kB/s | 54 kB 00:00 2024-04-23T18:06:13.287 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:06:13.314 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T18:06:13.326 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:06:13.428 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:06:13.567 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:06:13.629 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:06:13.721 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:06:13.759 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:06:13.791 INFO:teuthology.orchestra.run.smithi050.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 57 kB/s | 42 kB 00:00 2024-04-23T18:06:13.873 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:06:13.910 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T18:06:13.922 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:06:13.964 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T18:06:13.992 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:06:14.010 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:06:14.011 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T18:06:14.011 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T18:06:14.011 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:14.040 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:06:14.095 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:06:14.118 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:06:14.181 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:06:14.203 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:06:14.221 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:06:14.437 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:06:14.466 INFO:teuthology.orchestra.run.smithi050.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 65 kB/s | 44 kB 00:00 2024-04-23T18:06:14.477 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:06:14.586 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T18:06:14.680 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T18:06:14.710 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T18:06:14.744 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T18:06:14.809 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T18:06:14.844 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T18:06:14.887 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T18:06:14.920 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T18:06:14.948 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T18:06:14.991 INFO:teuthology.orchestra.run.smithi050.stdout:(93/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 681 kB/s | 5.1 MB 00:07 2024-04-23T18:06:15.016 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T18:06:15.079 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T18:06:15.109 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T18:06:15.145 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T18:06:15.221 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T18:06:15.256 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-23T18:06:15.298 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-23T18:06:15.328 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-23T18:06:15.383 INFO:teuthology.orchestra.run.smithi050.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 241 kB/s | 221 kB 00:00 2024-04-23T18:06:15.399 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T18:06:15.451 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T18:06:15.534 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T18:06:15.608 INFO:teuthology.orchestra.run.smithi050.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 58 kB/s | 36 kB 00:00 2024-04-23T18:06:15.678 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T18:06:15.729 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T18:06:16.025 INFO:teuthology.orchestra.run.smithi050.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 74 kB/s | 47 kB 00:00 2024-04-23T18:06:16.263 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T18:06:16.350 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T18:06:16.379 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T18:06:16.421 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T18:06:16.459 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-23T18:06:16.501 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T18:06:16.601 INFO:teuthology.orchestra.run.smithi050.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 307 kB/s | 305 kB 00:00 2024-04-23T18:06:16.660 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T18:06:16.701 INFO:teuthology.orchestra.run.smithi050.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 86 kB/s | 58 kB 00:00 2024-04-23T18:06:16.721 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T18:06:16.776 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T18:06:16.874 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T18:06:16.913 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T18:06:16.950 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T18:06:16.997 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T18:06:17.033 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T18:06:17.076 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T18:06:17.119 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T18:06:17.254 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T18:06:17.334 INFO:teuthology.orchestra.run.smithi050.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 128 kB/s | 94 kB 00:00 2024-04-23T18:06:17.340 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T18:06:17.379 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T18:06:17.401 INFO:teuthology.orchestra.run.smithi050.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 91 kB/s | 64 kB 00:00 2024-04-23T18:06:17.429 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T18:06:17.510 INFO:teuthology.orchestra.run.smithi050.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 208 kB/s | 22 kB 00:00 2024-04-23T18:06:17.524 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T18:06:17.562 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T18:06:17.603 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T18:06:17.628 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T18:06:17.653 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T18:06:17.685 INFO:teuthology.orchestra.run.smithi050.stdout:(102/152): protobuf-compiler-3.14.0-13.el9.x86_ 4.8 MB/s | 863 kB 00:00 2024-04-23T18:06:17.728 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T18:06:17.864 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T18:06:17.902 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T18:06:17.939 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T18:06:17.973 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T18:06:18.029 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T18:06:18.044 INFO:teuthology.orchestra.run.smithi050.stdout:(103/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 63 kB/s | 45 kB 00:00 2024-04-23T18:06:18.060 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T18:06:18.090 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T18:06:18.125 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T18:06:18.158 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T18:06:18.200 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T18:06:18.202 INFO:teuthology.orchestra.run.smithi050.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-04-23T18:06:18.228 INFO:teuthology.orchestra.run.smithi050.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-04-23T18:06:18.234 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T18:06:18.270 INFO:teuthology.orchestra.run.smithi050.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-04-23T18:06:18.322 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T18:06:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T18:06:18.413 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T18:06:18.461 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T18:06:18.478 INFO:teuthology.orchestra.run.smithi050.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 125 kB/s | 26 kB 00:00 2024-04-23T18:06:18.537 INFO:teuthology.orchestra.run.smithi050.stdout:(108/152): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-23T18:06:18.559 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T18:06:18.562 INFO:teuthology.orchestra.run.smithi050.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 587 kB/s | 49 kB 00:00 2024-04-23T18:06:18.606 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T18:06:18.621 INFO:teuthology.orchestra.run.smithi050.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 806 kB/s | 67 kB 00:00 2024-04-23T18:06:18.654 INFO:teuthology.orchestra.run.smithi050.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-04-23T18:06:18.721 INFO:teuthology.orchestra.run.smithi050.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-04-23T18:06:18.755 INFO:teuthology.orchestra.run.smithi050.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-04-23T18:06:18.797 INFO:teuthology.orchestra.run.smithi050.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 578 kB/s | 43 kB 00:00 2024-04-23T18:06:18.831 INFO:teuthology.orchestra.run.smithi050.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-04-23T18:06:18.872 INFO:teuthology.orchestra.run.smithi050.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-04-23T18:06:18.914 INFO:teuthology.orchestra.run.smithi050.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-23T18:06:18.956 INFO:teuthology.orchestra.run.smithi050.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-23T18:06:19.006 INFO:teuthology.orchestra.run.smithi050.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-23T18:06:19.040 INFO:teuthology.orchestra.run.smithi050.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 422 kB/s | 35 kB 00:00 2024-04-23T18:06:19.099 INFO:teuthology.orchestra.run.smithi050.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.2 MB/s | 206 kB 00:00 2024-04-23T18:06:19.190 INFO:teuthology.orchestra.run.smithi050.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-23T18:06:19.216 INFO:teuthology.orchestra.run.smithi050.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-04-23T18:06:19.283 INFO:teuthology.orchestra.run.smithi050.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-04-23T18:06:19.308 INFO:teuthology.orchestra.run.smithi050.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 191 kB/s | 18 kB 00:00 2024-04-23T18:06:19.346 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T18:06:19.376 INFO:teuthology.orchestra.run.smithi050.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-04-23T18:06:19.395 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T18:06:19.409 INFO:teuthology.orchestra.run.smithi050.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-04-23T18:06:19.429 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T18:06:19.476 INFO:teuthology.orchestra.run.smithi050.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-23T18:06:19.530 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T18:06:19.543 INFO:teuthology.orchestra.run.smithi050.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-04-23T18:06:19.585 INFO:teuthology.orchestra.run.smithi050.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-04-23T18:06:19.587 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T18:06:19.618 INFO:teuthology.orchestra.run.smithi050.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-04-23T18:06:19.669 INFO:teuthology.orchestra.run.smithi050.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 692 kB/s | 58 kB 00:00 2024-04-23T18:06:19.676 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T18:06:19.702 INFO:teuthology.orchestra.run.smithi050.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-23T18:06:19.744 INFO:teuthology.orchestra.run.smithi050.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-04-23T18:06:19.756 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T18:06:19.778 INFO:teuthology.orchestra.run.smithi050.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-23T18:06:19.828 INFO:teuthology.orchestra.run.smithi050.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-04-23T18:06:19.853 INFO:teuthology.orchestra.run.smithi050.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-04-23T18:06:19.920 INFO:teuthology.orchestra.run.smithi050.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-04-23T18:06:19.936 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T18:06:19.937 INFO:teuthology.orchestra.run.smithi050.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 707 kB/s | 59 kB 00:00 2024-04-23T18:06:19.971 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T18:06:20.004 INFO:teuthology.orchestra.run.smithi050.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 428 kB/s | 36 kB 00:00 2024-04-23T18:06:20.021 INFO:teuthology.orchestra.run.smithi050.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 611 kB/s | 51 kB 00:00 2024-04-23T18:06:20.048 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T18:06:20.113 INFO:teuthology.orchestra.run.smithi050.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-23T18:06:20.139 INFO:teuthology.orchestra.run.smithi050.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 766 kB/s | 90 kB 00:00 2024-04-23T18:06:20.222 INFO:teuthology.orchestra.run.smithi050.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-23T18:06:20.239 INFO:teuthology.orchestra.run.smithi050.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-04-23T18:06:20.315 INFO:teuthology.orchestra.run.smithi050.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-23T18:06:20.373 INFO:teuthology.orchestra.run.smithi050.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-23T18:06:20.696 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T18:06:20.710 INFO:teuthology.orchestra.run.smithi050.stdout:(148/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 9.4 MB/s | 3.1 MB 00:00 2024-04-23T18:06:20.812 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T18:06:21.198 INFO:teuthology.orchestra.run.smithi050.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 2.7 MB/s | 1.3 MB 00:00 2024-04-23T18:06:21.632 INFO:teuthology.orchestra.run.smithi050.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 5.5 MB/s | 2.4 MB 00:00 2024-04-23T18:06:21.782 INFO:teuthology.orchestra.run.smithi050.stdout:(151/152): librados2-19.0.0-3156.gd69f13d8.el7. 2.3 MB/s | 3.4 MB 00:01 2024-04-23T18:06:22.055 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T18:06:22.159 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T18:06:22.188 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T18:06:22.220 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T18:06:22.534 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T18:06:22.575 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:06:22.666 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:06:22.696 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-23T18:06:22.732 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T18:06:23.221 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T18:06:23.253 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:06:23.324 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:06:23.346 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-23T18:06:25.429 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:06:25.429 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-04-23T18:06:25.429 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-04-23T18:06:25.429 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-04-23T18:06:25.429 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-04-23T18:06:25.429 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-04-23T18:06:25.430 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-04-23T18:06:25.430 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-04-23T18:06:25.430 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:25.629 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:06:25.645 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:06:25.645 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:25.645 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T18:06:25.645 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:25.657 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:06:25.701 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:06:26.243 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:06:26.288 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:06:26.387 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:06:26.387 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T18:06:26.387 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T18:06:26.387 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:06:26.422 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:06:27.139 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:06:27.165 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:06:27.165 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:27.165 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T18:06:27.165 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:06:27.166 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:06:27.166 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:27.390 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:06:27.413 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:06:27.413 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:27.413 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T18:06:27.413 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:06:27.413 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:06:27.413 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:27.857 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:06:27.880 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:06:27.880 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:27.881 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T18:06:27.881 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:06:27.881 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:06:27.881 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:27.979 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T18:06:28.027 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T18:06:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T18:06:28.268 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T18:06:28.308 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T18:06:28.386 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T18:06:28.474 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:06:28.614 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:06:28.658 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T18:06:28.690 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T18:06:28.718 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T18:06:29.162 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:06:29.192 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:06:29.232 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-23T18:06:29.294 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-23T18:06:29.323 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:06:30.363 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:06:30.425 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:06:31.387 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:06:31.417 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:06:31.518 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:06:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-23T18:06:31.719 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:06:31.743 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:06:31.743 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:31.743 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T18:06:31.743 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:06:31.743 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:06:31.743 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:31.786 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:06:31.798 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:06:31.895 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-23T18:06:31.943 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-23T18:06:32.865 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T18:06:32.897 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:06:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:06:32.920 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:32.920 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T18:06:32.920 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:06:32.920 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:06:32.920 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:32.964 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:06:32.989 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:06:32.989 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:32.989 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T18:06:32.989 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:33.032 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:06:33.054 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:06:33.055 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:33.055 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T18:06:33.055 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:06:33.055 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:06:33.055 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:33.357 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:06:33.380 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:06:33.380 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:33.380 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T18:06:33.381 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:06:33.381 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:06:33.381 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:35.831 INFO:teuthology.orchestra.run.smithi050.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 712 kB/s | 16 MB 00:23 2024-04-23T18:06:35.837 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:06:35.837 INFO:teuthology.orchestra.run.smithi050.stdout:Total 3.5 MB/s | 222 MB 01:03 2024-04-23T18:06:36.913 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-04-23T18:06:37.073 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-04-23T18:06:37.073 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-04-23T18:06:37.934 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:06:37.935 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2024-04-23T18:06:37.935 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2024-04-23T18:06:37.935 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2024-04-23T18:06:37.935 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2024-04-23T18:06:37.935 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2024-04-23T18:06:37.935 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /root 2024-04-23T18:06:37.935 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2024-04-23T18:06:37.935 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:06:37.978 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-23T18:06:38.019 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-23T18:06:38.060 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-23T18:06:38.092 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T18:06:38.119 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:06:38.119 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T18:06:38.133 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:06:38.134 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:38.134 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T18:06:38.134 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:06:38.302 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-23T18:06:38.349 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T18:06:38.379 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-23T18:06:38.954 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-04-23T18:06:38.954 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-04-23T18:06:38.990 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:06:39.027 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:06:39.047 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:06:39.597 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:06:39.623 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:06:39.623 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:39.623 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T18:06:39.623 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:06:39.623 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:06:39.623 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:06:39.847 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:06:39.870 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:06:39.870 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:39.870 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T18:06:39.870 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:06:39.870 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:06:39.870 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:06:39.873 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-23T18:06:39.965 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T18:06:40.005 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T18:06:40.086 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T18:06:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T18:06:40.119 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:06:40.203 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:06:40.236 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:06:40.236 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:06:40.315 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:06:40.326 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:06:40.337 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:06:40.337 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:40.337 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T18:06:40.337 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:06:40.337 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:06:40.337 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:06:40.392 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:06:40.445 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T18:06:40.493 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T18:06:40.549 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T18:06:40.734 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T18:06:40.791 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T18:06:40.878 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T18:06:40.974 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:06:41.114 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:06:41.158 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T18:06:41.190 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T18:06:41.226 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T18:06:41.661 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:06:41.691 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:06:41.724 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-23T18:06:41.795 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-23T18:06:41.831 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:06:42.872 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:06:42.908 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:06:43.869 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:06:43.919 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:06:44.020 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:06:44.181 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-23T18:06:44.211 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:06:44.233 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:06:44.234 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:44.234 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T18:06:44.234 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:06:44.234 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:06:44.234 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:06:44.278 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:06:44.289 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:06:44.387 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-23T18:06:44.443 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-23T18:06:45.129 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-04-23T18:06:45.210 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T18:06:45.249 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T18:06:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T18:06:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T18:06:45.369 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T18:06:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T18:06:45.405 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:06:45.421 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T18:06:45.428 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:06:45.428 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:45.428 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T18:06:45.428 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:06:45.428 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:06:45.428 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:06:45.473 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:06:45.495 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:06:45.495 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:45.495 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T18:06:45.496 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:06:45.541 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:06:45.562 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:06:45.563 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:45.563 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T18:06:45.563 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:06:45.563 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:06:45.563 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:06:45.766 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T18:06:45.870 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:06:45.874 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:06:45.897 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:06:45.897 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:06:45.897 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T18:06:45.897 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:06:45.897 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:06:45.897 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:06:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:06:46.114 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-23T18:06:46.115 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-23T18:06:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-23T18:06:46.117 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-23T18:06:46.117 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-23T18:06:46.117 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-23T18:06:46.117 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-23T18:06:46.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-23T18:06:46.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-23T18:06:46.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-23T18:06:46.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-23T18:06:46.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-23T18:06:46.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T18:06:46.130 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T18:06:46.131 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T18:06:46.132 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T18:06:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T18:06:46.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T18:06:46.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T18:06:46.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T18:06:46.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T18:06:46.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T18:06:46.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T18:06:46.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T18:06:46.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T18:06:46.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T18:06:46.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T18:06:46.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T18:06:46.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T18:06:46.135 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T18:06:46.136 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T18:06:46.137 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T18:06:46.138 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T18:06:46.138 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T18:06:46.138 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-23T18:06:46.138 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T18:06:46.138 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-23T18:06:46.138 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T18:06:46.138 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T18:06:46.138 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:06:46.138 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T18:06:46.163 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:06:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:06:46.293 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:06:46.451 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T18:06:46.498 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T18:06:46.526 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:06:46.614 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:06:46.644 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:06:46.706 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T18:06:46.728 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:06:46.753 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T18:06:46.937 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:06:46.970 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T18:06:47.080 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T18:06:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T18:06:47.318 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T18:06:47.354 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T18:06:47.430 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T18:06:47.469 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T18:06:47.514 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T18:06:47.546 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T18:06:47.573 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T18:06:47.645 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T18:06:47.705 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T18:06:47.743 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T18:06:47.787 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T18:06:47.865 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T18:06:47.897 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-23T18:06:47.941 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-23T18:06:47.969 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-23T18:06:48.043 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T18:06:48.093 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T18:06:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:06:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:48.163 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-04-23T18:06:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T18:06:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T18:06:48.163 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-04-23T18:06:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T18:06:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T18:06:48.163 INFO:teuthology.orchestra.run.smithi177.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:48.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T18:06:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T18:06:48.166 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T18:06:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T18:06:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T18:06:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T18:06:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T18:06:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T18:06:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T18:06:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T18:06:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T18:06:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T18:06:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T18:06:48.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T18:06:48.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T18:06:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T18:06:48.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T18:06:48.173 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T18:06:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:06:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T18:06:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T18:06:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T18:06:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T18:06:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T18:06:48.174 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T18:06:48.175 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T18:06:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T18:06:48.175 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-23T18:06:48.175 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-04-23T18:06:48.318 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T18:06:48.372 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T18:06:48.560 DEBUG:teuthology.parallel:result is None 2024-04-23T18:06:48.936 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T18:06:49.020 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T18:06:49.054 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T18:06:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T18:06:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-23T18:06:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T18:06:49.332 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T18:06:49.379 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T18:06:49.436 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T18:06:49.509 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T18:06:49.547 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T18:06:49.584 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T18:06:49.631 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T18:06:49.666 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T18:06:49.710 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T18:06:49.752 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T18:06:49.897 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T18:06:49.948 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T18:06:49.979 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T18:06:50.046 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T18:06:50.144 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T18:06:50.178 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T18:06:50.219 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T18:06:50.252 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T18:06:50.277 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T18:06:50.355 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T18:06:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T18:06:50.520 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-23T18:06:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T18:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T18:06:50.586 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-23T18:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T18:06:50.628 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T18:06:50.635 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-23T18:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T18:06:50.675 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T18:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T18:06:50.703 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T18:06:50.724 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T18:06:50.757 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T18:06:50.799 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T18:06:50.833 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T18:06:50.886 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-23T18:06:50.925 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T18:06:50.933 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T18:06:50.962 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-23T18:06:50.981 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T18:06:51.011 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T18:06:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T18:06:51.171 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T18:06:51.222 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T18:06:51.581 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:06:51.619 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:06:51.639 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:06:52.003 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T18:06:52.060 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T18:06:52.094 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T18:06:52.197 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T18:06:52.252 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T18:06:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T18:06:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T18:06:52.465 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-23T18:06:52.557 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T18:06:52.564 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T18:06:52.593 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T18:06:52.598 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T18:06:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T18:06:52.681 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T18:06:52.712 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T18:06:52.712 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:06:52.796 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:06:52.829 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:06:52.829 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:06:52.932 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:06:53.003 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:06:53.360 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T18:06:53.480 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T18:06:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T18:06:54.856 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T18:06:54.885 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T18:06:54.917 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T18:06:55.245 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T18:06:55.289 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:06:55.380 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T18:06:55.410 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-23T18:06:55.447 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T18:06:55.952 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T18:06:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:06:56.046 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T18:06:56.068 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-23T18:06:58.449 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:06:58.490 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:06:59.060 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T18:06:59.102 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:06:59.128 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:06:59.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-23T18:06:59.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-23T18:06:59.128 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-23T18:06:59.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-23T18:06:59.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-23T18:06:59.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-23T18:06:59.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-23T18:06:59.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:06:59.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:06:59.129 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:06:59.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-23T18:06:59.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-23T18:06:59.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:06:59.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-23T18:06:59.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-23T18:06:59.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-23T18:06:59.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-23T18:06:59.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-23T18:06:59.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-23T18:06:59.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-23T18:06:59.130 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-23T18:06:59.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-23T18:06:59.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-23T18:06:59.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-23T18:06:59.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-23T18:06:59.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-23T18:06:59.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-23T18:06:59.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-23T18:06:59.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-23T18:06:59.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-23T18:06:59.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-23T18:06:59.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-23T18:06:59.131 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T18:06:59.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T18:06:59.133 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T18:06:59.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T18:06:59.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T18:06:59.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T18:06:59.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T18:06:59.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T18:06:59.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T18:06:59.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T18:06:59.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T18:06:59.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T18:06:59.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T18:06:59.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T18:06:59.134 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T18:06:59.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T18:06:59.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T18:06:59.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T18:06:59.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T18:06:59.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T18:06:59.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T18:06:59.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T18:06:59.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T18:06:59.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T18:06:59.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T18:06:59.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T18:06:59.135 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T18:06:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T18:06:59.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T18:06:59.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T18:06:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T18:06:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T18:06:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T18:06:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T18:06:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T18:06:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T18:06:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T18:06:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T18:06:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T18:06:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T18:06:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T18:06:59.139 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T18:06:59.140 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-23T18:06:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T18:06:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-23T18:06:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T18:06:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T18:06:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:06:59.141 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T18:06:59.185 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T18:06:59.185 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T18:06:59.185 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T18:06:59.185 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:06:59.219 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:07:01.220 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:07:01.220 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:07:01.220 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-04-23T18:07:01.220 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.220 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.220 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T18:07:01.220 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.221 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.222 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T18:07:01.223 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.224 INFO:teuthology.orchestra.run.smithi188.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T18:07:01.225 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.226 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T18:07:01.227 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T18:07:01.228 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T18:07:01.229 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T18:07:01.230 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T18:07:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T18:07:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T18:07:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T18:07:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T18:07:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T18:07:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T18:07:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T18:07:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T18:07:01.231 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T18:07:01.232 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T18:07:01.232 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:07:01.232 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-23T18:07:01.601 DEBUG:teuthology.parallel:result is None 2024-04-23T18:07:11.043 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T18:07:11.043 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2024-04-23T18:07:11.043 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2024-04-23T18:07:11.043 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2024-04-23T18:07:11.044 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2024-04-23T18:07:11.044 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2024-04-23T18:07:11.044 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /root 2024-04-23T18:07:11.044 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2024-04-23T18:07:11.044 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:07:11.216 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:07:11.231 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T18:07:11.232 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:07:11.232 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T18:07:11.232 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:07:12.761 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:07:12.788 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T18:07:12.788 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:07:12.788 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T18:07:12.788 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:07:12.788 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T18:07:12.788 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:07:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:07:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T18:07:13.044 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:07:13.044 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T18:07:13.044 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:07:13.044 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T18:07:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:07:13.504 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:07:13.527 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T18:07:13.528 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:07:13.528 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T18:07:13.528 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:07:13.528 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T18:07:13.528 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:07:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T18:07:13.693 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T18:07:13.757 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T18:07:13.953 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T18:07:14.006 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T18:07:14.087 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T18:07:14.185 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:07:14.328 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:07:14.365 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T18:07:14.396 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T18:07:14.423 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T18:07:14.918 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:07:14.947 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T18:07:14.980 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-23T18:07:15.044 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-23T18:07:15.086 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:07:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T18:07:16.213 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:07:17.228 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T18:07:17.255 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:07:17.362 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T18:07:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-23T18:07:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:07:17.581 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T18:07:17.581 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:07:17.582 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T18:07:17.582 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:07:17.582 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T18:07:17.582 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:07:17.633 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:07:17.644 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T18:07:17.746 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-23T18:07:17.792 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-23T18:07:18.768 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T18:07:18.811 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:07:18.835 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T18:07:18.835 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:07:18.835 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T18:07:18.835 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:07:18.835 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T18:07:18.835 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:07:18.879 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:07:18.901 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T18:07:18.902 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:07:18.902 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T18:07:18.902 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:07:18.946 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:07:18.969 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T18:07:18.969 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:07:18.969 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T18:07:18.969 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:07:18.969 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T18:07:18.969 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:07:19.288 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:07:19.311 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T18:07:19.311 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T18:07:19.311 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T18:07:19.311 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:07:19.312 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T18:07:19.312 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:07:24.176 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-23T18:07:24.218 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-23T18:07:24.258 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-23T18:07:24.289 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T18:07:24.309 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T18:07:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-23T18:07:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T18:07:24.568 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-23T18:07:25.180 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:07:25.216 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:07:25.236 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T18:07:26.139 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-23T18:07:26.220 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T18:07:26.254 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T18:07:26.347 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T18:07:26.392 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T18:07:26.392 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:07:26.461 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T18:07:26.494 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:07:26.494 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:07:26.579 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T18:07:26.645 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:07:32.427 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:07:32.427 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-23T18:07:32.427 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-23T18:07:32.427 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-23T18:07:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-23T18:07:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-23T18:07:32.430 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-23T18:07:32.441 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-23T18:07:32.441 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-23T18:07:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-23T18:07:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-23T18:07:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-23T18:07:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-23T18:07:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-23T18:07:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-23T18:07:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-23T18:07:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T18:07:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T18:07:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T18:07:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T18:07:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T18:07:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T18:07:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T18:07:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T18:07:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T18:07:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T18:07:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T18:07:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T18:07:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T18:07:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T18:07:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T18:07:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T18:07:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T18:07:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T18:07:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T18:07:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T18:07:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T18:07:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T18:07:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T18:07:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T18:07:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T18:07:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T18:07:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T18:07:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T18:07:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T18:07:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T18:07:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T18:07:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T18:07:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T18:07:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T18:07:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T18:07:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T18:07:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T18:07:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T18:07:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T18:07:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T18:07:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T18:07:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T18:07:32.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T18:07:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T18:07:32.449 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T18:07:32.450 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T18:07:32.451 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T18:07:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T18:07:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:07:34.569 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2024-04-23T18:07:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T18:07:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T18:07:34.569 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-04-23T18:07:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T18:07:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T18:07:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T18:07:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T18:07:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T18:07:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T18:07:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T18:07:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T18:07:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T18:07:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T18:07:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T18:07:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T18:07:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T18:07:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T18:07:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T18:07:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T18:07:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T18:07:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T18:07:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T18:07:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T18:07:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T18:07:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T18:07:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T18:07:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T18:07:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T18:07:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:07:34.581 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-04-23T18:07:34.938 DEBUG:teuthology.parallel:result is None 2024-04-23T18:07:34.938 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:07:35.085 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T18:07:35.107 INFO:teuthology.orchestra.run.smithi050.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-23T18:07:35.107 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-23T18:07:35.108 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-23T18:07:35.110 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:07:35.320 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T18:07:35.346 INFO:teuthology.orchestra.run.smithi177.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-23T18:07:35.347 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-23T18:07:35.347 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-23T18:07:35.348 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:07:35.485 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T18:07:35.511 INFO:teuthology.orchestra.run.smithi188.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-23T18:07:35.512 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-23T18:07:35.512 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-23T18:07:35.514 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-23T18:07:35.514 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:07:35.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T18:07:35.541 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-23T18:07:35.541 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T18:07:35.572 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T18:07:35.572 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T18:07:35.601 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-23T18:07:35.601 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:07:35.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T18:07:35.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T18:07:35.691 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-23T18:07:35.691 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T18:07:35.720 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T18:07:35.799 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T18:07:35.799 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T18:07:35.826 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T18:07:35.892 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-23T18:07:35.892 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:07:35.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T18:07:35.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T18:07:35.981 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-23T18:07:35.981 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T18:07:36.004 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T18:07:36.068 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T18:07:36.068 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T18:07:36.091 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T18:07:36.155 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-23T18:07:36.156 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:07:36.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T18:07:36.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T18:07:36.247 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-23T18:07:36.247 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T18:07:36.274 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T18:07:36.337 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T18:07:36.337 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T18:07:36.360 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T18:07:36.423 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-23T18:07:36.535 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-23T18:07:36.535 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:07:36.536 INFO:tasks.cephadm:Cluster fsid is 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:07:36.536 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-23T18:07:36.536 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.50', 'mon.b': '172.21.15.177', 'mon.c': '172.21.15.188'} 2024-04-23T18:07:36.536 INFO:tasks.cephadm:First mon is mon.a on smithi050 2024-04-23T18:07:36.536 INFO:tasks.cephadm:First mgr is x 2024-04-23T18:07:36.537 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-23T18:07:36.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo hostname $(hostname -s) 2024-04-23T18:07:36.562 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname $(hostname -s) 2024-04-23T18:07:36.586 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname $(hostname -s) 2024-04-23T18:07:36.609 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-23T18:07:36.609 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:07:36.752 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78795/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-gd69f13d8', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.gd69f13d8'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 12:00:06.590653', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-23T18:07:36.892 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 d69f13d827952bb07bb9b7973bca14f97d764d6f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:07:36.893 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm 2024-04-23T18:07:36.893 DEBUG:teuthology.orchestra.run.smithi050:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T18:07:37.773 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 23 18:07 /home/ubuntu/cephtest/cephadm 2024-04-23T18:07:37.774 DEBUG:teuthology.orchestra.run.smithi177:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T18:07:38.379 INFO:teuthology.orchestra.run.smithi177.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 23 18:07 /home/ubuntu/cephtest/cephadm 2024-04-23T18:07:38.380 DEBUG:teuthology.orchestra.run.smithi188:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T18:07:39.240 INFO:teuthology.orchestra.run.smithi188.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 23 18:07 /home/ubuntu/cephtest/cephadm 2024-04-23T18:07:39.240 DEBUG:teuthology.orchestra.run.smithi050:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T18:07:39.260 DEBUG:teuthology.orchestra.run.smithi177:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T18:07:39.279 DEBUG:teuthology.orchestra.run.smithi188:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T18:07:39.316 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f on all hosts... 2024-04-23T18:07:39.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-23T18:07:39.320 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-23T18:07:39.322 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-23T18:07:40.825 INFO:teuthology.orchestra.run.smithi188.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:07:40.918 INFO:teuthology.orchestra.run.smithi177.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:07:41.027 INFO:teuthology.orchestra.run.smithi050.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:08:08.788 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-23T18:08:08.788 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-23T18:08:08.788 INFO:teuthology.orchestra.run.smithi177.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-23T18:08:08.788 INFO:teuthology.orchestra.run.smithi177.stdout: "repo_digests": [ 2024-04-23T18:08:08.788 INFO:teuthology.orchestra.run.smithi177.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-23T18:08:08.788 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-23T18:08:08.788 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-23T18:08:14.334 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:08:14.334 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-23T18:08:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-23T18:08:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: "repo_digests": [ 2024-04-23T18:08:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-23T18:08:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-23T18:08:14.335 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:08:19.199 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T18:08:19.199 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-23T18:08:19.199 INFO:teuthology.orchestra.run.smithi188.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-23T18:08:19.199 INFO:teuthology.orchestra.run.smithi188.stdout: "repo_digests": [ 2024-04-23T18:08:19.199 INFO:teuthology.orchestra.run.smithi188.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-23T18:08:19.200 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-23T18:08:19.200 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T18:08:19.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph 2024-04-23T18:08:19.257 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph 2024-04-23T18:08:19.288 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph 2024-04-23T18:08:19.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /etc/ceph 2024-04-23T18:08:19.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /etc/ceph 2024-04-23T18:08:19.371 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /etc/ceph 2024-04-23T18:08:19.395 INFO:tasks.cephadm:Writing seed config... 2024-04-23T18:08:19.396 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-23T18:08:19.396 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-23T18:08:19.396 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-23T18:08:19.396 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-23T18:08:19.396 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-23T18:08:19.396 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-23T18:08:19.396 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-23T18:08:19.397 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-23T18:08:19.397 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-23T18:08:19.397 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-23T18:08:19.397 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-23T18:08:19.397 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-23T18:08:19.397 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-23T18:08:19.397 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-23T18:08:19.397 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-23T18:08:19.397 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-23T18:08:19.397 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-23T18:08:19.397 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-23T18:08:19.398 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-23T18:08:19.399 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-23T18:08:19.400 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-23T18:08:19.400 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:08:19.400 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-23T18:08:19.419 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 5d677974-019c-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-23T18:08:19.419 DEBUG:teuthology.orchestra.run.smithi050:mon.a> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.a.service 2024-04-23T18:08:19.462 DEBUG:teuthology.orchestra.run.smithi050:mgr.x> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.x.service 2024-04-23T18:08:19.505 INFO:tasks.cephadm:Bootstrapping... 2024-04-23T18:08:19.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f -v bootstrap --fsid 5d677974-019c-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.50 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:08:19.713 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-04-23T18:08:19.713 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f', '-v', 'bootstrap', '--fsid', '5d677974-019c-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.50', '--skip-admin-label'] 2024-04-23T18:08:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-23T18:08:19.714 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying podman|docker is present... 2024-04-23T18:08:19.741 INFO:teuthology.orchestra.run.smithi050.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-23T18:08:19.741 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying lvm2 is present... 2024-04-23T18:08:19.741 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying time synchronization is in place... 2024-04-23T18:08:19.748 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-23T18:08:19.749 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T18:08:19.755 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-23T18:08:19.755 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-23T18:08:19.763 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout enabled 2024-04-23T18:08:19.769 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout active 2024-04-23T18:08:19.769 INFO:teuthology.orchestra.run.smithi050.stdout:Unit chronyd.service is enabled and running 2024-04-23T18:08:19.769 INFO:teuthology.orchestra.run.smithi050.stdout:Repeating the final host check... 2024-04-23T18:08:19.796 INFO:teuthology.orchestra.run.smithi050.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-23T18:08:19.796 INFO:teuthology.orchestra.run.smithi050.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-23T18:08:19.796 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl is present 2024-04-23T18:08:19.797 INFO:teuthology.orchestra.run.smithi050.stdout:lvcreate is present 2024-04-23T18:08:19.802 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-23T18:08:19.802 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T18:08:19.809 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-23T18:08:19.809 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-23T18:08:19.816 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout enabled 2024-04-23T18:08:19.823 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout active 2024-04-23T18:08:19.823 INFO:teuthology.orchestra.run.smithi050.stdout:Unit chronyd.service is enabled and running 2024-04-23T18:08:19.823 INFO:teuthology.orchestra.run.smithi050.stdout:Host looks OK 2024-04-23T18:08:19.823 INFO:teuthology.orchestra.run.smithi050.stdout:Cluster fsid: 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:19.823 INFO:teuthology.orchestra.run.smithi050.stdout:Acquiring lock 140713756773536 on /run/cephadm/5d677974-019c-11ef-bc93-c7b262605968.lock 2024-04-23T18:08:19.823 INFO:teuthology.orchestra.run.smithi050.stdout:Lock 140713756773536 acquired on /run/cephadm/5d677974-019c-11ef-bc93-c7b262605968.lock 2024-04-23T18:08:19.824 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying IP 172.21.15.50 port 3300 ... 2024-04-23T18:08:19.824 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying IP 172.21.15.50 port 6789 ... 2024-04-23T18:08:19.824 INFO:teuthology.orchestra.run.smithi050.stdout:Base mon IP(s) is [172.21.15.50:3300, 172.21.15.50:6789], mon addrv is [v2:172.21.15.50:3300,v1:172.21.15.50:6789] 2024-04-23T18:08:19.827 INFO:teuthology.orchestra.run.smithi050.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.50 metric 100 2024-04-23T18:08:19.828 INFO:teuthology.orchestra.run.smithi050.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.50 metric 100 2024-04-23T18:08:19.830 INFO:teuthology.orchestra.run.smithi050.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-23T18:08:19.830 INFO:teuthology.orchestra.run.smithi050.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-23T18:08:19.831 INFO:teuthology.orchestra.run.smithi050.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-23T18:08:19.833 INFO:teuthology.orchestra.run.smithi050.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-23T18:08:19.833 INFO:teuthology.orchestra.run.smithi050.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-23T18:08:19.833 INFO:teuthology.orchestra.run.smithi050.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-23T18:08:19.833 INFO:teuthology.orchestra.run.smithi050.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-23T18:08:19.833 INFO:teuthology.orchestra.run.smithi050.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1584/64 scope link noprefixroute 2024-04-23T18:08:19.833 INFO:teuthology.orchestra.run.smithi050.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-23T18:08:19.834 INFO:teuthology.orchestra.run.smithi050.stdout:Mon IP `172.21.15.50` is in CIDR network `172.21.0.0/20` 2024-04-23T18:08:19.834 INFO:teuthology.orchestra.run.smithi050.stdout:Mon IP `172.21.15.50` is in CIDR network `172.21.0.0/20` 2024-04-23T18:08:19.834 INFO:teuthology.orchestra.run.smithi050.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-23T18:08:19.834 INFO:teuthology.orchestra.run.smithi050.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-23T18:08:19.835 INFO:teuthology.orchestra.run.smithi050.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:08:20.320 INFO:teuthology.orchestra.run.smithi050.stdout:/bin/podman: stdout da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-23T18:08:20.320 INFO:teuthology.orchestra.run.smithi050.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T18:08:20.320 INFO:teuthology.orchestra.run.smithi050.stdout:/bin/podman: stderr Getting image source signatures 2024-04-23T18:08:20.320 INFO:teuthology.orchestra.run.smithi050.stdout:/bin/podman: stderr Copying blob sha256:7f03586bcf9c6a2b4d690c118c4aa5b771f9c6f252d75a20fb283dd07093a738 2024-04-23T18:08:20.320 INFO:teuthology.orchestra.run.smithi050.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-23T18:08:20.320 INFO:teuthology.orchestra.run.smithi050.stdout:/bin/podman: stderr Copying config sha256:da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-23T18:08:20.320 INFO:teuthology.orchestra.run.smithi050.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-23T18:08:21.224 INFO:teuthology.orchestra.run.smithi050.stdout:ceph: stdout ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:08:21.224 INFO:teuthology.orchestra.run.smithi050.stdout:Ceph version: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:08:21.224 INFO:teuthology.orchestra.run.smithi050.stdout:Extracting ceph user uid/gid from container image... 2024-04-23T18:08:22.031 INFO:teuthology.orchestra.run.smithi050.stdout:stat: stdout 167 167 2024-04-23T18:08:22.032 INFO:teuthology.orchestra.run.smithi050.stdout:Creating initial keys... 2024-04-23T18:08:26.087 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-authtool: stdout AQAW+Sdm6TuTFxAA2k08ptJMe/dwDp0RrkMWxA== 2024-04-23T18:08:33.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-authtool: stdout AQAc+SdmilPOORAAyZ1zvj8aefOhzApXE8uPnA== 2024-04-23T18:08:34.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-authtool: stdout AQAh+SdmSWVKGBAAOin4p/T1AusttRu0o+uxFA== 2024-04-23T18:08:34.290 INFO:teuthology.orchestra.run.smithi050.stdout:Creating initial monmap... 2024-04-23T18:08:35.221 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T18:08:35.221 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-23T18:08:35.221 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:35.221 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T18:08:35.221 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool for a [v2:172.21.15.50:3300,v1:172.21.15.50:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T18:08:35.221 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = quincy 2024-04-23T18:08:35.221 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: set fsid to 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:35.221 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T18:08:35.221 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:08:35.221 INFO:teuthology.orchestra.run.smithi050.stdout:Creating mon... 2024-04-23T18:08:36.109 INFO:teuthology.orchestra.run.smithi050.stdout:create mon.a on 2024-04-23T18:08:36.314 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-23T18:08:36.491 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-23T18:08:36.670 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-5d677974-019c-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-5d677974-019c-11ef-bc93-c7b262605968.target. 2024-04-23T18:08:36.670 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-5d677974-019c-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-5d677974-019c-11ef-bc93-c7b262605968.target. 2024-04-23T18:08:36.904 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.a 2024-04-23T18:08:36.905 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-23T18:08:37.077 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5d677974-019c-11ef-bc93-c7b262605968.target.wants/ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-5d677974-019c-11ef-bc93-c7b262605968@.service. 2024-04-23T18:08:37.599 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T18:08:37.599 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout disabled 2024-04-23T18:08:37.606 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T18:08:37.606 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-23T18:08:37.606 INFO:teuthology.orchestra.run.smithi050.stdout:firewalld.service is not enabled 2024-04-23T18:08:37.607 INFO:teuthology.orchestra.run.smithi050.stdout:Not possible to enable service . firewalld.service is not available 2024-04-23T18:08:37.607 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mon to start... 2024-04-23T18:08:37.607 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mon... 2024-04-23T18:08:37.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:37 smithi050 systemd[1]: Started Ceph mon.a for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:08:38.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:38 smithi050 ceph-mon[28817]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T18:08:38.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:38 smithi050 ceph-mon[28817]: monmap epoch 1 2024-04-23T18:08:38.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:38 smithi050 ceph-mon[28817]: fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:38.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:38 smithi050 ceph-mon[28817]: last_changed 2024-04-23T18:08:34.626890+0000 2024-04-23T18:08:38.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:38 smithi050 ceph-mon[28817]: created 2024-04-23T18:08:34.626890+0000 2024-04-23T18:08:38.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:38 smithi050 ceph-mon[28817]: min_mon_release 19 (squid) 2024-04-23T18:08:38.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:38 smithi050 ceph-mon[28817]: election_strategy: 1 2024-04-23T18:08:38.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:38 smithi050 ceph-mon[28817]: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-23T18:08:38.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:38 smithi050 ceph-mon[28817]: fsmap 2024-04-23T18:08:38.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:38 smithi050 ceph-mon[28817]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T18:08:38.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:38 smithi050 ceph-mon[28817]: mgrmap e1: no daemons active 2024-04-23T18:08:38.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:38 smithi050 ceph-mon[28817]: from='client.? 172.21.15.50:0/370350152' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-23T18:08:41.416 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout cluster: 2024-04-23T18:08:41.422 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout id: 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:41.422 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-23T18:08:41.422 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:41.422 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout services: 2024-04-23T18:08:41.422 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.558124s) 2024-04-23T18:08:41.422 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-23T18:08:41.422 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout data: 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout pgs: 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.044+0000 7f6757ca5700 1 Processor -- start 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.044+0000 7f6757ca5700 1 -- start start 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.045+0000 7f6757ca5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 0x7f675010b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.045+0000 7f6757ca5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f675007be60 con 0x7f6750109020 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.045+0000 7f6755a41700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 0x7f675010b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.045+0000 7f6755a41700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 0x7f675010b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44106/0 (socket says 172.21.15.50:44106) 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.045+0000 7f6755a41700 1 -- 172.21.15.50:0/3345349359 learned_addr learned my addr 172.21.15.50:0/3345349359 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.045+0000 7f6755a41700 1 -- 172.21.15.50:0/3345349359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675010b950 con 0x7f6750109020 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.045+0000 7f6755a41700 1 --2- 172.21.15.50:0/3345349359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 0x7f675010b410 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f67400098a0 tx=0x7f6740009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a7e05148514979 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.046+0000 7f6754a3f700 1 -- 172.21.15.50:0/3345349359 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6740004160 con 0x7f6750109020 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.046+0000 7f6754a3f700 1 -- 172.21.15.50:0/3345349359 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6740004300 con 0x7f6750109020 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.046+0000 7f6754a3f700 1 -- 172.21.15.50:0/3345349359 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67400036a0 con 0x7f6750109020 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.046+0000 7f6757ca5700 1 -- 172.21.15.50:0/3345349359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 msgr2=0x7f675010b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.046+0000 7f6757ca5700 1 --2- 172.21.15.50:0/3345349359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 0x7f675010b410 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f67400098a0 tx=0x7f6740009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.046+0000 7f6757ca5700 1 -- 172.21.15.50:0/3345349359 shutdown_connections 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.046+0000 7f6757ca5700 1 --2- 172.21.15.50:0/3345349359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 0x7f675010b410 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.046+0000 7f6757ca5700 1 -- 172.21.15.50:0/3345349359 >> 172.21.15.50:0/3345349359 conn(0x7f6750102a50 msgr2=0x7f6750104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.046+0000 7f6757ca5700 1 -- 172.21.15.50:0/3345349359 shutdown_connections 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.046+0000 7f6757ca5700 1 -- 172.21.15.50:0/3345349359 wait complete. 2024-04-23T18:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.047+0000 7f6757ca5700 1 Processor -- start 2024-04-23T18:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.047+0000 7f6757ca5700 1 -- start start 2024-04-23T18:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.047+0000 7f6757ca5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 0x7f67501088a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.047+0000 7f6757ca5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f675010bdc0 con 0x7f6750109020 2024-04-23T18:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.047+0000 7f6755a41700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 0x7f67501088a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.047+0000 7f6755a41700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 0x7f67501088a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44110/0 (socket says 172.21.15.50:44110) 2024-04-23T18:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.047+0000 7f6755a41700 1 -- 172.21.15.50:0/370350152 learned_addr learned my addr 172.21.15.50:0/370350152 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.048+0000 7f6755a41700 1 -- 172.21.15.50:0/370350152 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6750108de0 con 0x7f6750109020 2024-04-23T18:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.048+0000 7f6755a41700 1 --2- 172.21.15.50:0/370350152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 0x7f67501088a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6740000c00 tx=0x7f6740004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.048+0000 7f6746ffd700 1 -- 172.21.15.50:0/370350152 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6740003db0 con 0x7f6750109020 2024-04-23T18:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.048+0000 7f6746ffd700 1 -- 172.21.15.50:0/370350152 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6740011630 con 0x7f6750109020 2024-04-23T18:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.048+0000 7f6757ca5700 1 -- 172.21.15.50:0/370350152 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6750106fd0 con 0x7f6750109020 2024-04-23T18:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.048+0000 7f6746ffd700 1 -- 172.21.15.50:0/370350152 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6740011950 con 0x7f6750109020 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.048+0000 7f6757ca5700 1 -- 172.21.15.50:0/370350152 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6750107510 con 0x7f6750109020 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.048+0000 7f6746ffd700 1 -- 172.21.15.50:0/370350152 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f674001a460 con 0x7f6750109020 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.048+0000 7f6746ffd700 1 -- 172.21.15.50:0/370350152 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f674001b780 con 0x7f6750109020 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.049+0000 7f6757ca5700 1 -- 172.21.15.50:0/370350152 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67500640a0 con 0x7f6750109020 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.052+0000 7f6746ffd700 1 -- 172.21.15.50:0/370350152 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f674001ba20 con 0x7f6750109020 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.125+0000 7f6757ca5700 1 -- 172.21.15.50:0/370350152 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f67501956d0 con 0x7f6750109020 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.126+0000 7f6746ffd700 1 -- 172.21.15.50:0/370350152 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f674001bc00 con 0x7f6750109020 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.127+0000 7f6757ca5700 1 -- 172.21.15.50:0/370350152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 msgr2=0x7f67501088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.127+0000 7f6757ca5700 1 --2- 172.21.15.50:0/370350152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 0x7f67501088a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6740000c00 tx=0x7f6740004750 comp rx=0 tx=0).stop 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.127+0000 7f6757ca5700 1 -- 172.21.15.50:0/370350152 shutdown_connections 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.127+0000 7f6757ca5700 1 --2- 172.21.15.50:0/370350152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6750109020 0x7f67501088a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.127+0000 7f6757ca5700 1 -- 172.21.15.50:0/370350152 >> 172.21.15.50:0/370350152 conn(0x7f6750102a50 msgr2=0x7f6750103930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.127+0000 7f6757ca5700 1 -- 172.21.15.50:0/370350152 shutdown_connections 2024-04-23T18:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:38.127+0000 7f6757ca5700 1 -- 172.21.15.50:0/370350152 wait complete. 2024-04-23T18:08:41.427 INFO:teuthology.orchestra.run.smithi050.stdout:mon is available 2024-04-23T18:08:41.427 INFO:teuthology.orchestra.run.smithi050.stdout:Assimilating anything we can from ceph.conf... 2024-04-23T18:08:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:41 smithi050 ceph-mon[28817]: from='client.? 172.21.15.50:0/3824681398' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T18:08:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:41 smithi050 ceph-mon[28817]: from='client.? 172.21.15.50:0/3824681398' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-23T18:08:42.455 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:42.455 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [global] 2024-04-23T18:08:42.456 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout fsid = 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:42.456 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-23T18:08:42.456 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.50:3300,v1:172.21.15.50:6789] 2024-04-23T18:08:42.456 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-23T18:08:42.456 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-23T18:08:42.456 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-23T18:08:42.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-23T18:08:42.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:42.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-23T18:08:42.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-23T18:08:42.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:42.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [osd] 2024-04-23T18:08:42.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-23T18:08:42.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-23T18:08:42.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-23T18:08:42.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-23T18:08:42.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-23T18:08:42.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-23T18:08:42.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.842+0000 7fec0ff89700 1 Processor -- start 2024-04-23T18:08:42.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.842+0000 7fec0ff89700 1 -- start start 2024-04-23T18:08:42.461 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.843+0000 7fec0ff89700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 0x7fec0810b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:42.461 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.843+0000 7fec0ff89700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fec08063d60 con 0x7fec08108fb0 2024-04-23T18:08:42.461 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.843+0000 7fec0dd25700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 0x7fec0810b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.843+0000 7fec0dd25700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 0x7fec0810b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48682/0 (socket says 172.21.15.50:48682) 2024-04-23T18:08:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.843+0000 7fec0dd25700 1 -- 172.21.15.50:0/4043049365 learned_addr learned my addr 172.21.15.50:0/4043049365 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.843+0000 7fec0dd25700 1 -- 172.21.15.50:0/4043049365 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0810b8e0 con 0x7fec08108fb0 2024-04-23T18:08:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.843+0000 7fec0dd25700 1 --2- 172.21.15.50:0/4043049365 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 0x7fec0810b3a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fec040098a0 tx=0x7fec04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40be4be917db755c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.843+0000 7fec0cd23700 1 -- 172.21.15.50:0/4043049365 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec04004160 con 0x7fec08108fb0 2024-04-23T18:08:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.843+0000 7fec0cd23700 1 -- 172.21.15.50:0/4043049365 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec04004300 con 0x7fec08108fb0 2024-04-23T18:08:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.844+0000 7fec0cd23700 1 -- 172.21.15.50:0/4043049365 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec040036a0 con 0x7fec08108fb0 2024-04-23T18:08:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.844+0000 7fec0ff89700 1 -- 172.21.15.50:0/4043049365 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 msgr2=0x7fec0810b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:42.462 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.844+0000 7fec0ff89700 1 --2- 172.21.15.50:0/4043049365 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 0x7fec0810b3a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fec040098a0 tx=0x7fec04009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.844+0000 7fec0ff89700 1 -- 172.21.15.50:0/4043049365 shutdown_connections 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.844+0000 7fec0ff89700 1 --2- 172.21.15.50:0/4043049365 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 0x7fec0810b3a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.844+0000 7fec0ff89700 1 -- 172.21.15.50:0/4043049365 >> 172.21.15.50:0/4043049365 conn(0x7fec08102a40 msgr2=0x7fec08104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.844+0000 7fec0ff89700 1 -- 172.21.15.50:0/4043049365 shutdown_connections 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.844+0000 7fec0ff89700 1 -- 172.21.15.50:0/4043049365 wait complete. 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.845+0000 7fec0ff89700 1 Processor -- start 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.845+0000 7fec0ff89700 1 -- start start 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.845+0000 7fec0ff89700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 0x7fec0819bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.845+0000 7fec0ff89700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fec0810bdb0 con 0x7fec08108fb0 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.845+0000 7fec0dd25700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 0x7fec0819bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.845+0000 7fec0dd25700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 0x7fec0819bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48694/0 (socket says 172.21.15.50:48694) 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.845+0000 7fec0dd25700 1 -- 172.21.15.50:0/3824681398 learned_addr learned my addr 172.21.15.50:0/3824681398 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:42.463 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.845+0000 7fec0dd25700 1 -- 172.21.15.50:0/3824681398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0819c180 con 0x7fec08108fb0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.846+0000 7fec0dd25700 1 --2- 172.21.15.50:0/3824681398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 0x7fec0819bc40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fec04000c00 tx=0x7fec04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.846+0000 7febfeffd700 1 -- 172.21.15.50:0/3824681398 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec04003db0 con 0x7fec08108fb0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.846+0000 7febfeffd700 1 -- 172.21.15.50:0/3824681398 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec04011630 con 0x7fec08108fb0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.846+0000 7febfeffd700 1 -- 172.21.15.50:0/3824681398 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec04011950 con 0x7fec08108fb0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.846+0000 7fec0ff89700 1 -- 172.21.15.50:0/3824681398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec0819c410 con 0x7fec08108fb0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.846+0000 7fec0ff89700 1 -- 172.21.15.50:0/3824681398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec0819f0b0 con 0x7fec08108fb0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.846+0000 7febfeffd700 1 -- 172.21.15.50:0/3824681398 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fec04011af0 con 0x7fec08108fb0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.846+0000 7febfeffd700 1 -- 172.21.15.50:0/3824681398 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fec0401c450 con 0x7fec08108fb0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.847+0000 7fec0ff89700 1 -- 172.21.15.50:0/3824681398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec080640a0 con 0x7fec08108fb0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.849+0000 7febfeffd700 1 -- 172.21.15.50:0/3824681398 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fec04004260 con 0x7fec08108fb0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.920+0000 7fec0ff89700 1 -- 172.21.15.50:0/3824681398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fec0802e180 con 0x7fec08108fb0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.923+0000 7febfeffd700 1 -- 172.21.15.50:0/3824681398 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+492 (secure 0 0 0) 0x7fec0401b450 con 0x7fec08108fb0 2024-04-23T18:08:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.923+0000 7febfeffd700 1 -- 172.21.15.50:0/3824681398 <== mon.0 v2:172.21.15.50:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fec0401b630 con 0x7fec08108fb0 2024-04-23T18:08:42.465 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.924+0000 7fec0ff89700 1 -- 172.21.15.50:0/3824681398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 msgr2=0x7fec0819bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:42.465 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.924+0000 7fec0ff89700 1 --2- 172.21.15.50:0/3824681398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 0x7fec0819bc40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fec04000c00 tx=0x7fec04004750 comp rx=0 tx=0).stop 2024-04-23T18:08:42.465 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.924+0000 7fec0ff89700 1 -- 172.21.15.50:0/3824681398 shutdown_connections 2024-04-23T18:08:42.465 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.924+0000 7fec0ff89700 1 --2- 172.21.15.50:0/3824681398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec08108fb0 0x7fec0819bc40 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:42.465 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.924+0000 7fec0ff89700 1 -- 172.21.15.50:0/3824681398 >> 172.21.15.50:0/3824681398 conn(0x7fec08102a40 msgr2=0x7fec081038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:42.465 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.924+0000 7fec0ff89700 1 -- 172.21.15.50:0/3824681398 shutdown_connections 2024-04-23T18:08:42.465 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:41.924+0000 7fec0ff89700 1 -- 172.21.15.50:0/3824681398 wait complete. 2024-04-23T18:08:42.465 INFO:teuthology.orchestra.run.smithi050.stdout:Generating new minimal ceph.conf... 2024-04-23T18:08:43.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:42 smithi050 ceph-mon[28817]: from='client.? 172.21.15.50:0/2353360235' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:08:43.560 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.894+0000 7f4f40317700 1 Processor -- start 2024-04-23T18:08:43.560 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.894+0000 7f4f40317700 1 -- start start 2024-04-23T18:08:43.560 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.895+0000 7f4f40317700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 0x7f4f38108980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:43.560 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.895+0000 7f4f40317700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4f38108f50 con 0x7f4f38108560 2024-04-23T18:08:43.560 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.895+0000 7f4f3e0b3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 0x7f4f38108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:43.560 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.895+0000 7f4f3e0b3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 0x7f4f38108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48710/0 (socket says 172.21.15.50:48710) 2024-04-23T18:08:43.560 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.895+0000 7f4f3e0b3700 1 -- 172.21.15.50:0/3649912375 learned_addr learned my addr 172.21.15.50:0/3649912375 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.895+0000 7f4f3e0b3700 1 -- 172.21.15.50:0/3649912375 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f38109710 con 0x7f4f38108560 2024-04-23T18:08:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.895+0000 7f4f3e0b3700 1 --2- 172.21.15.50:0/3649912375 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 0x7f4f38108980 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4f340098a0 tx=0x7f4f34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e126b311e6284f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.895+0000 7f4f3d0b1700 1 -- 172.21.15.50:0/3649912375 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f34004160 con 0x7f4f38108560 2024-04-23T18:08:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.895+0000 7f4f3d0b1700 1 -- 172.21.15.50:0/3649912375 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4f34004300 con 0x7f4f38108560 2024-04-23T18:08:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.895+0000 7f4f3d0b1700 1 -- 172.21.15.50:0/3649912375 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f34011af0 con 0x7f4f38108560 2024-04-23T18:08:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.896+0000 7f4f40317700 1 -- 172.21.15.50:0/3649912375 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 msgr2=0x7f4f38108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.896+0000 7f4f40317700 1 --2- 172.21.15.50:0/3649912375 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 0x7f4f38108980 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4f340098a0 tx=0x7f4f34009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.896+0000 7f4f40317700 1 -- 172.21.15.50:0/3649912375 shutdown_connections 2024-04-23T18:08:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.896+0000 7f4f40317700 1 --2- 172.21.15.50:0/3649912375 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 0x7f4f38108980 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.896+0000 7f4f40317700 1 -- 172.21.15.50:0/3649912375 >> 172.21.15.50:0/3649912375 conn(0x7f4f38103b40 msgr2=0x7f4f38105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.896+0000 7f4f40317700 1 -- 172.21.15.50:0/3649912375 shutdown_connections 2024-04-23T18:08:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.896+0000 7f4f40317700 1 -- 172.21.15.50:0/3649912375 wait complete. 2024-04-23T18:08:43.562 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.897+0000 7f4f40317700 1 Processor -- start 2024-04-23T18:08:43.562 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.897+0000 7f4f40317700 1 -- start start 2024-04-23T18:08:43.562 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.897+0000 7f4f40317700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 0x7f4f3819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:43.562 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.897+0000 7f4f40317700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3810cac0 con 0x7f4f38108560 2024-04-23T18:08:43.562 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.897+0000 7f4f3e0b3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 0x7f4f3819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:43.562 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.897+0000 7f4f3e0b3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 0x7f4f3819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48716/0 (socket says 172.21.15.50:48716) 2024-04-23T18:08:43.562 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.897+0000 7f4f3e0b3700 1 -- 172.21.15.50:0/2353360235 learned_addr learned my addr 172.21.15.50:0/2353360235 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:43.562 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.897+0000 7f4f3e0b3700 1 -- 172.21.15.50:0/2353360235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3819ca50 con 0x7f4f38108560 2024-04-23T18:08:43.562 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.897+0000 7f4f3e0b3700 1 --2- 172.21.15.50:0/2353360235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 0x7f4f3819c510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009fb0 tx=0x7f4f34011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:43.562 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.898+0000 7f4f2f7fe700 1 -- 172.21.15.50:0/2353360235 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f340036a0 con 0x7f4f38108560 2024-04-23T18:08:43.562 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.898+0000 7f4f2f7fe700 1 -- 172.21.15.50:0/2353360235 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4f3401a430 con 0x7f4f38108560 2024-04-23T18:08:43.562 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.898+0000 7f4f40317700 1 -- 172.21.15.50:0/2353360235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3819cce0 con 0x7f4f38108560 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.898+0000 7f4f2f7fe700 1 -- 172.21.15.50:0/2353360235 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f34019550 con 0x7f4f38108560 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.898+0000 7f4f40317700 1 -- 172.21.15.50:0/2353360235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f3819f9d0 con 0x7f4f38108560 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.898+0000 7f4f2f7fe700 1 -- 172.21.15.50:0/2353360235 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4f34019b90 con 0x7f4f38108560 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.898+0000 7f4f2f7fe700 1 -- 172.21.15.50:0/2353360235 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4f34024730 con 0x7f4f38108560 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.899+0000 7f4f40317700 1 -- 172.21.15.50:0/2353360235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f380640a0 con 0x7f4f38108560 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.901+0000 7f4f2f7fe700 1 -- 172.21.15.50:0/2353360235 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4f34032070 con 0x7f4f38108560 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.971+0000 7f4f40317700 1 -- 172.21.15.50:0/2353360235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f4f3810c8c0 con 0x7f4f38108560 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.972+0000 7f4f2f7fe700 1 -- 172.21.15.50:0/2353360235 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f4f3401a5e0 con 0x7f4f38108560 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.972+0000 7f4f40317700 1 -- 172.21.15.50:0/2353360235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 msgr2=0x7f4f3819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.972+0000 7f4f40317700 1 --2- 172.21.15.50:0/2353360235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 0x7f4f3819c510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009fb0 tx=0x7f4f34011940 comp rx=0 tx=0).stop 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.973+0000 7f4f40317700 1 -- 172.21.15.50:0/2353360235 shutdown_connections 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.973+0000 7f4f40317700 1 --2- 172.21.15.50:0/2353360235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f38108560 0x7f4f3819c510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.973+0000 7f4f40317700 1 -- 172.21.15.50:0/2353360235 >> 172.21.15.50:0/2353360235 conn(0x7f4f38103b40 msgr2=0x7f4f381046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:43.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.973+0000 7f4f40317700 1 -- 172.21.15.50:0/2353360235 shutdown_connections 2024-04-23T18:08:43.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:42.973+0000 7f4f40317700 1 -- 172.21.15.50:0/2353360235 wait complete. 2024-04-23T18:08:43.564 INFO:teuthology.orchestra.run.smithi050.stdout:Restarting the monitor... 2024-04-23T18:08:43.815 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:43 smithi050 systemd[1]: Stopping Ceph mon.a for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:08:44.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:43 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a[28813]: 2024-04-23T18:08:43.820+0000 7f8d47d5d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:08:44.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:43 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a[28813]: 2024-04-23T18:08:43.820+0000 7f8d47d5d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-23T18:08:44.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:43 smithi050 podman[29063]: 2024-04-23 18:08:43.995290497 +0000 UTC m=+0.260931085 container died 400ad3ad6a3664ed6d309d2adf573d57d30f2680c3d9719e5c18fdb447499aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2) 2024-04-23T18:08:44.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:44 smithi050 podman[29063]: 2024-04-23 18:08:44.278918275 +0000 UTC m=+0.544558856 container cleanup 400ad3ad6a3664ed6d309d2adf573d57d30f2680c3d9719e5c18fdb447499aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-23T18:08:44.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:44 smithi050 bash[29063]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a 2024-04-23T18:08:44.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:44 smithi050 podman[29077]: 2024-04-23 18:08:44.521610658 +0000 UTC m=+0.523004646 container remove 400ad3ad6a3664ed6d309d2adf573d57d30f2680c3d9719e5c18fdb447499aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:08:44.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:44 smithi050 systemd[1]: ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-23T18:08:44.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:44 smithi050 systemd[1]: Stopped Ceph mon.a for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:08:44.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:44 smithi050 systemd[1]: Starting Ceph mon.a for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:08:45.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:44 smithi050 podman[29169]: 2024-04-23 18:08:44.771924312 +0000 UTC m=+0.018350224 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:08:45.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 podman[29169]: 2024-04-23 18:08:45.647777323 +0000 UTC m=+0.894203234 container create b09dd59349eaaeb88989a44baf823c9759e8afb28e129f28a624fcc728688dbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-23T18:08:45.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 podman[29169]: 2024-04-23 18:08:45.816561749 +0000 UTC m=+1.062987661 container init b09dd59349eaaeb88989a44baf823c9759e8afb28e129f28a624fcc728688dbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416) 2024-04-23T18:08:45.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 podman[29169]: 2024-04-23 18:08:45.819669215 +0000 UTC m=+1.066095135 container start b09dd59349eaaeb88989a44baf823c9759e8afb28e129f28a624fcc728688dbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T18:08:45.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T18:08:45.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-23T18:08:45.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: pidfile_write: ignore empty --pid-file 2024-04-23T18:08:45.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: load: jerasure load: lrc 2024-04-23T18:08:45.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: RocksDB version: 7.9.2 2024-04-23T18:08:45.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Git sha 0 2024-04-23T18:08:45.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-23T18:08:45.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: DB SUMMARY 2024-04-23T18:08:45.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: DB Session ID: R6JGALNZ94ATW6JAKP8N 2024-04-23T18:08:45.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: CURRENT file: CURRENT 2024-04-23T18:08:45.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T18:08:45.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105438 ; 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.error_if_exists: 0 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.create_if_missing: 0 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.paranoid_checks: 1 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.env: 0x561b3baefcc0 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.info_log: 0x561b3e0edbe0 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.statistics: (nil) 2024-04-23T18:08:45.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.use_fsync: 0 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_log_file_size: 0 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.allow_fallocate: 1 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.use_direct_reads: 0 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.db_log_dir: 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.wal_dir: 2024-04-23T18:08:45.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.write_buffer_manager: 0x561b3e0810e0 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T18:08:45.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T18:08:45.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T18:08:45.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T18:08:45.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T18:08:45.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.unordered_write: 0 2024-04-23T18:08:45.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T18:08:45.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T18:08:45.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T18:08:45.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.row_cache: None 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.wal_filter: None 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.two_write_queues: 0 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.wal_compression: 0 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.atomic_flush: 0 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.log_readahead_size: 0 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T18:08:45.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_background_jobs: 2 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_background_compactions: -1 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_subcompactions: 1 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T18:08:45.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_open_files: -1 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_background_flushes: -1 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Compression algorithms supported: 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: kZSTD supported: 0 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: kXpressCompression supported: 0 2024-04-23T18:08:45.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: kZlibCompression supported: 1 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: kSnappyCompression supported: 1 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: kLZ4Compression supported: 1 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: kBZip2Compression supported: 0 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.merge_operator: 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_filter: None 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T18:08:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561b3e0eddc0) 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout: cache_index_and_filter_blocks: 1 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout: pin_top_level_index_and_filter: 1 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout: index_type: 0 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout: data_block_index_type: 0 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout: index_shortening: 1 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout: checksum: 4 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout: no_block_cache: 0 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout: block_cache: 0x561b3e0b31f0 2024-04-23T18:08:45.905 INFO:journalctl@ceph.mon.a.smithi050.stdout: block_cache_name: BinnedLRUCache 2024-04-23T18:08:45.906 INFO:journalctl@ceph.mon.a.smithi050.stdout: block_cache_options: 2024-04-23T18:08:45.906 INFO:journalctl@ceph.mon.a.smithi050.stdout: capacity : 536870912 2024-04-23T18:08:45.906 INFO:journalctl@ceph.mon.a.smithi050.stdout: num_shard_bits : 4 2024-04-23T18:08:45.906 INFO:journalctl@ceph.mon.a.smithi050.stdout: strict_capacity_limit : 0 2024-04-23T18:08:45.906 INFO:journalctl@ceph.mon.a.smithi050.stdout: high_pri_pool_ratio: 0.000 2024-04-23T18:08:45.906 INFO:journalctl@ceph.mon.a.smithi050.stdout: block_cache_compressed: (nil) 2024-04-23T18:08:45.906 INFO:journalctl@ceph.mon.a.smithi050.stdout: persistent_cache: (nil) 2024-04-23T18:08:45.906 INFO:journalctl@ceph.mon.a.smithi050.stdout: block_size: 4096 2024-04-23T18:08:45.906 INFO:journalctl@ceph.mon.a.smithi050.stdout: block_size_deviation: 10 2024-04-23T18:08:45.906 INFO:journalctl@ceph.mon.a.smithi050.stdout: block_restart_interval: 16 2024-04-23T18:08:45.906 INFO:journalctl@ceph.mon.a.smithi050.stdout: index_block_restart_interval: 1 2024-04-23T18:08:45.906 INFO:journalctl@ceph.mon.a.smithi050.stdout: metadata_block_size: 4096 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: partition_filters: 0 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: use_delta_encoding: 1 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: filter_policy: bloomfilter 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: whole_key_filtering: 1 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: verify_compression: 0 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: read_amp_bytes_per_bit: 0 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: format_version: 5 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: enable_index_compression: 1 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: block_align: 0 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: max_auto_readahead_size: 262144 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: prepopulate_block_cache: 0 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: initial_auto_readahead_size: 8192 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T18:08:45.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compression: NoCompression 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.num_levels: 7 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T18:08:45.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T18:08:45.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T18:08:45.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T18:08:45.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.inplace_update_support: 0 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.bloom_locality: 0 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.max_successive_merges: 0 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T18:08:45.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.ttl: 2592000 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.enable_blob_files: false 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.min_blob_size: 0 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T18:08:45.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8ff4c7be-1f0f-483e-acf7-8cd899b9c9c9 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895725850568, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895725851803, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100550, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98435, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14264, "raw_average_key_size": 49, "raw_value_size": 90689, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713895725, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8ff4c7be-1f0f-483e-acf7-8cd899b9c9c9", "db_session_id": "R6JGALNZ94ATW6JAKP8N", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895725851893, "job": 1, "event": "recovery_finished"} 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561b3e1c8000 2024-04-23T18:08:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: DB pointer 0x561b3e1b2000 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: ** DB Stats ** 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: ** Compaction Stats [default] ** 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T18:08:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout: L0 2/0 100.05 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: Sum 2/0 100.05 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: ** Compaction Stats [default] ** 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 117.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T18:08:45.916 INFO:journalctl@ceph.mon.a.smithi050.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout: Cumulative compaction: 0.00 GB write, 27.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout: Interval compaction: 0.00 GB write, 27.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout: Block cache BinnedLRUCache@0x561b3e0b31f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout: 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: starting mon.a rank 0 at public addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] at bind addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: mon.a@-1(???) e1 preinit fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: mon.a@-1(???).mds e1 new map 2024-04-23T18:08:45.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: mon.a@-1(???).mds e1 print_map 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout: e1 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout: legacy client fscid: -1 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout: 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout: No filesystems configured 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 bash[29169]: b09dd59349eaaeb88989a44baf823c9759e8afb28e129f28a624fcc728688dbc 2024-04-23T18:08:45.918 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 systemd[1]: Started Ceph mon.a for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:08:45.919 INFO:teuthology.orchestra.run.smithi050.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-23T18:08:46.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T18:08:46.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: monmap epoch 1 2024-04-23T18:08:46.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:46.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: last_changed 2024-04-23T18:08:34.626890+0000 2024-04-23T18:08:46.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: created 2024-04-23T18:08:34.626890+0000 2024-04-23T18:08:46.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: min_mon_release 19 (squid) 2024-04-23T18:08:46.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: election_strategy: 1 2024-04-23T18:08:46.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-23T18:08:46.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: fsmap 2024-04-23T18:08:46.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T18:08:46.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:45 smithi050 ceph-mon[29187]: mgrmap e1: no daemons active 2024-04-23T18:08:46.973 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.316+0000 7f4fcc8fa700 1 Processor -- start 2024-04-23T18:08:46.973 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.316+0000 7f4fcc8fa700 1 -- start start 2024-04-23T18:08:46.973 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.317+0000 7f4fcc8fa700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 0x7f4fc8107d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:46.973 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.317+0000 7f4fcc8fa700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc8108350 con 0x7f4fc8107960 2024-04-23T18:08:46.973 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.317+0000 7f4fc659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 0x7f4fc8107d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.317+0000 7f4fc659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 0x7f4fc8107d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48730/0 (socket says 172.21.15.50:48730) 2024-04-23T18:08:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.317+0000 7f4fc659c700 1 -- 172.21.15.50:0/1294466411 learned_addr learned my addr 172.21.15.50:0/1294466411 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.317+0000 7f4fc659c700 1 -- 172.21.15.50:0/1294466411 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc8108680 con 0x7f4fc8107960 2024-04-23T18:08:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.317+0000 7f4fc659c700 1 --2- 172.21.15.50:0/1294466411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 0x7f4fc8107d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4fb800bb40 tx=0x7f4fb800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f420835d366da5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.318+0000 7f4fc559a700 1 -- 172.21.15.50:0/1294466411 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fb800dbe0 con 0x7f4fc8107960 2024-04-23T18:08:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.318+0000 7f4fc559a700 1 -- 172.21.15.50:0/1294466411 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4fb80044c0 con 0x7f4fc8107960 2024-04-23T18:08:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.318+0000 7f4fc559a700 1 -- 172.21.15.50:0/1294466411 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fb8010470 con 0x7f4fc8107960 2024-04-23T18:08:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.318+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/1294466411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 msgr2=0x7f4fc8107d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.318+0000 7f4fcc8fa700 1 --2- 172.21.15.50:0/1294466411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 0x7f4fc8107d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4fb800bb40 tx=0x7f4fb800be50 comp rx=0 tx=0).stop 2024-04-23T18:08:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.318+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/1294466411 shutdown_connections 2024-04-23T18:08:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.318+0000 7f4fcc8fa700 1 --2- 172.21.15.50:0/1294466411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 0x7f4fc8107d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.318+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/1294466411 >> 172.21.15.50:0/1294466411 conn(0x7f4fc81021a0 msgr2=0x7f4fc81045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.318+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/1294466411 shutdown_connections 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.318+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/1294466411 wait complete. 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.319+0000 7f4fcc8fa700 1 Processor -- start 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.319+0000 7f4fcc8fa700 1 -- start start 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.319+0000 7f4fcc8fa700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 0x7f4fc819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.319+0000 7f4fcc8fa700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc81064c0 con 0x7f4fc8107960 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.320+0000 7f4fc659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 0x7f4fc819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.320+0000 7f4fc659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 0x7f4fc819a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48732/0 (socket says 172.21.15.50:48732) 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.320+0000 7f4fc659c700 1 -- 172.21.15.50:0/4243060037 learned_addr learned my addr 172.21.15.50:0/4243060037 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.320+0000 7f4fc659c700 1 -- 172.21.15.50:0/4243060037 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc819a6e0 con 0x7f4fc8107960 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.320+0000 7f4fc659c700 1 --2- 172.21.15.50:0/4243060037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 0x7f4fc819a1a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4fb800b340 tx=0x7f4fb8003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.320+0000 7f4fb77fe700 1 -- 172.21.15.50:0/4243060037 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fb8027070 con 0x7f4fc8107960 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.320+0000 7f4fb77fe700 1 -- 172.21.15.50:0/4243060037 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4fb8010bb0 con 0x7f4fc8107960 2024-04-23T18:08:46.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.320+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/4243060037 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc819a970 con 0x7f4fc8107960 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.320+0000 7f4fb77fe700 1 -- 172.21.15.50:0/4243060037 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fb8022ac0 con 0x7f4fc8107960 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.320+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/4243060037 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc819d660 con 0x7f4fc8107960 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.320+0000 7f4fb77fe700 1 -- 172.21.15.50:0/4243060037 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4fb802a070 con 0x7f4fc8107960 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.320+0000 7f4fb77fe700 1 -- 172.21.15.50:0/4243060037 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4fb801f070 con 0x7f4fc8107960 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.321+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/4243060037 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc80640a0 con 0x7f4fc8107960 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.324+0000 7f4fb77fe700 1 -- 172.21.15.50:0/4243060037 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4fb8031030 con 0x7f4fc8107960 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.395+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/4243060037 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f4fc8193700 con 0x7f4fc8107960 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.396+0000 7f4fb77fe700 1 -- 172.21.15.50:0/4243060037 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f4fc8193700 con 0x7f4fc8107960 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.397+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/4243060037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 msgr2=0x7f4fc819a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.397+0000 7f4fcc8fa700 1 --2- 172.21.15.50:0/4243060037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 0x7f4fc819a1a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4fb800b340 tx=0x7f4fb8003dc0 comp rx=0 tx=0).stop 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.397+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/4243060037 shutdown_connections 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.397+0000 7f4fcc8fa700 1 --2- 172.21.15.50:0/4243060037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fc8107960 0x7f4fc819a1a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.397+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/4243060037 >> 172.21.15.50:0/4243060037 conn(0x7f4fc81021a0 msgr2=0x7f4fc81907c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:46.977 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.397+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/4243060037 shutdown_connections 2024-04-23T18:08:46.977 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:46.397+0000 7f4fcc8fa700 1 -- 172.21.15.50:0/4243060037 wait complete. 2024-04-23T18:08:46.977 INFO:teuthology.orchestra.run.smithi050.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-23T18:08:46.999 INFO:teuthology.orchestra.run.smithi050.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:08:46.999 INFO:teuthology.orchestra.run.smithi050.stdout:Creating mgr... 2024-04-23T18:08:47.000 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-23T18:08:47.000 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-23T18:08:47.254 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.x 2024-04-23T18:08:47.255 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-23T18:08:47.409 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5d677974-019c-11ef-bc93-c7b262605968.target.wants/ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-5d677974-019c-11ef-bc93-c7b262605968@.service. 2024-04-23T18:08:47.646 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:47 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4243060037' entity='client.admin' 2024-04-23T18:08:47.647 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:47 smithi050 podman[29404]: 2024-04-23 18:08:47.555777779 +0000 UTC m=+0.018585065 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T18:08:47.929 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:47 smithi050 podman[29404]: 2024-04-23 18:08:47.860378616 +0000 UTC m=+0.323185877 container init 6cdd679662e2945e6caca10a6a1930bc5d391b218a03f36f075a6014369916a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:08:47.929 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:47 smithi050 podman[29404]: 2024-04-23 18:08:47.863094441 +0000 UTC m=+0.325901703 container start 6cdd679662e2945e6caca10a6a1930bc5d391b218a03f36f075a6014369916a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-23T18:08:47.942 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T18:08:47.942 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout disabled 2024-04-23T18:08:47.949 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T18:08:47.949 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-23T18:08:47.949 INFO:teuthology.orchestra.run.smithi050.stdout:firewalld.service is not enabled 2024-04-23T18:08:47.949 INFO:teuthology.orchestra.run.smithi050.stdout:Not possible to enable service . firewalld.service is not available 2024-04-23T18:08:47.958 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T18:08:47.958 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout disabled 2024-04-23T18:08:47.965 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T18:08:47.966 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-23T18:08:47.966 INFO:teuthology.orchestra.run.smithi050.stdout:firewalld.service is not enabled 2024-04-23T18:08:47.966 INFO:teuthology.orchestra.run.smithi050.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-23T18:08:47.966 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mgr to start... 2024-04-23T18:08:47.966 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mgr... 2024-04-23T18:08:48.195 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:47 smithi050 bash[29404]: 6cdd679662e2945e6caca10a6a1930bc5d391b218a03f36f075a6014369916a5 2024-04-23T18:08:48.195 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:47 smithi050 systemd[1]: Started Ceph mgr.x for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:08:48.195 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:48.019+0000 7f76e9718200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:08:48.491 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:48.213+0000 7f76e9718200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:08:48.491 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:48.319+0000 7f76e9718200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:08:48.810 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:48 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/681951536' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T18:08:48.810 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:48.651+0000 7f76e9718200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:08:48.810 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:48.744+0000 7f76e9718200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:08:49.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:49.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:08:49.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:08:49.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T18:08:49.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T18:08:49.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T18:08:49.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T18:08:49.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:49.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T18:08:49.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T18:08:49.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 0 2024-04-23T18:08:49.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T18:08:49.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T18:08:49.093 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T18:08:49.093 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T18:08:49.093 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:49.094 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T18:08:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:49.096 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T18:08:37.570444+0000", 2024-04-23T18:08:49.096 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:08:49.096 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:49.096 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T18:08:49.096 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:08:49.096 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.415+0000 7f024066e700 1 Processor -- start 2024-04-23T18:08:49.096 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.415+0000 7f024066e700 1 -- start start 2024-04-23T18:08:49.096 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.416+0000 7f024066e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 0x7f0238107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:49.096 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.416+0000 7f024066e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0238107e30 con 0x7f0238107440 2024-04-23T18:08:49.096 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.416+0000 7f023e40a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 0x7f0238107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:49.096 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.416+0000 7f023e40a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 0x7f0238107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48754/0 (socket says 172.21.15.50:48754) 2024-04-23T18:08:49.096 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.416+0000 7f023e40a700 1 -- 172.21.15.50:0/3236543092 learned_addr learned my addr 172.21.15.50:0/3236543092 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.417+0000 7f023e40a700 1 -- 172.21.15.50:0/3236543092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0238108640 con 0x7f0238107440 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.417+0000 7f023e40a700 1 --2- 172.21.15.50:0/3236543092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 0x7f0238107860 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f02340098a0 tx=0x7f0234009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97c0485742d60b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.417+0000 7f023d408700 1 -- 172.21.15.50:0/3236543092 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0234004160 con 0x7f0238107440 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.417+0000 7f023d408700 1 -- 172.21.15.50:0/3236543092 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0234004300 con 0x7f0238107440 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.417+0000 7f023d408700 1 -- 172.21.15.50:0/3236543092 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0234011af0 con 0x7f0238107440 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.417+0000 7f024066e700 1 -- 172.21.15.50:0/3236543092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 msgr2=0x7f0238107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.417+0000 7f024066e700 1 --2- 172.21.15.50:0/3236543092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 0x7f0238107860 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f02340098a0 tx=0x7f0234009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.418+0000 7f024066e700 1 -- 172.21.15.50:0/3236543092 shutdown_connections 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.418+0000 7f024066e700 1 --2- 172.21.15.50:0/3236543092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 0x7f0238107860 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.418+0000 7f024066e700 1 -- 172.21.15.50:0/3236543092 >> 172.21.15.50:0/3236543092 conn(0x7f0238102a40 msgr2=0x7f0238104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.418+0000 7f024066e700 1 -- 172.21.15.50:0/3236543092 shutdown_connections 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.418+0000 7f024066e700 1 -- 172.21.15.50:0/3236543092 wait complete. 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.418+0000 7f024066e700 1 Processor -- start 2024-04-23T18:08:49.097 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.418+0000 7f024066e700 1 -- start start 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.418+0000 7f024066e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 0x7f023819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.418+0000 7f024066e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f023810b9f0 con 0x7f0238107440 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.419+0000 7f023e40a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 0x7f023819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.419+0000 7f023e40a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 0x7f023819c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48766/0 (socket says 172.21.15.50:48766) 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.419+0000 7f023e40a700 1 -- 172.21.15.50:0/681951536 learned_addr learned my addr 172.21.15.50:0/681951536 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.419+0000 7f023e40a700 1 -- 172.21.15.50:0/681951536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023819ca60 con 0x7f0238107440 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.419+0000 7f023e40a700 1 --2- 172.21.15.50:0/681951536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 0x7f023819c520 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0234003f40 tx=0x7f0234011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.419+0000 7f022f7fe700 1 -- 172.21.15.50:0/681951536 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02340038b0 con 0x7f0238107440 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.419+0000 7f024066e700 1 -- 172.21.15.50:0/681951536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f023819ccf0 con 0x7f0238107440 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.419+0000 7f022f7fe700 1 -- 172.21.15.50:0/681951536 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0234003a50 con 0x7f0238107440 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.419+0000 7f022f7fe700 1 -- 172.21.15.50:0/681951536 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f023401a5c0 con 0x7f0238107440 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.419+0000 7f024066e700 1 -- 172.21.15.50:0/681951536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023819f990 con 0x7f0238107440 2024-04-23T18:08:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.419+0000 7f022f7fe700 1 -- 172.21.15.50:0/681951536 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f023401ac50 con 0x7f0238107440 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.419+0000 7f022f7fe700 1 -- 172.21.15.50:0/681951536 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0234019700 con 0x7f0238107440 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.420+0000 7f024066e700 1 -- 172.21.15.50:0/681951536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02380640a0 con 0x7f0238107440 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.422+0000 7f022f7fe700 1 -- 172.21.15.50:0/681951536 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f02340199a0 con 0x7f0238107440 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.496+0000 7f024066e700 1 -- 172.21.15.50:0/681951536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f023802e180 con 0x7f0238107440 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.497+0000 7f022f7fe700 1 -- 172.21.15.50:0/681951536 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0234019b80 con 0x7f0238107440 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.498+0000 7f024066e700 1 -- 172.21.15.50:0/681951536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 msgr2=0x7f023819c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.498+0000 7f024066e700 1 --2- 172.21.15.50:0/681951536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 0x7f023819c520 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0234003f40 tx=0x7f0234011890 comp rx=0 tx=0).stop 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.498+0000 7f024066e700 1 -- 172.21.15.50:0/681951536 shutdown_connections 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.498+0000 7f024066e700 1 --2- 172.21.15.50:0/681951536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0238107440 0x7f023819c520 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.498+0000 7f024066e700 1 -- 172.21.15.50:0/681951536 >> 172.21.15.50:0/681951536 conn(0x7f0238102a40 msgr2=0x7f02381038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.498+0000 7f024066e700 1 -- 172.21.15.50:0/681951536 shutdown_connections 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:48.498+0000 7f024066e700 1 -- 172.21.15.50:0/681951536 wait complete. 2024-04-23T18:08:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:mgr not available, waiting (1/15)... 2024-04-23T18:08:50.352 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:50.006+0000 7f76e9718200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:08:50.352 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:50.182+0000 7f76e9718200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:08:50.352 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:50.357+0000 7f76e9718200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:08:50.906 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:50.660+0000 7f76e9718200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:08:50.907 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:50.759+0000 7f76e9718200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:08:51.177 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:50.975+0000 7f76e9718200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:08:51.438 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:51 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:51.182+0000 7f76e9718200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:08:51.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:51 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2690418253' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T18:08:52.184 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:51 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:51.838+0000 7f76e9718200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:08:52.196 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:52.196 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:08:52.196 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:08:52.196 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 0 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T18:08:52.197 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:52.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T18:08:52.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T18:08:37.570444+0000", 2024-04-23T18:08:52.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.488+0000 7fd5b896f700 1 Processor -- start 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.488+0000 7fd5b896f700 1 -- start start 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.488+0000 7fd5b896f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 0x7fd5b407b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.488+0000 7fd5b896f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b407b7d0 con 0x7fd5b407cda0 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.488+0000 7fd5b259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 0x7fd5b407b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.489+0000 7fd5b259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 0x7fd5b407b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48778/0 (socket says 172.21.15.50:48778) 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.489+0000 7fd5b259c700 1 -- 172.21.15.50:0/826284376 learned_addr learned my addr 172.21.15.50:0/826284376 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.489+0000 7fd5b259c700 1 -- 172.21.15.50:0/826284376 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b407b950 con 0x7fd5b407cda0 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.489+0000 7fd5b259c700 1 --2- 172.21.15.50:0/826284376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 0x7fd5b407b200 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40098a0 tx=0x7fd5a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f5b0e048896024 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.489+0000 7fd5b159a700 1 -- 172.21.15.50:0/826284376 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5a4004160 con 0x7fd5b407cda0 2024-04-23T18:08:52.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.489+0000 7fd5b159a700 1 -- 172.21.15.50:0/826284376 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd5a4004300 con 0x7fd5b407cda0 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.489+0000 7fd5b159a700 1 -- 172.21.15.50:0/826284376 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5a4011af0 con 0x7fd5b407cda0 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.490+0000 7fd5b896f700 1 -- 172.21.15.50:0/826284376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 msgr2=0x7fd5b407b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.490+0000 7fd5b896f700 1 --2- 172.21.15.50:0/826284376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 0x7fd5b407b200 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40098a0 tx=0x7fd5a4009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.490+0000 7fd5b896f700 1 -- 172.21.15.50:0/826284376 shutdown_connections 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.490+0000 7fd5b896f700 1 --2- 172.21.15.50:0/826284376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 0x7fd5b407b200 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.490+0000 7fd5b896f700 1 -- 172.21.15.50:0/826284376 >> 172.21.15.50:0/826284376 conn(0x7fd5b4103b50 msgr2=0x7fd5b4105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.490+0000 7fd5b896f700 1 -- 172.21.15.50:0/826284376 shutdown_connections 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.490+0000 7fd5b896f700 1 -- 172.21.15.50:0/826284376 wait complete. 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.491+0000 7fd5b896f700 1 Processor -- start 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.491+0000 7fd5b896f700 1 -- start start 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.492+0000 7fd5b896f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 0x7fd5b419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.492+0000 7fd5b896f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b410eae0 con 0x7fd5b407cda0 2024-04-23T18:08:52.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.492+0000 7fd5b259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 0x7fd5b419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.492+0000 7fd5b259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 0x7fd5b419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48784/0 (socket says 172.21.15.50:48784) 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.492+0000 7fd5b259c700 1 -- 172.21.15.50:0/2690418253 learned_addr learned my addr 172.21.15.50:0/2690418253 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.492+0000 7fd5b259c700 1 -- 172.21.15.50:0/2690418253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b419ebe0 con 0x7fd5b407cda0 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.492+0000 7fd5b259c700 1 --2- 172.21.15.50:0/2690418253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 0x7fd5b419e6a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4000c00 tx=0x7fd5a4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.492+0000 7fd5a37fe700 1 -- 172.21.15.50:0/2690418253 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5a4003b20 con 0x7fd5b407cda0 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.493+0000 7fd5b896f700 1 -- 172.21.15.50:0/2690418253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b419ee70 con 0x7fd5b407cda0 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.493+0000 7fd5a37fe700 1 -- 172.21.15.50:0/2690418253 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd5a401a8c0 con 0x7fd5b407cda0 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.493+0000 7fd5a37fe700 1 -- 172.21.15.50:0/2690418253 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5a4024940 con 0x7fd5b407cda0 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.493+0000 7fd5b896f700 1 -- 172.21.15.50:0/2690418253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b41a1b60 con 0x7fd5b407cda0 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.493+0000 7fd5a37fe700 1 -- 172.21.15.50:0/2690418253 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd5a4022020 con 0x7fd5b407cda0 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.494+0000 7fd5a37fe700 1 -- 172.21.15.50:0/2690418253 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd5a401a430 con 0x7fd5b407cda0 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.494+0000 7fd5b896f700 1 -- 172.21.15.50:0/2690418253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b40640a0 con 0x7fd5b407cda0 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.496+0000 7fd5a37fe700 1 -- 172.21.15.50:0/2690418253 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd5a401a630 con 0x7fd5b407cda0 2024-04-23T18:08:52.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.570+0000 7fd5b896f700 1 -- 172.21.15.50:0/2690418253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd5b41a1dd0 con 0x7fd5b407cda0 2024-04-23T18:08:52.204 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.570+0000 7fd5a37fe700 1 -- 172.21.15.50:0/2690418253 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd5a401aa70 con 0x7fd5b407cda0 2024-04-23T18:08:52.204 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.571+0000 7fd5b896f700 1 -- 172.21.15.50:0/2690418253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 msgr2=0x7fd5b419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:52.204 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.571+0000 7fd5b896f700 1 --2- 172.21.15.50:0/2690418253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 0x7fd5b419e6a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4000c00 tx=0x7fd5a4011b70 comp rx=0 tx=0).stop 2024-04-23T18:08:52.204 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.571+0000 7fd5b896f700 1 -- 172.21.15.50:0/2690418253 shutdown_connections 2024-04-23T18:08:52.204 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.571+0000 7fd5b896f700 1 --2- 172.21.15.50:0/2690418253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5b407cda0 0x7fd5b419e6a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:52.204 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.571+0000 7fd5b896f700 1 -- 172.21.15.50:0/2690418253 >> 172.21.15.50:0/2690418253 conn(0x7fd5b4103b50 msgr2=0x7fd5b4104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:52.204 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.571+0000 7fd5b896f700 1 -- 172.21.15.50:0/2690418253 shutdown_connections 2024-04-23T18:08:52.204 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:51.571+0000 7fd5b896f700 1 -- 172.21.15.50:0/2690418253 wait complete. 2024-04-23T18:08:52.204 INFO:teuthology.orchestra.run.smithi050.stdout:mgr not available, waiting (2/15)... 2024-04-23T18:08:52.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:52.278+0000 7f76e9718200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:08:52.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:52.377+0000 7f76e9718200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:08:52.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:52.586+0000 7f76e9718200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:08:53.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:53.132+0000 7f76e9718200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:08:53.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:53.226+0000 7f76e9718200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:08:53.829 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:53.477+0000 7f76e9718200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:08:53.829 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:53.616+0000 7f76e9718200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:08:53.829 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:53.714+0000 7f76e9718200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:08:53.829 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:53.834+0000 7f76e9718200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:08:54.400 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:54 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:54.146+0000 7f76e9718200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:08:54.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:54 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4262673399' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T18:08:54.946 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:54 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:54.794+0000 7f76e9718200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:08:55.229 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:55.229 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:08:55.229 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:08:55.229 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T18:08:55.229 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T18:08:55.229 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T18:08:55.229 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T18:08:55.229 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:55.229 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T18:08:55.229 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 0 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T18:08:55.230 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:55.231 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T18:08:55.231 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T18:08:55.231 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T18:08:55.239 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T18:08:55.239 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T18:08:55.239 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T18:08:55.240 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:55.240 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T18:08:55.240 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T18:08:55.240 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T18:08:55.240 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T18:08:55.240 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T18:08:55.240 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T18:08:55.240 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T18:08:55.240 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T18:08:55.240 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T18:08:55.240 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T18:08:55.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:08:55.242 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:08:55.242 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:55.242 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T18:08:55.242 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:55.242 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T18:08:37.570444+0000", 2024-04-23T18:08:55.242 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:08:55.242 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:55.242 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T18:08:55.242 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:08:55.242 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.610+0000 7f718e01e700 1 Processor -- start 2024-04-23T18:08:55.242 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.610+0000 7f718e01e700 1 -- start start 2024-04-23T18:08:55.243 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.610+0000 7f718e01e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 0x7f7188107250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:55.243 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.610+0000 7f718e01e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f718807be40 con 0x7f7188106e30 2024-04-23T18:08:55.243 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.611+0000 7f71877fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 0x7f7188107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:55.243 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.611+0000 7f71877fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 0x7f7188107250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34030/0 (socket says 172.21.15.50:34030) 2024-04-23T18:08:55.243 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.611+0000 7f71877fe700 1 -- 172.21.15.50:0/3446205656 learned_addr learned my addr 172.21.15.50:0/3446205656 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:55.243 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.611+0000 7f71877fe700 1 -- 172.21.15.50:0/3446205656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7188107790 con 0x7f7188106e30 2024-04-23T18:08:55.243 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.611+0000 7f71877fe700 1 --2- 172.21.15.50:0/3446205656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 0x7f7188107250 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f71780098a0 tx=0x7f7178009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941d0d5dae3e95c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:55.243 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.612+0000 7f71867fc700 1 -- 172.21.15.50:0/3446205656 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7178004160 con 0x7f7188106e30 2024-04-23T18:08:55.243 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.612+0000 7f71867fc700 1 -- 172.21.15.50:0/3446205656 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7178004300 con 0x7f7188106e30 2024-04-23T18:08:55.243 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.612+0000 7f71867fc700 1 -- 172.21.15.50:0/3446205656 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7178011af0 con 0x7f7188106e30 2024-04-23T18:08:55.243 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.613+0000 7f718e01e700 1 -- 172.21.15.50:0/3446205656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 msgr2=0x7f7188107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.613+0000 7f718e01e700 1 --2- 172.21.15.50:0/3446205656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 0x7f7188107250 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f71780098a0 tx=0x7f7178009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.613+0000 7f718e01e700 1 -- 172.21.15.50:0/3446205656 shutdown_connections 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.613+0000 7f718e01e700 1 --2- 172.21.15.50:0/3446205656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 0x7f7188107250 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.613+0000 7f718e01e700 1 -- 172.21.15.50:0/3446205656 >> 172.21.15.50:0/3446205656 conn(0x7f7188102a50 msgr2=0x7f7188104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.613+0000 7f718e01e700 1 -- 172.21.15.50:0/3446205656 shutdown_connections 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.613+0000 7f718e01e700 1 -- 172.21.15.50:0/3446205656 wait complete. 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.613+0000 7f718e01e700 1 Processor -- start 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.614+0000 7f718e01e700 1 -- start start 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.614+0000 7f718e01e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 0x7f718819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.614+0000 7f718e01e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f71881087c0 con 0x7f7188106e30 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.614+0000 7f71877fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 0x7f718819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.614+0000 7f71877fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 0x7f718819e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34040/0 (socket says 172.21.15.50:34040) 2024-04-23T18:08:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.614+0000 7f71877fe700 1 -- 172.21.15.50:0/4262673399 learned_addr learned my addr 172.21.15.50:0/4262673399 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.614+0000 7f71877fe700 1 -- 172.21.15.50:0/4262673399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718819ebe0 con 0x7f7188106e30 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.615+0000 7f71877fe700 1 --2- 172.21.15.50:0/4262673399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 0x7f718819e6a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7178009fb0 tx=0x7f7178011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.615+0000 7f7184ff9700 1 -- 172.21.15.50:0/4262673399 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71780036a0 con 0x7f7188106e30 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.615+0000 7f7184ff9700 1 -- 172.21.15.50:0/4262673399 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f717801a430 con 0x7f7188106e30 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.615+0000 7f718e01e700 1 -- 172.21.15.50:0/4262673399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f718819ee70 con 0x7f7188106e30 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.615+0000 7f718e01e700 1 -- 172.21.15.50:0/4262673399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71881a1b60 con 0x7f7188106e30 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.615+0000 7f7184ff9700 1 -- 172.21.15.50:0/4262673399 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7178019550 con 0x7f7188106e30 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.615+0000 7f7184ff9700 1 -- 172.21.15.50:0/4262673399 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7178019b90 con 0x7f7188106e30 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.615+0000 7f7184ff9700 1 -- 172.21.15.50:0/4262673399 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7178024730 con 0x7f7188106e30 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.616+0000 7f718e01e700 1 -- 172.21.15.50:0/4262673399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7168005360 con 0x7f7188106e30 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.618+0000 7f7184ff9700 1 -- 172.21.15.50:0/4262673399 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f717801a5e0 con 0x7f7188106e30 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.692+0000 7f718e01e700 1 -- 172.21.15.50:0/4262673399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7168005920 con 0x7f7188106e30 2024-04-23T18:08:55.245 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.693+0000 7f7184ff9700 1 -- 172.21.15.50:0/4262673399 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f717801a8e0 con 0x7f7188106e30 2024-04-23T18:08:55.246 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.694+0000 7f718e01e700 1 -- 172.21.15.50:0/4262673399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 msgr2=0x7f718819e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:55.246 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.694+0000 7f718e01e700 1 --2- 172.21.15.50:0/4262673399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 0x7f718819e6a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7178009fb0 tx=0x7f7178011940 comp rx=0 tx=0).stop 2024-04-23T18:08:55.246 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.694+0000 7f718e01e700 1 -- 172.21.15.50:0/4262673399 shutdown_connections 2024-04-23T18:08:55.246 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.694+0000 7f718e01e700 1 --2- 172.21.15.50:0/4262673399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7188106e30 0x7f718819e6a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:55.246 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.694+0000 7f718e01e700 1 -- 172.21.15.50:0/4262673399 >> 172.21.15.50:0/4262673399 conn(0x7f7188102a50 msgr2=0x7f7188195340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:55.246 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.694+0000 7f718e01e700 1 -- 172.21.15.50:0/4262673399 shutdown_connections 2024-04-23T18:08:55.246 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:54.694+0000 7f718e01e700 1 -- 172.21.15.50:0/4262673399 wait complete. 2024-04-23T18:08:55.246 INFO:teuthology.orchestra.run.smithi050.stdout:mgr not available, waiting (3/15)... 2024-04-23T18:08:55.296 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:08:55.019+0000 7f76e9718200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:08:56.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: Activating manager daemon x 2024-04-23T18:08:56.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: mgrmap e2: x(active, starting, since 0.00323013s) 2024-04-23T18:08:56.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: from='mgr.14100 172.21.15.50:0/1949263856' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:08:56.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: from='mgr.14100 172.21.15.50:0/1949263856' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:08:56.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: from='mgr.14100 172.21.15.50:0/1949263856' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:08:56.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: from='mgr.14100 172.21.15.50:0/1949263856' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:08:56.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: from='mgr.14100 172.21.15.50:0/1949263856' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:08:56.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: Manager daemon x is now available 2024-04-23T18:08:56.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: from='mgr.14100 172.21.15.50:0/1949263856' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:08:56.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: from='mgr.14100 172.21.15.50:0/1949263856' entity='mgr.x' 2024-04-23T18:08:56.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: from='mgr.14100 172.21.15.50:0/1949263856' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:08:56.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: from='mgr.14100 172.21.15.50:0/1949263856' entity='mgr.x' 2024-04-23T18:08:56.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:55 smithi050 ceph-mon[29187]: from='mgr.14100 172.21.15.50:0/1949263856' entity='mgr.x' 2024-04-23T18:08:57.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:57 smithi050 ceph-mon[29187]: mgrmap e3: x(active, since 1.00722s) 2024-04-23T18:08:58.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:58 smithi050 ceph-mon[29187]: mgrmap e4: x(active, since 2s) 2024-04-23T18:08:58.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:58 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2206969718' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T18:08:58.501 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:58.501 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:08:58.501 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:08:58.501 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T18:08:58.501 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T18:08:58.501 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 0 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T18:08:58.503 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T18:08:58.503 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:58.503 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T18:08:58.503 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:58.503 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T18:08:58.503 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T18:08:58.504 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T18:08:58.504 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T18:08:58.504 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T18:08:58.504 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T18:08:58.504 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:58.504 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T18:08:58.504 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T18:08:58.504 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T18:08:58.504 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T18:08:58.504 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T18:08:58.505 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T18:08:58.505 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T18:08:58.505 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T18:08:58.505 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T18:08:58.505 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:58.505 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T18:08:58.505 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:58.505 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T18:08:58.505 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T18:08:58.505 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:58.506 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T18:08:58.506 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T18:08:58.506 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T18:08:58.506 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T18:08:58.506 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T18:08:58.506 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T18:08:58.506 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T18:08:37.570444+0000", 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.694+0000 7f6bb0fed700 1 Processor -- start 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.694+0000 7f6bb0fed700 1 -- start start 2024-04-23T18:08:58.507 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.694+0000 7f6bb0fed700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 0x7f6bac1071e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.694+0000 7f6bb0fed700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac07be80 con 0x7f6bac106dc0 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.695+0000 7f6baa59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 0x7f6bac1071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.695+0000 7f6baa59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 0x7f6bac1071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34116/0 (socket says 172.21.15.50:34116) 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.695+0000 7f6baa59c700 1 -- 172.21.15.50:0/2580039030 learned_addr learned my addr 172.21.15.50:0/2580039030 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.695+0000 7f6baa59c700 1 -- 172.21.15.50:0/2580039030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bac107720 con 0x7f6bac106dc0 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.695+0000 7f6baa59c700 1 --2- 172.21.15.50:0/2580039030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 0x7f6bac1071e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c0098a0 tx=0x7f6b9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa07f09f36785ea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.695+0000 7f6ba959a700 1 -- 172.21.15.50:0/2580039030 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b9c004160 con 0x7f6bac106dc0 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.695+0000 7f6ba959a700 1 -- 172.21.15.50:0/2580039030 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6b9c004300 con 0x7f6bac106dc0 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.695+0000 7f6ba959a700 1 -- 172.21.15.50:0/2580039030 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b9c011af0 con 0x7f6bac106dc0 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.696+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2580039030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 msgr2=0x7f6bac1071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.696+0000 7f6bb0fed700 1 --2- 172.21.15.50:0/2580039030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 0x7f6bac1071e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c0098a0 tx=0x7f6b9c009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.696+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2580039030 shutdown_connections 2024-04-23T18:08:58.508 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.696+0000 7f6bb0fed700 1 --2- 172.21.15.50:0/2580039030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 0x7f6bac1071e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.696+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2580039030 >> 172.21.15.50:0/2580039030 conn(0x7f6bac102a00 msgr2=0x7f6bac104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.696+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2580039030 shutdown_connections 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.696+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2580039030 wait complete. 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.697+0000 7f6bb0fed700 1 Processor -- start 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.697+0000 7f6bb0fed700 1 -- start start 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.697+0000 7f6bb0fed700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 0x7f6bac19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.697+0000 7f6bb0fed700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac108750 con 0x7f6bac106dc0 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.697+0000 7f6baa59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 0x7f6bac19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.697+0000 7f6baa59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 0x7f6bac19e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34128/0 (socket says 172.21.15.50:34128) 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.697+0000 7f6baa59c700 1 -- 172.21.15.50:0/2206969718 learned_addr learned my addr 172.21.15.50:0/2206969718 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.698+0000 7f6baa59c700 1 -- 172.21.15.50:0/2206969718 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bac19ebf0 con 0x7f6bac106dc0 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.698+0000 7f6baa59c700 1 --2- 172.21.15.50:0/2206969718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 0x7f6bac19e6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c011ac0 tx=0x7f6b9c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.698+0000 7f6b9b7fe700 1 -- 172.21.15.50:0/2206969718 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b9c003970 con 0x7f6bac106dc0 2024-04-23T18:08:58.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.698+0000 7f6b9b7fe700 1 -- 172.21.15.50:0/2206969718 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6b9c003b10 con 0x7f6bac106dc0 2024-04-23T18:08:58.510 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.698+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2206969718 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bac19ee80 con 0x7f6bac106dc0 2024-04-23T18:08:58.510 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.698+0000 7f6b9b7fe700 1 -- 172.21.15.50:0/2206969718 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b9c024610 con 0x7f6bac106dc0 2024-04-23T18:08:58.510 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.698+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2206969718 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bac1a1b70 con 0x7f6bac106dc0 2024-04-23T18:08:58.510 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.699+0000 7f6b9b7fe700 1 -- 172.21.15.50:0/2206969718 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f6b9c024840 con 0x7f6bac106dc0 2024-04-23T18:08:58.510 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.699+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2206969718 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bac0640a0 con 0x7f6bac106dc0 2024-04-23T18:08:58.510 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.699+0000 7f6b9b7fe700 1 --2- 172.21.15.50:0/2206969718 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f6b940395b0 0x7f6b9403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:58.510 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.699+0000 7f6b9b7fe700 1 -- 172.21.15.50:0/2206969718 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6b9c04d880 con 0x7f6bac106dc0 2024-04-23T18:08:58.510 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.699+0000 7f6ba9d9b700 1 --2- 172.21.15.50:0/2206969718 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f6b940395b0 0x7f6b9403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:58.510 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.700+0000 7f6ba9d9b700 1 --2- 172.21.15.50:0/2206969718 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f6b940395b0 0x7f6b9403ba70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0006fd0 tx=0x7f6ba0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:58.510 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.705+0000 7f6b9b7fe700 1 -- 172.21.15.50:0/2206969718 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b9c0172b0 con 0x7f6bac106dc0 2024-04-23T18:08:58.510 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.998+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2206969718 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6bac0008d0 con 0x7f6bac106dc0 2024-04-23T18:08:58.510 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:57.998+0000 7f6b9b7fe700 1 -- 172.21.15.50:0/2206969718 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f6b9c04d2a0 con 0x7f6bac106dc0 2024-04-23T18:08:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.000+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2206969718 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f6b940395b0 msgr2=0x7f6b9403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.000+0000 7f6bb0fed700 1 --2- 172.21.15.50:0/2206969718 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f6b940395b0 0x7f6b9403ba70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0006fd0 tx=0x7f6ba0006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.000+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2206969718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 msgr2=0x7f6bac19e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.000+0000 7f6bb0fed700 1 --2- 172.21.15.50:0/2206969718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 0x7f6bac19e6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c011ac0 tx=0x7f6b9c011ba0 comp rx=0 tx=0).stop 2024-04-23T18:08:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.000+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2206969718 shutdown_connections 2024-04-23T18:08:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.000+0000 7f6bb0fed700 1 --2- 172.21.15.50:0/2206969718 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f6b940395b0 0x7f6b9403ba70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.000+0000 7f6bb0fed700 1 --2- 172.21.15.50:0/2206969718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bac106dc0 0x7f6bac19e6b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.000+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2206969718 >> 172.21.15.50:0/2206969718 conn(0x7f6bac102a00 msgr2=0x7f6bac195370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.000+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2206969718 shutdown_connections 2024-04-23T18:08:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.000+0000 7f6bb0fed700 1 -- 172.21.15.50:0/2206969718 wait complete. 2024-04-23T18:08:58.511 INFO:teuthology.orchestra.run.smithi050.stdout:mgr is available 2024-04-23T18:08:59.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:08:59 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/715185957' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T18:08:59.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:59.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [global] 2024-04-23T18:08:59.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout fsid = 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:08:59.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-23T18:08:59.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-23T18:08:59.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-23T18:08:59.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-23T18:08:59.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-23T18:08:59.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:59.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-23T18:08:59.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-23T18:08:59.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [osd] 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.944+0000 7ffa4d8d7700 1 Processor -- start 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.944+0000 7ffa4d8d7700 1 -- start start 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.944+0000 7ffa4d8d7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 0x7ffa4810b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.944+0000 7ffa4d8d7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48063d60 con 0x7ffa48108fb0 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.945+0000 7ffa46ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 0x7ffa4810b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.945+0000 7ffa46ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 0x7ffa4810b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34140/0 (socket says 172.21.15.50:34140) 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.945+0000 7ffa46ffd700 1 -- 172.21.15.50:0/4236774372 learned_addr learned my addr 172.21.15.50:0/4236774372 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.945+0000 7ffa46ffd700 1 -- 172.21.15.50:0/4236774372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4810b8e0 con 0x7ffa48108fb0 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.945+0000 7ffa46ffd700 1 --2- 172.21.15.50:0/4236774372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 0x7ffa4810b3a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffa300098a0 tx=0x7ffa30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f02ad2bd564fac6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.945+0000 7ffa45ffb700 1 -- 172.21.15.50:0/4236774372 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa30004160 con 0x7ffa48108fb0 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.945+0000 7ffa45ffb700 1 -- 172.21.15.50:0/4236774372 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ffa30004300 con 0x7ffa48108fb0 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.945+0000 7ffa45ffb700 1 -- 172.21.15.50:0/4236774372 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa30011af0 con 0x7ffa48108fb0 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.946+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/4236774372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 msgr2=0x7ffa4810b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.946+0000 7ffa4d8d7700 1 --2- 172.21.15.50:0/4236774372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 0x7ffa4810b3a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffa300098a0 tx=0x7ffa30009bb0 comp rx=0 tx=0).stop 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.946+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/4236774372 shutdown_connections 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.946+0000 7ffa4d8d7700 1 --2- 172.21.15.50:0/4236774372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 0x7ffa4810b3a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.946+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/4236774372 >> 172.21.15.50:0/4236774372 conn(0x7ffa48102a40 msgr2=0x7ffa48104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.946+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/4236774372 shutdown_connections 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.946+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/4236774372 wait complete. 2024-04-23T18:08:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.947+0000 7ffa4d8d7700 1 Processor -- start 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.947+0000 7ffa4d8d7700 1 -- start start 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.947+0000 7ffa4d8d7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 0x7ffa4819c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.947+0000 7ffa4d8d7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4810c8b0 con 0x7ffa48108fb0 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.947+0000 7ffa46ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 0x7ffa4819c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.947+0000 7ffa46ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 0x7ffa4819c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34148/0 (socket says 172.21.15.50:34148) 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.947+0000 7ffa46ffd700 1 -- 172.21.15.50:0/715185957 learned_addr learned my addr 172.21.15.50:0/715185957 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.947+0000 7ffa46ffd700 1 -- 172.21.15.50:0/715185957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4819c990 con 0x7ffa48108fb0 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.947+0000 7ffa46ffd700 1 --2- 172.21.15.50:0/715185957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 0x7ffa4819c450 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffa30011ac0 tx=0x7ffa30011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.948+0000 7ffa4c8d5700 1 -- 172.21.15.50:0/715185957 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa30003b20 con 0x7ffa48108fb0 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.948+0000 7ffa4c8d5700 1 -- 172.21.15.50:0/715185957 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ffa3001a8c0 con 0x7ffa48108fb0 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.948+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/715185957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4819cc20 con 0x7ffa48108fb0 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.948+0000 7ffa4c8d5700 1 -- 172.21.15.50:0/715185957 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa300248d0 con 0x7ffa48108fb0 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.948+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/715185957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa4819f910 con 0x7ffa48108fb0 2024-04-23T18:08:59.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.948+0000 7ffa4c8d5700 1 -- 172.21.15.50:0/715185957 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7ffa30022020 con 0x7ffa48108fb0 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.949+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/715185957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa480640a0 con 0x7ffa48108fb0 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.949+0000 7ffa4c8d5700 1 --2- 172.21.15.50:0/715185957 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7ffa340395b0 0x7ffa3403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.949+0000 7ffa4c8d5700 1 -- 172.21.15.50:0/715185957 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ffa3004d710 con 0x7ffa48108fb0 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.949+0000 7ffa467fc700 1 --2- 172.21.15.50:0/715185957 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7ffa340395b0 0x7ffa3403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.949+0000 7ffa467fc700 1 --2- 172.21.15.50:0/715185957 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7ffa340395b0 0x7ffa3403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffa38006fd0 tx=0x7ffa38006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:58.954+0000 7ffa4c8d5700 1 -- 172.21.15.50:0/715185957 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa3004d470 con 0x7ffa48108fb0 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:59.160+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/715185957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7ffa4819fae0 con 0x7ffa48108fb0 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:59.161+0000 7ffa4c8d5700 1 -- 172.21.15.50:0/715185957 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+436 (secure 0 0 0) 0x7ffa4819fae0 con 0x7ffa48108fb0 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:59.164+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/715185957 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7ffa340395b0 msgr2=0x7ffa3403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:59.164+0000 7ffa4d8d7700 1 --2- 172.21.15.50:0/715185957 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7ffa340395b0 0x7ffa3403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffa38006fd0 tx=0x7ffa38006e40 comp rx=0 tx=0).stop 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:59.164+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/715185957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 msgr2=0x7ffa4819c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:59.164+0000 7ffa4d8d7700 1 --2- 172.21.15.50:0/715185957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 0x7ffa4819c450 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffa30011ac0 tx=0x7ffa30011ba0 comp rx=0 tx=0).stop 2024-04-23T18:08:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:59.164+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/715185957 shutdown_connections 2024-04-23T18:08:59.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:59.164+0000 7ffa4d8d7700 1 --2- 172.21.15.50:0/715185957 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7ffa340395b0 0x7ffa3403ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:59.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:59.164+0000 7ffa4d8d7700 1 --2- 172.21.15.50:0/715185957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48108fb0 0x7ffa4819c450 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:08:59.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:59.164+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/715185957 >> 172.21.15.50:0/715185957 conn(0x7ffa48102a40 msgr2=0x7ffa481034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:08:59.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:59.164+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/715185957 shutdown_connections 2024-04-23T18:08:59.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:08:59.164+0000 7ffa4d8d7700 1 -- 172.21.15.50:0/715185957 wait complete. 2024-04-23T18:08:59.710 INFO:teuthology.orchestra.run.smithi050.stdout:Enabling cephadm module... 2024-04-23T18:09:00.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:00 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2974507753' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-23T18:09:00.696 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:00 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: ignoring --setuser ceph since I am not root 2024-04-23T18:09:00.696 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:00 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: ignoring --setgroup ceph since I am not root 2024-04-23T18:09:00.696 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:00 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:00.621+0000 7fc5495a5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:09:00.975 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:00 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:00.817+0000 7fc5495a5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:09:00.975 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:00 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:00.923+0000 7fc5495a5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:09:00.983 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.113+0000 7f456bb6d700 1 Processor -- start 2024-04-23T18:09:00.983 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.113+0000 7f456bb6d700 1 -- start start 2024-04-23T18:09:00.983 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.113+0000 7f456bb6d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 0x7f4564107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:00.983 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.113+0000 7f456bb6d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4564107e30 con 0x7f4564107440 2024-04-23T18:09:00.983 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.114+0000 7f4569909700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 0x7f4564107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:00.983 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.114+0000 7f4569909700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 0x7f4564107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34164/0 (socket says 172.21.15.50:34164) 2024-04-23T18:09:00.983 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.114+0000 7f4569909700 1 -- 172.21.15.50:0/4186843702 learned_addr learned my addr 172.21.15.50:0/4186843702 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:00.983 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.114+0000 7f4569909700 1 -- 172.21.15.50:0/4186843702 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4564108640 con 0x7f4564107440 2024-04-23T18:09:00.983 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.114+0000 7f4569909700 1 --2- 172.21.15.50:0/4186843702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 0x7f4564107860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f45600098a0 tx=0x7f4560009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b916a7d21c54cd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:00.984 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.114+0000 7f4568907700 1 -- 172.21.15.50:0/4186843702 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4560004160 con 0x7f4564107440 2024-04-23T18:09:00.984 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.114+0000 7f4568907700 1 -- 172.21.15.50:0/4186843702 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4560004300 con 0x7f4564107440 2024-04-23T18:09:00.984 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.114+0000 7f4568907700 1 -- 172.21.15.50:0/4186843702 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4560011af0 con 0x7f4564107440 2024-04-23T18:09:00.984 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.115+0000 7f456bb6d700 1 -- 172.21.15.50:0/4186843702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 msgr2=0x7f4564107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:00.984 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.115+0000 7f456bb6d700 1 --2- 172.21.15.50:0/4186843702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 0x7f4564107860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f45600098a0 tx=0x7f4560009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:00.984 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.115+0000 7f456bb6d700 1 -- 172.21.15.50:0/4186843702 shutdown_connections 2024-04-23T18:09:00.984 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.115+0000 7f456bb6d700 1 --2- 172.21.15.50:0/4186843702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 0x7f4564107860 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:00.984 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.115+0000 7f456bb6d700 1 -- 172.21.15.50:0/4186843702 >> 172.21.15.50:0/4186843702 conn(0x7f4564102a40 msgr2=0x7f4564104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:00.984 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.115+0000 7f456bb6d700 1 -- 172.21.15.50:0/4186843702 shutdown_connections 2024-04-23T18:09:00.984 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.115+0000 7f456bb6d700 1 -- 172.21.15.50:0/4186843702 wait complete. 2024-04-23T18:09:00.984 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.116+0000 7f456bb6d700 1 Processor -- start 2024-04-23T18:09:00.984 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.116+0000 7f456bb6d700 1 -- start start 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.116+0000 7f456bb6d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 0x7f45641925b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.116+0000 7f456bb6d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f456410b9f0 con 0x7f4564107440 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.116+0000 7f4569909700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 0x7f45641925b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.116+0000 7f4569909700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 0x7f45641925b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34176/0 (socket says 172.21.15.50:34176) 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.116+0000 7f4569909700 1 -- 172.21.15.50:0/2974507753 learned_addr learned my addr 172.21.15.50:0/2974507753 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.116+0000 7f4569909700 1 -- 172.21.15.50:0/2974507753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4564192af0 con 0x7f4564107440 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.117+0000 7f4569909700 1 --2- 172.21.15.50:0/2974507753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 0x7f45641925b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45600117b0 tx=0x7f4560011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.117+0000 7f455affd700 1 -- 172.21.15.50:0/2974507753 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4560003af0 con 0x7f4564107440 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.117+0000 7f455affd700 1 -- 172.21.15.50:0/2974507753 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4560020070 con 0x7f4564107440 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.117+0000 7f456bb6d700 1 -- 172.21.15.50:0/2974507753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4564190ce0 con 0x7f4564107440 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.117+0000 7f455affd700 1 -- 172.21.15.50:0/2974507753 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4560019630 con 0x7f4564107440 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.117+0000 7f456bb6d700 1 -- 172.21.15.50:0/2974507753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4564191220 con 0x7f4564107440 2024-04-23T18:09:00.985 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.118+0000 7f455affd700 1 -- 172.21.15.50:0/2974507753 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f45600198a0 con 0x7f4564107440 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.118+0000 7f456bb6d700 1 -- 172.21.15.50:0/2974507753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45640640a0 con 0x7f4564107440 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.118+0000 7f455affd700 1 --2- 172.21.15.50:0/2974507753 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f45500395b0 0x7f455003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.118+0000 7f455affd700 1 -- 172.21.15.50:0/2974507753 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f456004dbb0 con 0x7f4564107440 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.118+0000 7f4569108700 1 --2- 172.21.15.50:0/2974507753 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f45500395b0 0x7f455003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.119+0000 7f4569108700 1 --2- 172.21.15.50:0/2974507753 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f45500395b0 0x7f455003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4554006fd0 tx=0x7f4554006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.123+0000 7f455affd700 1 -- 172.21.15.50:0/2974507753 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f456004d5d0 con 0x7f4564107440 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.390+0000 7f456bb6d700 1 -- 172.21.15.50:0/2974507753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f4564196350 con 0x7f4564107440 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.442+0000 7f455affd700 1 -- 172.21.15.50:0/2974507753 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f4560023770 con 0x7f4564107440 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.444+0000 7f455affd700 1 -- 172.21.15.50:0/2974507753 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f456002dc70 con 0x7f4564107440 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.445+0000 7f456bb6d700 1 -- 172.21.15.50:0/2974507753 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f45500395b0 msgr2=0x7f455003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.445+0000 7f456bb6d700 1 --2- 172.21.15.50:0/2974507753 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f45500395b0 0x7f455003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4554006fd0 tx=0x7f4554006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.445+0000 7f456bb6d700 1 -- 172.21.15.50:0/2974507753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 msgr2=0x7f45641925b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:00.986 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.445+0000 7f456bb6d700 1 --2- 172.21.15.50:0/2974507753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 0x7f45641925b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45600117b0 tx=0x7f4560011b70 comp rx=0 tx=0).stop 2024-04-23T18:09:00.987 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.445+0000 7f456bb6d700 1 -- 172.21.15.50:0/2974507753 shutdown_connections 2024-04-23T18:09:00.987 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.445+0000 7f456bb6d700 1 --2- 172.21.15.50:0/2974507753 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f45500395b0 0x7f455003ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:00.987 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.445+0000 7f456bb6d700 1 --2- 172.21.15.50:0/2974507753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564107440 0x7f45641925b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:00.987 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.445+0000 7f456bb6d700 1 -- 172.21.15.50:0/2974507753 >> 172.21.15.50:0/2974507753 conn(0x7f4564102a40 msgr2=0x7f4564103930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:00.987 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.445+0000 7f456bb6d700 1 -- 172.21.15.50:0/2974507753 shutdown_connections 2024-04-23T18:09:00.987 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:00.445+0000 7f456bb6d700 1 -- 172.21.15.50:0/2974507753 wait complete. 2024-04-23T18:09:01.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:01 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2974507753' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-23T18:09:01.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:01 smithi050 ceph-mon[29187]: mgrmap e5: x(active, since 5s) 2024-04-23T18:09:01.611 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:01.256+0000 7fc5495a5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:09:01.611 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:01.345+0000 7fc5495a5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:09:02.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:09:02.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-23T18:09:02.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T18:09:02.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-23T18:09:02.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-23T18:09:02.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:09:02.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.410+0000 7fbef49a9700 1 Processor -- start 2024-04-23T18:09:02.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.410+0000 7fbef49a9700 1 -- start start 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.411+0000 7fbef49a9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 0x7fbef0107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.411+0000 7fbef49a9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbef0107e30 con 0x7fbef0107440 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.411+0000 7fbeee59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 0x7fbef0107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.411+0000 7fbeee59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 0x7fbef0107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34214/0 (socket says 172.21.15.50:34214) 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.411+0000 7fbeee59c700 1 -- 172.21.15.50:0/241421960 learned_addr learned my addr 172.21.15.50:0/241421960 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.411+0000 7fbeee59c700 1 -- 172.21.15.50:0/241421960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef0108640 con 0x7fbef0107440 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.411+0000 7fbeee59c700 1 --2- 172.21.15.50:0/241421960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 0x7fbef0107860 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbee00098a0 tx=0x7fbee0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=776cb0fca3fe62e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.412+0000 7fbeed59a700 1 -- 172.21.15.50:0/241421960 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbee0004160 con 0x7fbef0107440 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.412+0000 7fbeed59a700 1 -- 172.21.15.50:0/241421960 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbee0004300 con 0x7fbef0107440 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.412+0000 7fbeed59a700 1 -- 172.21.15.50:0/241421960 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbee0011af0 con 0x7fbef0107440 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.412+0000 7fbef49a9700 1 -- 172.21.15.50:0/241421960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 msgr2=0x7fbef0107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.412+0000 7fbef49a9700 1 --2- 172.21.15.50:0/241421960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 0x7fbef0107860 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbee00098a0 tx=0x7fbee0009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:02.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.412+0000 7fbef49a9700 1 -- 172.21.15.50:0/241421960 shutdown_connections 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.412+0000 7fbef49a9700 1 --2- 172.21.15.50:0/241421960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 0x7fbef0107860 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.412+0000 7fbef49a9700 1 -- 172.21.15.50:0/241421960 >> 172.21.15.50:0/241421960 conn(0x7fbef0102a40 msgr2=0x7fbef0104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.412+0000 7fbef49a9700 1 -- 172.21.15.50:0/241421960 shutdown_connections 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.412+0000 7fbef49a9700 1 -- 172.21.15.50:0/241421960 wait complete. 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.413+0000 7fbef49a9700 1 Processor -- start 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.413+0000 7fbef49a9700 1 -- start start 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.414+0000 7fbef49a9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 0x7fbef0063900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.414+0000 7fbef49a9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbef010b9f0 con 0x7fbef0107440 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.414+0000 7fbeee59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 0x7fbef0063900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.414+0000 7fbeee59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 0x7fbef0063900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34228/0 (socket says 172.21.15.50:34228) 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.414+0000 7fbeee59c700 1 -- 172.21.15.50:0/1900364301 learned_addr learned my addr 172.21.15.50:0/1900364301 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.414+0000 7fbeee59c700 1 -- 172.21.15.50:0/1900364301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef019f220 con 0x7fbef0107440 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.414+0000 7fbeee59c700 1 --2- 172.21.15.50:0/1900364301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 0x7fbef0063900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbee0003f40 tx=0x7fbee0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.414+0000 7fbee77fe700 1 -- 172.21.15.50:0/1900364301 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbee00038b0 con 0x7fbef0107440 2024-04-23T18:09:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.414+0000 7fbef49a9700 1 -- 172.21.15.50:0/1900364301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbef019f3f0 con 0x7fbef0107440 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.414+0000 7fbee77fe700 1 -- 172.21.15.50:0/1900364301 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbee0003a50 con 0x7fbef0107440 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.415+0000 7fbee77fe700 1 -- 172.21.15.50:0/1900364301 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbee001a5c0 con 0x7fbef0107440 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.415+0000 7fbef49a9700 1 -- 172.21.15.50:0/1900364301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbef019f790 con 0x7fbef0107440 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.415+0000 7fbee77fe700 1 -- 172.21.15.50:0/1900364301 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fbee001a8c0 con 0x7fbef0107440 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.415+0000 7fbef49a9700 1 -- 172.21.15.50:0/1900364301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbef00640a0 con 0x7fbef0107440 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.416+0000 7fbee77fe700 1 --2- 172.21.15.50:0/1900364301 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7fbedc039560 0x7fbedc03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.416+0000 7fbee77fe700 1 -- 172.21.15.50:0/1900364301 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbee004dae0 con 0x7fbef0107440 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.416+0000 7fbeedd9b700 1 -- 172.21.15.50:0/1900364301 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7fbedc039560 msgr2=0x7fbedc03ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1300452320 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.416+0000 7fbeedd9b700 1 --2- 172.21.15.50:0/1900364301 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7fbedc039560 0x7fbedc03ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.425+0000 7fbee77fe700 1 -- 172.21.15.50:0/1900364301 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbee0019c70 con 0x7fbef0107440 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.617+0000 7fbeedd9b700 1 -- 172.21.15.50:0/1900364301 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7fbedc039560 msgr2=0x7fbedc03ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1300452320 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.617+0000 7fbeedd9b700 1 --2- 172.21.15.50:0/1900364301 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7fbedc039560 0x7fbedc03ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.696+0000 7fbef49a9700 1 -- 172.21.15.50:0/1900364301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fbef019fba0 con 0x7fbef0107440 2024-04-23T18:09:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.697+0000 7fbee77fe700 1 -- 172.21.15.50:0/1900364301 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fbef019fba0 con 0x7fbef0107440 2024-04-23T18:09:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.699+0000 7fbef49a9700 1 -- 172.21.15.50:0/1900364301 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7fbedc039560 msgr2=0x7fbedc03ba20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:09:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.699+0000 7fbef49a9700 1 --2- 172.21.15.50:0/1900364301 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7fbedc039560 0x7fbedc03ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.699+0000 7fbef49a9700 1 -- 172.21.15.50:0/1900364301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 msgr2=0x7fbef0063900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.699+0000 7fbef49a9700 1 --2- 172.21.15.50:0/1900364301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 0x7fbef0063900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbee0003f40 tx=0x7fbee0011890 comp rx=0 tx=0).stop 2024-04-23T18:09:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.699+0000 7fbef49a9700 1 -- 172.21.15.50:0/1900364301 shutdown_connections 2024-04-23T18:09:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.699+0000 7fbef49a9700 1 --2- 172.21.15.50:0/1900364301 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7fbedc039560 0x7fbedc03ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.699+0000 7fbef49a9700 1 --2- 172.21.15.50:0/1900364301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbef0107440 0x7fbef0063900 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.699+0000 7fbef49a9700 1 -- 172.21.15.50:0/1900364301 >> 172.21.15.50:0/1900364301 conn(0x7fbef0102a40 msgr2=0x7fbef01035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.699+0000 7fbef49a9700 1 -- 172.21.15.50:0/1900364301 shutdown_connections 2024-04-23T18:09:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:01.699+0000 7fbef49a9700 1 -- 172.21.15.50:0/1900364301 wait complete. 2024-04-23T18:09:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for the mgr to restart... 2024-04-23T18:09:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mgr epoch 5... 2024-04-23T18:09:02.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:02 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1900364301' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T18:09:02.696 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:02 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:02.607+0000 7fc5495a5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:09:03.198 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:02 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:02.787+0000 7fc5495a5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:09:03.198 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:02 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:02.962+0000 7fc5495a5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:09:03.559 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:03 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:03.261+0000 7fc5495a5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:09:03.560 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:03 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:03.356+0000 7fc5495a5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:09:03.560 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:03 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:03.565+0000 7fc5495a5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:09:03.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:03 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:03.766+0000 7fc5495a5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:09:04.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:04 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:04.390+0000 7fc5495a5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:09:05.131 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:04 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:04.826+0000 7fc5495a5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:09:05.132 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:04 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:04.926+0000 7fc5495a5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:09:05.132 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:05 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:05.134+0000 7fc5495a5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:09:05.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:05 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:05.677+0000 7fc5495a5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:09:05.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:05 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:05.773+0000 7fc5495a5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:09:06.377 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:06 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:06.025+0000 7fc5495a5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:09:06.377 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:06 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:06.163+0000 7fc5495a5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:09:06.377 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:06 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:06.261+0000 7fc5495a5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:09:06.377 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:06 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:06.382+0000 7fc5495a5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:09:06.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:06 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:06.696+0000 7fc5495a5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:09:07.593 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:07.322+0000 7fc5495a5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:09:07.593 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:07.546+0000 7fc5495a5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:09:07.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: Active manager daemon x restarted 2024-04-23T18:09:07.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: Activating manager daemon x 2024-04-23T18:09:07.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: osdmap e2: 0 total, 0 up, 0 in 2024-04-23T18:09:07.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: mgrmap e6: x(active, starting, since 0.00343286s) 2024-04-23T18:09:07.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:09:07.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:09:07.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:09:07.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:09:07.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:09:07.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: Manager daemon x is now available 2024-04-23T18:09:07.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:07.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:07.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:07.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:07.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:07 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:09:08.845 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:08 smithi050 ceph-mon[29187]: Found migration_current of "None". Setting to last migration. 2024-04-23T18:09:08.845 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:08 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:09:08.846 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:08 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:08.846 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:08 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:08.846 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:08 smithi050 ceph-mon[29187]: mgrmap e7: x(active, since 1.00575s) 2024-04-23T18:09:09.137 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:09:09.138 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-23T18:09:09.138 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-23T18:09:09.138 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:09:09.138 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.761+0000 7f172235c700 1 Processor -- start 2024-04-23T18:09:09.139 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.761+0000 7f172235c700 1 -- start start 2024-04-23T18:09:09.139 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.761+0000 7f172235c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 0x7f171c108990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:09.139 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.761+0000 7f172235c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f171c108f60 con 0x7f171c108570 2024-04-23T18:09:09.139 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.761+0000 7f171bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 0x7f171c108990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:09.139 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.761+0000 7f171bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 0x7f171c108990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45484/0 (socket says 172.21.15.50:45484) 2024-04-23T18:09:09.139 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.761+0000 7f171bfff700 1 -- 172.21.15.50:0/2669618985 learned_addr learned my addr 172.21.15.50:0/2669618985 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:09.140 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.762+0000 7f171bfff700 1 -- 172.21.15.50:0/2669618985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171c109720 con 0x7f171c108570 2024-04-23T18:09:09.140 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.762+0000 7f171bfff700 1 --2- 172.21.15.50:0/2669618985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 0x7f171c108990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f170c0098a0 tx=0x7f170c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991160782368022 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:09.140 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.762+0000 7f171affd700 1 -- 172.21.15.50:0/2669618985 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f170c004160 con 0x7f171c108570 2024-04-23T18:09:09.140 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.762+0000 7f171affd700 1 -- 172.21.15.50:0/2669618985 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f170c004300 con 0x7f171c108570 2024-04-23T18:09:09.140 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.762+0000 7f171affd700 1 -- 172.21.15.50:0/2669618985 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f170c011af0 con 0x7f171c108570 2024-04-23T18:09:09.141 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.763+0000 7f172235c700 1 -- 172.21.15.50:0/2669618985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 msgr2=0x7f171c108990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:09.141 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.763+0000 7f172235c700 1 --2- 172.21.15.50:0/2669618985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 0x7f171c108990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f170c0098a0 tx=0x7f170c009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.141 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.763+0000 7f172235c700 1 -- 172.21.15.50:0/2669618985 shutdown_connections 2024-04-23T18:09:09.141 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.763+0000 7f172235c700 1 --2- 172.21.15.50:0/2669618985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 0x7f171c108990 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.141 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.763+0000 7f172235c700 1 -- 172.21.15.50:0/2669618985 >> 172.21.15.50:0/2669618985 conn(0x7f171c103b70 msgr2=0x7f171c105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:09.141 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.763+0000 7f172235c700 1 -- 172.21.15.50:0/2669618985 shutdown_connections 2024-04-23T18:09:09.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.763+0000 7f172235c700 1 -- 172.21.15.50:0/2669618985 wait complete. 2024-04-23T18:09:09.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.764+0000 7f172235c700 1 Processor -- start 2024-04-23T18:09:09.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.764+0000 7f172235c700 1 -- start start 2024-04-23T18:09:09.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.764+0000 7f172235c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 0x7f171c07db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:09.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.764+0000 7f172235c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f171c10cad0 con 0x7f171c108570 2024-04-23T18:09:09.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.764+0000 7f171bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 0x7f171c07db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:09.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.764+0000 7f171bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 0x7f171c07db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45500/0 (socket says 172.21.15.50:45500) 2024-04-23T18:09:09.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.764+0000 7f171bfff700 1 -- 172.21.15.50:0/68905482 learned_addr learned my addr 172.21.15.50:0/68905482 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:09.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.765+0000 7f171bfff700 1 -- 172.21.15.50:0/68905482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171c07e0c0 con 0x7f171c108570 2024-04-23T18:09:09.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.765+0000 7f171bfff700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 0x7f171c07db80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f170c0117b0 tx=0x7f170c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:09.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.765+0000 7f17197fa700 1 -- 172.21.15.50:0/68905482 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f170c003af0 con 0x7f171c108570 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.765+0000 7f172235c700 1 -- 172.21.15.50:0/68905482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171c07c2b0 con 0x7f171c108570 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.765+0000 7f17197fa700 1 -- 172.21.15.50:0/68905482 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f170c020070 con 0x7f171c108570 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.765+0000 7f17197fa700 1 -- 172.21.15.50:0/68905482 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f170c019630 con 0x7f171c108570 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.765+0000 7f172235c700 1 -- 172.21.15.50:0/68905482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f171c07c7f0 con 0x7f171c108570 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.766+0000 7f17197fa700 1 -- 172.21.15.50:0/68905482 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f170c0198a0 con 0x7f171c108570 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.766+0000 7f17197fa700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 0x7f170403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.766+0000 7f17197fa700 1 -- 172.21.15.50:0/68905482 --> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f170403c1d0 con 0x7f1704039600 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.766+0000 7f17197fa700 1 -- 172.21.15.50:0/68905482 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f170c04db00 con 0x7f171c108570 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.766+0000 7f171b7fe700 1 -- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 msgr2=0x7f170403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1300452320 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.766+0000 7f171b7fe700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 0x7f170403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.967+0000 7f171b7fe700 1 -- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 msgr2=0x7f170403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1300452320 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:02.967+0000 7f171b7fe700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 0x7f170403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:09:09.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:03.368+0000 7f171b7fe700 1 -- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 msgr2=0x7f170403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1300452320 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:03.368+0000 7f171b7fe700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 0x7f170403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:04.169+0000 7f171b7fe700 1 -- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 msgr2=0x7f170403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1300452320 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:04.169+0000 7f171b7fe700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 0x7f170403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:05.769+0000 7f171b7fe700 1 -- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 msgr2=0x7f170403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1300452320 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:05.769+0000 7f171b7fe700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 0x7f170403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:07.549+0000 7f17197fa700 1 -- 172.21.15.50:0/68905482 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f170c00b480 con 0x7f171c108570 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:07.549+0000 7f17197fa700 1 -- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 msgr2=0x7f170403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:07.549+0000 7f17197fa700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 0x7f170403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.550+0000 7f17197fa700 1 -- 172.21.15.50:0/68905482 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f170c04ef30 con 0x7f171c108570 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.551+0000 7f17197fa700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f170403cef0 0x7f170403f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.551+0000 7f17197fa700 1 -- 172.21.15.50:0/68905482 --> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f170c01fec0 con 0x7f170403cef0 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.551+0000 7f171b7fe700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f170403cef0 0x7f170403f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:09.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.552+0000 7f171b7fe700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f170403cef0 0x7f170403f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1710003a10 tx=0x7f17100092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.553+0000 7f17197fa700 1 -- 172.21.15.50:0/68905482 <== mgr.14120 v2:172.21.15.50:6800/103327972 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f170c01fec0 con 0x7f170403cef0 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.559+0000 7f172235c700 1 -- 172.21.15.50:0/68905482 --> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f171c069ba0 con 0x7f170403cef0 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.559+0000 7f17197fa700 1 -- 172.21.15.50:0/68905482 <== mgr.14120 v2:172.21.15.50:6800/103327972 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f171c069ba0 con 0x7f170403cef0 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.560+0000 7f172235c700 1 -- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f170403cef0 msgr2=0x7f170403f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.560+0000 7f172235c700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f170403cef0 0x7f170403f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1710003a10 tx=0x7f17100092b0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.560+0000 7f172235c700 1 -- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 msgr2=0x7f171c07db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.560+0000 7f172235c700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 0x7f171c07db80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f170c0117b0 tx=0x7f170c011b70 comp rx=0 tx=0).stop 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.560+0000 7f172235c700 1 -- 172.21.15.50:0/68905482 shutdown_connections 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.560+0000 7f172235c700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f170403cef0 0x7f170403f2e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.560+0000 7f172235c700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f171c108570 0x7f171c07db80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.560+0000 7f172235c700 1 --2- 172.21.15.50:0/68905482 >> [v2:172.21.15.50:6800/1300452320,v1:172.21.15.50:6801/1300452320] conn(0x7f1704039600 0x7f170403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.560+0000 7f172235c700 1 -- 172.21.15.50:0/68905482 >> 172.21.15.50:0/68905482 conn(0x7f171c103b70 msgr2=0x7f171c104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:09.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.560+0000 7f172235c700 1 -- 172.21.15.50:0/68905482 shutdown_connections 2024-04-23T18:09:09.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:08.560+0000 7f172235c700 1 -- 172.21.15.50:0/68905482 wait complete. 2024-04-23T18:09:09.149 INFO:teuthology.orchestra.run.smithi050.stdout:mgr epoch 5 is available 2024-04-23T18:09:09.149 INFO:teuthology.orchestra.run.smithi050.stdout:Setting orchestrator backend to cephadm... 2024-04-23T18:09:09.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:09 smithi050 ceph-mon[29187]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-23T18:09:09.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:09 smithi050 ceph-mon[29187]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-23T18:09:10.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.575+0000 7fc46912d700 1 Processor -- start 2024-04-23T18:09:10.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.575+0000 7fc46912d700 1 -- start start 2024-04-23T18:09:10.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.575+0000 7fc46912d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 0x7fc464107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.575+0000 7fc46912d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc464107e30 con 0x7fc464107440 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.575+0000 7fc462d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 0x7fc464107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.575+0000 7fc462d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 0x7fc464107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45562/0 (socket says 172.21.15.50:45562) 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.575+0000 7fc462d9d700 1 -- 172.21.15.50:0/2592968845 learned_addr learned my addr 172.21.15.50:0/2592968845 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.575+0000 7fc462d9d700 1 -- 172.21.15.50:0/2592968845 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc464108640 con 0x7fc464107440 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.576+0000 7fc462d9d700 1 --2- 172.21.15.50:0/2592968845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 0x7fc464107860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc454009e80 tx=0x7fc45400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42bec25a68998713 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.576+0000 7fc461d9b700 1 -- 172.21.15.50:0/2592968845 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc454017070 con 0x7fc464107440 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.576+0000 7fc461d9b700 1 -- 172.21.15.50:0/2592968845 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc45400b8c0 con 0x7fc464107440 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.576+0000 7fc461d9b700 1 -- 172.21.15.50:0/2592968845 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc454004490 con 0x7fc464107440 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.576+0000 7fc46912d700 1 -- 172.21.15.50:0/2592968845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 msgr2=0x7fc464107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.576+0000 7fc46912d700 1 --2- 172.21.15.50:0/2592968845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 0x7fc464107860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc454009e80 tx=0x7fc45400b330 comp rx=0 tx=0).stop 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.576+0000 7fc46912d700 1 -- 172.21.15.50:0/2592968845 shutdown_connections 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.576+0000 7fc46912d700 1 --2- 172.21.15.50:0/2592968845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 0x7fc464107860 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.576+0000 7fc46912d700 1 -- 172.21.15.50:0/2592968845 >> 172.21.15.50:0/2592968845 conn(0x7fc464102a40 msgr2=0x7fc464104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.576+0000 7fc46912d700 1 -- 172.21.15.50:0/2592968845 shutdown_connections 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.576+0000 7fc46912d700 1 -- 172.21.15.50:0/2592968845 wait complete. 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.577+0000 7fc46912d700 1 Processor -- start 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.577+0000 7fc46912d700 1 -- start start 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.577+0000 7fc46912d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 0x7fc46419c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.577+0000 7fc46912d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc46410b9f0 con 0x7fc464107440 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.578+0000 7fc462d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 0x7fc46419c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.578+0000 7fc462d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 0x7fc46419c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45566/0 (socket says 172.21.15.50:45566) 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.578+0000 7fc462d9d700 1 -- 172.21.15.50:0/1096332790 learned_addr learned my addr 172.21.15.50:0/1096332790 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.578+0000 7fc462d9d700 1 -- 172.21.15.50:0/1096332790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc46419ca60 con 0x7fc464107440 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.578+0000 7fc462d9d700 1 --2- 172.21.15.50:0/1096332790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 0x7fc46419c520 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc454004a60 tx=0x7fc45400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.578+0000 7fc45bfff700 1 -- 172.21.15.50:0/1096332790 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc454020020 con 0x7fc464107440 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.578+0000 7fc45bfff700 1 -- 172.21.15.50:0/1096332790 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc454012430 con 0x7fc464107440 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.578+0000 7fc46912d700 1 -- 172.21.15.50:0/1096332790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc46419ccf0 con 0x7fc464107440 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.578+0000 7fc45bfff700 1 -- 172.21.15.50:0/1096332790 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc454017040 con 0x7fc464107440 2024-04-23T18:09:10.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.578+0000 7fc46912d700 1 -- 172.21.15.50:0/1096332790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc46419f990 con 0x7fc464107440 2024-04-23T18:09:10.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.579+0000 7fc45bfff700 1 -- 172.21.15.50:0/1096332790 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fc454023020 con 0x7fc464107440 2024-04-23T18:09:10.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.579+0000 7fc45bfff700 1 --2- 172.21.15.50:0/1096332790 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fc450039490 0x7fc45003b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:10.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.579+0000 7fc45bfff700 1 -- 172.21.15.50:0/1096332790 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc45400f050 con 0x7fc464107440 2024-04-23T18:09:10.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.579+0000 7fc46912d700 1 -- 172.21.15.50:0/1096332790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4640640a0 con 0x7fc464107440 2024-04-23T18:09:10.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.579+0000 7fc46259c700 1 --2- 172.21.15.50:0/1096332790 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fc450039490 0x7fc45003b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:10.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.580+0000 7fc46259c700 1 --2- 172.21.15.50:0/1096332790 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fc450039490 0x7fc45003b950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc44c006fd0 tx=0x7fc44c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:10.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.585+0000 7fc45bfff700 1 -- 172.21.15.50:0/1096332790 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc45402a020 con 0x7fc464107440 2024-04-23T18:09:10.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.603+0000 7fc45bfff700 1 -- 172.21.15.50:0/1096332790 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc4540125e0 con 0x7fc464107440 2024-04-23T18:09:10.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.812+0000 7fc46912d700 1 -- 172.21.15.50:0/1096332790 --> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fc46402e320 con 0x7fc450039490 2024-04-23T18:09:10.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.817+0000 7fc45bfff700 1 -- 172.21.15.50:0/1096332790 <== mgr.14120 v2:172.21.15.50:6800/103327972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc46402e320 con 0x7fc450039490 2024-04-23T18:09:10.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.819+0000 7fc46912d700 1 -- 172.21.15.50:0/1096332790 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fc450039490 msgr2=0x7fc45003b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:10.347 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.819+0000 7fc46912d700 1 --2- 172.21.15.50:0/1096332790 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fc450039490 0x7fc45003b950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc44c006fd0 tx=0x7fc44c006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:10.347 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.819+0000 7fc46912d700 1 -- 172.21.15.50:0/1096332790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 msgr2=0x7fc46419c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:10.347 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.819+0000 7fc46912d700 1 --2- 172.21.15.50:0/1096332790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 0x7fc46419c520 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc454004a60 tx=0x7fc45400bd50 comp rx=0 tx=0).stop 2024-04-23T18:09:10.347 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.819+0000 7fc46912d700 1 -- 172.21.15.50:0/1096332790 shutdown_connections 2024-04-23T18:09:10.347 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.819+0000 7fc46912d700 1 --2- 172.21.15.50:0/1096332790 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fc450039490 0x7fc45003b950 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:10.347 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.819+0000 7fc46912d700 1 --2- 172.21.15.50:0/1096332790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc464107440 0x7fc46419c520 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:10.347 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.819+0000 7fc46912d700 1 -- 172.21.15.50:0/1096332790 >> 172.21.15.50:0/1096332790 conn(0x7fc464102a40 msgr2=0x7fc4641038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:10.347 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.819+0000 7fc46912d700 1 -- 172.21.15.50:0/1096332790 shutdown_connections 2024-04-23T18:09:10.347 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:09.819+0000 7fc46912d700 1 -- 172.21.15.50:0/1096332790 wait complete. 2024-04-23T18:09:10.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:10 smithi050 ceph-mon[29187]: [23/Apr/2024:18:09:09] ENGINE Bus STARTING 2024-04-23T18:09:10.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:10 smithi050 ceph-mon[29187]: mgrmap e8: x(active, since 2s) 2024-04-23T18:09:10.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:10 smithi050 ceph-mon[29187]: [23/Apr/2024:18:09:09] ENGINE Serving on https://172.21.15.50:7150 2024-04-23T18:09:10.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:10 smithi050 ceph-mon[29187]: [23/Apr/2024:18:09:09] ENGINE Serving on http://172.21.15.50:8765 2024-04-23T18:09:10.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:10 smithi050 ceph-mon[29187]: [23/Apr/2024:18:09:09] ENGINE Bus STARTED 2024-04-23T18:09:10.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:10 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:10.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:10 smithi050 ceph-mon[29187]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:10.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:10 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:10.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:10 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:11.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-23T18:09:11.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.756+0000 7f3195d0e700 1 Processor -- start 2024-04-23T18:09:11.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.756+0000 7f3195d0e700 1 -- start start 2024-04-23T18:09:11.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.756+0000 7f3195d0e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 0x7f3190107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:11.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.756+0000 7f3195d0e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3190107e30 con 0x7f3190107440 2024-04-23T18:09:11.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.756+0000 7f318f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 0x7f3190107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:11.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.756+0000 7f318f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 0x7f3190107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45572/0 (socket says 172.21.15.50:45572) 2024-04-23T18:09:11.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.756+0000 7f318f7fe700 1 -- 172.21.15.50:0/2687165634 learned_addr learned my addr 172.21.15.50:0/2687165634 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.757+0000 7f318f7fe700 1 -- 172.21.15.50:0/2687165634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3190108640 con 0x7f3190107440 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.757+0000 7f318f7fe700 1 --2- 172.21.15.50:0/2687165634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 0x7f3190107860 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f318000bb40 tx=0x7f318000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a681944df05b8088 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.757+0000 7f318e7fc700 1 -- 172.21.15.50:0/2687165634 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f318000dbe0 con 0x7f3190107440 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.757+0000 7f318e7fc700 1 -- 172.21.15.50:0/2687165634 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f31800044c0 con 0x7f3190107440 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.757+0000 7f318e7fc700 1 -- 172.21.15.50:0/2687165634 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3180010470 con 0x7f3190107440 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.757+0000 7f3195d0e700 1 -- 172.21.15.50:0/2687165634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 msgr2=0x7f3190107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.758+0000 7f3195d0e700 1 --2- 172.21.15.50:0/2687165634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 0x7f3190107860 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f318000bb40 tx=0x7f318000be50 comp rx=0 tx=0).stop 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.758+0000 7f3195d0e700 1 -- 172.21.15.50:0/2687165634 shutdown_connections 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.758+0000 7f3195d0e700 1 --2- 172.21.15.50:0/2687165634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 0x7f3190107860 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.758+0000 7f3195d0e700 1 -- 172.21.15.50:0/2687165634 >> 172.21.15.50:0/2687165634 conn(0x7f3190102a40 msgr2=0x7f3190104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.758+0000 7f3195d0e700 1 -- 172.21.15.50:0/2687165634 shutdown_connections 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.758+0000 7f3195d0e700 1 -- 172.21.15.50:0/2687165634 wait complete. 2024-04-23T18:09:11.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.758+0000 7f3195d0e700 1 Processor -- start 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.758+0000 7f3195d0e700 1 -- start start 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.759+0000 7f3195d0e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 0x7f319019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.759+0000 7f3195d0e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f319007afe0 con 0x7f3190107440 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.759+0000 7f318f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 0x7f319019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.759+0000 7f318f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 0x7f319019e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45576/0 (socket says 172.21.15.50:45576) 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.759+0000 7f318f7fe700 1 -- 172.21.15.50:0/817661877 learned_addr learned my addr 172.21.15.50:0/817661877 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.759+0000 7f318f7fe700 1 -- 172.21.15.50:0/817661877 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319019ebf0 con 0x7f3190107440 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.759+0000 7f318f7fe700 1 --2- 172.21.15.50:0/817661877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 0x7f319019e6b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f318000d650 tx=0x7f3180003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.759+0000 7f318cff9700 1 -- 172.21.15.50:0/817661877 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3180010770 con 0x7f3190107440 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.759+0000 7f318cff9700 1 -- 172.21.15.50:0/817661877 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f318001d070 con 0x7f3190107440 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.759+0000 7f3195d0e700 1 -- 172.21.15.50:0/817661877 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319019ee80 con 0x7f3190107440 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.759+0000 7f318cff9700 1 -- 172.21.15.50:0/817661877 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3180019c10 con 0x7f3190107440 2024-04-23T18:09:11.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.759+0000 7f3195d0e700 1 -- 172.21.15.50:0/817661877 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31901a1b20 con 0x7f3190107440 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.760+0000 7f318cff9700 1 -- 172.21.15.50:0/817661877 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f3180028060 con 0x7f3190107440 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.760+0000 7f3195d0e700 1 -- 172.21.15.50:0/817661877 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3190069ba0 con 0x7f3190107440 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.760+0000 7f318cff9700 1 --2- 172.21.15.50:0/817661877 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f31780395b0 0x7f317803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.761+0000 7f318effd700 1 --2- 172.21.15.50:0/817661877 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f31780395b0 0x7f317803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.761+0000 7f318cff9700 1 -- 172.21.15.50:0/817661877 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f318004db50 con 0x7f3190107440 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.761+0000 7f318effd700 1 --2- 172.21.15.50:0/817661877 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f31780395b0 0x7f317803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3184006fd0 tx=0x7f3184006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.766+0000 7f318cff9700 1 -- 172.21.15.50:0/817661877 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3180015070 con 0x7f3190107440 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.975+0000 7f3195d0e700 1 -- 172.21.15.50:0/817661877 --> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f3190075460 con 0x7f31780395b0 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.975+0000 7f318cff9700 1 -- 172.21.15.50:0/817661877 <== mgr.14120 v2:172.21.15.50:6800/103327972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f3190075460 con 0x7f31780395b0 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.977+0000 7f3195d0e700 1 -- 172.21.15.50:0/817661877 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f31780395b0 msgr2=0x7f317803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.977+0000 7f3195d0e700 1 --2- 172.21.15.50:0/817661877 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f31780395b0 0x7f317803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3184006fd0 tx=0x7f3184006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.977+0000 7f3195d0e700 1 -- 172.21.15.50:0/817661877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 msgr2=0x7f319019e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:11.528 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.977+0000 7f3195d0e700 1 --2- 172.21.15.50:0/817661877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 0x7f319019e6b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f318000d650 tx=0x7f3180003ff0 comp rx=0 tx=0).stop 2024-04-23T18:09:11.529 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.977+0000 7f3195d0e700 1 -- 172.21.15.50:0/817661877 shutdown_connections 2024-04-23T18:09:11.529 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.977+0000 7f3195d0e700 1 --2- 172.21.15.50:0/817661877 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f31780395b0 0x7f317803ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:11.529 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.977+0000 7f3195d0e700 1 --2- 172.21.15.50:0/817661877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3190107440 0x7f319019e6b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:11.529 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.977+0000 7f3195d0e700 1 -- 172.21.15.50:0/817661877 >> 172.21.15.50:0/817661877 conn(0x7f3190102a40 msgr2=0x7f31901908c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:11.529 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.978+0000 7f3195d0e700 1 -- 172.21.15.50:0/817661877 shutdown_connections 2024-04-23T18:09:11.529 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:10.978+0000 7f3195d0e700 1 -- 172.21.15.50:0/817661877 wait complete. 2024-04-23T18:09:11.529 INFO:teuthology.orchestra.run.smithi050.stdout:Generating ssh key... 2024-04-23T18:09:11.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:11 smithi050 ceph-mon[29187]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:12.614 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: Generating public/private rsa key pair. 2024-04-23T18:09:12.614 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: Your identification has been saved in /tmp/tmpu9jzp667/key. 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: Your public key has been saved in /tmp/tmpu9jzp667/key.pub. 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: The key fingerprint is: 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: SHA256:DRV0GysgucF+qcYh/180SEgDLEqqTkr9LABNR5BdjaM ceph-5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: The key's randomart image is: 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: +---[RSA 3072]----+ 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: | .=.++=+.+.o | 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: | + + Bo.= . + | 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: | = o + +o.o o | 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: |o o E + o+ o | 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: |o . + +S o o | 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: |.+ . = . . | 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: |= . o. . . | 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: |.. . o . . | 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: | . .. | 2024-04-23T18:09:12.615 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: +----[SHA256]-----+ 2024-04-23T18:09:12.856 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.932+0000 7fba9c307700 1 Processor -- start 2024-04-23T18:09:12.856 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.932+0000 7fba9c307700 1 -- start start 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.932+0000 7fba9c307700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 0x7fba9410b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.932+0000 7fba9c307700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fba9410b8e0 con 0x7fba94108fb0 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.932+0000 7fba9a0a3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 0x7fba9410b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.932+0000 7fba9a0a3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 0x7fba9410b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57214/0 (socket says 172.21.15.50:57214) 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.932+0000 7fba9a0a3700 1 -- 172.21.15.50:0/2784534035 learned_addr learned my addr 172.21.15.50:0/2784534035 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.933+0000 7fba9a0a3700 1 -- 172.21.15.50:0/2784534035 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba9410ba60 con 0x7fba94108fb0 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.933+0000 7fba9a0a3700 1 --2- 172.21.15.50:0/2784534035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 0x7fba9410b3a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fba900098a0 tx=0x7fba90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab56c440ed874b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.933+0000 7fba990a1700 1 -- 172.21.15.50:0/2784534035 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba90004160 con 0x7fba94108fb0 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.933+0000 7fba990a1700 1 -- 172.21.15.50:0/2784534035 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fba90004300 con 0x7fba94108fb0 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.933+0000 7fba990a1700 1 -- 172.21.15.50:0/2784534035 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba90011af0 con 0x7fba94108fb0 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.934+0000 7fba9c307700 1 -- 172.21.15.50:0/2784534035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 msgr2=0x7fba9410b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.934+0000 7fba9c307700 1 --2- 172.21.15.50:0/2784534035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 0x7fba9410b3a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fba900098a0 tx=0x7fba90009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:12.857 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.934+0000 7fba9c307700 1 -- 172.21.15.50:0/2784534035 shutdown_connections 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.934+0000 7fba9c307700 1 --2- 172.21.15.50:0/2784534035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 0x7fba9410b3a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.934+0000 7fba9c307700 1 -- 172.21.15.50:0/2784534035 >> 172.21.15.50:0/2784534035 conn(0x7fba94102a60 msgr2=0x7fba94104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.934+0000 7fba9c307700 1 -- 172.21.15.50:0/2784534035 shutdown_connections 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.934+0000 7fba9c307700 1 -- 172.21.15.50:0/2784534035 wait complete. 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.934+0000 7fba9c307700 1 Processor -- start 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.935+0000 7fba9c307700 1 -- start start 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.935+0000 7fba9c307700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 0x7fba9419e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.935+0000 7fba9c307700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fba9410c9c0 con 0x7fba94108fb0 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.935+0000 7fba9a0a3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 0x7fba9419e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.935+0000 7fba9a0a3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 0x7fba9419e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57216/0 (socket says 172.21.15.50:57216) 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.935+0000 7fba9a0a3700 1 -- 172.21.15.50:0/3033095088 learned_addr learned my addr 172.21.15.50:0/3033095088 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.935+0000 7fba9a0a3700 1 -- 172.21.15.50:0/3033095088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba9419ed50 con 0x7fba94108fb0 2024-04-23T18:09:12.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.935+0000 7fba9a0a3700 1 --2- 172.21.15.50:0/3033095088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 0x7fba9419e810 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fba90009fb0 tx=0x7fba90011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:12.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.935+0000 7fba8b7fe700 1 -- 172.21.15.50:0/3033095088 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba900036a0 con 0x7fba94108fb0 2024-04-23T18:09:12.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.935+0000 7fba9c307700 1 -- 172.21.15.50:0/3033095088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba9419efe0 con 0x7fba94108fb0 2024-04-23T18:09:12.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.936+0000 7fba8b7fe700 1 -- 172.21.15.50:0/3033095088 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fba9001a430 con 0x7fba94108fb0 2024-04-23T18:09:12.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.936+0000 7fba8b7fe700 1 -- 172.21.15.50:0/3033095088 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba90019550 con 0x7fba94108fb0 2024-04-23T18:09:12.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.936+0000 7fba9c307700 1 -- 172.21.15.50:0/3033095088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba941a1cd0 con 0x7fba94108fb0 2024-04-23T18:09:12.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.936+0000 7fba8b7fe700 1 -- 172.21.15.50:0/3033095088 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fba9001a8e0 con 0x7fba94108fb0 2024-04-23T18:09:12.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.937+0000 7fba9c307700 1 -- 172.21.15.50:0/3033095088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba940640a0 con 0x7fba94108fb0 2024-04-23T18:09:12.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.937+0000 7fba8b7fe700 1 --2- 172.21.15.50:0/3033095088 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fba800395b0 0x7fba8003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:12.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.937+0000 7fba8b7fe700 1 -- 172.21.15.50:0/3033095088 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fba9004d8d0 con 0x7fba94108fb0 2024-04-23T18:09:12.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.937+0000 7fba998a2700 1 --2- 172.21.15.50:0/3033095088 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fba800395b0 0x7fba8003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:12.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.937+0000 7fba998a2700 1 --2- 172.21.15.50:0/3033095088 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fba800395b0 0x7fba8003ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fba84006fd0 tx=0x7fba84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:11.942+0000 7fba8b7fe700 1 -- 172.21.15.50:0/3033095088 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba9000b3e0 con 0x7fba94108fb0 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:12.150+0000 7fba9c307700 1 -- 172.21.15.50:0/3033095088 --> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fba94103c40 con 0x7fba800395b0 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:12.327+0000 7fba8b7fe700 1 -- 172.21.15.50:0/3033095088 <== mgr.14120 v2:172.21.15.50:6800/103327972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fba94103c40 con 0x7fba800395b0 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:12.329+0000 7fba9c307700 1 -- 172.21.15.50:0/3033095088 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fba800395b0 msgr2=0x7fba8003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:12.329+0000 7fba9c307700 1 --2- 172.21.15.50:0/3033095088 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fba800395b0 0x7fba8003ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fba84006fd0 tx=0x7fba84006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:12.329+0000 7fba9c307700 1 -- 172.21.15.50:0/3033095088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 msgr2=0x7fba9419e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:12.329+0000 7fba9c307700 1 --2- 172.21.15.50:0/3033095088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 0x7fba9419e810 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fba90009fb0 tx=0x7fba90011940 comp rx=0 tx=0).stop 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:12.329+0000 7fba9c307700 1 -- 172.21.15.50:0/3033095088 shutdown_connections 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:12.329+0000 7fba9c307700 1 --2- 172.21.15.50:0/3033095088 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fba800395b0 0x7fba8003ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:12.329+0000 7fba9c307700 1 --2- 172.21.15.50:0/3033095088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba94108fb0 0x7fba9419e810 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:12.329+0000 7fba9c307700 1 -- 172.21.15.50:0/3033095088 >> 172.21.15.50:0/3033095088 conn(0x7fba94102a60 msgr2=0x7fba94103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:12.329+0000 7fba9c307700 1 -- 172.21.15.50:0/3033095088 shutdown_connections 2024-04-23T18:09:12.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:12.329+0000 7fba9c307700 1 -- 172.21.15.50:0/3033095088 wait complete. 2024-04-23T18:09:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:13 smithi050 ceph-mon[29187]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:13 smithi050 ceph-mon[29187]: Generating ssh key... 2024-04-23T18:09:13.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:13 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:13.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:13 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:14.066 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDChIwbFEtv1dkUgFWQyQoV70XvzXBgQC+X1OE0+aItjE3IzVx546Wj67WgN4Y8pT1E9pvq93715nIGQnTRJaxVnEGnEwOCW10XZYVHaPckeqeoD4koPZ5S+cDAJMV5Dg+umW9zIv/SKpzstTHKFBg8EjONzPDjGkpMFl4gaaDSqkU2CJFr6hSUhXRsqF9I9xNKyMCWb/4khIlIEK7Eh+aSgckT6cXo0YFgTTqGS4izOpegt/8/q9JRvVe5vfnuJIdIiaSDAbBb9L3+51KIWnFYAYCsLKwGCA2s79zLPCxI39UKplPvdwKh89BO6uqudUaldBaLn36tfP8MVlXA7oV/jqqEvP06gzQT2rXv3fL6fPNq5+n0OQaoyuNYqiXshIiVGYo0bd/hmUsD3zw2pqcoOYhCeFQoj+74Ws6Cno5u8YsBotcdRzXFHaJSHfGwnKhHltUNptyAk0PwGWAf9nN0gJUPbPczBAWOQgbonxcj1qDn41ImaGJR/lLw3dv4u1k= ceph-5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:14.067 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.297+0000 7f06dfb10700 1 Processor -- start 2024-04-23T18:09:14.067 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.297+0000 7f06dfb10700 1 -- start start 2024-04-23T18:09:14.067 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.298+0000 7f06dfb10700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 0x7f06d810b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:14.067 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.298+0000 7f06dfb10700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f06d807bec0 con 0x7f06d8108fb0 2024-04-23T18:09:14.067 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.298+0000 7f06dd8ac700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 0x7f06d810b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:14.067 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.298+0000 7f06dd8ac700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 0x7f06d810b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57218/0 (socket says 172.21.15.50:57218) 2024-04-23T18:09:14.067 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.298+0000 7f06dd8ac700 1 -- 172.21.15.50:0/2729178465 learned_addr learned my addr 172.21.15.50:0/2729178465 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:14.067 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.298+0000 7f06dd8ac700 1 -- 172.21.15.50:0/2729178465 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d810b8e0 con 0x7f06d8108fb0 2024-04-23T18:09:14.067 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.298+0000 7f06dd8ac700 1 --2- 172.21.15.50:0/2729178465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 0x7f06d810b3a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f06d40098a0 tx=0x7f06d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9f977b24a98175 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:14.067 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.298+0000 7f06dc8aa700 1 -- 172.21.15.50:0/2729178465 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06d4004160 con 0x7f06d8108fb0 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.299+0000 7f06dc8aa700 1 -- 172.21.15.50:0/2729178465 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f06d4004300 con 0x7f06d8108fb0 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.299+0000 7f06dc8aa700 1 -- 172.21.15.50:0/2729178465 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06d4011af0 con 0x7f06d8108fb0 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.299+0000 7f06dfb10700 1 -- 172.21.15.50:0/2729178465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 msgr2=0x7f06d810b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.299+0000 7f06dfb10700 1 --2- 172.21.15.50:0/2729178465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 0x7f06d810b3a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f06d40098a0 tx=0x7f06d4009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.299+0000 7f06dfb10700 1 -- 172.21.15.50:0/2729178465 shutdown_connections 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.299+0000 7f06dfb10700 1 --2- 172.21.15.50:0/2729178465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 0x7f06d810b3a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.299+0000 7f06dfb10700 1 -- 172.21.15.50:0/2729178465 >> 172.21.15.50:0/2729178465 conn(0x7f06d8102a40 msgr2=0x7f06d8104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.299+0000 7f06dfb10700 1 -- 172.21.15.50:0/2729178465 shutdown_connections 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.299+0000 7f06dfb10700 1 -- 172.21.15.50:0/2729178465 wait complete. 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.300+0000 7f06dfb10700 1 Processor -- start 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.300+0000 7f06dfb10700 1 -- start start 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.300+0000 7f06dfb10700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 0x7f06d819c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.300+0000 7f06dfb10700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f06d810c8b0 con 0x7f06d8108fb0 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.300+0000 7f06dd8ac700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 0x7f06d819c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.300+0000 7f06dd8ac700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 0x7f06d819c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57232/0 (socket says 172.21.15.50:57232) 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.300+0000 7f06dd8ac700 1 -- 172.21.15.50:0/230328891 learned_addr learned my addr 172.21.15.50:0/230328891 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.300+0000 7f06dd8ac700 1 -- 172.21.15.50:0/230328891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d819c9f0 con 0x7f06d8108fb0 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.301+0000 7f06dd8ac700 1 --2- 172.21.15.50:0/230328891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 0x7f06d819c4b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f06d4009fb0 tx=0x7f06d4011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.301+0000 7f06ceffd700 1 -- 172.21.15.50:0/230328891 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06d40036a0 con 0x7f06d8108fb0 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.301+0000 7f06ceffd700 1 -- 172.21.15.50:0/230328891 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f06d401a430 con 0x7f06d8108fb0 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.301+0000 7f06dfb10700 1 -- 172.21.15.50:0/230328891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d819cc80 con 0x7f06d8108fb0 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.301+0000 7f06ceffd700 1 -- 172.21.15.50:0/230328891 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06d4019550 con 0x7f06d8108fb0 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.301+0000 7f06dfb10700 1 -- 172.21.15.50:0/230328891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d819f970 con 0x7f06d8108fb0 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.302+0000 7f06ceffd700 1 -- 172.21.15.50:0/230328891 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f06d4003840 con 0x7f06d8108fb0 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.302+0000 7f06ceffd700 1 --2- 172.21.15.50:0/230328891 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f06c4039560 0x7f06c403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:14.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.302+0000 7f06dfb10700 1 -- 172.21.15.50:0/230328891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d80640a0 con 0x7f06d8108fb0 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.302+0000 7f06ceffd700 1 -- 172.21.15.50:0/230328891 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f06d404d930 con 0x7f06d8108fb0 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.302+0000 7f06dd0ab700 1 --2- 172.21.15.50:0/230328891 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f06c4039560 0x7f06c403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.302+0000 7f06dd0ab700 1 --2- 172.21.15.50:0/230328891 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f06c4039560 0x7f06c403ba20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f06c8006fd0 tx=0x7f06c8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.307+0000 7f06ceffd700 1 -- 172.21.15.50:0/230328891 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f06d4003b30 con 0x7f06d8108fb0 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.516+0000 7f06dfb10700 1 -- 172.21.15.50:0/230328891 --> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f06d8103d00 con 0x7f06c4039560 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.517+0000 7f06ceffd700 1 -- 172.21.15.50:0/230328891 <== mgr.14120 v2:172.21.15.50:6800/103327972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f06d8103d00 con 0x7f06c4039560 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.519+0000 7f06dfb10700 1 -- 172.21.15.50:0/230328891 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f06c4039560 msgr2=0x7f06c403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.519+0000 7f06dfb10700 1 --2- 172.21.15.50:0/230328891 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f06c4039560 0x7f06c403ba20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f06c8006fd0 tx=0x7f06c8006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.519+0000 7f06dfb10700 1 -- 172.21.15.50:0/230328891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 msgr2=0x7f06d819c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.519+0000 7f06dfb10700 1 --2- 172.21.15.50:0/230328891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 0x7f06d819c4b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f06d4009fb0 tx=0x7f06d4011940 comp rx=0 tx=0).stop 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.519+0000 7f06dfb10700 1 -- 172.21.15.50:0/230328891 shutdown_connections 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.519+0000 7f06dfb10700 1 --2- 172.21.15.50:0/230328891 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f06c4039560 0x7f06c403ba20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.519+0000 7f06dfb10700 1 --2- 172.21.15.50:0/230328891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d8108fb0 0x7f06d819c4b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.519+0000 7f06dfb10700 1 -- 172.21.15.50:0/230328891 >> 172.21.15.50:0/230328891 conn(0x7f06d8102a40 msgr2=0x7f06d81035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:14.071 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.520+0000 7f06dfb10700 1 -- 172.21.15.50:0/230328891 shutdown_connections 2024-04-23T18:09:14.071 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:13.520+0000 7f06dfb10700 1 -- 172.21.15.50:0/230328891 wait complete. 2024-04-23T18:09:14.071 INFO:teuthology.orchestra.run.smithi050.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-23T18:09:14.071 INFO:teuthology.orchestra.run.smithi050.stdout:Adding key to root@localhost authorized_keys... 2024-04-23T18:09:14.071 INFO:teuthology.orchestra.run.smithi050.stdout:Adding host smithi050... 2024-04-23T18:09:14.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:14 smithi050 ceph-mon[29187]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:15.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:15 smithi050 ceph-mon[29187]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi050", "addr": "172.21.15.50", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:15.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:15 smithi050 ceph-mon[29187]: Deploying cephadm binary to smithi050 2024-04-23T18:09:19.646 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout Added host 'smithi050' with addr '172.21.15.50' 2024-04-23T18:09:19.646 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.498+0000 7fda356c8700 1 Processor -- start 2024-04-23T18:09:19.647 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.498+0000 7fda356c8700 1 -- start start 2024-04-23T18:09:19.647 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.498+0000 7fda356c8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 0x7fda3010b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:19.647 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.498+0000 7fda356c8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fda3007be80 con 0x7fda30109000 2024-04-23T18:09:19.647 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.499+0000 7fda2effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 0x7fda3010b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:19.647 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.499+0000 7fda2effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 0x7fda3010b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57234/0 (socket says 172.21.15.50:57234) 2024-04-23T18:09:19.647 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.499+0000 7fda2effd700 1 -- 172.21.15.50:0/726128273 learned_addr learned my addr 172.21.15.50:0/726128273 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:19.647 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.499+0000 7fda2effd700 1 -- 172.21.15.50:0/726128273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3010b930 con 0x7fda30109000 2024-04-23T18:09:19.647 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.499+0000 7fda2effd700 1 --2- 172.21.15.50:0/726128273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 0x7fda3010b3f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fda20009e80 tx=0x7fda2000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53a918e5b6cc5c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:19.647 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.499+0000 7fda2dffb700 1 -- 172.21.15.50:0/726128273 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda20017070 con 0x7fda30109000 2024-04-23T18:09:19.647 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.499+0000 7fda2dffb700 1 -- 172.21.15.50:0/726128273 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fda2000b8c0 con 0x7fda30109000 2024-04-23T18:09:19.647 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.499+0000 7fda2dffb700 1 -- 172.21.15.50:0/726128273 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda20004490 con 0x7fda30109000 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.500+0000 7fda356c8700 1 -- 172.21.15.50:0/726128273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 msgr2=0x7fda3010b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.500+0000 7fda356c8700 1 --2- 172.21.15.50:0/726128273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 0x7fda3010b3f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fda20009e80 tx=0x7fda2000b330 comp rx=0 tx=0).stop 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.500+0000 7fda356c8700 1 -- 172.21.15.50:0/726128273 shutdown_connections 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.500+0000 7fda356c8700 1 --2- 172.21.15.50:0/726128273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 0x7fda3010b3f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.500+0000 7fda356c8700 1 -- 172.21.15.50:0/726128273 >> 172.21.15.50:0/726128273 conn(0x7fda30102a50 msgr2=0x7fda30104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.500+0000 7fda356c8700 1 -- 172.21.15.50:0/726128273 shutdown_connections 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.500+0000 7fda356c8700 1 -- 172.21.15.50:0/726128273 wait complete. 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.501+0000 7fda356c8700 1 Processor -- start 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.501+0000 7fda356c8700 1 -- start start 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.501+0000 7fda356c8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 0x7fda3019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.501+0000 7fda356c8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fda3010c8f0 con 0x7fda30109000 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.501+0000 7fda2effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 0x7fda3019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.501+0000 7fda2effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 0x7fda3019e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57246/0 (socket says 172.21.15.50:57246) 2024-04-23T18:09:19.648 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.501+0000 7fda2effd700 1 -- 172.21.15.50:0/3024241273 learned_addr learned my addr 172.21.15.50:0/3024241273 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.501+0000 7fda2effd700 1 -- 172.21.15.50:0/3024241273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3019eba0 con 0x7fda30109000 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.502+0000 7fda2effd700 1 --2- 172.21.15.50:0/3024241273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 0x7fda3019e660 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda2000b7b0 tx=0x7fda2000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.502+0000 7fda17fff700 1 -- 172.21.15.50:0/3024241273 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda20020020 con 0x7fda30109000 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.502+0000 7fda17fff700 1 -- 172.21.15.50:0/3024241273 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fda20012430 con 0x7fda30109000 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.502+0000 7fda356c8700 1 -- 172.21.15.50:0/3024241273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda3019ee30 con 0x7fda30109000 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.502+0000 7fda17fff700 1 -- 172.21.15.50:0/3024241273 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda20017040 con 0x7fda30109000 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.502+0000 7fda356c8700 1 -- 172.21.15.50:0/3024241273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda301a1b20 con 0x7fda30109000 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.503+0000 7fda17fff700 1 -- 172.21.15.50:0/3024241273 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fda20023020 con 0x7fda30109000 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.503+0000 7fda17fff700 1 --2- 172.21.15.50:0/3024241273 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fda180395b0 0x7fda1803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.503+0000 7fda356c8700 1 -- 172.21.15.50:0/3024241273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda300640a0 con 0x7fda30109000 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.503+0000 7fda17fff700 1 -- 172.21.15.50:0/3024241273 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fda2000f050 con 0x7fda30109000 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.503+0000 7fda2e7fc700 1 --2- 172.21.15.50:0/3024241273 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fda180395b0 0x7fda1803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.503+0000 7fda2e7fc700 1 --2- 172.21.15.50:0/3024241273 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fda180395b0 0x7fda1803ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fda24006fd0 tx=0x7fda24006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:19.649 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.508+0000 7fda17fff700 1 -- 172.21.15.50:0/3024241273 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda2002a020 con 0x7fda30109000 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:14.726+0000 7fda356c8700 1 -- 172.21.15.50:0/3024241273 --> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi050", "addr": "172.21.15.50", "target": ["mon-mgr", ""]}) -- 0x7fda30103be0 con 0x7fda180395b0 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:18.992+0000 7fda17fff700 1 -- 172.21.15.50:0/3024241273 <== mgr.14120 v2:172.21.15.50:6800/103327972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fda30103be0 con 0x7fda180395b0 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:18.995+0000 7fda356c8700 1 -- 172.21.15.50:0/3024241273 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fda180395b0 msgr2=0x7fda1803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:18.995+0000 7fda356c8700 1 --2- 172.21.15.50:0/3024241273 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fda180395b0 0x7fda1803ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fda24006fd0 tx=0x7fda24006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:18.995+0000 7fda356c8700 1 -- 172.21.15.50:0/3024241273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 msgr2=0x7fda3019e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:18.995+0000 7fda356c8700 1 --2- 172.21.15.50:0/3024241273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 0x7fda3019e660 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda2000b7b0 tx=0x7fda2000be20 comp rx=0 tx=0).stop 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:18.995+0000 7fda356c8700 1 -- 172.21.15.50:0/3024241273 shutdown_connections 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:18.995+0000 7fda356c8700 1 --2- 172.21.15.50:0/3024241273 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fda180395b0 0x7fda1803ba70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:18.995+0000 7fda356c8700 1 --2- 172.21.15.50:0/3024241273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30109000 0x7fda3019e660 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:18.995+0000 7fda356c8700 1 -- 172.21.15.50:0/3024241273 >> 172.21.15.50:0/3024241273 conn(0x7fda30102a50 msgr2=0x7fda301034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:18.995+0000 7fda356c8700 1 -- 172.21.15.50:0/3024241273 shutdown_connections 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:18.995+0000 7fda356c8700 1 -- 172.21.15.50:0/3024241273 wait complete. 2024-04-23T18:09:19.650 INFO:teuthology.orchestra.run.smithi050.stdout:Deploying unmanaged mon service... 2024-04-23T18:09:20.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:19 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:20.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:19 smithi050 ceph-mon[29187]: Added host smithi050 2024-04-23T18:09:20.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:19 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:20.874 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-23T18:09:20.875 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.111+0000 7f9a529fc700 1 Processor -- start 2024-04-23T18:09:20.875 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.111+0000 7f9a529fc700 1 -- start start 2024-04-23T18:09:20.875 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.111+0000 7f9a529fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 0x7f9a4c10b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:20.875 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.111+0000 7f9a529fc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4c07be80 con 0x7f9a4c108ff0 2024-04-23T18:09:20.875 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.111+0000 7f9a4bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 0x7f9a4c10b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:20.875 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.111+0000 7f9a4bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 0x7f9a4c10b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57256/0 (socket says 172.21.15.50:57256) 2024-04-23T18:09:20.876 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.111+0000 7f9a4bfff700 1 -- 172.21.15.50:0/3309662491 learned_addr learned my addr 172.21.15.50:0/3309662491 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:20.876 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.111+0000 7f9a4bfff700 1 -- 172.21.15.50:0/3309662491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a4c10b920 con 0x7f9a4c108ff0 2024-04-23T18:09:20.876 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.112+0000 7f9a4bfff700 1 --2- 172.21.15.50:0/3309662491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 0x7f9a4c10b3e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0098a0 tx=0x7f9a3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954b2f9ba11aba21 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:20.876 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.112+0000 7f9a4affd700 1 -- 172.21.15.50:0/3309662491 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a3c004160 con 0x7f9a4c108ff0 2024-04-23T18:09:20.876 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.112+0000 7f9a4affd700 1 -- 172.21.15.50:0/3309662491 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9a3c004300 con 0x7f9a4c108ff0 2024-04-23T18:09:20.876 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.112+0000 7f9a4affd700 1 -- 172.21.15.50:0/3309662491 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a3c011af0 con 0x7f9a4c108ff0 2024-04-23T18:09:20.877 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.112+0000 7f9a529fc700 1 -- 172.21.15.50:0/3309662491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 msgr2=0x7f9a4c10b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:20.877 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.112+0000 7f9a529fc700 1 --2- 172.21.15.50:0/3309662491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 0x7f9a4c10b3e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0098a0 tx=0x7f9a3c009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:20.877 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.112+0000 7f9a529fc700 1 -- 172.21.15.50:0/3309662491 shutdown_connections 2024-04-23T18:09:20.877 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.112+0000 7f9a529fc700 1 --2- 172.21.15.50:0/3309662491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 0x7f9a4c10b3e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:20.877 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.112+0000 7f9a529fc700 1 -- 172.21.15.50:0/3309662491 >> 172.21.15.50:0/3309662491 conn(0x7f9a4c102a40 msgr2=0x7f9a4c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:20.877 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.112+0000 7f9a529fc700 1 -- 172.21.15.50:0/3309662491 shutdown_connections 2024-04-23T18:09:20.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.112+0000 7f9a529fc700 1 -- 172.21.15.50:0/3309662491 wait complete. 2024-04-23T18:09:20.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.113+0000 7f9a529fc700 1 Processor -- start 2024-04-23T18:09:20.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.113+0000 7f9a529fc700 1 -- start start 2024-04-23T18:09:20.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.113+0000 7f9a529fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 0x7f9a4c19c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:20.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.113+0000 7f9a529fc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4c10c8e0 con 0x7f9a4c108ff0 2024-04-23T18:09:20.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.113+0000 7f9a4bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 0x7f9a4c19c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:20.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.113+0000 7f9a4bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 0x7f9a4c19c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57262/0 (socket says 172.21.15.50:57262) 2024-04-23T18:09:20.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.113+0000 7f9a4bfff700 1 -- 172.21.15.50:0/3799061797 learned_addr learned my addr 172.21.15.50:0/3799061797 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:20.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.114+0000 7f9a4bfff700 1 -- 172.21.15.50:0/3799061797 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a4c19c9d0 con 0x7f9a4c108ff0 2024-04-23T18:09:20.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.114+0000 7f9a4bfff700 1 --2- 172.21.15.50:0/3799061797 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 0x7f9a4c19c490 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009fb0 tx=0x7f9a3c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:20.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.114+0000 7f9a497fa700 1 -- 172.21.15.50:0/3799061797 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a3c0036a0 con 0x7f9a4c108ff0 2024-04-23T18:09:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.114+0000 7f9a497fa700 1 -- 172.21.15.50:0/3799061797 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9a3c01a430 con 0x7f9a4c108ff0 2024-04-23T18:09:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.114+0000 7f9a529fc700 1 -- 172.21.15.50:0/3799061797 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a4c19cc60 con 0x7f9a4c108ff0 2024-04-23T18:09:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.114+0000 7f9a497fa700 1 -- 172.21.15.50:0/3799061797 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a3c019550 con 0x7f9a4c108ff0 2024-04-23T18:09:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.114+0000 7f9a529fc700 1 -- 172.21.15.50:0/3799061797 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a4c19f950 con 0x7f9a4c108ff0 2024-04-23T18:09:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.115+0000 7f9a497fa700 1 -- 172.21.15.50:0/3799061797 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f9a3c019800 con 0x7f9a4c108ff0 2024-04-23T18:09:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.115+0000 7f9a529fc700 1 -- 172.21.15.50:0/3799061797 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a4c0640a0 con 0x7f9a4c108ff0 2024-04-23T18:09:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.115+0000 7f9a497fa700 1 --2- 172.21.15.50:0/3799061797 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f9a340395b0 0x7f9a3403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.115+0000 7f9a497fa700 1 -- 172.21.15.50:0/3799061797 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9a3c04da30 con 0x7f9a4c108ff0 2024-04-23T18:09:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.115+0000 7f9a4b7fe700 1 --2- 172.21.15.50:0/3799061797 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f9a340395b0 0x7f9a3403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.116+0000 7f9a4b7fe700 1 --2- 172.21.15.50:0/3799061797 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f9a340395b0 0x7f9a3403ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a40006fd0 tx=0x7f9a40006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.120+0000 7f9a497fa700 1 -- 172.21.15.50:0/3799061797 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a3c01a5e0 con 0x7f9a4c108ff0 2024-04-23T18:09:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.344+0000 7f9a529fc700 1 -- 172.21.15.50:0/3799061797 --> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f9a4c103cf0 con 0x7f9a340395b0 2024-04-23T18:09:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.350+0000 7f9a497fa700 1 -- 172.21.15.50:0/3799061797 <== mgr.14120 v2:172.21.15.50:6800/103327972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9a4c103cf0 con 0x7f9a340395b0 2024-04-23T18:09:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.353+0000 7f9a529fc700 1 -- 172.21.15.50:0/3799061797 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f9a340395b0 msgr2=0x7f9a3403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.353+0000 7f9a529fc700 1 --2- 172.21.15.50:0/3799061797 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f9a340395b0 0x7f9a3403ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a40006fd0 tx=0x7f9a40006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.353+0000 7f9a529fc700 1 -- 172.21.15.50:0/3799061797 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 msgr2=0x7f9a4c19c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.353+0000 7f9a529fc700 1 --2- 172.21.15.50:0/3799061797 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 0x7f9a4c19c490 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009fb0 tx=0x7f9a3c011940 comp rx=0 tx=0).stop 2024-04-23T18:09:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.353+0000 7f9a529fc700 1 -- 172.21.15.50:0/3799061797 shutdown_connections 2024-04-23T18:09:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.353+0000 7f9a529fc700 1 --2- 172.21.15.50:0/3799061797 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f9a340395b0 0x7f9a3403ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.353+0000 7f9a529fc700 1 --2- 172.21.15.50:0/3799061797 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a4c108ff0 0x7f9a4c19c490 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.353+0000 7f9a529fc700 1 -- 172.21.15.50:0/3799061797 >> 172.21.15.50:0/3799061797 conn(0x7f9a4c102a40 msgr2=0x7f9a4c1035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.353+0000 7f9a529fc700 1 -- 172.21.15.50:0/3799061797 shutdown_connections 2024-04-23T18:09:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:20.353+0000 7f9a529fc700 1 -- 172.21.15.50:0/3799061797 wait complete. 2024-04-23T18:09:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:Deploying unmanaged mgr service... 2024-04-23T18:09:21.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:21 smithi050 ceph-mon[29187]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:21.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:21 smithi050 ceph-mon[29187]: Saving service mon spec with placement count:5 2024-04-23T18:09:21.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:21 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:22.867 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:22 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:22.867 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:22 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:22.867 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:22 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:23.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-23T18:09:23.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.321+0000 7f2eb58c3700 1 Processor -- start 2024-04-23T18:09:23.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.321+0000 7f2eb58c3700 1 -- start start 2024-04-23T18:09:23.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.321+0000 7f2eb58c3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 0x7f2eb0108960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:23.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.321+0000 7f2eb58c3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb0108f30 con 0x7f2eb0108540 2024-04-23T18:09:23.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.322+0000 7f2eaeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 0x7f2eb0108960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:23.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.322+0000 7f2eaeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 0x7f2eb0108960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39854/0 (socket says 172.21.15.50:39854) 2024-04-23T18:09:23.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.322+0000 7f2eaeffd700 1 -- 172.21.15.50:0/563666009 learned_addr learned my addr 172.21.15.50:0/563666009 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:23.142 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.322+0000 7f2eaeffd700 1 -- 172.21.15.50:0/563666009 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb01096f0 con 0x7f2eb0108540 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.322+0000 7f2eaeffd700 1 --2- 172.21.15.50:0/563666009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 0x7f2eb0108960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00098a0 tx=0x7f2ea0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abf2ca0ef1ba367 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.322+0000 7f2eadffb700 1 -- 172.21.15.50:0/563666009 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ea0004160 con 0x7f2eb0108540 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.322+0000 7f2eadffb700 1 -- 172.21.15.50:0/563666009 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ea0004300 con 0x7f2eb0108540 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.322+0000 7f2eadffb700 1 -- 172.21.15.50:0/563666009 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ea0011a20 con 0x7f2eb0108540 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.323+0000 7f2eb58c3700 1 -- 172.21.15.50:0/563666009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 msgr2=0x7f2eb0108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.323+0000 7f2eb58c3700 1 --2- 172.21.15.50:0/563666009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 0x7f2eb0108960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00098a0 tx=0x7f2ea0009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.324+0000 7f2eb58c3700 1 -- 172.21.15.50:0/563666009 shutdown_connections 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.324+0000 7f2eb58c3700 1 --2- 172.21.15.50:0/563666009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 0x7f2eb0108960 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.324+0000 7f2eb58c3700 1 -- 172.21.15.50:0/563666009 >> 172.21.15.50:0/563666009 conn(0x7f2eb0103b40 msgr2=0x7f2eb0105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.324+0000 7f2eb58c3700 1 -- 172.21.15.50:0/563666009 shutdown_connections 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.324+0000 7f2eb58c3700 1 -- 172.21.15.50:0/563666009 wait complete. 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.325+0000 7f2eb58c3700 1 Processor -- start 2024-04-23T18:09:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.325+0000 7f2eb58c3700 1 -- start start 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.325+0000 7f2eb58c3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 0x7f2eb019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.325+0000 7f2eb58c3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb007c0f0 con 0x7f2eb0108540 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.325+0000 7f2eaeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 0x7f2eb019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.325+0000 7f2eaeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 0x7f2eb019a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39868/0 (socket says 172.21.15.50:39868) 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.325+0000 7f2eaeffd700 1 -- 172.21.15.50:0/774998593 learned_addr learned my addr 172.21.15.50:0/774998593 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.326+0000 7f2eaeffd700 1 -- 172.21.15.50:0/774998593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb019a7f0 con 0x7f2eb0108540 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.326+0000 7f2eaeffd700 1 --2- 172.21.15.50:0/774998593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 0x7f2eb019a2b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00119f0 tx=0x7f2ea0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.326+0000 7f2eb48c1700 1 -- 172.21.15.50:0/774998593 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ea00039e0 con 0x7f2eb0108540 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.326+0000 7f2eb58c3700 1 -- 172.21.15.50:0/774998593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb019aa80 con 0x7f2eb0108540 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.326+0000 7f2eb58c3700 1 -- 172.21.15.50:0/774998593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb019d770 con 0x7f2eb0108540 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.326+0000 7f2eb48c1700 1 -- 172.21.15.50:0/774998593 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ea001b8f0 con 0x7f2eb0108540 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.326+0000 7f2eb48c1700 1 -- 172.21.15.50:0/774998593 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ea00237a0 con 0x7f2eb0108540 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.327+0000 7f2eb48c1700 1 -- 172.21.15.50:0/774998593 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f2ea002c3f0 con 0x7f2eb0108540 2024-04-23T18:09:23.144 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.327+0000 7f2eb58c3700 1 -- 172.21.15.50:0/774998593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eb00640a0 con 0x7f2eb0108540 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.327+0000 7f2eb48c1700 1 --2- 172.21.15.50:0/774998593 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f2e9803d9e0 0x7f2e9803fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.328+0000 7f2eb48c1700 1 -- 172.21.15.50:0/774998593 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2ea004de50 con 0x7f2eb0108540 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.328+0000 7f2eae7fc700 1 --2- 172.21.15.50:0/774998593 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f2e9803d9e0 0x7f2e9803fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.328+0000 7f2eae7fc700 1 --2- 172.21.15.50:0/774998593 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f2e9803d9e0 0x7f2e9803fea0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4006fd0 tx=0x7f2ea4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.337+0000 7f2eb48c1700 1 -- 172.21.15.50:0/774998593 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ea001f030 con 0x7f2eb0108540 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.562+0000 7f2eb58c3700 1 -- 172.21.15.50:0/774998593 --> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f2eb002e240 con 0x7f2e9803d9e0 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.567+0000 7f2eb48c1700 1 -- 172.21.15.50:0/774998593 <== mgr.14120 v2:172.21.15.50:6800/103327972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2eb002e240 con 0x7f2e9803d9e0 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.569+0000 7f2eb58c3700 1 -- 172.21.15.50:0/774998593 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f2e9803d9e0 msgr2=0x7f2e9803fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.569+0000 7f2eb58c3700 1 --2- 172.21.15.50:0/774998593 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f2e9803d9e0 0x7f2e9803fea0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4006fd0 tx=0x7f2ea4006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.569+0000 7f2eb58c3700 1 -- 172.21.15.50:0/774998593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 msgr2=0x7f2eb019a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.569+0000 7f2eb58c3700 1 --2- 172.21.15.50:0/774998593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 0x7f2eb019a2b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00119f0 tx=0x7f2ea0011ad0 comp rx=0 tx=0).stop 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.569+0000 7f2eb58c3700 1 -- 172.21.15.50:0/774998593 shutdown_connections 2024-04-23T18:09:23.145 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.569+0000 7f2eb58c3700 1 --2- 172.21.15.50:0/774998593 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f2e9803d9e0 0x7f2e9803fea0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:23.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.569+0000 7f2eb58c3700 1 --2- 172.21.15.50:0/774998593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eb0108540 0x7f2eb019a2b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:23.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.569+0000 7f2eb58c3700 1 -- 172.21.15.50:0/774998593 >> 172.21.15.50:0/774998593 conn(0x7f2eb0103b40 msgr2=0x7f2eb0103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:23.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.569+0000 7f2eb58c3700 1 -- 172.21.15.50:0/774998593 shutdown_connections 2024-04-23T18:09:23.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:22.569+0000 7f2eb58c3700 1 -- 172.21.15.50:0/774998593 wait complete. 2024-04-23T18:09:24.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:23 smithi050 ceph-mon[29187]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:24.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:23 smithi050 ceph-mon[29187]: Saving service mgr spec with placement count:2 2024-04-23T18:09:24.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:23 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2246004532' entity='client.admin' 2024-04-23T18:09:24.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.543+0000 7fa86e586700 1 Processor -- start 2024-04-23T18:09:24.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.543+0000 7fa86e586700 1 -- start start 2024-04-23T18:09:24.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.543+0000 7fa86e586700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 0x7fa86810b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:24.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.543+0000 7fa86e586700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa86807bec0 con 0x7fa868108fc0 2024-04-23T18:09:24.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.544+0000 7fa867fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 0x7fa86810b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:24.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.544+0000 7fa867fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 0x7fa86810b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39884/0 (socket says 172.21.15.50:39884) 2024-04-23T18:09:24.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.544+0000 7fa867fff700 1 -- 172.21.15.50:0/2201228305 learned_addr learned my addr 172.21.15.50:0/2201228305 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:24.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.544+0000 7fa867fff700 1 -- 172.21.15.50:0/2201228305 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86810b8f0 con 0x7fa868108fc0 2024-04-23T18:09:24.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.544+0000 7fa867fff700 1 --2- 172.21.15.50:0/2201228305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 0x7fa86810b3b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa8580098a0 tx=0x7fa858009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce54ae53f9b2c46a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:24.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.544+0000 7fa866ffd700 1 -- 172.21.15.50:0/2201228305 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa858004160 con 0x7fa868108fc0 2024-04-23T18:09:24.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.545+0000 7fa866ffd700 1 -- 172.21.15.50:0/2201228305 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa858004300 con 0x7fa868108fc0 2024-04-23T18:09:24.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.545+0000 7fa866ffd700 1 -- 172.21.15.50:0/2201228305 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa858011a20 con 0x7fa868108fc0 2024-04-23T18:09:24.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.545+0000 7fa86e586700 1 -- 172.21.15.50:0/2201228305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 msgr2=0x7fa86810b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:24.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.545+0000 7fa86e586700 1 --2- 172.21.15.50:0/2201228305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 0x7fa86810b3b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa8580098a0 tx=0x7fa858009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:24.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.545+0000 7fa86e586700 1 -- 172.21.15.50:0/2201228305 shutdown_connections 2024-04-23T18:09:24.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.545+0000 7fa86e586700 1 --2- 172.21.15.50:0/2201228305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 0x7fa86810b3b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:24.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.545+0000 7fa86e586700 1 -- 172.21.15.50:0/2201228305 >> 172.21.15.50:0/2201228305 conn(0x7fa868102a50 msgr2=0x7fa868104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:24.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.545+0000 7fa86e586700 1 -- 172.21.15.50:0/2201228305 shutdown_connections 2024-04-23T18:09:24.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.545+0000 7fa86e586700 1 -- 172.21.15.50:0/2201228305 wait complete. 2024-04-23T18:09:24.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.546+0000 7fa86e586700 1 Processor -- start 2024-04-23T18:09:24.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.546+0000 7fa86e586700 1 -- start start 2024-04-23T18:09:24.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.547+0000 7fa86e586700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 0x7fa86819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:24.537 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.547+0000 7fa86e586700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa86810c900 con 0x7fa868108fc0 2024-04-23T18:09:24.537 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.547+0000 7fa867fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 0x7fa86819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:24.537 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.547+0000 7fa867fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 0x7fa86819e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39890/0 (socket says 172.21.15.50:39890) 2024-04-23T18:09:24.537 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.547+0000 7fa867fff700 1 -- 172.21.15.50:0/2246004532 learned_addr learned my addr 172.21.15.50:0/2246004532 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:24.537 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.547+0000 7fa867fff700 1 -- 172.21.15.50:0/2246004532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86819ebc0 con 0x7fa868108fc0 2024-04-23T18:09:24.537 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.547+0000 7fa867fff700 1 --2- 172.21.15.50:0/2246004532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 0x7fa86819e680 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa8580116e0 tx=0x7fa858011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:24.537 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.547+0000 7fa8657fa700 1 -- 172.21.15.50:0/2246004532 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa858003b20 con 0x7fa868108fc0 2024-04-23T18:09:24.537 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.547+0000 7fa8657fa700 1 -- 172.21.15.50:0/2246004532 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa85801b8f0 con 0x7fa868108fc0 2024-04-23T18:09:24.537 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.548+0000 7fa86e586700 1 -- 172.21.15.50:0/2246004532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa86819ee50 con 0x7fa868108fc0 2024-04-23T18:09:24.537 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.547+0000 7fa8657fa700 1 -- 172.21.15.50:0/2246004532 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8580238c0 con 0x7fa868108fc0 2024-04-23T18:09:24.537 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.548+0000 7fa86e586700 1 -- 172.21.15.50:0/2246004532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8681a1b40 con 0x7fa868108fc0 2024-04-23T18:09:24.538 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.548+0000 7fa8657fa700 1 -- 172.21.15.50:0/2246004532 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fa858023a60 con 0x7fa868108fc0 2024-04-23T18:09:24.538 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.549+0000 7fa8657fa700 1 --2- 172.21.15.50:0/2246004532 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fa8500395b0 0x7fa85003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:24.538 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.549+0000 7fa86e586700 1 -- 172.21.15.50:0/2246004532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8680640a0 con 0x7fa868108fc0 2024-04-23T18:09:24.538 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.549+0000 7fa8657fa700 1 -- 172.21.15.50:0/2246004532 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa85804e3d0 con 0x7fa868108fc0 2024-04-23T18:09:24.538 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.549+0000 7fa8677fe700 1 --2- 172.21.15.50:0/2246004532 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fa8500395b0 0x7fa85003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:24.538 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.549+0000 7fa8677fe700 1 --2- 172.21.15.50:0/2246004532 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fa8500395b0 0x7fa85003ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa85c006fd0 tx=0x7fa85c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:24.538 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.555+0000 7fa8657fa700 1 -- 172.21.15.50:0/2246004532 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa85801f030 con 0x7fa868108fc0 2024-04-23T18:09:24.538 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.782+0000 7fa86e586700 1 -- 172.21.15.50:0/2246004532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fa8681a2180 con 0x7fa868108fc0 2024-04-23T18:09:24.538 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.785+0000 7fa8657fa700 1 -- 172.21.15.50:0/2246004532 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fa858016d20 con 0x7fa868108fc0 2024-04-23T18:09:24.538 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.787+0000 7fa86e586700 1 -- 172.21.15.50:0/2246004532 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fa8500395b0 msgr2=0x7fa85003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:24.538 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.787+0000 7fa86e586700 1 --2- 172.21.15.50:0/2246004532 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fa8500395b0 0x7fa85003ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa85c006fd0 tx=0x7fa85c006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:24.538 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.787+0000 7fa86e586700 1 -- 172.21.15.50:0/2246004532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 msgr2=0x7fa86819e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:24.539 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.787+0000 7fa86e586700 1 --2- 172.21.15.50:0/2246004532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 0x7fa86819e680 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa8580116e0 tx=0x7fa858011aa0 comp rx=0 tx=0).stop 2024-04-23T18:09:24.539 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.788+0000 7fa86e586700 1 -- 172.21.15.50:0/2246004532 shutdown_connections 2024-04-23T18:09:24.539 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.788+0000 7fa86e586700 1 --2- 172.21.15.50:0/2246004532 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fa8500395b0 0x7fa85003ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:24.539 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.788+0000 7fa86e586700 1 --2- 172.21.15.50:0/2246004532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa868108fc0 0x7fa86819e680 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:24.539 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.788+0000 7fa86e586700 1 -- 172.21.15.50:0/2246004532 >> 172.21.15.50:0/2246004532 conn(0x7fa868102a50 msgr2=0x7fa868103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:24.539 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.788+0000 7fa86e586700 1 -- 172.21.15.50:0/2246004532 shutdown_connections 2024-04-23T18:09:24.539 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:23.788+0000 7fa86e586700 1 -- 172.21.15.50:0/2246004532 wait complete. 2024-04-23T18:09:25.841 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.925+0000 7f3d60f97700 1 Processor -- start 2024-04-23T18:09:25.841 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.925+0000 7f3d60f97700 1 -- start start 2024-04-23T18:09:25.841 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.925+0000 7f3d60f97700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 0x7f3d5c108980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:25.841 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.925+0000 7f3d60f97700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c108f50 con 0x7f3d5c108560 2024-04-23T18:09:25.841 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.925+0000 7f3d5a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 0x7f3d5c108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:25.841 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.925+0000 7f3d5a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 0x7f3d5c108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39902/0 (socket says 172.21.15.50:39902) 2024-04-23T18:09:25.841 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.925+0000 7f3d5a59c700 1 -- 172.21.15.50:0/2065169823 learned_addr learned my addr 172.21.15.50:0/2065169823 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:25.841 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.926+0000 7f3d5a59c700 1 -- 172.21.15.50:0/2065169823 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c109710 con 0x7f3d5c108560 2024-04-23T18:09:25.841 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.926+0000 7f3d5a59c700 1 --2- 172.21.15.50:0/2065169823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 0x7f3d5c108980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0098a0 tx=0x7f3d4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7850288bcca688 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.926+0000 7f3d5959a700 1 -- 172.21.15.50:0/2065169823 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d4c004160 con 0x7f3d5c108560 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.926+0000 7f3d5959a700 1 -- 172.21.15.50:0/2065169823 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d4c004300 con 0x7f3d5c108560 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.926+0000 7f3d5959a700 1 -- 172.21.15.50:0/2065169823 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d4c011a20 con 0x7f3d5c108560 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.927+0000 7f3d60f97700 1 -- 172.21.15.50:0/2065169823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 msgr2=0x7f3d5c108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.927+0000 7f3d60f97700 1 --2- 172.21.15.50:0/2065169823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 0x7f3d5c108980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0098a0 tx=0x7f3d4c009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.927+0000 7f3d60f97700 1 -- 172.21.15.50:0/2065169823 shutdown_connections 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.927+0000 7f3d60f97700 1 --2- 172.21.15.50:0/2065169823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 0x7f3d5c108980 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.927+0000 7f3d60f97700 1 -- 172.21.15.50:0/2065169823 >> 172.21.15.50:0/2065169823 conn(0x7f3d5c103b40 msgr2=0x7f3d5c105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.927+0000 7f3d60f97700 1 -- 172.21.15.50:0/2065169823 shutdown_connections 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.927+0000 7f3d60f97700 1 -- 172.21.15.50:0/2065169823 wait complete. 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.928+0000 7f3d60f97700 1 Processor -- start 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.928+0000 7f3d60f97700 1 -- start start 2024-04-23T18:09:25.842 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.928+0000 7f3d60f97700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 0x7f3d5c07db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.928+0000 7f3d60f97700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c10cac0 con 0x7f3d5c108560 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.928+0000 7f3d5a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 0x7f3d5c07db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.928+0000 7f3d5a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 0x7f3d5c07db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39918/0 (socket says 172.21.15.50:39918) 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.928+0000 7f3d5a59c700 1 -- 172.21.15.50:0/3642095454 learned_addr learned my addr 172.21.15.50:0/3642095454 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.929+0000 7f3d5a59c700 1 -- 172.21.15.50:0/3642095454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c07e0c0 con 0x7f3d5c108560 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.929+0000 7f3d5a59c700 1 --2- 172.21.15.50:0/3642095454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 0x7f3d5c07db80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0119f0 tx=0x7f3d4c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.929+0000 7f3d4b7fe700 1 -- 172.21.15.50:0/3642095454 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d4c0039e0 con 0x7f3d5c108560 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.929+0000 7f3d60f97700 1 -- 172.21.15.50:0/3642095454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5c07c2b0 con 0x7f3d5c108560 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.929+0000 7f3d4b7fe700 1 -- 172.21.15.50:0/3642095454 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d4c01b8f0 con 0x7f3d5c108560 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.929+0000 7f3d4b7fe700 1 -- 172.21.15.50:0/3642095454 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d4c0237a0 con 0x7f3d5c108560 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.929+0000 7f3d60f97700 1 -- 172.21.15.50:0/3642095454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d5c07c7f0 con 0x7f3d5c108560 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.930+0000 7f3d4b7fe700 1 -- 172.21.15.50:0/3642095454 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f3d4c02c3f0 con 0x7f3d5c108560 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.930+0000 7f3d60f97700 1 -- 172.21.15.50:0/3642095454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d5c0640a0 con 0x7f3d5c108560 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.930+0000 7f3d4b7fe700 1 --2- 172.21.15.50:0/3642095454 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f3d44039560 0x7f3d4403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:25.843 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.930+0000 7f3d4b7fe700 1 -- 172.21.15.50:0/3642095454 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3d4c04e080 con 0x7f3d5c108560 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.930+0000 7f3d59d9b700 1 --2- 172.21.15.50:0/3642095454 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f3d44039560 0x7f3d4403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.931+0000 7f3d59d9b700 1 --2- 172.21.15.50:0/3642095454 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f3d44039560 0x7f3d4403ba20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3d50006fd0 tx=0x7f3d50006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:24.935+0000 7f3d4b7fe700 1 -- 172.21.15.50:0/3642095454 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d4c01f030 con 0x7f3d5c108560 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:25.143+0000 7f3d60f97700 1 -- 172.21.15.50:0/3642095454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f3d5c07ca70 con 0x7f3d5c108560 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:25.146+0000 7f3d4b7fe700 1 -- 172.21.15.50:0/3642095454 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f3d4c02b3a0 con 0x7f3d5c108560 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:25.148+0000 7f3d60f97700 1 -- 172.21.15.50:0/3642095454 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f3d44039560 msgr2=0x7f3d4403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:25.148+0000 7f3d60f97700 1 --2- 172.21.15.50:0/3642095454 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f3d44039560 0x7f3d4403ba20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3d50006fd0 tx=0x7f3d50006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:25.148+0000 7f3d60f97700 1 -- 172.21.15.50:0/3642095454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 msgr2=0x7f3d5c07db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:25.148+0000 7f3d60f97700 1 --2- 172.21.15.50:0/3642095454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 0x7f3d5c07db80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0119f0 tx=0x7f3d4c011ad0 comp rx=0 tx=0).stop 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:25.148+0000 7f3d60f97700 1 -- 172.21.15.50:0/3642095454 shutdown_connections 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:25.148+0000 7f3d60f97700 1 --2- 172.21.15.50:0/3642095454 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f3d44039560 0x7f3d4403ba20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:25.148+0000 7f3d60f97700 1 --2- 172.21.15.50:0/3642095454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d5c108560 0x7f3d5c07db80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:25.148+0000 7f3d60f97700 1 -- 172.21.15.50:0/3642095454 >> 172.21.15.50:0/3642095454 conn(0x7f3d5c103b40 msgr2=0x7f3d5c1046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:25.844 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:25.148+0000 7f3d60f97700 1 -- 172.21.15.50:0/3642095454 shutdown_connections 2024-04-23T18:09:25.845 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:25.148+0000 7f3d60f97700 1 -- 172.21.15.50:0/3642095454 wait complete. 2024-04-23T18:09:25.845 INFO:teuthology.orchestra.run.smithi050.stdout:Enabling the dashboard module... 2024-04-23T18:09:26.202 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:26 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3642095454' entity='client.admin' 2024-04-23T18:09:26.203 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:26 smithi050 ceph-mon[29187]: from='mgr.14120 172.21.15.50:0/3052049018' entity='mgr.x' 2024-04-23T18:09:27.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:27 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3015302436' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-23T18:09:27.416 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:27 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: ignoring --setuser ceph since I am not root 2024-04-23T18:09:27.416 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:27 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: ignoring --setgroup ceph since I am not root 2024-04-23T18:09:27.416 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:27 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:27.364+0000 7ff70f3ee200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:09:27.674 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:27 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:27.573+0000 7ff70f3ee200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:09:27.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.314+0000 7fdbcefa1700 1 Processor -- start 2024-04-23T18:09:27.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.314+0000 7fdbcefa1700 1 -- start start 2024-04-23T18:09:27.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.314+0000 7fdbcefa1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 0x7fdbc8108e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.314+0000 7fdbcefa1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc8109430 con 0x7fdbc8108a40 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.314+0000 7fdbccd3d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 0x7fdbc8108e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.315+0000 7fdbccd3d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 0x7fdbc8108e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39926/0 (socket says 172.21.15.50:39926) 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.315+0000 7fdbccd3d700 1 -- 172.21.15.50:0/1504153624 learned_addr learned my addr 172.21.15.50:0/1504153624 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.315+0000 7fdbccd3d700 1 -- 172.21.15.50:0/1504153624 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc81076f0 con 0x7fdbc8108a40 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.315+0000 7fdbccd3d700 1 --2- 172.21.15.50:0/1504153624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 0x7fdbc8108e60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc0098a0 tx=0x7fdbbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bfb62f29a4e0622 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.315+0000 7fdbc77fe700 1 -- 172.21.15.50:0/1504153624 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbbc004160 con 0x7fdbc8108a40 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.315+0000 7fdbc77fe700 1 -- 172.21.15.50:0/1504153624 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbbc004300 con 0x7fdbc8108a40 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.315+0000 7fdbc77fe700 1 -- 172.21.15.50:0/1504153624 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbbc011a20 con 0x7fdbc8108a40 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.316+0000 7fdbcefa1700 1 -- 172.21.15.50:0/1504153624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 msgr2=0x7fdbc8108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.316+0000 7fdbcefa1700 1 --2- 172.21.15.50:0/1504153624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 0x7fdbc8108e60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc0098a0 tx=0x7fdbbc009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.316+0000 7fdbcefa1700 1 -- 172.21.15.50:0/1504153624 shutdown_connections 2024-04-23T18:09:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.316+0000 7fdbcefa1700 1 --2- 172.21.15.50:0/1504153624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 0x7fdbc8108e60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.316+0000 7fdbcefa1700 1 -- 172.21.15.50:0/1504153624 >> 172.21.15.50:0/1504153624 conn(0x7fdbc81021a0 msgr2=0x7fdbc81045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.316+0000 7fdbcefa1700 1 -- 172.21.15.50:0/1504153624 shutdown_connections 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.316+0000 7fdbcefa1700 1 -- 172.21.15.50:0/1504153624 wait complete. 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.317+0000 7fdbcefa1700 1 Processor -- start 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.317+0000 7fdbcefa1700 1 -- start start 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.317+0000 7fdbcefa1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 0x7fdbc8107060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.317+0000 7fdbcefa1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc810bab0 con 0x7fdbc8108a40 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.317+0000 7fdbccd3d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 0x7fdbc8107060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.317+0000 7fdbccd3d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 0x7fdbc8107060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39942/0 (socket says 172.21.15.50:39942) 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.317+0000 7fdbccd3d700 1 -- 172.21.15.50:0/3015302436 learned_addr learned my addr 172.21.15.50:0/3015302436 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.318+0000 7fdbccd3d700 1 -- 172.21.15.50:0/3015302436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc819cf60 con 0x7fdbc8108a40 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.318+0000 7fdbccd3d700 1 --2- 172.21.15.50:0/3015302436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 0x7fdbc8107060 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009520 tx=0x7fdbbc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.318+0000 7fdbc5ffb700 1 -- 172.21.15.50:0/3015302436 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbbc0039e0 con 0x7fdbc8108a40 2024-04-23T18:09:27.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.318+0000 7fdbc5ffb700 1 -- 172.21.15.50:0/3015302436 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbbc01b8f0 con 0x7fdbc8108a40 2024-04-23T18:09:27.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.318+0000 7fdbcefa1700 1 -- 172.21.15.50:0/3015302436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc81a1570 con 0x7fdbc8108a40 2024-04-23T18:09:27.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.318+0000 7fdbc5ffb700 1 -- 172.21.15.50:0/3015302436 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbbc019770 con 0x7fdbc8108a40 2024-04-23T18:09:27.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.318+0000 7fdbcefa1700 1 -- 172.21.15.50:0/3015302436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbc81a1910 con 0x7fdbc8108a40 2024-04-23T18:09:27.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.319+0000 7fdbc5ffb700 1 -- 172.21.15.50:0/3015302436 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fdbbc019a70 con 0x7fdbc8108a40 2024-04-23T18:09:27.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.319+0000 7fdbcefa1700 1 -- 172.21.15.50:0/3015302436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbc80640a0 con 0x7fdbc8108a40 2024-04-23T18:09:27.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.319+0000 7fdbc5ffb700 1 --2- 172.21.15.50:0/3015302436 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fdbb80395b0 0x7fdbb803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:27.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.319+0000 7fdbc5ffb700 1 -- 172.21.15.50:0/3015302436 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdbbc04def0 con 0x7fdbc8108a40 2024-04-23T18:09:27.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.319+0000 7fdbc7fff700 1 --2- 172.21.15.50:0/3015302436 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fdbb80395b0 0x7fdbb803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:27.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.320+0000 7fdbc7fff700 1 --2- 172.21.15.50:0/3015302436 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fdbb80395b0 0x7fdbb803ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4006fd0 tx=0x7fdbb4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:27.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.329+0000 7fdbc5ffb700 1 -- 172.21.15.50:0/3015302436 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdbbc01f030 con 0x7fdbc8108a40 2024-04-23T18:09:27.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:26.606+0000 7fdbcefa1700 1 -- 172.21.15.50:0/3015302436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fdbc81a1d70 con 0x7fdbc8108a40 2024-04-23T18:09:27.710 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:27.148+0000 7fdbc5ffb700 1 -- 172.21.15.50:0/3015302436 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fdbbc0172b0 con 0x7fdbc8108a40 2024-04-23T18:09:27.711 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:27.150+0000 7fdbc5ffb700 1 -- 172.21.15.50:0/3015302436 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fdbbc01a8f0 con 0x7fdbc8108a40 2024-04-23T18:09:27.711 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:27.152+0000 7fdbcefa1700 1 -- 172.21.15.50:0/3015302436 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fdbb80395b0 msgr2=0x7fdbb803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:27.711 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:27.152+0000 7fdbcefa1700 1 --2- 172.21.15.50:0/3015302436 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fdbb80395b0 0x7fdbb803ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4006fd0 tx=0x7fdbb4006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:27.711 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:27.152+0000 7fdbcefa1700 1 -- 172.21.15.50:0/3015302436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 msgr2=0x7fdbc8107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:27.711 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:27.152+0000 7fdbcefa1700 1 --2- 172.21.15.50:0/3015302436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 0x7fdbc8107060 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009520 tx=0x7fdbbc011aa0 comp rx=0 tx=0).stop 2024-04-23T18:09:27.711 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:27.152+0000 7fdbcefa1700 1 -- 172.21.15.50:0/3015302436 shutdown_connections 2024-04-23T18:09:27.711 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:27.152+0000 7fdbcefa1700 1 --2- 172.21.15.50:0/3015302436 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7fdbb80395b0 0x7fdbb803ba70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:27.711 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:27.152+0000 7fdbcefa1700 1 --2- 172.21.15.50:0/3015302436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc8108a40 0x7fdbc8107060 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:27.711 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:27.152+0000 7fdbcefa1700 1 -- 172.21.15.50:0/3015302436 >> 172.21.15.50:0/3015302436 conn(0x7fdbc81021a0 msgr2=0x7fdbc8102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:27.711 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:27.152+0000 7fdbcefa1700 1 -- 172.21.15.50:0/3015302436 shutdown_connections 2024-04-23T18:09:27.711 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:27.152+0000 7fdbcefa1700 1 -- 172.21.15.50:0/3015302436 wait complete. 2024-04-23T18:09:27.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:27 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:27.680+0000 7ff70f3ee200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:09:28.337 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:28 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3015302436' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-23T18:09:28.337 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:28 smithi050 ceph-mon[29187]: mgrmap e9: x(active, since 19s) 2024-04-23T18:09:28.337 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:28 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:28.007+0000 7ff70f3ee200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:09:28.337 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:28 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:28.097+0000 7ff70f3ee200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:09:28.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.136+0000 7f523718f700 1 Processor -- start 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.136+0000 7f523718f700 1 -- start start 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.136+0000 7f523718f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 0x7f5230107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.136+0000 7f523718f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5230107e30 con 0x7f5230107440 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.136+0000 7f5234f2b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 0x7f5230107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.136+0000 7f5234f2b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 0x7f5230107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39952/0 (socket says 172.21.15.50:39952) 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.136+0000 7f5234f2b700 1 -- 172.21.15.50:0/3178497203 learned_addr learned my addr 172.21.15.50:0/3178497203 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.137+0000 7f5234f2b700 1 -- 172.21.15.50:0/3178497203 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5230108640 con 0x7f5230107440 2024-04-23T18:09:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.137+0000 7f5234f2b700 1 --2- 172.21.15.50:0/3178497203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 0x7f5230107860 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f52240098a0 tx=0x7f5224009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1693fec045b902 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:28.942 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.137+0000 7f522f7fe700 1 -- 172.21.15.50:0/3178497203 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5224004160 con 0x7f5230107440 2024-04-23T18:09:28.942 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.137+0000 7f522f7fe700 1 -- 172.21.15.50:0/3178497203 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5224004300 con 0x7f5230107440 2024-04-23T18:09:28.942 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.137+0000 7f522f7fe700 1 -- 172.21.15.50:0/3178497203 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5224011a20 con 0x7f5230107440 2024-04-23T18:09:28.942 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.138+0000 7f523718f700 1 -- 172.21.15.50:0/3178497203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 msgr2=0x7f5230107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:28.942 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.138+0000 7f523718f700 1 --2- 172.21.15.50:0/3178497203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 0x7f5230107860 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f52240098a0 tx=0x7f5224009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:28.942 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.138+0000 7f523718f700 1 -- 172.21.15.50:0/3178497203 shutdown_connections 2024-04-23T18:09:28.942 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.138+0000 7f523718f700 1 --2- 172.21.15.50:0/3178497203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 0x7f5230107860 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:28.942 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.138+0000 7f523718f700 1 -- 172.21.15.50:0/3178497203 >> 172.21.15.50:0/3178497203 conn(0x7f5230102a40 msgr2=0x7f5230104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:28.942 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.138+0000 7f523718f700 1 -- 172.21.15.50:0/3178497203 shutdown_connections 2024-04-23T18:09:28.942 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.138+0000 7f523718f700 1 -- 172.21.15.50:0/3178497203 wait complete. 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.139+0000 7f523718f700 1 Processor -- start 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.139+0000 7f523718f700 1 -- start start 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.139+0000 7f523718f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 0x7f523019b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.139+0000 7f523718f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f523010b9f0 con 0x7f5230107440 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.139+0000 7f5234f2b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 0x7f523019b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.139+0000 7f5234f2b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 0x7f523019b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39958/0 (socket says 172.21.15.50:39958) 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.139+0000 7f5234f2b700 1 -- 172.21.15.50:0/4036388099 learned_addr learned my addr 172.21.15.50:0/4036388099 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.139+0000 7f5234f2b700 1 -- 172.21.15.50:0/4036388099 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523019b870 con 0x7f5230107440 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.140+0000 7f5234f2b700 1 --2- 172.21.15.50:0/4036388099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 0x7f523019b330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5224009520 tx=0x7f5224011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.140+0000 7f522dffb700 1 -- 172.21.15.50:0/4036388099 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52240039e0 con 0x7f5230107440 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.140+0000 7f522dffb700 1 -- 172.21.15.50:0/4036388099 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f522401b8f0 con 0x7f5230107440 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.140+0000 7f523718f700 1 -- 172.21.15.50:0/4036388099 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f523019bb00 con 0x7f5230107440 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.140+0000 7f522dffb700 1 -- 172.21.15.50:0/4036388099 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5224019770 con 0x7f5230107440 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.140+0000 7f523718f700 1 -- 172.21.15.50:0/4036388099 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f523019e7f0 con 0x7f5230107440 2024-04-23T18:09:28.943 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.141+0000 7f522dffb700 1 -- 172.21.15.50:0/4036388099 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f5224019a70 con 0x7f5230107440 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.141+0000 7f522dffb700 1 --2- 172.21.15.50:0/4036388099 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f5220039600 0x7f522003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.141+0000 7f522dffb700 1 -- 172.21.15.50:0/4036388099 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f522404e130 con 0x7f5230107440 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.141+0000 7f522ffff700 1 -- 172.21.15.50:0/4036388099 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f5220039600 msgr2=0x7f522003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/103327972 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.141+0000 7f522ffff700 1 --2- 172.21.15.50:0/4036388099 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f5220039600 0x7f522003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.141+0000 7f523718f700 1 -- 172.21.15.50:0/4036388099 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52300640a0 con 0x7f5230107440 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.146+0000 7f522dffb700 1 -- 172.21.15.50:0/4036388099 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f522401f030 con 0x7f5230107440 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.342+0000 7f522ffff700 1 -- 172.21.15.50:0/4036388099 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f5220039600 msgr2=0x7f522003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/103327972 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.342+0000 7f522ffff700 1 --2- 172.21.15.50:0/4036388099 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f5220039600 0x7f522003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.416+0000 7f523718f700 1 -- 172.21.15.50:0/4036388099 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f523019e9c0 con 0x7f5230107440 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.416+0000 7f522dffb700 1 -- 172.21.15.50:0/4036388099 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f52240172b0 con 0x7f5230107440 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.418+0000 7f523718f700 1 -- 172.21.15.50:0/4036388099 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f5220039600 msgr2=0x7f522003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.418+0000 7f523718f700 1 --2- 172.21.15.50:0/4036388099 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f5220039600 0x7f522003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.418+0000 7f523718f700 1 -- 172.21.15.50:0/4036388099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 msgr2=0x7f523019b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.418+0000 7f523718f700 1 --2- 172.21.15.50:0/4036388099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 0x7f523019b330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5224009520 tx=0x7f5224011aa0 comp rx=0 tx=0).stop 2024-04-23T18:09:28.944 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.418+0000 7f523718f700 1 -- 172.21.15.50:0/4036388099 shutdown_connections 2024-04-23T18:09:28.945 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.418+0000 7f523718f700 1 --2- 172.21.15.50:0/4036388099 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f5220039600 0x7f522003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:28.945 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.418+0000 7f523718f700 1 --2- 172.21.15.50:0/4036388099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5230107440 0x7f523019b330 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:28.945 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.418+0000 7f523718f700 1 -- 172.21.15.50:0/4036388099 >> 172.21.15.50:0/4036388099 conn(0x7f5230102a40 msgr2=0x7f5230103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:28.945 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.419+0000 7f523718f700 1 -- 172.21.15.50:0/4036388099 shutdown_connections 2024-04-23T18:09:28.945 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:28.419+0000 7f523718f700 1 -- 172.21.15.50:0/4036388099 wait complete. 2024-04-23T18:09:28.945 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for the mgr to restart... 2024-04-23T18:09:28.945 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mgr epoch 9... 2024-04-23T18:09:29.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:29 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4036388099' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T18:09:29.695 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:29 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:29.378+0000 7ff70f3ee200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:09:29.695 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:29 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:29.556+0000 7ff70f3ee200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:09:30.030 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:29 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:29.734+0000 7ff70f3ee200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:09:30.030 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:30.035+0000 7ff70f3ee200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:09:30.337 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:30.132+0000 7ff70f3ee200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:09:30.338 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:30.343+0000 7ff70f3ee200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:09:30.695 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:30.547+0000 7ff70f3ee200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:09:31.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:31 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:31.174+0000 7ff70f3ee200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:09:31.930 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:31 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:31.620+0000 7ff70f3ee200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:09:31.930 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:31 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:31.720+0000 7ff70f3ee200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:09:31.930 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:31 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:31.935+0000 7ff70f3ee200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:09:32.695 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:32 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:32.505+0000 7ff70f3ee200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:09:32.695 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:32 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:32.602+0000 7ff70f3ee200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:09:33.195 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:32 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:32.859+0000 7ff70f3ee200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:09:33.195 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:33 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:33.001+0000 7ff70f3ee200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:09:33.195 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:33 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:33.103+0000 7ff70f3ee200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:09:33.535 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:33 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:33.227+0000 7ff70f3ee200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:09:33.535 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:33 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:33.540+0000 7ff70f3ee200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:09:34.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:34.171+0000 7ff70f3ee200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:09:34.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:09:34.400+0000 7ff70f3ee200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:09:34.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-mon[29187]: Active manager daemon x restarted 2024-04-23T18:09:34.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-mon[29187]: Activating manager daemon x 2024-04-23T18:09:34.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-mon[29187]: osdmap e3: 0 total, 0 up, 0 in 2024-04-23T18:09:34.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-mon[29187]: mgrmap e10: x(active, starting, since 0.00344641s) 2024-04-23T18:09:34.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:09:34.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:09:34.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:09:34.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:09:34.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:09:34.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-mon[29187]: Manager daemon x is now available 2024-04-23T18:09:34.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:35.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:09:35.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:09:35.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:35 smithi050 ceph-mon[29187]: [23/Apr/2024:18:09:35] ENGINE Bus STARTING 2024-04-23T18:09:35.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:35 smithi050 ceph-mon[29187]: [23/Apr/2024:18:09:35] ENGINE Serving on http://172.21.15.50:8765 2024-04-23T18:09:35.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:35 smithi050 ceph-mon[29187]: mgrmap e11: x(active, since 1.00615s) 2024-04-23T18:09:36.067 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-23T18:09:36.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-23T18:09:36.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-23T18:09:36.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-23T18:09:36.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.360+0000 7f9417137700 1 Processor -- start 2024-04-23T18:09:36.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.360+0000 7f9417137700 1 -- start start 2024-04-23T18:09:36.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.360+0000 7f9417137700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 0x7f941010b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:36.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.360+0000 7f9417137700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f941007be60 con 0x7f9410109020 2024-04-23T18:09:36.068 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.361+0000 7f9414ed3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 0x7f941010b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:36.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.361+0000 7f9414ed3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 0x7f941010b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39974/0 (socket says 172.21.15.50:39974) 2024-04-23T18:09:36.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.361+0000 7f9414ed3700 1 -- 172.21.15.50:0/1790916989 learned_addr learned my addr 172.21.15.50:0/1790916989 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:36.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.361+0000 7f9414ed3700 1 -- 172.21.15.50:0/1790916989 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941010b950 con 0x7f9410109020 2024-04-23T18:09:36.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.361+0000 7f9414ed3700 1 --2- 172.21.15.50:0/1790916989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 0x7f941010b410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f94040098a0 tx=0x7f9404009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62740f01593b8685 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:36.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.361+0000 7f940f7fe700 1 -- 172.21.15.50:0/1790916989 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9404004160 con 0x7f9410109020 2024-04-23T18:09:36.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.361+0000 7f940f7fe700 1 -- 172.21.15.50:0/1790916989 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9404004300 con 0x7f9410109020 2024-04-23T18:09:36.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.361+0000 7f940f7fe700 1 -- 172.21.15.50:0/1790916989 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9404011a20 con 0x7f9410109020 2024-04-23T18:09:36.069 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.362+0000 7f9417137700 1 -- 172.21.15.50:0/1790916989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 msgr2=0x7f941010b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:36.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.362+0000 7f9417137700 1 --2- 172.21.15.50:0/1790916989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 0x7f941010b410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f94040098a0 tx=0x7f9404009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:36.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.362+0000 7f9417137700 1 -- 172.21.15.50:0/1790916989 shutdown_connections 2024-04-23T18:09:36.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.362+0000 7f9417137700 1 --2- 172.21.15.50:0/1790916989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 0x7f941010b410 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:36.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.362+0000 7f9417137700 1 -- 172.21.15.50:0/1790916989 >> 172.21.15.50:0/1790916989 conn(0x7f9410102a50 msgr2=0x7f9410104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:36.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.362+0000 7f9417137700 1 -- 172.21.15.50:0/1790916989 shutdown_connections 2024-04-23T18:09:36.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.362+0000 7f9417137700 1 -- 172.21.15.50:0/1790916989 wait complete. 2024-04-23T18:09:36.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.363+0000 7f9417137700 1 Processor -- start 2024-04-23T18:09:36.070 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.363+0000 7f9417137700 1 -- start start 2024-04-23T18:09:36.071 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.363+0000 7f9417137700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 0x7f941019c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:36.071 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.363+0000 7f9417137700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f941010c900 con 0x7f9410109020 2024-04-23T18:09:36.071 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.363+0000 7f9414ed3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 0x7f941019c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:36.071 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.363+0000 7f9414ed3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 0x7f941019c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39986/0 (socket says 172.21.15.50:39986) 2024-04-23T18:09:36.071 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.363+0000 7f9414ed3700 1 -- 172.21.15.50:0/2070758024 learned_addr learned my addr 172.21.15.50:0/2070758024 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:36.071 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.363+0000 7f9414ed3700 1 -- 172.21.15.50:0/2070758024 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941019c990 con 0x7f9410109020 2024-04-23T18:09:36.071 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.363+0000 7f9414ed3700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 0x7f941019c450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f94040116e0 tx=0x7f9404011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:36.071 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.364+0000 7f940dffb700 1 -- 172.21.15.50:0/2070758024 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9404003b20 con 0x7f9410109020 2024-04-23T18:09:36.072 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.364+0000 7f940dffb700 1 -- 172.21.15.50:0/2070758024 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f940401b8f0 con 0x7f9410109020 2024-04-23T18:09:36.072 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.364+0000 7f9417137700 1 -- 172.21.15.50:0/2070758024 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f941019cc20 con 0x7f9410109020 2024-04-23T18:09:36.072 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.364+0000 7f940dffb700 1 -- 172.21.15.50:0/2070758024 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94040238c0 con 0x7f9410109020 2024-04-23T18:09:36.072 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.364+0000 7f9417137700 1 -- 172.21.15.50:0/2070758024 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f941019f910 con 0x7f9410109020 2024-04-23T18:09:36.072 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.365+0000 7f940dffb700 1 -- 172.21.15.50:0/2070758024 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f9404023b70 con 0x7f9410109020 2024-04-23T18:09:36.072 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.365+0000 7f940dffb700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 0x7f940003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:36.072 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.365+0000 7f940dffb700 1 -- 172.21.15.50:0/2070758024 --> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f940003c180 con 0x7f94000395b0 2024-04-23T18:09:36.072 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.365+0000 7f940dffb700 1 -- 172.21.15.50:0/2070758024 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f940404e070 con 0x7f9410109020 2024-04-23T18:09:36.072 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.365+0000 7f940ffff700 1 -- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 msgr2=0x7f940003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/103327972 2024-04-23T18:09:36.073 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.365+0000 7f940ffff700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 0x7f940003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:09:36.073 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.566+0000 7f940ffff700 1 -- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 msgr2=0x7f940003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/103327972 2024-04-23T18:09:36.073 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.566+0000 7f940ffff700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 0x7f940003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:09:36.073 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.967+0000 7f940ffff700 1 -- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 msgr2=0x7f940003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/103327972 2024-04-23T18:09:36.073 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:29.967+0000 7f940ffff700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 0x7f940003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T18:09:36.073 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:30.768+0000 7f940ffff700 1 -- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 msgr2=0x7f940003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/103327972 2024-04-23T18:09:36.073 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:30.768+0000 7f940ffff700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 0x7f940003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T18:09:36.073 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:32.369+0000 7f940ffff700 1 -- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 msgr2=0x7f940003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/103327972 2024-04-23T18:09:36.073 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:32.369+0000 7f940ffff700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 0x7f940003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-23T18:09:36.074 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:34.404+0000 7f940dffb700 1 -- 172.21.15.50:0/2070758024 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f940401baa0 con 0x7f9410109020 2024-04-23T18:09:36.074 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:34.404+0000 7f940dffb700 1 -- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 msgr2=0x7f940003ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:09:36.074 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:34.404+0000 7f940dffb700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 0x7f940003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:36.074 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.406+0000 7f940dffb700 1 -- 172.21.15.50:0/2070758024 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f9404050490 con 0x7f9410109020 2024-04-23T18:09:36.074 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.406+0000 7f940dffb700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f940003cfe0 0x7f940003f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:36.074 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.407+0000 7f940dffb700 1 -- 172.21.15.50:0/2070758024 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f940402de50 con 0x7f940003cfe0 2024-04-23T18:09:36.074 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.407+0000 7f940ffff700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f940003cfe0 0x7f940003f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:36.075 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.407+0000 7f940ffff700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f940003cfe0 0x7f940003f3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f93fc003a10 tx=0x7f93fc0092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:36.075 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.408+0000 7f940dffb700 1 -- 172.21.15.50:0/2070758024 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f940402de50 con 0x7f940003cfe0 2024-04-23T18:09:36.075 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.414+0000 7f9417137700 1 -- 172.21.15.50:0/2070758024 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9410069ba0 con 0x7f940003cfe0 2024-04-23T18:09:36.075 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.415+0000 7f940dffb700 1 -- 172.21.15.50:0/2070758024 <== mgr.14152 v2:172.21.15.50:6800/1004018784 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f9410069ba0 con 0x7f940003cfe0 2024-04-23T18:09:36.075 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.415+0000 7f9417137700 1 -- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f940003cfe0 msgr2=0x7f940003f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:36.075 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.415+0000 7f9417137700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f940003cfe0 0x7f940003f3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f93fc003a10 tx=0x7f93fc0092b0 comp rx=0 tx=0).stop 2024-04-23T18:09:36.075 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.415+0000 7f9417137700 1 -- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 msgr2=0x7f941019c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:36.075 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.415+0000 7f9417137700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 0x7f941019c450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f94040116e0 tx=0x7f9404011aa0 comp rx=0 tx=0).stop 2024-04-23T18:09:36.075 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.415+0000 7f9417137700 1 -- 172.21.15.50:0/2070758024 shutdown_connections 2024-04-23T18:09:36.076 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.415+0000 7f9417137700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f940003cfe0 0x7f940003f3d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:36.076 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.415+0000 7f9417137700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9410109020 0x7f941019c450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:36.076 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.415+0000 7f9417137700 1 --2- 172.21.15.50:0/2070758024 >> [v2:172.21.15.50:6800/103327972,v1:172.21.15.50:6801/103327972] conn(0x7f94000395b0 0x7f940003ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:36.076 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.415+0000 7f9417137700 1 -- 172.21.15.50:0/2070758024 >> 172.21.15.50:0/2070758024 conn(0x7f9410102a50 msgr2=0x7f9410103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:36.076 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.415+0000 7f9417137700 1 -- 172.21.15.50:0/2070758024 shutdown_connections 2024-04-23T18:09:36.076 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:35.415+0000 7f9417137700 1 -- 172.21.15.50:0/2070758024 wait complete. 2024-04-23T18:09:36.076 INFO:teuthology.orchestra.run.smithi050.stdout:mgr epoch 9 is available 2024-04-23T18:09:36.076 INFO:teuthology.orchestra.run.smithi050.stdout:Generating a dashboard self-signed certificate... 2024-04-23T18:09:36.618 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:36 smithi050 ceph-mon[29187]: [23/Apr/2024:18:09:35] ENGINE Serving on https://172.21.15.50:7150 2024-04-23T18:09:36.618 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:36 smithi050 ceph-mon[29187]: [23/Apr/2024:18:09:35] ENGINE Bus STARTED 2024-04-23T18:09:36.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:36 smithi050 ceph-mon[29187]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-23T18:09:36.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:36 smithi050 ceph-mon[29187]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-23T18:09:37.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-23T18:09:37.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.485+0000 7f44011ef700 1 Processor -- start 2024-04-23T18:09:37.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.486+0000 7f44011ef700 1 -- start start 2024-04-23T18:09:37.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.486+0000 7f44011ef700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 0x7f43fc108980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:37.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.486+0000 7f44011ef700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc108f50 con 0x7f43fc108560 2024-04-23T18:09:37.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.486+0000 7f43fad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 0x7f43fc108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:37.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.486+0000 7f43fad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 0x7f43fc108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34294/0 (socket says 172.21.15.50:34294) 2024-04-23T18:09:37.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.486+0000 7f43fad9d700 1 -- 172.21.15.50:0/486431627 learned_addr learned my addr 172.21.15.50:0/486431627 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:37.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.487+0000 7f43fad9d700 1 -- 172.21.15.50:0/486431627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc109710 con 0x7f43fc108560 2024-04-23T18:09:37.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.487+0000 7f43fad9d700 1 --2- 172.21.15.50:0/486431627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 0x7f43fc108980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0098a0 tx=0x7f43ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f25e26243ce365a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:37.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.487+0000 7f43fa59c700 1 -- 172.21.15.50:0/486431627 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43ec004160 con 0x7f43fc108560 2024-04-23T18:09:37.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.487+0000 7f43fa59c700 1 -- 172.21.15.50:0/486431627 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43ec004300 con 0x7f43fc108560 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.487+0000 7f43fa59c700 1 -- 172.21.15.50:0/486431627 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43ec011a20 con 0x7f43fc108560 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.488+0000 7f44011ef700 1 -- 172.21.15.50:0/486431627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 msgr2=0x7f43fc108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.488+0000 7f44011ef700 1 --2- 172.21.15.50:0/486431627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 0x7f43fc108980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0098a0 tx=0x7f43ec009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.488+0000 7f44011ef700 1 -- 172.21.15.50:0/486431627 shutdown_connections 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.488+0000 7f44011ef700 1 --2- 172.21.15.50:0/486431627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 0x7f43fc108980 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.488+0000 7f44011ef700 1 -- 172.21.15.50:0/486431627 >> 172.21.15.50:0/486431627 conn(0x7f43fc103b40 msgr2=0x7f43fc105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.488+0000 7f44011ef700 1 -- 172.21.15.50:0/486431627 shutdown_connections 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.488+0000 7f44011ef700 1 -- 172.21.15.50:0/486431627 wait complete. 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.489+0000 7f44011ef700 1 Processor -- start 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.489+0000 7f44011ef700 1 -- start start 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.489+0000 7f44011ef700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 0x7f43fc19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.489+0000 7f44011ef700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc10cac0 con 0x7f43fc108560 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.489+0000 7f43fad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 0x7f43fc19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:37.458 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.489+0000 7f43fad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 0x7f43fc19c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34304/0 (socket says 172.21.15.50:34304) 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.489+0000 7f43fad9d700 1 -- 172.21.15.50:0/3054409031 learned_addr learned my addr 172.21.15.50:0/3054409031 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.489+0000 7f43fad9d700 1 -- 172.21.15.50:0/3054409031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc19c9c0 con 0x7f43fc108560 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.490+0000 7f43fad9d700 1 --2- 172.21.15.50:0/3054409031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 0x7f43fc19c480 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0119f0 tx=0x7f43ec011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.490+0000 7f43f8d99700 1 -- 172.21.15.50:0/3054409031 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43ec0039e0 con 0x7f43fc108560 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.490+0000 7f44011ef700 1 -- 172.21.15.50:0/3054409031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43fc19cc50 con 0x7f43fc108560 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.490+0000 7f43f8d99700 1 -- 172.21.15.50:0/3054409031 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43ec01b8f0 con 0x7f43fc108560 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.490+0000 7f43f8d99700 1 -- 172.21.15.50:0/3054409031 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43ec0237a0 con 0x7f43fc108560 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.490+0000 7f44011ef700 1 -- 172.21.15.50:0/3054409031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43fc19f940 con 0x7f43fc108560 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.491+0000 7f43f8d99700 1 -- 172.21.15.50:0/3054409031 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f43ec01b440 con 0x7f43fc108560 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.491+0000 7f44011ef700 1 -- 172.21.15.50:0/3054409031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43fc0640a0 con 0x7f43fc108560 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.491+0000 7f43f8d99700 1 --2- 172.21.15.50:0/3054409031 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43e80395b0 0x7f43e803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.491+0000 7f43f8d99700 1 -- 172.21.15.50:0/3054409031 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f43ec04e0f0 con 0x7f43fc108560 2024-04-23T18:09:37.459 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.491+0000 7f43f259c700 1 --2- 172.21.15.50:0/3054409031 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43e80395b0 0x7f43e803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.492+0000 7f43f259c700 1 --2- 172.21.15.50:0/3054409031 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43e80395b0 0x7f43e803ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f43e4006fd0 tx=0x7f43e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.496+0000 7f43f8d99700 1 -- 172.21.15.50:0/3054409031 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f43ec01f030 con 0x7f43fc108560 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.711+0000 7f44011ef700 1 -- 172.21.15.50:0/3054409031 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f43fc104dc0 con 0x7f43e80395b0 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.870+0000 7f43f8d99700 1 -- 172.21.15.50:0/3054409031 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f43fc104dc0 con 0x7f43e80395b0 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.873+0000 7f44011ef700 1 -- 172.21.15.50:0/3054409031 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43e80395b0 msgr2=0x7f43e803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.873+0000 7f44011ef700 1 --2- 172.21.15.50:0/3054409031 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43e80395b0 0x7f43e803ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f43e4006fd0 tx=0x7f43e4006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.873+0000 7f44011ef700 1 -- 172.21.15.50:0/3054409031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 msgr2=0x7f43fc19c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.873+0000 7f44011ef700 1 --2- 172.21.15.50:0/3054409031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 0x7f43fc19c480 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0119f0 tx=0x7f43ec011ad0 comp rx=0 tx=0).stop 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.873+0000 7f44011ef700 1 -- 172.21.15.50:0/3054409031 shutdown_connections 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.873+0000 7f44011ef700 1 --2- 172.21.15.50:0/3054409031 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43e80395b0 0x7f43e803ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.873+0000 7f44011ef700 1 --2- 172.21.15.50:0/3054409031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43fc108560 0x7f43fc19c480 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.873+0000 7f44011ef700 1 -- 172.21.15.50:0/3054409031 >> 172.21.15.50:0/3054409031 conn(0x7f43fc103b40 msgr2=0x7f43fc1046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:37.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.873+0000 7f44011ef700 1 -- 172.21.15.50:0/3054409031 shutdown_connections 2024-04-23T18:09:37.461 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:36.873+0000 7f44011ef700 1 -- 172.21.15.50:0/3054409031 wait complete. 2024-04-23T18:09:37.461 INFO:teuthology.orchestra.run.smithi050.stdout:Creating initial admin user... 2024-04-23T18:09:37.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:37 smithi050 ceph-mon[29187]: mgrmap e12: x(active, since 2s) 2024-04-23T18:09:37.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:37 smithi050 ceph-mon[29187]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:37.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:37.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:37.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:38.872 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$YfbmB/tTO9FMow/3vbhy2ugEUwUTsJbm5CgiwMPavIhHrHBuYhINa", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713895778, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-23T18:09:38.872 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.914+0000 7f461b526700 1 Processor -- start 2024-04-23T18:09:38.872 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.914+0000 7f461b526700 1 -- start start 2024-04-23T18:09:38.872 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.914+0000 7f461b526700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 0x7f4614107250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:38.872 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.914+0000 7f461b526700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f461407be40 con 0x7f4614106e30 2024-04-23T18:09:38.872 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.915+0000 7f46192c2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 0x7f4614107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:38.872 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.915+0000 7f46192c2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 0x7f4614107250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34318/0 (socket says 172.21.15.50:34318) 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.915+0000 7f46192c2700 1 -- 172.21.15.50:0/2152668407 learned_addr learned my addr 172.21.15.50:0/2152668407 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.915+0000 7f46192c2700 1 -- 172.21.15.50:0/2152668407 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4614107790 con 0x7f4614106e30 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.915+0000 7f46192c2700 1 --2- 172.21.15.50:0/2152668407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 0x7f4614107250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f46100098a0 tx=0x7f4610009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2589468218f8650 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.915+0000 7f4607fff700 1 -- 172.21.15.50:0/2152668407 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4610004160 con 0x7f4614106e30 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.915+0000 7f4607fff700 1 -- 172.21.15.50:0/2152668407 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4610004300 con 0x7f4614106e30 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.915+0000 7f4607fff700 1 -- 172.21.15.50:0/2152668407 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4610011a20 con 0x7f4614106e30 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.916+0000 7f461b526700 1 -- 172.21.15.50:0/2152668407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 msgr2=0x7f4614107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.916+0000 7f461b526700 1 --2- 172.21.15.50:0/2152668407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 0x7f4614107250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f46100098a0 tx=0x7f4610009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.916+0000 7f461b526700 1 -- 172.21.15.50:0/2152668407 shutdown_connections 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.916+0000 7f461b526700 1 --2- 172.21.15.50:0/2152668407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 0x7f4614107250 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.916+0000 7f461b526700 1 -- 172.21.15.50:0/2152668407 >> 172.21.15.50:0/2152668407 conn(0x7f4614102a50 msgr2=0x7f4614104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.916+0000 7f461b526700 1 -- 172.21.15.50:0/2152668407 shutdown_connections 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.916+0000 7f461b526700 1 -- 172.21.15.50:0/2152668407 wait complete. 2024-04-23T18:09:38.873 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.917+0000 7f461b526700 1 Processor -- start 2024-04-23T18:09:38.874 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.917+0000 7f461b526700 1 -- start start 2024-04-23T18:09:38.874 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.917+0000 7f461b526700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 0x7f461419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:38.874 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.917+0000 7f461b526700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4614108800 con 0x7f4614106e30 2024-04-23T18:09:38.874 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.917+0000 7f46192c2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 0x7f461419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:38.874 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.917+0000 7f46192c2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 0x7f461419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34322/0 (socket says 172.21.15.50:34322) 2024-04-23T18:09:38.874 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.917+0000 7f46192c2700 1 -- 172.21.15.50:0/4153604354 learned_addr learned my addr 172.21.15.50:0/4153604354 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:38.874 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.918+0000 7f46192c2700 1 -- 172.21.15.50:0/4153604354 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461419eb40 con 0x7f4614106e30 2024-04-23T18:09:38.874 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.918+0000 7f46192c2700 1 --2- 172.21.15.50:0/4153604354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 0x7f461419e600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f46100116e0 tx=0x7f4610011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:38.874 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.918+0000 7f46067fc700 1 -- 172.21.15.50:0/4153604354 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4610003b20 con 0x7f4614106e30 2024-04-23T18:09:38.875 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.918+0000 7f46067fc700 1 -- 172.21.15.50:0/4153604354 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f461001b8f0 con 0x7f4614106e30 2024-04-23T18:09:38.875 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.918+0000 7f461b526700 1 -- 172.21.15.50:0/4153604354 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461419edd0 con 0x7f4614106e30 2024-04-23T18:09:38.875 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.918+0000 7f46067fc700 1 -- 172.21.15.50:0/4153604354 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46100238c0 con 0x7f4614106e30 2024-04-23T18:09:38.875 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.918+0000 7f461b526700 1 -- 172.21.15.50:0/4153604354 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46141a1ac0 con 0x7f4614106e30 2024-04-23T18:09:38.875 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.919+0000 7f46067fc700 1 -- 172.21.15.50:0/4153604354 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f4610023a60 con 0x7f4614106e30 2024-04-23T18:09:38.875 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.919+0000 7f46067fc700 1 --2- 172.21.15.50:0/4153604354 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4600039610 0x7f460003bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.919+0000 7f461b526700 1 -- 172.21.15.50:0/4153604354 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46140640a0 con 0x7f4614106e30 2024-04-23T18:09:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.919+0000 7f46067fc700 1 -- 172.21.15.50:0/4153604354 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f461004e150 con 0x7f4614106e30 2024-04-23T18:09:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.919+0000 7f4618ac1700 1 --2- 172.21.15.50:0/4153604354 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4600039610 0x7f460003bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.920+0000 7f4618ac1700 1 --2- 172.21.15.50:0/4153604354 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4600039610 0x7f460003bad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4608006fd0 tx=0x7f4608006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:37.925+0000 7f46067fc700 1 -- 172.21.15.50:0/4153604354 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f461001f030 con 0x7f4614106e30 2024-04-23T18:09:38.877 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:38.150+0000 7f461b526700 1 -- 172.21.15.50:0/4153604354 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f4614198160 con 0x7f4600039610 2024-04-23T18:09:38.877 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:38.396+0000 7f46067fc700 1 -- 172.21.15.50:0/4153604354 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f4614198160 con 0x7f4600039610 2024-04-23T18:09:38.877 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:38.398+0000 7f461b526700 1 -- 172.21.15.50:0/4153604354 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4600039610 msgr2=0x7f460003bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:38.877 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:38.398+0000 7f461b526700 1 --2- 172.21.15.50:0/4153604354 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4600039610 0x7f460003bad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4608006fd0 tx=0x7f4608006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:38.877 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:38.398+0000 7f461b526700 1 -- 172.21.15.50:0/4153604354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 msgr2=0x7f461419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:38.877 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:38.398+0000 7f461b526700 1 --2- 172.21.15.50:0/4153604354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 0x7f461419e600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f46100116e0 tx=0x7f4610011aa0 comp rx=0 tx=0).stop 2024-04-23T18:09:38.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:38.398+0000 7f461b526700 1 -- 172.21.15.50:0/4153604354 shutdown_connections 2024-04-23T18:09:38.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:38.398+0000 7f461b526700 1 --2- 172.21.15.50:0/4153604354 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4600039610 0x7f460003bad0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:38.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:38.398+0000 7f461b526700 1 --2- 172.21.15.50:0/4153604354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4614106e30 0x7f461419e600 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:38.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:38.398+0000 7f461b526700 1 -- 172.21.15.50:0/4153604354 >> 172.21.15.50:0/4153604354 conn(0x7f4614102a50 msgr2=0x7f46141952a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:38.878 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:38.399+0000 7f461b526700 1 -- 172.21.15.50:0/4153604354 shutdown_connections 2024-04-23T18:09:38.879 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:38.399+0000 7f461b526700 1 -- 172.21.15.50:0/4153604354 wait complete. 2024-04-23T18:09:38.879 INFO:teuthology.orchestra.run.smithi050.stdout:Fetching dashboard port number... 2024-04-23T18:09:39.254 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:39 smithi050 ceph-mon[29187]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:39.254 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:39.254 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:40.189 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 8443 2024-04-23T18:09:40.189 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.259+0000 7f6c1a0e7700 1 Processor -- start 2024-04-23T18:09:40.189 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.259+0000 7f6c1a0e7700 1 -- start start 2024-04-23T18:09:40.189 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.259+0000 7f6c1a0e7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 0x7f6c14107210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:40.189 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.259+0000 7f6c1a0e7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1407be80 con 0x7f6c14106df0 2024-04-23T18:09:40.189 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.259+0000 7f6c137fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 0x7f6c14107210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:40.189 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.259+0000 7f6c137fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 0x7f6c14107210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34336/0 (socket says 172.21.15.50:34336) 2024-04-23T18:09:40.190 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.259+0000 7f6c137fe700 1 -- 172.21.15.50:0/2708352370 learned_addr learned my addr 172.21.15.50:0/2708352370 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:40.190 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.259+0000 7f6c137fe700 1 -- 172.21.15.50:0/2708352370 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c14107750 con 0x7f6c14106df0 2024-04-23T18:09:40.190 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.260+0000 7f6c137fe700 1 --2- 172.21.15.50:0/2708352370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 0x7f6c14107210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6c040098a0 tx=0x7f6c04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2929cc6cc9b14a62 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:40.190 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.260+0000 7f6c127fc700 1 -- 172.21.15.50:0/2708352370 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c04004160 con 0x7f6c14106df0 2024-04-23T18:09:40.190 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.260+0000 7f6c127fc700 1 -- 172.21.15.50:0/2708352370 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c04004300 con 0x7f6c14106df0 2024-04-23T18:09:40.190 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.260+0000 7f6c127fc700 1 -- 172.21.15.50:0/2708352370 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c04011a20 con 0x7f6c14106df0 2024-04-23T18:09:40.190 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.260+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/2708352370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 msgr2=0x7f6c14107210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:40.190 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.260+0000 7f6c1a0e7700 1 --2- 172.21.15.50:0/2708352370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 0x7f6c14107210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6c040098a0 tx=0x7f6c04009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:40.190 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.260+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/2708352370 shutdown_connections 2024-04-23T18:09:40.190 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.260+0000 7f6c1a0e7700 1 --2- 172.21.15.50:0/2708352370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 0x7f6c14107210 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:40.190 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.260+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/2708352370 >> 172.21.15.50:0/2708352370 conn(0x7f6c14102a50 msgr2=0x7f6c14104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:40.190 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.260+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/2708352370 shutdown_connections 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.261+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/2708352370 wait complete. 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.261+0000 7f6c1a0e7700 1 Processor -- start 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.261+0000 7f6c1a0e7700 1 -- start start 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.261+0000 7f6c1a0e7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 0x7f6c1419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.262+0000 7f6c1a0e7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6c141087c0 con 0x7f6c14106df0 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.262+0000 7f6c137fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 0x7f6c1419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.262+0000 7f6c137fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 0x7f6c1419e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34338/0 (socket says 172.21.15.50:34338) 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.262+0000 7f6c137fe700 1 -- 172.21.15.50:0/1525560214 learned_addr learned my addr 172.21.15.50:0/1525560214 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.262+0000 7f6c137fe700 1 -- 172.21.15.50:0/1525560214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1419eb70 con 0x7f6c14106df0 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.262+0000 7f6c137fe700 1 --2- 172.21.15.50:0/1525560214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 0x7f6c1419e630 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c040116e0 tx=0x7f6c04011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.262+0000 7f6c10ff9700 1 -- 172.21.15.50:0/1525560214 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c04003b20 con 0x7f6c14106df0 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.262+0000 7f6c10ff9700 1 -- 172.21.15.50:0/1525560214 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c0401b8f0 con 0x7f6c14106df0 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.262+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/1525560214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1419ee00 con 0x7f6c14106df0 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.262+0000 7f6c10ff9700 1 -- 172.21.15.50:0/1525560214 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c040238c0 con 0x7f6c14106df0 2024-04-23T18:09:40.191 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.262+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/1525560214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c141a1af0 con 0x7f6c14106df0 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.263+0000 7f6c10ff9700 1 -- 172.21.15.50:0/1525560214 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f6c0401baa0 con 0x7f6c14106df0 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.264+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/1525560214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c140640a0 con 0x7f6c14106df0 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.264+0000 7f6c10ff9700 1 --2- 172.21.15.50:0/1525560214 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6bfc039610 0x7f6bfc03bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.264+0000 7f6c10ff9700 1 -- 172.21.15.50:0/1525560214 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6c0404e1a0 con 0x7f6c14106df0 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.264+0000 7f6c12ffd700 1 --2- 172.21.15.50:0/1525560214 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6bfc039610 0x7f6bfc03bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.264+0000 7f6c12ffd700 1 --2- 172.21.15.50:0/1525560214 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6bfc039610 0x7f6bfc03bad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6c08006fd0 tx=0x7f6c08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.269+0000 7f6c10ff9700 1 -- 172.21.15.50:0/1525560214 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c0401f030 con 0x7f6c14106df0 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.497+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/1525560214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f6c1402e1d0 con 0x7f6c14106df0 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.497+0000 7f6c10ff9700 1 -- 172.21.15.50:0/1525560214 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f6c0401a3d0 con 0x7f6c14106df0 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.500+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/1525560214 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6bfc039610 msgr2=0x7f6bfc03bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.500+0000 7f6c1a0e7700 1 --2- 172.21.15.50:0/1525560214 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6bfc039610 0x7f6bfc03bad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6c08006fd0 tx=0x7f6c08006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.500+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/1525560214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 msgr2=0x7f6c1419e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:40.192 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.500+0000 7f6c1a0e7700 1 --2- 172.21.15.50:0/1525560214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 0x7f6c1419e630 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c040116e0 tx=0x7f6c04011aa0 comp rx=0 tx=0).stop 2024-04-23T18:09:40.193 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.500+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/1525560214 shutdown_connections 2024-04-23T18:09:40.193 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.500+0000 7f6c1a0e7700 1 --2- 172.21.15.50:0/1525560214 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6bfc039610 0x7f6bfc03bad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:40.193 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.500+0000 7f6c1a0e7700 1 --2- 172.21.15.50:0/1525560214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c14106df0 0x7f6c1419e630 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:40.193 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.500+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/1525560214 >> 172.21.15.50:0/1525560214 conn(0x7f6c14102a50 msgr2=0x7f6c14195250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:40.193 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.501+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/1525560214 shutdown_connections 2024-04-23T18:09:40.193 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:39.501+0000 7f6c1a0e7700 1 -- 172.21.15.50:0/1525560214 wait complete. 2024-04-23T18:09:40.197 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T18:09:40.197 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout disabled 2024-04-23T18:09:40.205 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T18:09:40.205 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-23T18:09:40.205 INFO:teuthology.orchestra.run.smithi050.stdout:firewalld.service is not enabled 2024-04-23T18:09:40.205 INFO:teuthology.orchestra.run.smithi050.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-23T18:09:40.207 INFO:teuthology.orchestra.run.smithi050.stdout:Ceph Dashboard is now available at: 2024-04-23T18:09:40.207 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:09:40.207 INFO:teuthology.orchestra.run.smithi050.stdout: URL: https://smithi050:8443/ 2024-04-23T18:09:40.207 INFO:teuthology.orchestra.run.smithi050.stdout: User: admin 2024-04-23T18:09:40.207 INFO:teuthology.orchestra.run.smithi050.stdout: Password: 6jpjuvvl6i 2024-04-23T18:09:40.207 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:09:40.208 INFO:teuthology.orchestra.run.smithi050.stdout:Saving cluster configuration to /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config directory 2024-04-23T18:09:40.208 INFO:teuthology.orchestra.run.smithi050.stdout:Enabling autotune for osd_memory_target 2024-04-23T18:09:40.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:40 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1525560214' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-23T18:09:41.540 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.659+0000 7fe5f562f700 1 Processor -- start 2024-04-23T18:09:41.540 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.659+0000 7fe5f562f700 1 -- start start 2024-04-23T18:09:41.540 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.659+0000 7fe5f562f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 0x7fe5f010b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:41.540 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.659+0000 7fe5f562f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f0063d00 con 0x7fe5f0109010 2024-04-23T18:09:41.540 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.659+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 0x7fe5f010b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:41.540 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.659+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 0x7fe5f010b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34342/0 (socket says 172.21.15.50:34342) 2024-04-23T18:09:41.540 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.659+0000 7fe5eeffd700 1 -- 172.21.15.50:0/436059035 learned_addr learned my addr 172.21.15.50:0/436059035 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:41.540 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.660+0000 7fe5eeffd700 1 -- 172.21.15.50:0/436059035 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f010b940 con 0x7fe5f0109010 2024-04-23T18:09:41.540 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.660+0000 7fe5eeffd700 1 --2- 172.21.15.50:0/436059035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 0x7fe5f010b400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40098a0 tx=0x7fe5e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43fe8389e4e2ae96 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:41.540 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.660+0000 7fe5edffb700 1 -- 172.21.15.50:0/436059035 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e4004160 con 0x7fe5f0109010 2024-04-23T18:09:41.540 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.660+0000 7fe5edffb700 1 -- 172.21.15.50:0/436059035 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5e4004300 con 0x7fe5f0109010 2024-04-23T18:09:41.540 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.660+0000 7fe5edffb700 1 -- 172.21.15.50:0/436059035 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e4011a20 con 0x7fe5f0109010 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.661+0000 7fe5f562f700 1 -- 172.21.15.50:0/436059035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 msgr2=0x7fe5f010b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.661+0000 7fe5f562f700 1 --2- 172.21.15.50:0/436059035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 0x7fe5f010b400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40098a0 tx=0x7fe5e4009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.661+0000 7fe5f562f700 1 -- 172.21.15.50:0/436059035 shutdown_connections 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.661+0000 7fe5f562f700 1 --2- 172.21.15.50:0/436059035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 0x7fe5f010b400 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.661+0000 7fe5f562f700 1 -- 172.21.15.50:0/436059035 >> 172.21.15.50:0/436059035 conn(0x7fe5f0102a40 msgr2=0x7fe5f0104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.661+0000 7fe5f562f700 1 -- 172.21.15.50:0/436059035 shutdown_connections 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.661+0000 7fe5f562f700 1 -- 172.21.15.50:0/436059035 wait complete. 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.661+0000 7fe5f562f700 1 Processor -- start 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.662+0000 7fe5f562f700 1 -- start start 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.662+0000 7fe5f562f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 0x7fe5f019c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.662+0000 7fe5f562f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f010c8f0 con 0x7fe5f0109010 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.662+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 0x7fe5f019c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:41.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.662+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 0x7fe5f019c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34352/0 (socket says 172.21.15.50:34352) 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.662+0000 7fe5eeffd700 1 -- 172.21.15.50:0/1635803195 learned_addr learned my addr 172.21.15.50:0/1635803195 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.662+0000 7fe5eeffd700 1 -- 172.21.15.50:0/1635803195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f019c930 con 0x7fe5f0109010 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.662+0000 7fe5eeffd700 1 --2- 172.21.15.50:0/1635803195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 0x7fe5f019c3f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4000c00 tx=0x7fe5e4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.663+0000 7fe5d7fff700 1 -- 172.21.15.50:0/1635803195 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e4003aa0 con 0x7fe5f0109010 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.663+0000 7fe5d7fff700 1 -- 172.21.15.50:0/1635803195 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5e401b8f0 con 0x7fe5f0109010 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.663+0000 7fe5f562f700 1 -- 172.21.15.50:0/1635803195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f019cbc0 con 0x7fe5f0109010 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.663+0000 7fe5d7fff700 1 -- 172.21.15.50:0/1635803195 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e4023870 con 0x7fe5f0109010 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.663+0000 7fe5f562f700 1 -- 172.21.15.50:0/1635803195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f019f8b0 con 0x7fe5f0109010 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.664+0000 7fe5d7fff700 1 -- 172.21.15.50:0/1635803195 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fe5e4023b70 con 0x7fe5f0109010 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.664+0000 7fe5f562f700 1 -- 172.21.15.50:0/1635803195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f00640a0 con 0x7fe5f0109010 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.664+0000 7fe5d7fff700 1 --2- 172.21.15.50:0/1635803195 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe5d8039600 0x7fe5d803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.664+0000 7fe5d7fff700 1 -- 172.21.15.50:0/1635803195 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe5e404e240 con 0x7fe5f0109010 2024-04-23T18:09:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.664+0000 7fe5ee7fc700 1 --2- 172.21.15.50:0/1635803195 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe5d8039600 0x7fe5d803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.664+0000 7fe5ee7fc700 1 --2- 172.21.15.50:0/1635803195 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe5d8039600 0x7fe5d803bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0006fd0 tx=0x7fe5e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.669+0000 7fe5d7fff700 1 -- 172.21.15.50:0/1635803195 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe5e401f030 con 0x7fe5f0109010 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.878+0000 7fe5f562f700 1 -- 172.21.15.50:0/1635803195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fe5f019ff10 con 0x7fe5f0109010 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.878+0000 7fe5d7fff700 1 -- 172.21.15.50:0/1635803195 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fe5e4018e30 con 0x7fe5f0109010 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.880+0000 7fe5f562f700 1 -- 172.21.15.50:0/1635803195 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe5d8039600 msgr2=0x7fe5d803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.880+0000 7fe5f562f700 1 --2- 172.21.15.50:0/1635803195 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe5d8039600 0x7fe5d803bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0006fd0 tx=0x7fe5e0006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.880+0000 7fe5f562f700 1 -- 172.21.15.50:0/1635803195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 msgr2=0x7fe5f019c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.880+0000 7fe5f562f700 1 --2- 172.21.15.50:0/1635803195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 0x7fe5f019c3f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4000c00 tx=0x7fe5e4011aa0 comp rx=0 tx=0).stop 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.880+0000 7fe5f562f700 1 -- 172.21.15.50:0/1635803195 shutdown_connections 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.880+0000 7fe5f562f700 1 --2- 172.21.15.50:0/1635803195 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe5d8039600 0x7fe5d803bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.880+0000 7fe5f562f700 1 --2- 172.21.15.50:0/1635803195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f0109010 0x7fe5f019c3f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.880+0000 7fe5f562f700 1 -- 172.21.15.50:0/1635803195 >> 172.21.15.50:0/1635803195 conn(0x7fe5f0102a40 msgr2=0x7fe5f0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.880+0000 7fe5f562f700 1 -- 172.21.15.50:0/1635803195 shutdown_connections 2024-04-23T18:09:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:40.880+0000 7fe5f562f700 1 -- 172.21.15.50:0/1635803195 wait complete. 2024-04-23T18:09:42.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:42 smithi050 ceph-mon[29187]: mgrmap e13: x(active, since 6s) 2024-04-23T18:09:43.198 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.268+0000 7fc2535e6700 1 Processor -- start 2024-04-23T18:09:43.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.268+0000 7fc2535e6700 1 -- start start 2024-04-23T18:09:43.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.268+0000 7fc2535e6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 0x7fc24c10b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:43.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.268+0000 7fc2535e6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c10b8e0 con 0x7fc24c108fb0 2024-04-23T18:09:43.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.268+0000 7fc251382700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 0x7fc24c10b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:43.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.268+0000 7fc251382700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 0x7fc24c10b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60294/0 (socket says 172.21.15.50:60294) 2024-04-23T18:09:43.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.268+0000 7fc251382700 1 -- 172.21.15.50:0/535314090 learned_addr learned my addr 172.21.15.50:0/535314090 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:43.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.269+0000 7fc251382700 1 -- 172.21.15.50:0/535314090 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c10ba60 con 0x7fc24c108fb0 2024-04-23T18:09:43.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.269+0000 7fc251382700 1 --2- 172.21.15.50:0/535314090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 0x7fc24c10b3a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc2400098a0 tx=0x7fc240009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c03dd34b38824e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:43.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.269+0000 7fc23ffff700 1 -- 172.21.15.50:0/535314090 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc240004160 con 0x7fc24c108fb0 2024-04-23T18:09:43.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.269+0000 7fc23ffff700 1 -- 172.21.15.50:0/535314090 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc240004300 con 0x7fc24c108fb0 2024-04-23T18:09:43.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.269+0000 7fc23ffff700 1 -- 172.21.15.50:0/535314090 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc240011a20 con 0x7fc24c108fb0 2024-04-23T18:09:43.199 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.270+0000 7fc2535e6700 1 -- 172.21.15.50:0/535314090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 msgr2=0x7fc24c10b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.270+0000 7fc2535e6700 1 --2- 172.21.15.50:0/535314090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 0x7fc24c10b3a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc2400098a0 tx=0x7fc240009bb0 comp rx=0 tx=0).stop 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.270+0000 7fc2535e6700 1 -- 172.21.15.50:0/535314090 shutdown_connections 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.270+0000 7fc2535e6700 1 --2- 172.21.15.50:0/535314090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 0x7fc24c10b3a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.270+0000 7fc2535e6700 1 -- 172.21.15.50:0/535314090 >> 172.21.15.50:0/535314090 conn(0x7fc24c102a60 msgr2=0x7fc24c104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.270+0000 7fc2535e6700 1 -- 172.21.15.50:0/535314090 shutdown_connections 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.270+0000 7fc2535e6700 1 -- 172.21.15.50:0/535314090 wait complete. 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.270+0000 7fc2535e6700 1 Processor -- start 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.270+0000 7fc2535e6700 1 -- start start 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.271+0000 7fc2535e6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 0x7fc24c108830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.271+0000 7fc2535e6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c10ca00 con 0x7fc24c108fb0 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.271+0000 7fc251382700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 0x7fc24c108830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.271+0000 7fc251382700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 0x7fc24c108830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60300/0 (socket says 172.21.15.50:60300) 2024-04-23T18:09:43.200 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.271+0000 7fc251382700 1 -- 172.21.15.50:0/3494217517 learned_addr learned my addr 172.21.15.50:0/3494217517 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.271+0000 7fc251382700 1 -- 172.21.15.50:0/3494217517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c108d70 con 0x7fc24c108fb0 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.271+0000 7fc251382700 1 --2- 172.21.15.50:0/3494217517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 0x7fc24c108830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc240004660 tx=0x7fc240011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.271+0000 7fc23e7fc700 1 -- 172.21.15.50:0/3494217517 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc240003aa0 con 0x7fc24c108fb0 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.271+0000 7fc23e7fc700 1 -- 172.21.15.50:0/3494217517 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc24001b8f0 con 0x7fc24c108fb0 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.271+0000 7fc2535e6700 1 -- 172.21.15.50:0/3494217517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24c106f60 con 0x7fc24c108fb0 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.271+0000 7fc23e7fc700 1 -- 172.21.15.50:0/3494217517 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc240023900 con 0x7fc24c108fb0 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.271+0000 7fc2535e6700 1 -- 172.21.15.50:0/3494217517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24c1074a0 con 0x7fc24c108fb0 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.272+0000 7fc23e7fc700 1 -- 172.21.15.50:0/3494217517 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc24001baa0 con 0x7fc24c108fb0 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.272+0000 7fc2535e6700 1 -- 172.21.15.50:0/3494217517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc24c0640a0 con 0x7fc24c108fb0 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.273+0000 7fc23e7fc700 1 --2- 172.21.15.50:0/3494217517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc238039700 0x7fc23803bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.273+0000 7fc23e7fc700 1 -- 172.21.15.50:0/3494217517 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc24004e080 con 0x7fc24c108fb0 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.273+0000 7fc250b81700 1 --2- 172.21.15.50:0/3494217517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc238039700 0x7fc23803bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.273+0000 7fc250b81700 1 --2- 172.21.15.50:0/3494217517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc238039700 0x7fc23803bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc248006fd0 tx=0x7fc248006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.278+0000 7fc23e7fc700 1 -- 172.21.15.50:0/3494217517 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc24001f030 con 0x7fc24c108fb0 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.605+0000 7fc2535e6700 1 -- 172.21.15.50:0/3494217517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fc24c107b20 con 0x7fc24c108fb0 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.607+0000 7fc23e7fc700 1 -- 172.21.15.50:0/3494217517 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7fc240018350 con 0x7fc24c108fb0 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.609+0000 7fc2535e6700 1 -- 172.21.15.50:0/3494217517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc238039700 msgr2=0x7fc23803bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.609+0000 7fc2535e6700 1 --2- 172.21.15.50:0/3494217517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc238039700 0x7fc23803bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc248006fd0 tx=0x7fc248006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.610+0000 7fc2535e6700 1 -- 172.21.15.50:0/3494217517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 msgr2=0x7fc24c108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.610+0000 7fc2535e6700 1 --2- 172.21.15.50:0/3494217517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 0x7fc24c108830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc240004660 tx=0x7fc240011aa0 comp rx=0 tx=0).stop 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.610+0000 7fc2535e6700 1 -- 172.21.15.50:0/3494217517 shutdown_connections 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.610+0000 7fc2535e6700 1 --2- 172.21.15.50:0/3494217517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc238039700 0x7fc23803bbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.610+0000 7fc2535e6700 1 --2- 172.21.15.50:0/3494217517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c108fb0 0x7fc24c108830 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.610+0000 7fc2535e6700 1 -- 172.21.15.50:0/3494217517 >> 172.21.15.50:0/3494217517 conn(0x7fc24c102a60 msgr2=0x7fc24c103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.610+0000 7fc2535e6700 1 -- 172.21.15.50:0/3494217517 shutdown_connections 2024-04-23T18:09:43.202 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-23T18:09:42.610+0000 7fc2535e6700 1 -- 172.21.15.50:0/3494217517 wait complete. 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout:Or, if you are only running a single cluster on this host: 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout: ceph telemetry on 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout:For more information see: 2024-04-23T18:09:43.203 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:09:43.204 INFO:teuthology.orchestra.run.smithi050.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-23T18:09:43.204 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:09:43.204 INFO:teuthology.orchestra.run.smithi050.stdout:Bootstrap complete. 2024-04-23T18:09:43.240 INFO:tasks.cephadm:Fetching config... 2024-04-23T18:09:43.241 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:09:43.241 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-23T18:09:43.257 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-23T18:09:43.257 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:09:43.257 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-23T18:09:43.314 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-23T18:09:43.314 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:09:43.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-23T18:09:43.379 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-23T18:09:43.379 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:09:43.379 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-23T18:09:43.435 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-23T18:09:43.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDChIwbFEtv1dkUgFWQyQoV70XvzXBgQC+X1OE0+aItjE3IzVx546Wj67WgN4Y8pT1E9pvq93715nIGQnTRJaxVnEGnEwOCW10XZYVHaPckeqeoD4koPZ5S+cDAJMV5Dg+umW9zIv/SKpzstTHKFBg8EjONzPDjGkpMFl4gaaDSqkU2CJFr6hSUhXRsqF9I9xNKyMCWb/4khIlIEK7Eh+aSgckT6cXo0YFgTTqGS4izOpegt/8/q9JRvVe5vfnuJIdIiaSDAbBb9L3+51KIWnFYAYCsLKwGCA2s79zLPCxI39UKplPvdwKh89BO6uqudUaldBaLn36tfP8MVlXA7oV/jqqEvP06gzQT2rXv3fL6fPNq5+n0OQaoyuNYqiXshIiVGYo0bd/hmUsD3zw2pqcoOYhCeFQoj+74Ws6Cno5u8YsBotcdRzXFHaJSHfGwnKhHltUNptyAk0PwGWAf9nN0gJUPbPczBAWOQgbonxcj1qDn41ImaGJR/lLw3dv4u1k= ceph-5d677974-019c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T18:09:43.508 INFO:teuthology.orchestra.run.smithi050.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDChIwbFEtv1dkUgFWQyQoV70XvzXBgQC+X1OE0+aItjE3IzVx546Wj67WgN4Y8pT1E9pvq93715nIGQnTRJaxVnEGnEwOCW10XZYVHaPckeqeoD4koPZ5S+cDAJMV5Dg+umW9zIv/SKpzstTHKFBg8EjONzPDjGkpMFl4gaaDSqkU2CJFr6hSUhXRsqF9I9xNKyMCWb/4khIlIEK7Eh+aSgckT6cXo0YFgTTqGS4izOpegt/8/q9JRvVe5vfnuJIdIiaSDAbBb9L3+51KIWnFYAYCsLKwGCA2s79zLPCxI39UKplPvdwKh89BO6uqudUaldBaLn36tfP8MVlXA7oV/jqqEvP06gzQT2rXv3fL6fPNq5+n0OQaoyuNYqiXshIiVGYo0bd/hmUsD3zw2pqcoOYhCeFQoj+74Ws6Cno5u8YsBotcdRzXFHaJSHfGwnKhHltUNptyAk0PwGWAf9nN0gJUPbPczBAWOQgbonxcj1qDn41ImaGJR/lLw3dv4u1k= ceph-5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:43.520 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDChIwbFEtv1dkUgFWQyQoV70XvzXBgQC+X1OE0+aItjE3IzVx546Wj67WgN4Y8pT1E9pvq93715nIGQnTRJaxVnEGnEwOCW10XZYVHaPckeqeoD4koPZ5S+cDAJMV5Dg+umW9zIv/SKpzstTHKFBg8EjONzPDjGkpMFl4gaaDSqkU2CJFr6hSUhXRsqF9I9xNKyMCWb/4khIlIEK7Eh+aSgckT6cXo0YFgTTqGS4izOpegt/8/q9JRvVe5vfnuJIdIiaSDAbBb9L3+51KIWnFYAYCsLKwGCA2s79zLPCxI39UKplPvdwKh89BO6uqudUaldBaLn36tfP8MVlXA7oV/jqqEvP06gzQT2rXv3fL6fPNq5+n0OQaoyuNYqiXshIiVGYo0bd/hmUsD3zw2pqcoOYhCeFQoj+74Ws6Cno5u8YsBotcdRzXFHaJSHfGwnKhHltUNptyAk0PwGWAf9nN0gJUPbPczBAWOQgbonxcj1qDn41ImaGJR/lLw3dv4u1k= ceph-5d677974-019c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T18:09:43.559 INFO:teuthology.orchestra.run.smithi177.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDChIwbFEtv1dkUgFWQyQoV70XvzXBgQC+X1OE0+aItjE3IzVx546Wj67WgN4Y8pT1E9pvq93715nIGQnTRJaxVnEGnEwOCW10XZYVHaPckeqeoD4koPZ5S+cDAJMV5Dg+umW9zIv/SKpzstTHKFBg8EjONzPDjGkpMFl4gaaDSqkU2CJFr6hSUhXRsqF9I9xNKyMCWb/4khIlIEK7Eh+aSgckT6cXo0YFgTTqGS4izOpegt/8/q9JRvVe5vfnuJIdIiaSDAbBb9L3+51KIWnFYAYCsLKwGCA2s79zLPCxI39UKplPvdwKh89BO6uqudUaldBaLn36tfP8MVlXA7oV/jqqEvP06gzQT2rXv3fL6fPNq5+n0OQaoyuNYqiXshIiVGYo0bd/hmUsD3zw2pqcoOYhCeFQoj+74Ws6Cno5u8YsBotcdRzXFHaJSHfGwnKhHltUNptyAk0PwGWAf9nN0gJUPbPczBAWOQgbonxcj1qDn41ImaGJR/lLw3dv4u1k= ceph-5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:43.571 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDChIwbFEtv1dkUgFWQyQoV70XvzXBgQC+X1OE0+aItjE3IzVx546Wj67WgN4Y8pT1E9pvq93715nIGQnTRJaxVnEGnEwOCW10XZYVHaPckeqeoD4koPZ5S+cDAJMV5Dg+umW9zIv/SKpzstTHKFBg8EjONzPDjGkpMFl4gaaDSqkU2CJFr6hSUhXRsqF9I9xNKyMCWb/4khIlIEK7Eh+aSgckT6cXo0YFgTTqGS4izOpegt/8/q9JRvVe5vfnuJIdIiaSDAbBb9L3+51KIWnFYAYCsLKwGCA2s79zLPCxI39UKplPvdwKh89BO6uqudUaldBaLn36tfP8MVlXA7oV/jqqEvP06gzQT2rXv3fL6fPNq5+n0OQaoyuNYqiXshIiVGYo0bd/hmUsD3zw2pqcoOYhCeFQoj+74Ws6Cno5u8YsBotcdRzXFHaJSHfGwnKhHltUNptyAk0PwGWAf9nN0gJUPbPczBAWOQgbonxcj1qDn41ImaGJR/lLw3dv4u1k= ceph-5d677974-019c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T18:09:43.609 INFO:teuthology.orchestra.run.smithi188.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDChIwbFEtv1dkUgFWQyQoV70XvzXBgQC+X1OE0+aItjE3IzVx546Wj67WgN4Y8pT1E9pvq93715nIGQnTRJaxVnEGnEwOCW10XZYVHaPckeqeoD4koPZ5S+cDAJMV5Dg+umW9zIv/SKpzstTHKFBg8EjONzPDjGkpMFl4gaaDSqkU2CJFr6hSUhXRsqF9I9xNKyMCWb/4khIlIEK7Eh+aSgckT6cXo0YFgTTqGS4izOpegt/8/q9JRvVe5vfnuJIdIiaSDAbBb9L3+51KIWnFYAYCsLKwGCA2s79zLPCxI39UKplPvdwKh89BO6uqudUaldBaLn36tfP8MVlXA7oV/jqqEvP06gzQT2rXv3fL6fPNq5+n0OQaoyuNYqiXshIiVGYo0bd/hmUsD3zw2pqcoOYhCeFQoj+74Ws6Cno5u8YsBotcdRzXFHaJSHfGwnKhHltUNptyAk0PwGWAf9nN0gJUPbPczBAWOQgbonxcj1qDn41ImaGJR/lLw3dv4u1k= ceph-5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:09:43.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-23T18:09:43.872 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:09:43.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:43 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3494217517' entity='client.admin' 2024-04-23T18:09:46.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.004+0000 7f12f9361700 1 -- 172.21.15.50:0/4045722107 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12f4102f30 msgr2=0x7f12f4105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:46.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.004+0000 7f12f9361700 1 --2- 172.21.15.50:0/4045722107 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12f4102f30 0x7f12f4105320 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009910 tx=0x7f12e4009c20 comp rx=0 tx=0).stop 2024-04-23T18:09:46.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.004+0000 7f12f9361700 1 -- 172.21.15.50:0/4045722107 shutdown_connections 2024-04-23T18:09:46.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.004+0000 7f12f9361700 1 --2- 172.21.15.50:0/4045722107 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12f4102f30 0x7f12f4105320 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:46.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.004+0000 7f12f9361700 1 -- 172.21.15.50:0/4045722107 >> 172.21.15.50:0/4045722107 conn(0x7f12f40fc940 msgr2=0x7f12f40fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:46.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.004+0000 7f12f9361700 1 -- 172.21.15.50:0/4045722107 shutdown_connections 2024-04-23T18:09:46.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.004+0000 7f12f9361700 1 -- 172.21.15.50:0/4045722107 wait complete. 2024-04-23T18:09:46.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.004+0000 7f12f9361700 1 Processor -- start 2024-04-23T18:09:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.005+0000 7f12f9361700 1 -- start start 2024-04-23T18:09:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.005+0000 7f12f9361700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12f4102f30 0x7f12f4197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.005+0000 7f12f9361700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f12f4108380 con 0x7f12f4102f30 2024-04-23T18:09:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.005+0000 7f12f2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12f4102f30 0x7f12f4197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.005+0000 7f12f2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12f4102f30 0x7f12f4197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60320/0 (socket says 172.21.15.50:60320) 2024-04-23T18:09:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.005+0000 7f12f2ffd700 1 -- 172.21.15.50:0/2475321816 learned_addr learned my addr 172.21.15.50:0/2475321816 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.005+0000 7f12f2ffd700 1 -- 172.21.15.50:0/2475321816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f4197f70 con 0x7f12f4102f30 2024-04-23T18:09:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.005+0000 7f12f2ffd700 1 --2- 172.21.15.50:0/2475321816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12f4102f30 0x7f12f4197a30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f12e4005b40 tx=0x7f12e4004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.006+0000 7f12dbfff700 1 -- 172.21.15.50:0/2475321816 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12e4027070 con 0x7f12f4102f30 2024-04-23T18:09:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.006+0000 7f12f9361700 1 -- 172.21.15.50:0/2475321816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f4198260 con 0x7f12f4102f30 2024-04-23T18:09:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.006+0000 7f12dbfff700 1 -- 172.21.15.50:0/2475321816 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12e400be10 con 0x7f12f4102f30 2024-04-23T18:09:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.006+0000 7f12dbfff700 1 -- 172.21.15.50:0/2475321816 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12e4022de0 con 0x7f12f4102f30 2024-04-23T18:09:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.006+0000 7f12f9361700 1 -- 172.21.15.50:0/2475321816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f419aef0 con 0x7f12f4102f30 2024-04-23T18:09:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.006+0000 7f12dbfff700 1 -- 172.21.15.50:0/2475321816 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f12e402a070 con 0x7f12f4102f30 2024-04-23T18:09:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.007+0000 7f12dbfff700 1 --2- 172.21.15.50:0/2475321816 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f12dc039650 0x7f12dc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.007+0000 7f12f9361700 1 -- 172.21.15.50:0/2475321816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f4108b00 con 0x7f12f4102f30 2024-04-23T18:09:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.007+0000 7f12dbfff700 1 -- 172.21.15.50:0/2475321816 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f12e401e030 con 0x7f12f4102f30 2024-04-23T18:09:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.007+0000 7f12f27fc700 1 --2- 172.21.15.50:0/2475321816 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f12dc039650 0x7f12dc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:46.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.007+0000 7f12f27fc700 1 --2- 172.21.15.50:0/2475321816 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f12dc039650 0x7f12dc03bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f12e8006fd0 tx=0x7f12e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:46.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.012+0000 7f12dbfff700 1 -- 172.21.15.50:0/2475321816 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f12e4013070 con 0x7f12f4102f30 2024-04-23T18:09:46.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.219+0000 7f12f9361700 1 -- 172.21.15.50:0/2475321816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f12f402e0d0 con 0x7f12f4102f30 2024-04-23T18:09:46.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.221+0000 7f12dbfff700 1 -- 172.21.15.50:0/2475321816 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f12e4031020 con 0x7f12f4102f30 2024-04-23T18:09:46.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.224+0000 7f12f9361700 1 -- 172.21.15.50:0/2475321816 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f12dc039650 msgr2=0x7f12dc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:46.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.224+0000 7f12f9361700 1 --2- 172.21.15.50:0/2475321816 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f12dc039650 0x7f12dc03bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f12e8006fd0 tx=0x7f12e8006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:46.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.224+0000 7f12f9361700 1 -- 172.21.15.50:0/2475321816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12f4102f30 msgr2=0x7f12f4197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:46.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.224+0000 7f12f9361700 1 --2- 172.21.15.50:0/2475321816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12f4102f30 0x7f12f4197a30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f12e4005b40 tx=0x7f12e4004f80 comp rx=0 tx=0).stop 2024-04-23T18:09:46.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.224+0000 7f12f9361700 1 -- 172.21.15.50:0/2475321816 shutdown_connections 2024-04-23T18:09:46.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.224+0000 7f12f9361700 1 --2- 172.21.15.50:0/2475321816 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f12dc039650 0x7f12dc03bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:46.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.224+0000 7f12f9361700 1 --2- 172.21.15.50:0/2475321816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12f4102f30 0x7f12f4197a30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:46.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.224+0000 7f12f9361700 1 -- 172.21.15.50:0/2475321816 >> 172.21.15.50:0/2475321816 conn(0x7f12f40fc940 msgr2=0x7f12f40fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:46.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.224+0000 7f12f9361700 1 -- 172.21.15.50:0/2475321816 shutdown_connections 2024-04-23T18:09:46.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:46.224+0000 7f12f9361700 1 -- 172.21.15.50:0/2475321816 wait complete. 2024-04-23T18:09:46.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:46 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:46.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:46 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:46.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:46 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi050", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:09:46.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:46 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:46.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:46 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:46.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:46 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:46.800 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-23T18:09:46.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-23T18:09:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:09:47.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:47 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2475321816' entity='client.admin' 2024-04-23T18:09:47.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:47 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:47.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:47 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:48.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.608+0000 7f8b3c008700 1 -- 172.21.15.50:0/2468638738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b341024e0 msgr2=0x7f8b34102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:48.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.608+0000 7f8b3c008700 1 --2- 172.21.15.50:0/2468638738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b341024e0 0x7f8b34102900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009910 tx=0x7f8b30009c20 comp rx=0 tx=0).stop 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.608+0000 7f8b3c008700 1 -- 172.21.15.50:0/2468638738 shutdown_connections 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.608+0000 7f8b3c008700 1 --2- 172.21.15.50:0/2468638738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b341024e0 0x7f8b34102900 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.608+0000 7f8b3c008700 1 -- 172.21.15.50:0/2468638738 >> 172.21.15.50:0/2468638738 conn(0x7f8b340fda40 msgr2=0x7f8b340ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.608+0000 7f8b3c008700 1 -- 172.21.15.50:0/2468638738 shutdown_connections 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.608+0000 7f8b3c008700 1 -- 172.21.15.50:0/2468638738 wait complete. 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.608+0000 7f8b3c008700 1 Processor -- start 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.608+0000 7f8b3c008700 1 -- start start 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.608+0000 7f8b3c008700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b341024e0 0x7f8b34197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.609+0000 7f8b3c008700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8b34108420 con 0x7f8b341024e0 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.609+0000 7f8b39da4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b341024e0 0x7f8b34197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.609+0000 7f8b39da4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b341024e0 0x7f8b34197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60338/0 (socket says 172.21.15.50:60338) 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.609+0000 7f8b39da4700 1 -- 172.21.15.50:0/3073383489 learned_addr learned my addr 172.21.15.50:0/3073383489 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.609+0000 7f8b39da4700 1 -- 172.21.15.50:0/3073383489 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b34198070 con 0x7f8b341024e0 2024-04-23T18:09:48.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.609+0000 7f8b39da4700 1 --2- 172.21.15.50:0/3073383489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b341024e0 0x7f8b34197b30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8b30005800 tx=0x7f8b30009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.609+0000 7f8b2affd700 1 -- 172.21.15.50:0/3073383489 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b30027070 con 0x7f8b341024e0 2024-04-23T18:09:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.609+0000 7f8b2affd700 1 -- 172.21.15.50:0/3073383489 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b3000be10 con 0x7f8b341024e0 2024-04-23T18:09:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.609+0000 7f8b2affd700 1 -- 172.21.15.50:0/3073383489 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b3000f460 con 0x7f8b341024e0 2024-04-23T18:09:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.609+0000 7f8b3c008700 1 -- 172.21.15.50:0/3073383489 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b34198300 con 0x7f8b341024e0 2024-04-23T18:09:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.609+0000 7f8b3c008700 1 -- 172.21.15.50:0/3073383489 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b3419afa0 con 0x7f8b341024e0 2024-04-23T18:09:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.610+0000 7f8b2affd700 1 -- 172.21.15.50:0/3073383489 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8b3002a070 con 0x7f8b341024e0 2024-04-23T18:09:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.610+0000 7f8b3c008700 1 -- 172.21.15.50:0/3073383489 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b34108ba0 con 0x7f8b341024e0 2024-04-23T18:09:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.610+0000 7f8b2affd700 1 --2- 172.21.15.50:0/3073383489 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f8b20039650 0x7f8b2003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.610+0000 7f8b2affd700 1 -- 172.21.15.50:0/3073383489 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8b3001e030 con 0x7f8b341024e0 2024-04-23T18:09:48.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.611+0000 7f8b395a3700 1 --2- 172.21.15.50:0/3073383489 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f8b20039650 0x7f8b2003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:48.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.611+0000 7f8b395a3700 1 --2- 172.21.15.50:0/3073383489 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f8b20039650 0x7f8b2003bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8b24006fd0 tx=0x7f8b24006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:48.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.615+0000 7f8b2affd700 1 -- 172.21.15.50:0/3073383489 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b30013070 con 0x7f8b341024e0 2024-04-23T18:09:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.826+0000 7f8b3c008700 1 -- 172.21.15.50:0/3073383489 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f8b3402e090 con 0x7f8b20039650 2024-04-23T18:09:48.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.829+0000 7f8b2affd700 1 -- 172.21.15.50:0/3073383489 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8b3402e090 con 0x7f8b20039650 2024-04-23T18:09:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.831+0000 7f8b3c008700 1 -- 172.21.15.50:0/3073383489 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f8b20039650 msgr2=0x7f8b2003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.831+0000 7f8b3c008700 1 --2- 172.21.15.50:0/3073383489 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f8b20039650 0x7f8b2003bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8b24006fd0 tx=0x7f8b24006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.831+0000 7f8b3c008700 1 -- 172.21.15.50:0/3073383489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b341024e0 msgr2=0x7f8b34197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.831+0000 7f8b3c008700 1 --2- 172.21.15.50:0/3073383489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b341024e0 0x7f8b34197b30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8b30005800 tx=0x7f8b30009f60 comp rx=0 tx=0).stop 2024-04-23T18:09:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.832+0000 7f8b3c008700 1 -- 172.21.15.50:0/3073383489 shutdown_connections 2024-04-23T18:09:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.832+0000 7f8b3c008700 1 --2- 172.21.15.50:0/3073383489 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f8b20039650 0x7f8b2003bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.832+0000 7f8b3c008700 1 --2- 172.21.15.50:0/3073383489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b341024e0 0x7f8b34197b30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.832+0000 7f8b3c008700 1 -- 172.21.15.50:0/3073383489 >> 172.21.15.50:0/3073383489 conn(0x7f8b340fda40 msgr2=0x7f8b340fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.832+0000 7f8b3c008700 1 -- 172.21.15.50:0/3073383489 shutdown_connections 2024-04-23T18:09:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:48.832+0000 7f8b3c008700 1 -- 172.21.15.50:0/3073383489 wait complete. 2024-04-23T18:09:49.516 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi177 2024-04-23T18:09:49.516 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-23T18:09:49.516 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.conf 2024-04-23T18:09:49.535 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-23T18:09:49.536 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:09:49.593 INFO:tasks.cephadm:Adding host smithi177 to orchestrator... 2024-04-23T18:09:49.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch host add smithi177 2024-04-23T18:09:49.838 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:09:50.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:49 smithi050 ceph-mon[29187]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:50.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:50.074 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:50.074 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:09:50.074 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:09:50.074 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:49 smithi050 ceph-mon[29187]: Updating smithi050:/etc/ceph/ceph.conf 2024-04-23T18:09:50.074 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:49 smithi050 ceph-mon[29187]: Updating smithi050:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:09:50.074 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:50.074 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:50.074 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:50.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:50 smithi050 ceph-mon[29187]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:09:50.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:50 smithi050 ceph-mon[29187]: Updating smithi050:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:09:51.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.414+0000 7f217bc3d700 1 -- 172.21.15.50:0/199885448 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2174103650 msgr2=0x7f2174103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:51.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.414+0000 7f217bc3d700 1 --2- 172.21.15.50:0/199885448 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2174103650 0x7f2174103a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2170009910 tx=0x7f2170009c20 comp rx=0 tx=0).stop 2024-04-23T18:09:51.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.414+0000 7f217bc3d700 1 -- 172.21.15.50:0/199885448 shutdown_connections 2024-04-23T18:09:51.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.414+0000 7f217bc3d700 1 --2- 172.21.15.50:0/199885448 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2174103650 0x7f2174103a30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.414+0000 7f217bc3d700 1 -- 172.21.15.50:0/199885448 >> 172.21.15.50:0/199885448 conn(0x7f21740fcea0 msgr2=0x7f21740ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.414+0000 7f217bc3d700 1 -- 172.21.15.50:0/199885448 shutdown_connections 2024-04-23T18:09:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.414+0000 7f217bc3d700 1 -- 172.21.15.50:0/199885448 wait complete. 2024-04-23T18:09:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.415+0000 7f217bc3d700 1 Processor -- start 2024-04-23T18:09:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.415+0000 7f217bc3d700 1 -- start start 2024-04-23T18:09:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.415+0000 7f217bc3d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2174103650 0x7f21741012b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.415+0000 7f217bc3d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f217410c9d0 con 0x7f2174103650 2024-04-23T18:09:51.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.415+0000 7f21799d9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2174103650 0x7f21741012b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:51.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.415+0000 7f21799d9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2174103650 0x7f21741012b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60354/0 (socket says 172.21.15.50:60354) 2024-04-23T18:09:51.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.415+0000 7f21799d9700 1 -- 172.21.15.50:0/1154681907 learned_addr learned my addr 172.21.15.50:0/1154681907 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:51.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.415+0000 7f21799d9700 1 -- 172.21.15.50:0/1154681907 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21741031a0 con 0x7f2174103650 2024-04-23T18:09:51.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.415+0000 7f21799d9700 1 --2- 172.21.15.50:0/1154681907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2174103650 0x7f21741012b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f21700056d0 tx=0x7f21700057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:51.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.416+0000 7f216affd700 1 -- 172.21.15.50:0/1154681907 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2170026070 con 0x7f2174103650 2024-04-23T18:09:51.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.416+0000 7f216affd700 1 -- 172.21.15.50:0/1154681907 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2170004370 con 0x7f2174103650 2024-04-23T18:09:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.416+0000 7f217bc3d700 1 -- 172.21.15.50:0/1154681907 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21741017f0 con 0x7f2174103650 2024-04-23T18:09:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.416+0000 7f216affd700 1 -- 172.21.15.50:0/1154681907 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21700039b0 con 0x7f2174103650 2024-04-23T18:09:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.416+0000 7f217bc3d700 1 -- 172.21.15.50:0/1154681907 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2174101cb0 con 0x7f2174103650 2024-04-23T18:09:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.416+0000 7f216affd700 1 -- 172.21.15.50:0/1154681907 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2170029070 con 0x7f2174103650 2024-04-23T18:09:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.417+0000 7f217bc3d700 1 -- 172.21.15.50:0/1154681907 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f217406b190 con 0x7f2174103650 2024-04-23T18:09:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.417+0000 7f216affd700 1 --2- 172.21.15.50:0/1154681907 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2160039650 0x7f216003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.417+0000 7f216affd700 1 -- 172.21.15.50:0/1154681907 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f217001e070 con 0x7f2174103650 2024-04-23T18:09:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.417+0000 7f21791d8700 1 --2- 172.21.15.50:0/1154681907 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2160039650 0x7f216003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.417+0000 7f21791d8700 1 --2- 172.21.15.50:0/1154681907 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2160039650 0x7f216003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2164006fd0 tx=0x7f2164006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:51.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.422+0000 7f216affd700 1 -- 172.21.15.50:0/1154681907 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2170013070 con 0x7f2174103650 2024-04-23T18:09:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:51.636+0000 7f217bc3d700 1 -- 172.21.15.50:0/1154681907 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}) -- 0x7f21741027e0 con 0x7f2160039650 2024-04-23T18:09:52.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:52 smithi050 ceph-mon[29187]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:09:53.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:53 smithi050 ceph-mon[29187]: Deploying cephadm binary to smithi177 2024-04-23T18:09:55.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:55 smithi050 ceph-mon[29187]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:55.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:55.895+0000 7f216affd700 1 -- 172.21.15.50:0/1154681907 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f21741027e0 con 0x7f2160039650 2024-04-23T18:09:55.890 INFO:teuthology.orchestra.run.smithi050.stdout:Added host 'smithi177' with addr '172.21.15.177' 2024-04-23T18:09:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:55.898+0000 7f217bc3d700 1 -- 172.21.15.50:0/1154681907 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2160039650 msgr2=0x7f216003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:55.898+0000 7f217bc3d700 1 --2- 172.21.15.50:0/1154681907 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2160039650 0x7f216003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2164006fd0 tx=0x7f2164006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:55.898+0000 7f217bc3d700 1 -- 172.21.15.50:0/1154681907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2174103650 msgr2=0x7f21741012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:55.898+0000 7f217bc3d700 1 --2- 172.21.15.50:0/1154681907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2174103650 0x7f21741012b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f21700056d0 tx=0x7f21700057b0 comp rx=0 tx=0).stop 2024-04-23T18:09:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:55.898+0000 7f217bc3d700 1 -- 172.21.15.50:0/1154681907 shutdown_connections 2024-04-23T18:09:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:55.898+0000 7f217bc3d700 1 --2- 172.21.15.50:0/1154681907 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2160039650 0x7f216003bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:55.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:55.898+0000 7f217bc3d700 1 --2- 172.21.15.50:0/1154681907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2174103650 0x7f21741012b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:55.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:55.898+0000 7f217bc3d700 1 -- 172.21.15.50:0/1154681907 >> 172.21.15.50:0/1154681907 conn(0x7f21740fcea0 msgr2=0x7f21741043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:55.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:55.898+0000 7f217bc3d700 1 -- 172.21.15.50:0/1154681907 shutdown_connections 2024-04-23T18:09:55.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:55.898+0000 7f217bc3d700 1 -- 172.21.15.50:0/1154681907 wait complete. 2024-04-23T18:09:56.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T18:09:56.691 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:09:57.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:56 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:57.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:56 smithi050 ceph-mon[29187]: Added host smithi177 2024-04-23T18:09:57.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:56 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:09:57.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:56 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:58.033 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:57 smithi050 ceph-mon[29187]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:58.033 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:57 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:58.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.278+0000 7f4d06f53700 1 -- 172.21.15.50:0/3572065497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00103610 msgr2=0x7f4d001039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:58.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.278+0000 7f4d06f53700 1 --2- 172.21.15.50:0/3572065497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00103610 0x7f4d001039f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009910 tx=0x7f4cf0009c20 comp rx=0 tx=0).stop 2024-04-23T18:09:58.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.278+0000 7f4d06f53700 1 -- 172.21.15.50:0/3572065497 shutdown_connections 2024-04-23T18:09:58.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.278+0000 7f4d06f53700 1 --2- 172.21.15.50:0/3572065497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00103610 0x7f4d001039f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:58.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.278+0000 7f4d06f53700 1 -- 172.21.15.50:0/3572065497 >> 172.21.15.50:0/3572065497 conn(0x7f4d000fce60 msgr2=0x7f4d000ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.278+0000 7f4d06f53700 1 -- 172.21.15.50:0/3572065497 shutdown_connections 2024-04-23T18:09:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.278+0000 7f4d06f53700 1 -- 172.21.15.50:0/3572065497 wait complete. 2024-04-23T18:09:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.279+0000 7f4d06f53700 1 Processor -- start 2024-04-23T18:09:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.279+0000 7f4d06f53700 1 -- start start 2024-04-23T18:09:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.279+0000 7f4d06f53700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00103610 0x7f4d00197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.279+0000 7f4d06f53700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0010c9b0 con 0x7f4d00103610 2024-04-23T18:09:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.279+0000 7f4d04cef700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00103610 0x7f4d00197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.279+0000 7f4d04cef700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00103610 0x7f4d00197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58554/0 (socket says 172.21.15.50:58554) 2024-04-23T18:09:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.279+0000 7f4d04cef700 1 -- 172.21.15.50:0/1016209118 learned_addr learned my addr 172.21.15.50:0/1016209118 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:09:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.279+0000 7f4d04cef700 1 -- 172.21.15.50:0/1016209118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d00198460 con 0x7f4d00103610 2024-04-23T18:09:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.279+0000 7f4d04cef700 1 --2- 172.21.15.50:0/1016209118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00103610 0x7f4d00197f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4cf00056d0 tx=0x7f4cf00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:58.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.279+0000 7f4cfdffb700 1 -- 172.21.15.50:0/1016209118 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cf0026070 con 0x7f4d00103610 2024-04-23T18:09:58.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.280+0000 7f4cfdffb700 1 -- 172.21.15.50:0/1016209118 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cf0004370 con 0x7f4d00103610 2024-04-23T18:09:58.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.280+0000 7f4d06f53700 1 -- 172.21.15.50:0/1016209118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d001986f0 con 0x7f4d00103610 2024-04-23T18:09:58.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.280+0000 7f4cfdffb700 1 -- 172.21.15.50:0/1016209118 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cf00039b0 con 0x7f4d00103610 2024-04-23T18:09:58.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.280+0000 7f4d06f53700 1 -- 172.21.15.50:0/1016209118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d00198b50 con 0x7f4d00103610 2024-04-23T18:09:58.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.280+0000 7f4cfdffb700 1 -- 172.21.15.50:0/1016209118 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4cf0029070 con 0x7f4d00103610 2024-04-23T18:09:58.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.281+0000 7f4cfdffb700 1 --2- 172.21.15.50:0/1016209118 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4ce80396b0 0x7f4ce803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:09:58.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.281+0000 7f4cfdffb700 1 -- 172.21.15.50:0/1016209118 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4cf001e070 con 0x7f4d00103610 2024-04-23T18:09:58.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.281+0000 7f4d06f53700 1 -- 172.21.15.50:0/1016209118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d0006b1d0 con 0x7f4d00103610 2024-04-23T18:09:58.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.281+0000 7f4cfffff700 1 --2- 172.21.15.50:0/1016209118 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4ce80396b0 0x7f4ce803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:09:58.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.281+0000 7f4cfffff700 1 --2- 172.21.15.50:0/1016209118 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4ce80396b0 0x7f4ce803bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4006fd0 tx=0x7f4cf4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:09:58.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.286+0000 7f4cfdffb700 1 -- 172.21.15.50:0/1016209118 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4cf0030020 con 0x7f4d00103610 2024-04-23T18:09:58.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.500+0000 7f4d06f53700 1 -- 172.21.15.50:0/1016209118 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d00199280 con 0x7f4ce80396b0 2024-04-23T18:09:58.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.502+0000 7f4cfdffb700 1 -- 172.21.15.50:0/1016209118 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f4d00199280 con 0x7f4ce80396b0 2024-04-23T18:09:58.496 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:09:58.496 INFO:teuthology.orchestra.run.smithi050.stdout:[{"addr": "172.21.15.50", "hostname": "smithi050", "labels": [], "status": ""}, {"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}] 2024-04-23T18:09:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.504+0000 7f4d06f53700 1 -- 172.21.15.50:0/1016209118 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4ce80396b0 msgr2=0x7f4ce803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.504+0000 7f4d06f53700 1 --2- 172.21.15.50:0/1016209118 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4ce80396b0 0x7f4ce803bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4006fd0 tx=0x7f4cf4006e40 comp rx=0 tx=0).stop 2024-04-23T18:09:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.504+0000 7f4d06f53700 1 -- 172.21.15.50:0/1016209118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00103610 msgr2=0x7f4d00197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:09:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.504+0000 7f4d06f53700 1 --2- 172.21.15.50:0/1016209118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00103610 0x7f4d00197f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4cf00056d0 tx=0x7f4cf00057b0 comp rx=0 tx=0).stop 2024-04-23T18:09:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.504+0000 7f4d06f53700 1 -- 172.21.15.50:0/1016209118 shutdown_connections 2024-04-23T18:09:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.504+0000 7f4d06f53700 1 --2- 172.21.15.50:0/1016209118 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4ce80396b0 0x7f4ce803bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.504+0000 7f4d06f53700 1 --2- 172.21.15.50:0/1016209118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d00103610 0x7f4d00197f20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:09:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.504+0000 7f4d06f53700 1 -- 172.21.15.50:0/1016209118 >> 172.21.15.50:0/1016209118 conn(0x7f4d000fce60 msgr2=0x7f4d001043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:09:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.504+0000 7f4d06f53700 1 -- 172.21.15.50:0/1016209118 shutdown_connections 2024-04-23T18:09:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:09:58.504+0000 7f4d06f53700 1 -- 172.21.15.50:0/1016209118 wait complete. 2024-04-23T18:09:59.067 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi188 2024-04-23T18:09:59.068 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T18:09:59.068 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.conf 2024-04-23T18:09:59.087 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T18:09:59.087 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:09:59.145 INFO:tasks.cephadm:Adding host smithi188 to orchestrator... 2024-04-23T18:09:59.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch host add smithi188 2024-04-23T18:09:59.392 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:09:59.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:59 smithi050 ceph-mon[29187]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:09:59.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:59 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:09:59.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:09:59 smithi050 ceph-mon[29187]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:10:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.899+0000 7f70a8aa6700 1 -- 172.21.15.50:0/1436445835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70a4103610 msgr2=0x7f70a41039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.899+0000 7f70a8aa6700 1 --2- 172.21.15.50:0/1436445835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70a4103610 0x7f70a41039f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7094009910 tx=0x7f7094009c20 comp rx=0 tx=0).stop 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.899+0000 7f70a8aa6700 1 -- 172.21.15.50:0/1436445835 shutdown_connections 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.899+0000 7f70a8aa6700 1 --2- 172.21.15.50:0/1436445835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70a4103610 0x7f70a41039f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.899+0000 7f70a8aa6700 1 -- 172.21.15.50:0/1436445835 >> 172.21.15.50:0/1436445835 conn(0x7f70a40fce40 msgr2=0x7f70a40ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.899+0000 7f70a8aa6700 1 -- 172.21.15.50:0/1436445835 shutdown_connections 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.899+0000 7f70a8aa6700 1 -- 172.21.15.50:0/1436445835 wait complete. 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.900+0000 7f70a8aa6700 1 Processor -- start 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.900+0000 7f70a8aa6700 1 -- start start 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.900+0000 7f70a8aa6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70a4103610 0x7f70a4101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.900+0000 7f70a8aa6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f70a410c990 con 0x7f70a4103610 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.900+0000 7f70a259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70a4103610 0x7f70a4101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.900+0000 7f70a259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70a4103610 0x7f70a4101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58574/0 (socket says 172.21.15.50:58574) 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.900+0000 7f70a259c700 1 -- 172.21.15.50:0/3086485068 learned_addr learned my addr 172.21.15.50:0/3086485068 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:10:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.900+0000 7f70a259c700 1 -- 172.21.15.50:0/3086485068 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a4103160 con 0x7f70a4103610 2024-04-23T18:10:00.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.900+0000 7f70a259c700 1 --2- 172.21.15.50:0/3086485068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70a4103610 0x7f70a4101270 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f70940056d0 tx=0x7f70940057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:00.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.901+0000 7f70937fe700 1 -- 172.21.15.50:0/3086485068 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7094026070 con 0x7f70a4103610 2024-04-23T18:10:00.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.901+0000 7f70a8aa6700 1 -- 172.21.15.50:0/3086485068 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a41017b0 con 0x7f70a4103610 2024-04-23T18:10:00.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.901+0000 7f70937fe700 1 -- 172.21.15.50:0/3086485068 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7094004370 con 0x7f70a4103610 2024-04-23T18:10:00.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.901+0000 7f70937fe700 1 -- 172.21.15.50:0/3086485068 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70940039b0 con 0x7f70a4103610 2024-04-23T18:10:00.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.901+0000 7f70a8aa6700 1 -- 172.21.15.50:0/3086485068 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a4101c70 con 0x7f70a4103610 2024-04-23T18:10:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.901+0000 7f70937fe700 1 -- 172.21.15.50:0/3086485068 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f7094029070 con 0x7f70a4103610 2024-04-23T18:10:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.902+0000 7f70a8aa6700 1 -- 172.21.15.50:0/3086485068 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a406b130 con 0x7f70a4103610 2024-04-23T18:10:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.902+0000 7f70937fe700 1 --2- 172.21.15.50:0/3086485068 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f708c039650 0x7f708c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.902+0000 7f70937fe700 1 -- 172.21.15.50:0/3086485068 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f709401e070 con 0x7f70a4103610 2024-04-23T18:10:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.902+0000 7f70a1d9b700 1 --2- 172.21.15.50:0/3086485068 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f708c039650 0x7f708c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.902+0000 7f70a1d9b700 1 --2- 172.21.15.50:0/3086485068 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f708c039650 0x7f708c03bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7098006fd0 tx=0x7f7098006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:00.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:00.907+0000 7f70937fe700 1 -- 172.21.15.50:0/3086485068 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7094013070 con 0x7f70a4103610 2024-04-23T18:10:01.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:01.121+0000 7f70a8aa6700 1 -- 172.21.15.50:0/3086485068 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}) -- 0x7f70a4101ea0 con 0x7f708c039650 2024-04-23T18:10:01.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:01 smithi050 ceph-mon[29187]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:01.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:01 smithi050 ceph-mon[29187]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:02.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:02.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:02.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:02.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:02.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:02.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:02.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:02.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:02 smithi050 ceph-mon[29187]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-23T18:10:02.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:02 smithi050 ceph-mon[29187]: Deploying cephadm binary to smithi188 2024-04-23T18:10:02.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:02 smithi050 ceph-mon[29187]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:02.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:02 smithi050 ceph-mon[29187]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:10:03.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:03 smithi050 ceph-mon[29187]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:05.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:05.301+0000 7f70937fe700 1 -- 172.21.15.50:0/3086485068 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f70a4101ea0 con 0x7f708c039650 2024-04-23T18:10:05.296 INFO:teuthology.orchestra.run.smithi050.stdout:Added host 'smithi188' with addr '172.21.15.188' 2024-04-23T18:10:05.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:05.303+0000 7f70a8aa6700 1 -- 172.21.15.50:0/3086485068 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f708c039650 msgr2=0x7f708c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:05.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:05.303+0000 7f70a8aa6700 1 --2- 172.21.15.50:0/3086485068 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f708c039650 0x7f708c03bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7098006fd0 tx=0x7f7098006e40 comp rx=0 tx=0).stop 2024-04-23T18:10:05.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:05.303+0000 7f70a8aa6700 1 -- 172.21.15.50:0/3086485068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70a4103610 msgr2=0x7f70a4101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:05.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:05.303+0000 7f70a8aa6700 1 --2- 172.21.15.50:0/3086485068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70a4103610 0x7f70a4101270 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f70940056d0 tx=0x7f70940057b0 comp rx=0 tx=0).stop 2024-04-23T18:10:05.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:05.303+0000 7f70a8aa6700 1 -- 172.21.15.50:0/3086485068 shutdown_connections 2024-04-23T18:10:05.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:05.303+0000 7f70a8aa6700 1 --2- 172.21.15.50:0/3086485068 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f708c039650 0x7f708c03bb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:05.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:05.303+0000 7f70a8aa6700 1 --2- 172.21.15.50:0/3086485068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70a4103610 0x7f70a4101270 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:05.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:05.303+0000 7f70a8aa6700 1 -- 172.21.15.50:0/3086485068 >> 172.21.15.50:0/3086485068 conn(0x7f70a40fce40 msgr2=0x7f70a41043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:05.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:05.303+0000 7f70a8aa6700 1 -- 172.21.15.50:0/3086485068 shutdown_connections 2024-04-23T18:10:05.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:05.303+0000 7f70a8aa6700 1 -- 172.21.15.50:0/3086485068 wait complete. 2024-04-23T18:10:05.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T18:10:06.139 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:10:06.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:05 smithi050 ceph-mon[29187]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:06.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:05 smithi050 ceph-mon[29187]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:10:06.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:05 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:06.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:05 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:06.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:05 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:06.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:05 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:06.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:05 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:06.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:05 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:07.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:06 smithi050 ceph-mon[29187]: Added host smithi188 2024-04-23T18:10:07.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.564+0000 7fe63293f700 1 -- 172.21.15.50:0/1321267203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe62c1035f0 msgr2=0x7fe62c1039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.564+0000 7fe63293f700 1 --2- 172.21.15.50:0/1321267203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe62c1035f0 0x7fe62c1039d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009910 tx=0x7fe61c009c20 comp rx=0 tx=0).stop 2024-04-23T18:10:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.564+0000 7fe63293f700 1 -- 172.21.15.50:0/1321267203 shutdown_connections 2024-04-23T18:10:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.564+0000 7fe63293f700 1 --2- 172.21.15.50:0/1321267203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe62c1035f0 0x7fe62c1039d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.564+0000 7fe63293f700 1 -- 172.21.15.50:0/1321267203 >> 172.21.15.50:0/1321267203 conn(0x7fe62c0fce40 msgr2=0x7fe62c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.564+0000 7fe63293f700 1 -- 172.21.15.50:0/1321267203 shutdown_connections 2024-04-23T18:10:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.564+0000 7fe63293f700 1 -- 172.21.15.50:0/1321267203 wait complete. 2024-04-23T18:10:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.565+0000 7fe63293f700 1 Processor -- start 2024-04-23T18:10:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.565+0000 7fe63293f700 1 -- start start 2024-04-23T18:10:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.565+0000 7fe63293f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe62c1035f0 0x7fe62c101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.565+0000 7fe63293f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c10c990 con 0x7fe62c1035f0 2024-04-23T18:10:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.565+0000 7fe62bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe62c1035f0 0x7fe62c101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.565+0000 7fe62bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe62c1035f0 0x7fe62c101250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46344/0 (socket says 172.21.15.50:46344) 2024-04-23T18:10:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.565+0000 7fe62bfff700 1 -- 172.21.15.50:0/1874624493 learned_addr learned my addr 172.21.15.50:0/1874624493 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:10:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.565+0000 7fe62bfff700 1 -- 172.21.15.50:0/1874624493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62c103140 con 0x7fe62c1035f0 2024-04-23T18:10:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.566+0000 7fe62bfff700 1 --2- 172.21.15.50:0/1874624493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe62c1035f0 0x7fe62c101250 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe61c006010 tx=0x7fe61c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.566+0000 7fe6297fa700 1 -- 172.21.15.50:0/1874624493 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe61c026070 con 0x7fe62c1035f0 2024-04-23T18:10:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.566+0000 7fe63293f700 1 -- 172.21.15.50:0/1874624493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe62c101790 con 0x7fe62c1035f0 2024-04-23T18:10:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.566+0000 7fe6297fa700 1 -- 172.21.15.50:0/1874624493 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe61c00bc00 con 0x7fe62c1035f0 2024-04-23T18:10:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.566+0000 7fe6297fa700 1 -- 172.21.15.50:0/1874624493 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe61c003910 con 0x7fe62c1035f0 2024-04-23T18:10:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.566+0000 7fe63293f700 1 -- 172.21.15.50:0/1874624493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe62c101c50 con 0x7fe62c1035f0 2024-04-23T18:10:07.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.567+0000 7fe6297fa700 1 -- 172.21.15.50:0/1874624493 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe61c029070 con 0x7fe62c1035f0 2024-04-23T18:10:07.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.567+0000 7fe63293f700 1 -- 172.21.15.50:0/1874624493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe62c06afd0 con 0x7fe62c1035f0 2024-04-23T18:10:07.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.567+0000 7fe6297fa700 1 --2- 172.21.15.50:0/1874624493 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe614039650 0x7fe61403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:07.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.567+0000 7fe6297fa700 1 -- 172.21.15.50:0/1874624493 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe61c01e070 con 0x7fe62c1035f0 2024-04-23T18:10:07.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.567+0000 7fe62b7fe700 1 --2- 172.21.15.50:0/1874624493 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe614039650 0x7fe61403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:07.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.568+0000 7fe62b7fe700 1 --2- 172.21.15.50:0/1874624493 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe614039650 0x7fe61403bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe620006fd0 tx=0x7fe620006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.572+0000 7fe6297fa700 1 -- 172.21.15.50:0/1874624493 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe61c013070 con 0x7fe62c1035f0 2024-04-23T18:10:07.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.785+0000 7fe63293f700 1 -- 172.21.15.50:0/1874624493 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe62c101e80 con 0x7fe614039650 2024-04-23T18:10:07.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.787+0000 7fe6297fa700 1 -- 172.21.15.50:0/1874624493 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7fe62c101e80 con 0x7fe614039650 2024-04-23T18:10:07.781 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:10:07.781 INFO:teuthology.orchestra.run.smithi050.stdout:[{"addr": "172.21.15.50", "hostname": "smithi050", "labels": [], "status": ""}, {"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}, {"addr": "172.21.15.188", "hostname": "smithi188", "labels": [], "status": ""}] 2024-04-23T18:10:07.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.789+0000 7fe63293f700 1 -- 172.21.15.50:0/1874624493 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe614039650 msgr2=0x7fe61403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:07.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.789+0000 7fe63293f700 1 --2- 172.21.15.50:0/1874624493 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe614039650 0x7fe61403bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe620006fd0 tx=0x7fe620006e40 comp rx=0 tx=0).stop 2024-04-23T18:10:07.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.789+0000 7fe63293f700 1 -- 172.21.15.50:0/1874624493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe62c1035f0 msgr2=0x7fe62c101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:07.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.789+0000 7fe63293f700 1 --2- 172.21.15.50:0/1874624493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe62c1035f0 0x7fe62c101250 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe61c006010 tx=0x7fe61c005780 comp rx=0 tx=0).stop 2024-04-23T18:10:07.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.789+0000 7fe63293f700 1 -- 172.21.15.50:0/1874624493 shutdown_connections 2024-04-23T18:10:07.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.789+0000 7fe63293f700 1 --2- 172.21.15.50:0/1874624493 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe614039650 0x7fe61403bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:07.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.789+0000 7fe63293f700 1 --2- 172.21.15.50:0/1874624493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe62c1035f0 0x7fe62c101250 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:07.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.789+0000 7fe63293f700 1 -- 172.21.15.50:0/1874624493 >> 172.21.15.50:0/1874624493 conn(0x7fe62c0fce40 msgr2=0x7fe62c1063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:07.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.789+0000 7fe63293f700 1 -- 172.21.15.50:0/1874624493 shutdown_connections 2024-04-23T18:10:07.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:07.789+0000 7fe63293f700 1 -- 172.21.15.50:0/1874624493 wait complete. 2024-04-23T18:10:08.113 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:07 smithi050 ceph-mon[29187]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:08.113 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:07 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:08.417 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-23T18:10:08.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-23T18:10:08.657 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:10:09.042 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:08 smithi050 ceph-mon[29187]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:10:10.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.161+0000 7f90a18f7700 1 -- 172.21.15.50:0/554408837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909c108550 msgr2=0x7f909c108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:10.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.161+0000 7f90a18f7700 1 --2- 172.21.15.50:0/554408837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909c108550 0x7f909c108930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9084009910 tx=0x7f9084009c20 comp rx=0 tx=0).stop 2024-04-23T18:10:10.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.161+0000 7f90a18f7700 1 -- 172.21.15.50:0/554408837 shutdown_connections 2024-04-23T18:10:10.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.161+0000 7f90a18f7700 1 --2- 172.21.15.50:0/554408837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909c108550 0x7f909c108930 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:10.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.161+0000 7f90a18f7700 1 -- 172.21.15.50:0/554408837 >> 172.21.15.50:0/554408837 conn(0x7f909c0fdf70 msgr2=0x7f909c100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:10.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.161+0000 7f90a18f7700 1 -- 172.21.15.50:0/554408837 shutdown_connections 2024-04-23T18:10:10.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.161+0000 7f90a18f7700 1 -- 172.21.15.50:0/554408837 wait complete. 2024-04-23T18:10:10.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.162+0000 7f90a18f7700 1 Processor -- start 2024-04-23T18:10:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.162+0000 7f90a18f7700 1 -- start start 2024-04-23T18:10:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.162+0000 7f90a18f7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909c108550 0x7f909c197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.162+0000 7f90a18f7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f909c10c9c0 con 0x7f909c108550 2024-04-23T18:10:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.162+0000 7f909affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909c108550 0x7f909c197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.162+0000 7f909affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909c108550 0x7f909c197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46370/0 (socket says 172.21.15.50:46370) 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.162+0000 7f909affd700 1 -- 172.21.15.50:0/806437517 learned_addr learned my addr 172.21.15.50:0/806437517 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.163+0000 7f909affd700 1 -- 172.21.15.50:0/806437517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909c1983a0 con 0x7f909c108550 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.163+0000 7f909affd700 1 --2- 172.21.15.50:0/806437517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909c108550 0x7f909c197e60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f90840056d0 tx=0x7f90840057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.163+0000 7f90a08f5700 1 -- 172.21.15.50:0/806437517 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9084026070 con 0x7f909c108550 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.163+0000 7f90a08f5700 1 -- 172.21.15.50:0/806437517 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f908400bc00 con 0x7f909c108550 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.163+0000 7f90a18f7700 1 -- 172.21.15.50:0/806437517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f909c198630 con 0x7f909c108550 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.163+0000 7f90a08f5700 1 -- 172.21.15.50:0/806437517 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9084003920 con 0x7f909c108550 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.163+0000 7f90a18f7700 1 -- 172.21.15.50:0/806437517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f909c19c730 con 0x7f909c108550 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.164+0000 7f90a08f5700 1 -- 172.21.15.50:0/806437517 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f9084029070 con 0x7f909c108550 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.164+0000 7f90a18f7700 1 -- 172.21.15.50:0/806437517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f909c103bc0 con 0x7f909c108550 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.164+0000 7f90a08f5700 1 --2- 172.21.15.50:0/806437517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f90880396b0 0x7f908803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.164+0000 7f90a08f5700 1 -- 172.21.15.50:0/806437517 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f908401e070 con 0x7f909c108550 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.164+0000 7f909a7fc700 1 --2- 172.21.15.50:0/806437517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f90880396b0 0x7f908803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.165+0000 7f909a7fc700 1 --2- 172.21.15.50:0/806437517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f90880396b0 0x7f908803bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f908c006fd0 tx=0x7f908c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:10.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.169+0000 7f90a08f5700 1 -- 172.21.15.50:0/806437517 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9084013070 con 0x7f909c108550 2024-04-23T18:10:10.164 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:09 smithi050 ceph-mon[29187]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:10.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.380+0000 7f90a18f7700 1 -- 172.21.15.50:0/806437517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f909c1991c0 con 0x7f909c108550 2024-04-23T18:10:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.919+0000 7f90a08f5700 1 -- 172.21.15.50:0/806437517 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f908400e3e0 con 0x7f909c108550 2024-04-23T18:10:10.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.923+0000 7f90a18f7700 1 -- 172.21.15.50:0/806437517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f90880396b0 msgr2=0x7f908803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:10.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.923+0000 7f90a18f7700 1 --2- 172.21.15.50:0/806437517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f90880396b0 0x7f908803bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f908c006fd0 tx=0x7f908c006e40 comp rx=0 tx=0).stop 2024-04-23T18:10:10.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.923+0000 7f90a18f7700 1 -- 172.21.15.50:0/806437517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909c108550 msgr2=0x7f909c197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:10.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.923+0000 7f90a18f7700 1 --2- 172.21.15.50:0/806437517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909c108550 0x7f909c197e60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f90840056d0 tx=0x7f90840057b0 comp rx=0 tx=0).stop 2024-04-23T18:10:10.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.923+0000 7f90a18f7700 1 -- 172.21.15.50:0/806437517 shutdown_connections 2024-04-23T18:10:10.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.923+0000 7f90a18f7700 1 --2- 172.21.15.50:0/806437517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f90880396b0 0x7f908803bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:10.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.923+0000 7f90a18f7700 1 --2- 172.21.15.50:0/806437517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909c108550 0x7f909c197e60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:10.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.923+0000 7f90a18f7700 1 -- 172.21.15.50:0/806437517 >> 172.21.15.50:0/806437517 conn(0x7f909c0fdf70 msgr2=0x7f909c0fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:10.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.923+0000 7f90a18f7700 1 -- 172.21.15.50:0/806437517 shutdown_connections 2024-04-23T18:10:10.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:10.923+0000 7f90a18f7700 1 -- 172.21.15.50:0/806437517 wait complete. 2024-04-23T18:10:10.919 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2024-04-23T18:10:11.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:10 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/806437517' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T18:10:11.542 INFO:tasks.cephadm:Adding mon.a on smithi050 2024-04-23T18:10:11.542 INFO:tasks.cephadm:Adding mon.b on smithi177 2024-04-23T18:10:11.542 INFO:tasks.cephadm:Adding mon.c on smithi188 2024-04-23T18:10:11.542 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi050:172.21.15.50=a;smithi177:172.21.15.177=b;smithi188:172.21.15.188=c' 2024-04-23T18:10:11.772 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:12.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:12.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/806437517' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T18:10:12.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-23T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:13.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:12 smithi050 ceph-mon[29187]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:10:13.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:12 smithi050 ceph-mon[29187]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:10:13.213 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.218+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/3170517590 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3cc1015b0 msgr2=0x7fc3cc101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:13.213 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.218+0000 7fc3d0e2b700 1 --2- 172.21.15.188:0/3170517590 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3cc1015b0 0x7fc3cc101990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc009910 tx=0x7fc3bc009c20 comp rx=0 tx=0).stop 2024-04-23T18:10:13.213 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.218+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/3170517590 shutdown_connections 2024-04-23T18:10:13.213 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.218+0000 7fc3d0e2b700 1 --2- 172.21.15.188:0/3170517590 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3cc1015b0 0x7fc3cc101990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:13.213 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.218+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/3170517590 >> 172.21.15.188:0/3170517590 conn(0x7fc3cc0fce60 msgr2=0x7fc3cc0ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:13.214 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.218+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/3170517590 shutdown_connections 2024-04-23T18:10:13.214 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.218+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/3170517590 wait complete. 2024-04-23T18:10:13.214 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.219+0000 7fc3d0e2b700 1 Processor -- start 2024-04-23T18:10:13.214 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.219+0000 7fc3d0e2b700 1 -- start start 2024-04-23T18:10:13.214 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.219+0000 7fc3d0e2b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3cc1015b0 0x7fc3cc198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:13.214 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.219+0000 7fc3d0e2b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc3cc10b830 con 0x7fc3cc1015b0 2024-04-23T18:10:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.220+0000 7fc3ca59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3cc1015b0 0x7fc3cc198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.220+0000 7fc3ca59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3cc1015b0 0x7fc3cc198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.188:55400/0 (socket says 172.21.15.188:55400) 2024-04-23T18:10:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.220+0000 7fc3ca59c700 1 -- 172.21.15.188:0/2254227312 learned_addr learned my addr 172.21.15.188:0/2254227312 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-23T18:10:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.220+0000 7fc3ca59c700 1 -- 172.21.15.188:0/2254227312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3cc199230 con 0x7fc3cc1015b0 2024-04-23T18:10:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.220+0000 7fc3ca59c700 1 --2- 172.21.15.188:0/2254227312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3cc1015b0 0x7fc3cc198cf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0056d0 tx=0x7fc3bc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.220+0000 7fc3bb7fe700 1 -- 172.21.15.188:0/2254227312 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3bc026070 con 0x7fc3cc1015b0 2024-04-23T18:10:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.220+0000 7fc3bb7fe700 1 -- 172.21.15.188:0/2254227312 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3bc00bba0 con 0x7fc3cc1015b0 2024-04-23T18:10:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.220+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/2254227312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3cc199520 con 0x7fc3cc1015b0 2024-04-23T18:10:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.221+0000 7fc3bb7fe700 1 -- 172.21.15.188:0/2254227312 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3bc003920 con 0x7fc3cc1015b0 2024-04-23T18:10:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.221+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/2254227312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3cc1999e0 con 0x7fc3cc1015b0 2024-04-23T18:10:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.221+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/2254227312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3cc102a90 con 0x7fc3cc1015b0 2024-04-23T18:10:13.216 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.222+0000 7fc3bb7fe700 1 -- 172.21.15.188:0/2254227312 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc3bc029070 con 0x7fc3cc1015b0 2024-04-23T18:10:13.217 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.222+0000 7fc3bb7fe700 1 --2- 172.21.15.188:0/2254227312 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3b40396b0 0x7fc3b403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:13.217 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.222+0000 7fc3bb7fe700 1 -- 172.21.15.188:0/2254227312 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc3bc01e070 con 0x7fc3cc1015b0 2024-04-23T18:10:13.217 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.222+0000 7fc3c9d9b700 1 --2- 172.21.15.188:0/2254227312 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3b40396b0 0x7fc3b403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:13.217 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.223+0000 7fc3c9d9b700 1 --2- 172.21.15.188:0/2254227312 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3b40396b0 0x7fc3b403bb70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0006fd0 tx=0x7fc3c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:13.221 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.228+0000 7fc3bb7fe700 1 -- 172.21.15.188:0/2254227312 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3bc013070 con 0x7fc3cc1015b0 2024-04-23T18:10:13.433 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.439+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/2254227312 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi050:172.21.15.50=a;smithi177:172.21.15.177=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}) -- 0x7fc3cc06aec0 con 0x7fc3b40396b0 2024-04-23T18:10:13.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.446+0000 7fc3bb7fe700 1 -- 172.21.15.188:0/2254227312 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc3cc06aec0 con 0x7fc3b40396b0 2024-04-23T18:10:13.440 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mon update... 2024-04-23T18:10:13.442 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.448+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/2254227312 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3b40396b0 msgr2=0x7fc3b403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:13.442 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.448+0000 7fc3d0e2b700 1 --2- 172.21.15.188:0/2254227312 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3b40396b0 0x7fc3b403bb70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0006fd0 tx=0x7fc3c0006e40 comp rx=0 tx=0).stop 2024-04-23T18:10:13.442 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.448+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/2254227312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3cc1015b0 msgr2=0x7fc3cc198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:13.442 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.448+0000 7fc3d0e2b700 1 --2- 172.21.15.188:0/2254227312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3cc1015b0 0x7fc3cc198cf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0056d0 tx=0x7fc3bc0057b0 comp rx=0 tx=0).stop 2024-04-23T18:10:13.442 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.448+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/2254227312 shutdown_connections 2024-04-23T18:10:13.442 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.448+0000 7fc3d0e2b700 1 --2- 172.21.15.188:0/2254227312 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3b40396b0 0x7fc3b403bb70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:13.442 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.448+0000 7fc3d0e2b700 1 --2- 172.21.15.188:0/2254227312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3cc1015b0 0x7fc3cc198cf0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:13.443 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.448+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/2254227312 >> 172.21.15.188:0/2254227312 conn(0x7fc3cc0fce60 msgr2=0x7fc3cc102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:13.443 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.448+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/2254227312 shutdown_connections 2024-04-23T18:10:13.443 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:13.448+0000 7fc3d0e2b700 1 -- 172.21.15.188:0/2254227312 wait complete. 2024-04-23T18:10:14.004 DEBUG:teuthology.orchestra.run.smithi177:mon.b> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.b.service 2024-04-23T18:10:14.007 DEBUG:teuthology.orchestra.run.smithi188:mon.c> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.c.service 2024-04-23T18:10:14.009 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T18:10:14.009 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T18:10:14.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:13 smithi050 ceph-mon[29187]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:14.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:14.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:14.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:14.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:14.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:14.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:14.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:14.795 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:15.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:14 smithi050 ceph-mon[29187]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi050:172.21.15.50=a;smithi177:172.21.15.177=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:15.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:14 smithi050 ceph-mon[29187]: Saving service mon spec with placement smithi050:172.21.15.50=a;smithi177:172.21.15.177=b;smithi188:172.21.15.188=c;count:3 2024-04-23T18:10:15.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:14 smithi050 ceph-mon[29187]: Deploying daemon mon.c on smithi188 2024-04-23T18:10:16.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:15 smithi050 ceph-mon[29187]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:17.883 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.887+0000 7f483458d700 1 -- 172.21.15.188:0/1982689463 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c101590 msgr2=0x7f482c101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:17.883 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.887+0000 7f483458d700 1 --2- 172.21.15.188:0/1982689463 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c101590 0x7f482c101970 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f481c009910 tx=0x7f481c009c20 comp rx=0 tx=0).stop 2024-04-23T18:10:17.883 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.887+0000 7f483458d700 1 -- 172.21.15.188:0/1982689463 shutdown_connections 2024-04-23T18:10:17.883 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.887+0000 7f483458d700 1 --2- 172.21.15.188:0/1982689463 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c101590 0x7f482c101970 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.887+0000 7f483458d700 1 -- 172.21.15.188:0/1982689463 >> 172.21.15.188:0/1982689463 conn(0x7f482c0fce40 msgr2=0x7f482c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.887+0000 7f483458d700 1 -- 172.21.15.188:0/1982689463 shutdown_connections 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.887+0000 7f483458d700 1 -- 172.21.15.188:0/1982689463 wait complete. 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.888+0000 7f483458d700 1 Processor -- start 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.888+0000 7f483458d700 1 -- start start 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.888+0000 7f483458d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c101590 0x7f482c196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.888+0000 7f483458d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f482c10b810 con 0x7f482c101590 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.888+0000 7f4832329700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c101590 0x7f482c196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.888+0000 7f4832329700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c101590 0x7f482c196d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.188:35900/0 (socket says 172.21.15.188:35900) 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.888+0000 7f4832329700 1 -- 172.21.15.188:0/2606132816 learned_addr learned my addr 172.21.15.188:0/2606132816 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.889+0000 7f4832329700 1 -- 172.21.15.188:0/2606132816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482c1972c0 con 0x7f482c101590 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.889+0000 7f4832329700 1 --2- 172.21.15.188:0/2606132816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c101590 0x7f482c196d80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f481c0056d0 tx=0x7f481c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:17.884 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.889+0000 7f48237fe700 1 -- 172.21.15.188:0/2606132816 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f481c026070 con 0x7f482c101590 2024-04-23T18:10:17.885 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.889+0000 7f483458d700 1 -- 172.21.15.188:0/2606132816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f482c1975b0 con 0x7f482c101590 2024-04-23T18:10:17.885 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.889+0000 7f48237fe700 1 -- 172.21.15.188:0/2606132816 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f481c00bba0 con 0x7f482c101590 2024-04-23T18:10:17.885 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.889+0000 7f48237fe700 1 -- 172.21.15.188:0/2606132816 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f481c003920 con 0x7f482c101590 2024-04-23T18:10:17.885 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.889+0000 7f483458d700 1 -- 172.21.15.188:0/2606132816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f482c19b650 con 0x7f482c101590 2024-04-23T18:10:17.886 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.890+0000 7f483458d700 1 -- 172.21.15.188:0/2606132816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f482c102a70 con 0x7f482c101590 2024-04-23T18:10:17.886 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.890+0000 7f48237fe700 1 -- 172.21.15.188:0/2606132816 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f481c029070 con 0x7f482c101590 2024-04-23T18:10:17.886 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.890+0000 7f48237fe700 1 --2- 172.21.15.188:0/2606132816 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4818039650 0x7f481803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:17.886 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.890+0000 7f48237fe700 1 -- 172.21.15.188:0/2606132816 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f481c01e070 con 0x7f482c101590 2024-04-23T18:10:17.886 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.890+0000 7f4831b28700 1 --2- 172.21.15.188:0/2606132816 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4818039650 0x7f481803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:17.886 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.891+0000 7f4831b28700 1 --2- 172.21.15.188:0/2606132816 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4818039650 0x7f481803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4828006fd0 tx=0x7f4828006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:17.890 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:17.896+0000 7f48237fe700 1 -- 172.21.15.188:0/2606132816 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f481c013070 con 0x7f482c101590 2024-04-23T18:10:18.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:17 smithi050 ceph-mon[29187]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:18.196 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:18 smithi188 systemd[1]: Starting Ceph mon.c for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:10:18.197 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:18.201+0000 7f483458d700 1 -- 172.21.15.188:0/2606132816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f482c1980e0 con 0x7f482c101590 2024-04-23T18:10:18.197 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:10:18.197 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","modified":"2024-04-23T18:08:34.626890Z","created":"2024-04-23T18:08:34.626890Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-23T18:10:18.197 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:18.202+0000 7f48237fe700 1 -- 172.21.15.188:0/2606132816 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f481c0282c0 con 0x7f482c101590 2024-04-23T18:10:18.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:18.205+0000 7f483458d700 1 -- 172.21.15.188:0/2606132816 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4818039650 msgr2=0x7f481803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:18.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:18.205+0000 7f483458d700 1 --2- 172.21.15.188:0/2606132816 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4818039650 0x7f481803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4828006fd0 tx=0x7f4828006e40 comp rx=0 tx=0).stop 2024-04-23T18:10:18.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:18.205+0000 7f483458d700 1 -- 172.21.15.188:0/2606132816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c101590 msgr2=0x7f482c196d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:18.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:18.205+0000 7f483458d700 1 --2- 172.21.15.188:0/2606132816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c101590 0x7f482c196d80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f481c0056d0 tx=0x7f481c0057b0 comp rx=0 tx=0).stop 2024-04-23T18:10:18.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:18.205+0000 7f483458d700 1 -- 172.21.15.188:0/2606132816 shutdown_connections 2024-04-23T18:10:18.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:18.205+0000 7f483458d700 1 --2- 172.21.15.188:0/2606132816 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f4818039650 0x7f481803bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:18.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:18.205+0000 7f483458d700 1 --2- 172.21.15.188:0/2606132816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c101590 0x7f482c196d80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:18.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:18.205+0000 7f483458d700 1 -- 172.21.15.188:0/2606132816 >> 172.21.15.188:0/2606132816 conn(0x7f482c0fce40 msgr2=0x7f482c102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:18.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:18.205+0000 7f483458d700 1 -- 172.21.15.188:0/2606132816 shutdown_connections 2024-04-23T18:10:18.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:18.205+0000 7f483458d700 1 -- 172.21.15.188:0/2606132816 wait complete. 2024-04-23T18:10:18.200 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-23T18:10:18.762 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:18 smithi188 podman[32674]: 2024-04-23 18:10:18.521580175 +0000 UTC m=+0.097566392 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:10:19.027 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:18 smithi188 podman[32674]: 2024-04-23 18:10:18.952306409 +0000 UTC m=+0.528292655 container create df051151edb949b3c073340ab59c88066970ae1322fb0a8190157a8d1d716997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:10:19.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:18 smithi050 ceph-mon[29187]: from='client.? 172.21.15.188:0/2606132816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:10:19.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 podman[32674]: 2024-04-23 18:10:19.137931066 +0000 UTC m=+0.713917273 container init df051151edb949b3c073340ab59c88066970ae1322fb0a8190157a8d1d716997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-23T18:10:19.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 podman[32674]: 2024-04-23 18:10:19.1411614 +0000 UTC m=+0.717147607 container start df051151edb949b3c073340ab59c88066970ae1322fb0a8190157a8d1d716997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-23T18:10:19.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T18:10:19.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-23T18:10:19.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: pidfile_write: ignore empty --pid-file 2024-04-23T18:10:19.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: load: jerasure load: lrc 2024-04-23T18:10:19.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: RocksDB version: 7.9.2 2024-04-23T18:10:19.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Git sha 0 2024-04-23T18:10:19.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-23T18:10:19.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: DB SUMMARY 2024-04-23T18:10:19.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: DB Session ID: J90DZM2PQMW31S1H9H5T 2024-04-23T18:10:19.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: CURRENT file: CURRENT 2024-04-23T18:10:19.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.error_if_exists: 0 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.create_if_missing: 0 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.paranoid_checks: 1 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.env: 0x56059ea54cc0 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.info_log: 0x5605a01e1be0 2024-04-23T18:10:19.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T18:10:19.334 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.statistics: (nil) 2024-04-23T18:10:19.334 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.use_fsync: 0 2024-04-23T18:10:19.334 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_log_file_size: 0 2024-04-23T18:10:19.335 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T18:10:19.335 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T18:10:19.335 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T18:10:19.335 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T18:10:19.335 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.allow_fallocate: 1 2024-04-23T18:10:19.335 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T18:10:19.335 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T18:10:19.336 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.use_direct_reads: 0 2024-04-23T18:10:19.336 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T18:10:19.336 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T18:10:19.336 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.db_log_dir: 2024-04-23T18:10:19.336 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.wal_dir: 2024-04-23T18:10:19.336 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T18:10:19.336 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T18:10:19.336 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T18:10:19.336 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T18:10:19.336 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.write_buffer_manager: 0x5605a01750e0 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T18:10:19.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.unordered_write: 0 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.row_cache: None 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.wal_filter: None 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.two_write_queues: 0 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.wal_compression: 0 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.atomic_flush: 0 2024-04-23T18:10:19.338 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T18:10:19.339 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T18:10:19.339 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T18:10:19.339 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.log_readahead_size: 0 2024-04-23T18:10:19.339 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T18:10:19.339 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T18:10:19.339 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T18:10:19.339 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T18:10:19.339 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T18:10:19.339 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T18:10:19.339 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_background_jobs: 2 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_background_compactions: -1 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_subcompactions: 1 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_open_files: -1 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T18:10:19.340 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T18:10:19.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T18:10:19.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T18:10:19.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_background_flushes: -1 2024-04-23T18:10:19.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Compression algorithms supported: 2024-04-23T18:10:19.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T18:10:19.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: kZSTD supported: 0 2024-04-23T18:10:19.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: kXpressCompression supported: 0 2024-04-23T18:10:19.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T18:10:19.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: kZlibCompression supported: 1 2024-04-23T18:10:19.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: kSnappyCompression supported: 1 2024-04-23T18:10:19.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: kLZ4Compression supported: 1 2024-04-23T18:10:19.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: kBZip2Compression supported: 0 2024-04-23T18:10:19.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T18:10:19.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T18:10:19.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-23T18:10:19.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T18:10:19.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T18:10:19.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.merge_operator: 2024-04-23T18:10:19.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_filter: None 2024-04-23T18:10:19.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T18:10:19.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T18:10:19.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T18:10:19.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T18:10:19.343 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5605a01e1dc0) 2024-04-23T18:10:19.343 INFO:journalctl@ceph.mon.c.smithi188.stdout: cache_index_and_filter_blocks: 1 2024-04-23T18:10:19.343 INFO:journalctl@ceph.mon.c.smithi188.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T18:10:19.343 INFO:journalctl@ceph.mon.c.smithi188.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T18:10:19.343 INFO:journalctl@ceph.mon.c.smithi188.stdout: pin_top_level_index_and_filter: 1 2024-04-23T18:10:19.343 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_type: 0 2024-04-23T18:10:19.343 INFO:journalctl@ceph.mon.c.smithi188.stdout: data_block_index_type: 0 2024-04-23T18:10:19.343 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_shortening: 1 2024-04-23T18:10:19.343 INFO:journalctl@ceph.mon.c.smithi188.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T18:10:19.343 INFO:journalctl@ceph.mon.c.smithi188.stdout: checksum: 4 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: no_block_cache: 0 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache: 0x5605a01a71f0 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_name: BinnedLRUCache 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_options: 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: capacity : 536870912 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: num_shard_bits : 4 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: strict_capacity_limit : 0 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: high_pri_pool_ratio: 0.000 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_compressed: (nil) 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: persistent_cache: (nil) 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_size: 4096 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_size_deviation: 10 2024-04-23T18:10:19.344 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_restart_interval: 16 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_block_restart_interval: 1 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: metadata_block_size: 4096 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: partition_filters: 0 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: use_delta_encoding: 1 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: filter_policy: bloomfilter 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: whole_key_filtering: 1 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: verify_compression: 0 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: read_amp_bytes_per_bit: 0 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: format_version: 5 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: enable_index_compression: 1 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_align: 0 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: max_auto_readahead_size: 262144 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: prepopulate_block_cache: 0 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: initial_auto_readahead_size: 8192 2024-04-23T18:10:19.345 INFO:journalctl@ceph.mon.c.smithi188.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T18:10:19.346 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T18:10:19.346 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T18:10:19.346 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compression: NoCompression 2024-04-23T18:10:19.346 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T18:10:19.346 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T18:10:19.346 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.num_levels: 7 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T18:10:19.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T18:10:19.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T18:10:19.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T18:10:19.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T18:10:19.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T18:10:19.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T18:10:19.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T18:10:19.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T18:10:19.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T18:10:19.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T18:10:19.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T18:10:19.350 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T18:10:19.350 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T18:10:19.350 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T18:10:19.350 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T18:10:19.350 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T18:10:19.350 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T18:10:19.350 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T18:10:19.350 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T18:10:19.350 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.inplace_update_support: 0 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T18:10:19.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.bloom_locality: 0 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.max_successive_merges: 0 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.ttl: 2592000 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.enable_blob_files: false 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.min_blob_size: 0 2024-04-23T18:10:19.352 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 06d79ad6-eaf2-4c6f-81d9-c21a52e73712 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895819171556, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895819172091, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713895819, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "06d79ad6-eaf2-4c6f-81d9-c21a52e73712", "db_session_id": "J90DZM2PQMW31S1H9H5T", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-23T18:10:19.353 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713895819172187, "job": 1, "event": "recovery_finished"} 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5605a02bc000 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: DB pointer 0x5605a02a6000 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** DB Stats ** 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T18:10:19.354 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T18:10:19.355 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-23T18:10:19.356 INFO:journalctl@ceph.mon.c.smithi188.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T18:10:19.356 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-23T18:10:19.356 INFO:journalctl@ceph.mon.c.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T18:10:19.356 INFO:journalctl@ceph.mon.c.smithi188.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T18:10:19.356 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T18:10:19.356 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T18:10:19.356 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T18:10:19.356 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T18:10:19.356 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T18:10:19.356 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T18:10:19.356 INFO:journalctl@ceph.mon.c.smithi188.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T18:10:19.357 INFO:journalctl@ceph.mon.c.smithi188.stdout: Block cache BinnedLRUCache@0x5605a01a71f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-23T18:10:19.357 INFO:journalctl@ceph.mon.c.smithi188.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-23T18:10:19.357 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-23T18:10:19.357 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T18:10:19.357 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-23T18:10:19.357 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: using public_addr v2:172.21.15.188:0/0 -> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2024-04-23T18:10:19.357 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: starting mon.c rank -1 at public addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] at bind addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:10:19.357 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(???) e0 preinit fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:10:19.357 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 bash[32674]: df051151edb949b3c073340ab59c88066970ae1322fb0a8190157a8d1d716997 2024-04-23T18:10:19.357 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).mds e1 new map 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout: e1 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout: legacy client fscid: -1 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout: No filesystems configured 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-23T18:10:19.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 systemd[1]: Started Ceph mon.c for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:19.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: Deploying cephadm binary to smithi188 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:19.360 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: Added host smithi188 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/806437517' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/806437517' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:19.362 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:19.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:19.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:19.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:19.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:19.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi050:172.21.15.50=a;smithi177:172.21.15.177=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:19.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: Saving service mon spec with placement smithi050:172.21.15.50=a;smithi177:172.21.15.177=b;smithi188:172.21.15.188=c;count:3 2024-04-23T18:10:19.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: Deploying daemon mon.c on smithi188 2024-04-23T18:10:19.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:19.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:19.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: from='client.? 172.21.15.188:0/2606132816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:10:19.363 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:19 smithi188 ceph-mon[32694]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-23T18:10:19.791 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T18:10:19.791 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T18:10:20.025 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:10:24.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.569+0000 7f667ec18700 1 -- 172.21.15.188:0/4208520283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66680035c0 msgr2=0x7f6668005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:24.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.569+0000 7f667ec18700 1 --2- 172.21.15.188:0/4208520283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66680035c0 0x7f6668005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6678078b70 tx=0x7f666c00feb0 comp rx=0 tx=0).stop 2024-04-23T18:10:24.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.569+0000 7f667ec18700 1 -- 172.21.15.188:0/4208520283 shutdown_connections 2024-04-23T18:10:24.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.569+0000 7f667ec18700 1 --2- 172.21.15.188:0/4208520283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66680035c0 0x7f6668005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:24.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.569+0000 7f667ec18700 1 -- 172.21.15.188:0/4208520283 >> 172.21.15.188:0/4208520283 conn(0x7f6678077880 msgr2=0x7f6678077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:24.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.569+0000 7f667ec18700 1 -- 172.21.15.188:0/4208520283 shutdown_connections 2024-04-23T18:10:24.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.569+0000 7f667ec18700 1 -- 172.21.15.188:0/4208520283 wait complete. 2024-04-23T18:10:24.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:24.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:24.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: mon.a calling monitor election 2024-04-23T18:10:24.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:24.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:24.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:24.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:24.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: mon.c calling monitor election 2024-04-23T18:10:24.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:24.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:24.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:24.565 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: monmap epoch 2 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: last_changed 2024-04-23T18:10:19.198145+0000 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: created 2024-04-23T18:08:34.626890+0000 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: min_mon_release 19 (squid) 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: election_strategy: 1 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: fsmap 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: mgrmap e13: x(active, since 49s) 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: overall HEALTH_OK 2024-04-23T18:10:24.566 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:24.567 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:24.567 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:24.567 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:24.567 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:24.569 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.570+0000 7f667ec18700 1 Processor -- start 2024-04-23T18:10:24.569 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.570+0000 7f667ec18700 1 -- start start 2024-04-23T18:10:24.569 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.570+0000 7f667ec18700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66680035c0 0x7f6678101df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:24.569 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.570+0000 7f667ec18700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6678102330 0x7f66781027b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:24.569 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.570+0000 7f667ec18700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f667810a480 con 0x7f66680035c0 2024-04-23T18:10:24.569 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.570+0000 7f667ec18700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f667810a300 con 0x7f6678102330 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.570+0000 7f6677fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6678102330 0x7f66781027b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.571+0000 7f6677fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6678102330 0x7f66781027b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:55382/0 (socket says 172.21.15.188:55382) 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.571+0000 7f667c9b4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66680035c0 0x7f6678101df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.571+0000 7f6677fff700 1 -- 172.21.15.188:0/763870339 learned_addr learned my addr 172.21.15.188:0/763870339 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.571+0000 7f6677fff700 1 -- 172.21.15.188:0/763870339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6678102330 msgr2=0x7f66781027b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.571+0000 7f6677fff700 1 -- 172.21.15.188:0/763870339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6678102330 msgr2=0x7f66781027b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.571+0000 7f6677fff700 1 --2- 172.21.15.188:0/763870339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6678102330 0x7f66781027b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.571+0000 7f6677fff700 1 --2- 172.21.15.188:0/763870339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6678102330 0x7f66781027b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.571+0000 7f667c9b4700 1 -- 172.21.15.188:0/763870339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6678102330 msgr2=0x7f66781027b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.571+0000 7f667c9b4700 1 --2- 172.21.15.188:0/763870339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6678102330 0x7f66781027b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.571+0000 7f667c9b4700 1 -- 172.21.15.188:0/763870339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6678103d10 con 0x7f66680035c0 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.571+0000 7f667c9b4700 1 --2- 172.21.15.188:0/763870339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66680035c0 0x7f6678101df0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f666c010010 tx=0x7f666c0050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:24.570 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.571+0000 7f6675ffb700 1 -- 172.21.15.188:0/763870339 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f666c004aa0 con 0x7f66680035c0 2024-04-23T18:10:24.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.572+0000 7f6675ffb700 1 -- 172.21.15.188:0/763870339 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f666c004030 con 0x7f66680035c0 2024-04-23T18:10:24.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.572+0000 7f6675ffb700 1 -- 172.21.15.188:0/763870339 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f666c011bd0 con 0x7f66680035c0 2024-04-23T18:10:24.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.572+0000 7f667ec18700 1 -- 172.21.15.188:0/763870339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66781a41a0 con 0x7f66680035c0 2024-04-23T18:10:24.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.572+0000 7f667ec18700 1 -- 172.21.15.188:0/763870339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66781a46e0 con 0x7f66680035c0 2024-04-23T18:10:24.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.573+0000 7f6675ffb700 1 -- 172.21.15.188:0/763870339 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f666c023070 con 0x7f66680035c0 2024-04-23T18:10:24.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.573+0000 7f6675ffb700 1 --2- 172.21.15.188:0/763870339 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f665c039440 0x7f665c03b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:24.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.573+0000 7f667ec18700 1 -- 172.21.15.188:0/763870339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f667806a9a0 con 0x7f66680035c0 2024-04-23T18:10:24.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.573+0000 7f6675ffb700 1 -- 172.21.15.188:0/763870339 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f666c020050 con 0x7f66680035c0 2024-04-23T18:10:24.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.573+0000 7f6677fff700 1 --2- 172.21.15.188:0/763870339 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f665c039440 0x7f665c03b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:24.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.574+0000 7f6677fff700 1 --2- 172.21.15.188:0/763870339 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f665c039440 0x7f665c03b900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6664009180 tx=0x7f6664008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:24.573 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.579+0000 7f6675ffb700 1 -- 172.21.15.188:0/763870339 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f666c028020 con 0x7f66680035c0 2024-04-23T18:10:24.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:24.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:24.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: mon.a calling monitor election 2024-04-23T18:10:24.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:24.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:24.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:24.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:24.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: mon.c calling monitor election 2024-04-23T18:10:24.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:24.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:24.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:24.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:24.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:10:24.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: monmap epoch 2 2024-04-23T18:10:24.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:10:24.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: last_changed 2024-04-23T18:10:19.198145+0000 2024-04-23T18:10:24.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: created 2024-04-23T18:08:34.626890+0000 2024-04-23T18:10:24.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: min_mon_release 19 (squid) 2024-04-23T18:10:24.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: election_strategy: 1 2024-04-23T18:10:24.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-23T18:10:24.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-23T18:10:24.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: fsmap 2024-04-23T18:10:24.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:10:24.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: mgrmap e13: x(active, since 49s) 2024-04-23T18:10:24.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: overall HEALTH_OK 2024-04-23T18:10:24.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:24.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:24.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:24.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:24.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:24.859 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.865+0000 7f667ec18700 1 -- 172.21.15.188:0/763870339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f667802df80 con 0x7f66680035c0 2024-04-23T18:10:24.859 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.865+0000 7f6675ffb700 1 -- 172.21.15.188:0/763870339 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f666c01a390 con 0x7f66680035c0 2024-04-23T18:10:24.860 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:10:24.860 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":2,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","modified":"2024-04-23T18:10:19.198145Z","created":"2024-04-23T18:08:34.626890Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T18:10:24.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.867+0000 7f667ec18700 1 -- 172.21.15.188:0/763870339 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f665c039440 msgr2=0x7f665c03b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:24.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.867+0000 7f667ec18700 1 --2- 172.21.15.188:0/763870339 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f665c039440 0x7f665c03b900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6664009180 tx=0x7f6664008040 comp rx=0 tx=0).stop 2024-04-23T18:10:24.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.867+0000 7f667ec18700 1 -- 172.21.15.188:0/763870339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66680035c0 msgr2=0x7f6678101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:24.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.867+0000 7f667ec18700 1 --2- 172.21.15.188:0/763870339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66680035c0 0x7f6678101df0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f666c010010 tx=0x7f666c0050d0 comp rx=0 tx=0).stop 2024-04-23T18:10:24.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.868+0000 7f667ec18700 1 -- 172.21.15.188:0/763870339 shutdown_connections 2024-04-23T18:10:24.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.868+0000 7f667ec18700 1 --2- 172.21.15.188:0/763870339 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f665c039440 0x7f665c03b900 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:24.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.868+0000 7f667ec18700 1 --2- 172.21.15.188:0/763870339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66680035c0 0x7f6678101df0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:24.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.868+0000 7f667ec18700 1 --2- 172.21.15.188:0/763870339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6678102330 0x7f66781027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:24.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.868+0000 7f667ec18700 1 -- 172.21.15.188:0/763870339 >> 172.21.15.188:0/763870339 conn(0x7f6678077880 msgr2=0x7f66780fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:24.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.868+0000 7f667ec18700 1 -- 172.21.15.188:0/763870339 shutdown_connections 2024-04-23T18:10:24.862 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:24.868+0000 7f667ec18700 1 -- 172.21.15.188:0/763870339 wait complete. 2024-04-23T18:10:24.863 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 2 2024-04-23T18:10:25.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:25 smithi188 ceph-mon[32694]: Deploying daemon mon.b on smithi177 2024-04-23T18:10:25.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:25 smithi188 ceph-mon[32694]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:25.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:25 smithi188 ceph-mon[32694]: from='client.? 172.21.15.188:0/763870339' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:10:25.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:25 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:25.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:25 smithi050 ceph-mon[29187]: Deploying daemon mon.b on smithi177 2024-04-23T18:10:25.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:25 smithi050 ceph-mon[29187]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:25.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:25 smithi050 ceph-mon[29187]: from='client.? 172.21.15.188:0/763870339' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T18:10:25.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:25 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:26.432 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T18:10:26.433 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T18:10:26.674 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:10:27.812 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:27 smithi188 ceph-mon[32694]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:27.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:27 smithi050 ceph-mon[29187]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:28.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:28 smithi177 systemd[1]: Started Ceph mon.b for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:10:33.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: mon.a calling monitor election 2024-04-23T18:10:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: mon.c calling monitor election 2024-04-23T18:10:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:33.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:10:33.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: monmap epoch 3 2024-04-23T18:10:33.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:10:33.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: last_changed 2024-04-23T18:10:28.147831+0000 2024-04-23T18:10:33.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: created 2024-04-23T18:08:34.626890+0000 2024-04-23T18:10:33.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: min_mon_release 19 (squid) 2024-04-23T18:10:33.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: election_strategy: 1 2024-04-23T18:10:33.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-23T18:10:33.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-23T18:10:33.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: 2: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-04-23T18:10:33.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: fsmap 2024-04-23T18:10:33.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:10:33.583 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: mgrmap e13: x(active, since 58s) 2024-04-23T18:10:33.583 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: overall HEALTH_OK 2024-04-23T18:10:33.583 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:33.583 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:33.583 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:33.583 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:33.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:33.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: mon.a calling monitor election 2024-04-23T18:10:33.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: mon.c calling monitor election 2024-04-23T18:10:33.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:33.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:33.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:33.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:33.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:33.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:33.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:33.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:33.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: monmap epoch 3 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: last_changed 2024-04-23T18:10:28.147831+0000 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: created 2024-04-23T18:08:34.626890+0000 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: min_mon_release 19 (squid) 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: election_strategy: 1 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: 2: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: fsmap 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: mgrmap e13: x(active, since 58s) 2024-04-23T18:10:33.697 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: overall HEALTH_OK 2024-04-23T18:10:33.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:33.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:33.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:33.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:34.151 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.157+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2507801147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6eac0062a0 msgr2=0x7f6eac006680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:34.151 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.157+0000 7f6ec6cd2700 1 --2- 172.21.15.188:0/2507801147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6eac0062a0 0x7f6eac006680 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6ec01024f0 tx=0x7f6eb80095d0 comp rx=0 tx=0).stop 2024-04-23T18:10:34.151 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.157+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2507801147 shutdown_connections 2024-04-23T18:10:34.151 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.157+0000 7f6ec6cd2700 1 --2- 172.21.15.188:0/2507801147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6eac0062a0 0x7f6eac006680 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:34.151 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.157+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2507801147 >> 172.21.15.188:0/2507801147 conn(0x7f6ec00fcf20 msgr2=0x7f6ec00ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:34.152 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.157+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2507801147 shutdown_connections 2024-04-23T18:10:34.152 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.157+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2507801147 wait complete. 2024-04-23T18:10:34.154 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec6cd2700 1 Processor -- start 2024-04-23T18:10:34.154 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec6cd2700 1 -- start start 2024-04-23T18:10:34.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec6cd2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6eac0062a0 0x7f6ec0197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:34.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec6cd2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ec01976b0 0x7f6ec019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:34.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec6cd2700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ec019bf80 0x7f6ec019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:34.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec6cd2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec0109320 con 0x7f6ec01976b0 2024-04-23T18:10:34.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec6cd2700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec01091a0 con 0x7f6ec019bf80 2024-04-23T18:10:34.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec6cd2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec01094a0 con 0x7f6eac0062a0 2024-04-23T18:10:34.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec4a6e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6eac0062a0 0x7f6ec0197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:34.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec4a6e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6eac0062a0 0x7f6ec0197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:55384/0 (socket says 172.21.15.188:55384) 2024-04-23T18:10:34.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec4a6e700 1 -- 172.21.15.188:0/2938509984 learned_addr learned my addr 172.21.15.188:0/2938509984 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-23T18:10:34.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ebffff700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ec01976b0 0x7f6ec019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:34.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec4a6e700 1 -- 172.21.15.188:0/2938509984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6eac0062a0 msgr2=0x7f6ec0197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-23T18:10:34.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec4a6e700 1 -- 172.21.15.188:0/2938509984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6eac0062a0 msgr2=0x7f6ec0197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T18:10:34.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec4a6e700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6eac0062a0 0x7f6ec0197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T18:10:34.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec4a6e700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6eac0062a0 0x7f6ec0197170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:10:34.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.158+0000 7f6ec526f700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ec019bf80 0x7f6ec019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:34.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ec526f700 1 -- 172.21.15.188:0/2938509984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ec019bf80 msgr2=0x7f6ec019c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-23T18:10:34.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ec526f700 1 -- 172.21.15.188:0/2938509984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ec019bf80 msgr2=0x7f6ec019c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T18:10:34.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ec526f700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ec019bf80 0x7f6ec019c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T18:10:34.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ebffff700 1 -- 172.21.15.188:0/2938509984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6eac0062a0 msgr2=0x7f6ec0197170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:10:34.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ec526f700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ec019bf80 0x7f6ec019c430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:10:34.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ebffff700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6eac0062a0 0x7f6ec0197170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:34.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ebffff700 1 -- 172.21.15.188:0/2938509984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ec019bf80 msgr2=0x7f6ec019c430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:10:34.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ebffff700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ec019bf80 0x7f6ec019c430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:34.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ebffff700 1 -- 172.21.15.188:0/2938509984 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec01a3cf0 con 0x7f6ec01976b0 2024-04-23T18:10:34.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ebffff700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ec01976b0 0x7f6ec019ba40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000b550 tx=0x7f6eb000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:34.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ebdffb700 1 -- 172.21.15.188:0/2938509984 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0004230 con 0x7f6ec01976b0 2024-04-23T18:10:34.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ebdffb700 1 -- 172.21.15.188:0/2938509984 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6eb00043d0 con 0x7f6ec01976b0 2024-04-23T18:10:34.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2938509984 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ec01a3ec0 con 0x7f6ec01976b0 2024-04-23T18:10:34.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ebdffb700 1 -- 172.21.15.188:0/2938509984 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0011b90 con 0x7f6ec01976b0 2024-04-23T18:10:34.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.159+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2938509984 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ec01a43a0 con 0x7f6ec01976b0 2024-04-23T18:10:34.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.160+0000 7f6ebdffb700 1 -- 172.21.15.188:0/2938509984 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6eb001a400 con 0x7f6ec01976b0 2024-04-23T18:10:34.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.160+0000 7f6ebdffb700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ea0039ae0 0x7f6ea003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:34.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.160+0000 7f6ebdffb700 1 -- 172.21.15.188:0/2938509984 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6eb004c420 con 0x7f6ec01976b0 2024-04-23T18:10:34.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.160+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2938509984 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ec00640a0 con 0x7f6ec01976b0 2024-04-23T18:10:34.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.161+0000 7f6ec4a6e700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ea0039ae0 0x7f6ea003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:34.158 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.161+0000 7f6ec4a6e700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ea0039ae0 0x7f6ea003bfa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6eb80062a0 tx=0x7f6eb8006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:34.160 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.166+0000 7f6ebdffb700 1 -- 172.21.15.188:0/2938509984 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6eb0018510 con 0x7f6ec01976b0 2024-04-23T18:10:34.450 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.456+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2938509984 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6ec01a4660 con 0x7f6ec01976b0 2024-04-23T18:10:34.450 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:34 smithi188 ceph-mon[32694]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:34.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:34 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:34.451 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:34 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:34.451 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.456+0000 7f6ebdffb700 1 -- 172.21.15.188:0/2938509984 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f6eb0018770 con 0x7f6ec01976b0 2024-04-23T18:10:34.451 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-23T18:10:34.451 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":3,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","modified":"2024-04-23T18:10:28.147831Z","created":"2024-04-23T18:08:34.626890Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T18:10:34.453 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.459+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2938509984 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ea0039ae0 msgr2=0x7f6ea003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:34.453 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.459+0000 7f6ec6cd2700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ea0039ae0 0x7f6ea003bfa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6eb80062a0 tx=0x7f6eb8006210 comp rx=0 tx=0).stop 2024-04-23T18:10:34.453 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.459+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2938509984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ec01976b0 msgr2=0x7f6ec019ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:34.453 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.459+0000 7f6ec6cd2700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ec01976b0 0x7f6ec019ba40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000b550 tx=0x7f6eb000b910 comp rx=0 tx=0).stop 2024-04-23T18:10:34.453 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.459+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2938509984 shutdown_connections 2024-04-23T18:10:34.453 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.459+0000 7f6ec6cd2700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ea0039ae0 0x7f6ea003bfa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:34.454 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.459+0000 7f6ec6cd2700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ec019bf80 0x7f6ec019c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:34.454 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.459+0000 7f6ec6cd2700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6eac0062a0 0x7f6ec0197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:34.454 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.459+0000 7f6ec6cd2700 1 --2- 172.21.15.188:0/2938509984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ec01976b0 0x7f6ec019ba40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:34.454 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.459+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2938509984 >> 172.21.15.188:0/2938509984 conn(0x7f6ec00fcf20 msgr2=0x7f6ec0102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:34.454 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.459+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2938509984 shutdown_connections 2024-04-23T18:10:34.454 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:34.459+0000 7f6ec6cd2700 1 -- 172.21.15.188:0/2938509984 wait complete. 2024-04-23T18:10:34.454 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 3 2024-04-23T18:10:34.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:34 smithi050 ceph-mon[29187]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:34.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:34.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:35.291 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-23T18:10:35.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-23T18:10:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:10:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: mon.a calling monitor election 2024-04-23T18:10:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: mon.c calling monitor election 2024-04-23T18:10:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:10:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: monmap epoch 3 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: last_changed 2024-04-23T18:10:28.147831+0000 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: created 2024-04-23T18:08:34.626890+0000 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: min_mon_release 19 (squid) 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: election_strategy: 1 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: 2: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: fsmap 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: mgrmap e13: x(active, since 58s) 2024-04-23T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: overall HEALTH_OK 2024-04-23T18:10:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:35.555 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:10:35.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: mon.b calling monitor election 2024-04-23T18:10:35.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:35.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: Updating smithi050:/etc/ceph/ceph.conf 2024-04-23T18:10:35.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-23T18:10:35.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-23T18:10:35.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: Updating smithi050:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:35.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:35.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: mon.b calling monitor election 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: mon.a calling monitor election 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: mon.c calling monitor election 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: monmap epoch 3 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: last_changed 2024-04-23T18:10:28.147831+0000 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: created 2024-04-23T18:08:34.626890+0000 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: min_mon_release 19 (squid) 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: election_strategy: 1 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: 2: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-04-23T18:10:35.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: fsmap 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: mgrmap e13: x(active, since 60s) 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: overall HEALTH_OK 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:35.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:35.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:10:35.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:36.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: mon.b calling monitor election 2024-04-23T18:10:36.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:36.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: Updating smithi050:/etc/ceph/ceph.conf 2024-04-23T18:10:36.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-23T18:10:36.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: Updating smithi050:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: mon.b calling monitor election 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: mon.a calling monitor election 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: mon.c calling monitor election 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: monmap epoch 3 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: last_changed 2024-04-23T18:10:28.147831+0000 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: created 2024-04-23T18:08:34.626890+0000 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: min_mon_release 19 (squid) 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: election_strategy: 1 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-23T18:10:36.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: 2: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: fsmap 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: mgrmap e13: x(active, since 60s) 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: overall HEALTH_OK 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.042 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.042 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:36.042 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:10:36.042 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: mon.b calling monitor election 2024-04-23T18:10:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: Updating smithi050:/etc/ceph/ceph.conf 2024-04-23T18:10:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-23T18:10:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-23T18:10:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: Updating smithi050:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:10:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: mon.b calling monitor election 2024-04-23T18:10:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: mon.a calling monitor election 2024-04-23T18:10:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: mon.c calling monitor election 2024-04-23T18:10:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: monmap epoch 3 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: last_changed 2024-04-23T18:10:28.147831+0000 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: created 2024-04-23T18:08:34.626890+0000 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: min_mon_release 19 (squid) 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: election_strategy: 1 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: 2: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: fsmap 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: mgrmap e13: x(active, since 60s) 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: overall HEALTH_OK 2024-04-23T18:10:36.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:10:36.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:36.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:36 smithi050 ceph-mon[29187]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T18:10:36.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:36 smithi050 ceph-mon[29187]: Reconfiguring daemon mon.a on smithi050 2024-04-23T18:10:36.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:36 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:37.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:36 smithi177 ceph-mon[32308]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T18:10:37.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:36 smithi177 ceph-mon[32308]: Reconfiguring daemon mon.a on smithi050 2024-04-23T18:10:37.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:36 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:37.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:36 smithi188 ceph-mon[32694]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T18:10:37.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:36 smithi188 ceph-mon[32694]: Reconfiguring daemon mon.a on smithi050 2024-04-23T18:10:37.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:36 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:10:37.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:37 smithi050 ceph-mon[29187]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:37.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:37.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:37.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:37.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:10:37.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:37.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:37 smithi177 ceph-mon[32308]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:37.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:37.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:37.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:37.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:10:37.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:38.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.056+0000 7f393b9e5700 1 -- 172.21.15.50:0/1181226647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934103450 msgr2=0x7f393410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:38.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.056+0000 7f393b9e5700 1 --2- 172.21.15.50:0/1181226647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934103450 0x7f393410f8b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f392c00b0a0 tx=0x7f392c00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:10:38.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.056+0000 7f393b9e5700 1 -- 172.21.15.50:0/1181226647 shutdown_connections 2024-04-23T18:10:38.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.056+0000 7f393b9e5700 1 --2- 172.21.15.50:0/1181226647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934103450 0x7f393410f8b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:38.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.056+0000 7f393b9e5700 1 --2- 172.21.15.50:0/1181226647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3934108a50 0x7f3934108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:38.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.056+0000 7f393b9e5700 1 --2- 172.21.15.50:0/1181226647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934102a90 0x7f3934102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:38.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.056+0000 7f393b9e5700 1 -- 172.21.15.50:0/1181226647 >> 172.21.15.50:0/1181226647 conn(0x7f3934079ba0 msgr2=0x7f3934079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:38.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.056+0000 7f393b9e5700 1 -- 172.21.15.50:0/1181226647 shutdown_connections 2024-04-23T18:10:38.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.056+0000 7f393b9e5700 1 -- 172.21.15.50:0/1181226647 wait complete. 2024-04-23T18:10:38.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.056+0000 7f393b9e5700 1 Processor -- start 2024-04-23T18:10:38.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f393b9e5700 1 -- start start 2024-04-23T18:10:38.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f393b9e5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934102a90 0x7f3934074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:38.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f393b9e5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934103450 0x7f3934075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:38.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f393b9e5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3934108a50 0x7f39340757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:38.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f393b9e5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3934112be0 con 0x7f3934102a90 2024-04-23T18:10:38.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f393b9e5700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3934112a60 con 0x7f3934108a50 2024-04-23T18:10:38.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f393b9e5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3934112d60 con 0x7f3934103450 2024-04-23T18:10:38.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f3939781700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934102a90 0x7f3934074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:38.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f3939781700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934102a90 0x7f3934074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35414/0 (socket says 172.21.15.50:35414) 2024-04-23T18:10:38.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f3939781700 1 -- 172.21.15.50:0/3037175679 learned_addr learned my addr 172.21.15.50:0/3037175679 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:10:38.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f3939f82700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3934108a50 0x7f39340757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:38.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f3939781700 1 -- 172.21.15.50:0/3037175679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934103450 msgr2=0x7f3934075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:38.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f3938f80700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934103450 0x7f3934075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:38.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f3939781700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934103450 0x7f3934075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:38.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f3939781700 1 -- 172.21.15.50:0/3037175679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3934108a50 msgr2=0x7f39340757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:38.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f3939781700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3934108a50 0x7f39340757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:38.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f3939781700 1 -- 172.21.15.50:0/3037175679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39341a7580 con 0x7f3934102a90 2024-04-23T18:10:38.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.057+0000 7f3939781700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934102a90 0x7f3934074d40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f392400d980 tx=0x7f392400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:38.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.058+0000 7f392a7fc700 1 -- 172.21.15.50:0/3037175679 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3924016070 con 0x7f3934102a90 2024-04-23T18:10:38.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.058+0000 7f392a7fc700 1 -- 172.21.15.50:0/3037175679 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3924004830 con 0x7f3934102a90 2024-04-23T18:10:38.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.058+0000 7f393b9e5700 1 -- 172.21.15.50:0/3037175679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39341a7870 con 0x7f3934102a90 2024-04-23T18:10:38.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.058+0000 7f392a7fc700 1 -- 172.21.15.50:0/3037175679 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39240054c0 con 0x7f3934102a90 2024-04-23T18:10:38.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.058+0000 7f393b9e5700 1 -- 172.21.15.50:0/3037175679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39341a7e00 con 0x7f3934102a90 2024-04-23T18:10:38.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.059+0000 7f392a7fc700 1 -- 172.21.15.50:0/3037175679 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3924004220 con 0x7f3934102a90 2024-04-23T18:10:38.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.059+0000 7f393b9e5700 1 -- 172.21.15.50:0/3037175679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3934069730 con 0x7f3934102a90 2024-04-23T18:10:38.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.059+0000 7f392a7fc700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3920039890 0x7f392003bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:38.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.059+0000 7f392a7fc700 1 -- 172.21.15.50:0/3037175679 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f392404c6e0 con 0x7f3934102a90 2024-04-23T18:10:38.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.059+0000 7f3938f80700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3920039890 0x7f392003bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:38.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.059+0000 7f3938f80700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3920039890 0x7f392003bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f393407a130 tx=0x7f3930005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:38.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.064+0000 7f392a7fc700 1 -- 172.21.15.50:0/3037175679 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f392401f020 con 0x7f3934102a90 2024-04-23T18:10:38.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:37 smithi188 ceph-mon[32694]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:38.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:38.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:38.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:38.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:10:38.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:38.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.275+0000 7f393b9e5700 1 -- 172.21.15.50:0/3037175679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f393402df80 con 0x7f3934102a90 2024-04-23T18:10:38.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.275+0000 7f392a7fc700 1 -- 172.21.15.50:0/3037175679 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f3924008230 con 0x7f3934102a90 2024-04-23T18:10:38.270 INFO:teuthology.orchestra.run.smithi050.stdout:# minimal ceph.conf for 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:10:38.270 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2024-04-23T18:10:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:10:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: mon_host = [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2024-04-23T18:10:38.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.277+0000 7f393b9e5700 1 -- 172.21.15.50:0/3037175679 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3920039890 msgr2=0x7f392003bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:38.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.277+0000 7f393b9e5700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3920039890 0x7f392003bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f393407a130 tx=0x7f3930005d50 comp rx=0 tx=0).stop 2024-04-23T18:10:38.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.277+0000 7f393b9e5700 1 -- 172.21.15.50:0/3037175679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934102a90 msgr2=0x7f3934074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:38.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.277+0000 7f393b9e5700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934102a90 0x7f3934074d40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f392400d980 tx=0x7f392400dc90 comp rx=0 tx=0).stop 2024-04-23T18:10:38.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.277+0000 7f393b9e5700 1 -- 172.21.15.50:0/3037175679 shutdown_connections 2024-04-23T18:10:38.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.277+0000 7f393b9e5700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3920039890 0x7f392003bd50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:38.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.277+0000 7f393b9e5700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3934108a50 0x7f39340757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:38.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.277+0000 7f393b9e5700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934102a90 0x7f3934074d40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:38.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.277+0000 7f393b9e5700 1 --2- 172.21.15.50:0/3037175679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934103450 0x7f3934075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:38.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.277+0000 7f393b9e5700 1 -- 172.21.15.50:0/3037175679 >> 172.21.15.50:0/3037175679 conn(0x7f3934079ba0 msgr2=0x7f3934101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:38.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.277+0000 7f393b9e5700 1 -- 172.21.15.50:0/3037175679 shutdown_connections 2024-04-23T18:10:38.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:38.277+0000 7f393b9e5700 1 -- 172.21.15.50:0/3037175679 wait complete. 2024-04-23T18:10:38.815 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:38 smithi050 ceph-mon[29187]: Reconfiguring mon.b (monmap changed)... 2024-04-23T18:10:38.815 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:38 smithi050 ceph-mon[29187]: Reconfiguring daemon mon.b on smithi177 2024-04-23T18:10:38.815 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:38 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3037175679' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:38.843 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-23T18:10:38.843 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:10:38.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T18:10:38.871 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:10:38.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:10:38.936 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-23T18:10:38.936 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T18:10:38.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:38 smithi177 ceph-mon[32308]: Reconfiguring mon.b (monmap changed)... 2024-04-23T18:10:38.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:38 smithi177 ceph-mon[32308]: Reconfiguring daemon mon.b on smithi177 2024-04-23T18:10:38.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:38 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3037175679' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:38.963 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-23T18:10:38.963 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:10:39.027 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T18:10:39.027 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T18:10:39.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:38 smithi188 ceph-mon[32694]: Reconfiguring mon.b (monmap changed)... 2024-04-23T18:10:39.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:38 smithi188 ceph-mon[32694]: Reconfiguring daemon mon.b on smithi177 2024-04-23T18:10:39.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:38 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3037175679' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:39.053 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T18:10:39.053 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:10:39.117 INFO:tasks.cephadm:Adding mgr.x on smithi050 2024-04-23T18:10:39.117 INFO:tasks.cephadm:Adding mgr.y on smithi177 2024-04-23T18:10:39.117 INFO:tasks.cephadm:Adding mgr.z on smithi188 2024-04-23T18:10:39.117 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi050=x;smithi177=y;smithi188=z' 2024-04-23T18:10:39.911 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:10:40.017 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:39 smithi188 ceph-mon[32694]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:40.018 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:40.018 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:40.018 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:39 smithi188 ceph-mon[32694]: Reconfiguring mon.c (monmap changed)... 2024-04-23T18:10:40.018 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:40.018 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:10:40.018 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:40.019 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:39 smithi188 ceph-mon[32694]: Reconfiguring daemon mon.c on smithi188 2024-04-23T18:10:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:39 smithi050 ceph-mon[29187]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:40.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:40.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:40.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:39 smithi050 ceph-mon[29187]: Reconfiguring mon.c (monmap changed)... 2024-04-23T18:10:40.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:40.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:10:40.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:40.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:39 smithi050 ceph-mon[29187]: Reconfiguring daemon mon.c on smithi188 2024-04-23T18:10:40.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:39 smithi177 ceph-mon[32308]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:40.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:40.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:40.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:39 smithi177 ceph-mon[32308]: Reconfiguring mon.c (monmap changed)... 2024-04-23T18:10:40.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T18:10:40.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T18:10:40.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:40.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:39 smithi177 ceph-mon[32308]: Reconfiguring daemon mon.c on smithi188 2024-04-23T18:10:41.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.652+0000 7f45b7d4c700 1 -- 172.21.15.188:0/395654343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b0104640 msgr2=0x7f45b010aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:41.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.652+0000 7f45b7d4c700 1 --2- 172.21.15.188:0/395654343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b0104640 0x7f45b010aef0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f45a800b0a0 tx=0x7f45a800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:10:41.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.652+0000 7f45b7d4c700 1 -- 172.21.15.188:0/395654343 shutdown_connections 2024-04-23T18:10:41.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.652+0000 7f45b7d4c700 1 --2- 172.21.15.188:0/395654343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b0104640 0x7f45b010aef0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.652+0000 7f45b7d4c700 1 --2- 172.21.15.188:0/395654343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b0102aa0 0x7f45b0102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.652+0000 7f45b7d4c700 1 --2- 172.21.15.188:0/395654343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b0103ca0 0x7f45b0104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.652+0000 7f45b7d4c700 1 -- 172.21.15.188:0/395654343 >> 172.21.15.188:0/395654343 conn(0x7f45b00fe040 msgr2=0x7f45b0100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.652+0000 7f45b7d4c700 1 -- 172.21.15.188:0/395654343 shutdown_connections 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.652+0000 7f45b7d4c700 1 -- 172.21.15.188:0/395654343 wait complete. 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.652+0000 7f45b7d4c700 1 Processor -- start 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.652+0000 7f45b7d4c700 1 -- start start 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b7d4c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b0102aa0 0x7f45b00767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b7d4c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b0103ca0 0x7f45b0074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b7d4c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b0104640 0x7f45b0075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b7d4c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f45b010e590 con 0x7f45b0104640 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b7d4c700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f45b010e410 con 0x7f45b0102aa0 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b7d4c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f45b010e710 con 0x7f45b0103ca0 2024-04-23T18:10:41.649 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b52e7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b0103ca0 0x7f45b0074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:41.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b52e7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b0103ca0 0x7f45b0074e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:57834/0 (socket says 172.21.15.188:57834) 2024-04-23T18:10:41.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b52e7700 1 -- 172.21.15.188:0/641514092 learned_addr learned my addr 172.21.15.188:0/641514092 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-23T18:10:41.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b62e9700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b0104640 0x7f45b0075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:41.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b5ae8700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b0102aa0 0x7f45b00767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:41.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b52e7700 1 -- 172.21.15.188:0/641514092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b0102aa0 msgr2=0x7f45b00767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:41.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b52e7700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b0102aa0 0x7f45b00767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:41.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b52e7700 1 -- 172.21.15.188:0/641514092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b0104640 msgr2=0x7f45b0075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:41.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b52e7700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b0104640 0x7f45b0075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:41.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b52e7700 1 -- 172.21.15.188:0/641514092 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b0075c20 con 0x7f45b0103ca0 2024-04-23T18:10:41.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b5ae8700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b0102aa0 0x7f45b00767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:10:41.651 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b52e7700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b0103ca0 0x7f45b0074e30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f45a000e9a0 tx=0x7f45a000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:41.651 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45a6ffd700 1 -- 172.21.15.188:0/641514092 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a000cbd0 con 0x7f45b0103ca0 2024-04-23T18:10:41.651 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b7d4c700 1 -- 172.21.15.188:0/641514092 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b01a2d60 con 0x7f45b0103ca0 2024-04-23T18:10:41.651 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45a6ffd700 1 -- 172.21.15.188:0/641514092 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a000cd70 con 0x7f45b0103ca0 2024-04-23T18:10:41.651 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45a6ffd700 1 -- 172.21.15.188:0/641514092 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a0018680 con 0x7f45b0103ca0 2024-04-23T18:10:41.651 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.653+0000 7f45b7d4c700 1 -- 172.21.15.188:0/641514092 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b01a3240 con 0x7f45b0103ca0 2024-04-23T18:10:41.651 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.654+0000 7f45b7d4c700 1 -- 172.21.15.188:0/641514092 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45b0069730 con 0x7f45b0103ca0 2024-04-23T18:10:41.652 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.655+0000 7f45a6ffd700 1 -- 172.21.15.188:0/641514092 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f45a0010430 con 0x7f45b0103ca0 2024-04-23T18:10:41.652 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.655+0000 7f45a6ffd700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f459c039890 0x7f459c03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:41.652 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.655+0000 7f45a6ffd700 1 -- 172.21.15.188:0/641514092 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f45a004cc00 con 0x7f45b0103ca0 2024-04-23T18:10:41.652 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.655+0000 7f45b5ae8700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f459c039890 0x7f459c03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:41.652 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.656+0000 7f45b5ae8700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f459c039890 0x7f459c03bd50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f45ac005950 tx=0x7f45ac0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:41.654 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.660+0000 7f45a6ffd700 1 -- 172.21.15.188:0/641514092 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45a0053080 con 0x7f45b0103ca0 2024-04-23T18:10:41.868 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.874+0000 7f45b7d4c700 1 -- 172.21.15.188:0/641514092 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi050=x;smithi177=y;smithi188=z", "target": ["mon-mgr", ""]}) -- 0x7f45b0062e80 con 0x7f459c039890 2024-04-23T18:10:41.873 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.879+0000 7f45a6ffd700 1 -- 172.21.15.188:0/641514092 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f45b0062e80 con 0x7f459c039890 2024-04-23T18:10:41.873 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mgr update... 2024-04-23T18:10:41.875 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.881+0000 7f45b7d4c700 1 -- 172.21.15.188:0/641514092 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f459c039890 msgr2=0x7f459c03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:41.875 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.881+0000 7f45b7d4c700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f459c039890 0x7f459c03bd50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f45ac005950 tx=0x7f45ac0058e0 comp rx=0 tx=0).stop 2024-04-23T18:10:41.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.881+0000 7f45b7d4c700 1 -- 172.21.15.188:0/641514092 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b0103ca0 msgr2=0x7f45b0074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:41.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.881+0000 7f45b7d4c700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b0103ca0 0x7f45b0074e30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f45a000e9a0 tx=0x7f45a000ecb0 comp rx=0 tx=0).stop 2024-04-23T18:10:41.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.881+0000 7f45b7d4c700 1 -- 172.21.15.188:0/641514092 shutdown_connections 2024-04-23T18:10:41.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.881+0000 7f45b7d4c700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b0104640 0x7f45b0075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:41.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.881+0000 7f45b7d4c700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f459c039890 0x7f459c03bd50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:41.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.881+0000 7f45b7d4c700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b0102aa0 0x7f45b00767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:41.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.881+0000 7f45b7d4c700 1 --2- 172.21.15.188:0/641514092 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b0103ca0 0x7f45b0074e30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:41.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.881+0000 7f45b7d4c700 1 -- 172.21.15.188:0/641514092 >> 172.21.15.188:0/641514092 conn(0x7f45b00fe040 msgr2=0x7f45b01070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:41.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.882+0000 7f45b7d4c700 1 -- 172.21.15.188:0/641514092 shutdown_connections 2024-04-23T18:10:41.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:10:41.882+0000 7f45b7d4c700 1 -- 172.21.15.188:0/641514092 wait complete. 2024-04-23T18:10:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:10:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:10:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:10:42.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:42.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:10:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:10:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:10:42.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:10:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:10:42.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:10:42.540 DEBUG:teuthology.orchestra.run.smithi177:mgr.y> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.y.service 2024-04-23T18:10:42.544 DEBUG:teuthology.orchestra.run.smithi188:mgr.z> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.z.service 2024-04-23T18:10:42.547 INFO:tasks.cephadm:Deploying OSDs... 2024-04-23T18:10:42.547 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:10:42.547 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T18:10:42.567 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T18:10:42.567 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2024-04-23T18:10:42.625 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T18:10:42.625 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:10:42.626 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-23T18:10:42.626 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:10:42.626 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:10:42.626 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-23 18:09:44.179038191 +0000 2024-04-23T18:10:42.626 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-23 18:04:26.714485382 +0000 2024-04-23T18:10:42.626 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-23 18:04:26.714485382 +0000 2024-04-23T18:10:42.626 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-04-23 18:04:26.714485382 +0000 2024-04-23T18:10:42.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T18:10:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-23T18:10:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-23T18:10:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000128455 s, 4.0 MB/s 2024-04-23T18:10:42.694 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T18:10:42.753 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2024-04-23T18:10:42.811 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T18:10:42.811 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:10:42.811 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-23T18:10:42.811 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:10:42.811 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:10:42.811 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-23 18:09:44.229036973 +0000 2024-04-23T18:10:42.811 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-23 18:04:27.047477281 +0000 2024-04-23T18:10:42.811 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-23 18:04:27.047477281 +0000 2024-04-23T18:10:42.811 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-04-23 18:04:27.047477281 +0000 2024-04-23T18:10:42.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T18:10:42.877 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-23T18:10:42.877 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-23T18:10:42.877 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000122759 s, 4.2 MB/s 2024-04-23T18:10:42.879 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T18:10:42.937 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2024-04-23T18:10:42.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:42 smithi050 ceph-mon[29187]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi050=x;smithi177=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:42.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:42 smithi050 ceph-mon[29187]: Saving service mgr spec with placement smithi050=x;smithi177=y;smithi188=z;count:3 2024-04-23T18:10:42.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:42 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:42.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:42 smithi050 ceph-mon[29187]: Deploying daemon mgr.z on smithi188 2024-04-23T18:10:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T18:10:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:10:42.962 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-23T18:10:42.962 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:10:42.962 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:10:42.962 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-23 18:09:44.281035707 +0000 2024-04-23T18:10:42.962 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-23 18:04:27.371469399 +0000 2024-04-23T18:10:42.962 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-23 18:04:27.371469399 +0000 2024-04-23T18:10:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-04-23 18:04:27.371469399 +0000 2024-04-23T18:10:42.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T18:10:43.025 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-23T18:10:43.025 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-23T18:10:43.025 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000106568 s, 4.8 MB/s 2024-04-23T18:10:43.030 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T18:10:43.088 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2024-04-23T18:10:43.145 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T18:10:43.145 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:10:43.145 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-23T18:10:43.145 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:10:43.145 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:10:43.146 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-23 18:09:44.329034538 +0000 2024-04-23T18:10:43.146 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-23 18:04:27.713461079 +0000 2024-04-23T18:10:43.146 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-23 18:04:27.713461079 +0000 2024-04-23T18:10:43.146 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-04-23 18:04:27.713461079 +0000 2024-04-23T18:10:43.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T18:10:43.208 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-23T18:10:43.209 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-23T18:10:43.209 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000107213 s, 4.8 MB/s 2024-04-23T18:10:43.210 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T18:10:43.267 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-23T18:10:43.267 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T18:10:43.286 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T18:10:43.287 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-04-23T18:10:43.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:42 smithi177 ceph-mon[32308]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi050=x;smithi177=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:43.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:42 smithi177 ceph-mon[32308]: Saving service mgr spec with placement smithi050=x;smithi177=y;smithi188=z;count:3 2024-04-23T18:10:43.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:42 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:43.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:42 smithi177 ceph-mon[32308]: Deploying daemon mgr.z on smithi188 2024-04-23T18:10:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T18:10:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:10:43.306 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-23T18:10:43.306 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:10:43.306 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:10:43.306 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-23 18:10:27.612460415 +0000 2024-04-23T18:10:43.306 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-23 18:04:51.570465902 +0000 2024-04-23T18:10:43.306 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-23 18:04:51.570465902 +0000 2024-04-23T18:10:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-23 18:04:51.570465902 +0000 2024-04-23T18:10:43.307 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T18:10:43.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:42 smithi188 ceph-mon[32694]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi050=x;smithi177=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:43.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:42 smithi188 ceph-mon[32694]: Saving service mgr spec with placement smithi050=x;smithi177=y;smithi188=z;count:3 2024-04-23T18:10:43.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:42 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:43.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:42 smithi188 ceph-mon[32694]: Deploying daemon mgr.z on smithi188 2024-04-23T18:10:43.371 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-23T18:10:43.371 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-23T18:10:43.371 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000303198 s, 1.7 MB/s 2024-04-23T18:10:43.372 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T18:10:43.430 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-04-23T18:10:43.487 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T18:10:43.487 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:10:43.487 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-23T18:10:43.487 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:10:43.487 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:10:43.487 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-23 18:10:27.612460415 +0000 2024-04-23T18:10:43.487 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-23 18:04:51.913456385 +0000 2024-04-23T18:10:43.487 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-23 18:04:51.913456385 +0000 2024-04-23T18:10:43.488 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-23 18:04:51.913456385 +0000 2024-04-23T18:10:43.488 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T18:10:43.551 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-23T18:10:43.551 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-23T18:10:43.551 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000240575 s, 2.1 MB/s 2024-04-23T18:10:43.552 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T18:10:43.609 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-04-23T18:10:43.665 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T18:10:43.665 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:10:43.665 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-23T18:10:43.665 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:10:43.665 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:10:43.665 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-23 18:10:27.612460415 +0000 2024-04-23T18:10:43.665 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-23 18:04:52.257446840 +0000 2024-04-23T18:10:43.666 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-23 18:04:52.257446840 +0000 2024-04-23T18:10:43.666 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-23 18:04:52.257446840 +0000 2024-04-23T18:10:43.666 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T18:10:43.729 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-23T18:10:43.729 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-23T18:10:43.729 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000217074 s, 2.4 MB/s 2024-04-23T18:10:43.730 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T18:10:43.787 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-04-23T18:10:43.842 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T18:10:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:10:43.843 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-23T18:10:43.843 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:10:43.843 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:10:43.843 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-23 18:10:27.613460388 +0000 2024-04-23T18:10:43.843 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-23 18:04:52.599437351 +0000 2024-04-23T18:10:43.843 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-23 18:04:52.599437351 +0000 2024-04-23T18:10:43.843 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-23 18:04:52.599437351 +0000 2024-04-23T18:10:43.843 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T18:10:43.908 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-23T18:10:43.910 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-23T18:10:43.910 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000200056 s, 2.6 MB/s 2024-04-23T18:10:43.911 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T18:10:43.968 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-23T18:10:43.968 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T18:10:43.984 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T18:10:43.984 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2024-04-23T18:10:44.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:43 smithi177 ceph-mon[32308]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:44.040 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T18:10:44.040 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:10:44.040 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-23T18:10:44.040 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:10:44.040 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:10:44.040 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-23 18:10:43.899045331 +0000 2024-04-23T18:10:44.041 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-23 18:04:47.028928272 +0000 2024-04-23T18:10:44.041 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-23 18:04:47.028928272 +0000 2024-04-23T18:10:44.041 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-23 18:04:47.028928272 +0000 2024-04-23T18:10:44.041 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T18:10:44.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:43 smithi188 ceph-mon[32694]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:44.081 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:43 smithi188 systemd[1]: Starting Ceph mgr.z for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:10:44.105 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-23T18:10:44.106 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-23T18:10:44.106 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000264467 s, 1.9 MB/s 2024-04-23T18:10:44.107 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T18:10:44.162 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2024-04-23T18:10:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:43 smithi050 ceph-mon[29187]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:44.218 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T18:10:44.218 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:10:44.218 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-23T18:10:44.218 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:10:44.218 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:10:44.218 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-23 18:10:43.899045331 +0000 2024-04-23T18:10:44.218 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-23 18:04:47.375922091 +0000 2024-04-23T18:10:44.218 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-23 18:04:47.375922091 +0000 2024-04-23T18:10:44.219 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-23 18:04:47.375922091 +0000 2024-04-23T18:10:44.219 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T18:10:44.280 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-23T18:10:44.280 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-23T18:10:44.280 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000269021 s, 1.9 MB/s 2024-04-23T18:10:44.281 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T18:10:44.338 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2024-04-23T18:10:44.358 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:44 smithi188 podman[34367]: 2024-04-23 18:10:44.081829232 +0000 UTC m=+0.017949528 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:10:44.358 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:44 smithi188 podman[34367]: 2024-04-23 18:10:44.220741268 +0000 UTC m=+0.156861560 container create 315dbf4c4ee4fdd584a4e4432871e5008e152a1afe3b14c4da4e4aa040fbb8f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:10:44.372 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T18:10:44.372 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:10:44.372 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-23T18:10:44.372 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:10:44.372 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:10:44.372 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-23 18:10:43.899045331 +0000 2024-04-23T18:10:44.372 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-23 18:04:47.708916160 +0000 2024-04-23T18:10:44.372 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-23 18:04:47.708916160 +0000 2024-04-23T18:10:44.373 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-23 18:04:47.708916160 +0000 2024-04-23T18:10:44.373 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T18:10:44.434 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-23T18:10:44.434 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-23T18:10:44.434 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000191326 s, 2.7 MB/s 2024-04-23T18:10:44.435 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T18:10:44.493 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2024-04-23T18:10:44.551 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T18:10:44.551 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T18:10:44.551 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-23T18:10:44.551 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:10:44.551 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T18:10:44.551 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-23 18:10:43.900045313 +0000 2024-04-23T18:10:44.552 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-23 18:04:48.038910282 +0000 2024-04-23T18:10:44.552 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-23 18:04:48.038910282 +0000 2024-04-23T18:10:44.552 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-23 18:04:48.038910282 +0000 2024-04-23T18:10:44.552 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T18:10:44.610 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:44 smithi188 podman[34367]: 2024-04-23 18:10:44.457104518 +0000 UTC m=+0.393224808 container init 315dbf4c4ee4fdd584a4e4432871e5008e152a1afe3b14c4da4e4aa040fbb8f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-04-23T18:10:44.610 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:44 smithi188 podman[34367]: 2024-04-23 18:10:44.45975495 +0000 UTC m=+0.395875249 container start 315dbf4c4ee4fdd584a4e4432871e5008e152a1afe3b14c4da4e4aa040fbb8f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-23T18:10:44.610 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:44 smithi188 ceph-mgr[34510]: -- 172.21.15.188:0/3639182424 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b2397c85a0 con 0x55b23979cc00 2024-04-23T18:10:44.610 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:44 smithi188 bash[34367]: 315dbf4c4ee4fdd584a4e4432871e5008e152a1afe3b14c4da4e4aa040fbb8f8 2024-04-23T18:10:44.610 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:44 smithi188 systemd[1]: Started Ceph mgr.z for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:10:44.615 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-23T18:10:44.616 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-23T18:10:44.616 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000198625 s, 2.6 MB/s 2024-04-23T18:10:44.618 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T18:10:44.675 INFO:tasks.cephadm:Deploying osd.0 on smithi050 with /dev/vg_nvme/lv_4... 2024-04-23T18:10:44.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T18:10:44.897 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:44 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:44.615+0000 7fd713c7c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:10:44.897 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:44 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:44.802+0000 7fd713c7c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:10:44.897 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:44 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:44.903+0000 7fd713c7c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:10:44.926 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:10:45.580 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:45 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:45.210+0000 7fd713c7c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:10:45.580 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:45 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:45.294+0000 7fd713c7c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:10:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:45 smithi050 ceph-mon[29187]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:45 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:45 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:45.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:45 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:45 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:10:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:45 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:10:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:45 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:10:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:45 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:45.915 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:45 smithi050 ceph-mon[29187]: Deploying daemon mgr.y on smithi177 2024-04-23T18:10:45.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:45 smithi188 ceph-mon[32694]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:45.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:45 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:45.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:45 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:45.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:45 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:45.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:45 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:10:45.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:45 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:10:45.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:45 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:10:45.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:45 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:45.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:45 smithi188 ceph-mon[32694]: Deploying daemon mgr.y on smithi177 2024-04-23T18:10:46.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:45 smithi177 ceph-mon[32308]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:46.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:45 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:46.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:45 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:46.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:45 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:46.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:45 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:10:46.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:45 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T18:10:46.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:45 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:10:46.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:45 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:46.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:45 smithi177 ceph-mon[32308]: Deploying daemon mgr.y on smithi177 2024-04-23T18:10:46.783 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T18:10:46.784 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T18:10:46.784 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-23T18:10:46.784 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-23T18:10:46.784 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00843352 s, 1.2 GB/s 2024-04-23T18:10:46.784 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-23T18:10:46.788 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:46 smithi177 systemd[1]: Starting Ceph mgr.y for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:10:46.830 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:46.519+0000 7fd713c7c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:10:46.830 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:46.689+0000 7fd713c7c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:10:47.071 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:46 smithi177 podman[33714]: 2024-04-23 18:10:46.799843068 +0000 UTC m=+0.109646549 container create 0f041e72f6eecf5a5511d1d4b228b21719b0e407848854282f703c6e3098ddfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:10:47.071 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:46 smithi177 podman[33714]: 2024-04-23 18:10:46.708144042 +0000 UTC m=+0.017947477 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:10:47.071 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:47 smithi177 podman[33714]: 2024-04-23 18:10:47.002271978 +0000 UTC m=+0.312075417 container init 0f041e72f6eecf5a5511d1d4b228b21719b0e407848854282f703c6e3098ddfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True) 2024-04-23T18:10:47.071 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:47 smithi177 podman[33714]: 2024-04-23 18:10:47.004750338 +0000 UTC m=+0.314553773 container start 0f041e72f6eecf5a5511d1d4b228b21719b0e407848854282f703c6e3098ddfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-23T18:10:47.071 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:47 smithi177 ceph-mgr[33733]: -- 172.21.15.177:0/3618068482 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ae7d0465a0 con 0x55ae7d01ac00 2024-04-23T18:10:47.071 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:47 smithi177 bash[33714]: 0f041e72f6eecf5a5511d1d4b228b21719b0e407848854282f703c6e3098ddfc 2024-04-23T18:10:47.144 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:46.861+0000 7fd713c7c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:10:47.144 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:47 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:47.148+0000 7fd713c7c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:10:47.339 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:47 smithi177 systemd[1]: Started Ceph mgr.y for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:10:47.339 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:47 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:47.156+0000 7f8d4ebe5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:10:47.339 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:47 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:47.343+0000 7f8d4ebe5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:10:47.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_4 2024-04-23T18:10:47.449 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:47 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:47.241+0000 7fd713c7c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:10:47.449 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:47 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:47.454+0000 7fd713c7c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:10:47.647 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:10:47.771 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:47 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:47.452+0000 7f8d4ebe5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:10:47.772 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:47 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:47.776+0000 7f8d4ebe5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:10:47.830 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:47 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:47.650+0000 7fd713c7c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:10:48.038 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:47 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:47.873+0000 7f8d4ebe5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:10:48.330 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:48 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:48.277+0000 7fd713c7c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:10:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:48 smithi188 ceph-mon[32694]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:48.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:48 smithi050 ceph-mon[29187]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:48.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:48.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:48.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:48.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:48.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:48.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:48 smithi177 ceph-mon[32308]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:48.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:48.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:48.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:48.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:48.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:48.994 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:48 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:48.705+0000 7fd713c7c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:10:48.994 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:48 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:48.802+0000 7fd713c7c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:10:48.995 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:48.999+0000 7fd713c7c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:10:49.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.197+0000 7fd47ab97700 1 -- 172.21.15.50:0/387463151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4741040b0 msgr2=0x7fd4741044d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:49.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.197+0000 7fd47ab97700 1 --2- 172.21.15.50:0/387463151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4741040b0 0x7fd4741044d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd468009990 tx=0x7fd468009ca0 comp rx=0 tx=0).stop 2024-04-23T18:10:49.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.197+0000 7fd47ab97700 1 -- 172.21.15.50:0/387463151 shutdown_connections 2024-04-23T18:10:49.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.197+0000 7fd47ab97700 1 --2- 172.21.15.50:0/387463151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4741014d0 0x7fd47410b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:49.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.197+0000 7fd47ab97700 1 --2- 172.21.15.50:0/387463151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4741040b0 0x7fd4741044d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:49.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.197+0000 7fd47ab97700 1 --2- 172.21.15.50:0/387463151 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd474100b30 0x7fd474100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:49.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.197+0000 7fd47ab97700 1 -- 172.21.15.50:0/387463151 >> 172.21.15.50:0/387463151 conn(0x7fd4740fc6d0 msgr2=0x7fd4740feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:10:49.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.197+0000 7fd47ab97700 1 -- 172.21.15.50:0/387463151 shutdown_connections 2024-04-23T18:10:49.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.197+0000 7fd47ab97700 1 -- 172.21.15.50:0/387463151 wait complete. 2024-04-23T18:10:49.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.198+0000 7fd47ab97700 1 Processor -- start 2024-04-23T18:10:49.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.198+0000 7fd47ab97700 1 -- start start 2024-04-23T18:10:49.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.198+0000 7fd47ab97700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd474100b30 0x7fd4741037b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:49.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.198+0000 7fd47ab97700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4741014d0 0x7fd474101e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:49.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.198+0000 7fd47ab97700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4741040b0 0x7fd474102340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:49.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.198+0000 7fd47ab97700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd47410e4f0 con 0x7fd4741040b0 2024-04-23T18:10:49.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.198+0000 7fd47ab97700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd47410e370 con 0x7fd4741014d0 2024-04-23T18:10:49.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.198+0000 7fd47ab97700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd47410e670 con 0x7fd474100b30 2024-04-23T18:10:49.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd479134700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4741040b0 0x7fd474102340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:49.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd479134700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4741040b0 0x7fd474102340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42620/0 (socket says 172.21.15.50:42620) 2024-04-23T18:10:49.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd479134700 1 -- 172.21.15.50:0/1012717439 learned_addr learned my addr 172.21.15.50:0/1012717439 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:10:49.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd478933700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd474100b30 0x7fd4741037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:49.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd473fff700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4741014d0 0x7fd474101e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:49.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd479134700 1 -- 172.21.15.50:0/1012717439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd474100b30 msgr2=0x7fd4741037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:49.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd479134700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd474100b30 0x7fd4741037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:49.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd479134700 1 -- 172.21.15.50:0/1012717439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4741014d0 msgr2=0x7fd474101e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:10:49.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd479134700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4741014d0 0x7fd474101e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:10:49.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd479134700 1 -- 172.21.15.50:0/1012717439 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd474102bf0 con 0x7fd4741040b0 2024-04-23T18:10:49.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd479134700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4741040b0 0x7fd474102340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00ef30 tx=0x7fd46c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:49.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd471ffb700 1 -- 172.21.15.50:0/1012717439 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c0092e0 con 0x7fd4741040b0 2024-04-23T18:10:49.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd471ffb700 1 -- 172.21.15.50:0/1012717439 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd46c004bb0 con 0x7fd4741040b0 2024-04-23T18:10:49.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd47ab97700 1 -- 172.21.15.50:0/1012717439 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4741a2cd0 con 0x7fd4741040b0 2024-04-23T18:10:49.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.199+0000 7fd471ffb700 1 -- 172.21.15.50:0/1012717439 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c004020 con 0x7fd4741040b0 2024-04-23T18:10:49.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.200+0000 7fd47ab97700 1 -- 172.21.15.50:0/1012717439 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4741a3190 con 0x7fd4741040b0 2024-04-23T18:10:49.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.200+0000 7fd471ffb700 1 -- 172.21.15.50:0/1012717439 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd46c01b030 con 0x7fd4741040b0 2024-04-23T18:10:49.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.200+0000 7fd471ffb700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd464039890 0x7fd46403bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:10:49.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.201+0000 7fd471ffb700 1 -- 172.21.15.50:0/1012717439 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd46c04c500 con 0x7fd4741040b0 2024-04-23T18:10:49.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.201+0000 7fd478933700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd464039890 0x7fd46403bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:10:49.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.201+0000 7fd47ab97700 1 -- 172.21.15.50:0/1012717439 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd47402df80 con 0x7fd4741040b0 2024-04-23T18:10:49.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.201+0000 7fd478933700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd464039890 0x7fd46403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd46800b2e0 tx=0x7fd468005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:10:49.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.206+0000 7fd471ffb700 1 -- 172.21.15.50:0/1012717439 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd46c022020 con 0x7fd4741040b0 2024-04-23T18:10:49.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:49.435+0000 7fd47ab97700 1 -- 172.21.15.50:0/1012717439 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd474062e80 con 0x7fd464039890 2024-04-23T18:10:49.434 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:49.099+0000 7f8d4ebe5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:10:49.435 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:49.269+0000 7f8d4ebe5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:10:49.435 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:49.439+0000 7f8d4ebe5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:10:49.722 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:49.722 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: Reconfiguring daemon mgr.x on smithi050 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:49.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:49.724 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:49.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:49.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:49.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T18:10:49.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:10:49.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:10:49.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:49.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: Reconfiguring daemon mgr.x on smithi050 2024-04-23T18:10:49.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:49.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:49.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:49.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.830 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:49.516+0000 7fd713c7c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:10:49.830 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:49.606+0000 7fd713c7c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:10:49.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:49.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:49.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:49.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:49.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T18:10:49.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T18:10:49.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T18:10:49.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:49.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: Reconfiguring daemon mgr.x on smithi050 2024-04-23T18:10:49.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:10:49.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:10:49.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:49.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:50.016 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:49.727+0000 7f8d4ebe5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:10:50.017 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:49.818+0000 7f8d4ebe5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:10:50.017 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:50 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:50.021+0000 7f8d4ebe5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:10:50.172 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:49.843+0000 7fd713c7c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:10:50.172 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:49 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:49.972+0000 7fd713c7c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:10:50.173 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:50 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:50.064+0000 7fd713c7c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:10:50.173 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:50 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:50.177+0000 7fd713c7c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:10:50.288 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:50 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:50.213+0000 7f8d4ebe5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:10:50.830 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:50 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:50.478+0000 7fd713c7c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:10:50.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:50 smithi188 ceph-mon[32694]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:50.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:50 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:50.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:50 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:50.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:50 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:50.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:50 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:50.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:50 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:50.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:50 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:50.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:50 smithi050 ceph-mon[29187]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:50.934 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:50 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:50.934 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:50 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:50.934 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:50 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:50.934 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:50 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:50.934 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:50 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:50.934 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:50 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:51.038 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:50 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:50.811+0000 7f8d4ebe5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:10:51.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:50 smithi177 ceph-mon[32308]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:10:51.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:50 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:51.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:50 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:51.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:50 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:10:51.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:50 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:10:51.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:50 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:10:51.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:50 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:10:51.331 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:51 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:51.074+0000 7fd713c7c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:10:51.331 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:10:51 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:10:51.291+0000 7fd713c7c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:10:51.519 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:51 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:51.230+0000 7f8d4ebe5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:10:51.520 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:51 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:51.324+0000 7f8d4ebe5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:10:51.520 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:51 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:51.524+0000 7f8d4ebe5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:10:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:51.628+0000 7fd471ffb700 1 -- 172.21.15.50:0/1012717439 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7fd46c012050 con 0x7fd4741040b0 2024-04-23T18:10:51.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:51 smithi177 ceph-mon[32308]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:51.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:51 smithi177 ceph-mon[32308]: Standby manager daemon z started 2024-04-23T18:10:51.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:51 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.188:0/3749948948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:10:51.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:51 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.188:0/3749948948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:10:51.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:51 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.188:0/3749948948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:10:51.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:51 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.188:0/3749948948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:10:51.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:51 smithi050 ceph-mon[29187]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:51.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:51 smithi050 ceph-mon[29187]: Standby manager daemon z started 2024-04-23T18:10:51.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:51 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.188:0/3749948948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:10:51.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:51 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.188:0/3749948948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:10:51.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:51 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.188:0/3749948948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:10:51.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:51 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.188:0/3749948948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:10:52.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:51 smithi188 ceph-mon[32694]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:52.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:51 smithi188 ceph-mon[32694]: Standby manager daemon z started 2024-04-23T18:10:52.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:51 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.188:0/3749948948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:10:52.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:51 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.188:0/3749948948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:10:52.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:51 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.188:0/3749948948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:10:52.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:51 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.188:0/3749948948' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:10:52.368 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:52 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:52.045+0000 7f8d4ebe5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:10:52.368 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:52 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:52.135+0000 7f8d4ebe5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:10:52.629 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:52 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:52.373+0000 7f8d4ebe5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:10:52.629 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:52 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:52.506+0000 7f8d4ebe5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:10:52.629 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:52 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:52.600+0000 7f8d4ebe5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:10:52.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:52 smithi050 ceph-mon[29187]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-23T18:10:52.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:10:53.018 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:52 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:52.715+0000 7f8d4ebe5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:10:53.018 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:52 smithi177 ceph-mon[32308]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-23T18:10:53.018 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:10:53.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:52 smithi188 ceph-mon[32694]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-23T18:10:53.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:10:53.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:53 smithi050 ceph-mon[29187]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:53.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:53 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/798820094' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3cb7067-1f57-4e79-a884-bfae449d5508"}]: dispatch 2024-04-23T18:10:53.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:53 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/798820094' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3cb7067-1f57-4e79-a884-bfae449d5508"}]': finished 2024-04-23T18:10:53.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:53 smithi050 ceph-mon[29187]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T18:10:53.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:53 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:10:53.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:53 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3516105378' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:54.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:53 smithi177 ceph-mon[32308]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:54.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:53 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/798820094' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3cb7067-1f57-4e79-a884-bfae449d5508"}]: dispatch 2024-04-23T18:10:54.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:53 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/798820094' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3cb7067-1f57-4e79-a884-bfae449d5508"}]': finished 2024-04-23T18:10:54.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:53 smithi177 ceph-mon[32308]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T18:10:54.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:53 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:10:54.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:53 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3516105378' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:54.039 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:53 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:53.023+0000 7f8d4ebe5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:10:54.039 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:53 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:53.619+0000 7f8d4ebe5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:10:54.039 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:10:53 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:10:53.832+0000 7f8d4ebe5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:10:54.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:53 smithi188 ceph-mon[32694]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:54.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:53 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/798820094' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3cb7067-1f57-4e79-a884-bfae449d5508"}]: dispatch 2024-04-23T18:10:54.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:53 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/798820094' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3cb7067-1f57-4e79-a884-bfae449d5508"}]': finished 2024-04-23T18:10:54.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:53 smithi188 ceph-mon[32694]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T18:10:54.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:53 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:10:54.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:53 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3516105378' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:10:54.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:10:54.648+0000 7fd471ffb700 1 -- 172.21.15.50:0/1012717439 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fd46c018c10 con 0x7fd4741040b0 2024-04-23T18:10:54.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:54 smithi050 ceph-mon[29187]: Standby manager daemon y started 2024-04-23T18:10:54.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:54 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.177:0/1618442969' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:10:54.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:54 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.177:0/1618442969' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:10:54.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:54 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.177:0/1618442969' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:10:54.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:54 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.177:0/1618442969' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:10:55.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:54 smithi177 ceph-mon[32308]: Standby manager daemon y started 2024-04-23T18:10:55.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:54 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.177:0/1618442969' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:10:55.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:54 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.177:0/1618442969' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:10:55.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:54 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.177:0/1618442969' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:10:55.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:54 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.177:0/1618442969' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:10:55.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:54 smithi188 ceph-mon[32694]: Standby manager daemon y started 2024-04-23T18:10:55.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:54 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.177:0/1618442969' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:10:55.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:54 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.177:0/1618442969' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:10:55.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:54 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.177:0/1618442969' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:10:55.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:54 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.177:0/1618442969' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:10:55.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:55 smithi050 ceph-mon[29187]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:55.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:55 smithi050 ceph-mon[29187]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-23T18:10:55.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:55 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:10:56.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:55 smithi177 ceph-mon[32308]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:56.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:55 smithi177 ceph-mon[32308]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-23T18:10:56.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:55 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:10:56.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:55 smithi188 ceph-mon[32694]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:56.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:55 smithi188 ceph-mon[32694]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-23T18:10:56.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:55 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:10:57.916 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:57 smithi050 ceph-mon[29187]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:58.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:57 smithi177 ceph-mon[32308]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:58.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:57 smithi188 ceph-mon[32694]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:10:59.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:10:59 smithi050 ceph-mon[29187]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:00.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:10:59 smithi177 ceph-mon[32308]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:00.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:10:59 smithi188 ceph-mon[32694]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:00.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:00 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T18:11:00.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:00 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:00.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:00 smithi050 ceph-mon[29187]: Deploying daemon osd.0 on smithi050 2024-04-23T18:11:01.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:00 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T18:11:01.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:00 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:01.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:00 smithi177 ceph-mon[32308]: Deploying daemon osd.0 on smithi050 2024-04-23T18:11:01.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:00 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T18:11:01.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:00 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:01.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:00 smithi188 ceph-mon[32694]: Deploying daemon osd.0 on smithi050 2024-04-23T18:11:01.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:01 smithi050 ceph-mon[29187]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:02.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:01 smithi177 ceph-mon[32308]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:02.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:01 smithi188 ceph-mon[32694]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:03.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:03 smithi050 ceph-mon[29187]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:04.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:03 smithi177 ceph-mon[32308]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:04.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:03 smithi188 ceph-mon[32694]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:05.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:05 smithi050 ceph-mon[29187]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:06.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:05 smithi177 ceph-mon[32308]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:06.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:05 smithi188 ceph-mon[32694]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:06.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:06.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:06.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:07.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:07.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:07.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:07.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:07.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:07.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:07.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:07 smithi050 ceph-mon[29187]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:08.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:07 smithi177 ceph-mon[32308]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:08.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:07 smithi188 ceph-mon[32694]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.393+0000 7fd471ffb700 1 -- 172.21.15.50:0/1012717439 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd474062e80 con 0x7fd464039890 2024-04-23T18:11:09.388 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 0 on host 'smithi050' 2024-04-23T18:11:09.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.395+0000 7fd47ab97700 1 -- 172.21.15.50:0/1012717439 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd464039890 msgr2=0x7fd46403bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:09.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.395+0000 7fd47ab97700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd464039890 0x7fd46403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd46800b2e0 tx=0x7fd468005fb0 comp rx=0 tx=0).stop 2024-04-23T18:11:09.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.395+0000 7fd47ab97700 1 -- 172.21.15.50:0/1012717439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4741040b0 msgr2=0x7fd474102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.395+0000 7fd47ab97700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4741040b0 0x7fd474102340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00ef30 tx=0x7fd46c00c3a0 comp rx=0 tx=0).stop 2024-04-23T18:11:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.396+0000 7fd47ab97700 1 -- 172.21.15.50:0/1012717439 shutdown_connections 2024-04-23T18:11:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.396+0000 7fd47ab97700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4741040b0 0x7fd474102340 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.396+0000 7fd47ab97700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd474100b30 0x7fd4741037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.396+0000 7fd47ab97700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd464039890 0x7fd46403bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.396+0000 7fd47ab97700 1 --2- 172.21.15.50:0/1012717439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4741014d0 0x7fd474101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.396+0000 7fd47ab97700 1 -- 172.21.15.50:0/1012717439 >> 172.21.15.50:0/1012717439 conn(0x7fd4740fc6d0 msgr2=0x7fd474107270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.396+0000 7fd47ab97700 1 -- 172.21.15.50:0/1012717439 shutdown_connections 2024-04-23T18:11:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:11:09.396+0000 7fd47ab97700 1 -- 172.21.15.50:0/1012717439 wait complete. 2024-04-23T18:11:09.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:09 smithi050 ceph-mon[29187]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:09.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:09 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:09 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:09 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:09.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:09 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:09.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:09 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.589 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:09 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.589 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:09 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.589 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:09 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:09.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:09 smithi177 ceph-mon[32308]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:09.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:09 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:09 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:09 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:09.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:09 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:09.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:09 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:09 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:09 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:09 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:09.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:09 smithi188 ceph-mon[32694]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:09.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:09 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:09 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:09 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:09.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:09 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:09.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:09 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:09 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:09 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:09.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:09 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:10.159 DEBUG:teuthology.orchestra.run.smithi050:osd.0> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.0.service 2024-04-23T18:11:10.162 INFO:tasks.cephadm:Deploying osd.1 on smithi177 with /dev/vg_nvme/lv_4... 2024-04-23T18:11:10.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T18:11:10.404 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:11:10.771 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:10 smithi177 ceph-mon[32308]: from='osd.0 [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T18:11:10.771 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:10 smithi177 ceph-mon[32308]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T18:11:10.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:10 smithi188 ceph-mon[32694]: from='osd.0 [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T18:11:10.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:10 smithi188 ceph-mon[32694]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T18:11:10.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:10 smithi050 ceph-mon[29187]: from='osd.0 [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T18:11:10.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:10 smithi050 ceph-mon[29187]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T18:11:11.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:11 smithi050 ceph-mon[29187]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:11.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:11 smithi050 ceph-mon[29187]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T18:11:11.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:11 smithi050 ceph-mon[29187]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T18:11:11.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:11.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:11 smithi050 ceph-mon[29187]: from='osd.0 [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:11:11.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:11 smithi050 ceph-mon[29187]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:11:11.701 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:11 smithi177 ceph-mon[32308]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:11.701 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:11 smithi177 ceph-mon[32308]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T18:11:11.701 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:11 smithi177 ceph-mon[32308]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T18:11:11.701 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:11 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:11.701 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:11 smithi177 ceph-mon[32308]: from='osd.0 [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:11:11.701 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:11 smithi177 ceph-mon[32308]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:11:11.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:11 smithi188 ceph-mon[32694]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:11.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:11 smithi188 ceph-mon[32694]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T18:11:11.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:11 smithi188 ceph-mon[32694]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T18:11:11.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:11 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:11.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:11 smithi188 ceph-mon[32694]: from='osd.0 [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:11:11.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:11 smithi188 ceph-mon[32694]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:11:12.276 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T18:11:12.276 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T18:11:12.276 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-23T18:11:12.276 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-23T18:11:12.276 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111067 s, 944 MB/s 2024-04-23T18:11:12.276 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-23T18:11:12.624 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:12 smithi177 ceph-mon[32308]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-23T18:11:12.625 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:12 smithi177 ceph-mon[32308]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T18:11:12.625 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:12 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:12.625 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:12 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:12.625 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:12 smithi177 ceph-mon[32308]: from='osd.0 ' entity='osd.0' 2024-04-23T18:11:12.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:12 smithi050 ceph-mon[29187]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-23T18:11:12.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:12 smithi050 ceph-mon[29187]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T18:11:12.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:12 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:12.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:12 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:12.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:12 smithi050 ceph-mon[29187]: from='osd.0 ' entity='osd.0' 2024-04-23T18:11:12.696 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:11:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0[41799]: 2024-04-23T18:11:12.302+0000 7fc018940700 -1 osd.0 0 waiting for initial osdmap 2024-04-23T18:11:12.697 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:11:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0[41799]: 2024-04-23T18:11:12.306+0000 7fc01072b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:11:12.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:12 smithi188 ceph-mon[32694]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-23T18:11:12.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:12 smithi188 ceph-mon[32694]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T18:11:12.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:12 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:12.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:12 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:12.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:12 smithi188 ceph-mon[32694]: from='osd.0 ' entity='osd.0' 2024-04-23T18:11:12.927 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_4 2024-04-23T18:11:13.165 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:11:13.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: purged_snaps scrub starts 2024-04-23T18:11:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: purged_snaps scrub ok 2024-04-23T18:11:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: Detected new or changed devices on smithi050 2024-04-23T18:11:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi050 to 17093M 2024-04-23T18:11:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:13.790 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:13.790 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: osd.0 [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948] boot 2024-04-23T18:11:13.790 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T18:11:13.790 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:13.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: purged_snaps scrub starts 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: purged_snaps scrub ok 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: Detected new or changed devices on smithi050 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi050 to 17093M 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: osd.0 [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948] boot 2024-04-23T18:11:13.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T18:11:13.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:13.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: purged_snaps scrub starts 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: purged_snaps scrub ok 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: Detected new or changed devices on smithi050 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi050 to 17093M 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:13.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: osd.0 [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948] boot 2024-04-23T18:11:13.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T18:11:13.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.569+0000 7febdb7ac700 1 -- 172.21.15.177:0/3527528180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febd41019a0 msgr2=0x7febd4101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.569+0000 7febdb7ac700 1 --2- 172.21.15.177:0/3527528180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febd41019a0 0x7febd4101dc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7febd00099c0 tx=0x7febd0009cd0 comp rx=0 tx=0).stop 2024-04-23T18:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.569+0000 7febdb7ac700 1 -- 172.21.15.177:0/3527528180 shutdown_connections 2024-04-23T18:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.569+0000 7febdb7ac700 1 --2- 172.21.15.177:0/3527528180 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febd4103540 0x7febd4109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.569+0000 7febdb7ac700 1 --2- 172.21.15.177:0/3527528180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febd41019a0 0x7febd4101dc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:14.565 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.569+0000 7febdb7ac700 1 --2- 172.21.15.177:0/3527528180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd4102ba0 0x7febd4103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.569+0000 7febdb7ac700 1 -- 172.21.15.177:0/3527528180 >> 172.21.15.177:0/3527528180 conn(0x7febd40fcf40 msgr2=0x7febd40ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.569+0000 7febdb7ac700 1 -- 172.21.15.177:0/3527528180 shutdown_connections 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.569+0000 7febdb7ac700 1 -- 172.21.15.177:0/3527528180 wait complete. 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febdb7ac700 1 Processor -- start 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febdb7ac700 1 -- start start 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febdb7ac700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febd41019a0 0x7febd4105a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febdb7ac700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febd4102ba0 0x7febd4104120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febdb7ac700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd4103540 0x7febd4104660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febdb7ac700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7febd410d410 con 0x7febd4103540 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febdb7ac700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7febd410d290 con 0x7febd4102ba0 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febdb7ac700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7febd410d590 con 0x7febd41019a0 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febd8d47700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febd4102ba0 0x7febd4104120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:14.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febd9548700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febd41019a0 0x7febd4105a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febd8d47700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febd4102ba0 0x7febd4104120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:40754/0 (socket says 172.21.15.177:40754) 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febd8d47700 1 -- 172.21.15.177:0/955428251 learned_addr learned my addr 172.21.15.177:0/955428251 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febd9d49700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd4103540 0x7febd4104660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febd8d47700 1 -- 172.21.15.177:0/955428251 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febd41019a0 msgr2=0x7febd4105a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febd8d47700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febd41019a0 0x7febd4105a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febd8d47700 1 -- 172.21.15.177:0/955428251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd4103540 msgr2=0x7febd4104660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febd8d47700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd4103540 0x7febd4104660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.570+0000 7febd8d47700 1 -- 172.21.15.177:0/955428251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd4104f30 con 0x7febd4102ba0 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.571+0000 7febd8d47700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febd4102ba0 0x7febd4104120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7febc400c920 tx=0x7febc400cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.571+0000 7febca7fc700 1 -- 172.21.15.177:0/955428251 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc4007b00 con 0x7febd4102ba0 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.571+0000 7febdb7ac700 1 -- 172.21.15.177:0/955428251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd41a2d10 con 0x7febd4102ba0 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.571+0000 7febca7fc700 1 -- 172.21.15.177:0/955428251 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febc4004ce0 con 0x7febd4102ba0 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.571+0000 7febca7fc700 1 -- 172.21.15.177:0/955428251 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc4005500 con 0x7febd4102ba0 2024-04-23T18:11:14.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.571+0000 7febdb7ac700 1 -- 172.21.15.177:0/955428251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd41a31c0 con 0x7febd4102ba0 2024-04-23T18:11:14.568 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.572+0000 7febdb7ac700 1 -- 172.21.15.177:0/955428251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febd41a3450 con 0x7febd4102ba0 2024-04-23T18:11:14.568 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.573+0000 7febca7fc700 1 -- 172.21.15.177:0/955428251 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7febc4007ca0 con 0x7febd4102ba0 2024-04-23T18:11:14.568 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.573+0000 7febca7fc700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7febc00a3bb0 0x7febc00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:14.569 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.573+0000 7febca7fc700 1 -- 172.21.15.177:0/955428251 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7febc40ce950 con 0x7febd4102ba0 2024-04-23T18:11:14.569 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.574+0000 7febd9548700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7febc00a3bb0 0x7febc00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:14.569 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.574+0000 7febd9548700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7febc00a3bb0 0x7febc00a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7febd0006010 tx=0x7febd0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:14.571 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.577+0000 7febca7fc700 1 -- 172.21.15.177:0/955428251 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7febc4014030 con 0x7febd4102ba0 2024-04-23T18:11:14.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:14.782+0000 7febdb7ac700 1 -- 172.21.15.177:0/955428251 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7febd406ae30 con 0x7febc00a3bb0 2024-04-23T18:11:15.670 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:15 smithi177 ceph-mon[32308]: pgmap v48: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:15.670 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:15 smithi177 ceph-mon[32308]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T18:11:15.670 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:15 smithi177 ceph-mon[32308]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:15.670 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:15.670 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:15.670 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:15.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:15 smithi188 ceph-mon[32694]: pgmap v48: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:15.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:15 smithi188 ceph-mon[32694]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T18:11:15.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:15 smithi188 ceph-mon[32694]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:15.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:15.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:15.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:15.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:15 smithi050 ceph-mon[29187]: pgmap v48: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:15.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:15 smithi050 ceph-mon[29187]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T18:11:15.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:15 smithi050 ceph-mon[29187]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:15.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:17.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:17 smithi177 ceph-mon[32308]: pgmap v50: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:17.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:17 smithi177 ceph-mon[32308]: from='client.? 172.21.15.177:0/3481287234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b334ea9-1d7a-49b5-92bc-01416987b3ba"}]: dispatch 2024-04-23T18:11:17.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:17 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b334ea9-1d7a-49b5-92bc-01416987b3ba"}]: dispatch 2024-04-23T18:11:17.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:17 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b334ea9-1d7a-49b5-92bc-01416987b3ba"}]': finished 2024-04-23T18:11:17.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:17 smithi177 ceph-mon[32308]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T18:11:17.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:17 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:17.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:17 smithi188 ceph-mon[32694]: pgmap v50: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:17.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:17 smithi188 ceph-mon[32694]: from='client.? 172.21.15.177:0/3481287234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b334ea9-1d7a-49b5-92bc-01416987b3ba"}]: dispatch 2024-04-23T18:11:17.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:17 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b334ea9-1d7a-49b5-92bc-01416987b3ba"}]: dispatch 2024-04-23T18:11:17.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:17 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b334ea9-1d7a-49b5-92bc-01416987b3ba"}]': finished 2024-04-23T18:11:17.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:17 smithi188 ceph-mon[32694]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T18:11:17.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:17 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:17.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:17 smithi050 ceph-mon[29187]: pgmap v50: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:17.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:17 smithi050 ceph-mon[29187]: from='client.? 172.21.15.177:0/3481287234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b334ea9-1d7a-49b5-92bc-01416987b3ba"}]: dispatch 2024-04-23T18:11:17.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:17 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b334ea9-1d7a-49b5-92bc-01416987b3ba"}]: dispatch 2024-04-23T18:11:17.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:17 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b334ea9-1d7a-49b5-92bc-01416987b3ba"}]': finished 2024-04-23T18:11:17.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:17 smithi050 ceph-mon[29187]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T18:11:17.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:17 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:18.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:18 smithi188 ceph-mon[32694]: from='client.? 172.21.15.177:0/2594330454' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:18.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:18 smithi050 ceph-mon[29187]: from='client.? 172.21.15.177:0/2594330454' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:19.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:18 smithi177 ceph-mon[32308]: from='client.? 172.21.15.177:0/2594330454' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:19.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:19 smithi188 ceph-mon[32694]: pgmap v52: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:19.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:19 smithi050 ceph-mon[29187]: pgmap v52: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:20.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:19 smithi177 ceph-mon[32308]: pgmap v52: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:21.781 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:21 smithi177 ceph-mon[32308]: pgmap v53: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:21.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:21 smithi188 ceph-mon[32694]: pgmap v53: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:21 smithi050 ceph-mon[29187]: pgmap v53: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:23.605 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:23 smithi177 ceph-mon[32308]: pgmap v54: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:23.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:23 smithi188 ceph-mon[32694]: pgmap v54: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:23.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:23 smithi050 ceph-mon[29187]: pgmap v54: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:25.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:25 smithi177 ceph-mon[32308]: pgmap v55: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:25.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:25 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T18:11:25.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:25 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:25.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:25 smithi177 ceph-mon[32308]: Deploying daemon osd.1 on smithi177 2024-04-23T18:11:25.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:25 smithi188 ceph-mon[32694]: pgmap v55: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:25.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:25 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T18:11:25.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:25 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:25.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:25 smithi188 ceph-mon[32694]: Deploying daemon osd.1 on smithi177 2024-04-23T18:11:25.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:25 smithi050 ceph-mon[29187]: pgmap v55: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:25.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:25 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T18:11:25.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:25 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:25.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:25 smithi050 ceph-mon[29187]: Deploying daemon osd.1 on smithi177 2024-04-23T18:11:27.733 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:27 smithi177 ceph-mon[32308]: pgmap v56: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:27.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:27 smithi188 ceph-mon[32694]: pgmap v56: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:27.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:27 smithi050 ceph-mon[29187]: pgmap v56: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:29.815 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:29 smithi177 ceph-mon[32308]: pgmap v57: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:29.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:29 smithi188 ceph-mon[32694]: pgmap v57: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:29.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:29 smithi050 ceph-mon[29187]: pgmap v57: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:31.279 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:31.279 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:31.279 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:31.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:31.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:31.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:32.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:31 smithi177 ceph-mon[32308]: pgmap v58: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:32.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:31 smithi188 ceph-mon[32694]: pgmap v58: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:32.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:31 smithi050 ceph-mon[29187]: pgmap v58: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:33.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:33 smithi177 ceph-mon[32308]: pgmap v59: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:33.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:33 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:33.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:33 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:33.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:33 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:33.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:33 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:33.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:33 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:33.594 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.599+0000 7febca7fc700 1 -- 172.21.15.177:0/955428251 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7febd406ae30 con 0x7febc00a3bb0 2024-04-23T18:11:33.594 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 1 on host 'smithi177' 2024-04-23T18:11:33.596 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.601+0000 7febdb7ac700 1 -- 172.21.15.177:0/955428251 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7febc00a3bb0 msgr2=0x7febc00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:33.597 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.601+0000 7febdb7ac700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7febc00a3bb0 0x7febc00a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7febd0006010 tx=0x7febd0000bc0 comp rx=0 tx=0).stop 2024-04-23T18:11:33.597 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.601+0000 7febdb7ac700 1 -- 172.21.15.177:0/955428251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febd4102ba0 msgr2=0x7febd4104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:33.597 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.601+0000 7febdb7ac700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febd4102ba0 0x7febd4104120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7febc400c920 tx=0x7febc400cc30 comp rx=0 tx=0).stop 2024-04-23T18:11:33.597 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.601+0000 7febdb7ac700 1 -- 172.21.15.177:0/955428251 shutdown_connections 2024-04-23T18:11:33.597 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.601+0000 7febdb7ac700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd4103540 0x7febd4104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:33.597 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.601+0000 7febdb7ac700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febd41019a0 0x7febd4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:33.597 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.601+0000 7febdb7ac700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7febc00a3bb0 0x7febc00a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:33.597 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.601+0000 7febdb7ac700 1 --2- 172.21.15.177:0/955428251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febd4102ba0 0x7febd4104120 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:33.597 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.601+0000 7febdb7ac700 1 -- 172.21.15.177:0/955428251 >> 172.21.15.177:0/955428251 conn(0x7febd40fcf40 msgr2=0x7febd406a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:33.597 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.602+0000 7febdb7ac700 1 -- 172.21.15.177:0/955428251 shutdown_connections 2024-04-23T18:11:33.597 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:11:33.602+0000 7febdb7ac700 1 -- 172.21.15.177:0/955428251 wait complete. 2024-04-23T18:11:33.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:33 smithi188 ceph-mon[32694]: pgmap v59: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:33.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:33.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:33.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:33.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:33.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:33.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:33 smithi050 ceph-mon[29187]: pgmap v59: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:33.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:33.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:33.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:33.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:33.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:34.218 DEBUG:teuthology.orchestra.run.smithi177:osd.1> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.1.service 2024-04-23T18:11:34.220 INFO:tasks.cephadm:Deploying osd.2 on smithi188 with /dev/vg_nvme/lv_4... 2024-04-23T18:11:34.221 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T18:11:34.461 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:11:34.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:34 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:34.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:34 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:34.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:34 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:34.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:34 smithi177 ceph-mon[32308]: from='osd.1 [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:11:34.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:34 smithi177 ceph-mon[32308]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:11:34.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:34 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:34.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:34 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:34.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:34 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:34.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:34 smithi188 ceph-mon[32694]: from='osd.1 [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:11:34.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:34 smithi188 ceph-mon[32694]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:11:34.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:34.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:34.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:34.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:34 smithi050 ceph-mon[29187]: from='osd.1 [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:11:34.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:34 smithi050 ceph-mon[29187]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T18:11:35.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:35 smithi177 ceph-mon[32308]: pgmap v60: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:35.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:35 smithi177 ceph-mon[32308]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T18:11:35.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:35 smithi177 ceph-mon[32308]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T18:11:35.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:35.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:35 smithi177 ceph-mon[32308]: from='osd.1 [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:11:35.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:35 smithi177 ceph-mon[32308]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:11:35.789 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:11:35 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1[36954]: 2024-04-23T18:11:35.580+0000 7fd722e46700 -1 osd.1 0 waiting for initial osdmap 2024-04-23T18:11:35.790 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:11:35 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1[36954]: 2024-04-23T18:11:35.584+0000 7fd71ec39700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:11:35.819 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:35 smithi188 ceph-mon[32694]: pgmap v60: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:35.819 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:35 smithi188 ceph-mon[32694]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T18:11:35.819 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:35 smithi188 ceph-mon[32694]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T18:11:35.819 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:35.819 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:35 smithi188 ceph-mon[32694]: from='osd.1 [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:11:35.819 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:35 smithi188 ceph-mon[32694]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:11:35.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:35 smithi050 ceph-mon[29187]: pgmap v60: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:35.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:35 smithi050 ceph-mon[29187]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T18:11:35.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:35 smithi050 ceph-mon[29187]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T18:11:35.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:35.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:35 smithi050 ceph-mon[29187]: from='osd.1 [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:11:35.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:35 smithi050 ceph-mon[29187]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:11:36.482 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T18:11:36.482 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T18:11:36.482 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-23T18:11:36.482 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-23T18:11:36.482 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116808 s, 898 MB/s 2024-04-23T18:11:36.482 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-23T18:11:36.758 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:36 smithi188 ceph-mon[32694]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-23T18:11:36.758 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:36 smithi188 ceph-mon[32694]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T18:11:36.758 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:36 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:36.758 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:36 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:36.846 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:36 smithi177 ceph-mon[32308]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-23T18:11:36.846 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:36 smithi177 ceph-mon[32308]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T18:11:36.846 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:36 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:36.846 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:36 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:36.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:36 smithi050 ceph-mon[29187]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-23T18:11:36.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:36 smithi050 ceph-mon[29187]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T18:11:36.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:36 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:36.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:36 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:37.069 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_4 2024-04-23T18:11:37.310 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:11:37.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: purged_snaps scrub starts 2024-04-23T18:11:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: purged_snaps scrub ok 2024-04-23T18:11:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: pgmap v63: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: osd.1 [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435] boot 2024-04-23T18:11:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T18:11:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: Detected new or changed devices on smithi177 2024-04-23T18:11:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi177 to 17088M 2024-04-23T18:11:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:37.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:37.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:37.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:37.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: purged_snaps scrub starts 2024-04-23T18:11:37.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: purged_snaps scrub ok 2024-04-23T18:11:37.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: pgmap v63: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:37.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: osd.1 [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435] boot 2024-04-23T18:11:37.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T18:11:37.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:37.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: Detected new or changed devices on smithi177 2024-04-23T18:11:37.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:37.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:37.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:37.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi177 to 17088M 2024-04-23T18:11:37.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:37.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:37.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:37.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:38.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: purged_snaps scrub starts 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: purged_snaps scrub ok 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: pgmap v63: 0 pgs: ; 0 B data, 39 MiB used, 89 GiB / 89 GiB avail 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: osd.1 [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435] boot 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: Detected new or changed devices on smithi177 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi177 to 17088M 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:38.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:11:38.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:38.676 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:38 smithi188 ceph-mon[32694]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T18:11:38.696 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.702+0000 7f43aee54700 1 -- 172.21.15.188:0/312738633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43a8104640 msgr2=0x7f43a810aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:38.696 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.702+0000 7f43aee54700 1 --2- 172.21.15.188:0/312738633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43a8104640 0x7f43a810aef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f43a000b0a0 tx=0x7f43a000b3b0 comp rx=0 tx=0).stop 2024-04-23T18:11:38.696 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.702+0000 7f43aee54700 1 -- 172.21.15.188:0/312738633 shutdown_connections 2024-04-23T18:11:38.696 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.702+0000 7f43aee54700 1 --2- 172.21.15.188:0/312738633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43a8104640 0x7f43a810aef0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:38.696 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.702+0000 7f43aee54700 1 --2- 172.21.15.188:0/312738633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43a8102aa0 0x7f43a8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:38.696 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.702+0000 7f43aee54700 1 --2- 172.21.15.188:0/312738633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43a8103ca0 0x7f43a8104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:38.696 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.702+0000 7f43aee54700 1 -- 172.21.15.188:0/312738633 >> 172.21.15.188:0/312738633 conn(0x7f43a80fe040 msgr2=0x7f43a8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:38.696 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.702+0000 7f43aee54700 1 -- 172.21.15.188:0/312738633 shutdown_connections 2024-04-23T18:11:38.696 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.702+0000 7f43aee54700 1 -- 172.21.15.188:0/312738633 wait complete. 2024-04-23T18:11:38.698 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.702+0000 7f43aee54700 1 Processor -- start 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.702+0000 7f43aee54700 1 -- start start 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43aee54700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43a8102aa0 0x7f43a8198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43aee54700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43a8103ca0 0x7f43a81990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43aee54700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43a8104640 0x7f43a81a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43aee54700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f43a810e570 con 0x7f43a8103ca0 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43aee54700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f43a810e3f0 con 0x7f43a8102aa0 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43aee54700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f43a810e6f0 con 0x7f43a8104640 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43ad3f1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43a8104640 0x7f43a81a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43ad3f1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43a8104640 0x7f43a81a0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:40154/0 (socket says 172.21.15.188:40154) 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43ad3f1700 1 -- 172.21.15.188:0/1659516614 learned_addr learned my addr 172.21.15.188:0/1659516614 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43acbf0700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43a8102aa0 0x7f43a8198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43a7fff700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43a8103ca0 0x7f43a81990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:38.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43ad3f1700 1 -- 172.21.15.188:0/1659516614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43a8102aa0 msgr2=0x7f43a8198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:38.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43ad3f1700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43a8102aa0 0x7f43a8198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:38.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43ad3f1700 1 -- 172.21.15.188:0/1659516614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43a8103ca0 msgr2=0x7f43a81990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:38.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43ad3f1700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43a8103ca0 0x7f43a81990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:38.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43ad3f1700 1 -- 172.21.15.188:0/1659516614 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a81a0810 con 0x7f43a8104640 2024-04-23T18:11:38.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43a7fff700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43a8103ca0 0x7f43a81990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:11:38.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43ad3f1700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43a8104640 0x7f43a81a0170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f43a000b070 tx=0x7f43a0009c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:38.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43a5ffb700 1 -- 172.21.15.188:0/1659516614 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0021070 con 0x7f43a8104640 2024-04-23T18:11:38.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43a5ffb700 1 -- 172.21.15.188:0/1659516614 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43a0007d40 con 0x7f43a8104640 2024-04-23T18:11:38.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.703+0000 7f43a5ffb700 1 -- 172.21.15.188:0/1659516614 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a001d450 con 0x7f43a8104640 2024-04-23T18:11:38.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.704+0000 7f43aee54700 1 -- 172.21.15.188:0/1659516614 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a81a0aa0 con 0x7f43a8104640 2024-04-23T18:11:38.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.704+0000 7f43aee54700 1 -- 172.21.15.188:0/1659516614 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a81a0fa0 con 0x7f43a8104640 2024-04-23T18:11:38.700 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.705+0000 7f43aee54700 1 -- 172.21.15.188:0/1659516614 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43a8069730 con 0x7f43a8104640 2024-04-23T18:11:38.701 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.706+0000 7f43a5ffb700 1 -- 172.21.15.188:0/1659516614 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f43a00040b0 con 0x7f43a8104640 2024-04-23T18:11:38.701 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.706+0000 7f43a5ffb700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43980a3c30 0x7f43980a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:11:38.701 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.706+0000 7f43a5ffb700 1 -- 172.21.15.188:0/1659516614 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f43a000f020 con 0x7f43a8104640 2024-04-23T18:11:38.701 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.706+0000 7f43acbf0700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43980a3c30 0x7f43980a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:11:38.701 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.707+0000 7f43acbf0700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43980a3c30 0x7f43980a60f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f43a8103b00 tx=0x7f439c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:11:38.705 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.711+0000 7f43a5ffb700 1 -- 172.21.15.188:0/1659516614 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f43a002a020 con 0x7f43a8104640 2024-04-23T18:11:38.910 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:38.916+0000 7f43aee54700 1 -- 172.21.15.188:0/1659516614 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f43a8062e80 con 0x7f43980a3c30 2024-04-23T18:11:38.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:38 smithi050 ceph-mon[29187]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T18:11:39.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:38 smithi177 ceph-mon[32308]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T18:11:39.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:39 smithi188 ceph-mon[32694]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:39.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:39 smithi188 ceph-mon[32694]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:39.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:39.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:39.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:39.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:39 smithi050 ceph-mon[29187]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:39.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:39 smithi050 ceph-mon[29187]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:39.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:39.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:39.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:40.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:39 smithi177 ceph-mon[32308]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:40.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:39 smithi177 ceph-mon[32308]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:11:40.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:11:40.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:11:40.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:41.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:41 smithi050 ceph-mon[29187]: pgmap v67: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:41.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:41 smithi050 ceph-mon[29187]: from='client.? 172.21.15.188:0/2404385167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a46cc56-b2f7-431d-87c3-5160a3db721d"}]: dispatch 2024-04-23T18:11:41.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:41 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a46cc56-b2f7-431d-87c3-5160a3db721d"}]: dispatch 2024-04-23T18:11:41.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:41 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a46cc56-b2f7-431d-87c3-5160a3db721d"}]': finished 2024-04-23T18:11:41.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:41 smithi050 ceph-mon[29187]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T18:11:41.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:11:42.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:41 smithi177 ceph-mon[32308]: pgmap v67: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:42.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:41 smithi177 ceph-mon[32308]: from='client.? 172.21.15.188:0/2404385167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a46cc56-b2f7-431d-87c3-5160a3db721d"}]: dispatch 2024-04-23T18:11:42.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:41 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a46cc56-b2f7-431d-87c3-5160a3db721d"}]: dispatch 2024-04-23T18:11:42.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:41 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a46cc56-b2f7-431d-87c3-5160a3db721d"}]': finished 2024-04-23T18:11:42.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:41 smithi177 ceph-mon[32308]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T18:11:42.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:11:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:41 smithi188 ceph-mon[32694]: pgmap v67: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:41 smithi188 ceph-mon[32694]: from='client.? 172.21.15.188:0/2404385167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a46cc56-b2f7-431d-87c3-5160a3db721d"}]: dispatch 2024-04-23T18:11:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:41 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a46cc56-b2f7-431d-87c3-5160a3db721d"}]: dispatch 2024-04-23T18:11:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:41 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a46cc56-b2f7-431d-87c3-5160a3db721d"}]': finished 2024-04-23T18:11:42.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:41 smithi188 ceph-mon[32694]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T18:11:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:11:42.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:42 smithi050 ceph-mon[29187]: from='client.? 172.21.15.188:0/1927937234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:43.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:42 smithi177 ceph-mon[32308]: from='client.? 172.21.15.188:0/1927937234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:43.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:42 smithi188 ceph-mon[32694]: from='client.? 172.21.15.188:0/1927937234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:11:43.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:43 smithi050 ceph-mon[29187]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:44.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:43 smithi177 ceph-mon[32308]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:44.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:43 smithi188 ceph-mon[32694]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:45.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:45 smithi188 ceph-mon[32694]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:45.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:45 smithi050 ceph-mon[29187]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:46.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:45 smithi177 ceph-mon[32308]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:47.746 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:47 smithi188 ceph-mon[32694]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:47.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:47 smithi050 ceph-mon[29187]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:48.031 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:47 smithi177 ceph-mon[32308]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:49.881 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:49 smithi188 ceph-mon[32694]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:49.881 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T18:11:49.881 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:49.881 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:49 smithi188 ceph-mon[32694]: Deploying daemon osd.2 on smithi188 2024-04-23T18:11:49.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:49 smithi050 ceph-mon[29187]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:49.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T18:11:49.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:49.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:49 smithi050 ceph-mon[29187]: Deploying daemon osd.2 on smithi188 2024-04-23T18:11:50.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:49 smithi177 ceph-mon[32308]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:50.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T18:11:50.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:11:50.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:49 smithi177 ceph-mon[32308]: Deploying daemon osd.2 on smithi188 2024-04-23T18:11:51.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:51 smithi188 ceph-mon[32694]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:51.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:51 smithi050 ceph-mon[29187]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:52.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:51 smithi177 ceph-mon[32308]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:53.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:53 smithi188 ceph-mon[32694]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:53.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:53 smithi050 ceph-mon[29187]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:54.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:53 smithi177 ceph-mon[32308]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:55.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:55 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:55.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:55 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:55.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:55 smithi050 ceph-mon[29187]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:55.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:55 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:55.766 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:55 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:55.766 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:55 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:55.766 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:55 smithi188 ceph-mon[32694]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:55.767 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:55 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:55.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:55 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:55.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:55 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:55.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:55 smithi177 ceph-mon[32308]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:55.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:55 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:11:57.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:57 smithi188 ceph-mon[32694]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:57.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:57 smithi177 ceph-mon[32308]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:57.829 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 2 on host 'smithi188' 2024-04-23T18:11:57.830 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.835+0000 7f43a5ffb700 1 -- 172.21.15.188:0/1659516614 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f43a8062e80 con 0x7f43980a3c30 2024-04-23T18:11:57.832 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.838+0000 7f43aee54700 1 -- 172.21.15.188:0/1659516614 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43980a3c30 msgr2=0x7f43980a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:57.832 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.838+0000 7f43aee54700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43980a3c30 0x7f43980a60f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f43a8103b00 tx=0x7f439c00a380 comp rx=0 tx=0).stop 2024-04-23T18:11:57.832 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.838+0000 7f43aee54700 1 -- 172.21.15.188:0/1659516614 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43a8104640 msgr2=0x7f43a81a0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:11:57.832 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.838+0000 7f43aee54700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43a8104640 0x7f43a81a0170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f43a000b070 tx=0x7f43a0009c70 comp rx=0 tx=0).stop 2024-04-23T18:11:57.832 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.838+0000 7f43aee54700 1 -- 172.21.15.188:0/1659516614 shutdown_connections 2024-04-23T18:11:57.832 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.838+0000 7f43aee54700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43a8104640 0x7f43a81a0170 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:57.832 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.838+0000 7f43aee54700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43980a3c30 0x7f43980a60f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:57.832 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.838+0000 7f43aee54700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43a8102aa0 0x7f43a8198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:57.833 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.838+0000 7f43aee54700 1 --2- 172.21.15.188:0/1659516614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43a8103ca0 0x7f43a81990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:11:57.833 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.838+0000 7f43aee54700 1 -- 172.21.15.188:0/1659516614 >> 172.21.15.188:0/1659516614 conn(0x7f43a80fe040 msgr2=0x7f43a81070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:11:57.833 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.838+0000 7f43aee54700 1 -- 172.21.15.188:0/1659516614 shutdown_connections 2024-04-23T18:11:57.833 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:11:57.838+0000 7f43aee54700 1 -- 172.21.15.188:0/1659516614 wait complete. 2024-04-23T18:11:57.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:57 smithi050 ceph-mon[29187]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:11:58.442 DEBUG:teuthology.orchestra.run.smithi188:osd.2> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.2.service 2024-04-23T18:11:58.444 INFO:tasks.cephadm:Deploying osd.3 on smithi050 with /dev/vg_nvme/lv_3... 2024-04-23T18:11:58.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T18:11:58.699 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:11:59.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:58 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:59.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:58 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:59.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:58 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:59.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:58 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:59.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:58 smithi188 ceph-mon[32694]: from='osd.2 [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:11:59.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:58 smithi188 ceph-mon[32694]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:11:59.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:58 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:59.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:58 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:59.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:58 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:59.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:58 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:59.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:58 smithi050 ceph-mon[29187]: from='osd.2 [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:11:59.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:58 smithi050 ceph-mon[29187]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:11:59.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:58 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:59.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:58 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:59.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:58 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:59.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:58 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:11:59.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:58 smithi177 ceph-mon[32308]: from='osd.2 [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:11:59.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:58 smithi177 ceph-mon[32308]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T18:12:00.048 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:59 smithi050 ceph-mon[29187]: pgmap v77: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:12:00.049 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:59 smithi050 ceph-mon[29187]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T18:12:00.049 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:59 smithi050 ceph-mon[29187]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T18:12:00.049 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:59 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:00.049 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:59 smithi050 ceph-mon[29187]: from='osd.2 [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:12:00.049 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:11:59 smithi050 ceph-mon[29187]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:12:00.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:59 smithi188 ceph-mon[32694]: pgmap v77: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:12:00.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:59 smithi188 ceph-mon[32694]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T18:12:00.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:59 smithi188 ceph-mon[32694]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T18:12:00.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:59 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:00.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:59 smithi188 ceph-mon[32694]: from='osd.2 [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:12:00.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:11:59 smithi188 ceph-mon[32694]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:12:00.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:59 smithi177 ceph-mon[32308]: pgmap v77: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:12:00.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:59 smithi177 ceph-mon[32308]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T18:12:00.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:59 smithi177 ceph-mon[32308]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T18:12:00.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:59 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:00.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:59 smithi177 ceph-mon[32308]: from='osd.2 [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:12:00.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:11:59 smithi177 ceph-mon[32308]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:12:00.622 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T18:12:00.622 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T18:12:00.622 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-23T18:12:00.622 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-23T18:12:00.622 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00838466 s, 1.3 GB/s 2024-04-23T18:12:00.622 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-23T18:12:01.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:00 smithi188 ceph-mon[32694]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T18:12:01.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:00 smithi188 ceph-mon[32694]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T18:12:01.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:00 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:01.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:00 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:01.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:00 smithi188 ceph-mon[32694]: from='osd.2 ' entity='osd.2' 2024-04-23T18:12:01.082 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:12:00 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2[37874]: 2024-04-23T18:12:00.826+0000 7f4d0d257700 -1 osd.2 0 waiting for initial osdmap 2024-04-23T18:12:01.082 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:12:00 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2[37874]: 2024-04-23T18:12:00.946+0000 7f4d05042700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:12:01.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:00 smithi050 ceph-mon[29187]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T18:12:01.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:00 smithi050 ceph-mon[29187]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T18:12:01.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:00 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:01.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:00 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:01.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:00 smithi050 ceph-mon[29187]: from='osd.2 ' entity='osd.2' 2024-04-23T18:12:01.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_3 2024-04-23T18:12:01.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:00 smithi177 ceph-mon[32308]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T18:12:01.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:00 smithi177 ceph-mon[32308]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T18:12:01.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:00 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:01.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:00 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:01.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:00 smithi177 ceph-mon[32308]: from='osd.2 ' entity='osd.2' 2024-04-23T18:12:01.546 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:12:02.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: purged_snaps scrub starts 2024-04-23T18:12:02.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: purged_snaps scrub ok 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: pgmap v80: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:02.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: purged_snaps scrub starts 2024-04-23T18:12:02.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: purged_snaps scrub ok 2024-04-23T18:12:02.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: pgmap v80: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:12:02.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:02.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:02.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:02.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:02.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:02.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:02.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:02.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: purged_snaps scrub starts 2024-04-23T18:12:02.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: purged_snaps scrub ok 2024-04-23T18:12:02.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: pgmap v80: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:12:02.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:02.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:02.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:02.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:02.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:02.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:02.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:02.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:02.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:03.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.122+0000 7fa765d0d700 1 -- 172.21.15.50:0/2931551096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa760102ac0 msgr2=0x7fa760102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:03.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.122+0000 7fa765d0d700 1 --2- 172.21.15.50:0/2931551096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa760102ac0 0x7fa760102ee0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa7500099c0 tx=0x7fa750009cd0 comp rx=0 tx=0).stop 2024-04-23T18:12:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.123+0000 7fa765d0d700 1 -- 172.21.15.50:0/2931551096 shutdown_connections 2024-04-23T18:12:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.123+0000 7fa765d0d700 1 --2- 172.21.15.50:0/2931551096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa760104660 0x7fa76010af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.123+0000 7fa765d0d700 1 --2- 172.21.15.50:0/2931551096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa760102ac0 0x7fa760102ee0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.123+0000 7fa765d0d700 1 --2- 172.21.15.50:0/2931551096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa760103cc0 0x7fa760104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.123+0000 7fa765d0d700 1 -- 172.21.15.50:0/2931551096 >> 172.21.15.50:0/2931551096 conn(0x7fa7600fe060 msgr2=0x7fa7601004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.123+0000 7fa765d0d700 1 -- 172.21.15.50:0/2931551096 shutdown_connections 2024-04-23T18:12:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.123+0000 7fa765d0d700 1 -- 172.21.15.50:0/2931551096 wait complete. 2024-04-23T18:12:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.123+0000 7fa765d0d700 1 Processor -- start 2024-04-23T18:12:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.123+0000 7fa765d0d700 1 -- start start 2024-04-23T18:12:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa765d0d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa760102ac0 0x7fa760198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa765d0d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa760103cc0 0x7fa7601990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa765d0d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa760104660 0x7fa7601a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa765d0d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa76010e530 con 0x7fa760104660 2024-04-23T18:12:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa765d0d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa76010e3b0 con 0x7fa760103cc0 2024-04-23T18:12:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa765d0d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa76010e6b0 con 0x7fa760102ac0 2024-04-23T18:12:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa75ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa760104660 0x7fa7601a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa75ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa760104660 0x7fa7601a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58530/0 (socket says 172.21.15.50:58530) 2024-04-23T18:12:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa75ffff700 1 -- 172.21.15.50:0/1912658150 learned_addr learned my addr 172.21.15.50:0/1912658150 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:12:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa75f7fe700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa760102ac0 0x7fa760198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa75effd700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa760103cc0 0x7fa7601990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa75ffff700 1 -- 172.21.15.50:0/1912658150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa760102ac0 msgr2=0x7fa760198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa75ffff700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa760102ac0 0x7fa760198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa75ffff700 1 -- 172.21.15.50:0/1912658150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa760103cc0 msgr2=0x7fa7601990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa75ffff700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa760103cc0 0x7fa7601990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa75ffff700 1 -- 172.21.15.50:0/1912658150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7601a07f0 con 0x7fa760104660 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa75ffff700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa760104660 0x7fa7601a0150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa758007da0 tx=0x7fa75800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa75cff9700 1 -- 172.21.15.50:0/1912658150 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758016020 con 0x7fa760104660 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.124+0000 7fa765d0d700 1 -- 172.21.15.50:0/1912658150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7601a0ae0 con 0x7fa760104660 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.125+0000 7fa765d0d700 1 -- 172.21.15.50:0/1912658150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7601a1070 con 0x7fa760104660 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.125+0000 7fa75cff9700 1 -- 172.21.15.50:0/1912658150 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa75800ac50 con 0x7fa760104660 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.125+0000 7fa75cff9700 1 -- 172.21.15.50:0/1912658150 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7580115e0 con 0x7fa760104660 2024-04-23T18:12:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.125+0000 7fa765d0d700 1 -- 172.21.15.50:0/1912658150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa74c005360 con 0x7fa760104660 2024-04-23T18:12:03.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.127+0000 7fa75cff9700 1 -- 172.21.15.50:0/1912658150 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fa758019020 con 0x7fa760104660 2024-04-23T18:12:03.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.128+0000 7fa75cff9700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa7480a3bb0 0x7fa7480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:03.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.128+0000 7fa75f7fe700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa7480a3bb0 0x7fa7480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:03.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.128+0000 7fa75cff9700 1 -- 172.21.15.50:0/1912658150 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7fa7580cf270 con 0x7fa760104660 2024-04-23T18:12:03.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.128+0000 7fa75f7fe700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa7480a3bb0 0x7fa7480a6070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa7500038c0 tx=0x7fa75000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:03.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.132+0000 7fa75cff9700 1 -- 172.21.15.50:0/1912658150 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa758020020 con 0x7fa760104660 2024-04-23T18:12:03.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:02 smithi050 ceph-mon[29187]: Detected new or changed devices on smithi188 2024-04-23T18:12:03.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:02 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi188 to 17088M 2024-04-23T18:12:03.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:03.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:02 smithi050 ceph-mon[29187]: osd.2 [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369] boot 2024-04-23T18:12:03.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:02 smithi050 ceph-mon[29187]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T18:12:03.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:03.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:12:03.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:02 smithi177 ceph-mon[32308]: Detected new or changed devices on smithi188 2024-04-23T18:12:03.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:02 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi188 to 17088M 2024-04-23T18:12:03.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:02 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:03.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:02 smithi177 ceph-mon[32308]: osd.2 [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369] boot 2024-04-23T18:12:03.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:02 smithi177 ceph-mon[32308]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T18:12:03.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:02 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:03.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:02 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:12:03.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:02 smithi188 ceph-mon[32694]: Detected new or changed devices on smithi188 2024-04-23T18:12:03.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:02 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi188 to 17088M 2024-04-23T18:12:03.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:02 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:03.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:02 smithi188 ceph-mon[32694]: osd.2 [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369] boot 2024-04-23T18:12:03.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:02 smithi188 ceph-mon[32694]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T18:12:03.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:02 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:12:03.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:02 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:12:03.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:03.351+0000 7fa765d0d700 1 -- 172.21.15.50:0/1912658150 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa74c000bf0 con 0x7fa7480a3bb0 2024-04-23T18:12:04.178 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:03 smithi050 ceph-mon[29187]: pgmap v82: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:12:04.178 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:03 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T18:12:04.178 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:03 smithi050 ceph-mon[29187]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T18:12:04.178 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:03 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:12:04.178 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:03 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:04.178 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:03 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:04.179 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:03 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:04.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:03 smithi177 ceph-mon[32308]: pgmap v82: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:12:04.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:03 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T18:12:04.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:03 smithi177 ceph-mon[32308]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T18:12:04.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:03 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:12:04.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:03 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:04.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:03 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:04.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:03 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:04.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:03 smithi188 ceph-mon[32694]: pgmap v82: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T18:12:04.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:03 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T18:12:04.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:03 smithi188 ceph-mon[32694]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T18:12:04.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:03 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:12:04.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:03 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:04.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:03 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:04.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:03 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:04.788 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40668]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:12:04.788 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40668]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:12:04.788 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40668]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:12:04.789 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40668]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:12:04.789 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40671]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:12:04.789 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40671]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:12:04.789 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40671]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:12:04.789 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40671]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:12:04.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40674]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:12:04.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40674]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:12:04.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40674]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:12:04.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40674]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:12:04.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40677]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:12:04.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40677]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:12:04.790 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40677]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:12:04.790 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 sudo[40677]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:12:04.916 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:12:04.916 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:12:04.916 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46658]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:12:04.916 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46658]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:12:04.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46661]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:12:04.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46661]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:12:04.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46661]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:12:04.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46661]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:12:04.917 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46650]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:12:04.917 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46650]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:12:04.917 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46650]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:12:04.917 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46650]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:12:04.917 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46653]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:12:04.918 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46653]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:12:04.918 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46653]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:12:04.918 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:12:04 smithi050 sudo[46653]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:12:04.918 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41659]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:12:04.918 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41659]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:12:04.918 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41659]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:12:04.918 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41659]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:12:04.918 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41662]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:12:04.919 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41662]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:12:04.919 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41662]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:12:04.919 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41662]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:12:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41665]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T18:12:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41665]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:12:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41665]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:12:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41665]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:12:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41668]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T18:12:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41668]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41668]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 sudo[41668]: pam_unix(sudo:session): session closed for user root 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:12:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:12:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:12:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:12:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:12:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:12:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:12:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:12:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:12:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:12:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:12:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:04 smithi188 ceph-mon[32694]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:12:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:04.936+0000 7fa75cff9700 1 -- 172.21.15.50:0/1912658150 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa758099020 con 0x7fa760104660 2024-04-23T18:12:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:12:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T18:12:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:12:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:12:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:04 smithi050 ceph-mon[29187]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:12:05.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:12:05.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:12:05.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:12:05.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T18:12:05.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:12:05.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:12:05.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:12:05.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:04 smithi177 ceph-mon[32308]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T18:12:06.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:05 smithi050 ceph-mon[29187]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:06.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:05 smithi050 ceph-mon[29187]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T18:12:06.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:05 smithi050 ceph-mon[29187]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-23T18:12:06.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:05 smithi177 ceph-mon[32308]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:06.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:05 smithi177 ceph-mon[32308]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T18:12:06.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:05 smithi177 ceph-mon[32308]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-23T18:12:06.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:05 smithi188 ceph-mon[32694]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:06.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:05 smithi188 ceph-mon[32694]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T18:12:06.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:05 smithi188 ceph-mon[32694]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-23T18:12:07.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:06 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/658281278' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02828390-a4c5-468a-8425-092eb1513bdd"}]: dispatch 2024-04-23T18:12:07.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:06 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/658281278' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02828390-a4c5-468a-8425-092eb1513bdd"}]': finished 2024-04-23T18:12:07.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:06 smithi050 ceph-mon[29187]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T18:12:07.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:07.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:06 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2013482420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:07.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:06 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/658281278' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02828390-a4c5-468a-8425-092eb1513bdd"}]: dispatch 2024-04-23T18:12:07.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:06 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/658281278' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02828390-a4c5-468a-8425-092eb1513bdd"}]': finished 2024-04-23T18:12:07.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:06 smithi177 ceph-mon[32308]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T18:12:07.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:07.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:06 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2013482420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:07.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:06 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/658281278' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02828390-a4c5-468a-8425-092eb1513bdd"}]: dispatch 2024-04-23T18:12:07.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:06 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/658281278' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02828390-a4c5-468a-8425-092eb1513bdd"}]': finished 2024-04-23T18:12:07.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:06 smithi188 ceph-mon[32694]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T18:12:07.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:07.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:06 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2013482420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:08.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:08 smithi188 ceph-mon[32694]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:08.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:08 smithi050 ceph-mon[29187]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:08 smithi177 ceph-mon[32308]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:10.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:10 smithi188 ceph-mon[32694]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:10.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:10 smithi050 ceph-mon[29187]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:10.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:10 smithi177 ceph-mon[32308]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:12.276 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:12 smithi050 ceph-mon[29187]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:12.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:12 smithi188 ceph-mon[32694]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:12.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:12 smithi177 ceph-mon[32308]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:14.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:14 smithi188 ceph-mon[32694]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:14.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:14 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T18:12:14.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:14 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:14.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:14 smithi050 ceph-mon[29187]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:14.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:14 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T18:12:14.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:14 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:14.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:14 smithi177 ceph-mon[32308]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:14.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:14 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T18:12:14.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:14 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:15.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:15 smithi050 ceph-mon[29187]: Deploying daemon osd.3 on smithi050 2024-04-23T18:12:15.438 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:15 smithi177 ceph-mon[32308]: Deploying daemon osd.3 on smithi050 2024-04-23T18:12:15.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:15 smithi188 ceph-mon[32694]: Deploying daemon osd.3 on smithi050 2024-04-23T18:12:16.341 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:16 smithi050 ceph-mon[29187]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:16.507 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:16 smithi188 ceph-mon[32694]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:16.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:16 smithi177 ceph-mon[32308]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:18.166 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:18 smithi050 ceph-mon[29187]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:18.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:18 smithi177 ceph-mon[32308]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:18.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:18 smithi188 ceph-mon[32694]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:20.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:20 smithi050 ceph-mon[29187]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:20.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:20 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:20.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:20 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:20.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:20 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:20.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:20 smithi188 ceph-mon[32694]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:20.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:20 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:20.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:20 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:20.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:20 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:20.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:20 smithi177 ceph-mon[32308]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:20.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:20 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:20.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:20 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:20.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:20 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:22.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:22 smithi050 ceph-mon[29187]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:22.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:22 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:22.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:22 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:22.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:22 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:22.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:22 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:22.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:22 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:22.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:22 smithi177 ceph-mon[32308]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:22.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:22 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:22.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:22 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:22.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:22 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:22.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:22 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:22.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:22 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:22.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:22 smithi188 ceph-mon[32694]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:22.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:22 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:22.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:22 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:22.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:22 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:22.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:22 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:22.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:22 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:22.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.709+0000 7fa75cff9700 1 -- 172.21.15.50:0/1912658150 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa74c000bf0 con 0x7fa7480a3bb0 2024-04-23T18:12:22.705 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 3 on host 'smithi050' 2024-04-23T18:12:22.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.711+0000 7fa765d0d700 1 -- 172.21.15.50:0/1912658150 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa7480a3bb0 msgr2=0x7fa7480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:22.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.711+0000 7fa765d0d700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa7480a3bb0 0x7fa7480a6070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa7500038c0 tx=0x7fa75000b540 comp rx=0 tx=0).stop 2024-04-23T18:12:22.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.711+0000 7fa765d0d700 1 -- 172.21.15.50:0/1912658150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa760104660 msgr2=0x7fa7601a0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:22.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.711+0000 7fa765d0d700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa760104660 0x7fa7601a0150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa758007da0 tx=0x7fa75800a340 comp rx=0 tx=0).stop 2024-04-23T18:12:22.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.711+0000 7fa765d0d700 1 -- 172.21.15.50:0/1912658150 shutdown_connections 2024-04-23T18:12:22.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.711+0000 7fa765d0d700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa760104660 0x7fa7601a0150 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:22.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.711+0000 7fa765d0d700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa760102ac0 0x7fa760198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:22.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.711+0000 7fa765d0d700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa7480a3bb0 0x7fa7480a6070 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:22.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.711+0000 7fa765d0d700 1 --2- 172.21.15.50:0/1912658150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa760103cc0 0x7fa7601990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:22.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.711+0000 7fa765d0d700 1 -- 172.21.15.50:0/1912658150 >> 172.21.15.50:0/1912658150 conn(0x7fa7600fe060 msgr2=0x7fa760107060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:22.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.712+0000 7fa765d0d700 1 -- 172.21.15.50:0/1912658150 shutdown_connections 2024-04-23T18:12:22.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:12:22.712+0000 7fa765d0d700 1 -- 172.21.15.50:0/1912658150 wait complete. 2024-04-23T18:12:23.282 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:23 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:23.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:23 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:23.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:23 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:23.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:23 smithi050 ceph-mon[29187]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T18:12:23.352 DEBUG:teuthology.orchestra.run.smithi050:osd.3> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.3.service 2024-04-23T18:12:23.354 INFO:tasks.cephadm:Deploying osd.4 on smithi177 with /dev/vg_nvme/lv_3... 2024-04-23T18:12:23.354 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T18:12:23.381 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:23 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:23.381 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:23 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:23.381 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:23 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:23.381 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:23 smithi177 ceph-mon[32308]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T18:12:23.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:23 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:23.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:23 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:23.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:23 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:23.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:23 smithi188 ceph-mon[32694]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T18:12:23.720 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:12:24.414 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:24 smithi050 ceph-mon[29187]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:24.414 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:24 smithi050 ceph-mon[29187]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T18:12:24.414 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:24 smithi050 ceph-mon[29187]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T18:12:24.414 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:24 smithi050 ceph-mon[29187]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:12:24.414 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:24.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:24 smithi177 ceph-mon[32308]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:24.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:24 smithi177 ceph-mon[32308]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T18:12:24.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:24 smithi177 ceph-mon[32308]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T18:12:24.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:24 smithi177 ceph-mon[32308]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:12:24.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:24 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:24.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:24 smithi188 ceph-mon[32694]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:24.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:24 smithi188 ceph-mon[32694]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T18:12:24.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:24 smithi188 ceph-mon[32694]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T18:12:24.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:24 smithi188 ceph-mon[32694]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:12:24.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:25.557 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T18:12:25.557 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T18:12:25.557 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-23T18:12:25.557 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-23T18:12:25.557 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110709 s, 947 MB/s 2024-04-23T18:12:25.557 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-23T18:12:25.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:25 smithi050 ceph-mon[29187]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:25.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:25 smithi050 ceph-mon[29187]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-23T18:12:25.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:25 smithi050 ceph-mon[29187]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T18:12:25.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:25 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:25.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:25 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:25.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:25 smithi050 ceph-mon[29187]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' 2024-04-23T18:12:25.696 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:12:25 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3[48756]: 2024-04-23T18:12:25.434+0000 7fa5fef3c700 -1 osd.3 0 waiting for initial osdmap 2024-04-23T18:12:25.696 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:12:25 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3[48756]: 2024-04-23T18:12:25.442+0000 7fa5f6d27700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:12:25.978 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:25 smithi177 ceph-mon[32308]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:25.979 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:25 smithi177 ceph-mon[32308]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-23T18:12:25.979 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:25 smithi177 ceph-mon[32308]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T18:12:25.979 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:25 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:25.979 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:25 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:25.979 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:25 smithi177 ceph-mon[32308]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' 2024-04-23T18:12:26.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:25 smithi188 ceph-mon[32694]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-23T18:12:26.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:25 smithi188 ceph-mon[32694]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-23T18:12:26.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:25 smithi188 ceph-mon[32694]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T18:12:26.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:25 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:26.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:25 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:26.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:25 smithi188 ceph-mon[32694]: from='osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303]' entity='osd.3' 2024-04-23T18:12:26.273 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_3 2024-04-23T18:12:26.556 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:12:26.873 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: purged_snaps scrub starts 2024-04-23T18:12:26.873 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: purged_snaps scrub ok 2024-04-23T18:12:26.874 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303] boot 2024-04-23T18:12:26.874 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T18:12:26.874 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:26.874 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T18:12:26.874 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:26.874 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:26.874 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:26.874 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:26.874 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:26.874 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:26.874 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:26.874 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:26.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: purged_snaps scrub starts 2024-04-23T18:12:26.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: purged_snaps scrub ok 2024-04-23T18:12:26.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303] boot 2024-04-23T18:12:26.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T18:12:26.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:26.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T18:12:26.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:26.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:26.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:26.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:26.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:26.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:26.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:26.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:27.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: purged_snaps scrub starts 2024-04-23T18:12:27.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: purged_snaps scrub ok 2024-04-23T18:12:27.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: osd.3 [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303] boot 2024-04-23T18:12:27.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T18:12:27.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:12:27.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T18:12:27.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:27.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:27.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:27.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:27.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:27.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:27.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:27.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:28.045 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.049+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/1654765821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b44102aa0 msgr2=0x7f2b44102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:28.045 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.049+0000 7f2b4b4d6700 1 --2- 172.21.15.177:0/1654765821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b44102aa0 0x7f2b44102ec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2b400099c0 tx=0x7f2b40009cd0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.045 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.049+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/1654765821 shutdown_connections 2024-04-23T18:12:28.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.049+0000 7f2b4b4d6700 1 --2- 172.21.15.177:0/1654765821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b44104640 0x7f2b4410aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.049+0000 7f2b4b4d6700 1 --2- 172.21.15.177:0/1654765821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b44102aa0 0x7f2b44102ec0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.049+0000 7f2b4b4d6700 1 --2- 172.21.15.177:0/1654765821 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b44103ca0 0x7f2b44104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.049+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/1654765821 >> 172.21.15.177:0/1654765821 conn(0x7f2b440fe040 msgr2=0x7f2b44100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:28.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/1654765821 shutdown_connections 2024-04-23T18:12:28.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/1654765821 wait complete. 2024-04-23T18:12:28.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b4b4d6700 1 Processor -- start 2024-04-23T18:12:28.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b4b4d6700 1 -- start start 2024-04-23T18:12:28.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b4b4d6700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b44102aa0 0x7f2b44198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:28.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b4b4d6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b44103ca0 0x7f2b441990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:28.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b4b4d6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b44104640 0x7f2b441a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:28.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b4b4d6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4410e510 con 0x7f2b44103ca0 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b4b4d6700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4410e390 con 0x7f2b44102aa0 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b4b4d6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4410e690 con 0x7f2b44104640 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b49272700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b44102aa0 0x7f2b44198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b49272700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b44102aa0 0x7f2b44198b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:39208/0 (socket says 172.21.15.177:39208) 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.050+0000 7f2b49272700 1 -- 172.21.15.177:0/2204910708 learned_addr learned my addr 172.21.15.177:0/2204910708 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.051+0000 7f2b48a71700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b44103ca0 0x7f2b441990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.051+0000 7f2b49272700 1 -- 172.21.15.177:0/2204910708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b44104640 msgr2=0x7f2b441a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.051+0000 7f2b49272700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b44104640 0x7f2b441a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.051+0000 7f2b49272700 1 -- 172.21.15.177:0/2204910708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b44103ca0 msgr2=0x7f2b441990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.051+0000 7f2b49272700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b44103ca0 0x7f2b441990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.051+0000 7f2b49272700 1 -- 172.21.15.177:0/2204910708 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b441a07c0 con 0x7f2b44102aa0 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.051+0000 7f2b49272700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b44102aa0 0x7f2b44198b60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2b40009f50 tx=0x7f2b40009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:28.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.051+0000 7f2b367fc700 1 -- 172.21.15.177:0/2204910708 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b40027070 con 0x7f2b44102aa0 2024-04-23T18:12:28.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.051+0000 7f2b367fc700 1 -- 172.21.15.177:0/2204910708 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b40022470 con 0x7f2b44102aa0 2024-04-23T18:12:28.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.051+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/2204910708 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b441a0a50 con 0x7f2b44102aa0 2024-04-23T18:12:28.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.051+0000 7f2b367fc700 1 -- 172.21.15.177:0/2204910708 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4000f6c0 con 0x7f2b44102aa0 2024-04-23T18:12:28.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.051+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/2204910708 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b441a0f30 con 0x7f2b44102aa0 2024-04-23T18:12:28.049 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.052+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/2204910708 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b44069730 con 0x7f2b44102aa0 2024-04-23T18:12:28.049 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.053+0000 7f2b367fc700 1 -- 172.21.15.177:0/2204910708 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2b4002a070 con 0x7f2b44102aa0 2024-04-23T18:12:28.049 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.054+0000 7f2b367fc700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2b300a3c40 0x7f2b300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:28.049 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.054+0000 7f2b367fc700 1 -- 172.21.15.177:0/2204910708 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f2b400d04b0 con 0x7f2b44102aa0 2024-04-23T18:12:28.049 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.054+0000 7f2b48a71700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2b300a3c40 0x7f2b300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:28.049 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.055+0000 7f2b48a71700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2b300a3c40 0x7f2b300a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2b38005ea0 tx=0x7f2b38005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:28.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.058+0000 7f2b367fc700 1 -- 172.21.15.177:0/2204910708 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b40031080 con 0x7f2b44102aa0 2024-04-23T18:12:28.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:27 smithi050 ceph-mon[29187]: Detected new or changed devices on smithi050 2024-04-23T18:12:28.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:27 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi050 to 8546M 2024-04-23T18:12:28.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:27 smithi050 ceph-mon[29187]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 526 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:28.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:27 smithi050 ceph-mon[29187]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T18:12:28.260 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:28.266+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/2204910708 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2b44062f40 con 0x7f2b300a3c40 2024-04-23T18:12:28.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:27 smithi177 ceph-mon[32308]: Detected new or changed devices on smithi050 2024-04-23T18:12:28.261 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:27 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi050 to 8546M 2024-04-23T18:12:28.261 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:27 smithi177 ceph-mon[32308]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 526 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:28.261 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:27 smithi177 ceph-mon[32308]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T18:12:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:27 smithi188 ceph-mon[32694]: Detected new or changed devices on smithi050 2024-04-23T18:12:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:27 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi050 to 8546M 2024-04-23T18:12:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:27 smithi188 ceph-mon[32694]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 526 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:27 smithi188 ceph-mon[32694]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T18:12:29.190 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:28 smithi177 ceph-mon[32308]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T18:12:29.190 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:28 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:29.190 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:28 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:29.190 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:28 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:29.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:28 smithi050 ceph-mon[29187]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T18:12:29.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:28 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:29.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:28 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:29.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:28 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:29.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:28 smithi188 ceph-mon[32694]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T18:12:29.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:28 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:29.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:28 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:29.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:28 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:30.085 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:29 smithi177 ceph-mon[32308]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:30.085 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:29 smithi177 ceph-mon[32308]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 525 MiB used, 357 GiB / 358 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-04-23T18:12:30.085 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:29 smithi177 ceph-mon[32308]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T18:12:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:29 smithi050 ceph-mon[29187]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:29 smithi050 ceph-mon[29187]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 525 MiB used, 357 GiB / 358 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-04-23T18:12:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:29 smithi050 ceph-mon[29187]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T18:12:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:29 smithi188 ceph-mon[32694]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:29 smithi188 ceph-mon[32694]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 525 MiB used, 357 GiB / 358 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-04-23T18:12:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:29 smithi188 ceph-mon[32694]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T18:12:32.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:31 smithi050 ceph-mon[29187]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 524 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-23T18:12:32.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:31 smithi050 ceph-mon[29187]: from='client.? 172.21.15.177:0/2723057667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88f4c3e4-72bc-459f-a052-6a41dd0a0e73"}]: dispatch 2024-04-23T18:12:32.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:31 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88f4c3e4-72bc-459f-a052-6a41dd0a0e73"}]: dispatch 2024-04-23T18:12:32.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:31 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88f4c3e4-72bc-459f-a052-6a41dd0a0e73"}]': finished 2024-04-23T18:12:32.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:31 smithi050 ceph-mon[29187]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T18:12:32.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:31 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:32.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:31 smithi050 ceph-mon[29187]: from='client.? 172.21.15.177:0/3897216178' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:32.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:31 smithi177 ceph-mon[32308]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 524 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-23T18:12:32.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:31 smithi177 ceph-mon[32308]: from='client.? 172.21.15.177:0/2723057667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88f4c3e4-72bc-459f-a052-6a41dd0a0e73"}]: dispatch 2024-04-23T18:12:32.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:31 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88f4c3e4-72bc-459f-a052-6a41dd0a0e73"}]: dispatch 2024-04-23T18:12:32.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:31 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88f4c3e4-72bc-459f-a052-6a41dd0a0e73"}]': finished 2024-04-23T18:12:32.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:31 smithi177 ceph-mon[32308]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T18:12:32.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:31 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:32.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:31 smithi177 ceph-mon[32308]: from='client.? 172.21.15.177:0/3897216178' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:32.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:31 smithi188 ceph-mon[32694]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 524 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-23T18:12:32.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:31 smithi188 ceph-mon[32694]: from='client.? 172.21.15.177:0/2723057667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88f4c3e4-72bc-459f-a052-6a41dd0a0e73"}]: dispatch 2024-04-23T18:12:32.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:31 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88f4c3e4-72bc-459f-a052-6a41dd0a0e73"}]: dispatch 2024-04-23T18:12:32.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:31 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88f4c3e4-72bc-459f-a052-6a41dd0a0e73"}]': finished 2024-04-23T18:12:32.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:31 smithi188 ceph-mon[32694]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T18:12:32.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:31 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:32.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:31 smithi188 ceph-mon[32694]: from='client.? 172.21.15.177:0/3897216178' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:34.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:33 smithi050 ceph-mon[29187]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-23T18:12:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:33 smithi177 ceph-mon[32308]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-23T18:12:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:33 smithi188 ceph-mon[32694]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-23T18:12:34.695 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:12:34 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:12:34.420+0000 7ff69ec5f700 -1 calc_pg_upmaps abort due to max <= 0 2024-04-23T18:12:36.193 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:35 smithi177 ceph-mon[32308]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:36.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:35 smithi050 ceph-mon[29187]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:35 smithi188 ceph-mon[32694]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:38.104 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:37 smithi177 ceph-mon[32308]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:38.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:37 smithi050 ceph-mon[29187]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:38.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:37 smithi188 ceph-mon[32694]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:39.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:38 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T18:12:39.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:38 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:39.265 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:38 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T18:12:39.266 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:38 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:39.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:38 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T18:12:39.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:38 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:39 smithi050 ceph-mon[29187]: Deploying daemon osd.4 on smithi177 2024-04-23T18:12:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:39 smithi050 ceph-mon[29187]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:40.267 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:39 smithi177 ceph-mon[32308]: Deploying daemon osd.4 on smithi177 2024-04-23T18:12:40.267 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:39 smithi177 ceph-mon[32308]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:40.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:39 smithi188 ceph-mon[32694]: Deploying daemon osd.4 on smithi177 2024-04-23T18:12:40.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:39 smithi188 ceph-mon[32694]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:42.237 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:41 smithi177 ceph-mon[32308]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:42.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:41 smithi188 ceph-mon[32694]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:42.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:41 smithi050 ceph-mon[29187]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:44.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:43 smithi177 ceph-mon[32308]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:44.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:44.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:44.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:43 smithi188 ceph-mon[32694]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:44.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:43 smithi050 ceph-mon[29187]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:44.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:44.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:44.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:46.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:45 smithi177 ceph-mon[32308]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:46.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:45 smithi188 ceph-mon[32694]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:46.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:45 smithi050 ceph-mon[29187]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:47.129 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 4 on host 'smithi177' 2024-04-23T18:12:47.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.133+0000 7f2b367fc700 1 -- 172.21.15.177:0/2204910708 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2b44062f40 con 0x7f2b300a3c40 2024-04-23T18:12:47.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.136+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/2204910708 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2b300a3c40 msgr2=0x7f2b300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:47.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.136+0000 7f2b4b4d6700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2b300a3c40 0x7f2b300a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2b38005ea0 tx=0x7f2b38005e30 comp rx=0 tx=0).stop 2024-04-23T18:12:47.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.136+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/2204910708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b44102aa0 msgr2=0x7f2b44198b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:47.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.136+0000 7f2b4b4d6700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b44102aa0 0x7f2b44198b60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2b40009f50 tx=0x7f2b40009510 comp rx=0 tx=0).stop 2024-04-23T18:12:47.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.136+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/2204910708 shutdown_connections 2024-04-23T18:12:47.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.136+0000 7f2b4b4d6700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2b44104640 0x7f2b441a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.136+0000 7f2b4b4d6700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2b300a3c40 0x7f2b300a6100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.136+0000 7f2b4b4d6700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b44102aa0 0x7f2b44198b60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.136+0000 7f2b4b4d6700 1 --2- 172.21.15.177:0/2204910708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b44103ca0 0x7f2b441990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:47.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.136+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/2204910708 >> 172.21.15.177:0/2204910708 conn(0x7f2b440fe040 msgr2=0x7f2b44107060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:47.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.136+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/2204910708 shutdown_connections 2024-04-23T18:12:47.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:12:47.136+0000 7f2b4b4d6700 1 -- 172.21.15.177:0/2204910708 wait complete. 2024-04-23T18:12:47.785 DEBUG:teuthology.orchestra.run.smithi177:osd.4> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.4.service 2024-04-23T18:12:47.787 INFO:tasks.cephadm:Deploying osd.5 on smithi188 with /dev/vg_nvme/lv_3... 2024-04-23T18:12:47.787 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T18:12:48.031 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:12:48.364 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:48 smithi188 ceph-mon[32694]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:48.364 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:48.364 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:48.364 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:48 smithi188 ceph-mon[32694]: from='osd.4 [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:12:48.364 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:48 smithi188 ceph-mon[32694]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:12:48.364 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:48.364 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:48.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:48 smithi050 ceph-mon[29187]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:48.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:48.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:48.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:48 smithi050 ceph-mon[29187]: from='osd.4 [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:12:48.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:48 smithi050 ceph-mon[29187]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:12:48.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:48.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:48.467 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:48 smithi177 ceph-mon[32308]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:48.467 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:48.467 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:48.467 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:48 smithi177 ceph-mon[32308]: from='osd.4 [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:12:48.467 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:48 smithi177 ceph-mon[32308]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T18:12:48.468 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:48.468 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:49.344 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:49 smithi177 ceph-mon[32308]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T18:12:49.344 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:49 smithi177 ceph-mon[32308]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T18:12:49.345 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:49.345 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:49 smithi177 ceph-mon[32308]: from='osd.4 [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:12:49.345 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:49 smithi177 ceph-mon[32308]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:12:49.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:49 smithi188 ceph-mon[32694]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T18:12:49.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:49 smithi188 ceph-mon[32694]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T18:12:49.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:49.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:49 smithi188 ceph-mon[32694]: from='osd.4 [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:12:49.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:49 smithi188 ceph-mon[32694]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:12:49.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:49 smithi050 ceph-mon[29187]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T18:12:49.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:49 smithi050 ceph-mon[29187]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T18:12:49.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:49.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:49 smithi050 ceph-mon[29187]: from='osd.4 [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:12:49.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:49 smithi050 ceph-mon[29187]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:12:49.973 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T18:12:49.974 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T18:12:49.974 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-23T18:12:49.974 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-23T18:12:49.974 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113562 s, 923 MB/s 2024-04-23T18:12:49.974 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-23T18:12:50.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:50 smithi177 ceph-mon[32308]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:50.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:50 smithi177 ceph-mon[32308]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-23T18:12:50.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:50 smithi177 ceph-mon[32308]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T18:12:50.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:50 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:50.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:50 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:50.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:50 smithi177 ceph-mon[32308]: from='osd.4 ' entity='osd.4' 2024-04-23T18:12:50.290 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:12:50 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4[43498]: 2024-04-23T18:12:50.028+0000 7f1ba4f64700 -1 osd.4 0 waiting for initial osdmap 2024-04-23T18:12:50.290 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:12:50 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4[43498]: 2024-04-23T18:12:50.175+0000 7f1b9c54e700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:12:50.366 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:50 smithi188 ceph-mon[32694]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:50.366 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:50 smithi188 ceph-mon[32694]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-23T18:12:50.366 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:50 smithi188 ceph-mon[32694]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T18:12:50.366 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:50 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:50.366 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:50 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:50.366 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:50 smithi188 ceph-mon[32694]: from='osd.4 ' entity='osd.4' 2024-04-23T18:12:50.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:50 smithi050 ceph-mon[29187]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:50.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:50 smithi050 ceph-mon[29187]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-23T18:12:50.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:50 smithi050 ceph-mon[29187]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T18:12:50.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:50 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:50.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:50 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:50.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:50 smithi050 ceph-mon[29187]: from='osd.4 ' entity='osd.4' 2024-04-23T18:12:50.653 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_3 2024-04-23T18:12:50.896 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:12:51.136 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:51 smithi177 ceph-mon[32308]: purged_snaps scrub starts 2024-04-23T18:12:51.136 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:51 smithi177 ceph-mon[32308]: purged_snaps scrub ok 2024-04-23T18:12:51.136 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:51 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:51.310 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:51 smithi188 ceph-mon[32694]: purged_snaps scrub starts 2024-04-23T18:12:51.310 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:51 smithi188 ceph-mon[32694]: purged_snaps scrub ok 2024-04-23T18:12:51.310 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:51 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:51.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:51 smithi050 ceph-mon[29187]: purged_snaps scrub starts 2024-04-23T18:12:51.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:51 smithi050 ceph-mon[29187]: purged_snaps scrub ok 2024-04-23T18:12:51.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:51 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:52.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:52.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: osd.4 [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686] boot 2024-04-23T18:12:52.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T18:12:52.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: Detected new or changed devices on smithi177 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi177 to 8544M 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:52.221 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:52.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.436 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.442+0000 7f29e27d5700 1 -- 172.21.15.188:0/2739656917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29dc1019b0 msgr2=0x7f29dc101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:52.436 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.442+0000 7f29e27d5700 1 --2- 172.21.15.188:0/2739656917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29dc1019b0 0x7f29dc101dd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009990 tx=0x7f29cc009ca0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.437 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.442+0000 7f29e27d5700 1 -- 172.21.15.188:0/2739656917 shutdown_connections 2024-04-23T18:12:52.437 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.442+0000 7f29e27d5700 1 --2- 172.21.15.188:0/2739656917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29dc103550 0x7f29dc109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.437 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.442+0000 7f29e27d5700 1 --2- 172.21.15.188:0/2739656917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29dc1019b0 0x7f29dc101dd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.437 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.442+0000 7f29e27d5700 1 --2- 172.21.15.188:0/2739656917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29dc102bb0 0x7f29dc103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.437 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.442+0000 7f29e27d5700 1 -- 172.21.15.188:0/2739656917 >> 172.21.15.188:0/2739656917 conn(0x7f29dc0fcf70 msgr2=0x7f29dc0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:12:52.437 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.442+0000 7f29e27d5700 1 -- 172.21.15.188:0/2739656917 shutdown_connections 2024-04-23T18:12:52.437 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.442+0000 7f29e27d5700 1 -- 172.21.15.188:0/2739656917 wait complete. 2024-04-23T18:12:52.437 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29e27d5700 1 Processor -- start 2024-04-23T18:12:52.437 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29e27d5700 1 -- start start 2024-04-23T18:12:52.439 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29e27d5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29dc1019b0 0x7f29dc06bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:52.439 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29e27d5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29dc102bb0 0x7f29dc19d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29e27d5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29dc103550 0x7f29dc19fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29e27d5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f29dc10d420 con 0x7f29dc103550 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29e27d5700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f29dc10d2a0 con 0x7f29dc102bb0 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29e27d5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f29dc10d5a0 con 0x7f29dc1019b0 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29dbfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29dc1019b0 0x7f29dc06bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29dbfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29dc1019b0 0x7f29dc06bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:44636/0 (socket says 172.21.15.188:44636) 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29dbfff700 1 -- 172.21.15.188:0/2591627165 learned_addr learned my addr 172.21.15.188:0/2591627165 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29e0d72700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29dc103550 0x7f29dc19fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.443+0000 7f29db7fe700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29dc102bb0 0x7f29dc19d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.444+0000 7f29dbfff700 1 -- 172.21.15.188:0/2591627165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29dc102bb0 msgr2=0x7f29dc19d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.444+0000 7f29dbfff700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29dc102bb0 0x7f29dc19d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.444+0000 7f29dbfff700 1 -- 172.21.15.188:0/2591627165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29dc103550 msgr2=0x7f29dc19fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.444+0000 7f29dbfff700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29dc103550 0x7f29dc19fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:12:52.440 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.444+0000 7f29dbfff700 1 -- 172.21.15.188:0/2591627165 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29dc1a07c0 con 0x7f29dc1019b0 2024-04-23T18:12:52.441 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.444+0000 7f29db7fe700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29dc102bb0 0x7f29dc19d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:12:52.441 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.444+0000 7f29dbfff700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29dc1019b0 0x7f29dc06bd10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f29cc0059d0 tx=0x7f29cc0039f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:52.441 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.444+0000 7f29d97fa700 1 -- 172.21.15.188:0/2591627165 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc027070 con 0x7f29dc1019b0 2024-04-23T18:12:52.441 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.444+0000 7f29e27d5700 1 -- 172.21.15.188:0/2591627165 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29dc1a0a50 con 0x7f29dc1019b0 2024-04-23T18:12:52.441 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.444+0000 7f29d97fa700 1 -- 172.21.15.188:0/2591627165 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29cc022470 con 0x7f29dc1019b0 2024-04-23T18:12:52.441 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.444+0000 7f29d97fa700 1 -- 172.21.15.188:0/2591627165 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc00f6c0 con 0x7f29dc1019b0 2024-04-23T18:12:52.441 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.444+0000 7f29e27d5700 1 -- 172.21.15.188:0/2591627165 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29dc1a0f00 con 0x7f29dc1019b0 2024-04-23T18:12:52.441 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.445+0000 7f29e27d5700 1 -- 172.21.15.188:0/2591627165 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29dc069730 con 0x7f29dc1019b0 2024-04-23T18:12:52.442 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.447+0000 7f29d97fa700 1 -- 172.21.15.188:0/2591627165 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f29cc02a070 con 0x7f29dc1019b0 2024-04-23T18:12:52.442 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.447+0000 7f29d97fa700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f29c80a3c40 0x7f29c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:12:52.442 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.448+0000 7f29d97fa700 1 -- 172.21.15.188:0/2591627165 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f29cc0d0450 con 0x7f29dc1019b0 2024-04-23T18:12:52.442 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.448+0000 7f29db7fe700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f29c80a3c40 0x7f29c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:12:52.442 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.448+0000 7f29db7fe700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f29c80a3c40 0x7f29c80a6100 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f29c4005fd0 tx=0x7f29c4005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:12:52.445 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.451+0000 7f29d97fa700 1 -- 172.21.15.188:0/2591627165 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29cc013070 con 0x7f29dc1019b0 2024-04-23T18:12:52.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:52.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: osd.4 [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686] boot 2024-04-23T18:12:52.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T18:12:52.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:52.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: Detected new or changed devices on smithi177 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi177 to 8544M 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:52.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-23T18:12:52.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: osd.4 [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686] boot 2024-04-23T18:12:52.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T18:12:52.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:12:52.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: Detected new or changed devices on smithi177 2024-04-23T18:12:52.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:52.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:12:52.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi177 to 8544M 2024-04-23T18:12:52.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:52.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:52.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:12:52.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:52.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:12:52.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:12:52.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:12:52.656+0000 7f29e27d5700 1 -- 172.21.15.188:0/2591627165 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f29dc062e80 con 0x7f29c80a3c40 2024-04-23T18:12:53.553 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:53 smithi188 ceph-mon[32694]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T18:12:53.554 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:53 smithi188 ceph-mon[32694]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:12:53.554 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:53 smithi188 ceph-mon[32694]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:53.554 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:53 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:53.554 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:53 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:53.554 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:53 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:53.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:53 smithi050 ceph-mon[29187]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T18:12:53.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:53 smithi050 ceph-mon[29187]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:12:53.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:53 smithi050 ceph-mon[29187]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:53.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:53 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:53.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:53 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:53.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:53 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:53.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:53 smithi177 ceph-mon[32308]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T18:12:53.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:53 smithi177 ceph-mon[32308]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:12:53.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:53 smithi177 ceph-mon[32308]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:12:53.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:53 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:12:53.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:53 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:12:53.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:53 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:12:55.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:55 smithi177 ceph-mon[32308]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:12:55.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:55 smithi177 ceph-mon[32308]: from='client.? 172.21.15.188:0/3592134446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fee2f23-6eec-465e-a935-970aa8ded943"}]: dispatch 2024-04-23T18:12:55.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:55 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fee2f23-6eec-465e-a935-970aa8ded943"}]: dispatch 2024-04-23T18:12:55.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:55 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fee2f23-6eec-465e-a935-970aa8ded943"}]': finished 2024-04-23T18:12:55.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:55 smithi177 ceph-mon[32308]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T18:12:55.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:55 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:12:55.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:55 smithi188 ceph-mon[32694]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:12:55.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:55 smithi188 ceph-mon[32694]: from='client.? 172.21.15.188:0/3592134446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fee2f23-6eec-465e-a935-970aa8ded943"}]: dispatch 2024-04-23T18:12:55.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:55 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fee2f23-6eec-465e-a935-970aa8ded943"}]: dispatch 2024-04-23T18:12:55.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:55 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fee2f23-6eec-465e-a935-970aa8ded943"}]': finished 2024-04-23T18:12:55.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:55 smithi188 ceph-mon[32694]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T18:12:55.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:55 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:12:55.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:55 smithi050 ceph-mon[29187]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:12:55.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:55 smithi050 ceph-mon[29187]: from='client.? 172.21.15.188:0/3592134446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fee2f23-6eec-465e-a935-970aa8ded943"}]: dispatch 2024-04-23T18:12:55.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:55 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fee2f23-6eec-465e-a935-970aa8ded943"}]: dispatch 2024-04-23T18:12:55.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:55 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fee2f23-6eec-465e-a935-970aa8ded943"}]': finished 2024-04-23T18:12:55.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:55 smithi050 ceph-mon[29187]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T18:12:55.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:55 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:12:56.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:56 smithi177 ceph-mon[32308]: from='client.? 172.21.15.188:0/1726132846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:56.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:56 smithi188 ceph-mon[32694]: from='client.? 172.21.15.188:0/1726132846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:56.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:56 smithi050 ceph-mon[29187]: from='client.? 172.21.15.188:0/1726132846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:12:57.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:57 smithi177 ceph-mon[32308]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:12:57.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:57 smithi188 ceph-mon[32694]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:12:57.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:57 smithi050 ceph-mon[29187]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:12:59.749 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:12:59 smithi188 ceph-mon[32694]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:12:59.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:12:59 smithi177 ceph-mon[32308]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:12:59.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:12:59 smithi050 ceph-mon[29187]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:01.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:01 smithi188 ceph-mon[32694]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:01.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:01 smithi177 ceph-mon[32308]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:01.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:01 smithi050 ceph-mon[29187]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:03.613 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:03 smithi188 ceph-mon[32694]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:03.613 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:03 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T18:13:03.614 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:03 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:03.614 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:03 smithi188 ceph-mon[32694]: Deploying daemon osd.5 on smithi188 2024-04-23T18:13:03.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:03 smithi177 ceph-mon[32308]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:03.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:03 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T18:13:03.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:03 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:03.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:03 smithi177 ceph-mon[32308]: Deploying daemon osd.5 on smithi188 2024-04-23T18:13:03.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:03 smithi050 ceph-mon[29187]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:03.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:03 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T18:13:03.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:03 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:03.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:03 smithi050 ceph-mon[29187]: Deploying daemon osd.5 on smithi188 2024-04-23T18:13:05.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:05 smithi177 ceph-mon[32308]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:05.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:05 smithi188 ceph-mon[32694]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:05.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:05 smithi050 ceph-mon[29187]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:07.646 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:07 smithi188 ceph-mon[32694]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:07.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:07 smithi177 ceph-mon[32308]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:07.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:07 smithi050 ceph-mon[29187]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:09.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:09 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:09.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:09 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:09.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:09 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:09.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:09 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:09.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:09 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:09.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:09 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:09.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:09 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:09.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:09 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:09.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:09 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:10.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:10 smithi188 ceph-mon[32694]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:10.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:10 smithi050 ceph-mon[29187]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:10.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:10 smithi177 ceph-mon[32308]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:11.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:11 smithi188 ceph-mon[32694]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:11.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:11 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:11.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:11 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:11.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:11 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:11.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:11 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:11.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:11 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:11.874 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.879+0000 7f29d97fa700 1 -- 172.21.15.188:0/2591627165 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f29dc062e80 con 0x7f29c80a3c40 2024-04-23T18:13:11.874 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 5 on host 'smithi188' 2024-04-23T18:13:11.875 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.881+0000 7f29e27d5700 1 -- 172.21.15.188:0/2591627165 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f29c80a3c40 msgr2=0x7f29c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.875 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.881+0000 7f29e27d5700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f29c80a3c40 0x7f29c80a6100 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f29c4005fd0 tx=0x7f29c4005e20 comp rx=0 tx=0).stop 2024-04-23T18:13:11.875 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.881+0000 7f29e27d5700 1 -- 172.21.15.188:0/2591627165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29dc1019b0 msgr2=0x7f29dc06bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:11.875 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.881+0000 7f29e27d5700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29dc1019b0 0x7f29dc06bd10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f29cc0059d0 tx=0x7f29cc0039f0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.881+0000 7f29e27d5700 1 -- 172.21.15.188:0/2591627165 shutdown_connections 2024-04-23T18:13:11.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.881+0000 7f29e27d5700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29dc103550 0x7f29dc19fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.881+0000 7f29e27d5700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29dc1019b0 0x7f29dc06bd10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.881+0000 7f29e27d5700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f29c80a3c40 0x7f29c80a6100 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.881+0000 7f29e27d5700 1 --2- 172.21.15.188:0/2591627165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29dc102bb0 0x7f29dc19d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:11.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.881+0000 7f29e27d5700 1 -- 172.21.15.188:0/2591627165 >> 172.21.15.188:0/2591627165 conn(0x7f29dc0fcf70 msgr2=0x7f29dc105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:11.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.882+0000 7f29e27d5700 1 -- 172.21.15.188:0/2591627165 shutdown_connections 2024-04-23T18:13:11.876 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:13:11.882+0000 7f29e27d5700 1 -- 172.21.15.188:0/2591627165 wait complete. 2024-04-23T18:13:12.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:11 smithi177 ceph-mon[32308]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:12.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:11 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:12.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:11 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:12.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:11 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:12.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:11 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:12.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:11 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:12.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:11 smithi050 ceph-mon[29187]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:12.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:12.550 DEBUG:teuthology.orchestra.run.smithi188:osd.5> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.5.service 2024-04-23T18:13:12.552 INFO:tasks.cephadm:Deploying osd.6 on smithi050 with /dev/vg_nvme/lv_2... 2024-04-23T18:13:12.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T18:13:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:12.935 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:12 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:12.935 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:12 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:12.935 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:12 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:12.935 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:12 smithi050 ceph-mon[29187]: from='osd.5 [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:13:12.935 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:12 smithi050 ceph-mon[29187]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:13:13.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:12 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:13.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:12 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:13.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:12 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:13.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:12 smithi188 ceph-mon[32694]: from='osd.5 [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:13:13.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:12 smithi188 ceph-mon[32694]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:13:13.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:12 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:13.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:12 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:13.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:12 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:13.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:12 smithi177 ceph-mon[32308]: from='osd.5 [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:13:13.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:12 smithi177 ceph-mon[32308]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T18:13:13.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:13 smithi188 ceph-mon[32694]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:13.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:13 smithi188 ceph-mon[32694]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T18:13:13.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:13 smithi188 ceph-mon[32694]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T18:13:13.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:13.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:13 smithi188 ceph-mon[32694]: from='osd.5 [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:13:13.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:13 smithi188 ceph-mon[32694]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:13:13.970 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:13:13 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:13:13.850+0000 7fdcf4456700 -1 osd.5 0 waiting for initial osdmap 2024-04-23T18:13:13.971 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:13:13 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:13:13.856+0000 7fdcf0249700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:13:14.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:13 smithi050 ceph-mon[29187]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:14.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:13 smithi050 ceph-mon[29187]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T18:13:14.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:13 smithi050 ceph-mon[29187]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T18:13:14.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:14.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:13 smithi050 ceph-mon[29187]: from='osd.5 [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:13:14.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:13 smithi050 ceph-mon[29187]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:13:14.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:13 smithi177 ceph-mon[32308]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:14.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:13 smithi177 ceph-mon[32308]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T18:13:14.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:13 smithi177 ceph-mon[32308]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T18:13:14.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:14.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:13 smithi177 ceph-mon[32308]: from='osd.5 [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:13:14.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:13 smithi177 ceph-mon[32308]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:13:15.019 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:14 smithi050 ceph-mon[29187]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T18:13:15.019 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:14 smithi050 ceph-mon[29187]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T18:13:15.019 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:14 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:15.019 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:14 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:15.020 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T18:13:15.020 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T18:13:15.020 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-23T18:13:15.020 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-23T18:13:15.020 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00834849 s, 1.3 GB/s 2024-04-23T18:13:15.020 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-23T18:13:15.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:14 smithi188 ceph-mon[32694]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T18:13:15.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:14 smithi188 ceph-mon[32694]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T18:13:15.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:14 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:15.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:14 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:15.189 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:14 smithi177 ceph-mon[32308]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T18:13:15.190 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:14 smithi177 ceph-mon[32308]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T18:13:15.190 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:14 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:15.190 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:14 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:15.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_2 2024-04-23T18:13:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:13:15.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: purged_snaps scrub starts 2024-04-23T18:13:15.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: purged_snaps scrub ok 2024-04-23T18:13:15.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:15.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: osd.5 [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768] boot 2024-04-23T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: osdmap e38: 6 total, 6 up, 6 in 2024-04-23T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:15 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:16.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: purged_snaps scrub starts 2024-04-23T18:13:16.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: purged_snaps scrub ok 2024-04-23T18:13:16.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: osd.5 [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768] boot 2024-04-23T18:13:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: osdmap e38: 6 total, 6 up, 6 in 2024-04-23T18:13:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:15 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:16.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: purged_snaps scrub starts 2024-04-23T18:13:16.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: purged_snaps scrub ok 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: osd.5 [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768] boot 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: osdmap e38: 6 total, 6 up, 6 in 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:15 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:17.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:16 smithi050 ceph-mon[29187]: Detected new or changed devices on smithi188 2024-04-23T18:13:17.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:16 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi188 to 8544M 2024-04-23T18:13:17.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:16 smithi050 ceph-mon[29187]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T18:13:17.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:16 smithi177 ceph-mon[32308]: Detected new or changed devices on smithi188 2024-04-23T18:13:17.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:16 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi188 to 8544M 2024-04-23T18:13:17.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:16 smithi177 ceph-mon[32308]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T18:13:17.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:16 smithi188 ceph-mon[32694]: Detected new or changed devices on smithi188 2024-04-23T18:13:17.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:16 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi188 to 8544M 2024-04-23T18:13:17.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:16 smithi188 ceph-mon[32694]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T18:13:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.599+0000 7f6fef716700 1 -- 172.21.15.50:0/3875545483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fe8101340 msgr2=0x7f6fe8101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.599+0000 7f6fef716700 1 --2- 172.21.15.50:0/3875545483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fe8101340 0x7f6fe8101760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009990 tx=0x7f6fe4009ca0 comp rx=0 tx=0).stop 2024-04-23T18:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.599+0000 7f6fef716700 1 -- 172.21.15.50:0/3875545483 shutdown_connections 2024-04-23T18:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.599+0000 7f6fef716700 1 --2- 172.21.15.50:0/3875545483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fe8102660 0x7f6fe8102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.599+0000 7f6fef716700 1 --2- 172.21.15.50:0/3875545483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fe8101340 0x7f6fe8101760 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.599+0000 7f6fef716700 1 --2- 172.21.15.50:0/3875545483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fe8101ca0 0x7f6fe8102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.599+0000 7f6fef716700 1 -- 172.21.15.50:0/3875545483 >> 172.21.15.50:0/3875545483 conn(0x7f6fe80fcf60 msgr2=0x7f6fe80ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.599+0000 7f6fef716700 1 -- 172.21.15.50:0/3875545483 shutdown_connections 2024-04-23T18:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.599+0000 7f6fef716700 1 -- 172.21.15.50:0/3875545483 wait complete. 2024-04-23T18:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fef716700 1 Processor -- start 2024-04-23T18:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fef716700 1 -- start start 2024-04-23T18:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fef716700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fe8101340 0x7f6fe819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fef716700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fe8101ca0 0x7f6fe819b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fef716700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fe8102660 0x7f6fe81a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fef716700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8110730 con 0x7f6fe8101340 2024-04-23T18:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fef716700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe81105b0 con 0x7f6fe8102660 2024-04-23T18:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fef716700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe81108b0 con 0x7f6fe8101ca0 2024-04-23T18:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fed4b2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fe8101340 0x7f6fe819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fed4b2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fe8101340 0x7f6fe819ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42978/0 (socket says 172.21.15.50:42978) 2024-04-23T18:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fed4b2700 1 -- 172.21.15.50:0/63653215 learned_addr learned my addr 172.21.15.50:0/63653215 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6feccb1700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fe8101ca0 0x7f6fe819b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fedcb3700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fe8102660 0x7f6fe81a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fed4b2700 1 -- 172.21.15.50:0/63653215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fe8101ca0 msgr2=0x7f6fe819b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fed4b2700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fe8101ca0 0x7f6fe819b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fed4b2700 1 -- 172.21.15.50:0/63653215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fe8102660 msgr2=0x7f6fe81a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fed4b2700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fe8102660 0x7f6fe81a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.600+0000 7f6fed4b2700 1 -- 172.21.15.50:0/63653215 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe81a2990 con 0x7f6fe8101340 2024-04-23T18:13:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.601+0000 7f6fed4b2700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fe8101340 0x7f6fe819ad90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4006010 tx=0x7f6fe40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.601+0000 7f6fde7fc700 1 -- 172.21.15.50:0/63653215 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4027070 con 0x7f6fe8101340 2024-04-23T18:13:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.601+0000 7f6fef716700 1 -- 172.21.15.50:0/63653215 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe81a2c20 con 0x7f6fe8101340 2024-04-23T18:13:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.601+0000 7f6fde7fc700 1 -- 172.21.15.50:0/63653215 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fe4004080 con 0x7f6fe8101340 2024-04-23T18:13:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.601+0000 7f6fde7fc700 1 -- 172.21.15.50:0/63653215 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4022e00 con 0x7f6fe8101340 2024-04-23T18:13:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.601+0000 7f6fef716700 1 -- 172.21.15.50:0/63653215 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe81a3070 con 0x7f6fe8101340 2024-04-23T18:13:17.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.602+0000 7f6fef716700 1 -- 172.21.15.50:0/63653215 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe8069730 con 0x7f6fe8101340 2024-04-23T18:13:17.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.603+0000 7f6fde7fc700 1 -- 172.21.15.50:0/63653215 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6fe402a070 con 0x7f6fe8101340 2024-04-23T18:13:17.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.604+0000 7f6fde7fc700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6fd40a3c40 0x7f6fd40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:17.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.604+0000 7f6feccb1700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6fd40a3c40 0x7f6fd40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:17.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.604+0000 7f6fde7fc700 1 -- 172.21.15.50:0/63653215 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f6fe40d03a0 con 0x7f6fe8101340 2024-04-23T18:13:17.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.604+0000 7f6feccb1700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6fd40a3c40 0x7f6fd40a6100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800ac50 tx=0x7f6fd800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:17.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.608+0000 7f6fde7fc700 1 -- 172.21.15.50:0/63653215 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6fe4013070 con 0x7f6fe8101340 2024-04-23T18:13:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:17.826+0000 7f6fef716700 1 -- 172.21.15.50:0/63653215 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6fe8062e80 con 0x7f6fd40a3c40 2024-04-23T18:13:18.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:17 smithi050 ceph-mon[29187]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:18.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:17 smithi050 ceph-mon[29187]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T18:13:18.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:17 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:18.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:17 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:18.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:17 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:18.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:17 smithi177 ceph-mon[32308]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:18.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:17 smithi177 ceph-mon[32308]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T18:13:18.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:17 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:18.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:17 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:18.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:17 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:18.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:17 smithi188 ceph-mon[32694]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:18.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:17 smithi188 ceph-mon[32694]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T18:13:18.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:17 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:18.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:17 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:18.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:17 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:19.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:18 smithi050 ceph-mon[29187]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:19.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:18 smithi050 ceph-mon[29187]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T18:13:19.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:18 smithi177 ceph-mon[32308]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:19.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:18 smithi177 ceph-mon[32308]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T18:13:19.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:18 smithi188 ceph-mon[32694]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:19.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:18 smithi188 ceph-mon[32694]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T18:13:20.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:19 smithi050 ceph-mon[29187]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:13:20.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:19 smithi050 ceph-mon[29187]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T18:13:20.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:19 smithi177 ceph-mon[32308]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:13:20.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:19 smithi177 ceph-mon[32308]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T18:13:20.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:19 smithi188 ceph-mon[32694]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:13:20.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:19 smithi188 ceph-mon[32694]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T18:13:20.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:20 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2293434817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e"}]: dispatch 2024-04-23T18:13:20.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:20 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2293434817' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e"}]': finished 2024-04-23T18:13:20.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:20 smithi050 ceph-mon[29187]: osdmap e43: 7 total, 6 up, 7 in 2024-04-23T18:13:20.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:20 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:21.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:20 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2293434817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e"}]: dispatch 2024-04-23T18:13:21.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:20 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2293434817' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e"}]': finished 2024-04-23T18:13:21.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:20 smithi177 ceph-mon[32308]: osdmap e43: 7 total, 6 up, 7 in 2024-04-23T18:13:21.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:20 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:21.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:20 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2293434817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e"}]: dispatch 2024-04-23T18:13:21.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:20 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2293434817' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e"}]': finished 2024-04-23T18:13:21.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:20 smithi188 ceph-mon[32694]: osdmap e43: 7 total, 6 up, 7 in 2024-04-23T18:13:21.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:20 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:22.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:21 smithi050 ceph-mon[29187]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-23T18:13:22.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:21 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2250473064' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:22.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:21 smithi177 ceph-mon[32308]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-23T18:13:22.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:21 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2250473064' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:22.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:21 smithi188 ceph-mon[32694]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-23T18:13:22.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:21 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2250473064' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:24.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:23 smithi050 ceph-mon[29187]: pgmap v147: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-23T18:13:24.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:23 smithi177 ceph-mon[32308]: pgmap v147: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-23T18:13:24.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:23 smithi188 ceph-mon[32694]: pgmap v147: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-23T18:13:25.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:25 smithi050 ceph-mon[29187]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:26.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:25 smithi177 ceph-mon[32308]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:26.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:25 smithi188 ceph-mon[32694]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:27.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:27 smithi050 ceph-mon[29187]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:28.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:27 smithi177 ceph-mon[32308]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:27 smithi188 ceph-mon[32694]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:29.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:28 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T18:13:29.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:28 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:29.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:28 smithi050 ceph-mon[29187]: Deploying daemon osd.6 on smithi050 2024-04-23T18:13:29.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:28 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T18:13:29.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:28 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:29.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:28 smithi177 ceph-mon[32308]: Deploying daemon osd.6 on smithi050 2024-04-23T18:13:29.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:28 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T18:13:29.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:28 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:29.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:28 smithi188 ceph-mon[32694]: Deploying daemon osd.6 on smithi050 2024-04-23T18:13:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:29 smithi050 ceph-mon[29187]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:30.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:29 smithi177 ceph-mon[32308]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:29 smithi188 ceph-mon[32694]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:32.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:31 smithi050 ceph-mon[29187]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:32.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:31 smithi177 ceph-mon[32308]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:32.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:31 smithi188 ceph-mon[32694]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:34.162 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:33 smithi050 ceph-mon[29187]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:34.162 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:34.162 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:34.162 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:33 smithi177 ceph-mon[32308]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:33 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:33 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:33 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:33 smithi188 ceph-mon[32694]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:36.193 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:35 smithi050 ceph-mon[29187]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:35 smithi177 ceph-mon[32308]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:35 smithi188 ceph-mon[32694]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:37.429 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 6 on host 'smithi050' 2024-04-23T18:13:37.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.434+0000 7f6fde7fc700 1 -- 172.21.15.50:0/63653215 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6fe8062e80 con 0x7f6fd40a3c40 2024-04-23T18:13:37.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.437+0000 7f6fef716700 1 -- 172.21.15.50:0/63653215 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6fd40a3c40 msgr2=0x7f6fd40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:37.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.437+0000 7f6fef716700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6fd40a3c40 0x7f6fd40a6100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800ac50 tx=0x7f6fd800a380 comp rx=0 tx=0).stop 2024-04-23T18:13:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.437+0000 7f6fef716700 1 -- 172.21.15.50:0/63653215 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fe8101340 msgr2=0x7f6fe819ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.437+0000 7f6fef716700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fe8101340 0x7f6fe819ad90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4006010 tx=0x7f6fe40047b0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.437+0000 7f6fef716700 1 -- 172.21.15.50:0/63653215 shutdown_connections 2024-04-23T18:13:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.437+0000 7f6fef716700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6fd40a3c40 0x7f6fd40a6100 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.437+0000 7f6fef716700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fe8102660 0x7f6fe81a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.437+0000 7f6fef716700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fe8101340 0x7f6fe819ad90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.437+0000 7f6fef716700 1 --2- 172.21.15.50:0/63653215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fe8101ca0 0x7f6fe819b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:37.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.437+0000 7f6fef716700 1 -- 172.21.15.50:0/63653215 >> 172.21.15.50:0/63653215 conn(0x7f6fe80fcf60 msgr2=0x7f6fe80fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:37.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.437+0000 7f6fef716700 1 -- 172.21.15.50:0/63653215 shutdown_connections 2024-04-23T18:13:37.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:13:37.437+0000 7f6fef716700 1 -- 172.21.15.50:0/63653215 wait complete. 2024-04-23T18:13:37.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:37 smithi050 ceph-mon[29187]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:37.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:37.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:37.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:37 smithi050 ceph-mon[29187]: from='osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T18:13:37.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:37 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:37.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:37 smithi177 ceph-mon[32308]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:37.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:37.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:37.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:37 smithi177 ceph-mon[32308]: from='osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T18:13:37.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:37 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:37.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:37 smithi188 ceph-mon[32694]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:37.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:37.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:37.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:37 smithi188 ceph-mon[32694]: from='osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T18:13:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:37.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:37 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:38.055 DEBUG:teuthology.orchestra.run.smithi050:osd.6> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.6.service 2024-04-23T18:13:38.058 INFO:tasks.cephadm:Deploying osd.7 on smithi177 with /dev/vg_nvme/lv_2... 2024-04-23T18:13:38.058 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T18:13:38.303 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:13:39.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:38 smithi177 ceph-mon[32308]: from='osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T18:13:39.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:38 smithi177 ceph-mon[32308]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T18:13:39.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:38 smithi177 ceph-mon[32308]: from='osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:13:39.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:38 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:39.170 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:38 smithi050 ceph-mon[29187]: from='osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T18:13:39.170 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:38 smithi050 ceph-mon[29187]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T18:13:39.170 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:38 smithi050 ceph-mon[29187]: from='osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:13:39.170 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:38 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:39.170 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:13:38 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:13:38.918+0000 7f2b2ed2c700 -1 osd.6 0 waiting for initial osdmap 2024-04-23T18:13:39.171 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:13:38 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:13:38.928+0000 7f2b2ab1f700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:13:39.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:38 smithi188 ceph-mon[32694]: from='osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T18:13:39.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:38 smithi188 ceph-mon[32694]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T18:13:39.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:38 smithi188 ceph-mon[32694]: from='osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:13:39.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:38 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:40.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:39 smithi177 ceph-mon[32308]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:40.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:39 smithi177 ceph-mon[32308]: from='osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-23T18:13:40.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:39 smithi177 ceph-mon[32308]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T18:13:40.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:40.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:39 smithi050 ceph-mon[29187]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:39 smithi050 ceph-mon[29187]: from='osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-23T18:13:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:39 smithi050 ceph-mon[29187]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T18:13:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:40.315 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T18:13:40.315 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T18:13:40.315 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-23T18:13:40.315 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-23T18:13:40.315 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111687 s, 939 MB/s 2024-04-23T18:13:40.315 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-23T18:13:40.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:39 smithi188 ceph-mon[32694]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 191 MiB used, 536 GiB / 536 GiB avail 2024-04-23T18:13:40.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:39 smithi188 ceph-mon[32694]: from='osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-23T18:13:40.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:39 smithi188 ceph-mon[32694]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T18:13:40.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:40.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:40.895 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_2 2024-04-23T18:13:41.151 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:13:41.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:40 smithi050 ceph-mon[29187]: purged_snaps scrub starts 2024-04-23T18:13:41.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:40 smithi050 ceph-mon[29187]: purged_snaps scrub ok 2024-04-23T18:13:41.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:40 smithi050 ceph-mon[29187]: osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415] boot 2024-04-23T18:13:41.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:40 smithi050 ceph-mon[29187]: osdmap e46: 7 total, 7 up, 7 in 2024-04-23T18:13:41.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:40 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:41.273 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:40 smithi177 ceph-mon[32308]: purged_snaps scrub starts 2024-04-23T18:13:41.274 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:40 smithi177 ceph-mon[32308]: purged_snaps scrub ok 2024-04-23T18:13:41.274 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:40 smithi177 ceph-mon[32308]: osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415] boot 2024-04-23T18:13:41.274 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:40 smithi177 ceph-mon[32308]: osdmap e46: 7 total, 7 up, 7 in 2024-04-23T18:13:41.274 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:40 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:41.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:40 smithi188 ceph-mon[32694]: purged_snaps scrub starts 2024-04-23T18:13:41.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:40 smithi188 ceph-mon[32694]: purged_snaps scrub ok 2024-04-23T18:13:41.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:40 smithi188 ceph-mon[32694]: osd.6 [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415] boot 2024-04-23T18:13:41.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:40 smithi188 ceph-mon[32694]: osdmap e46: 7 total, 7 up, 7 in 2024-04-23T18:13:41.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:40 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:13:42.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:42.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T18:13:42.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: Detected new or changed devices on smithi050 2024-04-23T18:13:42.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:42.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi050 to 5697M 2024-04-23T18:13:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:42.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:41 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:42.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:42.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T18:13:42.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: Detected new or changed devices on smithi050 2024-04-23T18:13:42.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:42.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:42.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:42.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:42.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:42.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi050 to 5697M 2024-04-23T18:13:42.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:42.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:42.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:42.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:41 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:42.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:42.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T18:13:42.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: Detected new or changed devices on smithi050 2024-04-23T18:13:42.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:42.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:42.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:42.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:42.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:13:42.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi050 to 5697M 2024-04-23T18:13:42.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:42.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:42.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:13:42.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:41 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:42.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.777+0000 7fafbeac8700 1 -- 172.21.15.177:0/3638153980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb81035b0 msgr2=0x7fafb81059a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.777+0000 7fafbeac8700 1 --2- 172.21.15.177:0/3638153980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb81035b0 0x7fafb81059a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fafb400b0a0 tx=0x7fafb400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.777+0000 7fafbeac8700 1 -- 172.21.15.177:0/3638153980 shutdown_connections 2024-04-23T18:13:42.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.777+0000 7fafbeac8700 1 --2- 172.21.15.177:0/3638153980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafb8108810 0x7fafb810ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.777+0000 7fafbeac8700 1 --2- 172.21.15.177:0/3638153980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb81035b0 0x7fafb81059a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.777+0000 7fafbeac8700 1 --2- 172.21.15.177:0/3638153980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafb8105ee0 0x7fafb81082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.777+0000 7fafbeac8700 1 -- 172.21.15.177:0/3638153980 >> 172.21.15.177:0/3638153980 conn(0x7fafb80fcf40 msgr2=0x7fafb80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:13:42.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.777+0000 7fafbeac8700 1 -- 172.21.15.177:0/3638153980 shutdown_connections 2024-04-23T18:13:42.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.777+0000 7fafbeac8700 1 -- 172.21.15.177:0/3638153980 wait complete. 2024-04-23T18:13:42.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.777+0000 7fafbeac8700 1 Processor -- start 2024-04-23T18:13:42.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafbeac8700 1 -- start start 2024-04-23T18:13:42.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafbeac8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafb81035b0 0x7fafb8198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafbeac8700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb8105ee0 0x7fafb81990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafbeac8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafb8108810 0x7fafb81a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafbeac8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fafb810e570 con 0x7fafb8108810 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafbeac8700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fafb810e3f0 con 0x7fafb8105ee0 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafbeac8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fafb810e6f0 con 0x7fafb81035b0 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafaffff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb8105ee0 0x7fafb81990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafaffff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb8105ee0 0x7fafb81990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:33858/0 (socket says 172.21.15.177:33858) 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafaffff700 1 -- 172.21.15.177:0/2642738876 learned_addr learned my addr 172.21.15.177:0/2642738876 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafbd065700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafb8108810 0x7fafb81a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafbc864700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafb81035b0 0x7fafb8198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafaffff700 1 -- 172.21.15.177:0/2642738876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafb81035b0 msgr2=0x7fafb8198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafaffff700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafb81035b0 0x7fafb8198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafaffff700 1 -- 172.21.15.177:0/2642738876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafb8108810 msgr2=0x7fafb81a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:13:42.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafaffff700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafb8108810 0x7fafb81a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:13:42.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafaffff700 1 -- 172.21.15.177:0/2642738876 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb81a07e0 con 0x7fafb8105ee0 2024-04-23T18:13:42.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafaffff700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb8105ee0 0x7fafb81990c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafa800c9b0 tx=0x7fafa800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:42.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.778+0000 7fafadffb700 1 -- 172.21.15.177:0/2642738876 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8007b00 con 0x7fafb8105ee0 2024-04-23T18:13:42.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.779+0000 7fafadffb700 1 -- 172.21.15.177:0/2642738876 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafa8007ca0 con 0x7fafb8105ee0 2024-04-23T18:13:42.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.779+0000 7fafbeac8700 1 -- 172.21.15.177:0/2642738876 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb81a0ad0 con 0x7fafb8105ee0 2024-04-23T18:13:42.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.779+0000 7fafadffb700 1 -- 172.21.15.177:0/2642738876 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8018660 con 0x7fafb8105ee0 2024-04-23T18:13:42.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.779+0000 7fafbeac8700 1 -- 172.21.15.177:0/2642738876 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb81a1010 con 0x7fafb8105ee0 2024-04-23T18:13:42.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.780+0000 7fafbeac8700 1 -- 172.21.15.177:0/2642738876 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafb8069730 con 0x7fafb8105ee0 2024-04-23T18:13:42.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.781+0000 7fafadffb700 1 -- 172.21.15.177:0/2642738876 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fafa800f450 con 0x7fafb8105ee0 2024-04-23T18:13:42.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.781+0000 7fafadffb700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fafa00a3c40 0x7fafa00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:13:42.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.781+0000 7fafadffb700 1 -- 172.21.15.177:0/2642738876 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fafa80cf6a0 con 0x7fafb8105ee0 2024-04-23T18:13:42.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.781+0000 7fafbc864700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fafa00a3c40 0x7fafa00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:13:42.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.782+0000 7fafbc864700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fafa00a3c40 0x7fafa00a6100 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009fd0 tx=0x7fafb400b7d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:13:42.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.786+0000 7fafadffb700 1 -- 172.21.15.177:0/2642738876 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fafa8014030 con 0x7fafb8105ee0 2024-04-23T18:13:42.988 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:13:42.994+0000 7fafbeac8700 1 -- 172.21.15.177:0/2642738876 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fafb8062e80 con 0x7fafa00a3c40 2024-04-23T18:13:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:43 smithi050 ceph-mon[29187]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:43 smithi050 ceph-mon[29187]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:43 smithi177 ceph-mon[32308]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:43 smithi177 ceph-mon[32308]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:44.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:44.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:43 smithi188 ceph-mon[32694]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:43 smithi188 ceph-mon[32694]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:13:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:13:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:13:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:46.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:45 smithi050 ceph-mon[29187]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:46.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:45 smithi050 ceph-mon[29187]: from='client.? 172.21.15.177:0/1025861968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6d24ee9-5db8-439b-907e-c5f2c7c4658a"}]: dispatch 2024-04-23T18:13:46.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:45 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6d24ee9-5db8-439b-907e-c5f2c7c4658a"}]: dispatch 2024-04-23T18:13:46.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:45 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6d24ee9-5db8-439b-907e-c5f2c7c4658a"}]': finished 2024-04-23T18:13:46.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:45 smithi050 ceph-mon[29187]: osdmap e48: 8 total, 7 up, 8 in 2024-04-23T18:13:46.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:45 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:13:46.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:45 smithi177 ceph-mon[32308]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:46.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:45 smithi177 ceph-mon[32308]: from='client.? 172.21.15.177:0/1025861968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6d24ee9-5db8-439b-907e-c5f2c7c4658a"}]: dispatch 2024-04-23T18:13:46.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:45 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6d24ee9-5db8-439b-907e-c5f2c7c4658a"}]: dispatch 2024-04-23T18:13:46.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:45 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6d24ee9-5db8-439b-907e-c5f2c7c4658a"}]': finished 2024-04-23T18:13:46.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:45 smithi177 ceph-mon[32308]: osdmap e48: 8 total, 7 up, 8 in 2024-04-23T18:13:46.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:45 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:13:46.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:45 smithi188 ceph-mon[32694]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:46.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:45 smithi188 ceph-mon[32694]: from='client.? 172.21.15.177:0/1025861968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6d24ee9-5db8-439b-907e-c5f2c7c4658a"}]: dispatch 2024-04-23T18:13:46.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:45 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6d24ee9-5db8-439b-907e-c5f2c7c4658a"}]: dispatch 2024-04-23T18:13:46.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:45 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6d24ee9-5db8-439b-907e-c5f2c7c4658a"}]': finished 2024-04-23T18:13:46.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:45 smithi188 ceph-mon[32694]: osdmap e48: 8 total, 7 up, 8 in 2024-04-23T18:13:46.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:45 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:13:47.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:46 smithi050 ceph-mon[29187]: from='client.? 172.21.15.177:0/704246499' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:47.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:46 smithi177 ceph-mon[32308]: from='client.? 172.21.15.177:0/704246499' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:47.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:46 smithi188 ceph-mon[32694]: from='client.? 172.21.15.177:0/704246499' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:13:48.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:47 smithi050 ceph-mon[29187]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:48.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:47 smithi177 ceph-mon[32308]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:48.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:47 smithi188 ceph-mon[32694]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:50.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:49 smithi050 ceph-mon[29187]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:50.207 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:49 smithi177 ceph-mon[32308]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:50.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:49 smithi188 ceph-mon[32694]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:52.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:51 smithi177 ceph-mon[32308]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:52.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:51 smithi050 ceph-mon[29187]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:52.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:51 smithi188 ceph-mon[32694]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:53.999 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:53 smithi177 ceph-mon[32308]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:53.999 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:53 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T18:13:53.999 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:53 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:53.999 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:53 smithi177 ceph-mon[32308]: Deploying daemon osd.7 on smithi177 2024-04-23T18:13:54.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:53 smithi050 ceph-mon[29187]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:54.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:53 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T18:13:54.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:53 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:54.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:53 smithi050 ceph-mon[29187]: Deploying daemon osd.7 on smithi177 2024-04-23T18:13:54.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:53 smithi188 ceph-mon[32694]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:54.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:53 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T18:13:54.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:53 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:13:54.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:53 smithi188 ceph-mon[32694]: Deploying daemon osd.7 on smithi177 2024-04-23T18:13:56.148 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:55 smithi177 ceph-mon[32308]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:56.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:55 smithi050 ceph-mon[29187]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:55 smithi188 ceph-mon[32694]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:58.172 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:57 smithi177 ceph-mon[32308]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:58.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:57 smithi050 ceph-mon[29187]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:58.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:57 smithi188 ceph-mon[32694]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:59.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:59 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:59.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:59 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:59.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:59 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:59.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:13:59 smithi050 ceph-mon[29187]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:59.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:59 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:59.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:59 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:59.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:59 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:59.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:13:59 smithi177 ceph-mon[32308]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:13:59.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:59 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:59.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:59 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:13:59.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:59 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:13:59.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:13:59 smithi188 ceph-mon[32694]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:01.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:01 smithi050 ceph-mon[29187]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:01.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:01.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:01.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:01.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:01.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:01.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:01 smithi177 ceph-mon[32308]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:01.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:01.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:01.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:01.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:01.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:01.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:01 smithi188 ceph-mon[32694]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:01.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:01.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:01.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:01.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:01.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:02.206 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.211+0000 7fafadffb700 1 -- 172.21.15.177:0/2642738876 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fafb8062e80 con 0x7fafa00a3c40 2024-04-23T18:14:02.206 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 7 on host 'smithi177' 2024-04-23T18:14:02.208 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.214+0000 7fafbeac8700 1 -- 172.21.15.177:0/2642738876 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fafa00a3c40 msgr2=0x7fafa00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:02.208 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.214+0000 7fafbeac8700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fafa00a3c40 0x7fafa00a6100 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009fd0 tx=0x7fafb400b7d0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.208 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.214+0000 7fafbeac8700 1 -- 172.21.15.177:0/2642738876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb8105ee0 msgr2=0x7fafb81990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:02.208 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.214+0000 7fafbeac8700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb8105ee0 0x7fafb81990c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafa800c9b0 tx=0x7fafa800ccc0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.208 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.214+0000 7fafbeac8700 1 -- 172.21.15.177:0/2642738876 shutdown_connections 2024-04-23T18:14:02.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.214+0000 7fafbeac8700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafb8108810 0x7fafb81a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.214+0000 7fafbeac8700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafb81035b0 0x7fafb8198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.214+0000 7fafbeac8700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fafa00a3c40 0x7fafa00a6100 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.214+0000 7fafbeac8700 1 --2- 172.21.15.177:0/2642738876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb8105ee0 0x7fafb81990c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:02.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.214+0000 7fafbeac8700 1 -- 172.21.15.177:0/2642738876 >> 172.21.15.177:0/2642738876 conn(0x7fafb80fcf40 msgr2=0x7fafb8103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:02.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.214+0000 7fafbeac8700 1 -- 172.21.15.177:0/2642738876 shutdown_connections 2024-04-23T18:14:02.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:02.214+0000 7fafbeac8700 1 -- 172.21.15.177:0/2642738876 wait complete. 2024-04-23T18:14:02.842 DEBUG:teuthology.orchestra.run.smithi177:osd.7> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.7.service 2024-04-23T18:14:02.845 INFO:tasks.cephadm:Deploying osd.8 on smithi188 with /dev/vg_nvme/lv_2... 2024-04-23T18:14:02.845 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T18:14:03.089 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:14:03.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:03 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:03.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:03 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:03.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:03 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:03.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:03 smithi188 ceph-mon[32694]: from='osd.7 [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:14:03.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:03 smithi188 ceph-mon[32694]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:14:03.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:03 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:03.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:03 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:03.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:03 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:03.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:03 smithi050 ceph-mon[29187]: from='osd.7 [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:14:03.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:03 smithi050 ceph-mon[29187]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:14:03.533 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:03 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:03.533 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:03 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:03.534 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:03 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:03.534 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:03 smithi177 ceph-mon[32308]: from='osd.7 [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:14:03.534 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:03 smithi177 ceph-mon[32308]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T18:14:04.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:04 smithi188 ceph-mon[32694]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:04.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:04 smithi188 ceph-mon[32694]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T18:14:04.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:04 smithi188 ceph-mon[32694]: from='osd.7 [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:14:04.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:04 smithi188 ceph-mon[32694]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T18:14:04.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:04.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:04 smithi188 ceph-mon[32694]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:14:04.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:04 smithi050 ceph-mon[29187]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:04.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:04 smithi050 ceph-mon[29187]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T18:14:04.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:04 smithi050 ceph-mon[29187]: from='osd.7 [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:14:04.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:04 smithi050 ceph-mon[29187]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T18:14:04.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:04.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:04 smithi050 ceph-mon[29187]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:14:04.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:04 smithi177 ceph-mon[32308]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:04.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:04 smithi177 ceph-mon[32308]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T18:14:04.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:04 smithi177 ceph-mon[32308]: from='osd.7 [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:14:04.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:04 smithi177 ceph-mon[32308]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T18:14:04.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:04.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:04 smithi177 ceph-mon[32308]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:14:04.936 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T18:14:04.936 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T18:14:04.936 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-23T18:14:04.936 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-23T18:14:04.936 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133022 s, 788 MB/s 2024-04-23T18:14:04.936 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-23T18:14:05.300 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:05 smithi177 ceph-mon[32308]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-23T18:14:05.300 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:05 smithi177 ceph-mon[32308]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T18:14:05.300 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:05 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:05.300 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:05 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:05.300 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:05 smithi177 ceph-mon[32308]: from='osd.7 ' entity='osd.7' 2024-04-23T18:14:05.301 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:14:05 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:14:05.111+0000 7fe554c31700 -1 osd.7 0 waiting for initial osdmap 2024-04-23T18:14:05.301 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:14:05 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:14:05.306+0000 7fe54ca1c700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:14:05.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:05 smithi050 ceph-mon[29187]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-23T18:14:05.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:05 smithi050 ceph-mon[29187]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T18:14:05.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:05 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:05.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:05 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:05.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:05 smithi050 ceph-mon[29187]: from='osd.7 ' entity='osd.7' 2024-04-23T18:14:05.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:05 smithi188 ceph-mon[32694]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-23T18:14:05.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:05 smithi188 ceph-mon[32694]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T18:14:05.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:05 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:05.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:05 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:05.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:05 smithi188 ceph-mon[32694]: from='osd.7 ' entity='osd.7' 2024-04-23T18:14:05.556 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_2 2024-04-23T18:14:05.803 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:14:06.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: purged_snaps scrub starts 2024-04-23T18:14:06.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: purged_snaps scrub ok 2024-04-23T18:14:06.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:06.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:06 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:06.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: purged_snaps scrub starts 2024-04-23T18:14:06.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: purged_snaps scrub ok 2024-04-23T18:14:06.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:06.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:06 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:06.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: purged_snaps scrub starts 2024-04-23T18:14:06.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: purged_snaps scrub ok 2024-04-23T18:14:06.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:06.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T18:14:06.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:06.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:06.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:06.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:06.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:06.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:06.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:06.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:06.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:06.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:06.585 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:06.585 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:06 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:07.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.381+0000 7f5499a29700 1 -- 172.21.15.188:0/3739752081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54941019e0 msgr2=0x7f5494101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:07.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.381+0000 7f5499a29700 1 --2- 172.21.15.188:0/3739752081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54941019e0 0x7f5494101e00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f548c009990 tx=0x7f548c009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.381+0000 7f5499a29700 1 -- 172.21.15.188:0/3739752081 shutdown_connections 2024-04-23T18:14:07.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.381+0000 7f5499a29700 1 --2- 172.21.15.188:0/3739752081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5494103580 0x7f5494109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.381+0000 7f5499a29700 1 --2- 172.21.15.188:0/3739752081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54941019e0 0x7f5494101e00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.381+0000 7f5499a29700 1 --2- 172.21.15.188:0/3739752081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5494102be0 0x7f5494103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.381+0000 7f5499a29700 1 -- 172.21.15.188:0/3739752081 >> 172.21.15.188:0/3739752081 conn(0x7f54940fcfa0 msgr2=0x7f54940ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:07.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.381+0000 7f5499a29700 1 -- 172.21.15.188:0/3739752081 shutdown_connections 2024-04-23T18:14:07.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.381+0000 7f5499a29700 1 -- 172.21.15.188:0/3739752081 wait complete. 2024-04-23T18:14:07.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.381+0000 7f5499a29700 1 Processor -- start 2024-04-23T18:14:07.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f5499a29700 1 -- start start 2024-04-23T18:14:07.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f5499a29700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54941019e0 0x7f549406bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:07.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f5499a29700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5494102be0 0x7f549419d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:07.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f5499a29700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5494103580 0x7f549419ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:07.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f5499a29700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f549410d450 con 0x7f5494103580 2024-04-23T18:14:07.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f5499a29700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f549410d2d0 con 0x7f54941019e0 2024-04-23T18:14:07.379 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f5499a29700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f549410d5d0 con 0x7f5494102be0 2024-04-23T18:14:07.379 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f54927fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5494102be0 0x7f549419d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:07.379 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f54927fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5494102be0 0x7f549419d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:41240/0 (socket says 172.21.15.188:41240) 2024-04-23T18:14:07.379 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f54927fc700 1 -- 172.21.15.188:0/3778435560 learned_addr learned my addr 172.21.15.188:0/3778435560 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-23T18:14:07.379 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f5492ffd700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54941019e0 0x7f549406bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:07.379 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f54937fe700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5494103580 0x7f549419ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:07.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f54927fc700 1 -- 172.21.15.188:0/3778435560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54941019e0 msgr2=0x7f549406bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:07.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f54927fc700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54941019e0 0x7f549406bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f54927fc700 1 -- 172.21.15.188:0/3778435560 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5494103580 msgr2=0x7f549419ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:07.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f54927fc700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5494103580 0x7f549419ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:07.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f54927fc700 1 -- 172.21.15.188:0/3778435560 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54941a0840 con 0x7f5494102be0 2024-04-23T18:14:07.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f54927fc700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5494102be0 0x7f549419d930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f547c00c920 tx=0x7f547c00cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:07.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.382+0000 7f5498a27700 1 -- 172.21.15.188:0/3778435560 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547c007b00 con 0x7f5494102be0 2024-04-23T18:14:07.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.383+0000 7f5498a27700 1 -- 172.21.15.188:0/3778435560 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f547c004ce0 con 0x7f5494102be0 2024-04-23T18:14:07.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.383+0000 7f5499a29700 1 -- 172.21.15.188:0/3778435560 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54941a0b30 con 0x7f5494102be0 2024-04-23T18:14:07.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.383+0000 7f5498a27700 1 -- 172.21.15.188:0/3778435560 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547c005650 con 0x7f5494102be0 2024-04-23T18:14:07.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.383+0000 7f5499a29700 1 -- 172.21.15.188:0/3778435560 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54941a1070 con 0x7f5494102be0 2024-04-23T18:14:07.381 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.384+0000 7f5499a29700 1 -- 172.21.15.188:0/3778435560 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5494069730 con 0x7f5494102be0 2024-04-23T18:14:07.381 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.385+0000 7f5498a27700 1 -- 172.21.15.188:0/3778435560 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f547c004750 con 0x7f5494102be0 2024-04-23T18:14:07.382 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.385+0000 7f5498a27700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f54800a3c40 0x7f54800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:07.382 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.385+0000 7f5492ffd700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f54800a3c40 0x7f54800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:07.382 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.385+0000 7f5498a27700 1 -- 172.21.15.188:0/3778435560 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f547c0cf4c0 con 0x7f5494102be0 2024-04-23T18:14:07.382 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.386+0000 7f5492ffd700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f54800a3c40 0x7f54800a6100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f548c00b5c0 tx=0x7f548c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:07.383 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.390+0000 7f5498a27700 1 -- 172.21.15.188:0/3778435560 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f547c014030 con 0x7f5494102be0 2024-04-23T18:14:07.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:07 smithi177 ceph-mon[32308]: Detected new or changed devices on smithi177 2024-04-23T18:14:07.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:07 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi177 to 5696M 2024-04-23T18:14:07.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:07 smithi177 ceph-mon[32308]: osd.7 [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913] boot 2024-04-23T18:14:07.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:07 smithi177 ceph-mon[32308]: osdmap e52: 8 total, 8 up, 8 in 2024-04-23T18:14:07.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:07 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:07.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:07 smithi188 ceph-mon[32694]: Detected new or changed devices on smithi177 2024-04-23T18:14:07.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:07 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi177 to 5696M 2024-04-23T18:14:07.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:07 smithi188 ceph-mon[32694]: osd.7 [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913] boot 2024-04-23T18:14:07.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:07 smithi188 ceph-mon[32694]: osdmap e52: 8 total, 8 up, 8 in 2024-04-23T18:14:07.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:07 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:07.591 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:07.598+0000 7f5499a29700 1 -- 172.21.15.188:0/3778435560 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5494062e80 con 0x7f54800a3c40 2024-04-23T18:14:07.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:07 smithi050 ceph-mon[29187]: Detected new or changed devices on smithi177 2024-04-23T18:14:07.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:07 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi177 to 5696M 2024-04-23T18:14:07.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:07 smithi050 ceph-mon[29187]: osd.7 [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913] boot 2024-04-23T18:14:07.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:07 smithi050 ceph-mon[29187]: osdmap e52: 8 total, 8 up, 8 in 2024-04-23T18:14:07.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:07 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:14:08.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:08 smithi188 ceph-mon[32694]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:08.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:08 smithi188 ceph-mon[32694]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T18:14:08.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:08 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:14:08.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:08 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:14:08.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:08 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:08 smithi177 ceph-mon[32308]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:08 smithi177 ceph-mon[32308]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T18:14:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:08 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:14:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:08 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:14:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:08 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:08.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:08 smithi050 ceph-mon[29187]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-23T18:14:08.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:08 smithi050 ceph-mon[29187]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T18:14:08.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:08 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:14:08.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:08 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:14:08.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:08 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:09.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:09 smithi188 ceph-mon[32694]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:09.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:09 smithi188 ceph-mon[32694]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T18:14:09.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:09 smithi177 ceph-mon[32308]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:09.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:09 smithi177 ceph-mon[32308]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T18:14:09.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:09 smithi050 ceph-mon[29187]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:09.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:09 smithi050 ceph-mon[29187]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T18:14:10.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:10 smithi177 ceph-mon[32308]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:10.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:10 smithi177 ceph-mon[32308]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T18:14:10.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:10 smithi188 ceph-mon[32694]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:10.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:10 smithi188 ceph-mon[32694]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T18:14:10.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:10 smithi050 ceph-mon[29187]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:10.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:10 smithi050 ceph-mon[29187]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T18:14:11.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:11 smithi177 ceph-mon[32308]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T18:14:11.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:11 smithi177 ceph-mon[32308]: from='client.? 172.21.15.188:0/392998675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb3e8deb-e2b5-479b-aed3-5decbd2b9044"}]: dispatch 2024-04-23T18:14:11.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:11 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb3e8deb-e2b5-479b-aed3-5decbd2b9044"}]: dispatch 2024-04-23T18:14:11.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:11 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb3e8deb-e2b5-479b-aed3-5decbd2b9044"}]': finished 2024-04-23T18:14:11.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:11 smithi177 ceph-mon[32308]: osdmap e57: 9 total, 8 up, 9 in 2024-04-23T18:14:11.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:11 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:11.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:11 smithi177 ceph-mon[32308]: from='client.? 172.21.15.188:0/2639520149' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:14:11.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:11 smithi188 ceph-mon[32694]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T18:14:11.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:11 smithi188 ceph-mon[32694]: from='client.? 172.21.15.188:0/392998675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb3e8deb-e2b5-479b-aed3-5decbd2b9044"}]: dispatch 2024-04-23T18:14:11.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:11 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb3e8deb-e2b5-479b-aed3-5decbd2b9044"}]: dispatch 2024-04-23T18:14:11.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:11 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb3e8deb-e2b5-479b-aed3-5decbd2b9044"}]': finished 2024-04-23T18:14:11.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:11 smithi188 ceph-mon[32694]: osdmap e57: 9 total, 8 up, 9 in 2024-04-23T18:14:11.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:11 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:11.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:11 smithi188 ceph-mon[32694]: from='client.? 172.21.15.188:0/2639520149' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:14:11.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:11 smithi050 ceph-mon[29187]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T18:14:11.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:11 smithi050 ceph-mon[29187]: from='client.? 172.21.15.188:0/392998675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb3e8deb-e2b5-479b-aed3-5decbd2b9044"}]: dispatch 2024-04-23T18:14:11.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:11 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb3e8deb-e2b5-479b-aed3-5decbd2b9044"}]: dispatch 2024-04-23T18:14:11.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:11 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb3e8deb-e2b5-479b-aed3-5decbd2b9044"}]': finished 2024-04-23T18:14:11.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:11 smithi050 ceph-mon[29187]: osdmap e57: 9 total, 8 up, 9 in 2024-04-23T18:14:11.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:11.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:11 smithi050 ceph-mon[29187]: from='client.? 172.21.15.188:0/2639520149' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:14:12.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:12 smithi177 ceph-mon[32308]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:12.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:12 smithi188 ceph-mon[32694]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:12.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:12 smithi050 ceph-mon[29187]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:14.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:14 smithi188 ceph-mon[32694]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:14.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:14 smithi177 ceph-mon[32308]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:14.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:14 smithi050 ceph-mon[29187]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:16.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:16 smithi188 ceph-mon[32694]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:16.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:16 smithi177 ceph-mon[32308]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:16.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:16 smithi050 ceph-mon[29187]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:18.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:18 smithi188 ceph-mon[32694]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:18.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T18:14:18.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:18.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:18 smithi177 ceph-mon[32308]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:18.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T18:14:18.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:18.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:18 smithi050 ceph-mon[29187]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:18.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T18:14:18.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:19.402 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:19 smithi188 ceph-mon[32694]: Deploying daemon osd.8 on smithi188 2024-04-23T18:14:19.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:19 smithi177 ceph-mon[32308]: Deploying daemon osd.8 on smithi188 2024-04-23T18:14:19.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:19 smithi050 ceph-mon[29187]: Deploying daemon osd.8 on smithi188 2024-04-23T18:14:20.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:20 smithi188 ceph-mon[32694]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:20.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:20 smithi177 ceph-mon[32308]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:20.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:20 smithi050 ceph-mon[29187]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:22.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:22 smithi188 ceph-mon[32694]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:22.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:22 smithi177 ceph-mon[32308]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:22.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:22 smithi050 ceph-mon[29187]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:24.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:24 smithi188 ceph-mon[32694]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:24.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:24.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:24.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:24.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:24 smithi177 ceph-mon[32308]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:24.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:24 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:24.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:24 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:24.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:24 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:24.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:24 smithi050 ceph-mon[29187]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:24.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:24.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:24.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:26.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:26 smithi177 ceph-mon[32308]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:26.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:26 smithi188 ceph-mon[32694]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:26.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:26 smithi050 ceph-mon[29187]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:26.965 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.971+0000 7f5498a27700 1 -- 172.21.15.188:0/3778435560 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5494062e80 con 0x7f54800a3c40 2024-04-23T18:14:26.966 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 8 on host 'smithi188' 2024-04-23T18:14:26.968 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.974+0000 7f5499a29700 1 -- 172.21.15.188:0/3778435560 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f54800a3c40 msgr2=0x7f54800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:26.968 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.974+0000 7f5499a29700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f54800a3c40 0x7f54800a6100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f548c00b5c0 tx=0x7f548c005fd0 comp rx=0 tx=0).stop 2024-04-23T18:14:26.968 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.974+0000 7f5499a29700 1 -- 172.21.15.188:0/3778435560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5494102be0 msgr2=0x7f549419d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:26.968 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.974+0000 7f5499a29700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5494102be0 0x7f549419d930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f547c00c920 tx=0x7f547c00cce0 comp rx=0 tx=0).stop 2024-04-23T18:14:26.968 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.974+0000 7f5499a29700 1 -- 172.21.15.188:0/3778435560 shutdown_connections 2024-04-23T18:14:26.968 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.974+0000 7f5499a29700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5494103580 0x7f549419ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:26.968 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.974+0000 7f5499a29700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f54800a3c40 0x7f54800a6100 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:26.968 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.974+0000 7f5499a29700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54941019e0 0x7f549406bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:26.968 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.974+0000 7f5499a29700 1 --2- 172.21.15.188:0/3778435560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5494102be0 0x7f549419d930 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:26.968 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.974+0000 7f5499a29700 1 -- 172.21.15.188:0/3778435560 >> 172.21.15.188:0/3778435560 conn(0x7f54940fcfa0 msgr2=0x7f5494105f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:26.968 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.974+0000 7f5499a29700 1 -- 172.21.15.188:0/3778435560 shutdown_connections 2024-04-23T18:14:26.968 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:14:26.974+0000 7f5499a29700 1 -- 172.21.15.188:0/3778435560 wait complete. 2024-04-23T18:14:27.615 DEBUG:teuthology.orchestra.run.smithi188:osd.8> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.8.service 2024-04-23T18:14:27.618 INFO:tasks.cephadm:Deploying osd.9 on smithi050 with /dev/vg_nvme/lv_1... 2024-04-23T18:14:27.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T18:14:27.890 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:28.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:27 smithi050 ceph-mon[29187]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:28.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:27 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:28.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:27 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:28.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:27 smithi050 ceph-mon[29187]: from='osd.8 [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:14:28.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:27 smithi050 ceph-mon[29187]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:14:28.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:27 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:28.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:27 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:28.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:27 smithi177 ceph-mon[32308]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:28.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:27 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:28.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:27 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:28.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:27 smithi177 ceph-mon[32308]: from='osd.8 [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:14:28.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:27 smithi177 ceph-mon[32308]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:14:28.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:27 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:28.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:27 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:27 smithi188 ceph-mon[32694]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:27 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:27 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:27 smithi188 ceph-mon[32694]: from='osd.8 [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:14:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:27 smithi188 ceph-mon[32694]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T18:14:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:27 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:27 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:29.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:29 smithi188 ceph-mon[32694]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T18:14:29.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:29 smithi188 ceph-mon[32694]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T18:14:29.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:29 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:29.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:29 smithi188 ceph-mon[32694]: from='osd.8 [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:14:29.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:29 smithi188 ceph-mon[32694]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:14:29.429 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:14:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:14:29.294+0000 7f45d5527700 -1 osd.8 0 waiting for initial osdmap 2024-04-23T18:14:29.429 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:14:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:14:29.306+0000 7f45d131a700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:14:29.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:29 smithi050 ceph-mon[29187]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T18:14:29.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:29 smithi050 ceph-mon[29187]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T18:14:29.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:29 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:29.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:29 smithi050 ceph-mon[29187]: from='osd.8 [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:14:29.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:29 smithi050 ceph-mon[29187]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:14:29.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:29 smithi177 ceph-mon[32308]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T18:14:29.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:29 smithi177 ceph-mon[32308]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T18:14:29.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:29 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:29.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:29 smithi177 ceph-mon[32308]: from='osd.8 [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:14:29.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:29 smithi177 ceph-mon[32308]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:14:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T18:14:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T18:14:30.016 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-23T18:14:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-23T18:14:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00835646 s, 1.3 GB/s 2024-04-23T18:14:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-23T18:14:30.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:30 smithi177 ceph-mon[32308]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:30.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:30 smithi177 ceph-mon[32308]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T18:14:30.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:30 smithi177 ceph-mon[32308]: osdmap e59: 9 total, 8 up, 9 in 2024-04-23T18:14:30.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:30.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:30.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:30 smithi188 ceph-mon[32694]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:30.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:30 smithi188 ceph-mon[32694]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T18:14:30.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:30 smithi188 ceph-mon[32694]: osdmap e59: 9 total, 8 up, 9 in 2024-04-23T18:14:30.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:30.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:30.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:30 smithi050 ceph-mon[29187]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-23T18:14:30.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:30 smithi050 ceph-mon[29187]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T18:14:30.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:30 smithi050 ceph-mon[29187]: osdmap e59: 9 total, 8 up, 9 in 2024-04-23T18:14:30.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:30.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:30.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_1 2024-04-23T18:14:30.935 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:14:31.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: purged_snaps scrub starts 2024-04-23T18:14:31.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: purged_snaps scrub ok 2024-04-23T18:14:31.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: osd.8 [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360] boot 2024-04-23T18:14:31.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: osdmap e60: 9 total, 9 up, 9 in 2024-04-23T18:14:31.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:31.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:31.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: osdmap e61: 9 total, 9 up, 9 in 2024-04-23T18:14:31.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: Detected new or changed devices on smithi188 2024-04-23T18:14:31.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:31.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:31.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:31.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:31.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:31.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi188 to 5696M 2024-04-23T18:14:31.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:31.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:31.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:31.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:31 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:31.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: purged_snaps scrub starts 2024-04-23T18:14:31.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: purged_snaps scrub ok 2024-04-23T18:14:31.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: osd.8 [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360] boot 2024-04-23T18:14:31.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: osdmap e60: 9 total, 9 up, 9 in 2024-04-23T18:14:31.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:31.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:31.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: osdmap e61: 9 total, 9 up, 9 in 2024-04-23T18:14:31.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: Detected new or changed devices on smithi188 2024-04-23T18:14:31.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:31.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:31.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:31.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:31.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:31.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi188 to 5696M 2024-04-23T18:14:31.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:31.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:31.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:31.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:31 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:31.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: purged_snaps scrub starts 2024-04-23T18:14:31.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: purged_snaps scrub ok 2024-04-23T18:14:31.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: osd.8 [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360] boot 2024-04-23T18:14:31.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: osdmap e60: 9 total, 9 up, 9 in 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: osdmap e61: 9 total, 9 up, 9 in 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: Detected new or changed devices on smithi188 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi188 to 5696M 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:31.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:31 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:32.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:32.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:32.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:32.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:32.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:32 smithi050 ceph-mon[29187]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T18:14:32.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:32.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:32.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:32.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:32.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:32 smithi177 ceph-mon[32308]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T18:14:32.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:32.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:32.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:32.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:32.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:32 smithi188 ceph-mon[32694]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T18:14:32.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.593+0000 7ff6f2188700 1 -- 172.21.15.50:0/1282044040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6ec1013a0 msgr2=0x7ff6ec1017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.593+0000 7ff6f2188700 1 --2- 172.21.15.50:0/1282044040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6ec1013a0 0x7ff6ec1017c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc009990 tx=0x7ff6dc009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.593+0000 7ff6f2188700 1 -- 172.21.15.50:0/1282044040 shutdown_connections 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.593+0000 7ff6f2188700 1 --2- 172.21.15.50:0/1282044040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6ec1026c0 0x7ff6ec102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.593+0000 7ff6f2188700 1 --2- 172.21.15.50:0/1282044040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6ec1013a0 0x7ff6ec1017c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.593+0000 7ff6f2188700 1 --2- 172.21.15.50:0/1282044040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6ec101d00 0x7ff6ec102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.593+0000 7ff6f2188700 1 -- 172.21.15.50:0/1282044040 >> 172.21.15.50:0/1282044040 conn(0x7ff6ec0fcf40 msgr2=0x7ff6ec0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.593+0000 7ff6f2188700 1 -- 172.21.15.50:0/1282044040 shutdown_connections 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.593+0000 7ff6f2188700 1 -- 172.21.15.50:0/1282044040 wait complete. 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6f2188700 1 Processor -- start 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6f2188700 1 -- start start 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6f2188700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6ec1013a0 0x7ff6ec19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6f2188700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6ec101d00 0x7ff6ec19b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6f2188700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6ec1026c0 0x7ff6ec1a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6f2188700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ec1106f0 con 0x7ff6ec101d00 2024-04-23T18:14:32.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6f2188700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ec110570 con 0x7ff6ec1026c0 2024-04-23T18:14:32.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6f2188700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ec110870 con 0x7ff6ec1013a0 2024-04-23T18:14:32.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6eaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6ec101d00 0x7ff6ec19b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6eaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6ec101d00 0x7ff6ec19b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45788/0 (socket says 172.21.15.50:45788) 2024-04-23T18:14:32.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6eaffd700 1 -- 172.21.15.50:0/861594982 learned_addr learned my addr 172.21.15.50:0/861594982 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:14:32.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6ebfff700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6ec1026c0 0x7ff6ec1a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.594+0000 7ff6eb7fe700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6ec1013a0 0x7ff6ec19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.595+0000 7ff6eaffd700 1 -- 172.21.15.50:0/861594982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6ec1013a0 msgr2=0x7ff6ec19ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.595+0000 7ff6eaffd700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6ec1013a0 0x7ff6ec19ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.595+0000 7ff6eaffd700 1 -- 172.21.15.50:0/861594982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6ec1026c0 msgr2=0x7ff6ec1a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:32.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.595+0000 7ff6eaffd700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6ec1026c0 0x7ff6ec1a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:32.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.595+0000 7ff6eaffd700 1 -- 172.21.15.50:0/861594982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ec1a2a60 con 0x7ff6ec101d00 2024-04-23T18:14:32.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.595+0000 7ff6eaffd700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6ec101d00 0x7ff6ec19b340 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000d980 tx=0x7ff6e000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:32.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.595+0000 7ff6e8ff9700 1 -- 172.21.15.50:0/861594982 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0016070 con 0x7ff6ec101d00 2024-04-23T18:14:32.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.595+0000 7ff6e8ff9700 1 -- 172.21.15.50:0/861594982 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6e0009d80 con 0x7ff6ec101d00 2024-04-23T18:14:32.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.595+0000 7ff6f2188700 1 -- 172.21.15.50:0/861594982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ec1a2d50 con 0x7ff6ec101d00 2024-04-23T18:14:32.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.595+0000 7ff6e8ff9700 1 -- 172.21.15.50:0/861594982 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e000f3c0 con 0x7ff6ec101d00 2024-04-23T18:14:32.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.595+0000 7ff6f2188700 1 -- 172.21.15.50:0/861594982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6ec1a3250 con 0x7ff6ec101d00 2024-04-23T18:14:32.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.596+0000 7ff6f2188700 1 -- 172.21.15.50:0/861594982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6ec02df80 con 0x7ff6ec101d00 2024-04-23T18:14:32.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.597+0000 7ff6e8ff9700 1 -- 172.21.15.50:0/861594982 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff6e0019070 con 0x7ff6ec101d00 2024-04-23T18:14:32.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.598+0000 7ff6e8ff9700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6d40a3cc0 0x7ff6d40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:32.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.598+0000 7ff6eb7fe700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6d40a3cc0 0x7ff6d40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:32.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.598+0000 7ff6e8ff9700 1 -- 172.21.15.50:0/861594982 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5858+0+0 (secure 0 0 0) 0x7ff6e00cf890 con 0x7ff6ec101d00 2024-04-23T18:14:32.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.598+0000 7ff6eb7fe700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6d40a3cc0 0x7ff6d40a6180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00b5c0 tx=0x7ff6dc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:32.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.603+0000 7ff6e8ff9700 1 -- 172.21.15.50:0/861594982 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6e009cb60 con 0x7ff6ec101d00 2024-04-23T18:14:32.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:32.824+0000 7ff6f2188700 1 -- 172.21.15.50:0/861594982 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff6ec062e80 con 0x7ff6d40a3cc0 2024-04-23T18:14:33.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:33 smithi050 ceph-mon[29187]: pgmap v201: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:14:33.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:33 smithi050 ceph-mon[29187]: from='client.14409 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:33.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:14:33.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:14:33.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:33.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:33 smithi050 ceph-mon[29187]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T18:14:33.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:33 smithi188 ceph-mon[32694]: pgmap v201: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:14:33.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:33 smithi188 ceph-mon[32694]: from='client.14409 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:33.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:14:33.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:14:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:33.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:33 smithi188 ceph-mon[32694]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T18:14:33.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:33 smithi177 ceph-mon[32308]: pgmap v201: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:14:33.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:33 smithi177 ceph-mon[32308]: from='client.14409 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:33.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:33 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:14:33.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:33 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:14:33.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:33 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:33.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:33 smithi177 ceph-mon[32308]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T18:14:34.592 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:14:34 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:14:34.426+0000 7ff69ec5f700 -1 calc_pg_upmaps abort due to max <= 0 2024-04-23T18:14:35.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:34 smithi050 ceph-mon[29187]: osdmap e64: 9 total, 9 up, 9 in 2024-04-23T18:14:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:34 smithi177 ceph-mon[32308]: osdmap e64: 9 total, 9 up, 9 in 2024-04-23T18:14:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:34 smithi188 ceph-mon[32694]: osdmap e64: 9 total, 9 up, 9 in 2024-04-23T18:14:36.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:35 smithi050 ceph-mon[29187]: pgmap v204: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:14:36.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:35 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3010871926' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46facc79-031d-4199-a0c0-e551f82182fa"}]: dispatch 2024-04-23T18:14:36.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:35 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3010871926' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46facc79-031d-4199-a0c0-e551f82182fa"}]': finished 2024-04-23T18:14:36.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:35 smithi050 ceph-mon[29187]: osdmap e65: 10 total, 9 up, 10 in 2024-04-23T18:14:36.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:35 smithi177 ceph-mon[32308]: pgmap v204: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:14:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:35 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3010871926' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46facc79-031d-4199-a0c0-e551f82182fa"}]: dispatch 2024-04-23T18:14:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:35 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3010871926' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46facc79-031d-4199-a0c0-e551f82182fa"}]': finished 2024-04-23T18:14:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:35 smithi177 ceph-mon[32308]: osdmap e65: 10 total, 9 up, 10 in 2024-04-23T18:14:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:35 smithi188 ceph-mon[32694]: pgmap v204: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T18:14:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:35 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3010871926' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46facc79-031d-4199-a0c0-e551f82182fa"}]: dispatch 2024-04-23T18:14:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:35 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3010871926' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46facc79-031d-4199-a0c0-e551f82182fa"}]': finished 2024-04-23T18:14:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:35 smithi188 ceph-mon[32694]: osdmap e65: 10 total, 9 up, 10 in 2024-04-23T18:14:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:37.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:36 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4181324745' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:14:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:36 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4181324745' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:14:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:36 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4181324745' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:14:38.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:37 smithi050 ceph-mon[29187]: pgmap v206: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-04-23T18:14:38.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:37 smithi177 ceph-mon[32308]: pgmap v206: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-04-23T18:14:38.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:37 smithi188 ceph-mon[32694]: pgmap v206: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-04-23T18:14:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:39 smithi050 ceph-mon[29187]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:40.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:39 smithi177 ceph-mon[32308]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:40.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:39 smithi188 ceph-mon[32694]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:42.179 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:41 smithi050 ceph-mon[29187]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:42.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:41 smithi177 ceph-mon[32308]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:42.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:41 smithi188 ceph-mon[32694]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:43 smithi050 ceph-mon[29187]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T18:14:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:43 smithi050 ceph-mon[29187]: Deploying daemon osd.9 on smithi050 2024-04-23T18:14:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:43 smithi177 ceph-mon[32308]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T18:14:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:43 smithi177 ceph-mon[32308]: Deploying daemon osd.9 on smithi050 2024-04-23T18:14:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:43 smithi188 ceph-mon[32694]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T18:14:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:43 smithi188 ceph-mon[32694]: Deploying daemon osd.9 on smithi050 2024-04-23T18:14:46.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:45 smithi050 ceph-mon[29187]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:46.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:45 smithi177 ceph-mon[32308]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:46.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:45 smithi188 ceph-mon[32694]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:48.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:47 smithi050 ceph-mon[29187]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:48.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:47 smithi177 ceph-mon[32308]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:47 smithi188 ceph-mon[32694]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:49.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:49 smithi050 ceph-mon[29187]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:49.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:49.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:49.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:49 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:50.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:49 smithi177 ceph-mon[32308]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:50.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:50.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:50.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:49 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:50.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:49 smithi188 ceph-mon[32694]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:50.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:50.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:50.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:49 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:51.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:51 smithi050 ceph-mon[29187]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:52.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:51 smithi177 ceph-mon[32308]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:52.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:51 smithi188 ceph-mon[32694]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:52.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.684+0000 7ff6e8ff9700 1 -- 172.21.15.50:0/861594982 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff6ec062e80 con 0x7ff6d40a3cc0 2024-04-23T18:14:52.679 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 9 on host 'smithi050' 2024-04-23T18:14:52.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.687+0000 7ff6f2188700 1 -- 172.21.15.50:0/861594982 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6d40a3cc0 msgr2=0x7ff6d40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:52.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.687+0000 7ff6f2188700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6d40a3cc0 0x7ff6d40a6180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00b5c0 tx=0x7ff6dc005fb0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.687+0000 7ff6f2188700 1 -- 172.21.15.50:0/861594982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6ec101d00 msgr2=0x7ff6ec19b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:52.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.687+0000 7ff6f2188700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6ec101d00 0x7ff6ec19b340 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000d980 tx=0x7ff6e000dc90 comp rx=0 tx=0).stop 2024-04-23T18:14:52.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.687+0000 7ff6f2188700 1 -- 172.21.15.50:0/861594982 shutdown_connections 2024-04-23T18:14:52.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.687+0000 7ff6f2188700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6d40a3cc0 0x7ff6d40a6180 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.687+0000 7ff6f2188700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6ec1026c0 0x7ff6ec1a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.687+0000 7ff6f2188700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6ec1013a0 0x7ff6ec19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.687+0000 7ff6f2188700 1 --2- 172.21.15.50:0/861594982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6ec101d00 0x7ff6ec19b340 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:52.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.687+0000 7ff6f2188700 1 -- 172.21.15.50:0/861594982 >> 172.21.15.50:0/861594982 conn(0x7ff6ec0fcf40 msgr2=0x7ff6ec0fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:52.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.687+0000 7ff6f2188700 1 -- 172.21.15.50:0/861594982 shutdown_connections 2024-04-23T18:14:52.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:14:52.687+0000 7ff6f2188700 1 -- 172.21.15.50:0/861594982 wait complete. 2024-04-23T18:14:52.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:52.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:52.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:52.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:52.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:52.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:52 smithi050 ceph-mon[29187]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T18:14:52.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:52.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:52.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:52 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:53.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:53.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:53.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:53.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:53.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:53.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:52 smithi177 ceph-mon[32308]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T18:14:53.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:53.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:53.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:52 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:53.295 DEBUG:teuthology.orchestra.run.smithi050:osd.9> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.9.service 2024-04-23T18:14:53.297 INFO:tasks.cephadm:Deploying osd.10 on smithi177 with /dev/vg_nvme/lv_1... 2024-04-23T18:14:53.298 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T18:14:53.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:53.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:53.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:53.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:53.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:53.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:52 smithi188 ceph-mon[32694]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T18:14:53.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:53.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:53.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:52 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:14:53.542 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:14:53.964 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:53 smithi177 ceph-mon[32308]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:53.965 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:53 smithi177 ceph-mon[32308]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T18:14:53.965 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:53 smithi177 ceph-mon[32308]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T18:14:53.965 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:53 smithi177 ceph-mon[32308]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:14:53.965 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:53 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:54.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:53 smithi050 ceph-mon[29187]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:54.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:53 smithi050 ceph-mon[29187]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T18:14:54.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:53 smithi050 ceph-mon[29187]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T18:14:54.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:53 smithi050 ceph-mon[29187]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:14:54.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:53 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:54.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:53 smithi188 ceph-mon[32694]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:54.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:53 smithi188 ceph-mon[32694]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T18:14:54.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:53 smithi188 ceph-mon[32694]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T18:14:54.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:53 smithi188 ceph-mon[32694]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-23T18:14:54.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:53 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:54.762 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:14:54 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:14:54.593+0000 7fd5e3fc2700 -1 osd.9 0 waiting for initial osdmap 2024-04-23T18:14:54.762 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:14:54 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:14:54.606+0000 7fd5dbdad700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:14:55.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:54 smithi050 ceph-mon[29187]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-23T18:14:55.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:54 smithi050 ceph-mon[29187]: osdmap e67: 10 total, 9 up, 10 in 2024-04-23T18:14:55.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:54 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:55.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:54 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:55.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:54 smithi050 ceph-mon[29187]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' 2024-04-23T18:14:55.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:54 smithi177 ceph-mon[32308]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-23T18:14:55.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:54 smithi177 ceph-mon[32308]: osdmap e67: 10 total, 9 up, 10 in 2024-04-23T18:14:55.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:54 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:55.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:54 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:55.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:54 smithi177 ceph-mon[32308]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' 2024-04-23T18:14:55.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:54 smithi188 ceph-mon[32694]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-23T18:14:55.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:54 smithi188 ceph-mon[32694]: osdmap e67: 10 total, 9 up, 10 in 2024-04-23T18:14:55.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:54 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:55.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:54 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:55.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:54 smithi188 ceph-mon[32694]: from='osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791]' entity='osd.9' 2024-04-23T18:14:55.422 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T18:14:55.422 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T18:14:55.422 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-23T18:14:55.422 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-23T18:14:55.422 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112034 s, 936 MB/s 2024-04-23T18:14:55.422 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-23T18:14:56.099 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:55 smithi177 ceph-mon[32308]: purged_snaps scrub starts 2024-04-23T18:14:56.099 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:55 smithi177 ceph-mon[32308]: purged_snaps scrub ok 2024-04-23T18:14:56.099 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:55 smithi177 ceph-mon[32308]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:56.099 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:55 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:56.099 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:55 smithi177 ceph-mon[32308]: osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791] boot 2024-04-23T18:14:56.099 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:55 smithi177 ceph-mon[32308]: osdmap e68: 10 total, 10 up, 10 in 2024-04-23T18:14:56.100 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:55 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:56.126 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_1 2024-04-23T18:14:56.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:55 smithi050 ceph-mon[29187]: purged_snaps scrub starts 2024-04-23T18:14:56.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:55 smithi050 ceph-mon[29187]: purged_snaps scrub ok 2024-04-23T18:14:56.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:55 smithi050 ceph-mon[29187]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:56.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:55 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:56.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:55 smithi050 ceph-mon[29187]: osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791] boot 2024-04-23T18:14:56.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:55 smithi050 ceph-mon[29187]: osdmap e68: 10 total, 10 up, 10 in 2024-04-23T18:14:56.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:55 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:55 smithi188 ceph-mon[32694]: purged_snaps scrub starts 2024-04-23T18:14:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:55 smithi188 ceph-mon[32694]: purged_snaps scrub ok 2024-04-23T18:14:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:55 smithi188 ceph-mon[32694]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 804 GiB / 805 GiB avail 2024-04-23T18:14:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:55 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:55 smithi188 ceph-mon[32694]: osd.9 [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791] boot 2024-04-23T18:14:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:55 smithi188 ceph-mon[32694]: osdmap e68: 10 total, 10 up, 10 in 2024-04-23T18:14:56.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:55 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:14:56.359 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:14:57.446 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: Detected new or changed devices on smithi050 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi050 to 4273M 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:14:57.447 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:57 smithi177 ceph-mon[32308]: osdmap e69: 10 total, 10 up, 10 in 2024-04-23T18:14:57.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: Detected new or changed devices on smithi050 2024-04-23T18:14:57.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:57.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:57.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:57.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:57.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:57.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:57.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi050 to 4273M 2024-04-23T18:14:57.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:57.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:57.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:57.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:57.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:14:57.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:57 smithi050 ceph-mon[29187]: osdmap e69: 10 total, 10 up, 10 in 2024-04-23T18:14:57.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: Detected new or changed devices on smithi050 2024-04-23T18:14:57.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:57.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:57.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:57.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:57.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:57.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:14:57.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi050 to 4273M 2024-04-23T18:14:57.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:57.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:57.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:14:57.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:14:57.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:14:57.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:57 smithi188 ceph-mon[32694]: osdmap e69: 10 total, 10 up, 10 in 2024-04-23T18:14:57.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.923+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/3433654676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ef0103d90 msgr2=0x7f2ef01041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.923+0000 7f2ef4a4f700 1 --2- 172.21.15.177:0/3433654676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ef0103d90 0x7f2ef01041f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4009990 tx=0x7f2ee4009ca0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.923+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/3433654676 shutdown_connections 2024-04-23T18:14:57.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.923+0000 7f2ef4a4f700 1 --2- 172.21.15.177:0/3433654676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ef0104730 0x7f2ef010afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.921 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.923+0000 7f2ef4a4f700 1 --2- 172.21.15.177:0/3433654676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef0100db0 0x7f2ef01011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.923+0000 7f2ef4a4f700 1 --2- 172.21.15.177:0/3433654676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ef0103d90 0x7f2ef01041f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.923+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/3433654676 >> 172.21.15.177:0/3433654676 conn(0x7f2ef00fc710 msgr2=0x7f2ef00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:14:57.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.924+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/3433654676 shutdown_connections 2024-04-23T18:14:57.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.924+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/3433654676 wait complete. 2024-04-23T18:14:57.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.924+0000 7f2ef4a4f700 1 Processor -- start 2024-04-23T18:14:57.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.924+0000 7f2ef4a4f700 1 -- start start 2024-04-23T18:14:57.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.924+0000 7f2ef4a4f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef0100db0 0x7f2ef019ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.924+0000 7f2ef4a4f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ef0103d90 0x7f2ef019b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.924+0000 7f2ef4a4f700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ef0104730 0x7f2ef01a2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2ef4a4f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef010e5f0 con 0x7f2ef0103d90 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2ef4a4f700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef010e470 con 0x7f2ef0104730 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2ef4a4f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef010e770 con 0x7f2ef0100db0 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2eeed9d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ef0104730 0x7f2ef01a2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2eeed9d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ef0104730 0x7f2ef01a2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:60940/0 (socket says 172.21.15.177:60940) 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2eeed9d700 1 -- 172.21.15.177:0/1499369999 learned_addr learned my addr 172.21.15.177:0/1499369999 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2eedd9b700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ef0103d90 0x7f2ef019b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2eee59c700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef0100db0 0x7f2ef019ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2eeed9d700 1 -- 172.21.15.177:0/1499369999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef0100db0 msgr2=0x7f2ef019ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2eeed9d700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef0100db0 0x7f2ef019ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2eeed9d700 1 -- 172.21.15.177:0/1499369999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ef0103d90 msgr2=0x7f2ef019b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:14:57.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2eeed9d700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ef0103d90 0x7f2ef019b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:14:57.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2eeed9d700 1 -- 172.21.15.177:0/1499369999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef01a2ad0 con 0x7f2ef0104730 2024-04-23T18:14:57.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2eeed9d700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ef0104730 0x7f2ef01a2430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8007ee0 tx=0x7f2ee800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:57.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2edf7fe700 1 -- 172.21.15.177:0/1499369999 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee800dbd0 con 0x7f2ef0104730 2024-04-23T18:14:57.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/1499369999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ef01a2dc0 con 0x7f2ef0104730 2024-04-23T18:14:57.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2edf7fe700 1 -- 172.21.15.177:0/1499369999 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ee800dd70 con 0x7f2ef0104730 2024-04-23T18:14:57.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2edf7fe700 1 -- 172.21.15.177:0/1499369999 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8005990 con 0x7f2ef0104730 2024-04-23T18:14:57.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.925+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/1499369999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ef01a32f0 con 0x7f2ef0104730 2024-04-23T18:14:57.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.926+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/1499369999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ef01a3520 con 0x7f2ef0104730 2024-04-23T18:14:57.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.927+0000 7f2edf7fe700 1 -- 172.21.15.177:0/1499369999 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2ee801a020 con 0x7f2ef0104730 2024-04-23T18:14:57.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.928+0000 7f2edf7fe700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2ed80a3c40 0x7f2ed80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:14:57.926 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.928+0000 7f2edf7fe700 1 -- 172.21.15.177:0/1499369999 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f2ee80cf3a0 con 0x7f2ef0104730 2024-04-23T18:14:57.926 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.928+0000 7f2eee59c700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2ed80a3c40 0x7f2ed80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:14:57.926 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.929+0000 7f2eee59c700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2ed80a3c40 0x7f2ed80a6100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0006fd0 tx=0x7f2ee0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:14:57.927 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:57.933+0000 7f2edf7fe700 1 -- 172.21.15.177:0/1499369999 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ee8010070 con 0x7f2ef0104730 2024-04-23T18:14:58.136 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:14:58.141+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/1499369999 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2ef01032a0 con 0x7f2ed80a3c40 2024-04-23T18:14:58.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:58 smithi177 ceph-mon[32308]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:58.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:58 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:14:58.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:58 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:14:58.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:58 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:58.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:58 smithi050 ceph-mon[29187]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:58.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:58 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:14:58.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:58 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:14:58.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:58 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:58.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:58 smithi188 ceph-mon[32694]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:14:58.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:58 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:14:58.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:58 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:14:58.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:58 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:14:59.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:14:59 smithi050 ceph-mon[29187]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:14:59.773 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:14:59 smithi177 ceph-mon[32308]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:14:59.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:14:59 smithi188 ceph-mon[32694]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:01.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:01 smithi177 ceph-mon[32308]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:01.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:01 smithi177 ceph-mon[32308]: from='client.? 172.21.15.177:0/1022166890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7af2e9f-fa4b-43d7-9567-013393024176"}]: dispatch 2024-04-23T18:15:01.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:01 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7af2e9f-fa4b-43d7-9567-013393024176"}]: dispatch 2024-04-23T18:15:01.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:01 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7af2e9f-fa4b-43d7-9567-013393024176"}]': finished 2024-04-23T18:15:01.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:01 smithi177 ceph-mon[32308]: osdmap e70: 11 total, 10 up, 11 in 2024-04-23T18:15:01.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:01 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:01.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:01 smithi188 ceph-mon[32694]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:01.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:01 smithi188 ceph-mon[32694]: from='client.? 172.21.15.177:0/1022166890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7af2e9f-fa4b-43d7-9567-013393024176"}]: dispatch 2024-04-23T18:15:01.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:01 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7af2e9f-fa4b-43d7-9567-013393024176"}]: dispatch 2024-04-23T18:15:01.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:01 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7af2e9f-fa4b-43d7-9567-013393024176"}]': finished 2024-04-23T18:15:01.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:01 smithi188 ceph-mon[32694]: osdmap e70: 11 total, 10 up, 11 in 2024-04-23T18:15:01.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:01 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:01.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:01 smithi050 ceph-mon[29187]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:01.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:01 smithi050 ceph-mon[29187]: from='client.? 172.21.15.177:0/1022166890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7af2e9f-fa4b-43d7-9567-013393024176"}]: dispatch 2024-04-23T18:15:01.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:01 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7af2e9f-fa4b-43d7-9567-013393024176"}]: dispatch 2024-04-23T18:15:01.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:01 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7af2e9f-fa4b-43d7-9567-013393024176"}]': finished 2024-04-23T18:15:01.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:01 smithi050 ceph-mon[29187]: osdmap e70: 11 total, 10 up, 11 in 2024-04-23T18:15:01.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:01 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:02.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:02 smithi177 ceph-mon[32308]: from='client.? 172.21.15.177:0/3669259107' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:15:02.815 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:02 smithi050 ceph-mon[29187]: from='client.? 172.21.15.177:0/3669259107' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:15:02.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:02 smithi188 ceph-mon[32694]: from='client.? 172.21.15.177:0/3669259107' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:15:03.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:03 smithi177 ceph-mon[32308]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:03.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:03 smithi188 ceph-mon[32694]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:03.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:03 smithi050 ceph-mon[29187]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:05.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:05 smithi188 ceph-mon[32694]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:05.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:05 smithi050 ceph-mon[29187]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:06.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:05 smithi177 ceph-mon[32308]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:07.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:07 smithi177 ceph-mon[32308]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:07.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:07 smithi188 ceph-mon[32694]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:07.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:07 smithi050 ceph-mon[29187]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:08.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:08 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T18:15:08.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:08 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:08.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:08 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T18:15:08.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:08 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:09.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:08 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T18:15:09.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:08 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:09.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:09 smithi177 ceph-mon[32308]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:09.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:09 smithi177 ceph-mon[32308]: Deploying daemon osd.10 on smithi177 2024-04-23T18:15:09.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:09 smithi188 ceph-mon[32694]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:09.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:09 smithi188 ceph-mon[32694]: Deploying daemon osd.10 on smithi177 2024-04-23T18:15:09.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:09 smithi050 ceph-mon[29187]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:09.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:09 smithi050 ceph-mon[29187]: Deploying daemon osd.10 on smithi177 2024-04-23T18:15:11.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:11 smithi177 ceph-mon[32308]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:11.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:11 smithi188 ceph-mon[32694]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:11.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:11 smithi050 ceph-mon[29187]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:13.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:13 smithi177 ceph-mon[32308]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:13.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:13 smithi188 ceph-mon[32694]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:13.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:13 smithi050 ceph-mon[29187]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:14.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:14 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:14.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:14 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:14.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:14 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:14.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:14 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:14.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:14 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:14.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:14 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:14.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:14 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:14.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:14 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:14.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:14 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:16.284 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:16 smithi177 ceph-mon[32308]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:16.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:16 smithi188 ceph-mon[32694]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:16.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:16 smithi050 ceph-mon[29187]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:17.858 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 10 on host 'smithi177' 2024-04-23T18:15:17.858 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.864+0000 7f2edf7fe700 1 -- 172.21.15.177:0/1499369999 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2ef01032a0 con 0x7f2ed80a3c40 2024-04-23T18:15:17.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.866+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/1499369999 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2ed80a3c40 msgr2=0x7f2ed80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:17.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.866+0000 7f2ef4a4f700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2ed80a3c40 0x7f2ed80a6100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0006fd0 tx=0x7f2ee0008040 comp rx=0 tx=0).stop 2024-04-23T18:15:17.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.866+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/1499369999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ef0104730 msgr2=0x7f2ef01a2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:17.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.866+0000 7f2ef4a4f700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ef0104730 0x7f2ef01a2430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8007ee0 tx=0x7f2ee800d3b0 comp rx=0 tx=0).stop 2024-04-23T18:15:17.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.866+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/1499369999 shutdown_connections 2024-04-23T18:15:17.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.866+0000 7f2ef4a4f700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2ed80a3c40 0x7f2ed80a6100 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:17.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.866+0000 7f2ef4a4f700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ef0104730 0x7f2ef01a2430 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:17.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.866+0000 7f2ef4a4f700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ef0100db0 0x7f2ef019ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:17.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.866+0000 7f2ef4a4f700 1 --2- 172.21.15.177:0/1499369999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ef0103d90 0x7f2ef019b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:17.860 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.866+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/1499369999 >> 172.21.15.177:0/1499369999 conn(0x7f2ef00fc710 msgr2=0x7f2ef0101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:17.861 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.866+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/1499369999 shutdown_connections 2024-04-23T18:15:17.861 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-23T18:15:17.866+0000 7f2ef4a4f700 1 -- 172.21.15.177:0/1499369999 wait complete. 2024-04-23T18:15:18.386 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:18 smithi177 ceph-mon[32308]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:18.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:18.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:18.387 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.388 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.388 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.388 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:18.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:18 smithi050 ceph-mon[29187]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:18.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:18.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:18.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:18.514 DEBUG:teuthology.orchestra.run.smithi177:osd.10> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.10.service 2024-04-23T18:15:18.518 INFO:tasks.cephadm:Deploying osd.11 on smithi188 with /dev/vg_nvme/lv_1... 2024-04-23T18:15:18.520 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T18:15:18.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:18 smithi188 ceph-mon[32694]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:18.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:18.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:18.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:18.547 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:18.766 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:15:19.237 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:19 smithi188 ceph-mon[32694]: from='osd.10 [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:15:19.237 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:19 smithi188 ceph-mon[32694]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:15:19.237 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:19 smithi188 ceph-mon[32694]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T18:15:19.237 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:19 smithi188 ceph-mon[32694]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T18:15:19.237 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:19.238 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:19 smithi188 ceph-mon[32694]: from='osd.10 [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:15:19.238 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:19 smithi188 ceph-mon[32694]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:15:19.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:19 smithi177 ceph-mon[32308]: from='osd.10 [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:15:19.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:19 smithi177 ceph-mon[32308]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:15:19.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:19 smithi177 ceph-mon[32308]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T18:15:19.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:19 smithi177 ceph-mon[32308]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T18:15:19.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:19 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:19.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:19 smithi177 ceph-mon[32308]: from='osd.10 [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:15:19.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:19 smithi177 ceph-mon[32308]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:15:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:19 smithi050 ceph-mon[29187]: from='osd.10 [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:15:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:19 smithi050 ceph-mon[29187]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T18:15:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:19 smithi050 ceph-mon[29187]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T18:15:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:19 smithi050 ceph-mon[29187]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T18:15:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:19 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:19 smithi050 ceph-mon[29187]: from='osd.10 [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:15:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:19 smithi050 ceph-mon[29187]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-23T18:15:20.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:20 smithi188 ceph-mon[32694]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:20.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:20 smithi188 ceph-mon[32694]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-23T18:15:20.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:20 smithi188 ceph-mon[32694]: osdmap e72: 11 total, 10 up, 11 in 2024-04-23T18:15:20.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:20 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:20.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:20 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:20.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:20 smithi177 ceph-mon[32308]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:20.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:20 smithi177 ceph-mon[32308]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-23T18:15:20.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:20 smithi177 ceph-mon[32308]: osdmap e72: 11 total, 10 up, 11 in 2024-04-23T18:15:20.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:20 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:20.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:20 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:20.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:20 smithi050 ceph-mon[29187]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:20.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:20 smithi050 ceph-mon[29187]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-23T18:15:20.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:20 smithi050 ceph-mon[29187]: osdmap e72: 11 total, 10 up, 11 in 2024-04-23T18:15:20.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:20 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:20.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:20 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:20.890 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T18:15:20.890 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T18:15:20.891 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-23T18:15:20.891 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-23T18:15:20.891 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112011 s, 936 MB/s 2024-04-23T18:15:20.891 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-23T18:15:21.038 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:15:20 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:15:20.745+0000 7f32a8ec1700 -1 osd.10 0 waiting for initial osdmap 2024-04-23T18:15:21.039 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:15:20 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:15:20.907+0000 7f32a0cac700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:15:21.652 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_1 2024-04-23T18:15:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: purged_snaps scrub starts 2024-04-23T18:15:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: purged_snaps scrub ok 2024-04-23T18:15:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='osd.10 ' entity='osd.10' 2024-04-23T18:15:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: osdmap e73: 11 total, 10 up, 11 in 2024-04-23T18:15:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:21.932 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:15:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: purged_snaps scrub starts 2024-04-23T18:15:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: purged_snaps scrub ok 2024-04-23T18:15:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='osd.10 ' entity='osd.10' 2024-04-23T18:15:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: osdmap e73: 11 total, 10 up, 11 in 2024-04-23T18:15:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:21.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:21.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:21.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:21.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:21.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:21.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:21.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:21.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:21.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:22.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: purged_snaps scrub starts 2024-04-23T18:15:22.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: purged_snaps scrub ok 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='osd.10 ' entity='osd.10' 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: osdmap e73: 11 total, 10 up, 11 in 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:22.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:22.040 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:23.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:22 smithi188 ceph-mon[32694]: Detected new or changed devices on smithi177 2024-04-23T18:15:23.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:22 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi177 to 4272M 2024-04-23T18:15:23.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:22 smithi188 ceph-mon[32694]: osd.10 [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803] boot 2024-04-23T18:15:23.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:22 smithi188 ceph-mon[32694]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T18:15:23.056 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:22 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:23.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:22 smithi050 ceph-mon[29187]: Detected new or changed devices on smithi177 2024-04-23T18:15:23.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:22 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi177 to 4272M 2024-04-23T18:15:23.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:22 smithi050 ceph-mon[29187]: osd.10 [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803] boot 2024-04-23T18:15:23.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:22 smithi050 ceph-mon[29187]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T18:15:23.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:22 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:23.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:22 smithi177 ceph-mon[32308]: Detected new or changed devices on smithi177 2024-04-23T18:15:23.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:22 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi177 to 4272M 2024-04-23T18:15:23.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:22 smithi177 ceph-mon[32308]: osd.10 [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803] boot 2024-04-23T18:15:23.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:22 smithi177 ceph-mon[32308]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T18:15:23.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:22 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:15:23.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.577+0000 7f237419d700 1 -- 172.21.15.188:0/1284797743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236c103ca0 msgr2=0x7f236c104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:23.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.577+0000 7f237419d700 1 --2- 172.21.15.188:0/1284797743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236c103ca0 0x7f236c104100 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f235c009990 tx=0x7f235c009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.571 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.577+0000 7f237419d700 1 -- 172.21.15.188:0/1284797743 shutdown_connections 2024-04-23T18:15:23.572 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.577+0000 7f237419d700 1 --2- 172.21.15.188:0/1284797743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f236c104640 0x7f236c10aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.572 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.577+0000 7f237419d700 1 --2- 172.21.15.188:0/1284797743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f236c102aa0 0x7f236c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.572 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.577+0000 7f237419d700 1 --2- 172.21.15.188:0/1284797743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236c103ca0 0x7f236c104100 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.572 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.577+0000 7f237419d700 1 -- 172.21.15.188:0/1284797743 >> 172.21.15.188:0/1284797743 conn(0x7f236c0fe040 msgr2=0x7f236c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:23.572 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.578+0000 7f237419d700 1 -- 172.21.15.188:0/1284797743 shutdown_connections 2024-04-23T18:15:23.572 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.578+0000 7f237419d700 1 -- 172.21.15.188:0/1284797743 wait complete. 2024-04-23T18:15:23.574 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.578+0000 7f237419d700 1 Processor -- start 2024-04-23T18:15:23.574 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.578+0000 7f237419d700 1 -- start start 2024-04-23T18:15:23.574 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.578+0000 7f237419d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236c102aa0 0x7f236c0767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:23.574 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.578+0000 7f237419d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f236c103ca0 0x7f236c074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:23.574 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.578+0000 7f237419d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f236c104640 0x7f236c075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:23.574 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.578+0000 7f237419d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f236c10e5d0 con 0x7f236c103ca0 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.578+0000 7f237419d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f236c10e450 con 0x7f236c104640 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.578+0000 7f237419d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f236c10e750 con 0x7f236c102aa0 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.578+0000 7f2371f39700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236c102aa0 0x7f236c0767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2371f39700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236c102aa0 0x7f236c0767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:57894/0 (socket says 172.21.15.188:57894) 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2371f39700 1 -- 172.21.15.188:0/618798551 learned_addr learned my addr 172.21.15.188:0/618798551 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f237273a700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f236c104640 0x7f236c075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2371738700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f236c103ca0 0x7f236c074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2371f39700 1 -- 172.21.15.188:0/618798551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f236c104640 msgr2=0x7f236c075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2371f39700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f236c104640 0x7f236c075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2371f39700 1 -- 172.21.15.188:0/618798551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f236c103ca0 msgr2=0x7f236c074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2371f39700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f236c103ca0 0x7f236c074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2371f39700 1 -- 172.21.15.188:0/618798551 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f236c075c20 con 0x7f236c102aa0 2024-04-23T18:15:23.575 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2371738700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f236c103ca0 0x7f236c074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:23.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2371f39700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236c102aa0 0x7f236c0767e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f236800b500 tx=0x7f236800b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:23.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2362ffd700 1 -- 172.21.15.188:0/618798551 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2368015020 con 0x7f236c102aa0 2024-04-23T18:15:23.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f237419d700 1 -- 172.21.15.188:0/618798551 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f236c1a2e00 con 0x7f236c102aa0 2024-04-23T18:15:23.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2362ffd700 1 -- 172.21.15.188:0/618798551 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2368011ce0 con 0x7f236c102aa0 2024-04-23T18:15:23.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f2362ffd700 1 -- 172.21.15.188:0/618798551 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2368010d20 con 0x7f236c102aa0 2024-04-23T18:15:23.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.579+0000 7f237419d700 1 -- 172.21.15.188:0/618798551 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f236c1a3200 con 0x7f236c102aa0 2024-04-23T18:15:23.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.580+0000 7f237419d700 1 -- 172.21.15.188:0/618798551 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f236c02df80 con 0x7f236c102aa0 2024-04-23T18:15:23.577 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.581+0000 7f2362ffd700 1 -- 172.21.15.188:0/618798551 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2368018020 con 0x7f236c102aa0 2024-04-23T18:15:23.577 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.582+0000 7f2362ffd700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f23580a3c40 0x7f23580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:23.577 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.582+0000 7f2362ffd700 1 -- 172.21.15.188:0/618798551 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6723+0+0 (secure 0 0 0) 0x7f23680cf510 con 0x7f236c102aa0 2024-04-23T18:15:23.577 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.582+0000 7f2371738700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f23580a3c40 0x7f23580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:23.577 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.583+0000 7f2371738700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f23580a3c40 0x7f23580a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f235c00b5c0 tx=0x7f235c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:23.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.586+0000 7f2362ffd700 1 -- 172.21.15.188:0/618798551 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f236801f020 con 0x7f236c102aa0 2024-04-23T18:15:23.785 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:23.791+0000 7f237419d700 1 -- 172.21.15.188:0/618798551 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f236c062e80 con 0x7f23580a3c40 2024-04-23T18:15:24.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:23 smithi188 ceph-mon[32694]: pgmap v238: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:24.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:23 smithi188 ceph-mon[32694]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T18:15:24.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:23 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:15:24.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:23 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:15:24.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:23 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:24.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:23 smithi050 ceph-mon[29187]: pgmap v238: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:24.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:23 smithi050 ceph-mon[29187]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T18:15:24.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:23 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:15:24.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:23 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:15:24.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:23 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:24.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:23 smithi177 ceph-mon[32308]: pgmap v238: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:24.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:23 smithi177 ceph-mon[32308]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T18:15:24.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:23 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T18:15:24.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:23 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T18:15:24.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:23 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:25.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:24 smithi188 ceph-mon[32694]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:25.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:24 smithi188 ceph-mon[32694]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T18:15:25.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:24 smithi050 ceph-mon[29187]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:25.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:24 smithi050 ceph-mon[29187]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T18:15:25.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:24 smithi177 ceph-mon[32308]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:25.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:24 smithi177 ceph-mon[32308]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T18:15:26.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:25 smithi188 ceph-mon[32694]: pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:26.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:25 smithi188 ceph-mon[32694]: osdmap e77: 11 total, 11 up, 11 in 2024-04-23T18:15:26.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:25 smithi050 ceph-mon[29187]: pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:26.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:25 smithi050 ceph-mon[29187]: osdmap e77: 11 total, 11 up, 11 in 2024-04-23T18:15:26.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:25 smithi177 ceph-mon[32308]: pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:26.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:25 smithi177 ceph-mon[32308]: osdmap e77: 11 total, 11 up, 11 in 2024-04-23T18:15:27.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:26 smithi050 ceph-mon[29187]: osdmap e78: 11 total, 11 up, 11 in 2024-04-23T18:15:27.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:26 smithi050 ceph-mon[29187]: from='client.? 172.21.15.188:0/785240434' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40d6c591-e063-4b2f-ae02-58b45f6935cd"}]: dispatch 2024-04-23T18:15:27.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:26 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40d6c591-e063-4b2f-ae02-58b45f6935cd"}]: dispatch 2024-04-23T18:15:27.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:26 smithi050 ceph-mon[29187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40d6c591-e063-4b2f-ae02-58b45f6935cd"}]': finished 2024-04-23T18:15:27.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:26 smithi050 ceph-mon[29187]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T18:15:27.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:27.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:26 smithi177 ceph-mon[32308]: osdmap e78: 11 total, 11 up, 11 in 2024-04-23T18:15:27.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:26 smithi177 ceph-mon[32308]: from='client.? 172.21.15.188:0/785240434' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40d6c591-e063-4b2f-ae02-58b45f6935cd"}]: dispatch 2024-04-23T18:15:27.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:26 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40d6c591-e063-4b2f-ae02-58b45f6935cd"}]: dispatch 2024-04-23T18:15:27.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:26 smithi177 ceph-mon[32308]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40d6c591-e063-4b2f-ae02-58b45f6935cd"}]': finished 2024-04-23T18:15:27.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:26 smithi177 ceph-mon[32308]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T18:15:27.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:27.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:26 smithi188 ceph-mon[32694]: osdmap e78: 11 total, 11 up, 11 in 2024-04-23T18:15:27.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:26 smithi188 ceph-mon[32694]: from='client.? 172.21.15.188:0/785240434' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40d6c591-e063-4b2f-ae02-58b45f6935cd"}]: dispatch 2024-04-23T18:15:27.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:26 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40d6c591-e063-4b2f-ae02-58b45f6935cd"}]: dispatch 2024-04-23T18:15:27.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:26 smithi188 ceph-mon[32694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40d6c591-e063-4b2f-ae02-58b45f6935cd"}]': finished 2024-04-23T18:15:27.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:26 smithi188 ceph-mon[32694]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T18:15:27.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:28.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:27 smithi050 ceph-mon[29187]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:28.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:27 smithi050 ceph-mon[29187]: from='client.? 172.21.15.188:0/2035906134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:15:28.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:27 smithi177 ceph-mon[32308]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:28.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:27 smithi177 ceph-mon[32308]: from='client.? 172.21.15.188:0/2035906134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:15:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:27 smithi188 ceph-mon[32694]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:27 smithi188 ceph-mon[32694]: from='client.? 172.21.15.188:0/2035906134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T18:15:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:29 smithi050 ceph-mon[29187]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-23T18:15:30.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:29 smithi177 ceph-mon[32308]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-23T18:15:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:29 smithi188 ceph-mon[32694]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-23T18:15:31.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:31 smithi050 ceph-mon[29187]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:15:32.135 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:31 smithi188 ceph-mon[32694]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:15:32.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:31 smithi177 ceph-mon[32308]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T18:15:34.153 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:33 smithi188 ceph-mon[32694]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-23T18:15:34.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:33 smithi050 ceph-mon[29187]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-23T18:15:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:33 smithi177 ceph-mon[32308]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-23T18:15:35.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:34 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T18:15:35.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:34 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:35.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:34 smithi188 ceph-mon[32694]: Deploying daemon osd.11 on smithi188 2024-04-23T18:15:35.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T18:15:35.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:34 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:35.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:34 smithi050 ceph-mon[29187]: Deploying daemon osd.11 on smithi188 2024-04-23T18:15:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:34 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T18:15:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:34 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:34 smithi177 ceph-mon[32308]: Deploying daemon osd.11 on smithi188 2024-04-23T18:15:36.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:35 smithi188 ceph-mon[32694]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-23T18:15:36.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:35 smithi050 ceph-mon[29187]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-23T18:15:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:35 smithi177 ceph-mon[32308]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-23T18:15:38.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:37 smithi188 ceph-mon[32694]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:15:38.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:37 smithi050 ceph-mon[29187]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:15:38.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:37 smithi177 ceph-mon[32308]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T18:15:40.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:39 smithi188 ceph-mon[32694]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T18:15:40.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:40.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:40.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:39 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:39 smithi050 ceph-mon[29187]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T18:15:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:40.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:39 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:40.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:39 smithi177 ceph-mon[32308]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T18:15:40.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:40.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:40.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:39 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:42.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:41 smithi188 ceph-mon[32694]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:42.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:41 smithi050 ceph-mon[29187]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:42.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:41 smithi177 ceph-mon[32308]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:43.811 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 11 on host 'smithi188' 2024-04-23T18:15:43.812 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.817+0000 7f2362ffd700 1 -- 172.21.15.188:0/618798551 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f236c062e80 con 0x7f23580a3c40 2024-04-23T18:15:43.813 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.819+0000 7f237419d700 1 -- 172.21.15.188:0/618798551 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f23580a3c40 msgr2=0x7f23580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:43.813 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.819+0000 7f237419d700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f23580a3c40 0x7f23580a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f235c00b5c0 tx=0x7f235c005fd0 comp rx=0 tx=0).stop 2024-04-23T18:15:43.813 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.819+0000 7f237419d700 1 -- 172.21.15.188:0/618798551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236c102aa0 msgr2=0x7f236c0767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:43.813 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.819+0000 7f237419d700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236c102aa0 0x7f236c0767e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f236800b500 tx=0x7f236800b810 comp rx=0 tx=0).stop 2024-04-23T18:15:43.814 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.819+0000 7f237419d700 1 -- 172.21.15.188:0/618798551 shutdown_connections 2024-04-23T18:15:43.814 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.819+0000 7f237419d700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f23580a3c40 0x7f23580a6100 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:43.814 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.819+0000 7f237419d700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f236c104640 0x7f236c075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:43.814 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.819+0000 7f237419d700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f236c102aa0 0x7f236c0767e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:43.814 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.819+0000 7f237419d700 1 --2- 172.21.15.188:0/618798551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f236c103ca0 0x7f236c074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:43.814 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.819+0000 7f237419d700 1 -- 172.21.15.188:0/618798551 >> 172.21.15.188:0/618798551 conn(0x7f236c0fe040 msgr2=0x7f236c107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:43.814 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.820+0000 7f237419d700 1 -- 172.21.15.188:0/618798551 shutdown_connections 2024-04-23T18:15:43.814 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:43.820+0000 7f237419d700 1 -- 172.21.15.188:0/618798551 wait complete. 2024-04-23T18:15:44.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:43 smithi188 ceph-mon[32694]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:44.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:44.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:44.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:43 smithi188 ceph-mon[32694]: from='osd.11 [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:15:44.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:43 smithi188 ceph-mon[32694]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:15:44.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:43 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:43 smithi050 ceph-mon[29187]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:43 smithi050 ceph-mon[29187]: from='osd.11 [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:15:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:43 smithi050 ceph-mon[29187]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:15:44.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:43 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:43 smithi177 ceph-mon[32308]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:43 smithi177 ceph-mon[32308]: from='osd.11 [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:15:44.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:43 smithi177 ceph-mon[32308]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T18:15:44.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:44.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:43 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:44.450 DEBUG:teuthology.orchestra.run.smithi188:osd.11> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.11.service 2024-04-23T18:15:44.452 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-23T18:15:44.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-23T18:15:44.713 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:45.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:44 smithi188 ceph-mon[32694]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T18:15:45.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:44 smithi188 ceph-mon[32694]: osdmap e80: 12 total, 11 up, 12 in 2024-04-23T18:15:45.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:44 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:45.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:44 smithi188 ceph-mon[32694]: from='osd.11 [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:15:45.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:44 smithi188 ceph-mon[32694]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:15:45.080 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:15:44 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:15:44.911+0000 7fe2df0c8700 -1 osd.11 0 waiting for initial osdmap 2024-04-23T18:15:45.081 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:15:44 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:15:44.928+0000 7fe2daebb700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T18:15:45.169 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:44 smithi050 ceph-mon[29187]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T18:15:45.169 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:44 smithi050 ceph-mon[29187]: osdmap e80: 12 total, 11 up, 12 in 2024-04-23T18:15:45.169 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:44 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:45.169 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:44 smithi050 ceph-mon[29187]: from='osd.11 [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:15:45.169 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:44 smithi050 ceph-mon[29187]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:15:45.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:44 smithi177 ceph-mon[32308]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T18:15:45.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:44 smithi177 ceph-mon[32308]: osdmap e80: 12 total, 11 up, 12 in 2024-04-23T18:15:45.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:44 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:45.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:44 smithi177 ceph-mon[32308]: from='osd.11 [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:15:45.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:44 smithi177 ceph-mon[32308]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-23T18:15:46.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:45 smithi188 ceph-mon[32694]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:46.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:45 smithi188 ceph-mon[32694]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T18:15:46.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:45 smithi188 ceph-mon[32694]: osdmap e81: 12 total, 11 up, 12 in 2024-04-23T18:15:46.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:45 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:46.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:45 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:46.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:45 smithi050 ceph-mon[29187]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:46.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:45 smithi050 ceph-mon[29187]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T18:15:46.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:45 smithi050 ceph-mon[29187]: osdmap e81: 12 total, 11 up, 12 in 2024-04-23T18:15:46.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:45 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:46.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:45 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:46.188 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:45 smithi177 ceph-mon[32308]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:46.188 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:45 smithi177 ceph-mon[32308]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-23T18:15:46.189 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:45 smithi177 ceph-mon[32308]: osdmap e81: 12 total, 11 up, 12 in 2024-04-23T18:15:46.189 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:45 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:46.189 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:45 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:46.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.390+0000 7f2391aa0700 1 -- 172.21.15.50:0/1307055347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f238c104730 msgr2=0x7f238c10afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:46.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.390+0000 7f2391aa0700 1 --2- 172.21.15.50:0/1307055347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f238c104730 0x7f238c10afe0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f238400b0a0 tx=0x7f238400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.390+0000 7f2391aa0700 1 -- 172.21.15.50:0/1307055347 shutdown_connections 2024-04-23T18:15:46.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.390+0000 7f2391aa0700 1 --2- 172.21.15.50:0/1307055347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f238c104730 0x7f238c10afe0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.390+0000 7f2391aa0700 1 --2- 172.21.15.50:0/1307055347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f238c100db0 0x7f238c1011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.390+0000 7f2391aa0700 1 --2- 172.21.15.50:0/1307055347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f238c103d90 0x7f238c1041f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.390+0000 7f2391aa0700 1 -- 172.21.15.50:0/1307055347 >> 172.21.15.50:0/1307055347 conn(0x7f238c0fc710 msgr2=0x7f238c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:46.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.390+0000 7f2391aa0700 1 -- 172.21.15.50:0/1307055347 shutdown_connections 2024-04-23T18:15:46.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.390+0000 7f2391aa0700 1 -- 172.21.15.50:0/1307055347 wait complete. 2024-04-23T18:15:46.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.390+0000 7f2391aa0700 1 Processor -- start 2024-04-23T18:15:46.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f2391aa0700 1 -- start start 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f2391aa0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f238c100db0 0x7f238c1036f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f2391aa0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f238c103d90 0x7f238c101d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f2391aa0700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f238c104730 0x7f238c102280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f2391aa0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f238c10e580 con 0x7f238c103d90 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f2391aa0700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f238c10e400 con 0x7f238c104730 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f2391aa0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f238c10e700 con 0x7f238c100db0 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f238affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f238c103d90 0x7f238c101d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f238affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f238c103d90 0x7f238c101d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40520/0 (socket says 172.21.15.50:40520) 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f238affd700 1 -- 172.21.15.50:0/2759553891 learned_addr learned my addr 172.21.15.50:0/2759553891 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f238bfff700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f238c104730 0x7f238c102280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f238b7fe700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f238c100db0 0x7f238c1036f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f238affd700 1 -- 172.21.15.50:0/2759553891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f238c100db0 msgr2=0x7f238c1036f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f238affd700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f238c100db0 0x7f238c1036f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f238affd700 1 -- 172.21.15.50:0/2759553891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f238c104730 msgr2=0x7f238c102280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:46.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f238affd700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f238c104730 0x7f238c102280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f238affd700 1 -- 172.21.15.50:0/2759553891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238c102b30 con 0x7f238c103d90 2024-04-23T18:15:46.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.391+0000 7f238affd700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f238c103d90 0x7f238c101d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f238000e9a0 tx=0x7f238000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:46.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.392+0000 7f2388ff9700 1 -- 172.21.15.50:0/2759553891 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238000cbd0 con 0x7f238c103d90 2024-04-23T18:15:46.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.392+0000 7f2391aa0700 1 -- 172.21.15.50:0/2759553891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f238c1a2d20 con 0x7f238c103d90 2024-04-23T18:15:46.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.392+0000 7f2388ff9700 1 -- 172.21.15.50:0/2759553891 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f238000cd70 con 0x7f238c103d90 2024-04-23T18:15:46.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.392+0000 7f2388ff9700 1 -- 172.21.15.50:0/2759553891 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2380018680 con 0x7f238c103d90 2024-04-23T18:15:46.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.392+0000 7f2391aa0700 1 -- 172.21.15.50:0/2759553891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f238c1a3200 con 0x7f238c103d90 2024-04-23T18:15:46.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.393+0000 7f2391aa0700 1 -- 172.21.15.50:0/2759553891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f238c069730 con 0x7f238c103d90 2024-04-23T18:15:46.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.394+0000 7f2388ff9700 1 -- 172.21.15.50:0/2759553891 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2380010430 con 0x7f238c103d90 2024-04-23T18:15:46.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.394+0000 7f2388ff9700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f23740a3c40 0x7f23740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:46.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.395+0000 7f238b7fe700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f23740a3c40 0x7f23740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:46.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.395+0000 7f2388ff9700 1 -- 172.21.15.50:0/2759553891 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f23800d01a0 con 0x7f238c103d90 2024-04-23T18:15:46.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.395+0000 7f238b7fe700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f23740a3c40 0x7f23740a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f238c06af70 tx=0x7f237c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:46.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.399+0000 7f2388ff9700 1 -- 172.21.15.50:0/2759553891 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f238009d020 con 0x7f238c103d90 2024-04-23T18:15:46.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.610+0000 7f2391aa0700 1 -- 172.21.15.50:0/2759553891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f238c063650 con 0x7f238c103d90 2024-04-23T18:15:46.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.610+0000 7f2388ff9700 1 -- 172.21.15.50:0/2759553891 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f238009c920 con 0x7f238c103d90 2024-04-23T18:15:46.604 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:15:46.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.612+0000 7f2391aa0700 1 -- 172.21.15.50:0/2759553891 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f23740a3c40 msgr2=0x7f23740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:46.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.612+0000 7f2391aa0700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f23740a3c40 0x7f23740a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f238c06af70 tx=0x7f237c009260 comp rx=0 tx=0).stop 2024-04-23T18:15:46.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.612+0000 7f2391aa0700 1 -- 172.21.15.50:0/2759553891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f238c103d90 msgr2=0x7f238c101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:46.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.612+0000 7f2391aa0700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f238c103d90 0x7f238c101d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f238000e9a0 tx=0x7f238000ecb0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.612+0000 7f2391aa0700 1 -- 172.21.15.50:0/2759553891 shutdown_connections 2024-04-23T18:15:46.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.612+0000 7f2391aa0700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f23740a3c40 0x7f23740a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.612+0000 7f2391aa0700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f238c104730 0x7f238c102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.612+0000 7f2391aa0700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f238c100db0 0x7f238c1036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.612+0000 7f2391aa0700 1 --2- 172.21.15.50:0/2759553891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f238c103d90 0x7f238c101d40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:46.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.612+0000 7f2391aa0700 1 -- 172.21.15.50:0/2759553891 >> 172.21.15.50:0/2759553891 conn(0x7f238c0fc710 msgr2=0x7f238c107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:46.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.612+0000 7f2391aa0700 1 -- 172.21.15.50:0/2759553891 shutdown_connections 2024-04-23T18:15:46.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:46.612+0000 7f2391aa0700 1 -- 172.21.15.50:0/2759553891 wait complete. 2024-04-23T18:15:47.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:46 smithi050 ceph-mon[29187]: purged_snaps scrub starts 2024-04-23T18:15:47.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:46 smithi050 ceph-mon[29187]: purged_snaps scrub ok 2024-04-23T18:15:47.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:46 smithi050 ceph-mon[29187]: osd.11 [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853] boot 2024-04-23T18:15:47.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:46 smithi050 ceph-mon[29187]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T18:15:47.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:46 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:47.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:46 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2759553891' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T18:15:47.201 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:46 smithi177 ceph-mon[32308]: purged_snaps scrub starts 2024-04-23T18:15:47.201 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:46 smithi177 ceph-mon[32308]: purged_snaps scrub ok 2024-04-23T18:15:47.201 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:46 smithi177 ceph-mon[32308]: osd.11 [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853] boot 2024-04-23T18:15:47.201 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:46 smithi177 ceph-mon[32308]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T18:15:47.202 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:46 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:47.202 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:46 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2759553891' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T18:15:47.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:46 smithi188 ceph-mon[32694]: purged_snaps scrub starts 2024-04-23T18:15:47.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:46 smithi188 ceph-mon[32694]: purged_snaps scrub ok 2024-04-23T18:15:47.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:46 smithi188 ceph-mon[32694]: osd.11 [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853] boot 2024-04-23T18:15:47.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:46 smithi188 ceph-mon[32694]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T18:15:47.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:46 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:15:47.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:46 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2759553891' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T18:15:47.353 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1713896145,"num_in_osds":12,"osd_in_since":1713896126,"num_remapped_pgs":0} 2024-04-23T18:15:47.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T18:15:47.608 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:48.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:48.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T18:15:48.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: Detected new or changed devices on smithi188 2024-04-23T18:15:48.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:48.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi188 to 4272M 2024-04-23T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:47 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:48.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:48.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T18:15:48.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: Detected new or changed devices on smithi188 2024-04-23T18:15:48.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:48.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:48.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:48.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:48.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:48.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:48.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi188 to 4272M 2024-04-23T18:15:48.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:48.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:48.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:48.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:47 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-23T18:15:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T18:15:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: Detected new or changed devices on smithi188 2024-04-23T18:15:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:15:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi188 to 4272M 2024-04-23T18:15:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:48.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:47 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.196+0000 7fc3c51d3700 1 -- 172.21.15.50:0/615401606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3c0101da0 msgr2=0x7fc3c010f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.196+0000 7fc3c51d3700 1 --2- 172.21.15.50:0/615401606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3c0101da0 0x7fc3c010f810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc3b800b0a0 tx=0x7fc3b800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.196+0000 7fc3c51d3700 1 -- 172.21.15.50:0/615401606 shutdown_connections 2024-04-23T18:15:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.196+0000 7fc3c51d3700 1 --2- 172.21.15.50:0/615401606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3c0101da0 0x7fc3c010f810 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.196+0000 7fc3c51d3700 1 --2- 172.21.15.50:0/615401606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3c0107b00 0x7fc3c0107ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.196+0000 7fc3c51d3700 1 --2- 172.21.15.50:0/615401606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c01013e0 0x7fc3c0101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.196+0000 7fc3c51d3700 1 -- 172.21.15.50:0/615401606 >> 172.21.15.50:0/615401606 conn(0x7fc3c0076c60 msgr2=0x7fc3c0077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:49.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.196+0000 7fc3c51d3700 1 -- 172.21.15.50:0/615401606 shutdown_connections 2024-04-23T18:15:49.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.196+0000 7fc3c51d3700 1 -- 172.21.15.50:0/615401606 wait complete. 2024-04-23T18:15:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.197+0000 7fc3c51d3700 1 Processor -- start 2024-04-23T18:15:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.197+0000 7fc3c51d3700 1 -- start start 2024-04-23T18:15:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.197+0000 7fc3c51d3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3c01013e0 0x7fc3c01025c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.197+0000 7fc3c51d3700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c0101da0 0x7fc3c0102b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.197+0000 7fc3c51d3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3c0107b00 0x7fc3c0103040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.197+0000 7fc3c51d3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c0112c90 con 0x7fc3c0107b00 2024-04-23T18:15:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.197+0000 7fc3c51d3700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c0112b10 con 0x7fc3c0101da0 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.197+0000 7fc3c51d3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c0112e10 con 0x7fc3c01013e0 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.197+0000 7fc3bf59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3c0107b00 0x7fc3c0103040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.197+0000 7fc3bf59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3c0107b00 0x7fc3c0103040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40534/0 (socket says 172.21.15.50:40534) 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.197+0000 7fc3bf59e700 1 -- 172.21.15.50:0/54472163 learned_addr learned my addr 172.21.15.50:0/54472163 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3bed9d700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3c01013e0 0x7fc3c01025c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3be59c700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c0101da0 0x7fc3c0102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3bf59e700 1 -- 172.21.15.50:0/54472163 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3c01013e0 msgr2=0x7fc3c01025c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3bf59e700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3c01013e0 0x7fc3c01025c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3bf59e700 1 -- 172.21.15.50:0/54472163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c0101da0 msgr2=0x7fc3c0102b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3bf59e700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c0101da0 0x7fc3c0102b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3bf59e700 1 -- 172.21.15.50:0/54472163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c01a7590 con 0x7fc3c0107b00 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3be59c700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c0101da0 0x7fc3c0102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3bf59e700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3c0107b00 0x7fc3c0103040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80062a0 tx=0x7fc3b8004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:49.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3a7fff700 1 -- 172.21.15.50:0/54472163 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8021070 con 0x7fc3c0107b00 2024-04-23T18:15:49.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3a7fff700 1 -- 172.21.15.50:0/54472163 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b8003c00 con 0x7fc3c0107b00 2024-04-23T18:15:49.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3c51d3700 1 -- 172.21.15.50:0/54472163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c01a7820 con 0x7fc3c0107b00 2024-04-23T18:15:49.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3a7fff700 1 -- 172.21.15.50:0/54472163 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8004830 con 0x7fc3c0107b00 2024-04-23T18:15:49.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.198+0000 7fc3c51d3700 1 -- 172.21.15.50:0/54472163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c01a7cd0 con 0x7fc3c0107b00 2024-04-23T18:15:49.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.199+0000 7fc3c51d3700 1 -- 172.21.15.50:0/54472163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3c0069730 con 0x7fc3c0107b00 2024-04-23T18:15:49.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.200+0000 7fc3a7fff700 1 -- 172.21.15.50:0/54472163 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc3b8007a00 con 0x7fc3c0107b00 2024-04-23T18:15:49.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.201+0000 7fc3a7fff700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3a80a3c40 0x7fc3a80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:49.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.201+0000 7fc3bed9d700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3a80a3c40 0x7fc3a80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:49.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.201+0000 7fc3a7fff700 1 -- 172.21.15.50:0/54472163 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc3b800f020 con 0x7fc3c0107b00 2024-04-23T18:15:49.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.201+0000 7fc3bed9d700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3a80a3c40 0x7fc3a80a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00781e0 tx=0x7fc3b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:49.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.204+0000 7fc3a7fff700 1 -- 172.21.15.50:0/54472163 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3b802a020 con 0x7fc3c0107b00 2024-04-23T18:15:49.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.413+0000 7fc3c51d3700 1 -- 172.21.15.50:0/54472163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3c002e1e0 con 0x7fc3c0107b00 2024-04-23T18:15:49.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.414+0000 7fc3a7fff700 1 -- 172.21.15.50:0/54472163 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18418 (secure 0 0 0) 0x7fc3b809e830 con 0x7fc3c0107b00 2024-04-23T18:15:49.409 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:15:49.409 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":83,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:15:46.911832+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:15:49.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.418+0000 7fc3c51d3700 1 -- 172.21.15.50:0/54472163 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3a80a3c40 msgr2=0x7fc3a80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.418+0000 7fc3c51d3700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3a80a3c40 0x7fc3a80a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00781e0 tx=0x7fc3b0009260 comp rx=0 tx=0).stop 2024-04-23T18:15:49.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.418+0000 7fc3c51d3700 1 -- 172.21.15.50:0/54472163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3c0107b00 msgr2=0x7fc3c0103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:49.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.418+0000 7fc3c51d3700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3c0107b00 0x7fc3c0103040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80062a0 tx=0x7fc3b8004d30 comp rx=0 tx=0).stop 2024-04-23T18:15:49.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.418+0000 7fc3c51d3700 1 -- 172.21.15.50:0/54472163 shutdown_connections 2024-04-23T18:15:49.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.418+0000 7fc3c51d3700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3c0107b00 0x7fc3c0103040 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.418+0000 7fc3c51d3700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3c01013e0 0x7fc3c01025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.418+0000 7fc3c51d3700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3a80a3c40 0x7fc3a80a6100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.418+0000 7fc3c51d3700 1 --2- 172.21.15.50:0/54472163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3c0101da0 0x7fc3c0102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:49.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.418+0000 7fc3c51d3700 1 -- 172.21.15.50:0/54472163 >> 172.21.15.50:0/54472163 conn(0x7fc3c0076c60 msgr2=0x7fc3c0100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:49.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.418+0000 7fc3c51d3700 1 -- 172.21.15.50:0/54472163 shutdown_connections 2024-04-23T18:15:49.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:49.418+0000 7fc3c51d3700 1 -- 172.21.15.50:0/54472163 wait complete. 2024-04-23T18:15:50.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:49 smithi050 ceph-mon[29187]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:50.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:49 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/54472163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:50.166 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:12:02.493647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:15:50.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-23T18:15:50.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:49 smithi177 ceph-mon[32308]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:50.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:49 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/54472163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:50.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:49 smithi188 ceph-mon[32694]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:50.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:49 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/54472163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:15:52.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.104+0000 7fafa9d88700 1 -- 172.21.15.50:0/4149386075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa4108a60 msgr2=0x7fafa4108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:52.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.104+0000 7fafa9d88700 1 --2- 172.21.15.50:0/4149386075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa4108a60 0x7fafa4108e40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faf94009990 tx=0x7faf94009ca0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.104+0000 7fafa9d88700 1 -- 172.21.15.50:0/4149386075 shutdown_connections 2024-04-23T18:15:52.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.104+0000 7fafa9d88700 1 --2- 172.21.15.50:0/4149386075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafa406bbb0 0x7fafa410f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.104+0000 7fafa9d88700 1 --2- 172.21.15.50:0/4149386075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa4108a60 0x7fafa4108e40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.104+0000 7fafa9d88700 1 --2- 172.21.15.50:0/4149386075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafa406b1f0 0x7fafa406b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.104+0000 7fafa9d88700 1 -- 172.21.15.50:0/4149386075 >> 172.21.15.50:0/4149386075 conn(0x7fafa40789e0 msgr2=0x7fafa4078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:52.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.104+0000 7fafa9d88700 1 -- 172.21.15.50:0/4149386075 shutdown_connections 2024-04-23T18:15:52.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.104+0000 7fafa9d88700 1 -- 172.21.15.50:0/4149386075 wait complete. 2024-04-23T18:15:52.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:51 smithi050 ceph-mon[29187]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:52.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.104+0000 7fafa9d88700 1 Processor -- start 2024-04-23T18:15:52.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.105+0000 7fafa9d88700 1 -- start start 2024-04-23T18:15:52.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.105+0000 7fafa9d88700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafa406b1f0 0x7fafa41991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.105+0000 7fafa9d88700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa406bbb0 0x7fafa4199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.105+0000 7fafa9d88700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafa4108a60 0x7fafa419da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.105+0000 7fafa9d88700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fafa4112ca0 con 0x7fafa406bbb0 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.105+0000 7fafa9d88700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fafa4112b20 con 0x7fafa4108a60 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.105+0000 7fafa9d88700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fafa4112e20 con 0x7fafa406b1f0 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.105+0000 7fafa2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa406bbb0 0x7fafa4199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.105+0000 7fafa2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa406bbb0 0x7fafa4199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39334/0 (socket says 172.21.15.50:39334) 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.105+0000 7fafa2ffd700 1 -- 172.21.15.50:0/144132088 learned_addr learned my addr 172.21.15.50:0/144132088 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.105+0000 7fafa37fe700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafa406b1f0 0x7fafa41991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.105+0000 7fafa3fff700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafa4108a60 0x7fafa419da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.106+0000 7fafa2ffd700 1 -- 172.21.15.50:0/144132088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafa406b1f0 msgr2=0x7fafa41991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.106+0000 7fafa2ffd700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafa406b1f0 0x7fafa41991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.106+0000 7fafa2ffd700 1 -- 172.21.15.50:0/144132088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafa4108a60 msgr2=0x7fafa419da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.106+0000 7fafa2ffd700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafa4108a60 0x7fafa419da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.106+0000 7fafa2ffd700 1 -- 172.21.15.50:0/144132088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa419e1b0 con 0x7fafa406bbb0 2024-04-23T18:15:52.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.106+0000 7fafa2ffd700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa406bbb0 0x7fafa4199730 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7faf9800b530 tx=0x7faf9800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:52.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.106+0000 7fafa0ff9700 1 -- 172.21.15.50:0/144132088 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98015020 con 0x7fafa406bbb0 2024-04-23T18:15:52.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.106+0000 7fafa9d88700 1 -- 172.21.15.50:0/144132088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa419e4a0 con 0x7fafa406bbb0 2024-04-23T18:15:52.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.106+0000 7fafa0ff9700 1 -- 172.21.15.50:0/144132088 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf98011ce0 con 0x7fafa406bbb0 2024-04-23T18:15:52.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.106+0000 7fafa0ff9700 1 -- 172.21.15.50:0/144132088 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98010dd0 con 0x7fafa406bbb0 2024-04-23T18:15:52.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.106+0000 7fafa9d88700 1 -- 172.21.15.50:0/144132088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa41a5d20 con 0x7fafa406bbb0 2024-04-23T18:15:52.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.107+0000 7fafa9d88700 1 -- 172.21.15.50:0/144132088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafa410b060 con 0x7fafa406bbb0 2024-04-23T18:15:52.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.109+0000 7fafa0ff9700 1 -- 172.21.15.50:0/144132088 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7faf98018020 con 0x7fafa406bbb0 2024-04-23T18:15:52.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.110+0000 7fafa0ff9700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faf8c0a3c40 0x7faf8c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:52.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.110+0000 7fafa37fe700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faf8c0a3c40 0x7faf8c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:52.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.110+0000 7fafa0ff9700 1 -- 172.21.15.50:0/144132088 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7faf980cfc70 con 0x7fafa406bbb0 2024-04-23T18:15:52.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.110+0000 7fafa37fe700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faf8c0a3c40 0x7faf8c0a6100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faf940098e0 tx=0x7faf94005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:52.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.114+0000 7fafa0ff9700 1 -- 172.21.15.50:0/144132088 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf9801f020 con 0x7fafa406bbb0 2024-04-23T18:15:52.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:51 smithi177 ceph-mon[32308]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:52.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.318+0000 7fafa9d88700 1 -- 172.21.15.50:0/144132088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fafa419a530 con 0x7fafa406bbb0 2024-04-23T18:15:52.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.319+0000 7fafa0ff9700 1 -- 172.21.15.50:0/144132088 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7fafa419a530 con 0x7fafa406bbb0 2024-04-23T18:15:52.312 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-23T18:15:52.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.321+0000 7fafa9d88700 1 -- 172.21.15.50:0/144132088 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faf8c0a3c40 msgr2=0x7faf8c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:52.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.321+0000 7fafa9d88700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faf8c0a3c40 0x7faf8c0a6100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faf940098e0 tx=0x7faf94005960 comp rx=0 tx=0).stop 2024-04-23T18:15:52.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.321+0000 7fafa9d88700 1 -- 172.21.15.50:0/144132088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa406bbb0 msgr2=0x7fafa4199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:52.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.321+0000 7fafa9d88700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa406bbb0 0x7fafa4199730 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7faf9800b530 tx=0x7faf9800b840 comp rx=0 tx=0).stop 2024-04-23T18:15:52.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.321+0000 7fafa9d88700 1 -- 172.21.15.50:0/144132088 shutdown_connections 2024-04-23T18:15:52.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.321+0000 7fafa9d88700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faf8c0a3c40 0x7faf8c0a6100 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.321+0000 7fafa9d88700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafa4108a60 0x7fafa419da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.321+0000 7fafa9d88700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafa406b1f0 0x7fafa41991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.321+0000 7fafa9d88700 1 --2- 172.21.15.50:0/144132088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa406bbb0 0x7fafa4199730 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:52.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.321+0000 7fafa9d88700 1 -- 172.21.15.50:0/144132088 >> 172.21.15.50:0/144132088 conn(0x7fafa40789e0 msgr2=0x7fafa40ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:52.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.321+0000 7fafa9d88700 1 -- 172.21.15.50:0/144132088 shutdown_connections 2024-04-23T18:15:52.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:52.321+0000 7fafa9d88700 1 -- 172.21.15.50:0/144132088 wait complete. 2024-04-23T18:15:52.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:51 smithi188 ceph-mon[32694]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:53.005 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:52 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/144132088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:53.007 INFO:tasks.cephadm:Adding mds.a on smithi050 2024-04-23T18:15:53.007 INFO:tasks.cephadm:Adding mds.d on smithi050 2024-04-23T18:15:53.007 INFO:tasks.cephadm:Adding mds.g on smithi050 2024-04-23T18:15:53.007 INFO:tasks.cephadm:Adding mds.j on smithi050 2024-04-23T18:15:53.007 INFO:tasks.cephadm:Adding mds.b on smithi177 2024-04-23T18:15:53.008 INFO:tasks.cephadm:Adding mds.e on smithi177 2024-04-23T18:15:53.008 INFO:tasks.cephadm:Adding mds.h on smithi177 2024-04-23T18:15:53.008 INFO:tasks.cephadm:Adding mds.k on smithi177 2024-04-23T18:15:53.008 INFO:tasks.cephadm:Adding mds.c on smithi188 2024-04-23T18:15:53.008 INFO:tasks.cephadm:Adding mds.f on smithi188 2024-04-23T18:15:53.008 INFO:tasks.cephadm:Adding mds.i on smithi188 2024-04-23T18:15:53.008 INFO:tasks.cephadm:Adding mds.l on smithi188 2024-04-23T18:15:53.008 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi050=a;smithi050=d;smithi050=g;smithi050=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l' 2024-04-23T18:15:53.258 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:15:53.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:52 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/144132088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:53.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:52 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/144132088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:54.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:53 smithi050 ceph-mon[29187]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:54.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:53 smithi177 ceph-mon[32308]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:54.306 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:53 smithi188 ceph-mon[32694]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:54.795 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.802+0000 7fd824ced700 1 -- 172.21.15.188:0/1869171458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8201014f0 msgr2=0x7fd82010b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:54.796 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.802+0000 7fd824ced700 1 --2- 172.21.15.188:0/1869171458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8201014f0 0x7fd82010b0a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd81800b0a0 tx=0x7fd81800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:15:54.796 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.802+0000 7fd824ced700 1 -- 172.21.15.188:0/1869171458 shutdown_connections 2024-04-23T18:15:54.796 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.802+0000 7fd824ced700 1 --2- 172.21.15.188:0/1869171458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8201014f0 0x7fd82010b0a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:54.796 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.802+0000 7fd824ced700 1 --2- 172.21.15.188:0/1869171458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8201040b0 0x7fd8201044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:54.796 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.802+0000 7fd824ced700 1 --2- 172.21.15.188:0/1869171458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd820100b30 0x7fd820100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:54.796 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.802+0000 7fd824ced700 1 -- 172.21.15.188:0/1869171458 >> 172.21.15.188:0/1869171458 conn(0x7fd8200fc6b0 msgr2=0x7fd8200feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:54.796 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.802+0000 7fd824ced700 1 -- 172.21.15.188:0/1869171458 shutdown_connections 2024-04-23T18:15:54.796 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.802+0000 7fd824ced700 1 -- 172.21.15.188:0/1869171458 wait complete. 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd824ced700 1 Processor -- start 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd824ced700 1 -- start start 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd824ced700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd820100b30 0x7fd820198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd824ced700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8201014f0 0x7fd820199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd824ced700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8201040b0 0x7fd8201a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd824ced700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd82010e540 con 0x7fd820100b30 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd824ced700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd82010e3c0 con 0x7fd8201014f0 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd824ced700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd82010e6c0 con 0x7fd8201040b0 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd81ed9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8201040b0 0x7fd8201a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd81ed9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8201040b0 0x7fd8201a00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:41852/0 (socket says 172.21.15.188:41852) 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd81ed9d700 1 -- 172.21.15.188:0/708744288 learned_addr learned my addr 172.21.15.188:0/708744288 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd81e59c700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd820100b30 0x7fd820198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:54.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd81dd9b700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8201014f0 0x7fd820199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd81ed9d700 1 -- 172.21.15.188:0/708744288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8201014f0 msgr2=0x7fd820199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd81ed9d700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8201014f0 0x7fd820199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd81ed9d700 1 -- 172.21.15.188:0/708744288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd820100b30 msgr2=0x7fd820198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd81ed9d700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd820100b30 0x7fd820198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd81ed9d700 1 -- 172.21.15.188:0/708744288 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8201a0780 con 0x7fd8201040b0 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.803+0000 7fd81dd9b700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8201014f0 0x7fd820199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.804+0000 7fd81ed9d700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8201040b0 0x7fd8201a00e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd818004e40 tx=0x7fd818004e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.804+0000 7fd80f7fe700 1 -- 172.21.15.188:0/708744288 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818021070 con 0x7fd8201040b0 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.804+0000 7fd80f7fe700 1 -- 172.21.15.188:0/708744288 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd818007c80 con 0x7fd8201040b0 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.804+0000 7fd80f7fe700 1 -- 172.21.15.188:0/708744288 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818004850 con 0x7fd8201040b0 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.804+0000 7fd824ced700 1 -- 172.21.15.188:0/708744288 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8201a0a10 con 0x7fd8201040b0 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.804+0000 7fd824ced700 1 -- 172.21.15.188:0/708744288 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8201a0e70 con 0x7fd8201040b0 2024-04-23T18:15:54.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.805+0000 7fd824ced700 1 -- 172.21.15.188:0/708744288 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd820069730 con 0x7fd8201040b0 2024-04-23T18:15:54.801 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.806+0000 7fd80f7fe700 1 -- 172.21.15.188:0/708744288 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd818004000 con 0x7fd8201040b0 2024-04-23T18:15:54.801 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.807+0000 7fd80f7fe700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd8080a3c40 0x7fd8080a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:54.801 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.807+0000 7fd80f7fe700 1 -- 172.21.15.188:0/708744288 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fd8180d1120 con 0x7fd8201040b0 2024-04-23T18:15:54.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.807+0000 7fd81e59c700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd8080a3c40 0x7fd8080a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:54.802 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.807+0000 7fd81e59c700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd8080a3c40 0x7fd8080a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd82006aef0 tx=0x7fd81000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:54.804 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:54.811+0000 7fd80f7fe700 1 -- 172.21.15.188:0/708744288 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd81809df90 con 0x7fd8201040b0 2024-04-23T18:15:55.010 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.016+0000 7fd824ced700 1 -- 172.21.15.188:0/708744288 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi050=a;smithi050=d;smithi050=g;smithi050=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}) -- 0x7fd820063080 con 0x7fd8080a3c40 2024-04-23T18:15:55.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.024+0000 7fd80f7fe700 1 -- 172.21.15.188:0/708744288 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fd820063080 con 0x7fd8080a3c40 2024-04-23T18:15:55.017 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mds.all update... 2024-04-23T18:15:55.019 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.026+0000 7fd824ced700 1 -- 172.21.15.188:0/708744288 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd8080a3c40 msgr2=0x7fd8080a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:55.019 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.026+0000 7fd824ced700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd8080a3c40 0x7fd8080a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd82006aef0 tx=0x7fd81000b410 comp rx=0 tx=0).stop 2024-04-23T18:15:55.020 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.026+0000 7fd824ced700 1 -- 172.21.15.188:0/708744288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8201040b0 msgr2=0x7fd8201a00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:55.020 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.026+0000 7fd824ced700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8201040b0 0x7fd8201a00e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd818004e40 tx=0x7fd818004e70 comp rx=0 tx=0).stop 2024-04-23T18:15:55.020 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.026+0000 7fd824ced700 1 -- 172.21.15.188:0/708744288 shutdown_connections 2024-04-23T18:15:55.020 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.026+0000 7fd824ced700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8201040b0 0x7fd8201a00e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:55.020 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.026+0000 7fd824ced700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd820100b30 0x7fd820198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:55.020 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.026+0000 7fd824ced700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd8080a3c40 0x7fd8080a6100 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:55.020 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.026+0000 7fd824ced700 1 --2- 172.21.15.188:0/708744288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8201014f0 0x7fd820199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:55.020 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.026+0000 7fd824ced700 1 -- 172.21.15.188:0/708744288 >> 172.21.15.188:0/708744288 conn(0x7fd8200fc6b0 msgr2=0x7fd8201070f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:55.020 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.026+0000 7fd824ced700 1 -- 172.21.15.188:0/708744288 shutdown_connections 2024-04-23T18:15:55.020 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-23T18:15:55.026+0000 7fd824ced700 1 -- 172.21.15.188:0/708744288 wait complete. 2024-04-23T18:15:55.688 DEBUG:teuthology.orchestra.run.smithi050:mds.a> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.a.service 2024-04-23T18:15:55.692 DEBUG:teuthology.orchestra.run.smithi050:mds.d> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.d.service 2024-04-23T18:15:55.735 DEBUG:teuthology.orchestra.run.smithi050:mds.g> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.g.service 2024-04-23T18:15:55.778 DEBUG:teuthology.orchestra.run.smithi050:mds.j> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.j.service 2024-04-23T18:15:55.821 DEBUG:teuthology.orchestra.run.smithi177:mds.b> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.b.service 2024-04-23T18:15:55.823 DEBUG:teuthology.orchestra.run.smithi177:mds.e> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.e.service 2024-04-23T18:15:55.866 DEBUG:teuthology.orchestra.run.smithi177:mds.h> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.h.service 2024-04-23T18:15:55.910 DEBUG:teuthology.orchestra.run.smithi177:mds.k> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.k.service 2024-04-23T18:15:55.953 DEBUG:teuthology.orchestra.run.smithi188:mds.c> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.c.service 2024-04-23T18:15:55.955 DEBUG:teuthology.orchestra.run.smithi188:mds.f> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.f.service 2024-04-23T18:15:55.998 DEBUG:teuthology.orchestra.run.smithi188:mds.i> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.i.service 2024-04-23T18:15:56.041 DEBUG:teuthology.orchestra.run.smithi188:mds.l> sudo journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.l.service 2024-04-23T18:15:56.084 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-23T18:15:56.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:15:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.225+0000 7f1c5b872640 1 -- 172.21.15.50:0/2328071787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c54107c90 msgr2=0x7f1c5410a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.225+0000 7f1c5b872640 1 --2- 172.21.15.50:0/2328071787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c54107c90 0x7f1c5410a0e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c009900 tx=0x7f1c4c02a3f0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.225+0000 7f1c5b872640 1 -- 172.21.15.50:0/2328071787 shutdown_connections 2024-04-23T18:15:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.225+0000 7f1c5b872640 1 --2- 172.21.15.50:0/2328071787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c54107c90 0x7f1c5410a0e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.225+0000 7f1c5b872640 1 --2- 172.21.15.50:0/2328071787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c54105360 0x7f1c54107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.225+0000 7f1c5b872640 1 --2- 172.21.15.50:0/2328071787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c54102a30 0x7f1c54104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.225+0000 7f1c5b872640 1 -- 172.21.15.50:0/2328071787 >> 172.21.15.50:0/2328071787 conn(0x7f1c540fc440 msgr2=0x7f1c540fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c5b872640 1 -- 172.21.15.50:0/2328071787 shutdown_connections 2024-04-23T18:15:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c5b872640 1 -- 172.21.15.50:0/2328071787 wait complete. 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c5b872640 1 Processor -- start 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c5b872640 1 -- start start 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c5b872640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c54102a30 0x7f1c54198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c5b872640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c54105360 0x7f1c54198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c5b872640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c54107c90 0x7f1c5419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c5b872640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5410db40 con 0x7f1c54107c90 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c5b872640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5410d9c0 con 0x7f1c54102a30 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c5b872640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5410dcc0 con 0x7f1c54105360 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c59e0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c54107c90 0x7f1c5419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c59e0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c54107c90 0x7f1c5419f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39352/0 (socket says 172.21.15.50:39352) 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.226+0000 7f1c59e0f640 1 -- 172.21.15.50:0/2572881110 learned_addr learned my addr 172.21.15.50:0/2572881110 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c58e0d640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c54105360 0x7f1c54198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c5960e640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c54102a30 0x7f1c54198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c59e0f640 1 -- 172.21.15.50:0/2572881110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c54105360 msgr2=0x7f1c54198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c59e0f640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c54105360 0x7f1c54198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c59e0f640 1 -- 172.21.15.50:0/2572881110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c54102a30 msgr2=0x7f1c54198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c59e0f640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c54102a30 0x7f1c54198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c59e0f640 1 -- 172.21.15.50:0/2572881110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5419fd60 con 0x7f1c54107c90 2024-04-23T18:15:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c59e0f640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c54107c90 0x7f1c5419f6f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c005ab0 tx=0x7f1c4c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c4a7fc640 1 -- 172.21.15.50:0/2572881110 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c03f070 con 0x7f1c54107c90 2024-04-23T18:15:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c4a7fc640 1 -- 172.21.15.50:0/2572881110 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c4c031c50 con 0x7f1c54107c90 2024-04-23T18:15:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c4a7fc640 1 -- 172.21.15.50:0/2572881110 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c035030 con 0x7f1c54107c90 2024-04-23T18:15:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c5b872640 1 -- 172.21.15.50:0/2572881110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c5419fff0 con 0x7f1c54107c90 2024-04-23T18:15:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.227+0000 7f1c5b872640 1 -- 172.21.15.50:0/2572881110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c541a0450 con 0x7f1c54107c90 2024-04-23T18:15:56.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.228+0000 7f1c5b872640 1 -- 172.21.15.50:0/2572881110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c540fdf30 con 0x7f1c54107c90 2024-04-23T18:15:56.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.229+0000 7f1c4a7fc640 1 -- 172.21.15.50:0/2572881110 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1c4c042070 con 0x7f1c54107c90 2024-04-23T18:15:56.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.230+0000 7f1c4a7fc640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1c400a3bb0 0x7f1c400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:56.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.230+0000 7f1c5960e640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1c400a3bb0 0x7f1c400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:56.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.230+0000 7f1c4a7fc640 1 -- 172.21.15.50:0/2572881110 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f1c4c0f07a0 con 0x7f1c54107c90 2024-04-23T18:15:56.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.230+0000 7f1c5960e640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1c400a3bb0 0x7f1c400a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1c50005fd0 tx=0x7f1c50009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:56.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.234+0000 7f1c4a7fc640 1 -- 172.21.15.50:0/2572881110 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c4c0bd620 con 0x7f1c54107c90 2024-04-23T18:15:56.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:56 smithi177 ceph-mon[32308]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:56.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:56 smithi177 ceph-mon[32308]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi050=a;smithi050=d;smithi050=g;smithi050=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:56.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:56 smithi177 ceph-mon[32308]: Saving service mds.all spec with placement smithi050=a;smithi050=d;smithi050=g;smithi050=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2024-04-23T18:15:56.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:56 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:56.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:56 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:56.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:56 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:56.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:56 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:56.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:56 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:56.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:56 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:15:56.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:56 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:15:56.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:56 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:56 smithi188 ceph-mon[32694]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:56 smithi188 ceph-mon[32694]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi050=a;smithi050=d;smithi050=g;smithi050=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:56 smithi188 ceph-mon[32694]: Saving service mds.all spec with placement smithi050=a;smithi050=d;smithi050=g;smithi050=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2024-04-23T18:15:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:56 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:56 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:56.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:56 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:56.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:56 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:56.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:56 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:56.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:56 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:15:56.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:56 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:15:56.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:56 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:56.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.421+0000 7f1c5b872640 1 -- 172.21.15.50:0/2572881110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c54062b50 con 0x7f1c54107c90 2024-04-23T18:15:56.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.422+0000 7f1c4a7fc640 1 -- 172.21.15.50:0/2572881110 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18418 (secure 0 0 0) 0x7f1c4c0bcf20 con 0x7f1c54107c90 2024-04-23T18:15:56.417 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:15:56.417 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":83,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:15:46.911832+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:15:56.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.424+0000 7f1c5b872640 1 -- 172.21.15.50:0/2572881110 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1c400a3bb0 msgr2=0x7f1c400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.424+0000 7f1c5b872640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1c400a3bb0 0x7f1c400a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1c50005fd0 tx=0x7f1c50009500 comp rx=0 tx=0).stop 2024-04-23T18:15:56.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.424+0000 7f1c5b872640 1 -- 172.21.15.50:0/2572881110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c54107c90 msgr2=0x7f1c5419f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.424+0000 7f1c5b872640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c54107c90 0x7f1c5419f6f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c005ab0 tx=0x7f1c4c003b90 comp rx=0 tx=0).stop 2024-04-23T18:15:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.424+0000 7f1c5b872640 1 -- 172.21.15.50:0/2572881110 shutdown_connections 2024-04-23T18:15:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.424+0000 7f1c5b872640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1c400a3bb0 0x7f1c400a6070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.424+0000 7f1c5b872640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c54107c90 0x7f1c5419f6f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.424+0000 7f1c5b872640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c54105360 0x7f1c54198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.424+0000 7f1c5b872640 1 --2- 172.21.15.50:0/2572881110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c54102a30 0x7f1c54198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.424+0000 7f1c5b872640 1 -- 172.21.15.50:0/2572881110 >> 172.21.15.50:0/2572881110 conn(0x7f1c540fc440 msgr2=0x7f1c54103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.424+0000 7f1c5b872640 1 -- 172.21.15.50:0/2572881110 shutdown_connections 2024-04-23T18:15:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.424+0000 7f1c5b872640 1 -- 172.21.15.50:0/2572881110 wait complete. 2024-04-23T18:15:56.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:56 smithi050 ceph-mon[29187]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:56.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:56 smithi050 ceph-mon[29187]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi050=a;smithi050=d;smithi050=g;smithi050=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:15:56.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:56 smithi050 ceph-mon[29187]: Saving service mds.all spec with placement smithi050=a;smithi050=d;smithi050=g;smithi050=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2024-04-23T18:15:56.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:56 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:56.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:56 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:15:56.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:56 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:56.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:56 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:15:56.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:56 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:56.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:56 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:15:56.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:56 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:15:56.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:56 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:56.430 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:12:02.493647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:15:56.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:15:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.542+0000 7f76f55c6640 1 -- 172.21.15.50:0/4092745841 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76f0107c90 msgr2=0x7f76f010a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.542+0000 7f76f55c6640 1 --2- 172.21.15.50:0/4092745841 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76f0107c90 0x7f76f010a0e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009ca0 tx=0x7f76e802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.542+0000 7f76f55c6640 1 -- 172.21.15.50:0/4092745841 shutdown_connections 2024-04-23T18:15:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.542+0000 7f76f55c6640 1 --2- 172.21.15.50:0/4092745841 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76f0107c90 0x7f76f010a0e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.542+0000 7f76f55c6640 1 --2- 172.21.15.50:0/4092745841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76f0105360 0x7f76f0107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.542+0000 7f76f55c6640 1 --2- 172.21.15.50:0/4092745841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0102a30 0x7f76f0104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.542+0000 7f76f55c6640 1 -- 172.21.15.50:0/4092745841 >> 172.21.15.50:0/4092745841 conn(0x7f76f00fc440 msgr2=0x7f76f00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.542+0000 7f76f55c6640 1 -- 172.21.15.50:0/4092745841 shutdown_connections 2024-04-23T18:15:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.542+0000 7f76f55c6640 1 -- 172.21.15.50:0/4092745841 wait complete. 2024-04-23T18:15:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.543+0000 7f76f55c6640 1 Processor -- start 2024-04-23T18:15:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.543+0000 7f76f55c6640 1 -- start start 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.543+0000 7f76f55c6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76f0102a30 0x7f76f01023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.543+0000 7f76f55c6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76f0105360 0x7f76f01009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.543+0000 7f76f55c6640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0107c90 0x7f76f0100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.543+0000 7f76f55c6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f76f010daf0 con 0x7f76f0105360 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.543+0000 7f76f55c6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f76f010d970 con 0x7f76f0102a30 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.543+0000 7f76f55c6640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f76f010dc70 con 0x7f76f0107c90 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.543+0000 7f76ee7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76f0105360 0x7f76f01009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.543+0000 7f76ee7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76f0105360 0x7f76f01009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39374/0 (socket says 172.21.15.50:39374) 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.543+0000 7f76ee7fc640 1 -- 172.21.15.50:0/1206778399 learned_addr learned my addr 172.21.15.50:0/1206778399 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.543+0000 7f76ef7fe640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0107c90 0x7f76f0100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76eeffd640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76f0102a30 0x7f76f01023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76ee7fc640 1 -- 172.21.15.50:0/1206778399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0107c90 msgr2=0x7f76f0100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76ee7fc640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0107c90 0x7f76f0100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76ee7fc640 1 -- 172.21.15.50:0/1206778399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76f0102a30 msgr2=0x7f76f01023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76ee7fc640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76f0102a30 0x7f76f01023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76ee7fc640 1 -- 172.21.15.50:0/1206778399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f01017b0 con 0x7f76f0105360 2024-04-23T18:15:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76eeffd640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76f0102a30 0x7f76f01023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:15:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76ee7fc640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76f0105360 0x7f76f01009e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76e400e9c0 tx=0x7f76e400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76d7fff640 1 -- 172.21.15.50:0/1206778399 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e400cd80 con 0x7f76f0105360 2024-04-23T18:15:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76d7fff640 1 -- 172.21.15.50:0/1206778399 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e40054d0 con 0x7f76f0105360 2024-04-23T18:15:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76f55c6640 1 -- 172.21.15.50:0/1206778399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f01a22f0 con 0x7f76f0105360 2024-04-23T18:15:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76d7fff640 1 -- 172.21.15.50:0/1206778399 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e4010430 con 0x7f76f0105360 2024-04-23T18:15:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.544+0000 7f76f55c6640 1 -- 172.21.15.50:0/1206778399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f01a2840 con 0x7f76f0105360 2024-04-23T18:15:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.545+0000 7f76f55c6640 1 -- 172.21.15.50:0/1206778399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f00fdf30 con 0x7f76f0105360 2024-04-23T18:15:56.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.547+0000 7f76d7fff640 1 -- 172.21.15.50:0/1206778399 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f76e4004fb0 con 0x7f76f0105360 2024-04-23T18:15:56.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.547+0000 7f76d7fff640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f76d80a3bb0 0x7f76d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:56.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.547+0000 7f76eeffd640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f76d80a3bb0 0x7f76d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:56.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.547+0000 7f76d7fff640 1 -- 172.21.15.50:0/1206778399 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f76e40d0730 con 0x7f76f0105360 2024-04-23T18:15:56.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.548+0000 7f76eeffd640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f76d80a3bb0 0x7f76d80a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009b50 tx=0x7f76e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:56.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.551+0000 7f76d7fff640 1 -- 172.21.15.50:0/1206778399 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f76e4053ff0 con 0x7f76f0105360 2024-04-23T18:15:56.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.737+0000 7f76f55c6640 1 -- 172.21.15.50:0/1206778399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f76f002d830 con 0x7f76f0105360 2024-04-23T18:15:56.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.738+0000 7f76d7fff640 1 -- 172.21.15.50:0/1206778399 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f76e4016330 con 0x7f76f0105360 2024-04-23T18:15:56.732 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-23T18:15:56.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.740+0000 7f76f55c6640 1 -- 172.21.15.50:0/1206778399 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f76d80a3bb0 msgr2=0x7f76d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.740+0000 7f76f55c6640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f76d80a3bb0 0x7f76d80a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009b50 tx=0x7f76e0005f90 comp rx=0 tx=0).stop 2024-04-23T18:15:56.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.740+0000 7f76f55c6640 1 -- 172.21.15.50:0/1206778399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76f0105360 msgr2=0x7f76f01009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.740+0000 7f76f55c6640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76f0105360 0x7f76f01009e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76e400e9c0 tx=0x7f76e400ee80 comp rx=0 tx=0).stop 2024-04-23T18:15:56.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.740+0000 7f76f55c6640 1 -- 172.21.15.50:0/1206778399 shutdown_connections 2024-04-23T18:15:56.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.740+0000 7f76f55c6640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f76d80a3bb0 0x7f76d80a6070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.740+0000 7f76f55c6640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76f0107c90 0x7f76f0100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.740+0000 7f76f55c6640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76f0105360 0x7f76f01009e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.740+0000 7f76f55c6640 1 --2- 172.21.15.50:0/1206778399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76f0102a30 0x7f76f01023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.740+0000 7f76f55c6640 1 -- 172.21.15.50:0/1206778399 >> 172.21.15.50:0/1206778399 conn(0x7f76f00fc440 msgr2=0x7f76f01032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:56.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.740+0000 7f76f55c6640 1 -- 172.21.15.50:0/1206778399 shutdown_connections 2024-04-23T18:15:56.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.740+0000 7f76f55c6640 1 -- 172.21.15.50:0/1206778399 wait complete. 2024-04-23T18:15:56.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:15:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.853+0000 7f5181d15640 1 -- 172.21.15.50:0/3729023314 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f517c107c90 msgr2=0x7f517c10a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.853+0000 7f5181d15640 1 --2- 172.21.15.50:0/3729023314 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f517c107c90 0x7f517c10a0e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5174009ca0 tx=0x7f517402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.853+0000 7f5181d15640 1 -- 172.21.15.50:0/3729023314 shutdown_connections 2024-04-23T18:15:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.853+0000 7f5181d15640 1 --2- 172.21.15.50:0/3729023314 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f517c107c90 0x7f517c10a0e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.853+0000 7f5181d15640 1 --2- 172.21.15.50:0/3729023314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f517c105360 0x7f517c107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.853+0000 7f5181d15640 1 --2- 172.21.15.50:0/3729023314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f517c102a30 0x7f517c104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.853+0000 7f5181d15640 1 -- 172.21.15.50:0/3729023314 >> 172.21.15.50:0/3729023314 conn(0x7f517c0fc440 msgr2=0x7f517c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.853+0000 7f5181d15640 1 -- 172.21.15.50:0/3729023314 shutdown_connections 2024-04-23T18:15:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.853+0000 7f5181d15640 1 -- 172.21.15.50:0/3729023314 wait complete. 2024-04-23T18:15:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.853+0000 7f5181d15640 1 Processor -- start 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f5181d15640 1 -- start start 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f5181d15640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f517c102a30 0x7f517c198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f5181d15640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f517c105360 0x7f517c198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f5181d15640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f517c107c90 0x7f517c19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f5181d15640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f517c10daf0 con 0x7f517c102a30 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f5181d15640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f517c10d970 con 0x7f517c105360 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f5181d15640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f517c10dc70 con 0x7f517c107c90 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f517b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f517c102a30 0x7f517c198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f517b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f517c102a30 0x7f517c198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39400/0 (socket says 172.21.15.50:39400) 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f517b7fe640 1 -- 172.21.15.50:0/2821107880 learned_addr learned my addr 172.21.15.50:0/2821107880 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f517bfff640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f517c107c90 0x7f517c19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f517b7fe640 1 -- 172.21.15.50:0/2821107880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f517c107c90 msgr2=0x7f517c19f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f517b7fe640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f517c107c90 0x7f517c19f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f517b7fe640 1 -- 172.21.15.50:0/2821107880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f517c105360 msgr2=0x7f517c198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:56.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f517b7fe640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f517c105360 0x7f517c198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:56.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f517b7fe640 1 -- 172.21.15.50:0/2821107880 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517c19fd40 con 0x7f517c102a30 2024-04-23T18:15:56.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f517b7fe640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f517c102a30 0x7f517c198120 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f516c00e9f0 tx=0x7f516c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:56.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f5178ff9640 1 -- 172.21.15.50:0/2821107880 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c00cd80 con 0x7f517c102a30 2024-04-23T18:15:56.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f5178ff9640 1 -- 172.21.15.50:0/2821107880 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f516c0054d0 con 0x7f517c102a30 2024-04-23T18:15:56.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f5181d15640 1 -- 172.21.15.50:0/2821107880 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f517c19ffd0 con 0x7f517c102a30 2024-04-23T18:15:56.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f5178ff9640 1 -- 172.21.15.50:0/2821107880 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c010690 con 0x7f517c102a30 2024-04-23T18:15:56.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.854+0000 7f5181d15640 1 -- 172.21.15.50:0/2821107880 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f517c1a04f0 con 0x7f517c102a30 2024-04-23T18:15:56.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.855+0000 7f5181d15640 1 -- 172.21.15.50:0/2821107880 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f517c0fdf30 con 0x7f517c102a30 2024-04-23T18:15:56.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.856+0000 7f5178ff9640 1 -- 172.21.15.50:0/2821107880 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f516c004e90 con 0x7f517c102a30 2024-04-23T18:15:56.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.857+0000 7f5178ff9640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f51640a3c30 0x7f51640a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:56.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.857+0000 7f517affd640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f51640a3c30 0x7f51640a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:56.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.857+0000 7f5178ff9640 1 -- 172.21.15.50:0/2821107880 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f516c0d0490 con 0x7f517c102a30 2024-04-23T18:15:56.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.858+0000 7f517affd640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f51640a3c30 0x7f51640a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f517c199630 tx=0x7f5170005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:56.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:56.861+0000 7f5178ff9640 1 -- 172.21.15.50:0/2821107880 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f516c09d2b0 con 0x7f517c102a30 2024-04-23T18:15:57.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.089+0000 7f5181d15640 1 -- 172.21.15.50:0/2821107880 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f517c062c50 con 0x7f517c102a30 2024-04-23T18:15:57.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.089+0000 7f5178ff9640 1 -- 172.21.15.50:0/2821107880 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f516c09cbb0 con 0x7f517c102a30 2024-04-23T18:15:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:15:57.084 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-23T18:15:57.084 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 1 2024-04-23T18:15:57.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.091+0000 7f5181d15640 1 -- 172.21.15.50:0/2821107880 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f51640a3c30 msgr2=0x7f51640a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.091+0000 7f5181d15640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f51640a3c30 0x7f51640a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f517c199630 tx=0x7f5170005c70 comp rx=0 tx=0).stop 2024-04-23T18:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.091+0000 7f5181d15640 1 -- 172.21.15.50:0/2821107880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f517c102a30 msgr2=0x7f517c198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.091+0000 7f5181d15640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f517c102a30 0x7f517c198120 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f516c00e9f0 tx=0x7f516c00edf0 comp rx=0 tx=0).stop 2024-04-23T18:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.091+0000 7f5181d15640 1 -- 172.21.15.50:0/2821107880 shutdown_connections 2024-04-23T18:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.091+0000 7f5181d15640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f51640a3c30 0x7f51640a60f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.091+0000 7f5181d15640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f517c107c90 0x7f517c19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.091+0000 7f5181d15640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f517c105360 0x7f517c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.091+0000 7f5181d15640 1 --2- 172.21.15.50:0/2821107880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f517c102a30 0x7f517c198120 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.091+0000 7f5181d15640 1 -- 172.21.15.50:0/2821107880 >> 172.21.15.50:0/2821107880 conn(0x7f517c0fc440 msgr2=0x7f517c103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.091+0000 7f5181d15640 1 -- 172.21.15.50:0/2821107880 shutdown_connections 2024-04-23T18:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:57.091+0000 7f5181d15640 1 -- 172.21.15.50:0/2821107880 wait complete. 2024-04-23T18:15:57.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:57 smithi177 ceph-mon[32308]: Deploying daemon mds.l on smithi188 2024-04-23T18:15:57.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:57 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2572881110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:57.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:57 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1206778399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:57.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:57 smithi188 ceph-mon[32694]: Deploying daemon mds.l on smithi188 2024-04-23T18:15:57.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:57 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2572881110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:57.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:57 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1206778399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:57.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:57 smithi050 ceph-mon[29187]: Deploying daemon mds.l on smithi188 2024-04-23T18:15:57.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:57 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2572881110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:15:57.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:57 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1206778399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:15:58.023 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:15:57 smithi188 systemd[1]: Starting Ceph mds.l for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:15:58.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:58 smithi177 ceph-mon[32308]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:58.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:58 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2821107880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:58.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:58 smithi188 ceph-mon[32694]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:58.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:58 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2821107880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:58.300 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:15:58 smithi188 podman[63074]: 2024-04-23 18:15:57.975724964 +0000 UTC m=+0.017759599 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:15:58.300 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:15:58 smithi188 podman[63074]: 2024-04-23 18:15:58.079892105 +0000 UTC m=+0.121926723 container create 80cd05b82f1cf7aec72a0017579b1865e537811e78fc55567ecccf63b5f3a768 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-23T18:15:58.301 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:15:58 smithi188 podman[63074]: 2024-04-23 18:15:58.307555219 +0000 UTC m=+0.349589840 container init 80cd05b82f1cf7aec72a0017579b1865e537811e78fc55567ecccf63b5f3a768 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-l, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-23T18:15:58.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:58 smithi050 ceph-mon[29187]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:58.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:58 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2821107880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:15:58.580 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:15:58 smithi188 podman[63074]: 2024-04-23 18:15:58.310133907 +0000 UTC m=+0.352168526 container start 80cd05b82f1cf7aec72a0017579b1865e537811e78fc55567ecccf63b5f3a768 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-23T18:15:58.580 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:15:58 smithi188 ceph-mds[63093]: -- 172.21.15.188:0/3068003903 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c3fad75a40 con 0x55c3fad48c00 2024-04-23T18:15:58.580 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:15:58 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-l[63089]: starting mds.l at 2024-04-23T18:15:58.580 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:15:58 smithi188 bash[63074]: 80cd05b82f1cf7aec72a0017579b1865e537811e78fc55567ecccf63b5f3a768 2024-04-23T18:15:58.580 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:15:58 smithi188 systemd[1]: Started Ceph mds.l for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:15:59.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:15:59.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.217+0000 7fa6bc843640 1 -- 172.21.15.50:0/3763138622 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6b81031b0 msgr2=0x7fa6b8103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:59.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.217+0000 7fa6bc843640 1 --2- 172.21.15.50:0/3763138622 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6b81031b0 0x7fa6b8103630 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009900 tx=0x7fa6ac02a410 comp rx=0 tx=0).stop 2024-04-23T18:15:59.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.217+0000 7fa6bc843640 1 -- 172.21.15.50:0/3763138622 shutdown_connections 2024-04-23T18:15:59.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.217+0000 7fa6bc843640 1 --2- 172.21.15.50:0/3763138622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6b8103b70 0x7fa6b810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:59.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.217+0000 7fa6bc843640 1 --2- 172.21.15.50:0/3763138622 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6b81031b0 0x7fa6b8103630 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:59.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.217+0000 7fa6bc843640 1 --2- 172.21.15.50:0/3763138622 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6b8101fc0 0x7fa6b81023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:59.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.217+0000 7fa6bc843640 1 -- 172.21.15.50:0/3763138622 >> 172.21.15.50:0/3763138622 conn(0x7fa6b80fd550 msgr2=0x7fa6b80ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:59.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.217+0000 7fa6bc843640 1 -- 172.21.15.50:0/3763138622 shutdown_connections 2024-04-23T18:15:59.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.217+0000 7fa6bc843640 1 -- 172.21.15.50:0/3763138622 wait complete. 2024-04-23T18:15:59.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6bc843640 1 Processor -- start 2024-04-23T18:15:59.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6bc843640 1 -- start start 2024-04-23T18:15:59.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6bc843640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6b8101fc0 0x7fa6b8075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:59.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6bc843640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6b81031b0 0x7fa6b8074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:59.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6bc843640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6b8103b70 0x7fa6b8074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:59.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6bc843640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b810db00 con 0x7fa6b81031b0 2024-04-23T18:15:59.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6bc843640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b810d980 con 0x7fa6b8103b70 2024-04-23T18:15:59.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6bc843640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b810dc80 con 0x7fa6b8101fc0 2024-04-23T18:15:59.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6b5d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6b81031b0 0x7fa6b8074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6b5d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6b81031b0 0x7fa6b8074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39422/0 (socket says 172.21.15.50:39422) 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6b5d9b640 1 -- 172.21.15.50:0/1876083566 learned_addr learned my addr 172.21.15.50:0/1876083566 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6b659c640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6b8101fc0 0x7fa6b8075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6b6d9d640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6b8103b70 0x7fa6b8074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6b5d9b640 1 -- 172.21.15.50:0/1876083566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6b8101fc0 msgr2=0x7fa6b8075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6b5d9b640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6b8101fc0 0x7fa6b8075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6b5d9b640 1 -- 172.21.15.50:0/1876083566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6b8103b70 msgr2=0x7fa6b8074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6b5d9b640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6b8103b70 0x7fa6b8074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.218+0000 7fa6b5d9b640 1 -- 172.21.15.50:0/1876083566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b8075110 con 0x7fa6b81031b0 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.219+0000 7fa6b5d9b640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6b81031b0 0x7fa6b8074340 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac005940 tx=0x7fa6ac031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.219+0000 7fa6a77fe640 1 -- 172.21.15.50:0/1876083566 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac03f030 con 0x7fa6b81031b0 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.219+0000 7fa6bc843640 1 -- 172.21.15.50:0/1876083566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6b81a2280 con 0x7fa6b81031b0 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.219+0000 7fa6a77fe640 1 -- 172.21.15.50:0/1876083566 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6ac003aa0 con 0x7fa6b81031b0 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.219+0000 7fa6a77fe640 1 -- 172.21.15.50:0/1876083566 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac035070 con 0x7fa6b81031b0 2024-04-23T18:15:59.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.219+0000 7fa6bc843640 1 -- 172.21.15.50:0/1876083566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6b81a2820 con 0x7fa6b81031b0 2024-04-23T18:15:59.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.220+0000 7fa6bc843640 1 -- 172.21.15.50:0/1876083566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6b8109350 con 0x7fa6b81031b0 2024-04-23T18:15:59.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.221+0000 7fa6a77fe640 1 -- 172.21.15.50:0/1876083566 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa6ac042030 con 0x7fa6b81031b0 2024-04-23T18:15:59.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.222+0000 7fa6a77fe640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa6a00a3bb0 0x7fa6a00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:15:59.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.222+0000 7fa6b659c640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa6a00a3bb0 0x7fa6a00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:15:59.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.222+0000 7fa6a77fe640 1 -- 172.21.15.50:0/1876083566 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fa6ac0f04f0 con 0x7fa6b81031b0 2024-04-23T18:15:59.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.222+0000 7fa6b659c640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa6a00a3bb0 0x7fa6a00a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8103010 tx=0x7fa6a800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:15:59.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.226+0000 7fa6a77fe640 1 -- 172.21.15.50:0/1876083566 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa6ac0bcd20 con 0x7fa6b81031b0 2024-04-23T18:15:59.383 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:59 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.188:6832/3630824399,v1:172.21.15.188:6833/3630824399] up:boot 2024-04-23T18:15:59.383 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:59 smithi177 ceph-mon[32308]: fsmap 1 up:standby 2024-04-23T18:15:59.383 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:59 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:15:59.383 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:59 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:59.383 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:59 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:59.383 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:59 smithi177 ceph-mon[32308]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:59.383 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:59 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:59.384 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:59 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:15:59.384 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:59 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:15:59.384 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:59 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:59.384 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:15:59 smithi177 ceph-mon[32308]: Deploying daemon mds.k on smithi177 2024-04-23T18:15:59.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:59 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.188:6832/3630824399,v1:172.21.15.188:6833/3630824399] up:boot 2024-04-23T18:15:59.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:59 smithi050 ceph-mon[29187]: fsmap 1 up:standby 2024-04-23T18:15:59.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:59 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:15:59.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:59 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:59.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:59 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:59.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:59 smithi050 ceph-mon[29187]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:59.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:59 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:59.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:59 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:15:59.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:59 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:15:59.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:59 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:59.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:15:59 smithi050 ceph-mon[29187]: Deploying daemon mds.k on smithi177 2024-04-23T18:15:59.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.453+0000 7fa6bc843640 1 -- 172.21.15.50:0/1876083566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6b802d7c0 con 0x7fa6b81031b0 2024-04-23T18:15:59.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.453+0000 7fa6a77fe640 1 -- 172.21.15.50:0/1876083566 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fa6ac0c1020 con 0x7fa6b81031b0 2024-04-23T18:15:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:15:59.448 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T18:15:59.448 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 2 2024-04-23T18:15:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.455+0000 7fa6bc843640 1 -- 172.21.15.50:0/1876083566 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa6a00a3bb0 msgr2=0x7fa6a00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.455+0000 7fa6bc843640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa6a00a3bb0 0x7fa6a00a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8103010 tx=0x7fa6a800b390 comp rx=0 tx=0).stop 2024-04-23T18:15:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.455+0000 7fa6bc843640 1 -- 172.21.15.50:0/1876083566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6b81031b0 msgr2=0x7fa6b8074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:15:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.455+0000 7fa6bc843640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6b81031b0 0x7fa6b8074340 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac005940 tx=0x7fa6ac031a50 comp rx=0 tx=0).stop 2024-04-23T18:15:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.455+0000 7fa6bc843640 1 -- 172.21.15.50:0/1876083566 shutdown_connections 2024-04-23T18:15:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.455+0000 7fa6bc843640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa6a00a3bb0 0x7fa6a00a6070 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.455+0000 7fa6bc843640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6b8103b70 0x7fa6b8074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.455+0000 7fa6bc843640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6b81031b0 0x7fa6b8074340 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.455+0000 7fa6bc843640 1 --2- 172.21.15.50:0/1876083566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6b8101fc0 0x7fa6b8075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:15:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.455+0000 7fa6bc843640 1 -- 172.21.15.50:0/1876083566 >> 172.21.15.50:0/1876083566 conn(0x7fa6b80fd550 msgr2=0x7fa6b8106520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:15:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.455+0000 7fa6bc843640 1 -- 172.21.15.50:0/1876083566 shutdown_connections 2024-04-23T18:15:59.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:15:59.455+0000 7fa6bc843640 1 -- 172.21.15.50:0/1876083566 wait complete. 2024-04-23T18:15:59.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:59 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.188:6832/3630824399,v1:172.21.15.188:6833/3630824399] up:boot 2024-04-23T18:15:59.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:59 smithi188 ceph-mon[32694]: fsmap 1 up:standby 2024-04-23T18:15:59.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:59 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:15:59.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:59 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:59.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:59 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:59.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:59 smithi188 ceph-mon[32694]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:15:59.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:59 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:15:59.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:59 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:15:59.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:59 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:15:59.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:59 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:15:59.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:15:59 smithi188 ceph-mon[32694]: Deploying daemon mds.k on smithi177 2024-04-23T18:16:00.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:00 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1876083566' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:00.767 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:00 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1876083566' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:00.767 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:16:00 smithi177 systemd[1]: Starting Ceph mds.k for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:16:00.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:00 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1876083566' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:01.038 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:16:00 smithi177 podman[62054]: 2024-04-23 18:16:00.676260674 +0000 UTC m=+0.018058612 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:16:01.038 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:16:00 smithi177 podman[62054]: 2024-04-23 18:16:00.823612851 +0000 UTC m=+0.165427536 container create 6e3b8564481eb3abc27dd45376f912963cb174ca3e0294181f3c69f51bcd1c69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:16:01.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:01.538 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:16:01 smithi177 podman[62054]: 2024-04-23 18:16:01.131209153 +0000 UTC m=+0.473007075 container init 6e3b8564481eb3abc27dd45376f912963cb174ca3e0294181f3c69f51bcd1c69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-23T18:16:01.538 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:16:01 smithi177 podman[62054]: 2024-04-23 18:16:01.133752504 +0000 UTC m=+0.475550434 container start 6e3b8564481eb3abc27dd45376f912963cb174ca3e0294181f3c69f51bcd1c69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-23T18:16:01.538 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:16:01 smithi177 ceph-mds[62073]: -- 172.21.15.177:0/3290200251 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56454e3f5a40 con 0x56454e3c9000 2024-04-23T18:16:01.538 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:16:01 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-k[62069]: starting mds.k at 2024-04-23T18:16:01.538 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:16:01 smithi177 bash[62054]: 6e3b8564481eb3abc27dd45376f912963cb174ca3e0294181f3c69f51bcd1c69 2024-04-23T18:16:01.539 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:16:01 smithi177 systemd[1]: Started Ceph mds.k for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:16:01.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 -- 172.21.15.50:0/4185191667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f321c1011e0 msgr2=0x7f321c101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:01.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 --2- 172.21.15.50:0/4185191667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f321c1011e0 0x7f321c101640 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3208009900 tx=0x7f320802a480 comp rx=0 tx=0).stop 2024-04-23T18:16:01.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 -- 172.21.15.50:0/4185191667 shutdown_connections 2024-04-23T18:16:01.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 --2- 172.21.15.50:0/4185191667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f321c101b80 0x7f321c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:01.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 --2- 172.21.15.50:0/4185191667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f321c1011e0 0x7f321c101640 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 --2- 172.21.15.50:0/4185191667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f321c100880 0x7f321c100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 -- 172.21.15.50:0/4185191667 >> 172.21.15.50:0/4185191667 conn(0x7f321c0fc400 msgr2=0x7f321c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 -- 172.21.15.50:0/4185191667 shutdown_connections 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 -- 172.21.15.50:0/4185191667 wait complete. 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 Processor -- start 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 -- start start 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f321c100880 0x7f321c19a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.570+0000 7f322309a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f321c1011e0 0x7f321c19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f322309a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f321c101b80 0x7f321c1a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f322309a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f321c10fcd0 con 0x7f321c101b80 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f322309a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f321c10fb50 con 0x7f321c100880 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f322309a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f321c10fe50 con 0x7f321c1011e0 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3221637640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f321c101b80 0x7f321c1a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:01.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3221637640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f321c101b80 0x7f321c1a19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39448/0 (socket says 172.21.15.50:39448) 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3221637640 1 -- 172.21.15.50:0/1984788881 learned_addr learned my addr 172.21.15.50:0/1984788881 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f321bfff640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f321c1011e0 0x7f321c19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3220e36640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f321c100880 0x7f321c19a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3221637640 1 -- 172.21.15.50:0/1984788881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f321c1011e0 msgr2=0x7f321c19a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3221637640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f321c1011e0 0x7f321c19a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3221637640 1 -- 172.21.15.50:0/1984788881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f321c100880 msgr2=0x7f321c19a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3221637640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f321c100880 0x7f321c19a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3221637640 1 -- 172.21.15.50:0/1984788881 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321c1a2010 con 0x7f321c101b80 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3220e36640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f321c100880 0x7f321c19a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3221637640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f321c101b80 0x7f321c1a19a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3214009fd0 tx=0x7f321400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3219ffb640 1 -- 172.21.15.50:0/1984788881 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3214017070 con 0x7f321c101b80 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3219ffb640 1 -- 172.21.15.50:0/1984788881 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3214003e20 con 0x7f321c101b80 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f3219ffb640 1 -- 172.21.15.50:0/1984788881 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3214010460 con 0x7f321c101b80 2024-04-23T18:16:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.571+0000 7f322309a640 1 -- 172.21.15.50:0/1984788881 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f321c1a2300 con 0x7f321c101b80 2024-04-23T18:16:01.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.572+0000 7f322309a640 1 -- 172.21.15.50:0/1984788881 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f321c1a27d0 con 0x7f321c101b80 2024-04-23T18:16:01.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.573+0000 7f322309a640 1 -- 172.21.15.50:0/1984788881 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f321c10b860 con 0x7f321c101b80 2024-04-23T18:16:01.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.573+0000 7f3219ffb640 1 -- 172.21.15.50:0/1984788881 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3214003970 con 0x7f321c101b80 2024-04-23T18:16:01.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.574+0000 7f3219ffb640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f320c0a3bb0 0x7f320c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:01.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.574+0000 7f3220e36640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f320c0a3bb0 0x7f320c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:01.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.574+0000 7f3219ffb640 1 -- 172.21.15.50:0/1984788881 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f32140cfdc0 con 0x7f321c101b80 2024-04-23T18:16:01.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.575+0000 7f3220e36640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f320c0a3bb0 0x7f320c0a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f32100058c0 tx=0x7f3210005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:01.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.579+0000 7f3219ffb640 1 -- 172.21.15.50:0/1984788881 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f321409cba0 con 0x7f321c101b80 2024-04-23T18:16:01.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.811+0000 7f322309a640 1 -- 172.21.15.50:0/1984788881 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f321c062b50 con 0x7f321c101b80 2024-04-23T18:16:01.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.811+0000 7f3219ffb640 1 -- 172.21.15.50:0/1984788881 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f321409c4a0 con 0x7f321c101b80 2024-04-23T18:16:01.805 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:01.806 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T18:16:01.806 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 3 2024-04-23T18:16:01.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.814+0000 7f322309a640 1 -- 172.21.15.50:0/1984788881 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f320c0a3bb0 msgr2=0x7f320c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:01.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.814+0000 7f322309a640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f320c0a3bb0 0x7f320c0a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f32100058c0 tx=0x7f3210005850 comp rx=0 tx=0).stop 2024-04-23T18:16:01.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.814+0000 7f322309a640 1 -- 172.21.15.50:0/1984788881 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f321c101b80 msgr2=0x7f321c1a19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:01.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.814+0000 7f322309a640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f321c101b80 0x7f321c1a19a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3214009fd0 tx=0x7f321400b260 comp rx=0 tx=0).stop 2024-04-23T18:16:01.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.814+0000 7f322309a640 1 -- 172.21.15.50:0/1984788881 shutdown_connections 2024-04-23T18:16:01.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.814+0000 7f322309a640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f320c0a3bb0 0x7f320c0a6070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:01.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.814+0000 7f322309a640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f321c101b80 0x7f321c1a19a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:01.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.814+0000 7f322309a640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f321c1011e0 0x7f321c19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:01.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.814+0000 7f322309a640 1 --2- 172.21.15.50:0/1984788881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f321c100880 0x7f321c19a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:01.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.814+0000 7f322309a640 1 -- 172.21.15.50:0/1984788881 >> 172.21.15.50:0/1984788881 conn(0x7f321c0fc400 msgr2=0x7f321c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:01.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.814+0000 7f322309a640 1 -- 172.21.15.50:0/1984788881 shutdown_connections 2024-04-23T18:16:01.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:01.814+0000 7f322309a640 1 -- 172.21.15.50:0/1984788881 wait complete. 2024-04-23T18:16:02.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:02 smithi050 ceph-mon[29187]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:02.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:02.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:02.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:02.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:02.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:02.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:02.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:02 smithi050 ceph-mon[29187]: Deploying daemon mds.j on smithi050 2024-04-23T18:16:02.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:02 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6832/878222815,v1:172.21.15.177:6833/878222815] up:boot 2024-04-23T18:16:02.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:02 smithi050 ceph-mon[29187]: fsmap 2 up:standby 2024-04-23T18:16:02.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:02 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:16:02.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:02 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1984788881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:02.533 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:02 smithi177 ceph-mon[32308]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:02.533 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:02 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:02.533 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:02 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:02.533 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:02 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:02.533 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:02 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:02.533 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:02 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:02.533 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:02 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:02.534 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:02 smithi177 ceph-mon[32308]: Deploying daemon mds.j on smithi050 2024-04-23T18:16:02.534 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:02 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6832/878222815,v1:172.21.15.177:6833/878222815] up:boot 2024-04-23T18:16:02.534 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:02 smithi177 ceph-mon[32308]: fsmap 2 up:standby 2024-04-23T18:16:02.534 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:02 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:16:02.534 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:02 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1984788881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:02.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:02 smithi188 ceph-mon[32694]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:02.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:02 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:02.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:02 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:02.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:02 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:02.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:02 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:02.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:02 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:02.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:02 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:02.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:02 smithi188 ceph-mon[32694]: Deploying daemon mds.j on smithi050 2024-04-23T18:16:02.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:02 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6832/878222815,v1:172.21.15.177:6833/878222815] up:boot 2024-04-23T18:16:02.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:02 smithi188 ceph-mon[32694]: fsmap 2 up:standby 2024-04-23T18:16:02.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:02 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:16:02.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:02 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1984788881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:03.633 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:16:03 smithi050 systemd[1]: Starting Ceph mds.j for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:16:03.633 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:16:03 smithi050 podman[67400]: 2024-04-23 18:16:03.542245227 +0000 UTC m=+0.018693191 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:16:03.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:03.909 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:16:03 smithi050 podman[67400]: 2024-04-23 18:16:03.663728424 +0000 UTC m=+0.140176395 container create cea48831c1140f23d1e57b2595929366498d7464b4f04a5837f35c5f544fdb96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-j, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:16:03.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.936+0000 7fbb33835640 1 -- 172.21.15.50:0/1184363713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb2c101fe0 msgr2=0x7fbb2c102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.936+0000 7fbb33835640 1 --2- 172.21.15.50:0/1184363713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb2c101fe0 0x7fbb2c102460 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c009900 tx=0x7fbb1c02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:03.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.937+0000 7fbb33835640 1 -- 172.21.15.50:0/1184363713 shutdown_connections 2024-04-23T18:16:03.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.937+0000 7fbb33835640 1 --2- 172.21.15.50:0/1184363713 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb2c1029a0 0x7fbb2c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.937+0000 7fbb33835640 1 --2- 172.21.15.50:0/1184363713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb2c101fe0 0x7fbb2c102460 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.937+0000 7fbb33835640 1 --2- 172.21.15.50:0/1184363713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb2c100e90 0x7fbb2c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.937+0000 7fbb33835640 1 -- 172.21.15.50:0/1184363713 >> 172.21.15.50:0/1184363713 conn(0x7fbb2c0fc400 msgr2=0x7fbb2c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.937+0000 7fbb33835640 1 -- 172.21.15.50:0/1184363713 shutdown_connections 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.937+0000 7fbb33835640 1 -- 172.21.15.50:0/1184363713 wait complete. 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.937+0000 7fbb33835640 1 Processor -- start 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.937+0000 7fbb33835640 1 -- start start 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.937+0000 7fbb33835640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb2c100e90 0x7fbb2c1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.937+0000 7fbb33835640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb2c101fe0 0x7fbb2c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb33835640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb2c1029a0 0x7fbb2c19f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb33835640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c10c9d0 con 0x7fbb2c1029a0 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb33835640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c10c850 con 0x7fbb2c101fe0 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb33835640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c10cb50 con 0x7fbb2c100e90 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb31dd2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb2c1029a0 0x7fbb2c19f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb31dd2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb2c1029a0 0x7fbb2c19f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51106/0 (socket says 172.21.15.50:51106) 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb31dd2640 1 -- 172.21.15.50:0/2436538545 learned_addr learned my addr 172.21.15.50:0/2436538545 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb315d1640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb2c100e90 0x7fbb2c1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb30dd0640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb2c101fe0 0x7fbb2c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb31dd2640 1 -- 172.21.15.50:0/2436538545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb2c100e90 msgr2=0x7fbb2c1981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb31dd2640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb2c100e90 0x7fbb2c1981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb31dd2640 1 -- 172.21.15.50:0/2436538545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb2c101fe0 msgr2=0x7fbb2c1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb31dd2640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb2c101fe0 0x7fbb2c1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb31dd2640 1 -- 172.21.15.50:0/2436538545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2c19fdc0 con 0x7fbb2c1029a0 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb30dd0640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb2c101fe0 0x7fbb2c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb31dd2640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb2c1029a0 0x7fbb2c19f750 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbb2400cce0 tx=0x7fbb2400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb227fc640 1 -- 172.21.15.50:0/2436538545 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2400f450 con 0x7fbb2c1029a0 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb33835640 1 -- 172.21.15.50:0/2436538545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb2c1a00b0 con 0x7fbb2c1029a0 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb227fc640 1 -- 172.21.15.50:0/2436538545 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb2400faa0 con 0x7fbb2c1029a0 2024-04-23T18:16:03.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb227fc640 1 -- 172.21.15.50:0/2436538545 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24018b70 con 0x7fbb2c1029a0 2024-04-23T18:16:03.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.938+0000 7fbb33835640 1 -- 172.21.15.50:0/2436538545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb2c1a0580 con 0x7fbb2c1029a0 2024-04-23T18:16:03.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.939+0000 7fbb33835640 1 -- 172.21.15.50:0/2436538545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb2c1a0840 con 0x7fbb2c1029a0 2024-04-23T18:16:03.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.941+0000 7fbb227fc640 1 -- 172.21.15.50:0/2436538545 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbb24018d10 con 0x7fbb2c1029a0 2024-04-23T18:16:03.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.942+0000 7fbb227fc640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fbb180a3bb0 0x7fbb180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:03.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.942+0000 7fbb315d1640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fbb180a3bb0 0x7fbb180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:03.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.942+0000 7fbb227fc640 1 -- 172.21.15.50:0/2436538545 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fbb240d0450 con 0x7fbb2c1029a0 2024-04-23T18:16:03.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.943+0000 7fbb315d1640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fbb180a3bb0 0x7fbb180a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800bbe0 tx=0x7fbb2800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:03.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:03.945+0000 7fbb227fc640 1 -- 172.21.15.50:0/2436538545 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbb24014030 con 0x7fbb2c1029a0 2024-04-23T18:16:04.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.193+0000 7fbb33835640 1 -- 172.21.15.50:0/2436538545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb2c1a0a10 con 0x7fbb2c1029a0 2024-04-23T18:16:04.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.193+0000 7fbb227fc640 1 -- 172.21.15.50:0/2436538545 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fbb2409d360 con 0x7fbb2c1029a0 2024-04-23T18:16:04.188 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:04.188 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T18:16:04.189 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 3 2024-04-23T18:16:04.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.196+0000 7fbb33835640 1 -- 172.21.15.50:0/2436538545 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fbb180a3bb0 msgr2=0x7fbb180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.196+0000 7fbb33835640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fbb180a3bb0 0x7fbb180a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800bbe0 tx=0x7fbb2800b390 comp rx=0 tx=0).stop 2024-04-23T18:16:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.196+0000 7fbb33835640 1 -- 172.21.15.50:0/2436538545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb2c1029a0 msgr2=0x7fbb2c19f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.196+0000 7fbb33835640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb2c1029a0 0x7fbb2c19f750 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbb2400cce0 tx=0x7fbb2400a620 comp rx=0 tx=0).stop 2024-04-23T18:16:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.196+0000 7fbb33835640 1 -- 172.21.15.50:0/2436538545 shutdown_connections 2024-04-23T18:16:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.196+0000 7fbb33835640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fbb180a3bb0 0x7fbb180a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.196+0000 7fbb33835640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb2c1029a0 0x7fbb2c19f750 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.196+0000 7fbb33835640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb2c101fe0 0x7fbb2c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.196+0000 7fbb33835640 1 --2- 172.21.15.50:0/2436538545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb2c100e90 0x7fbb2c1981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.196+0000 7fbb33835640 1 -- 172.21.15.50:0/2436538545 >> 172.21.15.50:0/2436538545 conn(0x7fbb2c0fc400 msgr2=0x7fbb2c0fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.196+0000 7fbb33835640 1 -- 172.21.15.50:0/2436538545 shutdown_connections 2024-04-23T18:16:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:04.196+0000 7fbb33835640 1 -- 172.21.15.50:0/2436538545 wait complete. 2024-04-23T18:16:04.194 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:16:03 smithi050 podman[67400]: 2024-04-23 18:16:03.916121083 +0000 UTC m=+0.392569009 container init cea48831c1140f23d1e57b2595929366498d7464b4f04a5837f35c5f544fdb96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-23T18:16:04.195 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:16:03 smithi050 podman[67400]: 2024-04-23 18:16:03.920387996 +0000 UTC m=+0.396835925 container start cea48831c1140f23d1e57b2595929366498d7464b4f04a5837f35c5f544fdb96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:16:04.195 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:16:03 smithi050 ceph-mds[67439]: -- 172.21.15.50:0/4279096970 <== mon.0 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f19a785a40 con 0x55f19a758c00 2024-04-23T18:16:04.195 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:16:03 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-j[67415]: starting mds.j at 2024-04-23T18:16:04.195 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:16:04 smithi050 bash[67400]: cea48831c1140f23d1e57b2595929366498d7464b4f04a5837f35c5f544fdb96 2024-04-23T18:16:04.195 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:16:04 smithi050 systemd[1]: Started Ceph mds.j for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:16:04.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:04 smithi177 ceph-mon[32308]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:04.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:04.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:04.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:04.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:04.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:04.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:04 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:04.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:04 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2436538545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:04.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:04 smithi188 ceph-mon[32694]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:04.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:04.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:04.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:04.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:04.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:04.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:04 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:04.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:04 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2436538545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:04.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:04 smithi050 ceph-mon[29187]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:04.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:04.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:04.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:04.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:04.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:04.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:04 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:04.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:04 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2436538545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:05.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:05 smithi188 ceph-mon[32694]: Deploying daemon mds.d on smithi050 2024-04-23T18:16:05.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:05 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.50:6834/878281924,v1:172.21.15.50:6835/878281924] up:boot 2024-04-23T18:16:05.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:05 smithi188 ceph-mon[32694]: fsmap 3 up:standby 2024-04-23T18:16:05.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:05 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:16:05.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:05 smithi050 ceph-mon[29187]: Deploying daemon mds.d on smithi050 2024-04-23T18:16:05.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:05 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.50:6834/878281924,v1:172.21.15.50:6835/878281924] up:boot 2024-04-23T18:16:05.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:05 smithi050 ceph-mon[29187]: fsmap 3 up:standby 2024-04-23T18:16:05.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:05 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:16:05.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:05 smithi177 ceph-mon[32308]: Deploying daemon mds.d on smithi050 2024-04-23T18:16:05.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:05 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.50:6834/878281924,v1:172.21.15.50:6835/878281924] up:boot 2024-04-23T18:16:05.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:05 smithi177 ceph-mon[32308]: fsmap 3 up:standby 2024-04-23T18:16:05.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:05 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:16:06.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:06.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.335+0000 7fd2aba91640 1 -- 172.21.15.50:0/2240414533 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2a41053e0 msgr2=0x7fd2a41077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:06.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.335+0000 7fd2aba91640 1 --2- 172.21.15.50:0/2240414533 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2a41053e0 0x7fd2a41077d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd294009900 tx=0x7fd29402a410 comp rx=0 tx=0).stop 2024-04-23T18:16:06.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.335+0000 7fd2aba91640 1 -- 172.21.15.50:0/2240414533 shutdown_connections 2024-04-23T18:16:06.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.335+0000 7fd2aba91640 1 --2- 172.21.15.50:0/2240414533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4107d10 0x7fd2a410a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:06.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.335+0000 7fd2aba91640 1 --2- 172.21.15.50:0/2240414533 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2a41053e0 0x7fd2a41077d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:06.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.335+0000 7fd2aba91640 1 --2- 172.21.15.50:0/2240414533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2a4102ab0 0x7fd2a4104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:06.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.335+0000 7fd2aba91640 1 -- 172.21.15.50:0/2240414533 >> 172.21.15.50:0/2240414533 conn(0x7fd2a40fc440 msgr2=0x7fd2a40fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:06.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.336+0000 7fd2aba91640 1 -- 172.21.15.50:0/2240414533 shutdown_connections 2024-04-23T18:16:06.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.336+0000 7fd2aba91640 1 -- 172.21.15.50:0/2240414533 wait complete. 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2aba91640 1 Processor -- start 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2aba91640 1 -- start start 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2aba91640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4102ab0 0x7fd2a4198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2aba91640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2a41053e0 0x7fd2a41987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2aba91640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2a4107d10 0x7fd2a419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2aba91640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a410db10 con 0x7fd2a41053e0 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2aba91640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a410d990 con 0x7fd2a4107d10 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2aba91640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a410dc90 con 0x7fd2a4102ab0 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2a902c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2a41053e0 0x7fd2a41987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2a902c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2a41053e0 0x7fd2a41987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51152/0 (socket says 172.21.15.50:51152) 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2a902c640 1 -- 172.21.15.50:0/3273340553 learned_addr learned my addr 172.21.15.50:0/3273340553 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2a982d640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4102ab0 0x7fd2a4198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.337+0000 7fd2aa02e640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2a4107d10 0x7fd2a419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.338+0000 7fd2a902c640 1 -- 172.21.15.50:0/3273340553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4102ab0 msgr2=0x7fd2a4198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:06.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.338+0000 7fd2a902c640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4102ab0 0x7fd2a4198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:06.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.338+0000 7fd2a902c640 1 -- 172.21.15.50:0/3273340553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2a4107d10 msgr2=0x7fd2a419f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:06.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.338+0000 7fd2a902c640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2a4107d10 0x7fd2a419f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:06.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.338+0000 7fd2a902c640 1 -- 172.21.15.50:0/3273340553 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a4100920 con 0x7fd2a41053e0 2024-04-23T18:16:06.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.338+0000 7fd2a902c640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2a41053e0 0x7fd2a41987b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd294006010 tx=0x7fd2940042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:06.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.338+0000 7fd29affd640 1 -- 172.21.15.50:0/3273340553 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29403f030 con 0x7fd2a41053e0 2024-04-23T18:16:06.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.338+0000 7fd29affd640 1 -- 172.21.15.50:0/3273340553 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd294031910 con 0x7fd2a41053e0 2024-04-23T18:16:06.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.338+0000 7fd29affd640 1 -- 172.21.15.50:0/3273340553 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294035070 con 0x7fd2a41053e0 2024-04-23T18:16:06.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.338+0000 7fd2aba91640 1 -- 172.21.15.50:0/3273340553 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a4100bb0 con 0x7fd2a41053e0 2024-04-23T18:16:06.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.338+0000 7fd2aba91640 1 -- 172.21.15.50:0/3273340553 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a4101050 con 0x7fd2a41053e0 2024-04-23T18:16:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.340+0000 7fd2aba91640 1 -- 172.21.15.50:0/3273340553 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a402d5c0 con 0x7fd2a41053e0 2024-04-23T18:16:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.341+0000 7fd29affd640 1 -- 172.21.15.50:0/3273340553 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd294042030 con 0x7fd2a41053e0 2024-04-23T18:16:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.342+0000 7fd29affd640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd2900a3c30 0x7fd2900a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:06.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.342+0000 7fd2a982d640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd2900a3c30 0x7fd2900a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:06.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.342+0000 7fd29affd640 1 -- 172.21.15.50:0/3273340553 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fd2940f07d0 con 0x7fd2a41053e0 2024-04-23T18:16:06.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.342+0000 7fd2a982d640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd2900a3c30 0x7fd2900a60f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009bd0 tx=0x7fd2a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:06.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.346+0000 7fd29affd640 1 -- 172.21.15.50:0/3273340553 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd2940bcf60 con 0x7fd2a41053e0 2024-04-23T18:16:06.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:06 smithi050 ceph-mon[29187]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:06.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:06 smithi177 ceph-mon[32308]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:06.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:06 smithi188 ceph-mon[32694]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:06.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.588+0000 7fd2aba91640 1 -- 172.21.15.50:0/3273340553 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2a402d7e0 con 0x7fd2a41053e0 2024-04-23T18:16:06.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.588+0000 7fd29affd640 1 -- 172.21.15.50:0/3273340553 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7fd2940c1260 con 0x7fd2a41053e0 2024-04-23T18:16:06.582 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:06.582 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T18:16:06.583 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 4 2024-04-23T18:16:06.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.590+0000 7fd2aba91640 1 -- 172.21.15.50:0/3273340553 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd2900a3c30 msgr2=0x7fd2900a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:06.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.590+0000 7fd2aba91640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd2900a3c30 0x7fd2900a60f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009bd0 tx=0x7fd2a0005f90 comp rx=0 tx=0).stop 2024-04-23T18:16:06.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.590+0000 7fd2aba91640 1 -- 172.21.15.50:0/3273340553 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2a41053e0 msgr2=0x7fd2a41987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:06.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.590+0000 7fd2aba91640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2a41053e0 0x7fd2a41987b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd294006010 tx=0x7fd2940042f0 comp rx=0 tx=0).stop 2024-04-23T18:16:06.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.590+0000 7fd2aba91640 1 -- 172.21.15.50:0/3273340553 shutdown_connections 2024-04-23T18:16:06.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.590+0000 7fd2aba91640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd2900a3c30 0x7fd2900a60f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:06.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.590+0000 7fd2aba91640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2a4107d10 0x7fd2a419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:06.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.590+0000 7fd2aba91640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2a41053e0 0x7fd2a41987b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:06.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.590+0000 7fd2aba91640 1 --2- 172.21.15.50:0/3273340553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a4102ab0 0x7fd2a4198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:06.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.591+0000 7fd2aba91640 1 -- 172.21.15.50:0/3273340553 >> 172.21.15.50:0/3273340553 conn(0x7fd2a40fc440 msgr2=0x7fd2a4104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:06.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.591+0000 7fd2aba91640 1 -- 172.21.15.50:0/3273340553 shutdown_connections 2024-04-23T18:16:06.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:06.591+0000 7fd2aba91640 1 -- 172.21.15.50:0/3273340553 wait complete. 2024-04-23T18:16:07.195 INFO:journalctl@ceph.mds.d.smithi050.stdout:Apr 23 18:16:06 smithi050 systemd[1]: Started Ceph mds.d for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:16:07.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:07 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3273340553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:07.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:07 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:07.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:07 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:07.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:07 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:07.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:07 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:07.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:07 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:07.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:07 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:07.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:07 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3273340553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:07.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:07 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:07.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:07 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:07.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:07 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:07.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:07 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:07.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:07 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:07.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:07 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:07.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:07 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3273340553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:07.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:07 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:07.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:07 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:07.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:07 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:07.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:07 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:07.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:07 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:07.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:07 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:08 smithi177 ceph-mon[32308]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:08 smithi177 ceph-mon[32308]: Deploying daemon mds.e on smithi177 2024-04-23T18:16:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:08 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.50:6836/1480144576,v1:172.21.15.50:6837/1480144576] up:boot 2024-04-23T18:16:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:08 smithi177 ceph-mon[32308]: fsmap 4 up:standby 2024-04-23T18:16:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:08 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:16:08.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:08 smithi188 ceph-mon[32694]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:08.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:08 smithi188 ceph-mon[32694]: Deploying daemon mds.e on smithi177 2024-04-23T18:16:08.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:08 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.50:6836/1480144576,v1:172.21.15.50:6837/1480144576] up:boot 2024-04-23T18:16:08.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:08 smithi188 ceph-mon[32694]: fsmap 4 up:standby 2024-04-23T18:16:08.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:08 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:16:08.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:08 smithi050 ceph-mon[29187]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:08 smithi050 ceph-mon[29187]: Deploying daemon mds.e on smithi177 2024-04-23T18:16:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:08 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.50:6836/1480144576,v1:172.21.15.50:6837/1480144576] up:boot 2024-04-23T18:16:08.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:08 smithi050 ceph-mon[29187]: fsmap 4 up:standby 2024-04-23T18:16:08.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:08 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:16:08.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.721+0000 7fb33247c640 1 -- 172.21.15.50:0/4027730275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb32c103b70 msgr2=0x7fb32c10a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:08.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.721+0000 7fb33247c640 1 --2- 172.21.15.50:0/4027730275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb32c103b70 0x7fb32c10a400 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb324009ca0 tx=0x7fb32402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.721+0000 7fb33247c640 1 -- 172.21.15.50:0/4027730275 shutdown_connections 2024-04-23T18:16:08.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.721+0000 7fb33247c640 1 --2- 172.21.15.50:0/4027730275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb32c103b70 0x7fb32c10a400 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.721+0000 7fb33247c640 1 --2- 172.21.15.50:0/4027730275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb32c1031b0 0x7fb32c103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.721+0000 7fb33247c640 1 --2- 172.21.15.50:0/4027730275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb32c101fc0 0x7fb32c1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.721+0000 7fb33247c640 1 -- 172.21.15.50:0/4027730275 >> 172.21.15.50:0/4027730275 conn(0x7fb32c0fd550 msgr2=0x7fb32c0ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:08.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.721+0000 7fb33247c640 1 -- 172.21.15.50:0/4027730275 shutdown_connections 2024-04-23T18:16:08.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.721+0000 7fb33247c640 1 -- 172.21.15.50:0/4027730275 wait complete. 2024-04-23T18:16:08.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb33247c640 1 Processor -- start 2024-04-23T18:16:08.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb33247c640 1 -- start start 2024-04-23T18:16:08.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb33247c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb32c101fc0 0x7fb32c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:08.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb33247c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb32c1031b0 0x7fb32c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:08.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb33247c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb32c103b70 0x7fb32c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb33247c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c10db00 con 0x7fb32c103b70 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb33247c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c10d980 con 0x7fb32c101fc0 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb33247c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c10dc80 con 0x7fb32c1031b0 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb330a19640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb32c103b70 0x7fb32c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb330a19640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb32c103b70 0x7fb32c19f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51196/0 (socket says 172.21.15.50:51196) 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb330a19640 1 -- 172.21.15.50:0/2386331519 learned_addr learned my addr 172.21.15.50:0/2386331519 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb32bfff640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb32c101fc0 0x7fb32c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.722+0000 7fb32b7fe640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb32c1031b0 0x7fb32c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.723+0000 7fb330a19640 1 -- 172.21.15.50:0/2386331519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb32c1031b0 msgr2=0x7fb32c1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.723+0000 7fb330a19640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb32c1031b0 0x7fb32c1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.723+0000 7fb330a19640 1 -- 172.21.15.50:0/2386331519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb32c101fc0 msgr2=0x7fb32c198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.723+0000 7fb330a19640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb32c101fc0 0x7fb32c198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.723+0000 7fb330a19640 1 -- 172.21.15.50:0/2386331519 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32c19fd90 con 0x7fb32c103b70 2024-04-23T18:16:08.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.723+0000 7fb330a19640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb32c103b70 0x7fb32c19f720 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb324031a20 tx=0x7fb324003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:08.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.723+0000 7fb3297fa640 1 -- 172.21.15.50:0/2386331519 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb324004240 con 0x7fb32c103b70 2024-04-23T18:16:08.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.723+0000 7fb33247c640 1 -- 172.21.15.50:0/2386331519 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb32c1a0020 con 0x7fb32c103b70 2024-04-23T18:16:08.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.723+0000 7fb3297fa640 1 -- 172.21.15.50:0/2386331519 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb324003e10 con 0x7fb32c103b70 2024-04-23T18:16:08.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.723+0000 7fb3297fa640 1 -- 172.21.15.50:0/2386331519 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3240305c0 con 0x7fb32c103b70 2024-04-23T18:16:08.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.723+0000 7fb33247c640 1 -- 172.21.15.50:0/2386331519 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb32c1a05c0 con 0x7fb32c103b70 2024-04-23T18:16:08.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.724+0000 7fb33247c640 1 -- 172.21.15.50:0/2386331519 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb32c109380 con 0x7fb32c103b70 2024-04-23T18:16:08.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.725+0000 7fb3297fa640 1 -- 172.21.15.50:0/2386331519 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb32402f660 con 0x7fb32c103b70 2024-04-23T18:16:08.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.726+0000 7fb3297fa640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3140a3bb0 0x7fb3140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:08.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.726+0000 7fb32bfff640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3140a3bb0 0x7fb3140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:08.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.726+0000 7fb3297fa640 1 -- 172.21.15.50:0/2386331519 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fb3240f18e0 con 0x7fb32c103b70 2024-04-23T18:16:08.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.727+0000 7fb32bfff640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3140a3bb0 0x7fb3140a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0058c0 tx=0x7fb31c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:08.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.730+0000 7fb3297fa640 1 -- 172.21.15.50:0/2386331519 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3240be7f0 con 0x7fb32c103b70 2024-04-23T18:16:08.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.955+0000 7fb33247c640 1 -- 172.21.15.50:0/2386331519 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb32c062b50 con 0x7fb32c103b70 2024-04-23T18:16:08.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.956+0000 7fb3297fa640 1 -- 172.21.15.50:0/2386331519 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7fb3240be0f0 con 0x7fb32c103b70 2024-04-23T18:16:08.950 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:08.950 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T18:16:08.950 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 5 2024-04-23T18:16:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.958+0000 7fb33247c640 1 -- 172.21.15.50:0/2386331519 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3140a3bb0 msgr2=0x7fb3140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.958+0000 7fb33247c640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3140a3bb0 0x7fb3140a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0058c0 tx=0x7fb31c00a2d0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.958+0000 7fb33247c640 1 -- 172.21.15.50:0/2386331519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb32c103b70 msgr2=0x7fb32c19f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.958+0000 7fb33247c640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb32c103b70 0x7fb32c19f720 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb324031a20 tx=0x7fb324003c20 comp rx=0 tx=0).stop 2024-04-23T18:16:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.958+0000 7fb33247c640 1 -- 172.21.15.50:0/2386331519 shutdown_connections 2024-04-23T18:16:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.958+0000 7fb33247c640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3140a3bb0 0x7fb3140a6070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.958+0000 7fb33247c640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb32c103b70 0x7fb32c19f720 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.958+0000 7fb33247c640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb32c1031b0 0x7fb32c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.958+0000 7fb33247c640 1 --2- 172.21.15.50:0/2386331519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb32c101fc0 0x7fb32c198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.958+0000 7fb33247c640 1 -- 172.21.15.50:0/2386331519 >> 172.21.15.50:0/2386331519 conn(0x7fb32c0fd550 msgr2=0x7fb32c106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.958+0000 7fb33247c640 1 -- 172.21.15.50:0/2386331519 shutdown_connections 2024-04-23T18:16:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:08.958+0000 7fb33247c640 1 -- 172.21.15.50:0/2386331519 wait complete. 2024-04-23T18:16:09.284 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:16:09 smithi177 systemd[1]: Starting Ceph mds.e for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:16:09.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:09 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2386331519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:09.538 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:16:09 smithi177 podman[62377]: 2024-04-23 18:16:09.322418721 +0000 UTC m=+0.108426548 container create 18dff365af235e26491c68160cfa6301c51821148ea5f32083e997f955815ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T18:16:09.538 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:16:09 smithi177 podman[62377]: 2024-04-23 18:16:09.232236217 +0000 UTC m=+0.018244058 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:16:09.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:09 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2386331519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:09.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:09 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2386331519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:10.038 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:16:09 smithi177 podman[62377]: 2024-04-23 18:16:09.549428463 +0000 UTC m=+0.335436290 container init 18dff365af235e26491c68160cfa6301c51821148ea5f32083e997f955815ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:16:10.038 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:16:09 smithi177 podman[62377]: 2024-04-23 18:16:09.552025616 +0000 UTC m=+0.338033444 container start 18dff365af235e26491c68160cfa6301c51821148ea5f32083e997f955815ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-23T18:16:10.038 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:16:09 smithi177 ceph-mds[62396]: -- 172.21.15.177:0/2179807397 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d724217a40 con 0x55d7241eac00 2024-04-23T18:16:10.038 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:16:09 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e[62392]: starting mds.e at 2024-04-23T18:16:10.038 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:16:09 smithi177 bash[62377]: 18dff365af235e26491c68160cfa6301c51821148ea5f32083e997f955815ccb 2024-04-23T18:16:10.038 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:16:09 smithi177 systemd[1]: Started Ceph mds.e for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:16:10.536 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:10 smithi188 ceph-mon[32694]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:10.536 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:10 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:10.536 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:10 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:10.536 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:10 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:10.536 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:10 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:10.536 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:10 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:10.536 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:10 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:10.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:10 smithi177 ceph-mon[32308]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:10.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:10 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:10.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:10 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:10.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:10 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:10.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:10 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:10.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:10 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:10.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:10 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:10.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:10 smithi050 ceph-mon[29187]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:10.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:10 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:10.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:10 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:10.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:10 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:10.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:10 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:10.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:10 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:10.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:10 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:10.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:11.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.085+0000 7fd617679640 1 -- 172.21.15.50:0/792347219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd610105400 msgr2=0x7fd6101077f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.085+0000 7fd617679640 1 --2- 172.21.15.50:0/792347219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd610105400 0x7fd6101077f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd600009900 tx=0x7fd60002a410 comp rx=0 tx=0).stop 2024-04-23T18:16:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.085+0000 7fd617679640 1 -- 172.21.15.50:0/792347219 shutdown_connections 2024-04-23T18:16:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.085+0000 7fd617679640 1 --2- 172.21.15.50:0/792347219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd610107d30 0x7fd61010a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.085+0000 7fd617679640 1 --2- 172.21.15.50:0/792347219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd610105400 0x7fd6101077f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.085+0000 7fd617679640 1 --2- 172.21.15.50:0/792347219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd610102ad0 0x7fd610104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.085+0000 7fd617679640 1 -- 172.21.15.50:0/792347219 >> 172.21.15.50:0/792347219 conn(0x7fd6100fc440 msgr2=0x7fd6100fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.085+0000 7fd617679640 1 -- 172.21.15.50:0/792347219 shutdown_connections 2024-04-23T18:16:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.085+0000 7fd617679640 1 -- 172.21.15.50:0/792347219 wait complete. 2024-04-23T18:16:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd617679640 1 Processor -- start 2024-04-23T18:16:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd617679640 1 -- start start 2024-04-23T18:16:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd617679640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd610102ad0 0x7fd610198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd617679640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd610105400 0x7fd610198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd617679640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd610107d30 0x7fd61019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd617679640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd61010db10 con 0x7fd610107d30 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd617679640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd61010d990 con 0x7fd610102ad0 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd617679640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd61010dc90 con 0x7fd610105400 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd615c16640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd610107d30 0x7fd61019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd615c16640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd610107d30 0x7fd61019f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51208/0 (socket says 172.21.15.50:51208) 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd615c16640 1 -- 172.21.15.50:0/315208363 learned_addr learned my addr 172.21.15.50:0/315208363 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd615415640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd610102ad0 0x7fd610198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.086+0000 7fd614c14640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd610105400 0x7fd610198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.087+0000 7fd615c16640 1 -- 172.21.15.50:0/315208363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd610105400 msgr2=0x7fd610198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.087+0000 7fd615c16640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd610105400 0x7fd610198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.087+0000 7fd615c16640 1 -- 172.21.15.50:0/315208363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd610102ad0 msgr2=0x7fd610198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.087+0000 7fd615c16640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd610102ad0 0x7fd610198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.087+0000 7fd615c16640 1 -- 172.21.15.50:0/315208363 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61019fe40 con 0x7fd610107d30 2024-04-23T18:16:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.087+0000 7fd615c16640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd610107d30 0x7fd61019f7d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd60800eaa0 tx=0x7fd60800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.087+0000 7fd6067fc640 1 -- 172.21.15.50:0/315208363 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60800cdb0 con 0x7fd610107d30 2024-04-23T18:16:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.087+0000 7fd6067fc640 1 -- 172.21.15.50:0/315208363 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd608005020 con 0x7fd610107d30 2024-04-23T18:16:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.087+0000 7fd617679640 1 -- 172.21.15.50:0/315208363 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6101a0130 con 0x7fd610107d30 2024-04-23T18:16:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.087+0000 7fd6067fc640 1 -- 172.21.15.50:0/315208363 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6080116d0 con 0x7fd610107d30 2024-04-23T18:16:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.087+0000 7fd617679640 1 -- 172.21.15.50:0/315208363 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6101a0650 con 0x7fd610107d30 2024-04-23T18:16:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.088+0000 7fd617679640 1 -- 172.21.15.50:0/315208363 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6100fdf30 con 0x7fd610107d30 2024-04-23T18:16:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.089+0000 7fd6067fc640 1 -- 172.21.15.50:0/315208363 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd6080051d0 con 0x7fd610107d30 2024-04-23T18:16:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.090+0000 7fd6067fc640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd5fc0a3c30 0x7fd5fc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.090+0000 7fd6067fc640 1 -- 172.21.15.50:0/315208363 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fd6080d03d0 con 0x7fd610107d30 2024-04-23T18:16:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.090+0000 7fd615415640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd5fc0a3c30 0x7fd5fc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.091+0000 7fd615415640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd5fc0a3c30 0x7fd5fc0a60f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd60c00bbe0 tx=0x7fd60c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.094+0000 7fd6067fc640 1 -- 172.21.15.50:0/315208363 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd608016030 con 0x7fd610107d30 2024-04-23T18:16:11.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.322+0000 7fd617679640 1 -- 172.21.15.50:0/315208363 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd61002d5c0 con 0x7fd610107d30 2024-04-23T18:16:11.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.322+0000 7fd6067fc640 1 -- 172.21.15.50:0/315208363 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7fd60809cc80 con 0x7fd610107d30 2024-04-23T18:16:11.317 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:11.317 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-23T18:16:11.317 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 6 2024-04-23T18:16:11.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.325+0000 7fd617679640 1 -- 172.21.15.50:0/315208363 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd5fc0a3c30 msgr2=0x7fd5fc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:11.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.325+0000 7fd617679640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd5fc0a3c30 0x7fd5fc0a60f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd60c00bbe0 tx=0x7fd60c00b390 comp rx=0 tx=0).stop 2024-04-23T18:16:11.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.325+0000 7fd617679640 1 -- 172.21.15.50:0/315208363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd610107d30 msgr2=0x7fd61019f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:11.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.325+0000 7fd617679640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd610107d30 0x7fd61019f7d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd60800eaa0 tx=0x7fd60800ef60 comp rx=0 tx=0).stop 2024-04-23T18:16:11.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.325+0000 7fd617679640 1 -- 172.21.15.50:0/315208363 shutdown_connections 2024-04-23T18:16:11.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.325+0000 7fd617679640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd5fc0a3c30 0x7fd5fc0a60f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:11.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.325+0000 7fd617679640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd610107d30 0x7fd61019f7d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:11.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.325+0000 7fd617679640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd610105400 0x7fd610198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:11.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.325+0000 7fd617679640 1 --2- 172.21.15.50:0/315208363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd610102ad0 0x7fd610198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:11.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.325+0000 7fd617679640 1 -- 172.21.15.50:0/315208363 >> 172.21.15.50:0/315208363 conn(0x7fd6100fc440 msgr2=0x7fd610103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:11.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.325+0000 7fd617679640 1 -- 172.21.15.50:0/315208363 shutdown_connections 2024-04-23T18:16:11.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:11.325+0000 7fd617679640 1 -- 172.21.15.50:0/315208363 wait complete. 2024-04-23T18:16:11.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:11 smithi177 ceph-mon[32308]: Deploying daemon mds.f on smithi188 2024-04-23T18:16:11.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:11 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6834/475975073,v1:172.21.15.177:6835/475975073] up:boot 2024-04-23T18:16:11.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:11 smithi177 ceph-mon[32308]: fsmap 5 up:standby 2024-04-23T18:16:11.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:11 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:16:11.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:11 smithi188 ceph-mon[32694]: Deploying daemon mds.f on smithi188 2024-04-23T18:16:11.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:11 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6834/475975073,v1:172.21.15.177:6835/475975073] up:boot 2024-04-23T18:16:11.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:11 smithi188 ceph-mon[32694]: fsmap 5 up:standby 2024-04-23T18:16:11.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:11 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:16:11.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:11 smithi050 ceph-mon[29187]: Deploying daemon mds.f on smithi188 2024-04-23T18:16:11.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:11 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6834/475975073,v1:172.21.15.177:6835/475975073] up:boot 2024-04-23T18:16:11.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:11 smithi050 ceph-mon[29187]: fsmap 5 up:standby 2024-04-23T18:16:11.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:11 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:16:11.992 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:16:11 smithi188 systemd[1]: Starting Ceph mds.f for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:16:12.252 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:16:11 smithi188 podman[63376]: 2024-04-23 18:16:11.998622067 +0000 UTC m=+0.114531167 container create 3323d250daaa620e5f6439f47bf6aab87e42acdd3e44cf29c6fae511ecc82a84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-23T18:16:12.253 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:16:11 smithi188 podman[63376]: 2024-04-23 18:16:11.902059596 +0000 UTC m=+0.017968647 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:16:12.253 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:16:12 smithi188 podman[63376]: 2024-04-23 18:16:12.259744256 +0000 UTC m=+0.375653298 container init 3323d250daaa620e5f6439f47bf6aab87e42acdd3e44cf29c6fae511ecc82a84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:16:12.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:12 smithi177 ceph-mon[32308]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:12.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:12 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/315208363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:12.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:12 smithi188 ceph-mon[32694]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:12.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:12 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/315208363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:12.580 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:16:12 smithi188 podman[63376]: 2024-04-23 18:16:12.262286255 +0000 UTC m=+0.378195302 container start 3323d250daaa620e5f6439f47bf6aab87e42acdd3e44cf29c6fae511ecc82a84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-04-23T18:16:12.580 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:16:12 smithi188 ceph-mds[63395]: -- 172.21.15.188:0/1612205475 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56190f8d1a40 con 0x56190f8a4c00 2024-04-23T18:16:12.580 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:16:12 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f[63391]: starting mds.f at 2024-04-23T18:16:12.580 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:16:12 smithi188 bash[63376]: 3323d250daaa620e5f6439f47bf6aab87e42acdd3e44cf29c6fae511ecc82a84 2024-04-23T18:16:12.580 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:16:12 smithi188 systemd[1]: Started Ceph mds.f for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:16:12.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:12 smithi050 ceph-mon[29187]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:12.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:12 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/315208363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:13.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:13 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.188:6834/1248812622,v1:172.21.15.188:6835/1248812622] up:boot 2024-04-23T18:16:13.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:13 smithi188 ceph-mon[32694]: fsmap 6 up:standby 2024-04-23T18:16:13.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:16:13.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:13.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:13.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:13.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:13.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:13.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:13 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:13.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:13 smithi188 ceph-mon[32694]: Deploying daemon mds.c on smithi188 2024-04-23T18:16:13.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:13 smithi188 ceph-mon[32694]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:13.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.452+0000 7f52d1395640 1 -- 172.21.15.50:0/992911541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc102a00 msgr2=0x7f52cc109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.452+0000 7f52d1395640 1 --2- 172.21.15.50:0/992911541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc102a00 0x7f52cc109320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f52c4009ca0 tx=0x7f52c402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.452+0000 7f52d1395640 1 -- 172.21.15.50:0/992911541 shutdown_connections 2024-04-23T18:16:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.452+0000 7f52d1395640 1 --2- 172.21.15.50:0/992911541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc102a00 0x7f52cc109320 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.452+0000 7f52d1395640 1 --2- 172.21.15.50:0/992911541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52cc102040 0x7f52cc1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.452+0000 7f52d1395640 1 --2- 172.21.15.50:0/992911541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52cc100e60 0x7f52cc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.452+0000 7f52d1395640 1 -- 172.21.15.50:0/992911541 >> 172.21.15.50:0/992911541 conn(0x7f52cc0fc440 msgr2=0x7f52cc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.452+0000 7f52d1395640 1 -- 172.21.15.50:0/992911541 shutdown_connections 2024-04-23T18:16:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.452+0000 7f52d1395640 1 -- 172.21.15.50:0/992911541 wait complete. 2024-04-23T18:16:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.453+0000 7f52d1395640 1 Processor -- start 2024-04-23T18:16:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.453+0000 7f52d1395640 1 -- start start 2024-04-23T18:16:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.453+0000 7f52d1395640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52cc100e60 0x7f52cc06b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.453+0000 7f52d1395640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc102040 0x7f52cc19f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.453+0000 7f52d1395640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52cc102a00 0x7f52cc1a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.453+0000 7f52d1395640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc10ca40 con 0x7f52cc102040 2024-04-23T18:16:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.453+0000 7f52d1395640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc10c8c0 con 0x7f52cc100e60 2024-04-23T18:16:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.453+0000 7f52d1395640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc10cbc0 con 0x7f52cc102a00 2024-04-23T18:16:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.453+0000 7f52ca7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc102040 0x7f52cc19f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.453+0000 7f52ca7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc102040 0x7f52cc19f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34260/0 (socket says 172.21.15.50:34260) 2024-04-23T18:16:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.453+0000 7f52ca7fc640 1 -- 172.21.15.50:0/2202465161 learned_addr learned my addr 172.21.15.50:0/2202465161 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.453+0000 7f52caffd640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52cc100e60 0x7f52cc06b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.454+0000 7f52cb7fe640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52cc102a00 0x7f52cc1a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.454+0000 7f52ca7fc640 1 -- 172.21.15.50:0/2202465161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52cc102a00 msgr2=0x7f52cc1a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.454+0000 7f52ca7fc640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52cc102a00 0x7f52cc1a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.454+0000 7f52ca7fc640 1 -- 172.21.15.50:0/2202465161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52cc100e60 msgr2=0x7f52cc06b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.454+0000 7f52ca7fc640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52cc100e60 0x7f52cc06b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.454+0000 7f52ca7fc640 1 -- 172.21.15.50:0/2202465161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc1a1f60 con 0x7f52cc102040 2024-04-23T18:16:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.454+0000 7f52ca7fc640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc102040 0x7f52cc19f210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f52c000e9f0 tx=0x7f52c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.454+0000 7f52b3fff640 1 -- 172.21.15.50:0/2202465161 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c000cd80 con 0x7f52cc102040 2024-04-23T18:16:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.454+0000 7f52b3fff640 1 -- 172.21.15.50:0/2202465161 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52c00108e0 con 0x7f52cc102040 2024-04-23T18:16:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.454+0000 7f52b3fff640 1 -- 172.21.15.50:0/2202465161 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c0018980 con 0x7f52cc102040 2024-04-23T18:16:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.454+0000 7f52d1395640 1 -- 172.21.15.50:0/2202465161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52cc1a2250 con 0x7f52cc102040 2024-04-23T18:16:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.454+0000 7f52d1395640 1 -- 172.21.15.50:0/2202465161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52cc1a27f0 con 0x7f52cc102040 2024-04-23T18:16:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.455+0000 7f52d1395640 1 -- 172.21.15.50:0/2202465161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52cc1081a0 con 0x7f52cc102040 2024-04-23T18:16:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.457+0000 7f52b3fff640 1 -- 172.21.15.50:0/2202465161 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f52c0010430 con 0x7f52cc102040 2024-04-23T18:16:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.458+0000 7f52b3fff640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52b40a3c30 0x7f52b40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.458+0000 7f52caffd640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52b40a3c30 0x7f52b40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.458+0000 7f52b3fff640 1 -- 172.21.15.50:0/2202465161 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f52c00d0740 con 0x7f52cc102040 2024-04-23T18:16:13.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.458+0000 7f52caffd640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52b40a3c30 0x7f52b40a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f52bc005fd0 tx=0x7f52bc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.462+0000 7f52b3fff640 1 -- 172.21.15.50:0/2202465161 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f52c009d520 con 0x7f52cc102040 2024-04-23T18:16:13.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.690+0000 7f52d1395640 1 -- 172.21.15.50:0/2202465161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52cc02d5c0 con 0x7f52cc102040 2024-04-23T18:16:13.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.690+0000 7f52b3fff640 1 -- 172.21.15.50:0/2202465161 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7f52c009ce20 con 0x7f52cc102040 2024-04-23T18:16:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:13.685 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-23T18:16:13.685 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 7 2024-04-23T18:16:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.692+0000 7f52d1395640 1 -- 172.21.15.50:0/2202465161 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52b40a3c30 msgr2=0x7f52b40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.692+0000 7f52d1395640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52b40a3c30 0x7f52b40a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f52bc005fd0 tx=0x7f52bc00b410 comp rx=0 tx=0).stop 2024-04-23T18:16:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.692+0000 7f52d1395640 1 -- 172.21.15.50:0/2202465161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc102040 msgr2=0x7f52cc19f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.692+0000 7f52d1395640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc102040 0x7f52cc19f210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f52c000e9f0 tx=0x7f52c000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:16:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.692+0000 7f52d1395640 1 -- 172.21.15.50:0/2202465161 shutdown_connections 2024-04-23T18:16:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.692+0000 7f52d1395640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52b40a3c30 0x7f52b40a60f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.692+0000 7f52d1395640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52cc102a00 0x7f52cc1a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.692+0000 7f52d1395640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc102040 0x7f52cc19f210 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.692+0000 7f52d1395640 1 --2- 172.21.15.50:0/2202465161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52cc100e60 0x7f52cc06b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.692+0000 7f52d1395640 1 -- 172.21.15.50:0/2202465161 >> 172.21.15.50:0/2202465161 conn(0x7f52cc0fc440 msgr2=0x7f52cc105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.693+0000 7f52d1395640 1 -- 172.21.15.50:0/2202465161 shutdown_connections 2024-04-23T18:16:13.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:13.693+0000 7f52d1395640 1 -- 172.21.15.50:0/2202465161 wait complete. 2024-04-23T18:16:13.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:13 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.188:6834/1248812622,v1:172.21.15.188:6835/1248812622] up:boot 2024-04-23T18:16:13.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:13 smithi050 ceph-mon[29187]: fsmap 6 up:standby 2024-04-23T18:16:13.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:16:13.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:13.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:13.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:13.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:13.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:13.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:13 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:13.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:13 smithi050 ceph-mon[29187]: Deploying daemon mds.c on smithi188 2024-04-23T18:16:13.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:13 smithi050 ceph-mon[29187]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:13.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:13 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.188:6834/1248812622,v1:172.21.15.188:6835/1248812622] up:boot 2024-04-23T18:16:13.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:13 smithi177 ceph-mon[32308]: fsmap 6 up:standby 2024-04-23T18:16:13.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:16:13.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:13.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:13.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:13.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:13.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:13.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:13 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:13 smithi177 ceph-mon[32308]: Deploying daemon mds.c on smithi188 2024-04-23T18:16:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:13 smithi177 ceph-mon[32308]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:14.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:14 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2202465161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:14.580 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:16:14 smithi188 systemd[1]: Starting Ceph mds.c for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:16:14.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:14 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2202465161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:14.788 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:14 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2202465161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:14.915 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:16:14 smithi188 podman[63685]: 2024-04-23 18:16:14.668854334 +0000 UTC m=+0.094658722 container create 9ced8839238f399099b97e16c52bbedcc533082d605f316e3eb5d87749009e93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-04-23T18:16:14.915 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:16:14 smithi188 podman[63685]: 2024-04-23 18:16:14.592571196 +0000 UTC m=+0.018375539 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:16:14.916 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:16:14 smithi188 podman[63685]: 2024-04-23 18:16:14.89623436 +0000 UTC m=+0.322038703 container init 9ced8839238f399099b97e16c52bbedcc533082d605f316e3eb5d87749009e93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-23T18:16:14.916 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:16:14 smithi188 podman[63685]: 2024-04-23 18:16:14.898749672 +0000 UTC m=+0.324554018 container start 9ced8839238f399099b97e16c52bbedcc533082d605f316e3eb5d87749009e93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-23T18:16:14.916 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:16:14 smithi188 ceph-mds[63704]: -- 172.21.15.188:0/2211168850 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ab036cba40 con 0x55ab0369f000 2024-04-23T18:16:15.330 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:16:14 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-c[63700]: starting mds.c at 2024-04-23T18:16:15.330 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:16:14 smithi188 bash[63685]: 9ced8839238f399099b97e16c52bbedcc533082d605f316e3eb5d87749009e93 2024-04-23T18:16:15.330 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:16:14 smithi188 systemd[1]: Started Ceph mds.c for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:16:15.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:15.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.819+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/1484419021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c481031b0 msgr2=0x7f1c48103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.819+0000 7f1c4d0a0640 1 --2- 172.21.15.50:0/1484419021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c481031b0 0x7f1c48103630 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c009900 tx=0x7f1c3c02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.819+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/1484419021 shutdown_connections 2024-04-23T18:16:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.819+0000 7f1c4d0a0640 1 --2- 172.21.15.50:0/1484419021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c48103b70 0x7f1c4810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.819+0000 7f1c4d0a0640 1 --2- 172.21.15.50:0/1484419021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c481031b0 0x7f1c48103630 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.819+0000 7f1c4d0a0640 1 --2- 172.21.15.50:0/1484419021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c48101fc0 0x7f1c481023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.819+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/1484419021 >> 172.21.15.50:0/1484419021 conn(0x7f1c480fd550 msgr2=0x7f1c480ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.819+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/1484419021 shutdown_connections 2024-04-23T18:16:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.819+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/1484419021 wait complete. 2024-04-23T18:16:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.820+0000 7f1c4d0a0640 1 Processor -- start 2024-04-23T18:16:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.820+0000 7f1c4d0a0640 1 -- start start 2024-04-23T18:16:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.820+0000 7f1c4d0a0640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c48101fc0 0x7f1c48198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.820+0000 7f1c4d0a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c481031b0 0x7f1c481987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4d0a0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c48103b70 0x7f1c4819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4d0a0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4810db00 con 0x7f1c481031b0 2024-04-23T18:16:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4d0a0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4810d980 con 0x7f1c48103b70 2024-04-23T18:16:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4d0a0640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4810dc80 con 0x7f1c48101fc0 2024-04-23T18:16:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c481031b0 0x7f1c481987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c481031b0 0x7f1c481987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34266/0 (socket says 172.21.15.50:34266) 2024-04-23T18:16:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4659c640 1 -- 172.21.15.50:0/169839274 learned_addr learned my addr 172.21.15.50:0/169839274 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c46d9d640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c48101fc0 0x7f1c48198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4759e640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c48103b70 0x7f1c4819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4659c640 1 -- 172.21.15.50:0/169839274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c48101fc0 msgr2=0x7f1c48198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4659c640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c48101fc0 0x7f1c48198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4659c640 1 -- 172.21.15.50:0/169839274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c48103b70 msgr2=0x7f1c4819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4659c640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c48103b70 0x7f1c4819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4659c640 1 -- 172.21.15.50:0/169839274 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4819fe00 con 0x7f1c481031b0 2024-04-23T18:16:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4759e640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c48103b70 0x7f1c4819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4659c640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c481031b0 0x7f1c481987c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c005940 tx=0x7f1c3c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c2ffff640 1 -- 172.21.15.50:0/169839274 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c03f030 con 0x7f1c481031b0 2024-04-23T18:16:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/169839274 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c481a0090 con 0x7f1c481031b0 2024-04-23T18:16:15.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c2ffff640 1 -- 172.21.15.50:0/169839274 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c3c003aa0 con 0x7f1c481031b0 2024-04-23T18:16:15.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.821+0000 7f1c2ffff640 1 -- 172.21.15.50:0/169839274 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c035070 con 0x7f1c481031b0 2024-04-23T18:16:15.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.822+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/169839274 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c481a05d0 con 0x7f1c481031b0 2024-04-23T18:16:15.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.823+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/169839274 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c48109300 con 0x7f1c481031b0 2024-04-23T18:16:15.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.824+0000 7f1c2ffff640 1 -- 172.21.15.50:0/169839274 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1c3c042030 con 0x7f1c481031b0 2024-04-23T18:16:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.824+0000 7f1c2ffff640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1c300a3bb0 0x7f1c300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.824+0000 7f1c2ffff640 1 -- 172.21.15.50:0/169839274 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f1c3c0f0650 con 0x7f1c481031b0 2024-04-23T18:16:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.824+0000 7f1c46d9d640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1c300a3bb0 0x7f1c300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.825+0000 7f1c46d9d640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1c300a3bb0 0x7f1c300a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1c38005fd0 tx=0x7f1c38005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:15.829+0000 7f1c2ffff640 1 -- 172.21.15.50:0/169839274 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c3c0bd4d0 con 0x7f1c481031b0 2024-04-23T18:16:16.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.054+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/169839274 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c4802d770 con 0x7f1c481031b0 2024-04-23T18:16:16.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.055+0000 7f1c2ffff640 1 -- 172.21.15.50:0/169839274 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f1c3c0bcdd0 con 0x7f1c481031b0 2024-04-23T18:16:16.049 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:16.049 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-23T18:16:16.049 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 8 2024-04-23T18:16:16.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.057+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/169839274 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1c300a3bb0 msgr2=0x7f1c300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.057+0000 7f1c4d0a0640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1c300a3bb0 0x7f1c300a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1c38005fd0 tx=0x7f1c38005850 comp rx=0 tx=0).stop 2024-04-23T18:16:16.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.057+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/169839274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c481031b0 msgr2=0x7f1c481987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:16.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.057+0000 7f1c4d0a0640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c481031b0 0x7f1c481987c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c005940 tx=0x7f1c3c031a50 comp rx=0 tx=0).stop 2024-04-23T18:16:16.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.057+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/169839274 shutdown_connections 2024-04-23T18:16:16.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.057+0000 7f1c4d0a0640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1c300a3bb0 0x7f1c300a6070 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.057+0000 7f1c4d0a0640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c48103b70 0x7f1c4819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.057+0000 7f1c4d0a0640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1c481031b0 0x7f1c481987c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.057+0000 7f1c4d0a0640 1 --2- 172.21.15.50:0/169839274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c48101fc0 0x7f1c48198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:16.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.057+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/169839274 >> 172.21.15.50:0/169839274 conn(0x7f1c480fd550 msgr2=0x7f1c481064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:16.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.057+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/169839274 shutdown_connections 2024-04-23T18:16:16.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:16.057+0000 7f1c4d0a0640 1 -- 172.21.15.50:0/169839274 wait complete. 2024-04-23T18:16:16.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:16 smithi177 ceph-mon[32308]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:16.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:16 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:16.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:16 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:16 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:16 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:16 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:16 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:16 smithi177 ceph-mon[32308]: Deploying daemon mds.h on smithi177 2024-04-23T18:16:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:16 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] up:boot 2024-04-23T18:16:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:16 smithi177 ceph-mon[32308]: fsmap 7 up:standby 2024-04-23T18:16:16.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:16 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:16:16.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:16 smithi188 ceph-mon[32694]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:16.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:16 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:16.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:16 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:16.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:16 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:16.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:16 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:16 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:16 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:16 smithi188 ceph-mon[32694]: Deploying daemon mds.h on smithi177 2024-04-23T18:16:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:16 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] up:boot 2024-04-23T18:16:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:16 smithi188 ceph-mon[32694]: fsmap 7 up:standby 2024-04-23T18:16:16.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:16 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:16:16.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:16 smithi050 ceph-mon[29187]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:16.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:16 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:16.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:16 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:16.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:16 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:16.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:16 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:16.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:16 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:16.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:16 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:16.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:16 smithi050 ceph-mon[29187]: Deploying daemon mds.h on smithi177 2024-04-23T18:16:16.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:16 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] up:boot 2024-04-23T18:16:16.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:16 smithi050 ceph-mon[29187]: fsmap 7 up:standby 2024-04-23T18:16:16.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:16 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:16:17.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:17 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/169839274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:17.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:16:17 smithi177 systemd[1]: Starting Ceph mds.h for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:16:17.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:17 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/169839274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:17.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:17 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/169839274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:17.645 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:16:17 smithi177 podman[62689]: 2024-04-23 18:16:17.398109672 +0000 UTC m=+0.102052695 container create 9b09d288c4b24034e1b47bac5ae3eeed90f9a0097a3ebd8fca5eeeff13a72236 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:16:17.645 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:16:17 smithi177 podman[62689]: 2024-04-23 18:16:17.314393951 +0000 UTC m=+0.018336981 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:16:17.645 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:16:17 smithi177 podman[62689]: 2024-04-23 18:16:17.625840126 +0000 UTC m=+0.329783149 container init 9b09d288c4b24034e1b47bac5ae3eeed90f9a0097a3ebd8fca5eeeff13a72236 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-23T18:16:17.645 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:16:17 smithi177 podman[62689]: 2024-04-23 18:16:17.628571561 +0000 UTC m=+0.332514591 container start 9b09d288c4b24034e1b47bac5ae3eeed90f9a0097a3ebd8fca5eeeff13a72236 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-23T18:16:18.038 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:16:17 smithi177 ceph-mds[62716]: -- 172.21.15.177:0/1628460372 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5616eba03a40 con 0x5616eb9d6c00 2024-04-23T18:16:18.038 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:16:17 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: starting mds.h at 2024-04-23T18:16:18.038 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:16:17 smithi177 bash[62689]: 9b09d288c4b24034e1b47bac5ae3eeed90f9a0097a3ebd8fca5eeeff13a72236 2024-04-23T18:16:18.038 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:16:17 smithi177 systemd[1]: Started Ceph mds.h for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:16:18.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:18.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.183+0000 7fb19bd29640 1 -- 172.21.15.50:0/3099430710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb194100270 msgr2=0x7fb194100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:18.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.183+0000 7fb19bd29640 1 --2- 172.21.15.50:0/3099430710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb194100270 0x7fb194100690 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb190009930 tx=0x7fb19002a420 comp rx=0 tx=0).stop 2024-04-23T18:16:18.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.183+0000 7fb19bd29640 1 -- 172.21.15.50:0/3099430710 shutdown_connections 2024-04-23T18:16:18.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.183+0000 7fb19bd29640 1 --2- 172.21.15.50:0/3099430710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb194103c00 0x7fb19410a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.183+0000 7fb19bd29640 1 --2- 172.21.15.50:0/3099430710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb194103240 0x7fb1941036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.183+0000 7fb19bd29640 1 --2- 172.21.15.50:0/3099430710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb194100270 0x7fb194100690 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.183+0000 7fb19bd29640 1 -- 172.21.15.50:0/3099430710 >> 172.21.15.50:0/3099430710 conn(0x7fb1940fbbc0 msgr2=0x7fb1940fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:18.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.183+0000 7fb19bd29640 1 -- 172.21.15.50:0/3099430710 shutdown_connections 2024-04-23T18:16:18.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.183+0000 7fb19bd29640 1 -- 172.21.15.50:0/3099430710 wait complete. 2024-04-23T18:16:18.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb19bd29640 1 Processor -- start 2024-04-23T18:16:18.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb19bd29640 1 -- start start 2024-04-23T18:16:18.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb19bd29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb194100270 0x7fb194198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:18.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb19bd29640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb194103240 0x7fb194198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:18.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb19bd29640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb194103c00 0x7fb19419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:18.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb19bd29640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb19410db10 con 0x7fb194100270 2024-04-23T18:16:18.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb19bd29640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb19410d990 con 0x7fb194103c00 2024-04-23T18:16:18.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb19bd29640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb19410dc90 con 0x7fb194103240 2024-04-23T18:16:18.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb199ac5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb194100270 0x7fb194198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:18.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb199ac5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb194100270 0x7fb194198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34292/0 (socket says 172.21.15.50:34292) 2024-04-23T18:16:18.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb199ac5640 1 -- 172.21.15.50:0/684696167 learned_addr learned my addr 172.21.15.50:0/684696167 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:18.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb1992c4640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb194103240 0x7fb194198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.184+0000 7fb19a2c6640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb194103c00 0x7fb19419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.185+0000 7fb199ac5640 1 -- 172.21.15.50:0/684696167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb194103240 msgr2=0x7fb194198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.185+0000 7fb199ac5640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb194103240 0x7fb194198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.185+0000 7fb199ac5640 1 -- 172.21.15.50:0/684696167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb194103c00 msgr2=0x7fb19419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.185+0000 7fb199ac5640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb194103c00 0x7fb19419f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.185+0000 7fb199ac5640 1 -- 172.21.15.50:0/684696167 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19419fe40 con 0x7fb194100270 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.185+0000 7fb1992c4640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb194103240 0x7fb194198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.185+0000 7fb199ac5640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb194100270 0x7fb194198220 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb190009230 tx=0x7fb1900042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.185+0000 7fb18affd640 1 -- 172.21.15.50:0/684696167 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19003f030 con 0x7fb194100270 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.185+0000 7fb18affd640 1 -- 172.21.15.50:0/684696167 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb190032e50 con 0x7fb194100270 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.185+0000 7fb19bd29640 1 -- 172.21.15.50:0/684696167 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1941a00d0 con 0x7fb194100270 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.185+0000 7fb18affd640 1 -- 172.21.15.50:0/684696167 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190035070 con 0x7fb194100270 2024-04-23T18:16:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.185+0000 7fb19bd29640 1 -- 172.21.15.50:0/684696167 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1941a05c0 con 0x7fb194100270 2024-04-23T18:16:18.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.186+0000 7fb19bd29640 1 -- 172.21.15.50:0/684696167 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1941094e0 con 0x7fb194100270 2024-04-23T18:16:18.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.187+0000 7fb18affd640 1 -- 172.21.15.50:0/684696167 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb190042030 con 0x7fb194100270 2024-04-23T18:16:18.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.188+0000 7fb18affd640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb1800a3bb0 0x7fb1800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:18.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.189+0000 7fb1992c4640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb1800a3bb0 0x7fb1800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:18.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.189+0000 7fb18affd640 1 -- 172.21.15.50:0/684696167 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fb1900f0890 con 0x7fb194100270 2024-04-23T18:16:18.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.189+0000 7fb1992c4640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb1800a3bb0 0x7fb1800a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb1840095f0 tx=0x7fb184005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:18.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.192+0000 7fb18affd640 1 -- 172.21.15.50:0/684696167 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb1900bd0c0 con 0x7fb194100270 2024-04-23T18:16:18.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:18 smithi188 ceph-mon[32694]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:18.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:18.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:18.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:18.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:18.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:18.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:18 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:18.382 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:18 smithi177 ceph-mon[32308]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:18.382 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:18.382 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:18.383 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:18.383 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:18.383 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:18.383 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:18 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:18.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.421+0000 7fb19bd29640 1 -- 172.21.15.50:0/684696167 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb19402d5c0 con 0x7fb194100270 2024-04-23T18:16:18.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.422+0000 7fb18affd640 1 -- 172.21.15.50:0/684696167 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7fb1900c13c0 con 0x7fb194100270 2024-04-23T18:16:18.416 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:18.416 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-23T18:16:18.416 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 9 2024-04-23T18:16:18.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.424+0000 7fb19bd29640 1 -- 172.21.15.50:0/684696167 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb1800a3bb0 msgr2=0x7fb1800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:18.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.424+0000 7fb19bd29640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb1800a3bb0 0x7fb1800a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb1840095f0 tx=0x7fb184005f90 comp rx=0 tx=0).stop 2024-04-23T18:16:18.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.424+0000 7fb19bd29640 1 -- 172.21.15.50:0/684696167 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb194100270 msgr2=0x7fb194198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:18.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.424+0000 7fb19bd29640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb194100270 0x7fb194198220 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb190009230 tx=0x7fb1900042f0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.424+0000 7fb19bd29640 1 -- 172.21.15.50:0/684696167 shutdown_connections 2024-04-23T18:16:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.424+0000 7fb19bd29640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb1800a3bb0 0x7fb1800a6070 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.424+0000 7fb19bd29640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb194103c00 0x7fb19419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.424+0000 7fb19bd29640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb194103240 0x7fb194198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.424+0000 7fb19bd29640 1 --2- 172.21.15.50:0/684696167 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb194100270 0x7fb194198220 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.424+0000 7fb19bd29640 1 -- 172.21.15.50:0/684696167 >> 172.21.15.50:0/684696167 conn(0x7fb1940fbbc0 msgr2=0x7fb1941066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.424+0000 7fb19bd29640 1 -- 172.21.15.50:0/684696167 shutdown_connections 2024-04-23T18:16:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:18.424+0000 7fb19bd29640 1 -- 172.21.15.50:0/684696167 wait complete. 2024-04-23T18:16:18.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:18 smithi050 ceph-mon[29187]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:18.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:18.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:18.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:18.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:18.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:18.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:18 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:19.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:19 smithi188 ceph-mon[32694]: Deploying daemon mds.i on smithi188 2024-04-23T18:16:19.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:19 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6836/1048893636,v1:172.21.15.177:6837/1048893636] up:boot 2024-04-23T18:16:19.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:19 smithi188 ceph-mon[32694]: fsmap 8 up:standby 2024-04-23T18:16:19.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:19 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:16:19.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:19 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/684696167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:19 smithi050 ceph-mon[29187]: Deploying daemon mds.i on smithi188 2024-04-23T18:16:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:19 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6836/1048893636,v1:172.21.15.177:6837/1048893636] up:boot 2024-04-23T18:16:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:19 smithi050 ceph-mon[29187]: fsmap 8 up:standby 2024-04-23T18:16:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:19 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:16:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:19 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/684696167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:19.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:19 smithi177 ceph-mon[32308]: Deploying daemon mds.i on smithi188 2024-04-23T18:16:19.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:19 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6836/1048893636,v1:172.21.15.177:6837/1048893636] up:boot 2024-04-23T18:16:19.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:19 smithi177 ceph-mon[32308]: fsmap 8 up:standby 2024-04-23T18:16:19.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:19 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:16:19.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:19 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/684696167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:20.053 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:16:19 smithi188 systemd[1]: Starting Ceph mds.i for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:16:20.318 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:20 smithi188 ceph-mon[32694]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:20.319 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:16:20 smithi188 podman[64005]: 2024-04-23 18:16:20.072136963 +0000 UTC m=+0.019730772 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:16:20.319 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:16:20 smithi188 podman[64005]: 2024-04-23 18:16:20.20554909 +0000 UTC m=+0.153142889 container create d24dc11dcb0a5663065f53b9a9bbeead0e14e63ce4d3ac39bab1909ea3fca65a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-23T18:16:20.346 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:20 smithi050 ceph-mon[29187]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:20.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:20.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:20 smithi177 ceph-mon[32308]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:20.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.554+0000 7fcbddce1640 1 -- 172.21.15.50:0/434405203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcbd8101fe0 msgr2=0x7fcbd8102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.554+0000 7fcbddce1640 1 --2- 172.21.15.50:0/434405203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcbd8101fe0 0x7fcbd8102460 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009900 tx=0x7fcbcc02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.554+0000 7fcbddce1640 1 -- 172.21.15.50:0/434405203 shutdown_connections 2024-04-23T18:16:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.554+0000 7fcbddce1640 1 --2- 172.21.15.50:0/434405203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbd81029a0 0x7fcbd81092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.554+0000 7fcbddce1640 1 --2- 172.21.15.50:0/434405203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcbd8101fe0 0x7fcbd8102460 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.554+0000 7fcbddce1640 1 --2- 172.21.15.50:0/434405203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd8100e90 0x7fcbd81012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.554+0000 7fcbddce1640 1 -- 172.21.15.50:0/434405203 >> 172.21.15.50:0/434405203 conn(0x7fcbd80fc400 msgr2=0x7fcbd80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.554+0000 7fcbddce1640 1 -- 172.21.15.50:0/434405203 shutdown_connections 2024-04-23T18:16:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.554+0000 7fcbddce1640 1 -- 172.21.15.50:0/434405203 wait complete. 2024-04-23T18:16:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.555+0000 7fcbddce1640 1 Processor -- start 2024-04-23T18:16:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.555+0000 7fcbddce1640 1 -- start start 2024-04-23T18:16:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.555+0000 7fcbddce1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcbd8100e90 0x7fcbd8198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.555+0000 7fcbddce1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbd8101fe0 0x7fcbd81987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.555+0000 7fcbddce1640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd81029a0 0x7fcbd819f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.555+0000 7fcbddce1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd810c9f0 con 0x7fcbd8100e90 2024-04-23T18:16:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.555+0000 7fcbddce1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd810c870 con 0x7fcbd8101fe0 2024-04-23T18:16:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.555+0000 7fcbddce1640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd810cb70 con 0x7fcbd81029a0 2024-04-23T18:16:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.556+0000 7fcbd77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcbd8100e90 0x7fcbd8198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.556+0000 7fcbd7fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd81029a0 0x7fcbd819f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.556+0000 7fcbd77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcbd8100e90 0x7fcbd8198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34308/0 (socket says 172.21.15.50:34308) 2024-04-23T18:16:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.556+0000 7fcbd77fe640 1 -- 172.21.15.50:0/1663104195 learned_addr learned my addr 172.21.15.50:0/1663104195 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.556+0000 7fcbd6ffd640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbd8101fe0 0x7fcbd81987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.556+0000 7fcbd7fff640 1 -- 172.21.15.50:0/1663104195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbd8101fe0 msgr2=0x7fcbd81987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.556+0000 7fcbd7fff640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbd8101fe0 0x7fcbd81987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.556+0000 7fcbd7fff640 1 -- 172.21.15.50:0/1663104195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcbd8100e90 msgr2=0x7fcbd8198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.556+0000 7fcbd7fff640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcbd8100e90 0x7fcbd8198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.556+0000 7fcbd7fff640 1 -- 172.21.15.50:0/1663104195 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd819fea0 con 0x7fcbd81029a0 2024-04-23T18:16:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.556+0000 7fcbd77fe640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcbd8100e90 0x7fcbd8198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.556+0000 7fcbd7fff640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd81029a0 0x7fcbd819f830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000e970 tx=0x7fcbd000ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.559+0000 7fcbd4ff9640 1 -- 172.21.15.50:0/1663104195 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd000ccc0 con 0x7fcbd81029a0 2024-04-23T18:16:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.559+0000 7fcbd4ff9640 1 -- 172.21.15.50:0/1663104195 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbd0005020 con 0x7fcbd81029a0 2024-04-23T18:16:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.559+0000 7fcbd4ff9640 1 -- 172.21.15.50:0/1663104195 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0004a00 con 0x7fcbd81029a0 2024-04-23T18:16:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.559+0000 7fcbddce1640 1 -- 172.21.15.50:0/1663104195 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd81a0190 con 0x7fcbd81029a0 2024-04-23T18:16:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.559+0000 7fcbddce1640 1 -- 172.21.15.50:0/1663104195 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd81a06b0 con 0x7fcbd81029a0 2024-04-23T18:16:20.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.560+0000 7fcbddce1640 1 -- 172.21.15.50:0/1663104195 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbd81a0970 con 0x7fcbd81029a0 2024-04-23T18:16:20.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.561+0000 7fcbd4ff9640 1 -- 172.21.15.50:0/1663104195 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcbd0004ba0 con 0x7fcbd81029a0 2024-04-23T18:16:20.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.562+0000 7fcbd4ff9640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcbc00a3c30 0x7fcbc00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:20.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.562+0000 7fcbd77fe640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcbc00a3c30 0x7fcbc00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:20.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.563+0000 7fcbd77fe640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcbc00a3c30 0x7fcbc00a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80049d0 tx=0x7fcbc8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:20.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.563+0000 7fcbd4ff9640 1 -- 172.21.15.50:0/1663104195 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fcbd00cdf20 con 0x7fcbd81029a0 2024-04-23T18:16:20.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.568+0000 7fcbd4ff9640 1 -- 172.21.15.50:0/1663104195 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcbd009ccf0 con 0x7fcbd81029a0 2024-04-23T18:16:20.580 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:16:20 smithi188 podman[64005]: 2024-04-23 18:16:20.433113793 +0000 UTC m=+0.380707592 container init d24dc11dcb0a5663065f53b9a9bbeead0e14e63ce4d3ac39bab1909ea3fca65a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-23T18:16:20.580 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:16:20 smithi188 podman[64005]: 2024-04-23 18:16:20.435625868 +0000 UTC m=+0.383219668 container start d24dc11dcb0a5663065f53b9a9bbeead0e14e63ce4d3ac39bab1909ea3fca65a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T18:16:20.580 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:16:20 smithi188 ceph-mds[64024]: -- 172.21.15.188:0/2955343960 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5620e34c1a40 con 0x5620e3494c00 2024-04-23T18:16:20.580 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:16:20 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-i[64020]: starting mds.i at 2024-04-23T18:16:20.580 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:16:20 smithi188 bash[64005]: d24dc11dcb0a5663065f53b9a9bbeead0e14e63ce4d3ac39bab1909ea3fca65a 2024-04-23T18:16:20.580 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:16:20 smithi188 systemd[1]: Started Ceph mds.i for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:16:20.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.805+0000 7fcbddce1640 1 -- 172.21.15.50:0/1663104195 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbd81a0be0 con 0x7fcbd81029a0 2024-04-23T18:16:20.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.806+0000 7fcbd4ff9640 1 -- 172.21.15.50:0/1663104195 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7fcbd009c5f0 con 0x7fcbd81029a0 2024-04-23T18:16:20.800 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:20.800 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-23T18:16:20.800 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 9 2024-04-23T18:16:20.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.808+0000 7fcbddce1640 1 -- 172.21.15.50:0/1663104195 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcbc00a3c30 msgr2=0x7fcbc00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.808+0000 7fcbddce1640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcbc00a3c30 0x7fcbc00a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80049d0 tx=0x7fcbc8009500 comp rx=0 tx=0).stop 2024-04-23T18:16:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.808+0000 7fcbddce1640 1 -- 172.21.15.50:0/1663104195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd81029a0 msgr2=0x7fcbd819f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.808+0000 7fcbddce1640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd81029a0 0x7fcbd819f830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000e970 tx=0x7fcbd000ee30 comp rx=0 tx=0).stop 2024-04-23T18:16:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.808+0000 7fcbddce1640 1 -- 172.21.15.50:0/1663104195 shutdown_connections 2024-04-23T18:16:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.808+0000 7fcbddce1640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcbc00a3c30 0x7fcbc00a60f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.808+0000 7fcbddce1640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd81029a0 0x7fcbd819f830 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.808+0000 7fcbddce1640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbd8101fe0 0x7fcbd81987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.808+0000 7fcbddce1640 1 --2- 172.21.15.50:0/1663104195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcbd8100e90 0x7fcbd8198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.808+0000 7fcbddce1640 1 -- 172.21.15.50:0/1663104195 >> 172.21.15.50:0/1663104195 conn(0x7fcbd80fc400 msgr2=0x7fcbd80fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.808+0000 7fcbddce1640 1 -- 172.21.15.50:0/1663104195 shutdown_connections 2024-04-23T18:16:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:20.808+0000 7fcbddce1640 1 -- 172.21.15.50:0/1663104195 wait complete. 2024-04-23T18:16:21.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:21 smithi188 ceph-mon[32694]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:21.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:21.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:21.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:21.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:21.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:21.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:21.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:21 smithi188 ceph-mon[32694]: Deploying daemon mds.g on smithi050 2024-04-23T18:16:21.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:21 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1663104195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:21.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:21 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] up:boot 2024-04-23T18:16:21.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:21 smithi188 ceph-mon[32694]: fsmap 9 up:standby 2024-04-23T18:16:21.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:21 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:16:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:21 smithi050 ceph-mon[29187]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:21 smithi050 ceph-mon[29187]: Deploying daemon mds.g on smithi050 2024-04-23T18:16:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:21 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1663104195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:21.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:21 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] up:boot 2024-04-23T18:16:21.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:21 smithi050 ceph-mon[29187]: fsmap 9 up:standby 2024-04-23T18:16:21.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:21 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:16:22.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:21 smithi177 ceph-mon[32308]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:22.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:22.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:22.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:22.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:22.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:22.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:22.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:21 smithi177 ceph-mon[32308]: Deploying daemon mds.g on smithi050 2024-04-23T18:16:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:21 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1663104195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:21 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] up:boot 2024-04-23T18:16:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:21 smithi177 ceph-mon[32308]: fsmap 9 up:standby 2024-04-23T18:16:22.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:21 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:16:22.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:22.841 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:16:22 smithi050 systemd[1]: Starting Ceph mds.g for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:16:22.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.931+0000 7fc38f1c6640 1 -- 172.21.15.50:0/4067647966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc388102a00 msgr2=0x7fc388109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:22.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.931+0000 7fc38f1c6640 1 --2- 172.21.15.50:0/4067647966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc388102a00 0x7fc388109320 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc380009ca0 tx=0x7fc38002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:22.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.932+0000 7fc38f1c6640 1 -- 172.21.15.50:0/4067647966 shutdown_connections 2024-04-23T18:16:22.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.932+0000 7fc38f1c6640 1 --2- 172.21.15.50:0/4067647966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc388102a00 0x7fc388109320 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:22.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.932+0000 7fc38f1c6640 1 --2- 172.21.15.50:0/4067647966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc388102040 0x7fc3881024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:22.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.932+0000 7fc38f1c6640 1 --2- 172.21.15.50:0/4067647966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc388100e60 0x7fc388101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:22.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.932+0000 7fc38f1c6640 1 -- 172.21.15.50:0/4067647966 >> 172.21.15.50:0/4067647966 conn(0x7fc3880fc440 msgr2=0x7fc3880fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:22.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.932+0000 7fc38f1c6640 1 -- 172.21.15.50:0/4067647966 shutdown_connections 2024-04-23T18:16:22.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.932+0000 7fc38f1c6640 1 -- 172.21.15.50:0/4067647966 wait complete. 2024-04-23T18:16:22.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.932+0000 7fc38f1c6640 1 Processor -- start 2024-04-23T18:16:22.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.932+0000 7fc38f1c6640 1 -- start start 2024-04-23T18:16:22.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc38f1c6640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc388100e60 0x7fc388104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:22.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc38f1c6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc388102040 0x7fc3881034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc38f1c6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc388102a00 0x7fc3881039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc38f1c6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc38810ca40 con 0x7fc388102040 2024-04-23T18:16:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc38f1c6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc38810c8c0 con 0x7fc388102a00 2024-04-23T18:16:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc38f1c6640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc38810cbc0 con 0x7fc388100e60 2024-04-23T18:16:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc387fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc388102040 0x7fc3881034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc387fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc388102040 0x7fc3881034b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46618/0 (socket says 172.21.15.50:46618) 2024-04-23T18:16:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc387fff640 1 -- 172.21.15.50:0/1414301797 learned_addr learned my addr 172.21.15.50:0/1414301797 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc38cf62640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc388100e60 0x7fc388104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc38d763640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc388102a00 0x7fc3881039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc387fff640 1 -- 172.21.15.50:0/1414301797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc388100e60 msgr2=0x7fc388104e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc387fff640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc388100e60 0x7fc388104e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc387fff640 1 -- 172.21.15.50:0/1414301797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc388102a00 msgr2=0x7fc3881039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:22.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc387fff640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc388102a00 0x7fc3881039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:22.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc387fff640 1 -- 172.21.15.50:0/1414301797 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc388104280 con 0x7fc388102040 2024-04-23T18:16:22.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc38d763640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc388102a00 0x7fc3881039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:22.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc387fff640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc388102040 0x7fc3881034b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc37c00d950 tx=0x7fc37c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:22.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc385ffb640 1 -- 172.21.15.50:0/1414301797 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c017030 con 0x7fc388102040 2024-04-23T18:16:22.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc385ffb640 1 -- 172.21.15.50:0/1414301797 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc37c004020 con 0x7fc388102040 2024-04-23T18:16:22.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc38f1c6640 1 -- 172.21.15.50:0/1414301797 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3881a2280 con 0x7fc388102040 2024-04-23T18:16:22.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.933+0000 7fc385ffb640 1 -- 172.21.15.50:0/1414301797 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c0047a0 con 0x7fc388102040 2024-04-23T18:16:22.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.934+0000 7fc38f1c6640 1 -- 172.21.15.50:0/1414301797 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3881a2820 con 0x7fc388102040 2024-04-23T18:16:22.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.935+0000 7fc38f1c6640 1 -- 172.21.15.50:0/1414301797 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3881a2b40 con 0x7fc388102040 2024-04-23T18:16:22.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.936+0000 7fc385ffb640 1 -- 172.21.15.50:0/1414301797 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc37c009a30 con 0x7fc388102040 2024-04-23T18:16:22.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.936+0000 7fc385ffb640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3700a3c30 0x7fc3700a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:22.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.937+0000 7fc38cf62640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3700a3c30 0x7fc3700a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:22.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.937+0000 7fc385ffb640 1 -- 172.21.15.50:0/1414301797 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc37c0cf990 con 0x7fc388102040 2024-04-23T18:16:22.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.937+0000 7fc38cf62640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3700a3c30 0x7fc3700a60f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc388101bb0 tx=0x7fc37800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:22.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:22.941+0000 7fc385ffb640 1 -- 172.21.15.50:0/1414301797 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc37c09c810 con 0x7fc388102040 2024-04-23T18:16:23.093 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:16:22 smithi050 podman[68349]: 2024-04-23 18:16:22.851410084 +0000 UTC m=+0.020010225 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:16:23.093 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:16:22 smithi050 podman[68349]: 2024-04-23 18:16:22.981306392 +0000 UTC m=+0.149906522 container create ba154573e0276b92162134c9b40df7dc9658ddf4625d57df92f42b7eb2418226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0) 2024-04-23T18:16:23.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.169+0000 7fc38f1c6640 1 -- 172.21.15.50:0/1414301797 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3881a2d10 con 0x7fc388102040 2024-04-23T18:16:23.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.170+0000 7fc385ffb640 1 -- 172.21.15.50:0/1414301797 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7fc37c09c110 con 0x7fc388102040 2024-04-23T18:16:23.164 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:23.164 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-23T18:16:23.164 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 10 2024-04-23T18:16:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.172+0000 7fc38f1c6640 1 -- 172.21.15.50:0/1414301797 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3700a3c30 msgr2=0x7fc3700a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.172+0000 7fc38f1c6640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3700a3c30 0x7fc3700a60f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc388101bb0 tx=0x7fc37800b390 comp rx=0 tx=0).stop 2024-04-23T18:16:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.172+0000 7fc38f1c6640 1 -- 172.21.15.50:0/1414301797 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc388102040 msgr2=0x7fc3881034b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.172+0000 7fc38f1c6640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc388102040 0x7fc3881034b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc37c00d950 tx=0x7fc37c00de10 comp rx=0 tx=0).stop 2024-04-23T18:16:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.172+0000 7fc38f1c6640 1 -- 172.21.15.50:0/1414301797 shutdown_connections 2024-04-23T18:16:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.172+0000 7fc38f1c6640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc3700a3c30 0x7fc3700a60f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.172+0000 7fc38f1c6640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc388102a00 0x7fc3881039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.172+0000 7fc38f1c6640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc388102040 0x7fc3881034b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.172+0000 7fc38f1c6640 1 --2- 172.21.15.50:0/1414301797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc388100e60 0x7fc388104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:23.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.172+0000 7fc38f1c6640 1 -- 172.21.15.50:0/1414301797 >> 172.21.15.50:0/1414301797 conn(0x7fc3880fc440 msgr2=0x7fc3880fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:23.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.172+0000 7fc38f1c6640 1 -- 172.21.15.50:0/1414301797 shutdown_connections 2024-04-23T18:16:23.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:23.172+0000 7fc38f1c6640 1 -- 172.21.15.50:0/1414301797 wait complete. 2024-04-23T18:16:23.446 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:16:23 smithi050 podman[68349]: 2024-04-23 18:16:23.200474499 +0000 UTC m=+0.369074629 container init ba154573e0276b92162134c9b40df7dc9658ddf4625d57df92f42b7eb2418226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:16:23.446 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:16:23 smithi050 podman[68349]: 2024-04-23 18:16:23.20327894 +0000 UTC m=+0.371879077 container start ba154573e0276b92162134c9b40df7dc9658ddf4625d57df92f42b7eb2418226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-23T18:16:23.447 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:16:23 smithi050 ceph-mds[68409]: -- 172.21.15.50:0/2832935202 <== mon.0 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56393ef07a40 con 0x56393eedb000 2024-04-23T18:16:23.447 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:16:23 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-g[68403]: starting mds.g at 2024-04-23T18:16:23.447 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:16:23 smithi050 bash[68349]: ba154573e0276b92162134c9b40df7dc9658ddf4625d57df92f42b7eb2418226 2024-04-23T18:16:23.447 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:16:23 smithi050 systemd[1]: Started Ceph mds.g for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:16:23.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:23 smithi188 ceph-mon[32694]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:23.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:23 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1414301797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:23.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:23 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:23.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:23 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:23.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:23 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:23.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:23 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:23.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:23 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:23.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:23 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:23.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:23 smithi050 ceph-mon[29187]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:23.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:23 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1414301797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:23.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:23 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:23.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:23 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:23.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:23 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:23.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:23 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:23.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:23 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:23.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:23 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:24.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:23 smithi177 ceph-mon[32308]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:24.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:23 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1414301797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:24.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:23 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:24.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:23 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:24.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:23 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:24.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:23 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:24.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:23 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:24.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:23 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:24.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:24 smithi050 ceph-mon[29187]: Deploying daemon mds.a on smithi050 2024-04-23T18:16:24.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:24 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.50:6838/2730935833,v1:172.21.15.50:6839/2730935833] up:boot 2024-04-23T18:16:24.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:24 smithi050 ceph-mon[29187]: fsmap 10 up:standby 2024-04-23T18:16:24.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:24 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:16:24.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:24 smithi188 ceph-mon[32694]: Deploying daemon mds.a on smithi050 2024-04-23T18:16:24.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:24 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.50:6838/2730935833,v1:172.21.15.50:6839/2730935833] up:boot 2024-04-23T18:16:24.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:24 smithi188 ceph-mon[32694]: fsmap 10 up:standby 2024-04-23T18:16:24.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:24 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:16:25.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:24 smithi177 ceph-mon[32308]: Deploying daemon mds.a on smithi050 2024-04-23T18:16:25.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:24 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.50:6838/2730935833,v1:172.21.15.50:6839/2730935833] up:boot 2024-04-23T18:16:25.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:24 smithi177 ceph-mon[32308]: fsmap 10 up:standby 2024-04-23T18:16:25.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:24 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:16:25.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.330+0000 7f49d6043640 1 -- 172.21.15.50:0/630181685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0102a00 msgr2=0x7f49d0109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.330+0000 7f49d6043640 1 --2- 172.21.15.50:0/630181685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0102a00 0x7f49d0109320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009ca0 tx=0x7f49c802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.330+0000 7f49d6043640 1 -- 172.21.15.50:0/630181685 shutdown_connections 2024-04-23T18:16:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.330+0000 7f49d6043640 1 --2- 172.21.15.50:0/630181685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0102a00 0x7f49d0109320 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.330+0000 7f49d6043640 1 --2- 172.21.15.50:0/630181685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d0102040 0x7f49d01024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.330+0000 7f49d6043640 1 --2- 172.21.15.50:0/630181685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49d0100e60 0x7f49d0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.330+0000 7f49d6043640 1 -- 172.21.15.50:0/630181685 >> 172.21.15.50:0/630181685 conn(0x7f49d00fc440 msgr2=0x7f49d00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.330+0000 7f49d6043640 1 -- 172.21.15.50:0/630181685 shutdown_connections 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.330+0000 7f49d6043640 1 -- 172.21.15.50:0/630181685 wait complete. 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49d6043640 1 Processor -- start 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49d6043640 1 -- start start 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49d6043640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d0100e60 0x7f49d006b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49d6043640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49d0102040 0x7f49d019ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49d6043640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0102a00 0x7f49d019f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49d6043640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f49d010ca10 con 0x7f49d0102a00 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49d6043640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f49d010c890 con 0x7f49d0100e60 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49d6043640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f49d010cb90 con 0x7f49d0102040 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49cffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0102a00 0x7f49d019f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49cffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0102a00 0x7f49d019f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46644/0 (socket says 172.21.15.50:46644) 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49cffff640 1 -- 172.21.15.50:0/768385388 learned_addr learned my addr 172.21.15.50:0/768385388 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49c6ffd640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49d0102040 0x7f49d019ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.331+0000 7f49cf7fe640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d0100e60 0x7f49d006b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.332+0000 7f49cffff640 1 -- 172.21.15.50:0/768385388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49d0102040 msgr2=0x7f49d019ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.332+0000 7f49cffff640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49d0102040 0x7f49d019ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.332+0000 7f49cffff640 1 -- 172.21.15.50:0/768385388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d0100e60 msgr2=0x7f49d006b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.332+0000 7f49cffff640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d0100e60 0x7f49d006b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.332+0000 7f49cffff640 1 -- 172.21.15.50:0/768385388 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d019fe00 con 0x7f49d0102a00 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.332+0000 7f49cf7fe640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d0100e60 0x7f49d006b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.332+0000 7f49cffff640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0102a00 0x7f49d019f4f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f49c80097b0 tx=0x7f49c8031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.332+0000 7f49cd7fa640 1 -- 172.21.15.50:0/768385388 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c8003bf0 con 0x7f49d0102a00 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.332+0000 7f49d6043640 1 -- 172.21.15.50:0/768385388 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d01a0030 con 0x7f49d0102a00 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.332+0000 7f49d6043640 1 -- 172.21.15.50:0/768385388 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d01a05d0 con 0x7f49d0102a00 2024-04-23T18:16:25.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.332+0000 7f49cd7fa640 1 -- 172.21.15.50:0/768385388 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49c8003d90 con 0x7f49d0102a00 2024-04-23T18:16:25.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.332+0000 7f49cd7fa640 1 -- 172.21.15.50:0/768385388 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c802fc70 con 0x7f49d0102a00 2024-04-23T18:16:25.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.333+0000 7f49d6043640 1 -- 172.21.15.50:0/768385388 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49d00691c0 con 0x7f49d0102a00 2024-04-23T18:16:25.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.334+0000 7f49cd7fa640 1 -- 172.21.15.50:0/768385388 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f49c8004040 con 0x7f49d0102a00 2024-04-23T18:16:25.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.335+0000 7f49cd7fa640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f49bc0a3bb0 0x7f49bc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:25.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.335+0000 7f49cf7fe640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f49bc0a3bb0 0x7f49bc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:25.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.335+0000 7f49cd7fa640 1 -- 172.21.15.50:0/768385388 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f49c80f1780 con 0x7f49d0102a00 2024-04-23T18:16:25.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.335+0000 7f49cf7fe640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f49bc0a3bb0 0x7f49bc0a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f49c00058c0 tx=0x7f49c0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:25.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.343+0000 7f49cd7fa640 1 -- 172.21.15.50:0/768385388 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f49c80be600 con 0x7f49d0102a00 2024-04-23T18:16:25.564 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:16:25 smithi050 systemd[1]: Starting Ceph mds.a for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:16:25.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.587+0000 7f49d6043640 1 -- 172.21.15.50:0/768385388 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49d0108200 con 0x7f49d0102a00 2024-04-23T18:16:25.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.588+0000 7f49cd7fa640 1 -- 172.21.15.50:0/768385388 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f49c80bdf00 con 0x7f49d0102a00 2024-04-23T18:16:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:25.582 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-23T18:16:25.583 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 11 2024-04-23T18:16:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.590+0000 7f49d6043640 1 -- 172.21.15.50:0/768385388 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f49bc0a3bb0 msgr2=0x7f49bc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.590+0000 7f49d6043640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f49bc0a3bb0 0x7f49bc0a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f49c00058c0 tx=0x7f49c0005850 comp rx=0 tx=0).stop 2024-04-23T18:16:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.590+0000 7f49d6043640 1 -- 172.21.15.50:0/768385388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0102a00 msgr2=0x7f49d019f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.590+0000 7f49d6043640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0102a00 0x7f49d019f4f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f49c80097b0 tx=0x7f49c8031840 comp rx=0 tx=0).stop 2024-04-23T18:16:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.590+0000 7f49d6043640 1 -- 172.21.15.50:0/768385388 shutdown_connections 2024-04-23T18:16:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.591+0000 7f49d6043640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f49bc0a3bb0 0x7f49bc0a6070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.591+0000 7f49d6043640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0102a00 0x7f49d019f4f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.591+0000 7f49d6043640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49d0102040 0x7f49d019ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.591+0000 7f49d6043640 1 --2- 172.21.15.50:0/768385388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d0100e60 0x7f49d006b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.591+0000 7f49d6043640 1 -- 172.21.15.50:0/768385388 >> 172.21.15.50:0/768385388 conn(0x7f49d00fc440 msgr2=0x7f49d01053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.591+0000 7f49d6043640 1 -- 172.21.15.50:0/768385388 shutdown_connections 2024-04-23T18:16:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:25.591+0000 7f49d6043640 1 -- 172.21.15.50:0/768385388 wait complete. 2024-04-23T18:16:25.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:25 smithi050 ceph-mon[29187]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:25.817 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:16:25 smithi050 podman[68738]: 2024-04-23 18:16:25.705551062 +0000 UTC m=+0.115697139 container create 48c5cf119a73dc3c8f3638e74ecdee1e39f1fa9ab781aad568100372ca0501ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-23T18:16:25.817 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:16:25 smithi050 podman[68738]: 2024-04-23 18:16:25.609288441 +0000 UTC m=+0.019434462 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:16:25.829 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:25 smithi188 ceph-mon[32694]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:26.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:25 smithi177 ceph-mon[32308]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:26.195 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:16:25 smithi050 podman[68738]: 2024-04-23 18:16:25.941619501 +0000 UTC m=+0.351765520 container init 48c5cf119a73dc3c8f3638e74ecdee1e39f1fa9ab781aad568100372ca0501ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-23T18:16:26.195 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:16:25 smithi050 podman[68738]: 2024-04-23 18:16:25.944258972 +0000 UTC m=+0.354404993 container start 48c5cf119a73dc3c8f3638e74ecdee1e39f1fa9ab781aad568100372ca0501ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-23T18:16:26.195 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:16:25 smithi050 ceph-mds[68759]: -- 172.21.15.50:0/1628785454 <== mon.0 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5561e37e5a40 con 0x5561e37b8800 2024-04-23T18:16:26.195 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:16:25 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-a[68755]: starting mds.a at 2024-04-23T18:16:26.195 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:16:26 smithi050 bash[68738]: 48c5cf119a73dc3c8f3638e74ecdee1e39f1fa9ab781aad568100372ca0501ce 2024-04-23T18:16:26.195 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:16:26 smithi050 systemd[1]: Started Ceph mds.a for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:16:26.826 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:26 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/768385388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:26.826 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:26.827 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:26.827 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:26.827 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:26.827 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:26.827 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:26 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:26.827 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:26 smithi177 ceph-mon[32308]: Deploying daemon mds.b on smithi177 2024-04-23T18:16:26.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:26 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/768385388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:26.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:26.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:26.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:26.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:26.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:26.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:26 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:26.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:26 smithi188 ceph-mon[32694]: Deploying daemon mds.b on smithi177 2024-04-23T18:16:26.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:26 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/768385388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:26.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:26.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:26.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:26.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T18:16:26.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T18:16:26.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:26 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:26.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:26 smithi050 ceph-mon[29187]: Deploying daemon mds.b on smithi177 2024-04-23T18:16:27.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.722+0000 7f250342a640 1 -- 172.21.15.50:0/3924043800 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24fc103b70 msgr2=0x7f24fc10a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.722+0000 7f250342a640 1 --2- 172.21.15.50:0/3924043800 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24fc103b70 0x7f24fc10a400 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f24f4009ca0 tx=0x7f24f402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.722+0000 7f250342a640 1 -- 172.21.15.50:0/3924043800 shutdown_connections 2024-04-23T18:16:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.722+0000 7f250342a640 1 --2- 172.21.15.50:0/3924043800 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24fc103b70 0x7f24fc10a400 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.722+0000 7f250342a640 1 --2- 172.21.15.50:0/3924043800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24fc1031b0 0x7f24fc103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.722+0000 7f250342a640 1 --2- 172.21.15.50:0/3924043800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24fc101fc0 0x7f24fc1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.722+0000 7f250342a640 1 -- 172.21.15.50:0/3924043800 >> 172.21.15.50:0/3924043800 conn(0x7f24fc0fd550 msgr2=0x7f24fc0ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.722+0000 7f250342a640 1 -- 172.21.15.50:0/3924043800 shutdown_connections 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.722+0000 7f250342a640 1 -- 172.21.15.50:0/3924043800 wait complete. 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.722+0000 7f250342a640 1 Processor -- start 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.722+0000 7f250342a640 1 -- start start 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f250342a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24fc101fc0 0x7f24fc198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f250342a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24fc1031b0 0x7f24fc1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f250342a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24fc103b70 0x7f24fc19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f250342a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f24fc10db00 con 0x7f24fc103b70 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f250342a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f24fc10d980 con 0x7f24fc101fc0 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f250342a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f24fc10dc80 con 0x7f24fc1031b0 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f25019c7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24fc103b70 0x7f24fc19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f25019c7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24fc103b70 0x7f24fc19f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46682/0 (socket says 172.21.15.50:46682) 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f25019c7640 1 -- 172.21.15.50:0/895809555 learned_addr learned my addr 172.21.15.50:0/895809555 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f25011c6640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24fc101fc0 0x7f24fc198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f25009c5640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24fc1031b0 0x7f24fc1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f25019c7640 1 -- 172.21.15.50:0/895809555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24fc1031b0 msgr2=0x7f24fc1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f25019c7640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24fc1031b0 0x7f24fc1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f25019c7640 1 -- 172.21.15.50:0/895809555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24fc101fc0 msgr2=0x7f24fc198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f25019c7640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24fc101fc0 0x7f24fc198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f25019c7640 1 -- 172.21.15.50:0/895809555 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24fc19fd90 con 0x7f24fc103b70 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f25011c6640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24fc101fc0 0x7f24fc198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f25019c7640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24fc103b70 0x7f24fc19f720 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f24f40097b0 tx=0x7f24f4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f24f27fc640 1 -- 172.21.15.50:0/895809555 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f4004240 con 0x7f24fc103b70 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f24f27fc640 1 -- 172.21.15.50:0/895809555 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24f40317c0 con 0x7f24fc103b70 2024-04-23T18:16:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f24f27fc640 1 -- 172.21.15.50:0/895809555 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f402fd80 con 0x7f24fc103b70 2024-04-23T18:16:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.723+0000 7f250342a640 1 -- 172.21.15.50:0/895809555 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24fc1a0020 con 0x7f24fc103b70 2024-04-23T18:16:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.724+0000 7f250342a640 1 -- 172.21.15.50:0/895809555 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24fc1a0570 con 0x7f24fc103b70 2024-04-23T18:16:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.724+0000 7f250342a640 1 -- 172.21.15.50:0/895809555 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24fc109380 con 0x7f24fc103b70 2024-04-23T18:16:27.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.726+0000 7f24f27fc640 1 -- 172.21.15.50:0/895809555 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f24f40043e0 con 0x7f24fc103b70 2024-04-23T18:16:27.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.726+0000 7f24f27fc640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f24e80a3bb0 0x7f24e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:27.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.726+0000 7f25011c6640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f24e80a3bb0 0x7f24e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:27.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.726+0000 7f24f27fc640 1 -- 172.21.15.50:0/895809555 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f24f40f1ac0 con 0x7f24fc103b70 2024-04-23T18:16:27.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.727+0000 7f25011c6640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f24e80a3bb0 0x7f24e80a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f24f80058c0 tx=0x7f24f8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:27.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.730+0000 7f24f27fc640 1 -- 172.21.15.50:0/895809555 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f24f40bea60 con 0x7f24fc103b70 2024-04-23T18:16:27.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:27 smithi188 ceph-mon[32694]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:27.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:27 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.50:6840/3198542689,v1:172.21.15.50:6841/3198542689] up:boot 2024-04-23T18:16:27.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:27 smithi188 ceph-mon[32694]: fsmap 11 up:standby 2024-04-23T18:16:27.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:27 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:16:27.887 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:27 smithi177 ceph-mon[32308]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:27.887 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:27 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.50:6840/3198542689,v1:172.21.15.50:6841/3198542689] up:boot 2024-04-23T18:16:27.887 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:27 smithi177 ceph-mon[32308]: fsmap 11 up:standby 2024-04-23T18:16:27.887 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:27 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:16:27.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:27 smithi050 ceph-mon[29187]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:27.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:27 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.50:6840/3198542689,v1:172.21.15.50:6841/3198542689] up:boot 2024-04-23T18:16:27.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:27 smithi050 ceph-mon[29187]: fsmap 11 up:standby 2024-04-23T18:16:27.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:27 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:16:27.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.960+0000 7f250342a640 1 -- 172.21.15.50:0/895809555 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24fc02d5c0 con 0x7f24fc103b70 2024-04-23T18:16:27.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.960+0000 7f24f27fc640 1 -- 172.21.15.50:0/895809555 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9659 (secure 0 0 0) 0x7f24f40be360 con 0x7f24fc103b70 2024-04-23T18:16:27.954 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:27.955 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-23T18:16:27.955 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 12 2024-04-23T18:16:27.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.963+0000 7f250342a640 1 -- 172.21.15.50:0/895809555 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f24e80a3bb0 msgr2=0x7f24e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:27.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.963+0000 7f250342a640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f24e80a3bb0 0x7f24e80a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f24f80058c0 tx=0x7f24f8005850 comp rx=0 tx=0).stop 2024-04-23T18:16:27.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.963+0000 7f250342a640 1 -- 172.21.15.50:0/895809555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24fc103b70 msgr2=0x7f24fc19f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:27.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.963+0000 7f250342a640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24fc103b70 0x7f24fc19f720 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f24f40097b0 tx=0x7f24f4003c20 comp rx=0 tx=0).stop 2024-04-23T18:16:27.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.963+0000 7f250342a640 1 -- 172.21.15.50:0/895809555 shutdown_connections 2024-04-23T18:16:27.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.963+0000 7f250342a640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f24e80a3bb0 0x7f24e80a6070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.963+0000 7f250342a640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24fc103b70 0x7f24fc19f720 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.963+0000 7f250342a640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24fc1031b0 0x7f24fc1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.963+0000 7f250342a640 1 --2- 172.21.15.50:0/895809555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24fc101fc0 0x7f24fc198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:27.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.963+0000 7f250342a640 1 -- 172.21.15.50:0/895809555 >> 172.21.15.50:0/895809555 conn(0x7f24fc0fd550 msgr2=0x7f24fc106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:27.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.963+0000 7f250342a640 1 -- 172.21.15.50:0/895809555 shutdown_connections 2024-04-23T18:16:27.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:27.963+0000 7f250342a640 1 -- 172.21.15.50:0/895809555 wait complete. 2024-04-23T18:16:28.464 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:16:28 smithi177 systemd[1]: Starting Ceph mds.b for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:16:28.763 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:28 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/895809555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:28.763 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:16:28 smithi177 podman[63009]: 2024-04-23 18:16:28.559951429 +0000 UTC m=+0.112817450 container create 87e740c0e35fe04487285f419c10974e84f25e1bb199cc30c738da247ba93f63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-23T18:16:28.764 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:16:28 smithi177 podman[63009]: 2024-04-23 18:16:28.472093405 +0000 UTC m=+0.024959396 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:16:28.764 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:16:28 smithi177 podman[63009]: 2024-04-23 18:16:28.770591505 +0000 UTC m=+0.323457483 container init 87e740c0e35fe04487285f419c10974e84f25e1bb199cc30c738da247ba93f63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD) 2024-04-23T18:16:28.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:28 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/895809555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:28.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:28 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/895809555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:29.038 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:16:28 smithi177 podman[63009]: 2024-04-23 18:16:28.773603516 +0000 UTC m=+0.326469495 container start 87e740c0e35fe04487285f419c10974e84f25e1bb199cc30c738da247ba93f63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-23T18:16:29.038 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:16:28 smithi177 ceph-mds[63028]: -- 172.21.15.177:0/2270680201 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5592169fba40 con 0x5592169cf000 2024-04-23T18:16:29.038 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:16:28 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b[63024]: starting mds.b at 2024-04-23T18:16:29.038 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:16:28 smithi177 bash[63009]: 87e740c0e35fe04487285f419c10974e84f25e1bb199cc30c738da247ba93f63 2024-04-23T18:16:29.038 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:16:28 smithi177 systemd[1]: Started Ceph mds.b for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:16:29.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:30.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.083+0000 7f10f6c23640 1 -- 172.21.15.50:0/1689817357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10f01029a0 msgr2=0x7f10f01092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.083+0000 7f10f6c23640 1 --2- 172.21.15.50:0/1689817357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10f01029a0 0x7f10f01092c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009ca0 tx=0x7f10e802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 -- 172.21.15.50:0/1689817357 shutdown_connections 2024-04-23T18:16:30.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 --2- 172.21.15.50:0/1689817357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10f01029a0 0x7f10f01092c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 --2- 172.21.15.50:0/1689817357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0101fe0 0x7f10f0102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 --2- 172.21.15.50:0/1689817357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10f0100e90 0x7f10f01012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 -- 172.21.15.50:0/1689817357 >> 172.21.15.50:0/1689817357 conn(0x7f10f00fc400 msgr2=0x7f10f00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 -- 172.21.15.50:0/1689817357 shutdown_connections 2024-04-23T18:16:30.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 -- 172.21.15.50:0/1689817357 wait complete. 2024-04-23T18:16:30.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 Processor -- start 2024-04-23T18:16:30.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 -- start start 2024-04-23T18:16:30.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0100e90 0x7f10f01981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10f0101fe0 0x7f10f0198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10f01029a0 0x7f10f019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f10f010c9f0 con 0x7f10f0101fe0 2024-04-23T18:16:30.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10f010c870 con 0x7f10f0100e90 2024-04-23T18:16:30.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.084+0000 7f10f6c23640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f10f010cb70 con 0x7f10f01029a0 2024-04-23T18:16:30.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10effff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10f0101fe0 0x7f10f0198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10effff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10f0101fe0 0x7f10f0198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46690/0 (socket says 172.21.15.50:46690) 2024-04-23T18:16:30.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10effff640 1 -- 172.21.15.50:0/4194277285 learned_addr learned my addr 172.21.15.50:0/4194277285 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:30.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10f49bf640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0100e90 0x7f10f01981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10f51c0640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10f01029a0 0x7f10f019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10effff640 1 -- 172.21.15.50:0/4194277285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10f01029a0 msgr2=0x7f10f019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10effff640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10f01029a0 0x7f10f019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10effff640 1 -- 172.21.15.50:0/4194277285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0100e90 msgr2=0x7f10f01981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10effff640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0100e90 0x7f10f01981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10effff640 1 -- 172.21.15.50:0/4194277285 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f019fde0 con 0x7f10f0101fe0 2024-04-23T18:16:30.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10f51c0640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10f01029a0 0x7f10f019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:30.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10effff640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10f0101fe0 0x7f10f0198700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00d770 tx=0x7f10dc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10edffb640 1 -- 172.21.15.50:0/4194277285 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc017070 con 0x7f10f0101fe0 2024-04-23T18:16:30.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10edffb640 1 -- 172.21.15.50:0/4194277285 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10dc009c30 con 0x7f10f0101fe0 2024-04-23T18:16:30.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10f6c23640 1 -- 172.21.15.50:0/4194277285 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f01a00d0 con 0x7f10f0101fe0 2024-04-23T18:16:30.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.085+0000 7f10edffb640 1 -- 172.21.15.50:0/4194277285 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc003cc0 con 0x7f10f0101fe0 2024-04-23T18:16:30.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.086+0000 7f10f6c23640 1 -- 172.21.15.50:0/4194277285 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f01a0620 con 0x7f10f0101fe0 2024-04-23T18:16:30.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.087+0000 7f10f6c23640 1 -- 172.21.15.50:0/4194277285 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10f01a0880 con 0x7f10f0101fe0 2024-04-23T18:16:30.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.088+0000 7f10edffb640 1 -- 172.21.15.50:0/4194277285 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f10dc003e60 con 0x7f10f0101fe0 2024-04-23T18:16:30.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.088+0000 7f10edffb640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f10e00a3bb0 0x7f10e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.088+0000 7f10f49bf640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f10e00a3bb0 0x7f10e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.089+0000 7f10edffb640 1 -- 172.21.15.50:0/4194277285 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f10dc0cf8f0 con 0x7f10f0101fe0 2024-04-23T18:16:30.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.089+0000 7f10f49bf640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f10e00a3bb0 0x7f10e00a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f10e4005fd0 tx=0x7f10e400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.093+0000 7f10edffb640 1 -- 172.21.15.50:0/4194277285 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f10dc09c770 con 0x7f10f0101fe0 2024-04-23T18:16:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:29 smithi050 ceph-mon[29187]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:29 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:29 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:29 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:29 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:29 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:29 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:16:30.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:29 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:29 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] up:boot 2024-04-23T18:16:30.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:29 smithi050 ceph-mon[29187]: fsmap 12 up:standby 2024-04-23T18:16:30.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:29 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:16:30.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:29 smithi177 ceph-mon[32308]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:30.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:29 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:29 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:29 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:29 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:29 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:29 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:16:30.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:29 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:29 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] up:boot 2024-04-23T18:16:30.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:29 smithi177 ceph-mon[32308]: fsmap 12 up:standby 2024-04-23T18:16:30.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:29 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:16:30.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.319+0000 7f10f6c23640 1 -- 172.21.15.50:0/4194277285 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10f01a0a50 con 0x7f10f0101fe0 2024-04-23T18:16:30.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.319+0000 7f10edffb640 1 -- 172.21.15.50:0/4194277285 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10488 (secure 0 0 0) 0x7f10dc009de0 con 0x7f10f0101fe0 2024-04-23T18:16:30.313 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:30.314 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-23T18:16:30.314 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 13 2024-04-23T18:16:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.321+0000 7f10f6c23640 1 -- 172.21.15.50:0/4194277285 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f10e00a3bb0 msgr2=0x7f10e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.321+0000 7f10f6c23640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f10e00a3bb0 0x7f10e00a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f10e4005fd0 tx=0x7f10e400b390 comp rx=0 tx=0).stop 2024-04-23T18:16:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.321+0000 7f10f6c23640 1 -- 172.21.15.50:0/4194277285 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10f0101fe0 msgr2=0x7f10f0198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.321+0000 7f10f6c23640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10f0101fe0 0x7f10f0198700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00d770 tx=0x7f10dc00dc30 comp rx=0 tx=0).stop 2024-04-23T18:16:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.321+0000 7f10f6c23640 1 -- 172.21.15.50:0/4194277285 shutdown_connections 2024-04-23T18:16:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.321+0000 7f10f6c23640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f10e00a3bb0 0x7f10e00a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.321+0000 7f10f6c23640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10f01029a0 0x7f10f019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.321+0000 7f10f6c23640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10f0101fe0 0x7f10f0198700 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.321+0000 7f10f6c23640 1 --2- 172.21.15.50:0/4194277285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f0100e90 0x7f10f01981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.321+0000 7f10f6c23640 1 -- 172.21.15.50:0/4194277285 >> 172.21.15.50:0/4194277285 conn(0x7f10f00fc400 msgr2=0x7f10f00fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.322+0000 7f10f6c23640 1 -- 172.21.15.50:0/4194277285 shutdown_connections 2024-04-23T18:16:30.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.322+0000 7f10f6c23640 1 -- 172.21.15.50:0/4194277285 wait complete. 2024-04-23T18:16:30.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:16:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:29 smithi188 ceph-mon[32694]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:29 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:29 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:29 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:29 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:29 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:29 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:16:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:29 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:29 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] up:boot 2024-04-23T18:16:30.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:29 smithi188 ceph-mon[32694]: fsmap 12 up:standby 2024-04-23T18:16:30.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:29 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:16:30.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.442+0000 7f525be59640 1 -- 172.21.15.50:0/1517464856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5254103190 msgr2=0x7f5254103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.442+0000 7f525be59640 1 --2- 172.21.15.50:0/1517464856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5254103190 0x7f5254103610 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5244009900 tx=0x7f524402a410 comp rx=0 tx=0).stop 2024-04-23T18:16:30.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.442+0000 7f525be59640 1 -- 172.21.15.50:0/1517464856 shutdown_connections 2024-04-23T18:16:30.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.442+0000 7f525be59640 1 --2- 172.21.15.50:0/1517464856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5254103b50 0x7f525410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.442+0000 7f525be59640 1 --2- 172.21.15.50:0/1517464856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5254103190 0x7f5254103610 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.442+0000 7f525be59640 1 --2- 172.21.15.50:0/1517464856 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5254101fa0 0x7f52541023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.442+0000 7f525be59640 1 -- 172.21.15.50:0/1517464856 >> 172.21.15.50:0/1517464856 conn(0x7f52540fd530 msgr2=0x7f52540ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.442+0000 7f525be59640 1 -- 172.21.15.50:0/1517464856 shutdown_connections 2024-04-23T18:16:30.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.442+0000 7f525be59640 1 -- 172.21.15.50:0/1517464856 wait complete. 2024-04-23T18:16:30.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f525be59640 1 Processor -- start 2024-04-23T18:16:30.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f525be59640 1 -- start start 2024-04-23T18:16:30.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f525be59640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5254101fa0 0x7f52541981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f525be59640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5254103190 0x7f52541986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f525be59640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5254103b50 0x7f525419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f525be59640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f525410dae0 con 0x7f5254101fa0 2024-04-23T18:16:30.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f525be59640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f525410d960 con 0x7f5254103b50 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f525be59640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f525410dc60 con 0x7f5254103190 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f5259bf5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5254101fa0 0x7f52541981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f5259bf5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5254101fa0 0x7f52541981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46708/0 (socket says 172.21.15.50:46708) 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f5259bf5640 1 -- 172.21.15.50:0/1464399978 learned_addr learned my addr 172.21.15.50:0/1464399978 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f52593f4640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5254103190 0x7f52541986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f525a3f6640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5254103b50 0x7f525419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f5259bf5640 1 -- 172.21.15.50:0/1464399978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5254103190 msgr2=0x7f52541986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f5259bf5640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5254103190 0x7f52541986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f5259bf5640 1 -- 172.21.15.50:0/1464399978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5254103b50 msgr2=0x7f525419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f5259bf5640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5254103b50 0x7f525419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f5259bf5640 1 -- 172.21.15.50:0/1464399978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525419fdc0 con 0x7f5254101fa0 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.443+0000 7f5259bf5640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5254101fa0 0x7f52541981a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f525000e9c0 tx=0x7f525000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.444+0000 7f524affd640 1 -- 172.21.15.50:0/1464399978 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525000cd80 con 0x7f5254101fa0 2024-04-23T18:16:30.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.444+0000 7f524affd640 1 -- 172.21.15.50:0/1464399978 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52500035f0 con 0x7f5254101fa0 2024-04-23T18:16:30.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.444+0000 7f525be59640 1 -- 172.21.15.50:0/1464399978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52541a00b0 con 0x7f5254101fa0 2024-04-23T18:16:30.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.444+0000 7f524affd640 1 -- 172.21.15.50:0/1464399978 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250010690 con 0x7f5254101fa0 2024-04-23T18:16:30.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.444+0000 7f525be59640 1 -- 172.21.15.50:0/1464399978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52541a0620 con 0x7f5254101fa0 2024-04-23T18:16:30.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.445+0000 7f525be59640 1 -- 172.21.15.50:0/1464399978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5254109330 con 0x7f5254101fa0 2024-04-23T18:16:30.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.446+0000 7f524affd640 1 -- 172.21.15.50:0/1464399978 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f52500037a0 con 0x7f5254101fa0 2024-04-23T18:16:30.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.447+0000 7f524affd640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52400a3c30 0x7f52400a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.447+0000 7f52593f4640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52400a3c30 0x7f52400a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.447+0000 7f524affd640 1 -- 172.21.15.50:0/1464399978 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f52500d0350 con 0x7f5254101fa0 2024-04-23T18:16:30.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.447+0000 7f52593f4640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52400a3c30 0x7f52400a60f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5244000c00 tx=0x7f52440314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.451+0000 7f524affd640 1 -- 172.21.15.50:0/1464399978 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f525009d1d0 con 0x7f5254101fa0 2024-04-23T18:16:30.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.661+0000 7f525be59640 1 -- 172.21.15.50:0/1464399978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5254062b50 con 0x7f5254101fa0 2024-04-23T18:16:30.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.662+0000 7f524affd640 1 -- 172.21.15.50:0/1464399978 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18418 (secure 0 0 0) 0x7f525009cad0 con 0x7f5254101fa0 2024-04-23T18:16:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:30.657 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":83,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:15:46.911832+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:16:30.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.664+0000 7f525be59640 1 -- 172.21.15.50:0/1464399978 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52400a3c30 msgr2=0x7f52400a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.664+0000 7f525be59640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52400a3c30 0x7f52400a60f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5244000c00 tx=0x7f52440314a0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.664+0000 7f525be59640 1 -- 172.21.15.50:0/1464399978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5254101fa0 msgr2=0x7f52541981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.664+0000 7f525be59640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5254101fa0 0x7f52541981a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f525000e9c0 tx=0x7f525000ee80 comp rx=0 tx=0).stop 2024-04-23T18:16:30.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.664+0000 7f525be59640 1 -- 172.21.15.50:0/1464399978 shutdown_connections 2024-04-23T18:16:30.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.664+0000 7f525be59640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52400a3c30 0x7f52400a60f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.664+0000 7f525be59640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5254103b50 0x7f525419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.664+0000 7f525be59640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5254103190 0x7f52541986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.664+0000 7f525be59640 1 --2- 172.21.15.50:0/1464399978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5254101fa0 0x7f52541981a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.664+0000 7f525be59640 1 -- 172.21.15.50:0/1464399978 >> 172.21.15.50:0/1464399978 conn(0x7f52540fd530 msgr2=0x7f5254106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.665+0000 7f525be59640 1 -- 172.21.15.50:0/1464399978 shutdown_connections 2024-04-23T18:16:30.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.665+0000 7f525be59640 1 -- 172.21.15.50:0/1464399978 wait complete. 2024-04-23T18:16:30.671 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:12:02.493647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:16:30.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:16:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.788+0000 7f5860a23640 1 -- 172.21.15.50:0/887082022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f585c103b50 msgr2=0x7f585c10a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.788+0000 7f5860a23640 1 --2- 172.21.15.50:0/887082022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f585c103b50 0x7f585c10a3e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5854009ca0 tx=0x7f585402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.788+0000 7f5860a23640 1 -- 172.21.15.50:0/887082022 shutdown_connections 2024-04-23T18:16:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.788+0000 7f5860a23640 1 --2- 172.21.15.50:0/887082022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f585c103b50 0x7f585c10a3e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.788+0000 7f5860a23640 1 --2- 172.21.15.50:0/887082022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585c103190 0x7f585c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.788+0000 7f5860a23640 1 --2- 172.21.15.50:0/887082022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f585c101fa0 0x7f585c1023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.788+0000 7f5860a23640 1 -- 172.21.15.50:0/887082022 >> 172.21.15.50:0/887082022 conn(0x7f585c0fd530 msgr2=0x7f585c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.788+0000 7f5860a23640 1 -- 172.21.15.50:0/887082022 shutdown_connections 2024-04-23T18:16:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.788+0000 7f5860a23640 1 -- 172.21.15.50:0/887082022 wait complete. 2024-04-23T18:16:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f5860a23640 1 Processor -- start 2024-04-23T18:16:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f5860a23640 1 -- start start 2024-04-23T18:16:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f5860a23640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f585c101fa0 0x7f585c198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f5860a23640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f585c103190 0x7f585c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f5860a23640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585c103b50 0x7f585c19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f5860a23640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f585c10db30 con 0x7f585c101fa0 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f5860a23640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f585c10d9b0 con 0x7f585c103b50 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f5860a23640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f585c10dcb0 con 0x7f585c103190 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f585a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f585c101fa0 0x7f585c198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f585a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f585c101fa0 0x7f585c198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46726/0 (socket says 172.21.15.50:46726) 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f585a59c640 1 -- 172.21.15.50:0/1053723699 learned_addr learned my addr 172.21.15.50:0/1053723699 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f5851d9b640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f585c103190 0x7f585c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.789+0000 7f585ad9d640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585c103b50 0x7f585c19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.790+0000 7f585a59c640 1 -- 172.21.15.50:0/1053723699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f585c103190 msgr2=0x7f585c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.790+0000 7f585a59c640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f585c103190 0x7f585c198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.790+0000 7f585a59c640 1 -- 172.21.15.50:0/1053723699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585c103b50 msgr2=0x7f585c19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.790+0000 7f585a59c640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585c103b50 0x7f585c19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.790+0000 7f585a59c640 1 -- 172.21.15.50:0/1053723699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f585c19fd70 con 0x7f585c101fa0 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.790+0000 7f5851d9b640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f585c103190 0x7f585c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.790+0000 7f585a59c640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f585c101fa0 0x7f585c198150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f584c00e9c0 tx=0x7f584c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.790+0000 7f5853fff640 1 -- 172.21.15.50:0/1053723699 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c00cd80 con 0x7f585c101fa0 2024-04-23T18:16:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.790+0000 7f5853fff640 1 -- 172.21.15.50:0/1053723699 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f584c0035f0 con 0x7f585c101fa0 2024-04-23T18:16:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.790+0000 7f5860a23640 1 -- 172.21.15.50:0/1053723699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f585c1a0060 con 0x7f585c101fa0 2024-04-23T18:16:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.790+0000 7f5853fff640 1 -- 172.21.15.50:0/1053723699 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c010690 con 0x7f585c101fa0 2024-04-23T18:16:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.790+0000 7f5860a23640 1 -- 172.21.15.50:0/1053723699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f585c1a0560 con 0x7f585c101fa0 2024-04-23T18:16:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.791+0000 7f5860a23640 1 -- 172.21.15.50:0/1053723699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f585c109340 con 0x7f585c101fa0 2024-04-23T18:16:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.792+0000 7f5853fff640 1 -- 172.21.15.50:0/1053723699 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f584c004000 con 0x7f585c101fa0 2024-04-23T18:16:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.793+0000 7f5853fff640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58480a3bb0 0x7f58480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.793+0000 7f5851d9b640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58480a3bb0 0x7f58480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.793+0000 7f5853fff640 1 -- 172.21.15.50:0/1053723699 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f584c0d0680 con 0x7f585c101fa0 2024-04-23T18:16:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.794+0000 7f5851d9b640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58480a3bb0 0x7f58480a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f585c199660 tx=0x7f584400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:30.800+0000 7f5853fff640 1 -- 172.21.15.50:0/1053723699 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f584c09d500 con 0x7f585c101fa0 2024-04-23T18:16:31.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.012+0000 7f5860a23640 1 -- 172.21.15.50:0/1053723699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f585c062b50 con 0x7f585c101fa0 2024-04-23T18:16:31.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.012+0000 7f5853fff640 1 -- 172.21.15.50:0/1053723699 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f584c09ce00 con 0x7f585c101fa0 2024-04-23T18:16:31.005 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-23T18:16:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.014+0000 7f5860a23640 1 -- 172.21.15.50:0/1053723699 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58480a3bb0 msgr2=0x7f58480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.014+0000 7f5860a23640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58480a3bb0 0x7f58480a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f585c199660 tx=0x7f584400b390 comp rx=0 tx=0).stop 2024-04-23T18:16:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.014+0000 7f5860a23640 1 -- 172.21.15.50:0/1053723699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f585c101fa0 msgr2=0x7f585c198150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.014+0000 7f5860a23640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f585c101fa0 0x7f585c198150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f584c00e9c0 tx=0x7f584c00ee80 comp rx=0 tx=0).stop 2024-04-23T18:16:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.014+0000 7f5860a23640 1 -- 172.21.15.50:0/1053723699 shutdown_connections 2024-04-23T18:16:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.014+0000 7f5860a23640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58480a3bb0 0x7f58480a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.014+0000 7f5860a23640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585c103b50 0x7f585c19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.014+0000 7f5860a23640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f585c103190 0x7f585c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.014+0000 7f5860a23640 1 --2- 172.21.15.50:0/1053723699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f585c101fa0 0x7f585c198150 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.014+0000 7f5860a23640 1 -- 172.21.15.50:0/1053723699 >> 172.21.15.50:0/1053723699 conn(0x7f585c0fd530 msgr2=0x7f585c106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.015+0000 7f5860a23640 1 -- 172.21.15.50:0/1053723699 shutdown_connections 2024-04-23T18:16:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.015+0000 7f5860a23640 1 -- 172.21.15.50:0/1053723699 wait complete. 2024-04-23T18:16:31.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-23T18:16:31.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.130+0000 7f25f977d640 1 -- 172.21.15.50:0/3858468808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25f41029a0 msgr2=0x7f25f41092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.130+0000 7f25f977d640 1 --2- 172.21.15.50:0/3858468808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25f41029a0 0x7f25f41092c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f25ec009ca0 tx=0x7f25ec02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.130+0000 7f25f977d640 1 -- 172.21.15.50:0/3858468808 shutdown_connections 2024-04-23T18:16:31.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.130+0000 7f25f977d640 1 --2- 172.21.15.50:0/3858468808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25f41029a0 0x7f25f41092c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.130+0000 7f25f977d640 1 --2- 172.21.15.50:0/3858468808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25f4101fe0 0x7f25f4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.130+0000 7f25f977d640 1 --2- 172.21.15.50:0/3858468808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25f4100e90 0x7f25f41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.130+0000 7f25f977d640 1 -- 172.21.15.50:0/3858468808 >> 172.21.15.50:0/3858468808 conn(0x7f25f40fc400 msgr2=0x7f25f40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:31.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.130+0000 7f25f977d640 1 -- 172.21.15.50:0/3858468808 shutdown_connections 2024-04-23T18:16:31.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.130+0000 7f25f977d640 1 -- 172.21.15.50:0/3858468808 wait complete. 2024-04-23T18:16:31.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f977d640 1 Processor -- start 2024-04-23T18:16:31.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f977d640 1 -- start start 2024-04-23T18:16:31.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f977d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25f4100e90 0x7f25f41981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:31.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f977d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25f4101fe0 0x7f25f41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:31.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f977d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25f41029a0 0x7f25f419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:31.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f977d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f25f410c9f0 con 0x7f25f4100e90 2024-04-23T18:16:31.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f977d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f25f410c870 con 0x7f25f41029a0 2024-04-23T18:16:31.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f977d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f25f410cb70 con 0x7f25f4101fe0 2024-04-23T18:16:31.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25f4100e90 0x7f25f41981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25f4100e90 0x7f25f41981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46752/0 (socket says 172.21.15.50:46752) 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f2ffd640 1 -- 172.21.15.50:0/3776652039 learned_addr learned my addr 172.21.15.50:0/3776652039 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f37fe640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25f41029a0 0x7f25f419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f27fc640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25f4101fe0 0x7f25f41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f2ffd640 1 -- 172.21.15.50:0/3776652039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25f4101fe0 msgr2=0x7f25f41986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f2ffd640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25f4101fe0 0x7f25f41986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f2ffd640 1 -- 172.21.15.50:0/3776652039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25f41029a0 msgr2=0x7f25f419f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.131+0000 7f25f2ffd640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25f41029a0 0x7f25f419f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.132+0000 7f25f2ffd640 1 -- 172.21.15.50:0/3776652039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f419fdd0 con 0x7f25f4100e90 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.132+0000 7f25f37fe640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25f41029a0 0x7f25f419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.132+0000 7f25f2ffd640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25f4100e90 0x7f25f41981b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f25e400e9f0 tx=0x7f25e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.132+0000 7f25dbfff640 1 -- 172.21.15.50:0/3776652039 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e400cd80 con 0x7f25f4100e90 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.132+0000 7f25f977d640 1 -- 172.21.15.50:0/3776652039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25f41a00c0 con 0x7f25f4100e90 2024-04-23T18:16:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.132+0000 7f25f977d640 1 -- 172.21.15.50:0/3776652039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25f41a0610 con 0x7f25f4100e90 2024-04-23T18:16:31.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.132+0000 7f25dbfff640 1 -- 172.21.15.50:0/3776652039 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25e40054d0 con 0x7f25f4100e90 2024-04-23T18:16:31.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.132+0000 7f25dbfff640 1 -- 172.21.15.50:0/3776652039 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e4010430 con 0x7f25f4100e90 2024-04-23T18:16:31.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.133+0000 7f25f977d640 1 -- 172.21.15.50:0/3776652039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e0005290 con 0x7f25f4100e90 2024-04-23T18:16:31.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.134+0000 7f25dbfff640 1 -- 172.21.15.50:0/3776652039 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f25e40046c0 con 0x7f25f4100e90 2024-04-23T18:16:31.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.135+0000 7f25dbfff640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f25dc0a3bb0 0x7f25dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:31.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.135+0000 7f25f27fc640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f25dc0a3bb0 0x7f25dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:31.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.135+0000 7f25dbfff640 1 -- 172.21.15.50:0/3776652039 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f25e40d0490 con 0x7f25f4100e90 2024-04-23T18:16:31.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.135+0000 7f25f27fc640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f25dc0a3bb0 0x7f25dc0a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f25e80095f0 tx=0x7f25e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:31.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.143+0000 7f25dbfff640 1 -- 172.21.15.50:0/3776652039 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f25e4053ff0 con 0x7f25f4100e90 2024-04-23T18:16:31.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:30 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4194277285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:31.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:30 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1464399978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:31.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:31.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:16:31.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:30 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:30 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4194277285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:30 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1464399978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:16:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:30 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.334+0000 7f25f977d640 1 -- 172.21.15.50:0/3776652039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f25e0005850 con 0x7f25f4100e90 2024-04-23T18:16:31.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.335+0000 7f25dbfff640 1 -- 172.21.15.50:0/3776652039 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) ==== 84+0+65 (secure 0 0 0) 0x7f25e409d270 con 0x7f25f4100e90 2024-04-23T18:16:31.328 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:31.328 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2024-04-23T18:16:31.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:16:31.328 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2024-04-23T18:16:31.328 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2024-04-23T18:16:31.328 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:16:31.328 INFO:teuthology.orchestra.run.smithi050.stdout:] 2024-04-23T18:16:31.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:30 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4194277285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:30 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1464399978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:16:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:30 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.337+0000 7f25f977d640 1 -- 172.21.15.50:0/3776652039 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f25dc0a3bb0 msgr2=0x7f25dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.337+0000 7f25f977d640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f25dc0a3bb0 0x7f25dc0a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f25e80095f0 tx=0x7f25e8005f90 comp rx=0 tx=0).stop 2024-04-23T18:16:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.337+0000 7f25f977d640 1 -- 172.21.15.50:0/3776652039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25f4100e90 msgr2=0x7f25f41981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.337+0000 7f25f977d640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25f4100e90 0x7f25f41981b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f25e400e9f0 tx=0x7f25e400edf0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.337+0000 7f25f977d640 1 -- 172.21.15.50:0/3776652039 shutdown_connections 2024-04-23T18:16:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.337+0000 7f25f977d640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f25dc0a3bb0 0x7f25dc0a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.337+0000 7f25f977d640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25f41029a0 0x7f25f419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.337+0000 7f25f977d640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25f4101fe0 0x7f25f41986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.337+0000 7f25f977d640 1 --2- 172.21.15.50:0/3776652039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25f4100e90 0x7f25f41981b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.337+0000 7f25f977d640 1 -- 172.21.15.50:0/3776652039 >> 172.21.15.50:0/3776652039 conn(0x7f25f40fc400 msgr2=0x7f25f40fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:31.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.337+0000 7f25f977d640 1 -- 172.21.15.50:0/3776652039 shutdown_connections 2024-04-23T18:16:31.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.337+0000 7f25f977d640 1 -- 172.21.15.50:0/3776652039 wait complete. 2024-04-23T18:16:31.341 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-23T18:16:31.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-23T18:16:31.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 -- 172.21.15.50:0/2043860148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4a8103190 msgr2=0x7fe4a8103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 --2- 172.21.15.50:0/2043860148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4a8103190 0x7fe4a8103610 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe498009900 tx=0x7fe49802a410 comp rx=0 tx=0).stop 2024-04-23T18:16:31.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 -- 172.21.15.50:0/2043860148 shutdown_connections 2024-04-23T18:16:31.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 --2- 172.21.15.50:0/2043860148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a8103b50 0x7fe4a810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 --2- 172.21.15.50:0/2043860148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4a8103190 0x7fe4a8103610 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 --2- 172.21.15.50:0/2043860148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8101fa0 0x7fe4a81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 -- 172.21.15.50:0/2043860148 >> 172.21.15.50:0/2043860148 conn(0x7fe4a80fd530 msgr2=0x7fe4a80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:31.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 -- 172.21.15.50:0/2043860148 shutdown_connections 2024-04-23T18:16:31.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 -- 172.21.15.50:0/2043860148 wait complete. 2024-04-23T18:16:31.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 Processor -- start 2024-04-23T18:16:31.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 -- start start 2024-04-23T18:16:31.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a8101fa0 0x7fe4a81981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:31.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8103190 0x7fe4a8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:31.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4a8103b50 0x7fe4a819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a810dae0 con 0x7fe4a8103b50 2024-04-23T18:16:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a810d960 con 0x7fe4a8103190 2024-04-23T18:16:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.456+0000 7fe4af623640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a810dc60 con 0x7fe4a8101fa0 2024-04-23T18:16:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4ad3bf640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a8101fa0 0x7fe4a81981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4adbc0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4a8103b50 0x7fe4a819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4ad3bf640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a8101fa0 0x7fe4a81981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:37214/0 (socket says 172.21.15.50:37214) 2024-04-23T18:16:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4ad3bf640 1 -- 172.21.15.50:0/3524563662 learned_addr learned my addr 172.21.15.50:0/3524563662 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4adbc0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4a8103b50 0x7fe4a819f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46764/0 (socket says 172.21.15.50:46764) 2024-04-23T18:16:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4acbbe640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8103190 0x7fe4a8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4adbc0640 1 -- 172.21.15.50:0/3524563662 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a8101fa0 msgr2=0x7fe4a81981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4adbc0640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a8101fa0 0x7fe4a81981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4adbc0640 1 -- 172.21.15.50:0/3524563662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8103190 msgr2=0x7fe4a8198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4adbc0640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8103190 0x7fe4a8198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4adbc0640 1 -- 172.21.15.50:0/3524563662 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a819fde0 con 0x7fe4a8103b50 2024-04-23T18:16:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4adbc0640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4a8103b50 0x7fe4a819f770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009fd0 tx=0x7fe4a000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe49e7fc640 1 -- 172.21.15.50:0/3524563662 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a000ce20 con 0x7fe4a8103b50 2024-04-23T18:16:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe49e7fc640 1 -- 172.21.15.50:0/3524563662 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4a0005020 con 0x7fe4a8103b50 2024-04-23T18:16:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4af623640 1 -- 172.21.15.50:0/3524563662 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a81a00d0 con 0x7fe4a8103b50 2024-04-23T18:16:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe49e7fc640 1 -- 172.21.15.50:0/3524563662 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0010690 con 0x7fe4a8103b50 2024-04-23T18:16:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.457+0000 7fe4af623640 1 -- 172.21.15.50:0/3524563662 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a81a05f0 con 0x7fe4a8103b50 2024-04-23T18:16:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.458+0000 7fe4af623640 1 -- 172.21.15.50:0/3524563662 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4a8109330 con 0x7fe4a8103b50 2024-04-23T18:16:31.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.460+0000 7fe49e7fc640 1 -- 172.21.15.50:0/3524563662 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe4a00054d0 con 0x7fe4a8103b50 2024-04-23T18:16:31.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.460+0000 7fe49e7fc640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe4940a3bb0 0x7fe4940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:31.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.460+0000 7fe4ad3bf640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe4940a3bb0 0x7fe4940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:31.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.460+0000 7fe49e7fc640 1 -- 172.21.15.50:0/3524563662 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fe4a00d05b0 con 0x7fe4a8103b50 2024-04-23T18:16:31.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.461+0000 7fe4ad3bf640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe4940a3bb0 0x7fe4940a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40058c0 tx=0x7fe4a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:31.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.468+0000 7fe49e7fc640 1 -- 172.21.15.50:0/3524563662 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe4a009d430 con 0x7fe4a8103b50 2024-04-23T18:16:31.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.675+0000 7fe4af623640 1 -- 172.21.15.50:0/3524563662 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fe4a802d5c0 con 0x7fe4a8103b50 2024-04-23T18:16:31.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.941+0000 7fe49e7fc640 1 -- 172.21.15.50:0/3524563662 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) ==== 138+0+0 (secure 0 0 0) 0x7fe4a009cd30 con 0x7fe4a8103b50 2024-04-23T18:16:31.934 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2024-04-23T18:16:31.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.943+0000 7fe4af623640 1 -- 172.21.15.50:0/3524563662 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe4940a3bb0 msgr2=0x7fe4940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.943+0000 7fe4af623640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe4940a3bb0 0x7fe4940a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40058c0 tx=0x7fe4a4005f90 comp rx=0 tx=0).stop 2024-04-23T18:16:31.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.943+0000 7fe4af623640 1 -- 172.21.15.50:0/3524563662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4a8103b50 msgr2=0x7fe4a819f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:31.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.943+0000 7fe4af623640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4a8103b50 0x7fe4a819f770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009fd0 tx=0x7fe4a000ef10 comp rx=0 tx=0).stop 2024-04-23T18:16:31.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.943+0000 7fe4af623640 1 -- 172.21.15.50:0/3524563662 shutdown_connections 2024-04-23T18:16:31.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.943+0000 7fe4af623640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe4940a3bb0 0x7fe4940a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.944+0000 7fe4af623640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4a8103b50 0x7fe4a819f770 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.944+0000 7fe4af623640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8103190 0x7fe4a8198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.944+0000 7fe4af623640 1 --2- 172.21.15.50:0/3524563662 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a8101fa0 0x7fe4a81981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:31.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.944+0000 7fe4af623640 1 -- 172.21.15.50:0/3524563662 >> 172.21.15.50:0/3524563662 conn(0x7fe4a80fd530 msgr2=0x7fe4a8106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:31.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.944+0000 7fe4af623640 1 -- 172.21.15.50:0/3524563662 shutdown_connections 2024-04-23T18:16:31.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:31.944+0000 7fe4af623640 1 -- 172.21.15.50:0/3524563662 wait complete. 2024-04-23T18:16:31.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-23T18:16:32.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:31 smithi177 ceph-mon[32308]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:32.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:31 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1053723699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:32.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:31 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:16:32.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:31 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3776652039' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T18:16:32.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:31 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3524563662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T18:16:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.069+0000 7f1d09ed0640 1 -- 172.21.15.50:0/4023774478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d04107cf0 msgr2=0x7f1d0410a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.069+0000 7f1d09ed0640 1 --2- 172.21.15.50:0/4023774478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d04107cf0 0x7f1d0410a0e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009ca0 tx=0x7f1cfc02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.069+0000 7f1d09ed0640 1 -- 172.21.15.50:0/4023774478 shutdown_connections 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.069+0000 7f1d09ed0640 1 --2- 172.21.15.50:0/4023774478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d04107cf0 0x7f1d0410a0e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.069+0000 7f1d09ed0640 1 --2- 172.21.15.50:0/4023774478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d041053c0 0x7f1d041077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.069+0000 7f1d09ed0640 1 --2- 172.21.15.50:0/4023774478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d04102a90 0x7f1d04104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.069+0000 7f1d09ed0640 1 -- 172.21.15.50:0/4023774478 >> 172.21.15.50:0/4023774478 conn(0x7f1d040fc440 msgr2=0x7f1d040fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.069+0000 7f1d09ed0640 1 -- 172.21.15.50:0/4023774478 shutdown_connections 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.069+0000 7f1d09ed0640 1 -- 172.21.15.50:0/4023774478 wait complete. 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d09ed0640 1 Processor -- start 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d09ed0640 1 -- start start 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d09ed0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d04102a90 0x7f1d0419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d09ed0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d041053c0 0x7f1d0419a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d09ed0640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d04107cf0 0x7f1d041a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d09ed0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0410daf0 con 0x7f1d04102a90 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d09ed0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0410d970 con 0x7f1d041053c0 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d09ed0640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0410dc70 con 0x7f1d04107cf0 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d037fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d04102a90 0x7f1d0419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d037fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d04102a90 0x7f1d0419a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38552/0 (socket says 172.21.15.50:38552) 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d037fe640 1 -- 172.21.15.50:0/1288714359 learned_addr learned my addr 172.21.15.50:0/1288714359 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d03fff640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d04107cf0 0x7f1d041a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1cfbfff640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d041053c0 0x7f1d0419a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d037fe640 1 -- 172.21.15.50:0/1288714359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d04107cf0 msgr2=0x7f1d041a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d037fe640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d04107cf0 0x7f1d041a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d037fe640 1 -- 172.21.15.50:0/1288714359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d041053c0 msgr2=0x7f1d0419a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d037fe640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d041053c0 0x7f1d0419a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.070+0000 7f1d037fe640 1 -- 172.21.15.50:0/1288714359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d041a1f60 con 0x7f1d04102a90 2024-04-23T18:16:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.071+0000 7f1d037fe640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d04102a90 0x7f1d0419a340 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1cf400c920 tx=0x7f1cf400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.071+0000 7f1d017fa640 1 -- 172.21.15.50:0/1288714359 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4007c70 con 0x7f1d04102a90 2024-04-23T18:16:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.071+0000 7f1d017fa640 1 -- 172.21.15.50:0/1288714359 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cf4007e10 con 0x7f1d04102a90 2024-04-23T18:16:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.071+0000 7f1d09ed0640 1 -- 172.21.15.50:0/1288714359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d041a2250 con 0x7f1d04102a90 2024-04-23T18:16:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.071+0000 7f1d017fa640 1 -- 172.21.15.50:0/1288714359 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4018820 con 0x7f1d04102a90 2024-04-23T18:16:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.071+0000 7f1d09ed0640 1 -- 172.21.15.50:0/1288714359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d041a27a0 con 0x7f1d04102a90 2024-04-23T18:16:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.072+0000 7f1d09ed0640 1 -- 172.21.15.50:0/1288714359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d041097e0 con 0x7f1d04102a90 2024-04-23T18:16:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.073+0000 7f1d017fa640 1 -- 172.21.15.50:0/1288714359 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1cf400f450 con 0x7f1d04102a90 2024-04-23T18:16:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.074+0000 7f1d017fa640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1cf00a3bb0 0x7f1cf00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.074+0000 7f1cfbfff640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1cf00a3bb0 0x7f1cf00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.074+0000 7f1d017fa640 1 -- 172.21.15.50:0/1288714359 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7f1cf40d04a0 con 0x7f1d04102a90 2024-04-23T18:16:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.074+0000 7f1cfbfff640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1cf00a3bb0 0x7f1cf00a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1d0419b850 tx=0x7f1cec00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.077+0000 7f1d017fa640 1 -- 172.21.15.50:0/1288714359 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1cf4014030 con 0x7f1d04102a90 2024-04-23T18:16:32.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:31 smithi050 ceph-mon[29187]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:32.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:31 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1053723699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:32.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:31 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:16:32.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:31 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3776652039' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T18:16:32.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:31 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3524563662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T18:16:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.265+0000 7f1d09ed0640 1 -- 172.21.15.50:0/1288714359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f1d0402d5c0 con 0x7f1d04102a90 2024-04-23T18:16:32.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:31 smithi188 ceph-mon[32694]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:32.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:31 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1053723699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:16:32.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:31 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:16:32.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:31 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3776652039' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T18:16:32.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:31 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3524563662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T18:16:32.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.943+0000 7f1d017fa640 1 -- 172.21.15.50:0/1288714359 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v85) ==== 170+0+0 (secure 0 0 0) 0x7f1cf409d1a0 con 0x7f1d04102a90 2024-04-23T18:16:32.936 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2024-04-23T18:16:32.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.945+0000 7f1d09ed0640 1 -- 172.21.15.50:0/1288714359 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1cf00a3bb0 msgr2=0x7f1cf00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.945+0000 7f1d09ed0640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1cf00a3bb0 0x7f1cf00a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1d0419b850 tx=0x7f1cec00b390 comp rx=0 tx=0).stop 2024-04-23T18:16:32.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.945+0000 7f1d09ed0640 1 -- 172.21.15.50:0/1288714359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d04102a90 msgr2=0x7f1d0419a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:32.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.945+0000 7f1d09ed0640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d04102a90 0x7f1d0419a340 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1cf400c920 tx=0x7f1cf400cde0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.946+0000 7f1d09ed0640 1 -- 172.21.15.50:0/1288714359 shutdown_connections 2024-04-23T18:16:32.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.946+0000 7f1d09ed0640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1cf00a3bb0 0x7f1cf00a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.946+0000 7f1d09ed0640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d04107cf0 0x7f1d041a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.946+0000 7f1d09ed0640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d041053c0 0x7f1d0419a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.946+0000 7f1d09ed0640 1 --2- 172.21.15.50:0/1288714359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d04102a90 0x7f1d0419a340 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:32.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.946+0000 7f1d09ed0640 1 -- 172.21.15.50:0/1288714359 >> 172.21.15.50:0/1288714359 conn(0x7f1d040fc440 msgr2=0x7f1d04102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:32.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.947+0000 7f1d09ed0640 1 -- 172.21.15.50:0/1288714359 shutdown_connections 2024-04-23T18:16:32.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:32.947+0000 7f1d09ed0640 1 -- 172.21.15.50:0/1288714359 wait complete. 2024-04-23T18:16:32.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-23T18:16:33.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.079+0000 7fc0700cb640 1 -- 172.21.15.50:0/3467010828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc068107cd0 msgr2=0x7fc06810a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.079+0000 7fc0700cb640 1 --2- 172.21.15.50:0/3467010828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc068107cd0 0x7fc06810a0c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc060009ca0 tx=0x7fc06002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 -- 172.21.15.50:0/3467010828 shutdown_connections 2024-04-23T18:16:33.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 --2- 172.21.15.50:0/3467010828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc068107cd0 0x7fc06810a0c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 --2- 172.21.15.50:0/3467010828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0681053a0 0x7fc068107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 --2- 172.21.15.50:0/3467010828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc068102a70 0x7fc068104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 -- 172.21.15.50:0/3467010828 >> 172.21.15.50:0/3467010828 conn(0x7fc0680fc400 msgr2=0x7fc0680fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 -- 172.21.15.50:0/3467010828 shutdown_connections 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 -- 172.21.15.50:0/3467010828 wait complete. 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 Processor -- start 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 -- start start 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc068102a70 0x7fc068198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0681053a0 0x7fc068198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc068107cd0 0x7fc06819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc06810dad0 con 0x7fc068102a70 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc06810d950 con 0x7fc0681053a0 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.080+0000 7fc0700cb640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc06810dc50 con 0x7fc068107cd0 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc06de67640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc068102a70 0x7fc068198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc06de67640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc068102a70 0x7fc068198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38570/0 (socket says 172.21.15.50:38570) 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc06de67640 1 -- 172.21.15.50:0/972586903 learned_addr learned my addr 172.21.15.50:0/972586903 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc06e668640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc068107cd0 0x7fc06819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc06d666640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0681053a0 0x7fc068198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc06de67640 1 -- 172.21.15.50:0/972586903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc068107cd0 msgr2=0x7fc06819f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc06de67640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc068107cd0 0x7fc06819f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc06de67640 1 -- 172.21.15.50:0/972586903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0681053a0 msgr2=0x7fc068198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc06de67640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0681053a0 0x7fc068198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc06de67640 1 -- 172.21.15.50:0/972586903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06819fe10 con 0x7fc068102a70 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc06d666640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0681053a0 0x7fc068198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc06de67640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc068102a70 0x7fc068198240 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc06400d950 tx=0x7fc06400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc05effd640 1 -- 172.21.15.50:0/972586903 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc064017070 con 0x7fc068102a70 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc05effd640 1 -- 172.21.15.50:0/972586903 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc064009c30 con 0x7fc068102a70 2024-04-23T18:16:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc0700cb640 1 -- 172.21.15.50:0/972586903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0681a0100 con 0x7fc068102a70 2024-04-23T18:16:33.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc05effd640 1 -- 172.21.15.50:0/972586903 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06400f620 con 0x7fc068102a70 2024-04-23T18:16:33.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.081+0000 7fc0700cb640 1 -- 172.21.15.50:0/972586903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0681a06a0 con 0x7fc068102a70 2024-04-23T18:16:33.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.082+0000 7fc0700cb640 1 -- 172.21.15.50:0/972586903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0680fdef0 con 0x7fc068102a70 2024-04-23T18:16:33.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.084+0000 7fc05effd640 1 -- 172.21.15.50:0/972586903 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc064010460 con 0x7fc068102a70 2024-04-23T18:16:33.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.084+0000 7fc05effd640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc0540a3c30 0x7fc0540a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:33.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.084+0000 7fc06d666640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc0540a3c30 0x7fc0540a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:33.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.084+0000 7fc05effd640 1 -- 172.21.15.50:0/972586903 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7879+0+0 (secure 0 0 0) 0x7fc0640cfde0 con 0x7fc068102a70 2024-04-23T18:16:33.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.085+0000 7fc06d666640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc0540a3c30 0x7fc0540a60f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc068199700 tx=0x7fc058005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:33.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.088+0000 7fc05effd640 1 -- 172.21.15.50:0/972586903 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc06409c9f0 con 0x7fc068102a70 2024-04-23T18:16:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:32 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3524563662' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T18:16:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:32 smithi177 ceph-mon[32308]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T18:16:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:32 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1288714359' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T18:16:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:33.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:16:33.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:32 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.316+0000 7fc0700cb640 1 -- 172.21.15.50:0/972586903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc06802d5c0 con 0x7fc068102a70 2024-04-23T18:16:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:32 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3524563662' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T18:16:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:32 smithi188 ceph-mon[32694]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T18:16:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:32 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1288714359' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T18:16:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:16:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:32 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:32 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3524563662' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T18:16:33.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:32 smithi050 ceph-mon[29187]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T18:16:33.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:32 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1288714359' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T18:16:33.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:16:33.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:16:33.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:32 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:16:33.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.966+0000 7fc05effd640 1 -- 172.21.15.50:0/972586903 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fc064009de0 con 0x7fc068102a70 2024-04-23T18:16:33.959 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 2 and data pool 3 2024-04-23T18:16:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.969+0000 7fc0700cb640 1 -- 172.21.15.50:0/972586903 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc0540a3c30 msgr2=0x7fc0540a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.969+0000 7fc0700cb640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc0540a3c30 0x7fc0540a60f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc068199700 tx=0x7fc058005e50 comp rx=0 tx=0).stop 2024-04-23T18:16:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.969+0000 7fc0700cb640 1 -- 172.21.15.50:0/972586903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc068102a70 msgr2=0x7fc068198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.969+0000 7fc0700cb640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc068102a70 0x7fc068198240 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc06400d950 tx=0x7fc06400de10 comp rx=0 tx=0).stop 2024-04-23T18:16:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.969+0000 7fc0700cb640 1 -- 172.21.15.50:0/972586903 shutdown_connections 2024-04-23T18:16:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.969+0000 7fc0700cb640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc0540a3c30 0x7fc0540a60f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.969+0000 7fc0700cb640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc068107cd0 0x7fc06819f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.969+0000 7fc0700cb640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0681053a0 0x7fc068198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.969+0000 7fc0700cb640 1 --2- 172.21.15.50:0/972586903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc068102a70 0x7fc068198240 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.969+0000 7fc0700cb640 1 -- 172.21.15.50:0/972586903 >> 172.21.15.50:0/972586903 conn(0x7fc0680fc400 msgr2=0x7fc0681033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.970+0000 7fc0700cb640 1 -- 172.21.15.50:0/972586903 shutdown_connections 2024-04-23T18:16:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:33.970+0000 7fc0700cb640 1 -- 172.21.15.50:0/972586903 wait complete. 2024-04-23T18:16:33.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:16:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.103+0000 7f6c11fd8640 1 -- 172.21.15.50:0/984932183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0c105360 msgr2=0x7f6c0c107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.103+0000 7f6c11fd8640 1 --2- 172.21.15.50:0/984932183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0c105360 0x7f6c0c107750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc009900 tx=0x7f6bfc02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.103+0000 7f6c11fd8640 1 -- 172.21.15.50:0/984932183 shutdown_connections 2024-04-23T18:16:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.103+0000 7f6c11fd8640 1 --2- 172.21.15.50:0/984932183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c107c90 0x7f6c0c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.103+0000 7f6c11fd8640 1 --2- 172.21.15.50:0/984932183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0c105360 0x7f6c0c107750 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.103+0000 7f6c11fd8640 1 --2- 172.21.15.50:0/984932183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c102a30 0x7f6c0c104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.103+0000 7f6c11fd8640 1 -- 172.21.15.50:0/984932183 >> 172.21.15.50:0/984932183 conn(0x7f6c0c0fc440 msgr2=0x7f6c0c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.103+0000 7f6c11fd8640 1 -- 172.21.15.50:0/984932183 shutdown_connections 2024-04-23T18:16:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.103+0000 7f6c11fd8640 1 -- 172.21.15.50:0/984932183 wait complete. 2024-04-23T18:16:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c11fd8640 1 Processor -- start 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c11fd8640 1 -- start start 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c11fd8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0c102a30 0x7f6c0c198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c11fd8640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c105360 0x7f6c0c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c11fd8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c107c90 0x7f6c0c19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c11fd8640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c10dab0 con 0x7f6c0c102a30 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c11fd8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c10d930 con 0x7f6c0c107c90 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c11fd8640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c10dc30 con 0x7f6c0c105360 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c0b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0c102a30 0x7f6c0c198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c0b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0c102a30 0x7f6c0c198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38586/0 (socket says 172.21.15.50:38586) 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c0b7fe640 1 -- 172.21.15.50:0/1098246585 learned_addr learned my addr 172.21.15.50:0/1098246585 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c0affd640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c105360 0x7f6c0c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.104+0000 7f6c0bfff640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c107c90 0x7f6c0c19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.105+0000 7f6c0b7fe640 1 -- 172.21.15.50:0/1098246585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c105360 msgr2=0x7f6c0c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.105+0000 7f6c0b7fe640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c105360 0x7f6c0c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.105+0000 7f6c0b7fe640 1 -- 172.21.15.50:0/1098246585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c107c90 msgr2=0x7f6c0c19f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.105+0000 7f6c0b7fe640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c107c90 0x7f6c0c19f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.105+0000 7f6c0b7fe640 1 -- 172.21.15.50:0/1098246585 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c19fe40 con 0x7f6c0c102a30 2024-04-23T18:16:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.105+0000 7f6c0b7fe640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0c102a30 0x7f6c0c198220 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6c0000d950 tx=0x7f6c0000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.105+0000 7f6c08ff9640 1 -- 172.21.15.50:0/1098246585 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c00017030 con 0x7f6c0c102a30 2024-04-23T18:16:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.105+0000 7f6c08ff9640 1 -- 172.21.15.50:0/1098246585 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c00004020 con 0x7f6c0c102a30 2024-04-23T18:16:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.105+0000 7f6c11fd8640 1 -- 172.21.15.50:0/1098246585 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c0c1a0130 con 0x7f6c0c102a30 2024-04-23T18:16:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.105+0000 7f6c08ff9640 1 -- 172.21.15.50:0/1098246585 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c000047a0 con 0x7f6c0c102a30 2024-04-23T18:16:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.105+0000 7f6c11fd8640 1 -- 172.21.15.50:0/1098246585 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c0c1a0650 con 0x7f6c0c102a30 2024-04-23T18:16:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.106+0000 7f6c11fd8640 1 -- 172.21.15.50:0/1098246585 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c0c0fdf30 con 0x7f6c0c102a30 2024-04-23T18:16:34.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.107+0000 7f6c08ff9640 1 -- 172.21.15.50:0/1098246585 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6c00009a30 con 0x7f6c0c102a30 2024-04-23T18:16:34.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.108+0000 7f6c08ff9640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6bf40a3bb0 0x7f6bf40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.108+0000 7f6c0affd640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6bf40a3bb0 0x7f6bf40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.108+0000 7f6c08ff9640 1 -- 172.21.15.50:0/1098246585 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f6c000d0330 con 0x7f6c0c102a30 2024-04-23T18:16:34.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.108+0000 7f6c0affd640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6bf40a3bb0 0x7f6bf40a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc009230 tx=0x7f6bfc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.111+0000 7f6c08ff9640 1 -- 172.21.15.50:0/1098246585 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c0009ce60 con 0x7f6c0c102a30 2024-04-23T18:16:34.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:33 smithi050 ceph-mon[29187]: pgmap v283: 33 pgs: 6 creating+peering, 26 unknown, 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:34.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:33 smithi050 ceph-mon[29187]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T18:16:34.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:33 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1288714359' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T18:16:34.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:33 smithi050 ceph-mon[29187]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T18:16:34.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:33 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/972586903' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T18:16:34.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:33 smithi050 ceph-mon[29187]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T18:16:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:33 smithi177 ceph-mon[32308]: pgmap v283: 33 pgs: 6 creating+peering, 26 unknown, 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:33 smithi177 ceph-mon[32308]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T18:16:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:33 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1288714359' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T18:16:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:33 smithi177 ceph-mon[32308]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T18:16:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:33 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/972586903' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T18:16:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:33 smithi177 ceph-mon[32308]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T18:16:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.303+0000 7f6c11fd8640 1 -- 172.21.15.50:0/1098246585 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c0c02d5c0 con 0x7f6c0c102a30 2024-04-23T18:16:34.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.305+0000 7f6c08ff9640 1 -- 172.21.15.50:0/1098246585 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22194 (secure 0 0 0) 0x7f6c0009c760 con 0x7f6c0c102a30 2024-04-23T18:16:34.298 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:34.298 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":86,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:16:33.316505+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:16:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.307+0000 7f6c11fd8640 1 -- 172.21.15.50:0/1098246585 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6bf40a3bb0 msgr2=0x7f6bf40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.307+0000 7f6c11fd8640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6bf40a3bb0 0x7f6bf40a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc009230 tx=0x7f6bfc0314a0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.307+0000 7f6c11fd8640 1 -- 172.21.15.50:0/1098246585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0c102a30 msgr2=0x7f6c0c198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.307+0000 7f6c11fd8640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0c102a30 0x7f6c0c198220 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6c0000d950 tx=0x7f6c0000de10 comp rx=0 tx=0).stop 2024-04-23T18:16:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.307+0000 7f6c11fd8640 1 -- 172.21.15.50:0/1098246585 shutdown_connections 2024-04-23T18:16:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.307+0000 7f6c11fd8640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6bf40a3bb0 0x7f6bf40a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.307+0000 7f6c11fd8640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c107c90 0x7f6c0c19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.307+0000 7f6c11fd8640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c105360 0x7f6c0c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.307+0000 7f6c11fd8640 1 --2- 172.21.15.50:0/1098246585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0c102a30 0x7f6c0c198220 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.307+0000 7f6c11fd8640 1 -- 172.21.15.50:0/1098246585 >> 172.21.15.50:0/1098246585 conn(0x7f6c0c0fc440 msgr2=0x7f6c0c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.307+0000 7f6c11fd8640 1 -- 172.21.15.50:0/1098246585 shutdown_connections 2024-04-23T18:16:34.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.307+0000 7f6c11fd8640 1 -- 172.21.15.50:0/1098246585 wait complete. 2024-04-23T18:16:34.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:16:34.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:33 smithi188 ceph-mon[32694]: pgmap v283: 33 pgs: 6 creating+peering, 26 unknown, 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:33 smithi188 ceph-mon[32694]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T18:16:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:33 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1288714359' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T18:16:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:33 smithi188 ceph-mon[32694]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T18:16:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:33 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/972586903' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T18:16:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:33 smithi188 ceph-mon[32694]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T18:16:34.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.420+0000 7fec873ad640 1 -- 172.21.15.50:0/1745532256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec801031b0 msgr2=0x7fec80103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.420+0000 7fec873ad640 1 --2- 172.21.15.50:0/1745532256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec801031b0 0x7fec80103630 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fec70009900 tx=0x7fec7002a410 comp rx=0 tx=0).stop 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.420+0000 7fec873ad640 1 -- 172.21.15.50:0/1745532256 shutdown_connections 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.420+0000 7fec873ad640 1 --2- 172.21.15.50:0/1745532256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec80103b70 0x7fec8010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.420+0000 7fec873ad640 1 --2- 172.21.15.50:0/1745532256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec801031b0 0x7fec80103630 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.420+0000 7fec873ad640 1 --2- 172.21.15.50:0/1745532256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec80101fc0 0x7fec801023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.420+0000 7fec873ad640 1 -- 172.21.15.50:0/1745532256 >> 172.21.15.50:0/1745532256 conn(0x7fec800fd550 msgr2=0x7fec800ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.420+0000 7fec873ad640 1 -- 172.21.15.50:0/1745532256 shutdown_connections 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.420+0000 7fec873ad640 1 -- 172.21.15.50:0/1745532256 wait complete. 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec873ad640 1 Processor -- start 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec873ad640 1 -- start start 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec873ad640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec80101fc0 0x7fec80198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec873ad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec801031b0 0x7fec801986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec873ad640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec80103b70 0x7fec8019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec873ad640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fec8010db00 con 0x7fec801031b0 2024-04-23T18:16:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec873ad640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fec8010d980 con 0x7fec80101fc0 2024-04-23T18:16:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec873ad640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fec8010dc80 con 0x7fec80103b70 2024-04-23T18:16:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec84948640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec801031b0 0x7fec801986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec84948640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec801031b0 0x7fec801986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38614/0 (socket says 172.21.15.50:38614) 2024-04-23T18:16:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec84948640 1 -- 172.21.15.50:0/1876544298 learned_addr learned my addr 172.21.15.50:0/1876544298 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec8594a640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec80103b70 0x7fec8019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec85149640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec80101fc0 0x7fec80198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec84948640 1 -- 172.21.15.50:0/1876544298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec80103b70 msgr2=0x7fec8019f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec84948640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec80103b70 0x7fec8019f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec84948640 1 -- 172.21.15.50:0/1876544298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec80101fc0 msgr2=0x7fec80198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec84948640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec80101fc0 0x7fec80198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.421+0000 7fec84948640 1 -- 172.21.15.50:0/1876544298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8019fdb0 con 0x7fec801031b0 2024-04-23T18:16:34.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.422+0000 7fec84948640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec801031b0 0x7fec801986d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fec70005940 tx=0x7fec70031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.422+0000 7fec767fc640 1 -- 172.21.15.50:0/1876544298 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7003f030 con 0x7fec801031b0 2024-04-23T18:16:34.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.422+0000 7fec767fc640 1 -- 172.21.15.50:0/1876544298 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec70003aa0 con 0x7fec801031b0 2024-04-23T18:16:34.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.422+0000 7fec873ad640 1 -- 172.21.15.50:0/1876544298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec801a0040 con 0x7fec801031b0 2024-04-23T18:16:34.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.422+0000 7fec767fc640 1 -- 172.21.15.50:0/1876544298 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70035070 con 0x7fec801031b0 2024-04-23T18:16:34.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.422+0000 7fec873ad640 1 -- 172.21.15.50:0/1876544298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec801a0580 con 0x7fec801031b0 2024-04-23T18:16:34.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.423+0000 7fec873ad640 1 -- 172.21.15.50:0/1876544298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec801092e0 con 0x7fec801031b0 2024-04-23T18:16:34.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.424+0000 7fec767fc640 1 -- 172.21.15.50:0/1876544298 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fec70042030 con 0x7fec801031b0 2024-04-23T18:16:34.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.425+0000 7fec767fc640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fec6c0a3c30 0x7fec6c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.425+0000 7fec85149640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fec6c0a3c30 0x7fec6c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.425+0000 7fec767fc640 1 -- 172.21.15.50:0/1876544298 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fec700f0910 con 0x7fec801031b0 2024-04-23T18:16:34.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.425+0000 7fec85149640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fec6c0a3c30 0x7fec6c0a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec7c005d00 tx=0x7fec7c005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.429+0000 7fec767fc640 1 -- 172.21.15.50:0/1876544298 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec700bd3a0 con 0x7fec801031b0 2024-04-23T18:16:34.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.618+0000 7fec873ad640 1 -- 172.21.15.50:0/1876544298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fec80062b50 con 0x7fec801031b0 2024-04-23T18:16:34.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.620+0000 7fec767fc640 1 -- 172.21.15.50:0/1876544298 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22194 (secure 0 0 0) 0x7fec700bcca0 con 0x7fec801031b0 2024-04-23T18:16:34.613 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:34.613 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":86,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:16:33.316505+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:16:34.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.622+0000 7fec873ad640 1 -- 172.21.15.50:0/1876544298 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fec6c0a3c30 msgr2=0x7fec6c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.622+0000 7fec873ad640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fec6c0a3c30 0x7fec6c0a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec7c005d00 tx=0x7fec7c005c50 comp rx=0 tx=0).stop 2024-04-23T18:16:34.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.622+0000 7fec873ad640 1 -- 172.21.15.50:0/1876544298 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec801031b0 msgr2=0x7fec801986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.622+0000 7fec873ad640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec801031b0 0x7fec801986d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fec70005940 tx=0x7fec70031a50 comp rx=0 tx=0).stop 2024-04-23T18:16:34.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.622+0000 7fec873ad640 1 -- 172.21.15.50:0/1876544298 shutdown_connections 2024-04-23T18:16:34.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.622+0000 7fec873ad640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fec6c0a3c30 0x7fec6c0a60f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.622+0000 7fec873ad640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec80103b70 0x7fec8019f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.622+0000 7fec873ad640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec801031b0 0x7fec801986d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.622+0000 7fec873ad640 1 --2- 172.21.15.50:0/1876544298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec80101fc0 0x7fec80198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.622+0000 7fec873ad640 1 -- 172.21.15.50:0/1876544298 >> 172.21.15.50:0/1876544298 conn(0x7fec800fd550 msgr2=0x7fec801064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.622+0000 7fec873ad640 1 -- 172.21.15.50:0/1876544298 shutdown_connections 2024-04-23T18:16:34.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.622+0000 7fec873ad640 1 -- 172.21.15.50:0/1876544298 wait complete. 2024-04-23T18:16:34.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-23T18:16:34.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.738+0000 7f7e40d08640 1 -- 172.21.15.50:0/3919126617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e3c100e60 msgr2=0x7f7e3c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.738+0000 7f7e40d08640 1 --2- 172.21.15.50:0/3919126617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e3c100e60 0x7f7e3c101280 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009900 tx=0x7f7e2402a410 comp rx=0 tx=0).stop 2024-04-23T18:16:34.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.738+0000 7f7e40d08640 1 -- 172.21.15.50:0/3919126617 shutdown_connections 2024-04-23T18:16:34.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.738+0000 7f7e40d08640 1 --2- 172.21.15.50:0/3919126617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e3c102a00 0x7f7e3c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.738+0000 7f7e40d08640 1 --2- 172.21.15.50:0/3919126617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e3c102040 0x7f7e3c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.738+0000 7f7e40d08640 1 --2- 172.21.15.50:0/3919126617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e3c100e60 0x7f7e3c101280 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.738+0000 7f7e40d08640 1 -- 172.21.15.50:0/3919126617 >> 172.21.15.50:0/3919126617 conn(0x7f7e3c0fc440 msgr2=0x7f7e3c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.738+0000 7f7e40d08640 1 -- 172.21.15.50:0/3919126617 shutdown_connections 2024-04-23T18:16:34.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.738+0000 7f7e40d08640 1 -- 172.21.15.50:0/3919126617 wait complete. 2024-04-23T18:16:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e40d08640 1 Processor -- start 2024-04-23T18:16:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e40d08640 1 -- start start 2024-04-23T18:16:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e40d08640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e3c100e60 0x7f7e3c104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e40d08640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e3c102040 0x7f7e3c1034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e40d08640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e3c102a00 0x7f7e3c1039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e40d08640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c10c9f0 con 0x7f7e3c102a00 2024-04-23T18:16:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e40d08640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c10c870 con 0x7f7e3c102040 2024-04-23T18:16:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e40d08640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c10cb70 con 0x7f7e3c100e60 2024-04-23T18:16:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e3ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e3c102a00 0x7f7e3c1039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e3ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e3c102a00 0x7f7e3c1039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38636/0 (socket says 172.21.15.50:38636) 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e3ad9d640 1 -- 172.21.15.50:0/1921138055 learned_addr learned my addr 172.21.15.50:0/1921138055 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e3a59c640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e3c100e60 0x7f7e3c104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e39d9b640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e3c102040 0x7f7e3c1034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e3ad9d640 1 -- 172.21.15.50:0/1921138055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e3c100e60 msgr2=0x7f7e3c104e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e3ad9d640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e3c100e60 0x7f7e3c104e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e3ad9d640 1 -- 172.21.15.50:0/1921138055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e3c102040 msgr2=0x7f7e3c1034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e3ad9d640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e3c102040 0x7f7e3c1034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.739+0000 7f7e3ad9d640 1 -- 172.21.15.50:0/1921138055 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3c104280 con 0x7f7e3c102a00 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.740+0000 7f7e3ad9d640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e3c102a00 0x7f7e3c1039f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400eb10 tx=0x7f7e3400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.740+0000 7f7e337fe640 1 -- 172.21.15.50:0/1921138055 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34004b20 con 0x7f7e3c102a00 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.740+0000 7f7e40d08640 1 -- 172.21.15.50:0/1921138055 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e3c1a2330 con 0x7f7e3c102a00 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.740+0000 7f7e337fe640 1 -- 172.21.15.50:0/1921138055 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e34004cc0 con 0x7f7e3c102a00 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.740+0000 7f7e337fe640 1 -- 172.21.15.50:0/1921138055 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34010670 con 0x7f7e3c102a00 2024-04-23T18:16:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.740+0000 7f7e40d08640 1 -- 172.21.15.50:0/1921138055 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e3c1a2830 con 0x7f7e3c102a00 2024-04-23T18:16:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.741+0000 7f7e40d08640 1 -- 172.21.15.50:0/1921138055 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e3c1a2b50 con 0x7f7e3c102a00 2024-04-23T18:16:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.742+0000 7f7e337fe640 1 -- 172.21.15.50:0/1921138055 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7e34005020 con 0x7f7e3c102a00 2024-04-23T18:16:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.742+0000 7f7e337fe640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7e280a3bb0 0x7f7e280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.743+0000 7f7e3a59c640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7e280a3bb0 0x7f7e280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.743+0000 7f7e337fe640 1 -- 172.21.15.50:0/1921138055 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f7e340d08a0 con 0x7f7e3c102a00 2024-04-23T18:16:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.743+0000 7f7e3a59c640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7e280a3bb0 0x7f7e280a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009230 tx=0x7f7e240314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:34.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.747+0000 7f7e337fe640 1 -- 172.21.15.50:0/1921138055 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e3409d3d0 con 0x7f7e3c102a00 2024-04-23T18:16:34.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.976+0000 7f7e40d08640 1 -- 172.21.15.50:0/1921138055 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f7e3c1a2e20 con 0x7f7e3c102a00 2024-04-23T18:16:34.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.982+0000 7f7e337fe640 1 -- 172.21.15.50:0/1921138055 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f7e3409ccd0 con 0x7f7e3c102a00 2024-04-23T18:16:34.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.985+0000 7f7e40d08640 1 -- 172.21.15.50:0/1921138055 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7e280a3bb0 msgr2=0x7f7e280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.985+0000 7f7e40d08640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7e280a3bb0 0x7f7e280a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009230 tx=0x7f7e240314a0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.985+0000 7f7e40d08640 1 -- 172.21.15.50:0/1921138055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e3c102a00 msgr2=0x7f7e3c1039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:34.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.985+0000 7f7e40d08640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e3c102a00 0x7f7e3c1039f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400eb10 tx=0x7f7e3400c2d0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.985+0000 7f7e40d08640 1 -- 172.21.15.50:0/1921138055 shutdown_connections 2024-04-23T18:16:34.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.985+0000 7f7e40d08640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7e280a3bb0 0x7f7e280a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.985+0000 7f7e40d08640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e3c102a00 0x7f7e3c1039f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.985+0000 7f7e40d08640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e3c102040 0x7f7e3c1034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.985+0000 7f7e40d08640 1 --2- 172.21.15.50:0/1921138055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e3c100e60 0x7f7e3c104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:34.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.985+0000 7f7e40d08640 1 -- 172.21.15.50:0/1921138055 >> 172.21.15.50:0/1921138055 conn(0x7f7e3c0fc440 msgr2=0x7f7e3c06a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:34.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.985+0000 7f7e40d08640 1 -- 172.21.15.50:0/1921138055 shutdown_connections 2024-04-23T18:16:34.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:34.985+0000 7f7e40d08640 1 -- 172.21.15.50:0/1921138055 wait complete. 2024-04-23T18:16:34.989 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-23T18:16:34.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-23T18:16:35.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.101+0000 7f783dd76640 1 -- 172.21.15.50:0/1321773396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7838103190 msgr2=0x7f7838103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.101+0000 7f783dd76640 1 --2- 172.21.15.50:0/1321773396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7838103190 0x7f7838103610 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7820009900 tx=0x7f782002a410 comp rx=0 tx=0).stop 2024-04-23T18:16:35.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.101+0000 7f783dd76640 1 -- 172.21.15.50:0/1321773396 shutdown_connections 2024-04-23T18:16:35.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.101+0000 7f783dd76640 1 --2- 172.21.15.50:0/1321773396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7838103b50 0x7f783810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.101+0000 7f783dd76640 1 --2- 172.21.15.50:0/1321773396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7838103190 0x7f7838103610 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.101+0000 7f783dd76640 1 --2- 172.21.15.50:0/1321773396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7838101fa0 0x7f78381023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.101+0000 7f783dd76640 1 -- 172.21.15.50:0/1321773396 >> 172.21.15.50:0/1321773396 conn(0x7f78380fd530 msgr2=0x7f78380ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.101+0000 7f783dd76640 1 -- 172.21.15.50:0/1321773396 shutdown_connections 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.101+0000 7f783dd76640 1 -- 172.21.15.50:0/1321773396 wait complete. 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.101+0000 7f783dd76640 1 Processor -- start 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.101+0000 7f783dd76640 1 -- start start 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f783dd76640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7838101fa0 0x7f78381981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f783dd76640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7838103190 0x7f78381986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f783dd76640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7838103b50 0x7f783819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f783dd76640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f783810dae0 con 0x7f7838103b50 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f783dd76640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f783810d960 con 0x7f7838103190 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f783dd76640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f783810dc60 con 0x7f7838101fa0 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f7837fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7838103b50 0x7f783819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f7837fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7838103b50 0x7f783819f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38654/0 (socket says 172.21.15.50:38654) 2024-04-23T18:16:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f7837fff640 1 -- 172.21.15.50:0/4290238380 learned_addr learned my addr 172.21.15.50:0/4290238380 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f782effd640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7838103190 0x7f78381986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f78377fe640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7838101fa0 0x7f78381981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f7837fff640 1 -- 172.21.15.50:0/4290238380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7838101fa0 msgr2=0x7f78381981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f7837fff640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7838101fa0 0x7f78381981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f7837fff640 1 -- 172.21.15.50:0/4290238380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7838103190 msgr2=0x7f78381986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f7837fff640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7838103190 0x7f78381986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f7837fff640 1 -- 172.21.15.50:0/4290238380 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783819fdd0 con 0x7f7838103b50 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f78377fe640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7838101fa0 0x7f78381981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f7837fff640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7838103b50 0x7f783819f760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f783000eaa0 tx=0x7f783000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.102+0000 7f78357fa640 1 -- 172.21.15.50:0/4290238380 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783000cdb0 con 0x7f7838103b50 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.103+0000 7f78357fa640 1 -- 172.21.15.50:0/4290238380 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7830005020 con 0x7f7838103b50 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.103+0000 7f783dd76640 1 -- 172.21.15.50:0/4290238380 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78381a00c0 con 0x7f7838103b50 2024-04-23T18:16:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.103+0000 7f78357fa640 1 -- 172.21.15.50:0/4290238380 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78300116d0 con 0x7f7838103b50 2024-04-23T18:16:35.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.103+0000 7f783dd76640 1 -- 172.21.15.50:0/4290238380 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78381a05e0 con 0x7f7838103b50 2024-04-23T18:16:35.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.104+0000 7f783dd76640 1 -- 172.21.15.50:0/4290238380 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78381093b0 con 0x7f7838103b50 2024-04-23T18:16:35.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.105+0000 7f78357fa640 1 -- 172.21.15.50:0/4290238380 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f78300051d0 con 0x7f7838103b50 2024-04-23T18:16:35.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.105+0000 7f78357fa640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f78240a3c30 0x7f78240a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:35.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.105+0000 7f78377fe640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f78240a3c30 0x7f78240a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:35.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.105+0000 7f78357fa640 1 -- 172.21.15.50:0/4290238380 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f78300d0470 con 0x7f7838103b50 2024-04-23T18:16:35.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.106+0000 7f78377fe640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f78240a3c30 0x7f78240a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f78280058c0 tx=0x7f7828005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:35.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.110+0000 7f78357fa640 1 -- 172.21.15.50:0/4290238380 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7830016030 con 0x7f7838103b50 2024-04-23T18:16:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:34 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/972586903' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T18:16:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:34 smithi177 ceph-mon[32308]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T18:16:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:34 smithi177 ceph-mon[32308]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T18:16:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:34 smithi177 ceph-mon[32308]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T18:16:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:34 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1098246585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:34 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1876544298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:34 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1921138055' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T18:16:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:34 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/972586903' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T18:16:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:34 smithi188 ceph-mon[32694]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T18:16:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:34 smithi188 ceph-mon[32694]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T18:16:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:34 smithi188 ceph-mon[32694]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T18:16:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:34 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1098246585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:34 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1876544298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:34 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1921138055' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T18:16:35.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.343+0000 7f783dd76640 1 -- 172.21.15.50:0/4290238380 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f7838062b50 con 0x7f7838103b50 2024-04-23T18:16:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:34 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/972586903' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T18:16:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:34 smithi050 ceph-mon[29187]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T18:16:35.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:34 smithi050 ceph-mon[29187]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T18:16:35.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:34 smithi050 ceph-mon[29187]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T18:16:35.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:34 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1098246585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:35.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:34 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1876544298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:35.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:34 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1921138055' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T18:16:35.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.947+0000 7f78357fa640 1 -- 172.21.15.50:0/4290238380 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7f783009cfa0 con 0x7f7838103b50 2024-04-23T18:16:35.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.950+0000 7f783dd76640 1 -- 172.21.15.50:0/4290238380 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f78240a3c30 msgr2=0x7f78240a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.950+0000 7f783dd76640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f78240a3c30 0x7f78240a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f78280058c0 tx=0x7f7828005850 comp rx=0 tx=0).stop 2024-04-23T18:16:35.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.950+0000 7f783dd76640 1 -- 172.21.15.50:0/4290238380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7838103b50 msgr2=0x7f783819f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:35.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.950+0000 7f783dd76640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7838103b50 0x7f783819f760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f783000eaa0 tx=0x7f783000ef60 comp rx=0 tx=0).stop 2024-04-23T18:16:35.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.951+0000 7f783dd76640 1 -- 172.21.15.50:0/4290238380 shutdown_connections 2024-04-23T18:16:35.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.951+0000 7f783dd76640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f78240a3c30 0x7f78240a60f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.951+0000 7f783dd76640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7838103b50 0x7f783819f760 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.951+0000 7f783dd76640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7838103190 0x7f78381986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.951+0000 7f783dd76640 1 --2- 172.21.15.50:0/4290238380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7838101fa0 0x7f78381981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:35.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.951+0000 7f783dd76640 1 -- 172.21.15.50:0/4290238380 >> 172.21.15.50:0/4290238380 conn(0x7f78380fd530 msgr2=0x7f7838106580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:35.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.951+0000 7f783dd76640 1 -- 172.21.15.50:0/4290238380 shutdown_connections 2024-04-23T18:16:35.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:35.951+0000 7f783dd76640 1 -- 172.21.15.50:0/4290238380 wait complete. 2024-04-23T18:16:35.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-23T18:16:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.082+0000 7f4320b20640 1 -- 172.21.15.50:0/2945006912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f431c101160 msgr2=0x7f431c1015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.082+0000 7f4320b20640 1 --2- 172.21.15.50:0/2945006912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f431c101160 0x7f431c1015c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4304009900 tx=0x7f430402a410 comp rx=0 tx=0).stop 2024-04-23T18:16:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.082+0000 7f4320b20640 1 -- 172.21.15.50:0/2945006912 shutdown_connections 2024-04-23T18:16:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.082+0000 7f4320b20640 1 --2- 172.21.15.50:0/2945006912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431c101b00 0x7f431c101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.082+0000 7f4320b20640 1 --2- 172.21.15.50:0/2945006912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f431c101160 0x7f431c1015c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.082+0000 7f4320b20640 1 --2- 172.21.15.50:0/2945006912 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f431c100800 0x7f431c100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.082+0000 7f4320b20640 1 -- 172.21.15.50:0/2945006912 >> 172.21.15.50:0/2945006912 conn(0x7f431c0fc400 msgr2=0x7f431c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.083+0000 7f4320b20640 1 -- 172.21.15.50:0/2945006912 shutdown_connections 2024-04-23T18:16:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.083+0000 7f4320b20640 1 -- 172.21.15.50:0/2945006912 wait complete. 2024-04-23T18:16:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.083+0000 7f4320b20640 1 Processor -- start 2024-04-23T18:16:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.083+0000 7f4320b20640 1 -- start start 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.083+0000 7f4320b20640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431c100800 0x7f431c19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.083+0000 7f4320b20640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f431c101160 0x7f431c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.083+0000 7f4320b20640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f431c101b00 0x7f431c1a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.083+0000 7f4320b20640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f431c10fce0 con 0x7f431c101160 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.083+0000 7f4320b20640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f431c10fb60 con 0x7f431c100800 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.083+0000 7f4320b20640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f431c10fe60 con 0x7f431c101b00 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f4319d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f431c101160 0x7f431c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f4319d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f431c101160 0x7f431c19a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38680/0 (socket says 172.21.15.50:38680) 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f4319d9b640 1 -- 172.21.15.50:0/2784644891 learned_addr learned my addr 172.21.15.50:0/2784644891 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f431ad9d640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f431c101b00 0x7f431c1a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f431a59c640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431c100800 0x7f431c19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f4319d9b640 1 -- 172.21.15.50:0/2784644891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f431c101b00 msgr2=0x7f431c1a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f4319d9b640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f431c101b00 0x7f431c1a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f4319d9b640 1 -- 172.21.15.50:0/2784644891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431c100800 msgr2=0x7f431c19a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f4319d9b640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431c100800 0x7f431c19a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f4319d9b640 1 -- 172.21.15.50:0/2784644891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431c1a2020 con 0x7f431c101160 2024-04-23T18:16:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f4319d9b640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f431c101160 0x7f431c19a940 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4304005940 tx=0x7f4304004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f43137fe640 1 -- 172.21.15.50:0/2784644891 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430403f030 con 0x7f431c101160 2024-04-23T18:16:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f43137fe640 1 -- 172.21.15.50:0/2784644891 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4304003cb0 con 0x7f431c101160 2024-04-23T18:16:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f4320b20640 1 -- 172.21.15.50:0/2784644891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f431c1a22b0 con 0x7f431c101160 2024-04-23T18:16:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f43137fe640 1 -- 172.21.15.50:0/2784644891 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304035070 con 0x7f431c101160 2024-04-23T18:16:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.084+0000 7f4320b20640 1 -- 172.21.15.50:0/2784644891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f431c1a27a0 con 0x7f431c101160 2024-04-23T18:16:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.085+0000 7f4320b20640 1 -- 172.21.15.50:0/2784644891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f431c10b7e0 con 0x7f431c101160 2024-04-23T18:16:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.087+0000 7f43137fe640 1 -- 172.21.15.50:0/2784644891 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4304042030 con 0x7f431c101160 2024-04-23T18:16:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.087+0000 7f43137fe640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43080a3bb0 0x7f43080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.088+0000 7f431a59c640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43080a3bb0 0x7f43080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.088+0000 7f43137fe640 1 -- 172.21.15.50:0/2784644891 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f43040f0c00 con 0x7f431c101160 2024-04-23T18:16:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.088+0000 7f431a59c640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43080a3bb0 0x7f43080a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4314005fd0 tx=0x7f4314005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:36.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.092+0000 7f43137fe640 1 -- 172.21.15.50:0/2784644891 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f43040f7080 con 0x7f431c101160 2024-04-23T18:16:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: pgmap v286: 97 pgs: 18 creating+peering, 78 unknown, 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T18:16:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: Cluster is now healthy 2024-04-23T18:16:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1921138055' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T18:16:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] up:active 2024-04-23T18:16:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:16:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T18:16:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4290238380' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T18:16:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-23T18:16:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-23T18:16:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4290238380' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-23T18:16:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-23T18:16:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:35 smithi177 ceph-mon[32308]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-23T18:16:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.320+0000 7f4320b20640 1 -- 172.21.15.50:0/2784644891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f431c02d810 con 0x7f431c101160 2024-04-23T18:16:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: pgmap v286: 97 pgs: 18 creating+peering, 78 unknown, 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T18:16:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: Cluster is now healthy 2024-04-23T18:16:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1921138055' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T18:16:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] up:active 2024-04-23T18:16:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:16:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T18:16:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4290238380' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T18:16:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-23T18:16:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-23T18:16:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4290238380' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-23T18:16:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-23T18:16:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:35 smithi188 ceph-mon[32694]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: pgmap v286: 97 pgs: 18 creating+peering, 78 unknown, 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: Cluster is now healthy 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1921138055' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] up:active 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4290238380' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4290238380' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-23T18:16:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:35 smithi050 ceph-mon[29187]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-23T18:16:36.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.954+0000 7f43137fe640 1 -- 172.21.15.50:0/2784644891 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7f4304003e60 con 0x7f431c101160 2024-04-23T18:16:36.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.957+0000 7f4320b20640 1 -- 172.21.15.50:0/2784644891 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43080a3bb0 msgr2=0x7f43080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:36.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.957+0000 7f4320b20640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43080a3bb0 0x7f43080a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4314005fd0 tx=0x7f4314005850 comp rx=0 tx=0).stop 2024-04-23T18:16:36.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.957+0000 7f4320b20640 1 -- 172.21.15.50:0/2784644891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f431c101160 msgr2=0x7f431c19a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:36.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.957+0000 7f4320b20640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f431c101160 0x7f431c19a940 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4304005940 tx=0x7f4304004430 comp rx=0 tx=0).stop 2024-04-23T18:16:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.957+0000 7f4320b20640 1 -- 172.21.15.50:0/2784644891 shutdown_connections 2024-04-23T18:16:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.957+0000 7f4320b20640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f43080a3bb0 0x7f43080a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.957+0000 7f4320b20640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f431c101b00 0x7f431c1a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.957+0000 7f4320b20640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f431c101160 0x7f431c19a940 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.957+0000 7f4320b20640 1 --2- 172.21.15.50:0/2784644891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431c100800 0x7f431c19a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.957+0000 7f4320b20640 1 -- 172.21.15.50:0/2784644891 >> 172.21.15.50:0/2784644891 conn(0x7f431c0fc400 msgr2=0x7f431c0fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.957+0000 7f4320b20640 1 -- 172.21.15.50:0/2784644891 shutdown_connections 2024-04-23T18:16:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:36.957+0000 7f4320b20640 1 -- 172.21.15.50:0/2784644891 wait complete. 2024-04-23T18:16:36.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-23T18:16:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.086+0000 7fe290b5d640 1 -- 172.21.15.50:0/399981822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28c1029a0 msgr2=0x7fe28c1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.086+0000 7fe290b5d640 1 --2- 172.21.15.50:0/399981822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28c1029a0 0x7fe28c1092c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe284009ca0 tx=0x7fe28402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.086+0000 7fe290b5d640 1 -- 172.21.15.50:0/399981822 shutdown_connections 2024-04-23T18:16:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.086+0000 7fe290b5d640 1 --2- 172.21.15.50:0/399981822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28c1029a0 0x7fe28c1092c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.086+0000 7fe290b5d640 1 --2- 172.21.15.50:0/399981822 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28c101fe0 0x7fe28c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.086+0000 7fe290b5d640 1 --2- 172.21.15.50:0/399981822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe28c100e90 0x7fe28c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.086+0000 7fe290b5d640 1 -- 172.21.15.50:0/399981822 >> 172.21.15.50:0/399981822 conn(0x7fe28c0fc400 msgr2=0x7fe28c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.086+0000 7fe290b5d640 1 -- 172.21.15.50:0/399981822 shutdown_connections 2024-04-23T18:16:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.086+0000 7fe290b5d640 1 -- 172.21.15.50:0/399981822 wait complete. 2024-04-23T18:16:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.086+0000 7fe290b5d640 1 Processor -- start 2024-04-23T18:16:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe290b5d640 1 -- start start 2024-04-23T18:16:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe290b5d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe28c100e90 0x7fe28c1970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe290b5d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28c101fe0 0x7fe28c197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe290b5d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28c1029a0 0x7fe28c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe290b5d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c10c9e0 con 0x7fe28c101fe0 2024-04-23T18:16:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe290b5d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c10c860 con 0x7fe28c100e90 2024-04-23T18:16:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe290b5d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c10cb60 con 0x7fe28c1029a0 2024-04-23T18:16:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe289d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28c101fe0 0x7fe28c197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe289d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28c101fe0 0x7fe28c197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38710/0 (socket says 172.21.15.50:38710) 2024-04-23T18:16:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe289d9b640 1 -- 172.21.15.50:0/2997998679 learned_addr learned my addr 172.21.15.50:0/2997998679 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe28a59c640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe28c100e90 0x7fe28c1970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe28ad9d640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28c1029a0 0x7fe28c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe289d9b640 1 -- 172.21.15.50:0/2997998679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28c1029a0 msgr2=0x7fe28c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe289d9b640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28c1029a0 0x7fe28c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe289d9b640 1 -- 172.21.15.50:0/2997998679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe28c100e90 msgr2=0x7fe28c1970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe289d9b640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe28c100e90 0x7fe28c1970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe289d9b640 1 -- 172.21.15.50:0/2997998679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c19ed00 con 0x7fe28c101fe0 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe28a59c640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe28c100e90 0x7fe28c1970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe289d9b640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28c101fe0 0x7fe28c197620 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe28000d950 tx=0x7fe28000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe27b7fe640 1 -- 172.21.15.50:0/2997998679 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280017070 con 0x7fe28c101fe0 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.087+0000 7fe27b7fe640 1 -- 172.21.15.50:0/2997998679 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe280009c30 con 0x7fe28c101fe0 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.088+0000 7fe290b5d640 1 -- 172.21.15.50:0/2997998679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28c19eff0 con 0x7fe28c101fe0 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.088+0000 7fe27b7fe640 1 -- 172.21.15.50:0/2997998679 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28000f620 con 0x7fe28c101fe0 2024-04-23T18:16:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.088+0000 7fe290b5d640 1 -- 172.21.15.50:0/2997998679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28c19f460 con 0x7fe28c101fe0 2024-04-23T18:16:37.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.089+0000 7fe290b5d640 1 -- 172.21.15.50:0/2997998679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe28c108210 con 0x7fe28c101fe0 2024-04-23T18:16:37.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.090+0000 7fe27b7fe640 1 -- 172.21.15.50:0/2997998679 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe28001a070 con 0x7fe28c101fe0 2024-04-23T18:16:37.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.090+0000 7fe27b7fe640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe2740a3bb0 0x7fe2740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:37.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.091+0000 7fe28a59c640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe2740a3bb0 0x7fe2740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:37.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.091+0000 7fe27b7fe640 1 -- 172.21.15.50:0/2997998679 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fe2800cfc80 con 0x7fe28c101fe0 2024-04-23T18:16:37.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.091+0000 7fe28a59c640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe2740a3bb0 0x7fe2740a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009fc0 tx=0x7fe27c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:37.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.095+0000 7fe27b7fe640 1 -- 172.21.15.50:0/2997998679 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe280009de0 con 0x7fe28c101fe0 2024-04-23T18:16:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:36 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2784644891' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T18:16:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:36 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2784644891' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T18:16:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:36 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] up:active 2024-04-23T18:16:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:36 smithi177 ceph-mon[32308]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-23T18:16:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:36 smithi177 ceph-mon[32308]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-23T18:16:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:36 smithi177 ceph-mon[32308]: Cluster is now healthy 2024-04-23T18:16:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:36 smithi177 ceph-mon[32308]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-23T18:16:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:36 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-23T18:16:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:36 smithi177 ceph-mon[32308]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-23T18:16:37.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.323+0000 7fe290b5d640 1 -- 172.21.15.50:0/2997998679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fe28c02d810 con 0x7fe28c101fe0 2024-04-23T18:16:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:36 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2784644891' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T18:16:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:36 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2784644891' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T18:16:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:36 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] up:active 2024-04-23T18:16:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:36 smithi188 ceph-mon[32694]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-23T18:16:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:36 smithi188 ceph-mon[32694]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-23T18:16:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:36 smithi188 ceph-mon[32694]: Cluster is now healthy 2024-04-23T18:16:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:36 smithi188 ceph-mon[32694]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-23T18:16:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:36 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-23T18:16:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:36 smithi188 ceph-mon[32694]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-23T18:16:37.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:36 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2784644891' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T18:16:37.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:36 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2784644891' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T18:16:37.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:36 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] up:active 2024-04-23T18:16:37.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:36 smithi050 ceph-mon[29187]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-23T18:16:37.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:36 smithi050 ceph-mon[29187]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-23T18:16:37.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:36 smithi050 ceph-mon[29187]: Cluster is now healthy 2024-04-23T18:16:37.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:36 smithi050 ceph-mon[29187]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-23T18:16:37.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:36 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-23T18:16:37.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:36 smithi050 ceph-mon[29187]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-23T18:16:37.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.960+0000 7fe27b7fe640 1 -- 172.21.15.50:0/2997998679 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) ==== 115+0+0 (secure 0 0 0) 0x7fe28009c160 con 0x7fe28c101fe0 2024-04-23T18:16:37.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.963+0000 7fe290b5d640 1 -- 172.21.15.50:0/2997998679 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe2740a3bb0 msgr2=0x7fe2740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:37.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.963+0000 7fe290b5d640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe2740a3bb0 0x7fe2740a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009fc0 tx=0x7fe27c005f90 comp rx=0 tx=0).stop 2024-04-23T18:16:37.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.963+0000 7fe290b5d640 1 -- 172.21.15.50:0/2997998679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28c101fe0 msgr2=0x7fe28c197620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:37.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.963+0000 7fe290b5d640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28c101fe0 0x7fe28c197620 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe28000d950 tx=0x7fe28000de10 comp rx=0 tx=0).stop 2024-04-23T18:16:37.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.963+0000 7fe290b5d640 1 -- 172.21.15.50:0/2997998679 shutdown_connections 2024-04-23T18:16:37.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.963+0000 7fe290b5d640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fe2740a3bb0 0x7fe2740a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.963+0000 7fe290b5d640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28c1029a0 0x7fe28c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.963+0000 7fe290b5d640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28c101fe0 0x7fe28c197620 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.963+0000 7fe290b5d640 1 --2- 172.21.15.50:0/2997998679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe28c100e90 0x7fe28c1970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:37.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.963+0000 7fe290b5d640 1 -- 172.21.15.50:0/2997998679 >> 172.21.15.50:0/2997998679 conn(0x7fe28c0fc400 msgr2=0x7fe28c1053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:37.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.963+0000 7fe290b5d640 1 -- 172.21.15.50:0/2997998679 shutdown_connections 2024-04-23T18:16:37.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:37.963+0000 7fe290b5d640 1 -- 172.21.15.50:0/2997998679 wait complete. 2024-04-23T18:16:37.971 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-23T18:16:37.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-23T18:16:38.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 -- 172.21.15.50:0/237558404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc200103b60 msgr2=0x7fc20010a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 --2- 172.21.15.50:0/237558404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc200103b60 0x7fc20010a3f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8009ca0 tx=0x7fc1f802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 -- 172.21.15.50:0/237558404 shutdown_connections 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 --2- 172.21.15.50:0/237558404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc200103b60 0x7fc20010a3f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 --2- 172.21.15.50:0/237558404 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2001031a0 0x7fc200103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 --2- 172.21.15.50:0/237558404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc200101fb0 0x7fc2001023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 -- 172.21.15.50:0/237558404 >> 172.21.15.50:0/237558404 conn(0x7fc2000fd520 msgr2=0x7fc2000ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 -- 172.21.15.50:0/237558404 shutdown_connections 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 -- 172.21.15.50:0/237558404 wait complete. 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 Processor -- start 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 -- start start 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc200101fb0 0x7fc200198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.090+0000 7fc2078e8640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2001031a0 0x7fc2001986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc2078e8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc200103b60 0x7fc20019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc2078e8640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc20010daf0 con 0x7fc200101fb0 2024-04-23T18:16:38.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc2078e8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc20010d970 con 0x7fc200103b60 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc2078e8640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc20010dc70 con 0x7fc2001031a0 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc205684640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc200101fb0 0x7fc200198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc205684640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc200101fb0 0x7fc200198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38734/0 (socket says 172.21.15.50:38734) 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc205684640 1 -- 172.21.15.50:0/1848824373 learned_addr learned my addr 172.21.15.50:0/1848824373 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc205e85640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc200103b60 0x7fc20019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc205684640 1 -- 172.21.15.50:0/1848824373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2001031a0 msgr2=0x7fc2001986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc204e83640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2001031a0 0x7fc2001986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc205684640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2001031a0 0x7fc2001986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc205684640 1 -- 172.21.15.50:0/1848824373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc200103b60 msgr2=0x7fc20019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc205684640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc200103b60 0x7fc20019f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc205684640 1 -- 172.21.15.50:0/1848824373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc20019fd90 con 0x7fc200101fb0 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc204e83640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2001031a0 0x7fc2001986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc205684640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc200101fb0 0x7fc200198170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc00e9c0 tx=0x7fc1fc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc1f67fc640 1 -- 172.21.15.50:0/1848824373 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc00cd80 con 0x7fc200101fb0 2024-04-23T18:16:38.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc2078e8640 1 -- 172.21.15.50:0/1848824373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2001a0080 con 0x7fc200101fb0 2024-04-23T18:16:38.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc1f67fc640 1 -- 172.21.15.50:0/1848824373 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1fc0035f0 con 0x7fc200101fb0 2024-04-23T18:16:38.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc1f67fc640 1 -- 172.21.15.50:0/1848824373 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc010690 con 0x7fc200101fb0 2024-04-23T18:16:38.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.091+0000 7fc2078e8640 1 -- 172.21.15.50:0/1848824373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2001a05d0 con 0x7fc200101fb0 2024-04-23T18:16:38.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.092+0000 7fc2078e8640 1 -- 172.21.15.50:0/1848824373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2001093a0 con 0x7fc200101fb0 2024-04-23T18:16:38.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.094+0000 7fc1f67fc640 1 -- 172.21.15.50:0/1848824373 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc1fc0037a0 con 0x7fc200101fb0 2024-04-23T18:16:38.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.094+0000 7fc1f67fc640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc1ec0a3bb0 0x7fc1ec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:38.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.094+0000 7fc204e83640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc1ec0a3bb0 0x7fc1ec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:38.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.094+0000 7fc1f67fc640 1 -- 172.21.15.50:0/1848824373 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fc1fc0d07e0 con 0x7fc200101fb0 2024-04-23T18:16:38.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.095+0000 7fc204e83640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc1ec0a3bb0 0x7fc1ec0a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc200199680 tx=0x7fc1f0005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:38.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.098+0000 7fc1f67fc640 1 -- 172.21.15.50:0/1848824373 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc1fc09d310 con 0x7fc200101fb0 2024-04-23T18:16:38.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:37 smithi177 ceph-mon[32308]: pgmap v288: 97 pgs: 18 creating+peering, 34 unknown, 45 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 898 B/s wr, 7 op/s 2024-04-23T18:16:38.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:37 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2997998679' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T18:16:38.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:37 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2997998679' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T18:16:38.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:37 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] up:active 2024-04-23T18:16:38.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:37 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-23T18:16:38.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:37 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:16:38.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.310+0000 7fc2078e8640 1 -- 172.21.15.50:0/1848824373 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc200062390 con 0x7fc1ec0a3bb0 2024-04-23T18:16:38.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:37 smithi050 ceph-mon[29187]: pgmap v288: 97 pgs: 18 creating+peering, 34 unknown, 45 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 898 B/s wr, 7 op/s 2024-04-23T18:16:38.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:37 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2997998679' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T18:16:38.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:37 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2997998679' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T18:16:38.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:37 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] up:active 2024-04-23T18:16:38.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:37 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-23T18:16:38.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:37 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:16:38.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:37 smithi188 ceph-mon[32694]: pgmap v288: 97 pgs: 18 creating+peering, 34 unknown, 45 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 898 B/s wr, 7 op/s 2024-04-23T18:16:38.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:37 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2997998679' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T18:16:38.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:37 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2997998679' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T18:16:38.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:37 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] up:active 2024-04-23T18:16:38.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:37 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-23T18:16:38.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:37 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:16:38.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.337+0000 7fc1f67fc640 1 -- 172.21.15.50:0/1848824373 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc200062390 con 0x7fc1ec0a3bb0 2024-04-23T18:16:38.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.339+0000 7fc2078e8640 1 -- 172.21.15.50:0/1848824373 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc1ec0a3bb0 msgr2=0x7fc1ec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.339+0000 7fc2078e8640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc1ec0a3bb0 0x7fc1ec0a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc200199680 tx=0x7fc1f0005d30 comp rx=0 tx=0).stop 2024-04-23T18:16:38.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.339+0000 7fc2078e8640 1 -- 172.21.15.50:0/1848824373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc200101fb0 msgr2=0x7fc200198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.339+0000 7fc2078e8640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc200101fb0 0x7fc200198170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc00e9c0 tx=0x7fc1fc00ee80 comp rx=0 tx=0).stop 2024-04-23T18:16:38.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.339+0000 7fc2078e8640 1 -- 172.21.15.50:0/1848824373 shutdown_connections 2024-04-23T18:16:38.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.339+0000 7fc2078e8640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc1ec0a3bb0 0x7fc1ec0a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.339+0000 7fc2078e8640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc200103b60 0x7fc20019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.339+0000 7fc2078e8640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2001031a0 0x7fc2001986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.339+0000 7fc2078e8640 1 --2- 172.21.15.50:0/1848824373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc200101fb0 0x7fc200198170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.339+0000 7fc2078e8640 1 -- 172.21.15.50:0/1848824373 >> 172.21.15.50:0/1848824373 conn(0x7fc2000fd520 msgr2=0x7fc200106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:38.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.339+0000 7fc2078e8640 1 -- 172.21.15.50:0/1848824373 shutdown_connections 2024-04-23T18:16:38.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.339+0000 7fc2078e8640 1 -- 172.21.15.50:0/1848824373 wait complete. 2024-04-23T18:16:38.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-23T18:16:38.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.456+0000 7ff9b3519640 1 -- 172.21.15.50:0/3481955925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9ac1029a0 msgr2=0x7ff9ac1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.456+0000 7ff9b3519640 1 --2- 172.21.15.50:0/3481955925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9ac1029a0 0x7ff9ac1092c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009ca0 tx=0x7ff9a802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.456+0000 7ff9b3519640 1 -- 172.21.15.50:0/3481955925 shutdown_connections 2024-04-23T18:16:38.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.456+0000 7ff9b3519640 1 --2- 172.21.15.50:0/3481955925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9ac1029a0 0x7ff9ac1092c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.456+0000 7ff9b3519640 1 --2- 172.21.15.50:0/3481955925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ac101fe0 0x7ff9ac102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.456+0000 7ff9b3519640 1 --2- 172.21.15.50:0/3481955925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9ac100e90 0x7ff9ac1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.456+0000 7ff9b3519640 1 -- 172.21.15.50:0/3481955925 >> 172.21.15.50:0/3481955925 conn(0x7ff9ac0fc400 msgr2=0x7ff9ac0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:38.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.457+0000 7ff9b3519640 1 -- 172.21.15.50:0/3481955925 shutdown_connections 2024-04-23T18:16:38.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.457+0000 7ff9b3519640 1 -- 172.21.15.50:0/3481955925 wait complete. 2024-04-23T18:16:38.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.457+0000 7ff9b3519640 1 Processor -- start 2024-04-23T18:16:38.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.457+0000 7ff9b3519640 1 -- start start 2024-04-23T18:16:38.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.457+0000 7ff9b3519640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9ac100e90 0x7ff9ac1970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:38.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.457+0000 7ff9b3519640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ac101fe0 0x7ff9ac197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:38.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.457+0000 7ff9b3519640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9ac1029a0 0x7ff9ac19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:38.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.457+0000 7ff9b3519640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ac10c9f0 con 0x7ff9ac1029a0 2024-04-23T18:16:38.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.457+0000 7ff9b3519640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ac10c870 con 0x7ff9ac101fe0 2024-04-23T18:16:38.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.457+0000 7ff9b3519640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ac10cb70 con 0x7ff9ac100e90 2024-04-23T18:16:38.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.457+0000 7ff9b1ab6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9ac1029a0 0x7ff9ac19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:38.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b1ab6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9ac1029a0 0x7ff9ac19e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38764/0 (socket says 172.21.15.50:38764) 2024-04-23T18:16:38.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b1ab6640 1 -- 172.21.15.50:0/3631494466 learned_addr learned my addr 172.21.15.50:0/3631494466 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:38.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b0ab4640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ac101fe0 0x7ff9ac197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:38.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b12b5640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9ac100e90 0x7ff9ac1970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:38.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b1ab6640 1 -- 172.21.15.50:0/3631494466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9ac100e90 msgr2=0x7ff9ac1970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b1ab6640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9ac100e90 0x7ff9ac1970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b1ab6640 1 -- 172.21.15.50:0/3631494466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ac101fe0 msgr2=0x7ff9ac197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b1ab6640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ac101fe0 0x7ff9ac197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b1ab6640 1 -- 172.21.15.50:0/3631494466 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9ac19ecf0 con 0x7ff9ac1029a0 2024-04-23T18:16:38.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b0ab4640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ac101fe0 0x7ff9ac197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:38.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b1ab6640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9ac1029a0 0x7ff9ac19e680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff9a80097b0 tx=0x7ff9a8031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:38.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9a27fc640 1 -- 172.21.15.50:0/3631494466 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8003bf0 con 0x7ff9ac1029a0 2024-04-23T18:16:38.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9a27fc640 1 -- 172.21.15.50:0/3631494466 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9a8003d90 con 0x7ff9ac1029a0 2024-04-23T18:16:38.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b3519640 1 -- 172.21.15.50:0/3631494466 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9ac19ef80 con 0x7ff9ac1029a0 2024-04-23T18:16:38.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9a27fc640 1 -- 172.21.15.50:0/3631494466 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a802fc70 con 0x7ff9ac1029a0 2024-04-23T18:16:38.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.458+0000 7ff9b3519640 1 -- 172.21.15.50:0/3631494466 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9ac19f490 con 0x7ff9ac1029a0 2024-04-23T18:16:38.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.459+0000 7ff9b3519640 1 -- 172.21.15.50:0/3631494466 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9ac1081c0 con 0x7ff9ac1029a0 2024-04-23T18:16:38.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.460+0000 7ff9a27fc640 1 -- 172.21.15.50:0/3631494466 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff9a8004040 con 0x7ff9ac1029a0 2024-04-23T18:16:38.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.461+0000 7ff9a27fc640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff9980a3c30 0x7ff9980a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:38.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.461+0000 7ff9b12b5640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff9980a3c30 0x7ff9980a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:38.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.461+0000 7ff9a27fc640 1 -- 172.21.15.50:0/3631494466 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7ff9a80f17b0 con 0x7ff9ac1029a0 2024-04-23T18:16:38.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.461+0000 7ff9b12b5640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff9980a3c30 0x7ff9980a60f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4005fd0 tx=0x7ff9a400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:38.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.465+0000 7ff9a27fc640 1 -- 172.21.15.50:0/3631494466 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff9a80be2e0 con 0x7ff9ac1029a0 2024-04-23T18:16:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.682+0000 7ff9b3519640 1 -- 172.21.15.50:0/3631494466 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff9ac062390 con 0x7ff9980a3c30 2024-04-23T18:16:38.695 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:16:38 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:16:38.315+0000 7ff6717e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:16:38.695 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:16:38 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:16:38.315+0000 7ff6717e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:16:38.695 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:16:38 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:16:38.315+0000 7ff6717e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:16:38.695 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:16:38 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:16:38.315+0000 7ff6717e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:16:38.695 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:16:38 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:16:38.315+0000 7ff6717e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:16:38.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.707+0000 7ff9a27fc640 1 -- 172.21.15.50:0/3631494466 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff9ac062390 con 0x7ff9980a3c30 2024-04-23T18:16:38.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.708+0000 7ff9b3519640 1 -- 172.21.15.50:0/3631494466 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff9980a3c30 msgr2=0x7ff9980a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.708+0000 7ff9b3519640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff9980a3c30 0x7ff9980a60f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4005fd0 tx=0x7ff9a400b390 comp rx=0 tx=0).stop 2024-04-23T18:16:38.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.708+0000 7ff9b3519640 1 -- 172.21.15.50:0/3631494466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9ac1029a0 msgr2=0x7ff9ac19e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.708+0000 7ff9b3519640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9ac1029a0 0x7ff9ac19e680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff9a80097b0 tx=0x7ff9a8031840 comp rx=0 tx=0).stop 2024-04-23T18:16:38.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.708+0000 7ff9b3519640 1 -- 172.21.15.50:0/3631494466 shutdown_connections 2024-04-23T18:16:38.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.709+0000 7ff9b3519640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff9980a3c30 0x7ff9980a60f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.709+0000 7ff9b3519640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9ac1029a0 0x7ff9ac19e680 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.709+0000 7ff9b3519640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ac101fe0 0x7ff9ac197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.709+0000 7ff9b3519640 1 --2- 172.21.15.50:0/3631494466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9ac100e90 0x7ff9ac1970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.709+0000 7ff9b3519640 1 -- 172.21.15.50:0/3631494466 >> 172.21.15.50:0/3631494466 conn(0x7ff9ac0fc400 msgr2=0x7ff9ac105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:38.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.709+0000 7ff9b3519640 1 -- 172.21.15.50:0/3631494466 shutdown_connections 2024-04-23T18:16:38.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.709+0000 7ff9b3519640 1 -- 172.21.15.50:0/3631494466 wait complete. 2024-04-23T18:16:38.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-23T18:16:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.826+0000 7fa88df4a640 1 -- 172.21.15.50:0/2119897701 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa888102a30 msgr2=0x7fa888104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.826+0000 7fa88df4a640 1 --2- 172.21.15.50:0/2119897701 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa888102a30 0x7fa888104e20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa878009900 tx=0x7fa87802a410 comp rx=0 tx=0).stop 2024-04-23T18:16:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 -- 172.21.15.50:0/2119897701 shutdown_connections 2024-04-23T18:16:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 --2- 172.21.15.50:0/2119897701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa888107c90 0x7fa88810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 --2- 172.21.15.50:0/2119897701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888105360 0x7fa888107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 --2- 172.21.15.50:0/2119897701 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa888102a30 0x7fa888104e20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 -- 172.21.15.50:0/2119897701 >> 172.21.15.50:0/2119897701 conn(0x7fa8880fc440 msgr2=0x7fa8880fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 -- 172.21.15.50:0/2119897701 shutdown_connections 2024-04-23T18:16:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 -- 172.21.15.50:0/2119897701 wait complete. 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 Processor -- start 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 -- start start 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa888102a30 0x7fa888198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa888105360 0x7fa8881987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888107c90 0x7fa88819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa88810daf0 con 0x7fa888102a30 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa88810d970 con 0x7fa888105360 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.827+0000 7fa88df4a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa88810dc70 con 0x7fa888107c90 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa8877fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa888102a30 0x7fa888198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa8877fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa888102a30 0x7fa888198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38782/0 (socket says 172.21.15.50:38782) 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa886ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa888105360 0x7fa8881987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa8877fe640 1 -- 172.21.15.50:0/3974956329 learned_addr learned my addr 172.21.15.50:0/3974956329 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa887fff640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888107c90 0x7fa88819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa8877fe640 1 -- 172.21.15.50:0/3974956329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888107c90 msgr2=0x7fa88819f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa8877fe640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888107c90 0x7fa88819f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa8877fe640 1 -- 172.21.15.50:0/3974956329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa888105360 msgr2=0x7fa8881987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa8877fe640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa888105360 0x7fa8881987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa8877fe640 1 -- 172.21.15.50:0/3974956329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88819fe90 con 0x7fa888102a30 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa886ffd640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa888105360 0x7fa8881987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa887fff640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888107c90 0x7fa88819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa8877fe640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa888102a30 0x7fa888198270 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa878005940 tx=0x7fa878031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa884ff9640 1 -- 172.21.15.50:0/3974956329 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87803f030 con 0x7fa888102a30 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa884ff9640 1 -- 172.21.15.50:0/3974956329 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa878003aa0 con 0x7fa888102a30 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa88df4a640 1 -- 172.21.15.50:0/3974956329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8881a0120 con 0x7fa888102a30 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa884ff9640 1 -- 172.21.15.50:0/3974956329 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878035070 con 0x7fa888102a30 2024-04-23T18:16:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.828+0000 7fa88df4a640 1 -- 172.21.15.50:0/3974956329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8881a0610 con 0x7fa888102a30 2024-04-23T18:16:38.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.829+0000 7fa88df4a640 1 -- 172.21.15.50:0/3974956329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8880fdf30 con 0x7fa888102a30 2024-04-23T18:16:38.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.831+0000 7fa884ff9640 1 -- 172.21.15.50:0/3974956329 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa878042030 con 0x7fa888102a30 2024-04-23T18:16:38.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.831+0000 7fa884ff9640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa8700a3bb0 0x7fa8700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:38.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.831+0000 7fa886ffd640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa8700a3bb0 0x7fa8700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:38.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.832+0000 7fa884ff9640 1 -- 172.21.15.50:0/3974956329 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fa8780f08f0 con 0x7fa888102a30 2024-04-23T18:16:38.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.832+0000 7fa886ffd640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa8700a3bb0 0x7fa8700a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa888199780 tx=0x7fa87c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:38.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:38.836+0000 7fa884ff9640 1 -- 172.21.15.50:0/3974956329 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa8780bd420 con 0x7fa888102a30 2024-04-23T18:16:38.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.003+0000 7fa884ff9640 1 -- 172.21.15.50:0/3974956329 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa8780b9390 con 0x7fa888102a30 2024-04-23T18:16:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.058+0000 7fa88df4a640 1 -- 172.21.15.50:0/3974956329 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fa888062420 con 0x7fa8700a3bb0 2024-04-23T18:16:39.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.063+0000 7fa884ff9640 1 -- 172.21.15.50:0/3974956329 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fa888062420 con 0x7fa8700a3bb0 2024-04-23T18:16:39.057 INFO:teuthology.orchestra.run.smithi050.stdout:/volumes/qa/sv_0/2d806385-8a0b-467a-a9ba-9bae1ebd38aa 2024-04-23T18:16:39.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.065+0000 7fa88df4a640 1 -- 172.21.15.50:0/3974956329 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa8700a3bb0 msgr2=0x7fa8700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.065+0000 7fa88df4a640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa8700a3bb0 0x7fa8700a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa888199780 tx=0x7fa87c00b390 comp rx=0 tx=0).stop 2024-04-23T18:16:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.065+0000 7fa88df4a640 1 -- 172.21.15.50:0/3974956329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa888102a30 msgr2=0x7fa888198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.065+0000 7fa88df4a640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa888102a30 0x7fa888198270 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa878005940 tx=0x7fa878031a50 comp rx=0 tx=0).stop 2024-04-23T18:16:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.065+0000 7fa88df4a640 1 -- 172.21.15.50:0/3974956329 shutdown_connections 2024-04-23T18:16:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.065+0000 7fa88df4a640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa8700a3bb0 0x7fa8700a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.065+0000 7fa88df4a640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa888107c90 0x7fa88819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.065+0000 7fa88df4a640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa888105360 0x7fa8881987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.065+0000 7fa88df4a640 1 --2- 172.21.15.50:0/3974956329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa888102a30 0x7fa888198270 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.065+0000 7fa88df4a640 1 -- 172.21.15.50:0/3974956329 >> 172.21.15.50:0/3974956329 conn(0x7fa8880fc440 msgr2=0x7fa888103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.066+0000 7fa88df4a640 1 -- 172.21.15.50:0/3974956329 shutdown_connections 2024-04-23T18:16:39.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.066+0000 7fa88df4a640 1 -- 172.21.15.50:0/3974956329 wait complete. 2024-04-23T18:16:39.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-23T18:16:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.184+0000 7ff61a4a0640 1 -- 172.21.15.50:0/3958423044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff614102a30 msgr2=0x7ff614104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.184+0000 7ff61a4a0640 1 --2- 172.21.15.50:0/3958423044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff614102a30 0x7ff614104e20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff604009900 tx=0x7ff60402a410 comp rx=0 tx=0).stop 2024-04-23T18:16:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.184+0000 7ff61a4a0640 1 -- 172.21.15.50:0/3958423044 shutdown_connections 2024-04-23T18:16:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.184+0000 7ff61a4a0640 1 --2- 172.21.15.50:0/3958423044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff614107c90 0x7ff61410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.184+0000 7ff61a4a0640 1 --2- 172.21.15.50:0/3958423044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff614105360 0x7ff614107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.184+0000 7ff61a4a0640 1 --2- 172.21.15.50:0/3958423044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff614102a30 0x7ff614104e20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.184+0000 7ff61a4a0640 1 -- 172.21.15.50:0/3958423044 >> 172.21.15.50:0/3958423044 conn(0x7ff6140fc440 msgr2=0x7ff6140fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.185+0000 7ff61a4a0640 1 -- 172.21.15.50:0/3958423044 shutdown_connections 2024-04-23T18:16:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.185+0000 7ff61a4a0640 1 -- 172.21.15.50:0/3958423044 wait complete. 2024-04-23T18:16:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.185+0000 7ff61a4a0640 1 Processor -- start 2024-04-23T18:16:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.185+0000 7ff61a4a0640 1 -- start start 2024-04-23T18:16:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.185+0000 7ff61a4a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff614102a30 0x7ff6141023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.185+0000 7ff61a4a0640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff614105360 0x7ff6141009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.185+0000 7ff61a4a0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff614107c90 0x7ff614100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.185+0000 7ff61a4a0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff61410daf0 con 0x7ff614102a30 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.185+0000 7ff61a4a0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff61410d970 con 0x7ff614107c90 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.185+0000 7ff61a4a0640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff61410dc70 con 0x7ff614105360 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff613fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff614102a30 0x7ff6141023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff613fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff614102a30 0x7ff6141023a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38796/0 (socket says 172.21.15.50:38796) 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff613fff640 1 -- 172.21.15.50:0/2602287092 learned_addr learned my addr 172.21.15.50:0/2602287092 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff618a3d640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff614107c90 0x7ff614100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff613fff640 1 -- 172.21.15.50:0/2602287092 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff614105360 msgr2=0x7ff6141009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff6137fe640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff614105360 0x7ff6141009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff613fff640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff614105360 0x7ff6141009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff613fff640 1 -- 172.21.15.50:0/2602287092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff614107c90 msgr2=0x7ff614100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff613fff640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff614107c90 0x7ff614100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff613fff640 1 -- 172.21.15.50:0/2602287092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6141017b0 con 0x7ff614102a30 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff613fff640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff614102a30 0x7ff6141023a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff604005940 tx=0x7ff604031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff6117fa640 1 -- 172.21.15.50:0/2602287092 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60403f030 con 0x7ff614102a30 2024-04-23T18:16:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff6117fa640 1 -- 172.21.15.50:0/2602287092 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff604003aa0 con 0x7ff614102a30 2024-04-23T18:16:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff61a4a0640 1 -- 172.21.15.50:0/2602287092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6141a22b0 con 0x7ff614102a30 2024-04-23T18:16:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff6117fa640 1 -- 172.21.15.50:0/2602287092 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604035070 con 0x7ff614102a30 2024-04-23T18:16:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.186+0000 7ff61a4a0640 1 -- 172.21.15.50:0/2602287092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6141a2800 con 0x7ff614102a30 2024-04-23T18:16:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.187+0000 7ff61a4a0640 1 -- 172.21.15.50:0/2602287092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6140fdf30 con 0x7ff614102a30 2024-04-23T18:16:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.188+0000 7ff6117fa640 1 -- 172.21.15.50:0/2602287092 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff604042030 con 0x7ff614102a30 2024-04-23T18:16:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.189+0000 7ff6117fa640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6000a3c40 0x7ff6000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.189+0000 7ff6137fe640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6000a3c40 0x7ff6000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.189+0000 7ff6117fa640 1 -- 172.21.15.50:0/2602287092 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7ff6040f0bd0 con 0x7ff614102a30 2024-04-23T18:16:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.190+0000 7ff6137fe640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6000a3c40 0x7ff6000a6100 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff614102110 tx=0x7ff5fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:39.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.194+0000 7ff6117fa640 1 -- 172.21.15.50:0/2602287092 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6040bd500 con 0x7ff614102a30 2024-04-23T18:16:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.414+0000 7ff61a4a0640 1 -- 172.21.15.50:0/2602287092 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff614062420 con 0x7ff6000a3c40 2024-04-23T18:16:39.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.438+0000 7ff6117fa640 1 -- 172.21.15.50:0/2602287092 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff614062420 con 0x7ff6000a3c40 2024-04-23T18:16:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.440+0000 7ff61a4a0640 1 -- 172.21.15.50:0/2602287092 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6000a3c40 msgr2=0x7ff6000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.440+0000 7ff61a4a0640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6000a3c40 0x7ff6000a6100 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff614102110 tx=0x7ff5fc005f90 comp rx=0 tx=0).stop 2024-04-23T18:16:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.440+0000 7ff61a4a0640 1 -- 172.21.15.50:0/2602287092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff614102a30 msgr2=0x7ff6141023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.440+0000 7ff61a4a0640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff614102a30 0x7ff6141023a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff604005940 tx=0x7ff604031a50 comp rx=0 tx=0).stop 2024-04-23T18:16:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.440+0000 7ff61a4a0640 1 -- 172.21.15.50:0/2602287092 shutdown_connections 2024-04-23T18:16:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.440+0000 7ff61a4a0640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6000a3c40 0x7ff6000a6100 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.440+0000 7ff61a4a0640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff614107c90 0x7ff614100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.440+0000 7ff61a4a0640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff614105360 0x7ff6141009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.440+0000 7ff61a4a0640 1 --2- 172.21.15.50:0/2602287092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff614102a30 0x7ff6141023a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.440+0000 7ff61a4a0640 1 -- 172.21.15.50:0/2602287092 >> 172.21.15.50:0/2602287092 conn(0x7ff6140fc440 msgr2=0x7ff614103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.440+0000 7ff61a4a0640 1 -- 172.21.15.50:0/2602287092 shutdown_connections 2024-04-23T18:16:39.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.440+0000 7ff61a4a0640 1 -- 172.21.15.50:0/2602287092 wait complete. 2024-04-23T18:16:39.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-23T18:16:39.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.557+0000 7fa407205640 1 -- 172.21.15.50:0/4255056185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa400100e90 msgr2=0x7fa4001012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.557+0000 7fa407205640 1 --2- 172.21.15.50:0/4255056185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa400100e90 0x7fa4001012b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009900 tx=0x7fa3f402a410 comp rx=0 tx=0).stop 2024-04-23T18:16:39.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 -- 172.21.15.50:0/4255056185 shutdown_connections 2024-04-23T18:16:39.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 --2- 172.21.15.50:0/4255056185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4001029a0 0x7fa4001092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 --2- 172.21.15.50:0/4255056185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa400101fe0 0x7fa400102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 --2- 172.21.15.50:0/4255056185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa400100e90 0x7fa4001012b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 -- 172.21.15.50:0/4255056185 >> 172.21.15.50:0/4255056185 conn(0x7fa4000fc400 msgr2=0x7fa4000fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:39.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 -- 172.21.15.50:0/4255056185 shutdown_connections 2024-04-23T18:16:39.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 -- 172.21.15.50:0/4255056185 wait complete. 2024-04-23T18:16:39.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 Processor -- start 2024-04-23T18:16:39.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 -- start start 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa400100e90 0x7fa400197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa400101fe0 0x7fa400197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4001029a0 0x7fa40019e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa40010c9c0 con 0x7fa4001029a0 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa40010c840 con 0x7fa400101fe0 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa407205640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa40010cb40 con 0x7fa400100e90 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa4057a2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4001029a0 0x7fa40019e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa4057a2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4001029a0 0x7fa40019e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38820/0 (socket says 172.21.15.50:38820) 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa4057a2640 1 -- 172.21.15.50:0/768966542 learned_addr learned my addr 172.21.15.50:0/768966542 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.558+0000 7fa404fa1640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa400100e90 0x7fa400197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.559+0000 7fa3fffff640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa400101fe0 0x7fa400197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.559+0000 7fa4057a2640 1 -- 172.21.15.50:0/768966542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa400100e90 msgr2=0x7fa400197030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.559+0000 7fa4057a2640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa400100e90 0x7fa400197030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.559+0000 7fa4057a2640 1 -- 172.21.15.50:0/768966542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa400101fe0 msgr2=0x7fa400197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.559+0000 7fa4057a2640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa400101fe0 0x7fa400197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.559+0000 7fa4057a2640 1 -- 172.21.15.50:0/768966542 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40019ec50 con 0x7fa4001029a0 2024-04-23T18:16:39.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.559+0000 7fa4057a2640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4001029a0 0x7fa40019e5e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800cce0 tx=0x7fa3f800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:39.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.559+0000 7fa3fdffb640 1 -- 172.21.15.50:0/768966542 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f800f450 con 0x7fa4001029a0 2024-04-23T18:16:39.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.559+0000 7fa3fdffb640 1 -- 172.21.15.50:0/768966542 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3f800faa0 con 0x7fa4001029a0 2024-04-23T18:16:39.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.559+0000 7fa407205640 1 -- 172.21.15.50:0/768966542 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa40019ef40 con 0x7fa4001029a0 2024-04-23T18:16:39.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.559+0000 7fa3fdffb640 1 -- 172.21.15.50:0/768966542 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8018b70 con 0x7fa4001029a0 2024-04-23T18:16:39.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.559+0000 7fa407205640 1 -- 172.21.15.50:0/768966542 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa40019f3b0 con 0x7fa4001029a0 2024-04-23T18:16:39.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.560+0000 7fa407205640 1 -- 172.21.15.50:0/768966542 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa400108220 con 0x7fa4001029a0 2024-04-23T18:16:39.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.561+0000 7fa3fdffb640 1 -- 172.21.15.50:0/768966542 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa3f8018d10 con 0x7fa4001029a0 2024-04-23T18:16:39.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.562+0000 7fa3fdffb640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa3f00a3c40 0x7fa3f00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:39.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.562+0000 7fa404fa1640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa3f00a3c40 0x7fa3f00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:39.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.562+0000 7fa3fdffb640 1 -- 172.21.15.50:0/768966542 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fa3f80d0930 con 0x7fa4001029a0 2024-04-23T18:16:39.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.562+0000 7fa404fa1640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa3f00a3c40 0x7fa3f00a6100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4000c00 tx=0x7fa3f40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:39.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.566+0000 7fa3fdffb640 1 -- 172.21.15.50:0/768966542 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa3f8014030 con 0x7fa4001029a0 2024-04-23T18:16:39.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.781+0000 7fa407205640 1 -- 172.21.15.50:0/768966542 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fa400062390 con 0x7fa3f00a3c40 2024-04-23T18:16:39.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.786+0000 7fa3fdffb640 1 -- 172.21.15.50:0/768966542 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fa400062390 con 0x7fa3f00a3c40 2024-04-23T18:16:39.780 INFO:teuthology.orchestra.run.smithi050.stdout:/volumes/qa/sv_1/0c4ee723-281e-4755-96b0-c936df51685c 2024-04-23T18:16:39.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.788+0000 7fa407205640 1 -- 172.21.15.50:0/768966542 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa3f00a3c40 msgr2=0x7fa3f00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.788+0000 7fa407205640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa3f00a3c40 0x7fa3f00a6100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4000c00 tx=0x7fa3f40314a0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.788+0000 7fa407205640 1 -- 172.21.15.50:0/768966542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4001029a0 msgr2=0x7fa40019e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.788+0000 7fa407205640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4001029a0 0x7fa40019e5e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800cce0 tx=0x7fa3f800a620 comp rx=0 tx=0).stop 2024-04-23T18:16:39.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.788+0000 7fa407205640 1 -- 172.21.15.50:0/768966542 shutdown_connections 2024-04-23T18:16:39.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.788+0000 7fa407205640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa3f00a3c40 0x7fa3f00a6100 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.788+0000 7fa407205640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4001029a0 0x7fa40019e5e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.788+0000 7fa407205640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa400101fe0 0x7fa400197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.788+0000 7fa407205640 1 --2- 172.21.15.50:0/768966542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa400100e90 0x7fa400197030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.788+0000 7fa407205640 1 -- 172.21.15.50:0/768966542 >> 172.21.15.50:0/768966542 conn(0x7fa4000fc400 msgr2=0x7fa4001053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:39.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.788+0000 7fa407205640 1 -- 172.21.15.50:0/768966542 shutdown_connections 2024-04-23T18:16:39.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.788+0000 7fa407205640 1 -- 172.21.15.50:0/768966542 wait complete. 2024-04-23T18:16:39.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:16:39.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.901+0000 7f8182e2a640 1 -- 172.21.15.50:0/1285645031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817c101fa0 msgr2=0x7f817c1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.901+0000 7f8182e2a640 1 --2- 172.21.15.50:0/1285645031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817c101fa0 0x7f817c1023c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8170009900 tx=0x7f817002a410 comp rx=0 tx=0).stop 2024-04-23T18:16:39.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.901+0000 7f8182e2a640 1 -- 172.21.15.50:0/1285645031 shutdown_connections 2024-04-23T18:16:39.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.901+0000 7f8182e2a640 1 --2- 172.21.15.50:0/1285645031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817c103b50 0x7f817c10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.901+0000 7f8182e2a640 1 --2- 172.21.15.50:0/1285645031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f817c103190 0x7f817c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.901+0000 7f8182e2a640 1 --2- 172.21.15.50:0/1285645031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817c101fa0 0x7f817c1023c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.901+0000 7f8182e2a640 1 -- 172.21.15.50:0/1285645031 >> 172.21.15.50:0/1285645031 conn(0x7f817c0fd530 msgr2=0x7f817c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.901+0000 7f8182e2a640 1 -- 172.21.15.50:0/1285645031 shutdown_connections 2024-04-23T18:16:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.901+0000 7f8182e2a640 1 -- 172.21.15.50:0/1285645031 wait complete. 2024-04-23T18:16:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.901+0000 7f8182e2a640 1 Processor -- start 2024-04-23T18:16:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.901+0000 7f8182e2a640 1 -- start start 2024-04-23T18:16:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8182e2a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817c101fa0 0x7f817c075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8182e2a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f817c103190 0x7f817c074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8182e2a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817c103b50 0x7f817c074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8182e2a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f817c10dae0 con 0x7f817c101fa0 2024-04-23T18:16:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8182e2a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f817c10d960 con 0x7f817c103b50 2024-04-23T18:16:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8182e2a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f817c10dc60 con 0x7f817c103190 2024-04-23T18:16:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8180bc6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817c101fa0 0x7f817c075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8180bc6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817c101fa0 0x7f817c075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38832/0 (socket says 172.21.15.50:38832) 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8180bc6640 1 -- 172.21.15.50:0/2933436550 learned_addr learned my addr 172.21.15.50:0/2933436550 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f81813c7640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817c103b50 0x7f817c074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f817bfff640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f817c103190 0x7f817c074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8180bc6640 1 -- 172.21.15.50:0/2933436550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f817c103190 msgr2=0x7f817c074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8180bc6640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f817c103190 0x7f817c074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8180bc6640 1 -- 172.21.15.50:0/2933436550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817c103b50 msgr2=0x7f817c074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8180bc6640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817c103b50 0x7f817c074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8180bc6640 1 -- 172.21.15.50:0/2933436550 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c075110 con 0x7f817c101fa0 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f81813c7640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817c103b50 0x7f817c074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8180bc6640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817c101fa0 0x7f817c075d00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8170006010 tx=0x7f81700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8179ffb640 1 -- 172.21.15.50:0/2933436550 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817003f030 con 0x7f817c101fa0 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8179ffb640 1 -- 172.21.15.50:0/2933436550 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8170003cb0 con 0x7f817c101fa0 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8182e2a640 1 -- 172.21.15.50:0/2933436550 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f817c1a22a0 con 0x7f817c101fa0 2024-04-23T18:16:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.902+0000 7f8179ffb640 1 -- 172.21.15.50:0/2933436550 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8170035070 con 0x7f817c101fa0 2024-04-23T18:16:39.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.903+0000 7f8182e2a640 1 -- 172.21.15.50:0/2933436550 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f817c1a2840 con 0x7f817c101fa0 2024-04-23T18:16:39.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.904+0000 7f8182e2a640 1 -- 172.21.15.50:0/2933436550 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f817c1093e0 con 0x7f817c101fa0 2024-04-23T18:16:39.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.905+0000 7f8179ffb640 1 -- 172.21.15.50:0/2933436550 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8170042030 con 0x7f817c101fa0 2024-04-23T18:16:39.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.905+0000 7f8179ffb640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f816c0a3cc0 0x7f816c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:39.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.905+0000 7f817bfff640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f816c0a3cc0 0x7f816c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:39.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.905+0000 7f8179ffb640 1 -- 172.21.15.50:0/2933436550 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f81700f0a70 con 0x7f817c101fa0 2024-04-23T18:16:39.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.906+0000 7f817bfff640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f816c0a3cc0 0x7f816c0a6180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8168005d00 tx=0x7f8168005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:39.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:39.910+0000 7f8179ffb640 1 -- 172.21.15.50:0/2933436550 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f81700bd4a0 con 0x7f817c101fa0 2024-04-23T18:16:39.990 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:16:39 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b[63024]: 2024-04-23T18:16:39.591+0000 7f0084ff7700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-23T18:16:39.990 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:16:39 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b[63024]: 2024-04-23T18:16:39.591+0000 7f0084ff7700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-23T18:16:39.990 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:16:39 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b[63024]: 2024-04-23T18:16:39.591+0000 7f0084ff7700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-23T18:16:40.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.140+0000 7f8182e2a640 1 -- 172.21.15.50:0/2933436550 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f817c02d5c0 con 0x7f817c101fa0 2024-04-23T18:16:40.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.141+0000 7f8179ffb640 1 -- 172.21.15.50:0/2933436550 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11875 (secure 0 0 0) 0x7f81700bcda0 con 0x7f817c101fa0 2024-04-23T18:16:40.135 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:40.135 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:16:37.965071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:16:40.135 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 20 2024-04-23T18:16:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.143+0000 7f8182e2a640 1 -- 172.21.15.50:0/2933436550 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f816c0a3cc0 msgr2=0x7f816c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.143+0000 7f8182e2a640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f816c0a3cc0 0x7f816c0a6180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8168005d00 tx=0x7f8168005c50 comp rx=0 tx=0).stop 2024-04-23T18:16:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.143+0000 7f8182e2a640 1 -- 172.21.15.50:0/2933436550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817c101fa0 msgr2=0x7f817c075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.143+0000 7f8182e2a640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817c101fa0 0x7f817c075d00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8170006010 tx=0x7f81700042f0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.143+0000 7f8182e2a640 1 -- 172.21.15.50:0/2933436550 shutdown_connections 2024-04-23T18:16:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.143+0000 7f8182e2a640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f816c0a3cc0 0x7f816c0a6180 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.143+0000 7f8182e2a640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817c103b50 0x7f817c074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.143+0000 7f8182e2a640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f817c103190 0x7f817c074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.143+0000 7f8182e2a640 1 --2- 172.21.15.50:0/2933436550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817c101fa0 0x7f817c075d00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.143+0000 7f8182e2a640 1 -- 172.21.15.50:0/2933436550 >> 172.21.15.50:0/2933436550 conn(0x7f817c0fd530 msgr2=0x7f817c1065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:40.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.143+0000 7f8182e2a640 1 -- 172.21.15.50:0/2933436550 shutdown_connections 2024-04-23T18:16:40.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.143+0000 7f8182e2a640 1 -- 172.21.15.50:0/2933436550 wait complete. 2024-04-23T18:16:40.147 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:39 smithi050 ceph-mon[29187]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:40.147 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:39 smithi050 ceph-mon[29187]: pgmap v289: 97 pgs: 8 creating+peering, 89 active+clean; 580 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s wr, 13 op/s 2024-04-23T18:16:40.147 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:39 smithi050 ceph-mon[29187]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:40.147 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:39 smithi050 ceph-mon[29187]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-23T18:16:40.147 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:39 smithi050 ceph-mon[29187]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:40.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:16:40.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.256+0000 7f7a85c31640 1 -- 172.21.15.50:0/1154479951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a80101fc0 msgr2=0x7f7a801023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.256+0000 7f7a85c31640 1 --2- 172.21.15.50:0/1154479951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a80101fc0 0x7f7a801023e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009900 tx=0x7f7a7402a410 comp rx=0 tx=0).stop 2024-04-23T18:16:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.256+0000 7f7a85c31640 1 -- 172.21.15.50:0/1154479951 shutdown_connections 2024-04-23T18:16:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.256+0000 7f7a85c31640 1 --2- 172.21.15.50:0/1154479951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a80103b70 0x7f7a8010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.256+0000 7f7a85c31640 1 --2- 172.21.15.50:0/1154479951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a801031b0 0x7f7a80103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.256+0000 7f7a85c31640 1 --2- 172.21.15.50:0/1154479951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a80101fc0 0x7f7a801023e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.256+0000 7f7a85c31640 1 -- 172.21.15.50:0/1154479951 >> 172.21.15.50:0/1154479951 conn(0x7f7a800fd550 msgr2=0x7f7a800ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.256+0000 7f7a85c31640 1 -- 172.21.15.50:0/1154479951 shutdown_connections 2024-04-23T18:16:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.256+0000 7f7a85c31640 1 -- 172.21.15.50:0/1154479951 wait complete. 2024-04-23T18:16:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a85c31640 1 Processor -- start 2024-04-23T18:16:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a85c31640 1 -- start start 2024-04-23T18:16:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a85c31640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a80101fc0 0x7f7a80075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a85c31640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a801031b0 0x7f7a80074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a85c31640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a80103b70 0x7f7a80074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a85c31640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8010db40 con 0x7f7a801031b0 2024-04-23T18:16:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a85c31640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8010d9c0 con 0x7f7a80101fc0 2024-04-23T18:16:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a85c31640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8010dcc0 con 0x7f7a80103b70 2024-04-23T18:16:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a7effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a801031b0 0x7f7a80074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a7effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a801031b0 0x7f7a80074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38844/0 (socket says 172.21.15.50:38844) 2024-04-23T18:16:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a7effd640 1 -- 172.21.15.50:0/1690086715 learned_addr learned my addr 172.21.15.50:0/1690086715 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a7f7fe640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a80101fc0 0x7f7a80075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a7ffff640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a80103b70 0x7f7a80074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a7effd640 1 -- 172.21.15.50:0/1690086715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a80103b70 msgr2=0x7f7a80074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a7effd640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a80103b70 0x7f7a80074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a7effd640 1 -- 172.21.15.50:0/1690086715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a80101fc0 msgr2=0x7f7a80075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a7effd640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a80101fc0 0x7f7a80075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.257+0000 7f7a7effd640 1 -- 172.21.15.50:0/1690086715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a80075110 con 0x7f7a801031b0 2024-04-23T18:16:40.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.258+0000 7f7a7effd640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a801031b0 0x7f7a80074340 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7a7000b4a0 tx=0x7f7a7000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:40.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.258+0000 7f7a7cff9640 1 -- 172.21.15.50:0/1690086715 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a70014070 con 0x7f7a801031b0 2024-04-23T18:16:40.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.258+0000 7f7a85c31640 1 -- 172.21.15.50:0/1690086715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a801a2360 con 0x7f7a801031b0 2024-04-23T18:16:40.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.258+0000 7f7a7cff9640 1 -- 172.21.15.50:0/1690086715 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a70010ac0 con 0x7f7a801031b0 2024-04-23T18:16:40.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.258+0000 7f7a7cff9640 1 -- 172.21.15.50:0/1690086715 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a70017c30 con 0x7f7a801031b0 2024-04-23T18:16:40.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.258+0000 7f7a85c31640 1 -- 172.21.15.50:0/1690086715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a801a2930 con 0x7f7a801031b0 2024-04-23T18:16:40.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.259+0000 7f7a85c31640 1 -- 172.21.15.50:0/1690086715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a80109360 con 0x7f7a801031b0 2024-04-23T18:16:40.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.260+0000 7f7a7cff9640 1 -- 172.21.15.50:0/1690086715 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7a70017070 con 0x7f7a801031b0 2024-04-23T18:16:40.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.261+0000 7f7a7cff9640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7a680a3c40 0x7f7a680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.261+0000 7f7a7f7fe640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7a680a3c40 0x7f7a680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.261+0000 7f7a7cff9640 1 -- 172.21.15.50:0/1690086715 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f7a700cf960 con 0x7f7a801031b0 2024-04-23T18:16:40.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.261+0000 7f7a7f7fe640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7a680a3c40 0x7f7a680a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009230 tx=0x7f7a740314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:40.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.265+0000 7f7a7cff9640 1 -- 172.21.15.50:0/1690086715 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7a7009be50 con 0x7f7a801031b0 2024-04-23T18:16:40.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:39 smithi177 ceph-mon[32308]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:40.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:39 smithi177 ceph-mon[32308]: pgmap v289: 97 pgs: 8 creating+peering, 89 active+clean; 580 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s wr, 13 op/s 2024-04-23T18:16:40.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:39 smithi177 ceph-mon[32308]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:40.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:39 smithi177 ceph-mon[32308]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-23T18:16:40.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:39 smithi177 ceph-mon[32308]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:40.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:39 smithi188 ceph-mon[32694]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:40.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:39 smithi188 ceph-mon[32694]: pgmap v289: 97 pgs: 8 creating+peering, 89 active+clean; 580 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s wr, 13 op/s 2024-04-23T18:16:40.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:39 smithi188 ceph-mon[32694]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:40.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:39 smithi188 ceph-mon[32694]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-23T18:16:40.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:39 smithi188 ceph-mon[32694]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:40.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.456+0000 7f7a85c31640 1 -- 172.21.15.50:0/1690086715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7a8002d810 con 0x7f7a801031b0 2024-04-23T18:16:40.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.458+0000 7f7a7cff9640 1 -- 172.21.15.50:0/1690086715 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22181 (secure 0 0 0) 0x7f7a700a0150 con 0x7f7a801031b0 2024-04-23T18:16:40.452 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:40.452 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":87,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:16:34.977786+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:16:40.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.460+0000 7f7a85c31640 1 -- 172.21.15.50:0/1690086715 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7a680a3c40 msgr2=0x7f7a680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.460+0000 7f7a85c31640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7a680a3c40 0x7f7a680a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009230 tx=0x7f7a740314a0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.460+0000 7f7a85c31640 1 -- 172.21.15.50:0/1690086715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a801031b0 msgr2=0x7f7a80074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.460+0000 7f7a85c31640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a801031b0 0x7f7a80074340 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7a7000b4a0 tx=0x7f7a7000b960 comp rx=0 tx=0).stop 2024-04-23T18:16:40.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.460+0000 7f7a85c31640 1 -- 172.21.15.50:0/1690086715 shutdown_connections 2024-04-23T18:16:40.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.460+0000 7f7a85c31640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7a680a3c40 0x7f7a680a6100 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.460+0000 7f7a85c31640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a80103b70 0x7f7a80074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.460+0000 7f7a85c31640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a801031b0 0x7f7a80074340 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.460+0000 7f7a85c31640 1 --2- 172.21.15.50:0/1690086715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a80101fc0 0x7f7a80075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.460+0000 7f7a85c31640 1 -- 172.21.15.50:0/1690086715 >> 172.21.15.50:0/1690086715 conn(0x7f7a800fd550 msgr2=0x7f7a80106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:40.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.460+0000 7f7a85c31640 1 -- 172.21.15.50:0/1690086715 shutdown_connections 2024-04-23T18:16:40.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.460+0000 7f7a85c31640 1 -- 172.21.15.50:0/1690086715 wait complete. 2024-04-23T18:16:40.465 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-23T18:16:40.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:16:40.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.574+0000 7f61aa944640 1 -- 172.21.15.50:0/1106589505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61a4105360 msgr2=0x7f61a4107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.574+0000 7f61aa944640 1 --2- 172.21.15.50:0/1106589505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61a4105360 0x7f61a4107750 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f618c009900 tx=0x7f618c02a410 comp rx=0 tx=0).stop 2024-04-23T18:16:40.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.574+0000 7f61aa944640 1 -- 172.21.15.50:0/1106589505 shutdown_connections 2024-04-23T18:16:40.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.574+0000 7f61aa944640 1 --2- 172.21.15.50:0/1106589505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61a4107c90 0x7f61a410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.574+0000 7f61aa944640 1 --2- 172.21.15.50:0/1106589505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61a4105360 0x7f61a4107750 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.574+0000 7f61aa944640 1 --2- 172.21.15.50:0/1106589505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a4102a30 0x7f61a4104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.574+0000 7f61aa944640 1 -- 172.21.15.50:0/1106589505 >> 172.21.15.50:0/1106589505 conn(0x7f61a40fc440 msgr2=0x7f61a40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.574+0000 7f61aa944640 1 -- 172.21.15.50:0/1106589505 shutdown_connections 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.574+0000 7f61aa944640 1 -- 172.21.15.50:0/1106589505 wait complete. 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61aa944640 1 Processor -- start 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61aa944640 1 -- start start 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61aa944640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61a4102a30 0x7f61a41023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61aa944640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a4105360 0x7f61a41009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61aa944640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61a4107c90 0x7f61a4100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61aa944640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f61a410daf0 con 0x7f61a4107c90 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61aa944640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f61a410d970 con 0x7f61a4102a30 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61aa944640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f61a410dc70 con 0x7f61a4105360 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61a8ee1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61a4107c90 0x7f61a4100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61a8ee1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61a4107c90 0x7f61a4100f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38854/0 (socket says 172.21.15.50:38854) 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61a8ee1640 1 -- 172.21.15.50:0/1996162075 learned_addr learned my addr 172.21.15.50:0/1996162075 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61a37fe640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a4105360 0x7f61a41009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61a3fff640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61a4102a30 0x7f61a41023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61a8ee1640 1 -- 172.21.15.50:0/1996162075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a4105360 msgr2=0x7f61a41009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61a8ee1640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a4105360 0x7f61a41009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61a8ee1640 1 -- 172.21.15.50:0/1996162075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61a4102a30 msgr2=0x7f61a41023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61a8ee1640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61a4102a30 0x7f61a41023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61a8ee1640 1 -- 172.21.15.50:0/1996162075 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a41017b0 con 0x7f61a4107c90 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.575+0000 7f61a3fff640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61a4102a30 0x7f61a41023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.576+0000 7f61a8ee1640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61a4107c90 0x7f61a4100f20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f619c00cad0 tx=0x7f619c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.576+0000 7f61a17fa640 1 -- 172.21.15.50:0/1996162075 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c00ae20 con 0x7f61a4107c90 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.576+0000 7f61a17fa640 1 -- 172.21.15.50:0/1996162075 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f619c005020 con 0x7f61a4107c90 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.576+0000 7f61aa944640 1 -- 172.21.15.50:0/1996162075 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a41a23b0 con 0x7f61a4107c90 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.576+0000 7f61a17fa640 1 -- 172.21.15.50:0/1996162075 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c00f6b0 con 0x7f61a4107c90 2024-04-23T18:16:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.576+0000 7f61aa944640 1 -- 172.21.15.50:0/1996162075 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a41a2880 con 0x7f61a4107c90 2024-04-23T18:16:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.577+0000 7f61aa944640 1 -- 172.21.15.50:0/1996162075 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61a40691c0 con 0x7f61a4107c90 2024-04-23T18:16:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.578+0000 7f61a17fa640 1 -- 172.21.15.50:0/1996162075 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f619c0051d0 con 0x7f61a4107c90 2024-04-23T18:16:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.578+0000 7f61a17fa640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f61900a3c40 0x7f61900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.579+0000 7f61a17fa640 1 -- 172.21.15.50:0/1996162075 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f619c0d04a0 con 0x7f61a4107c90 2024-04-23T18:16:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.579+0000 7f61a3fff640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f61900a3c40 0x7f61900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.579+0000 7f61a3fff640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f61900a3c40 0x7f61900a6100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f61940058c0 tx=0x7f6194005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:40.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.583+0000 7f61a17fa640 1 -- 172.21.15.50:0/1996162075 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f619c014030 con 0x7f61a4107c90 2024-04-23T18:16:40.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.768+0000 7f61aa944640 1 -- 172.21.15.50:0/1996162075 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f61a4062390 con 0x7f61900a3c40 2024-04-23T18:16:40.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.776+0000 7f61a17fa640 1 -- 172.21.15.50:0/1996162075 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258772 (secure 0 0 0) 0x7f61a4062390 con 0x7f61900a3c40 2024-04-23T18:16:40.772 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:40.774 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-23T18:16:40.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.779+0000 7f61aa944640 1 -- 172.21.15.50:0/1996162075 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f61900a3c40 msgr2=0x7f61900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.779+0000 7f61aa944640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f61900a3c40 0x7f61900a6100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f61940058c0 tx=0x7f6194005850 comp rx=0 tx=0).stop 2024-04-23T18:16:40.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.779+0000 7f61aa944640 1 -- 172.21.15.50:0/1996162075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61a4107c90 msgr2=0x7f61a4100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.779+0000 7f61aa944640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61a4107c90 0x7f61a4100f20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f619c00cad0 tx=0x7f619c00cf90 comp rx=0 tx=0).stop 2024-04-23T18:16:40.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.779+0000 7f61aa944640 1 -- 172.21.15.50:0/1996162075 shutdown_connections 2024-04-23T18:16:40.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.779+0000 7f61aa944640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f61900a3c40 0x7f61900a6100 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.779+0000 7f61aa944640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61a4107c90 0x7f61a4100f20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.779+0000 7f61aa944640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a4105360 0x7f61a41009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.779+0000 7f61aa944640 1 --2- 172.21.15.50:0/1996162075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61a4102a30 0x7f61a41023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.779+0000 7f61aa944640 1 -- 172.21.15.50:0/1996162075 >> 172.21.15.50:0/1996162075 conn(0x7f61a40fc440 msgr2=0x7f61a41032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:40.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.779+0000 7f61aa944640 1 -- 172.21.15.50:0/1996162075 shutdown_connections 2024-04-23T18:16:40.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.779+0000 7f61aa944640 1 -- 172.21.15.50:0/1996162075 wait complete. 2024-04-23T18:16:40.784 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-04-23T18:16:40.501319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594296,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":143,"num_read_kb":126,"num_write":282,"num_write_kb":2972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":206,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":387036,"kb_used_data":12880,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124554788,"statfs":{"total":1151940427776,"available":1151544102912,"internally_reserved":0,"allocated":13189120,"data_stored":7164049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3928,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":101,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.557677"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966580+0000","last_change":"2024-04-23T18:16:33.987102+0000","last_active":"2024-04-23T18:16:35.966580+0000","last_peered":"2024-04-23T18:16:35.966580+0000","last_clean":"2024-04-23T18:16:35.966580+0000","last_became_active":"2024-04-23T18:16:33.986716+0000","last_became_peered":"2024-04-23T18:16:33.986716+0000","last_unstale":"2024-04-23T18:16:35.966580+0000","last_undegraded":"2024-04-23T18:16:35.966580+0000","last_fullsized":"2024-04-23T18:16:35.966580+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:41.895219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.439035+0000","last_change":"2024-04-23T18:16:33.995703+0000","last_active":"2024-04-23T18:16:35.439035+0000","last_peered":"2024-04-23T18:16:35.439035+0000","last_clean":"2024-04-23T18:16:35.439035+0000","last_became_active":"2024-04-23T18:16:33.995242+0000","last_became_peered":"2024-04-23T18:16:33.995242+0000","last_unstale":"2024-04-23T18:16:35.439035+0000","last_undegraded":"2024-04-23T18:16:35.439035+0000","last_fullsized":"2024-04-23T18:16:35.439035+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:57.813822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.438836+0000","last_change":"2024-04-23T18:16:34.000864+0000","last_active":"2024-04-23T18:16:35.438836+0000","last_peered":"2024-04-23T18:16:35.438836+0000","last_clean":"2024-04-23T18:16:35.438836+0000","last_became_active":"2024-04-23T18:16:33.998934+0000","last_became_peered":"2024-04-23T18:16:33.998934+0000","last_unstale":"2024-04-23T18:16:35.438836+0000","last_undegraded":"2024-04-23T18:16:35.438836+0000","last_fullsized":"2024-04-23T18:16:35.438836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:47:29.481697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961511+0000","last_change":"2024-04-23T18:16:33.989351+0000","last_active":"2024-04-23T18:16:35.961511+0000","last_peered":"2024-04-23T18:16:35.961511+0000","last_clean":"2024-04-23T18:16:35.961511+0000","last_became_active":"2024-04-23T18:16:33.989030+0000","last_became_peered":"2024-04-23T18:16:33.989030+0000","last_unstale":"2024-04-23T18:16:35.961511+0000","last_undegraded":"2024-04-23T18:16:35.961511+0000","last_fullsized":"2024-04-23T18:16:35.961511+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:29:03.023675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.989528+0000","last_change":"2024-04-23T18:16:33.996062+0000","last_active":"2024-04-23T18:16:34.989528+0000","last_peered":"2024-04-23T18:16:34.989528+0000","last_clean":"2024-04-23T18:16:34.989528+0000","last_became_active":"2024-04-23T18:16:33.995732+0000","last_became_peered":"2024-04-23T18:16:33.995732+0000","last_unstale":"2024-04-23T18:16:34.989528+0000","last_undegraded":"2024-04-23T18:16:34.989528+0000","last_fullsized":"2024-04-23T18:16:34.989528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:04.175580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961265+0000","last_change":"2024-04-23T18:16:33.989895+0000","last_active":"2024-04-23T18:16:35.961265+0000","last_peered":"2024-04-23T18:16:35.961265+0000","last_clean":"2024-04-23T18:16:35.961265+0000","last_became_active":"2024-04-23T18:16:33.989547+0000","last_became_peered":"2024-04-23T18:16:33.989547+0000","last_unstale":"2024-04-23T18:16:35.961265+0000","last_undegraded":"2024-04-23T18:16:35.961265+0000","last_fullsized":"2024-04-23T18:16:35.961265+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:12:37.317974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985478+0000","last_change":"2024-04-23T18:16:33.999636+0000","last_active":"2024-04-23T18:16:34.985478+0000","last_peered":"2024-04-23T18:16:34.985478+0000","last_clean":"2024-04-23T18:16:34.985478+0000","last_became_active":"2024-04-23T18:16:33.999282+0000","last_became_peered":"2024-04-23T18:16:33.999282+0000","last_unstale":"2024-04-23T18:16:34.985478+0000","last_undegraded":"2024-04-23T18:16:34.985478+0000","last_fullsized":"2024-04-23T18:16:34.985478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:28.841392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.999421+0000","last_change":"2024-04-23T18:16:33.996602+0000","last_active":"2024-04-23T18:16:33.999421+0000","last_peered":"2024-04-23T18:16:33.999421+0000","last_clean":"2024-04-23T18:16:33.999421+0000","last_became_active":"2024-04-23T18:16:33.994401+0000","last_became_peered":"2024-04-23T18:16:33.994401+0000","last_unstale":"2024-04-23T18:16:33.999421+0000","last_undegraded":"2024-04-23T18:16:33.999421+0000","last_fullsized":"2024-04-23T18:16:33.999421+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:58.575037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.800019+0000","last_change":"2024-04-23T18:16:33.994763+0000","last_active":"2024-04-23T18:16:35.800019+0000","last_peered":"2024-04-23T18:16:35.800019+0000","last_clean":"2024-04-23T18:16:35.800019+0000","last_became_active":"2024-04-23T18:16:33.994408+0000","last_became_peered":"2024-04-23T18:16:33.994408+0000","last_unstale":"2024-04-23T18:16:35.800019+0000","last_undegraded":"2024-04-23T18:16:35.800019+0000","last_fullsized":"2024-04-23T18:16:35.800019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:02.706160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961665+0000","last_change":"2024-04-23T18:16:33.993435+0000","last_active":"2024-04-23T18:16:35.961665+0000","last_peered":"2024-04-23T18:16:35.961665+0000","last_clean":"2024-04-23T18:16:35.961665+0000","last_became_active":"2024-04-23T18:16:33.993108+0000","last_became_peered":"2024-04-23T18:16:33.993108+0000","last_unstale":"2024-04-23T18:16:35.961665+0000","last_undegraded":"2024-04-23T18:16:35.961665+0000","last_fullsized":"2024-04-23T18:16:35.961665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:28:32.584896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985884+0000","last_change":"2024-04-23T18:16:33.997526+0000","last_active":"2024-04-23T18:16:34.985884+0000","last_peered":"2024-04-23T18:16:34.985884+0000","last_clean":"2024-04-23T18:16:34.985884+0000","last_became_active":"2024-04-23T18:16:33.997189+0000","last_became_peered":"2024-04-23T18:16:33.997189+0000","last_unstale":"2024-04-23T18:16:34.985884+0000","last_undegraded":"2024-04-23T18:16:34.985884+0000","last_fullsized":"2024-04-23T18:16:34.985884+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:11:41.208261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961359+0000","last_change":"2024-04-23T18:16:33.989775+0000","last_active":"2024-04-23T18:16:35.961359+0000","last_peered":"2024-04-23T18:16:35.961359+0000","last_clean":"2024-04-23T18:16:35.961359+0000","last_became_active":"2024-04-23T18:16:33.989382+0000","last_became_peered":"2024-04-23T18:16:33.989382+0000","last_unstale":"2024-04-23T18:16:35.961359+0000","last_undegraded":"2024-04-23T18:16:35.961359+0000","last_fullsized":"2024-04-23T18:16:35.961359+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:20:17.756258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.984796+0000","last_change":"2024-04-23T18:16:34.005871+0000","last_active":"2024-04-23T18:16:34.984796+0000","last_peered":"2024-04-23T18:16:34.984796+0000","last_clean":"2024-04-23T18:16:34.984796+0000","last_became_active":"2024-04-23T18:16:34.005218+0000","last_became_peered":"2024-04-23T18:16:34.005218+0000","last_unstale":"2024-04-23T18:16:34.984796+0000","last_undegraded":"2024-04-23T18:16:34.984796+0000","last_fullsized":"2024-04-23T18:16:34.984796+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:30:03.864306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.438680+0000","last_change":"2024-04-23T18:16:33.995054+0000","last_active":"2024-04-23T18:16:35.438680+0000","last_peered":"2024-04-23T18:16:35.438680+0000","last_clean":"2024-04-23T18:16:35.438680+0000","last_became_active":"2024-04-23T18:16:33.994389+0000","last_became_peered":"2024-04-23T18:16:33.994389+0000","last_unstale":"2024-04-23T18:16:35.438680+0000","last_undegraded":"2024-04-23T18:16:35.438680+0000","last_fullsized":"2024-04-23T18:16:35.438680+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:27.344616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.438661+0000","last_change":"2024-04-23T18:16:33.984490+0000","last_active":"2024-04-23T18:16:35.438661+0000","last_peered":"2024-04-23T18:16:35.438661+0000","last_clean":"2024-04-23T18:16:35.438661+0000","last_became_active":"2024-04-23T18:16:33.984071+0000","last_became_peered":"2024-04-23T18:16:33.984071+0000","last_unstale":"2024-04-23T18:16:35.438661+0000","last_undegraded":"2024-04-23T18:16:35.438661+0000","last_fullsized":"2024-04-23T18:16:35.438661+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:23.589290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.799608+0000","last_change":"2024-04-23T18:16:34.002163+0000","last_active":"2024-04-23T18:16:35.799608+0000","last_peered":"2024-04-23T18:16:35.799608+0000","last_clean":"2024-04-23T18:16:35.799608+0000","last_became_active":"2024-04-23T18:16:34.001764+0000","last_became_peered":"2024-04-23T18:16:34.001764+0000","last_unstale":"2024-04-23T18:16:35.799608+0000","last_undegraded":"2024-04-23T18:16:35.799608+0000","last_fullsized":"2024-04-23T18:16:35.799608+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:16:03.616358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966605+0000","last_change":"2024-04-23T18:16:33.987586+0000","last_active":"2024-04-23T18:16:35.966605+0000","last_peered":"2024-04-23T18:16:35.966605+0000","last_clean":"2024-04-23T18:16:35.966605+0000","last_became_active":"2024-04-23T18:16:33.987266+0000","last_became_peered":"2024-04-23T18:16:33.987266+0000","last_unstale":"2024-04-23T18:16:35.966605+0000","last_undegraded":"2024-04-23T18:16:35.966605+0000","last_fullsized":"2024-04-23T18:16:35.966605+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:41.610529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.993997+0000","last_change":"2024-04-23T18:16:33.992938+0000","last_active":"2024-04-23T18:16:33.993997+0000","last_peered":"2024-04-23T18:16:33.993997+0000","last_clean":"2024-04-23T18:16:33.993997+0000","last_became_active":"2024-04-23T18:16:33.990441+0000","last_became_peered":"2024-04-23T18:16:33.990441+0000","last_unstale":"2024-04-23T18:16:33.993997+0000","last_undegraded":"2024-04-23T18:16:33.993997+0000","last_fullsized":"2024-04-23T18:16:33.993997+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:54.914022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.959747+0000","last_change":"2024-04-23T18:16:33.994844+0000","last_active":"2024-04-23T18:16:35.959747+0000","last_peered":"2024-04-23T18:16:35.959747+0000","last_clean":"2024-04-23T18:16:35.959747+0000","last_became_active":"2024-04-23T18:16:33.994505+0000","last_became_peered":"2024-04-23T18:16:33.994505+0000","last_unstale":"2024-04-23T18:16:35.959747+0000","last_undegraded":"2024-04-23T18:16:35.959747+0000","last_fullsized":"2024-04-23T18:16:35.959747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:32:33.445266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.000893+0000","last_change":"2024-04-23T18:16:32.958841+0000","last_active":"2024-04-23T18:16:34.000893+0000","last_peered":"2024-04-23T18:16:34.000893+0000","last_clean":"2024-04-23T18:16:34.000893+0000","last_became_active":"2024-04-23T18:16:32.958529+0000","last_became_peered":"2024-04-23T18:16:32.958529+0000","last_unstale":"2024-04-23T18:16:34.000893+0000","last_undegraded":"2024-04-23T18:16:34.000893+0000","last_fullsized":"2024-04-23T18:16:34.000893+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:47:09.933786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.993360+0000","last_change":"2024-04-23T18:16:33.992108+0000","last_active":"2024-04-23T18:16:33.993360+0000","last_peered":"2024-04-23T18:16:33.993360+0000","last_clean":"2024-04-23T18:16:33.993360+0000","last_became_active":"2024-04-23T18:16:33.991751+0000","last_became_peered":"2024-04-23T18:16:33.991751+0000","last_unstale":"2024-04-23T18:16:33.993360+0000","last_undegraded":"2024-04-23T18:16:33.993360+0000","last_fullsized":"2024-04-23T18:16:33.993360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:48.631561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.003714+0000","last_change":"2024-04-23T18:16:32.960461+0000","last_active":"2024-04-23T18:16:34.003714+0000","last_peered":"2024-04-23T18:16:34.003714+0000","last_clean":"2024-04-23T18:16:34.003714+0000","last_became_active":"2024-04-23T18:16:32.960115+0000","last_became_peered":"2024-04-23T18:16:32.960115+0000","last_unstale":"2024-04-23T18:16:34.003714+0000","last_undegraded":"2024-04-23T18:16:34.003714+0000","last_fullsized":"2024-04-23T18:16:34.003714+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:30.533043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.439004+0000","last_change":"2024-04-23T18:16:34.001831+0000","last_active":"2024-04-23T18:16:35.439004+0000","last_peered":"2024-04-23T18:16:35.439004+0000","last_clean":"2024-04-23T18:16:35.439004+0000","last_became_active":"2024-04-23T18:16:34.001486+0000","last_became_peered":"2024-04-23T18:16:34.001486+0000","last_unstale":"2024-04-23T18:16:35.439004+0000","last_undegraded":"2024-04-23T18:16:35.439004+0000","last_fullsized":"2024-04-23T18:16:35.439004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:06:40.452366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.969781+0000","last_change":"2024-04-23T18:16:32.960387+0000","last_active":"2024-04-23T18:16:36.969781+0000","last_peered":"2024-04-23T18:16:36.969781+0000","last_clean":"2024-04-23T18:16:36.969781+0000","last_became_active":"2024-04-23T18:16:32.960046+0000","last_became_peered":"2024-04-23T18:16:32.960046+0000","last_unstale":"2024-04-23T18:16:36.969781+0000","last_undegraded":"2024-04-23T18:16:36.969781+0000","last_fullsized":"2024-04-23T18:16:36.969781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:03.296653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.799689+0000","last_change":"2024-04-23T18:16:33.996520+0000","last_active":"2024-04-23T18:16:35.799689+0000","last_peered":"2024-04-23T18:16:35.799689+0000","last_clean":"2024-04-23T18:16:35.799689+0000","last_became_active":"2024-04-23T18:16:33.995636+0000","last_became_peered":"2024-04-23T18:16:33.995636+0000","last_unstale":"2024-04-23T18:16:35.799689+0000","last_undegraded":"2024-04-23T18:16:35.799689+0000","last_fullsized":"2024-04-23T18:16:35.799689+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:29.665407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.989840+0000","last_change":"2024-04-23T18:16:32.956813+0000","last_active":"2024-04-23T18:16:34.989840+0000","last_peered":"2024-04-23T18:16:34.989840+0000","last_clean":"2024-04-23T18:16:34.989840+0000","last_became_active":"2024-04-23T18:16:32.956505+0000","last_became_peered":"2024-04-23T18:16:32.956505+0000","last_unstale":"2024-04-23T18:16:34.989840+0000","last_undegraded":"2024-04-23T18:16:34.989840+0000","last_fullsized":"2024-04-23T18:16:34.989840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:36:27.330353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.000115+0000","last_change":"2024-04-23T18:16:33.999754+0000","last_active":"2024-04-23T18:16:34.000115+0000","last_peered":"2024-04-23T18:16:34.000115+0000","last_clean":"2024-04-23T18:16:34.000115+0000","last_became_active":"2024-04-23T18:16:33.999034+0000","last_became_peered":"2024-04-23T18:16:33.999034+0000","last_unstale":"2024-04-23T18:16:34.000115+0000","last_undegraded":"2024-04-23T18:16:34.000115+0000","last_fullsized":"2024-04-23T18:16:34.000115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:20:04.819520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.001496+0000","last_change":"2024-04-23T18:16:32.961347+0000","last_active":"2024-04-23T18:16:34.001496+0000","last_peered":"2024-04-23T18:16:34.001496+0000","last_clean":"2024-04-23T18:16:34.001496+0000","last_became_active":"2024-04-23T18:16:32.961045+0000","last_became_peered":"2024-04-23T18:16:32.961045+0000","last_unstale":"2024-04-23T18:16:34.001496+0000","last_undegraded":"2024-04-23T18:16:34.001496+0000","last_fullsized":"2024-04-23T18:16:34.001496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:47:39.662705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.989516+0000","last_change":"2024-04-23T18:16:33.995166+0000","last_active":"2024-04-23T18:16:34.989516+0000","last_peered":"2024-04-23T18:16:34.989516+0000","last_clean":"2024-04-23T18:16:34.989516+0000","last_became_active":"2024-04-23T18:16:33.994347+0000","last_became_peered":"2024-04-23T18:16:33.994347+0000","last_unstale":"2024-04-23T18:16:34.989516+0000","last_undegraded":"2024-04-23T18:16:34.989516+0000","last_fullsized":"2024-04-23T18:16:34.989516+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.305837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961159+0000","last_change":"2024-04-23T18:16:32.959358+0000","last_active":"2024-04-23T18:16:35.961159+0000","last_peered":"2024-04-23T18:16:35.961159+0000","last_clean":"2024-04-23T18:16:35.961159+0000","last_became_active":"2024-04-23T18:16:32.959036+0000","last_became_peered":"2024-04-23T18:16:32.959036+0000","last_unstale":"2024-04-23T18:16:35.961159+0000","last_undegraded":"2024-04-23T18:16:35.961159+0000","last_fullsized":"2024-04-23T18:16:35.961159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:03:15.734754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.989070+0000","last_change":"2024-04-23T18:16:33.988578+0000","last_active":"2024-04-23T18:16:33.989070+0000","last_peered":"2024-04-23T18:16:33.989070+0000","last_clean":"2024-04-23T18:16:33.989070+0000","last_became_active":"2024-04-23T18:16:33.988248+0000","last_became_peered":"2024-04-23T18:16:33.988248+0000","last_unstale":"2024-04-23T18:16:33.989070+0000","last_undegraded":"2024-04-23T18:16:33.989070+0000","last_fullsized":"2024-04-23T18:16:33.989070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:50.122995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.969393+0000","last_change":"2024-04-23T18:16:32.952945+0000","last_active":"2024-04-23T18:16:36.969393+0000","last_peered":"2024-04-23T18:16:36.969393+0000","last_clean":"2024-04-23T18:16:36.969393+0000","last_became_active":"2024-04-23T18:16:32.952602+0000","last_became_peered":"2024-04-23T18:16:32.952602+0000","last_unstale":"2024-04-23T18:16:36.969393+0000","last_undegraded":"2024-04-23T18:16:36.969393+0000","last_fullsized":"2024-04-23T18:16:36.969393+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:20.037431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.999446+0000","last_change":"2024-04-23T18:16:33.996897+0000","last_active":"2024-04-23T18:16:33.999446+0000","last_peered":"2024-04-23T18:16:33.999446+0000","last_clean":"2024-04-23T18:16:33.999446+0000","last_became_active":"2024-04-23T18:16:33.996214+0000","last_became_peered":"2024-04-23T18:16:33.996214+0000","last_unstale":"2024-04-23T18:16:33.999446+0000","last_undegraded":"2024-04-23T18:16:33.999446+0000","last_fullsized":"2024-04-23T18:16:33.999446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:10:05.662144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.975990+0000","last_change":"2024-04-23T18:16:32.959637+0000","last_active":"2024-04-23T18:16:33.975990+0000","last_peered":"2024-04-23T18:16:33.975990+0000","last_clean":"2024-04-23T18:16:33.975990+0000","last_became_active":"2024-04-23T18:16:32.959332+0000","last_became_peered":"2024-04-23T18:16:32.959332+0000","last_unstale":"2024-04-23T18:16:33.975990+0000","last_undegraded":"2024-04-23T18:16:33.975990+0000","last_fullsized":"2024-04-23T18:16:33.975990+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:30:49.089101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.986793+0000","last_change":"2024-04-23T18:16:33.986288+0000","last_active":"2024-04-23T18:16:33.986793+0000","last_peered":"2024-04-23T18:16:33.986793+0000","last_clean":"2024-04-23T18:16:33.986793+0000","last_became_active":"2024-04-23T18:16:33.985866+0000","last_became_peered":"2024-04-23T18:16:33.985866+0000","last_unstale":"2024-04-23T18:16:33.986793+0000","last_undegraded":"2024-04-23T18:16:33.986793+0000","last_fullsized":"2024-04-23T18:16:33.986793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:22:10.864070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.965205+0000","last_change":"2024-04-23T18:16:32.958466+0000","last_active":"2024-04-23T18:16:35.965205+0000","last_peered":"2024-04-23T18:16:35.965205+0000","last_clean":"2024-04-23T18:16:35.965205+0000","last_became_active":"2024-04-23T18:16:32.958155+0000","last_became_peered":"2024-04-23T18:16:32.958155+0000","last_unstale":"2024-04-23T18:16:35.965205+0000","last_undegraded":"2024-04-23T18:16:35.965205+0000","last_fullsized":"2024-04-23T18:16:35.965205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:15:49.073934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.799958+0000","last_change":"2024-04-23T18:16:33.996156+0000","last_active":"2024-04-23T18:16:35.799958+0000","last_peered":"2024-04-23T18:16:35.799958+0000","last_clean":"2024-04-23T18:16:35.799958+0000","last_became_active":"2024-04-23T18:16:33.995711+0000","last_became_peered":"2024-04-23T18:16:33.995711+0000","last_unstale":"2024-04-23T18:16:35.799958+0000","last_undegraded":"2024-04-23T18:16:35.799958+0000","last_fullsized":"2024-04-23T18:16:35.799958+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:45.272598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.962041+0000","last_change":"2024-04-23T18:16:32.959395+0000","last_active":"2024-04-23T18:16:35.962041+0000","last_peered":"2024-04-23T18:16:35.962041+0000","last_clean":"2024-04-23T18:16:35.962041+0000","last_became_active":"2024-04-23T18:16:32.959096+0000","last_became_peered":"2024-04-23T18:16:32.959096+0000","last_unstale":"2024-04-23T18:16:35.962041+0000","last_undegraded":"2024-04-23T18:16:35.962041+0000","last_fullsized":"2024-04-23T18:16:35.962041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:24:46.910385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.438960+0000","last_change":"2024-04-23T18:16:33.995800+0000","last_active":"2024-04-23T18:16:35.438960+0000","last_peered":"2024-04-23T18:16:35.438960+0000","last_clean":"2024-04-23T18:16:35.438960+0000","last_became_active":"2024-04-23T18:16:33.995306+0000","last_became_peered":"2024-04-23T18:16:33.995306+0000","last_unstale":"2024-04-23T18:16:35.438960+0000","last_undegraded":"2024-04-23T18:16:35.438960+0000","last_fullsized":"2024-04-23T18:16:35.438960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:13:40.097771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.002649+0000","last_change":"2024-04-23T18:16:32.966725+0000","last_active":"2024-04-23T18:16:34.002649+0000","last_peered":"2024-04-23T18:16:34.002649+0000","last_clean":"2024-04-23T18:16:34.002649+0000","last_became_active":"2024-04-23T18:16:32.966304+0000","last_became_peered":"2024-04-23T18:16:32.966304+0000","last_unstale":"2024-04-23T18:16:34.002649+0000","last_undegraded":"2024-04-23T18:16:34.002649+0000","last_fullsized":"2024-04-23T18:16:34.002649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:45.191051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.799873+0000","last_change":"2024-04-23T18:16:33.994667+0000","last_active":"2024-04-23T18:16:35.799873+0000","last_peered":"2024-04-23T18:16:35.799873+0000","last_clean":"2024-04-23T18:16:35.799873+0000","last_became_active":"2024-04-23T18:16:33.994305+0000","last_became_peered":"2024-04-23T18:16:33.994305+0000","last_unstale":"2024-04-23T18:16:35.799873+0000","last_undegraded":"2024-04-23T18:16:35.799873+0000","last_fullsized":"2024-04-23T18:16:35.799873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:23.807075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'8","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.965773+0000","last_change":"2024-04-23T18:16:32.957946+0000","last_active":"2024-04-23T18:16:35.965773+0000","last_peered":"2024-04-23T18:16:35.965773+0000","last_clean":"2024-04-23T18:16:35.965773+0000","last_became_active":"2024-04-23T18:16:32.957478+0000","last_became_peered":"2024-04-23T18:16:32.957478+0000","last_unstale":"2024-04-23T18:16:35.965773+0000","last_undegraded":"2024-04-23T18:16:35.965773+0000","last_fullsized":"2024-04-23T18:16:35.965773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:16.850604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.989797+0000","last_change":"2024-04-23T18:16:33.989537+0000","last_active":"2024-04-23T18:16:34.989797+0000","last_peered":"2024-04-23T18:16:34.989797+0000","last_clean":"2024-04-23T18:16:34.989797+0000","last_became_active":"2024-04-23T18:16:33.989180+0000","last_became_peered":"2024-04-23T18:16:33.989180+0000","last_unstale":"2024-04-23T18:16:34.989797+0000","last_undegraded":"2024-04-23T18:16:34.989797+0000","last_fullsized":"2024-04-23T18:16:34.989797+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:53:18.700347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.969039+0000","last_change":"2024-04-23T18:16:32.967314+0000","last_active":"2024-04-23T18:16:36.969039+0000","last_peered":"2024-04-23T18:16:36.969039+0000","last_clean":"2024-04-23T18:16:36.969039+0000","last_became_active":"2024-04-23T18:16:32.966861+0000","last_became_peered":"2024-04-23T18:16:32.966861+0000","last_unstale":"2024-04-23T18:16:36.969039+0000","last_undegraded":"2024-04-23T18:16:36.969039+0000","last_fullsized":"2024-04-23T18:16:36.969039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:00:25.606402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.984593+0000","last_change":"2024-04-23T18:16:33.992593+0000","last_active":"2024-04-23T18:16:34.984593+0000","last_peered":"2024-04-23T18:16:34.984593+0000","last_clean":"2024-04-23T18:16:34.984593+0000","last_became_active":"2024-04-23T18:16:33.992228+0000","last_became_peered":"2024-04-23T18:16:33.992228+0000","last_unstale":"2024-04-23T18:16:34.984593+0000","last_undegraded":"2024-04-23T18:16:34.984593+0000","last_fullsized":"2024-04-23T18:16:34.984593+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:12:23.836829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.976815+0000","last_change":"2024-04-23T18:16:32.961740+0000","last_active":"2024-04-23T18:16:33.976815+0000","last_peered":"2024-04-23T18:16:33.976815+0000","last_clean":"2024-04-23T18:16:33.976815+0000","last_became_active":"2024-04-23T18:16:32.961331+0000","last_became_peered":"2024-04-23T18:16:32.961331+0000","last_unstale":"2024-04-23T18:16:33.976815+0000","last_undegraded":"2024-04-23T18:16:33.976815+0000","last_fullsized":"2024-04-23T18:16:33.976815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:59:17.665288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985806+0000","last_change":"2024-04-23T18:16:33.996572+0000","last_active":"2024-04-23T18:16:34.985806+0000","last_peered":"2024-04-23T18:16:34.985806+0000","last_clean":"2024-04-23T18:16:34.985806+0000","last_became_active":"2024-04-23T18:16:33.994839+0000","last_became_peered":"2024-04-23T18:16:33.994839+0000","last_unstale":"2024-04-23T18:16:34.985806+0000","last_undegraded":"2024-04-23T18:16:34.985806+0000","last_fullsized":"2024-04-23T18:16:34.985806+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:40:02.176521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.968358+0000","last_change":"2024-04-23T18:16:32.954935+0000","last_active":"2024-04-23T18:16:36.968358+0000","last_peered":"2024-04-23T18:16:36.968358+0000","last_clean":"2024-04-23T18:16:36.968358+0000","last_became_active":"2024-04-23T18:16:32.954595+0000","last_became_peered":"2024-04-23T18:16:32.954595+0000","last_unstale":"2024-04-23T18:16:36.968358+0000","last_undegraded":"2024-04-23T18:16:36.968358+0000","last_fullsized":"2024-04-23T18:16:36.968358+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:22.954101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985307+0000","last_change":"2024-04-23T18:16:34.000215+0000","last_active":"2024-04-23T18:16:34.985307+0000","last_peered":"2024-04-23T18:16:34.985307+0000","last_clean":"2024-04-23T18:16:34.985307+0000","last_became_active":"2024-04-23T18:16:33.999855+0000","last_became_peered":"2024-04-23T18:16:33.999855+0000","last_unstale":"2024-04-23T18:16:34.985307+0000","last_undegraded":"2024-04-23T18:16:34.985307+0000","last_fullsized":"2024-04-23T18:16:34.985307+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:14.686916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.971055+0000","last_change":"2024-04-23T18:16:32.966907+0000","last_active":"2024-04-23T18:16:36.971055+0000","last_peered":"2024-04-23T18:16:36.971055+0000","last_clean":"2024-04-23T18:16:36.971055+0000","last_became_active":"2024-04-23T18:16:32.966530+0000","last_became_peered":"2024-04-23T18:16:32.966530+0000","last_unstale":"2024-04-23T18:16:36.971055+0000","last_undegraded":"2024-04-23T18:16:36.971055+0000","last_fullsized":"2024-04-23T18:16:36.971055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:30.631851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.999398+0000","last_change":"2024-04-23T18:16:33.998817+0000","last_active":"2024-04-23T18:16:33.999398+0000","last_peered":"2024-04-23T18:16:33.999398+0000","last_clean":"2024-04-23T18:16:33.999398+0000","last_became_active":"2024-04-23T18:16:33.997982+0000","last_became_peered":"2024-04-23T18:16:33.997982+0000","last_unstale":"2024-04-23T18:16:33.999398+0000","last_undegraded":"2024-04-23T18:16:33.999398+0000","last_fullsized":"2024-04-23T18:16:33.999398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:10:32.635610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:37.964985+0000","last_change":"2024-04-23T18:16:32.959253+0000","last_active":"2024-04-23T18:16:37.964985+0000","last_peered":"2024-04-23T18:16:37.964985+0000","last_clean":"2024-04-23T18:16:37.964985+0000","last_became_active":"2024-04-23T18:16:32.958929+0000","last_became_peered":"2024-04-23T18:16:32.958929+0000","last_unstale":"2024-04-23T18:16:37.964985+0000","last_undegraded":"2024-04-23T18:16:37.964985+0000","last_fullsized":"2024-04-23T18:16:37.964985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:36.317516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966233+0000","last_change":"2024-04-23T18:16:33.979464+0000","last_active":"2024-04-23T18:16:35.966233+0000","last_peered":"2024-04-23T18:16:35.966233+0000","last_clean":"2024-04-23T18:16:35.966233+0000","last_became_active":"2024-04-23T18:16:33.979106+0000","last_became_peered":"2024-04-23T18:16:33.979106+0000","last_unstale":"2024-04-23T18:16:35.966233+0000","last_undegraded":"2024-04-23T18:16:35.966233+0000","last_fullsized":"2024-04-23T18:16:35.966233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:00.125514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.969236+0000","last_change":"2024-04-23T18:16:32.960812+0000","last_active":"2024-04-23T18:16:33.969236+0000","last_peered":"2024-04-23T18:16:33.969236+0000","last_clean":"2024-04-23T18:16:33.969236+0000","last_became_active":"2024-04-23T18:16:32.959807+0000","last_became_peered":"2024-04-23T18:16:32.959807+0000","last_unstale":"2024-04-23T18:16:33.969236+0000","last_undegraded":"2024-04-23T18:16:33.969236+0000","last_fullsized":"2024-04-23T18:16:33.969236+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:45.097443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961994+0000","last_change":"2024-04-23T18:16:33.992891+0000","last_active":"2024-04-23T18:16:35.961994+0000","last_peered":"2024-04-23T18:16:35.961994+0000","last_clean":"2024-04-23T18:16:35.961994+0000","last_became_active":"2024-04-23T18:16:33.992559+0000","last_became_peered":"2024-04-23T18:16:33.992559+0000","last_unstale":"2024-04-23T18:16:35.961994+0000","last_undegraded":"2024-04-23T18:16:35.961994+0000","last_fullsized":"2024-04-23T18:16:35.961994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:48.615158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985377+0000","last_change":"2024-04-23T18:16:32.957015+0000","last_active":"2024-04-23T18:16:34.985377+0000","last_peered":"2024-04-23T18:16:34.985377+0000","last_clean":"2024-04-23T18:16:34.985377+0000","last_became_active":"2024-04-23T18:16:32.956690+0000","last_became_peered":"2024-04-23T18:16:32.956690+0000","last_unstale":"2024-04-23T18:16:34.985377+0000","last_undegraded":"2024-04-23T18:16:34.985377+0000","last_fullsized":"2024-04-23T18:16:34.985377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:20:42.087521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961653+0000","last_change":"2024-04-23T18:16:33.998906+0000","last_active":"2024-04-23T18:16:35.961653+0000","last_peered":"2024-04-23T18:16:35.961653+0000","last_clean":"2024-04-23T18:16:35.961653+0000","last_became_active":"2024-04-23T18:16:33.997297+0000","last_became_peered":"2024-04-23T18:16:33.997297+0000","last_unstale":"2024-04-23T18:16:35.961653+0000","last_undegraded":"2024-04-23T18:16:35.961653+0000","last_fullsized":"2024-04-23T18:16:35.961653+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:06.668830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.987932+0000","last_change":"2024-04-23T18:16:33.987529+0000","last_active":"2024-04-23T18:16:33.987932+0000","last_peered":"2024-04-23T18:16:33.987932+0000","last_clean":"2024-04-23T18:16:33.987932+0000","last_became_active":"2024-04-23T18:16:33.987215+0000","last_became_peered":"2024-04-23T18:16:33.987215+0000","last_unstale":"2024-04-23T18:16:33.987932+0000","last_undegraded":"2024-04-23T18:16:33.987932+0000","last_fullsized":"2024-04-23T18:16:33.987932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:48.580489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:37.972347+0000","last_change":"2024-04-23T18:16:32.961423+0000","last_active":"2024-04-23T18:16:37.972347+0000","last_peered":"2024-04-23T18:16:37.972347+0000","last_clean":"2024-04-23T18:16:37.972347+0000","last_became_active":"2024-04-23T18:16:32.961116+0000","last_became_peered":"2024-04-23T18:16:32.961116+0000","last_unstale":"2024-04-23T18:16:37.972347+0000","last_undegraded":"2024-04-23T18:16:37.972347+0000","last_fullsized":"2024-04-23T18:16:37.972347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:28.525642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.993245+0000","last_change":"2024-04-23T18:16:33.990747+0000","last_active":"2024-04-23T18:16:33.993245+0000","last_peered":"2024-04-23T18:16:33.993245+0000","last_clean":"2024-04-23T18:16:33.993245+0000","last_became_active":"2024-04-23T18:16:33.989972+0000","last_became_peered":"2024-04-23T18:16:33.989972+0000","last_unstale":"2024-04-23T18:16:33.993245+0000","last_undegraded":"2024-04-23T18:16:33.993245+0000","last_fullsized":"2024-04-23T18:16:33.993245+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:15:03.047353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.965328+0000","last_change":"2024-04-23T18:16:32.952821+0000","last_active":"2024-04-23T18:16:35.965328+0000","last_peered":"2024-04-23T18:16:35.965328+0000","last_clean":"2024-04-23T18:16:35.965328+0000","last_became_active":"2024-04-23T18:16:32.952470+0000","last_became_peered":"2024-04-23T18:16:32.952470+0000","last_unstale":"2024-04-23T18:16:35.965328+0000","last_undegraded":"2024-04-23T18:16:35.965328+0000","last_fullsized":"2024-04-23T18:16:35.965328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:19.392670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":349,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985221+0000","last_change":"2024-04-23T18:15:25.867370+0000","last_active":"2024-04-23T18:16:34.985221+0000","last_peered":"2024-04-23T18:16:34.985221+0000","last_clean":"2024-04-23T18:16:34.985221+0000","last_became_active":"2024-04-23T18:15:25.866815+0000","last_became_peered":"2024-04-23T18:15:25.866815+0000","last_unstale":"2024-04-23T18:16:34.985221+0000","last_undegraded":"2024-04-23T18:16:34.985221+0000","last_fullsized":"2024-04-23T18:16:34.985221+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_clean_scrub_stamp":"2024-04-23T18:12:02.913350+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:42.482422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.984495+0000","last_change":"2024-04-23T18:16:33.992988+0000","last_active":"2024-04-23T18:16:34.984495+0000","last_peered":"2024-04-23T18:16:34.984495+0000","last_clean":"2024-04-23T18:16:34.984495+0000","last_became_active":"2024-04-23T18:16:33.992650+0000","last_became_peered":"2024-04-23T18:16:33.992650+0000","last_unstale":"2024-04-23T18:16:34.984495+0000","last_undegraded":"2024-04-23T18:16:34.984495+0000","last_fullsized":"2024-04-23T18:16:34.984495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:54.054708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.968944+0000","last_change":"2024-04-23T18:16:32.957318+0000","last_active":"2024-04-23T18:16:36.968944+0000","last_peered":"2024-04-23T18:16:36.968944+0000","last_clean":"2024-04-23T18:16:36.968944+0000","last_became_active":"2024-04-23T18:16:32.956973+0000","last_became_peered":"2024-04-23T18:16:32.956973+0000","last_unstale":"2024-04-23T18:16:36.968944+0000","last_undegraded":"2024-04-23T18:16:36.968944+0000","last_fullsized":"2024-04-23T18:16:36.968944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:35.684825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.958607+0000","last_change":"2024-04-23T18:16:32.966814+0000","last_active":"2024-04-23T18:16:35.958607+0000","last_peered":"2024-04-23T18:16:35.958607+0000","last_clean":"2024-04-23T18:16:35.958607+0000","last_became_active":"2024-04-23T18:16:32.966451+0000","last_became_peered":"2024-04-23T18:16:32.966451+0000","last_unstale":"2024-04-23T18:16:35.958607+0000","last_undegraded":"2024-04-23T18:16:35.958607+0000","last_fullsized":"2024-04-23T18:16:35.958607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:08:22.325576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.984938+0000","last_change":"2024-04-23T18:16:33.996345+0000","last_active":"2024-04-23T18:16:34.984938+0000","last_peered":"2024-04-23T18:16:34.984938+0000","last_clean":"2024-04-23T18:16:34.984938+0000","last_became_active":"2024-04-23T18:16:33.993802+0000","last_became_peered":"2024-04-23T18:16:33.993802+0000","last_unstale":"2024-04-23T18:16:34.984938+0000","last_undegraded":"2024-04-23T18:16:34.984938+0000","last_fullsized":"2024-04-23T18:16:34.984938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:32.410330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.969822+0000","last_change":"2024-04-23T18:16:32.957953+0000","last_active":"2024-04-23T18:16:33.969822+0000","last_peered":"2024-04-23T18:16:33.969822+0000","last_clean":"2024-04-23T18:16:33.969822+0000","last_became_active":"2024-04-23T18:16:32.957616+0000","last_became_peered":"2024-04-23T18:16:32.957616+0000","last_unstale":"2024-04-23T18:16:33.969822+0000","last_undegraded":"2024-04-23T18:16:33.969822+0000","last_fullsized":"2024-04-23T18:16:33.969822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:15.485373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.986335+0000","last_change":"2024-04-23T18:16:33.985685+0000","last_active":"2024-04-23T18:16:33.986335+0000","last_peered":"2024-04-23T18:16:33.986335+0000","last_clean":"2024-04-23T18:16:33.986335+0000","last_became_active":"2024-04-23T18:16:33.984631+0000","last_became_peered":"2024-04-23T18:16:33.984631+0000","last_unstale":"2024-04-23T18:16:33.986335+0000","last_undegraded":"2024-04-23T18:16:33.986335+0000","last_fullsized":"2024-04-23T18:16:33.986335+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:24.236851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.965256+0000","last_change":"2024-04-23T18:16:32.954281+0000","last_active":"2024-04-23T18:16:35.965256+0000","last_peered":"2024-04-23T18:16:35.965256+0000","last_clean":"2024-04-23T18:16:35.965256+0000","last_became_active":"2024-04-23T18:16:32.953988+0000","last_became_peered":"2024-04-23T18:16:32.953988+0000","last_unstale":"2024-04-23T18:16:35.965256+0000","last_undegraded":"2024-04-23T18:16:35.965256+0000","last_fullsized":"2024-04-23T18:16:35.965256+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:09.049861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.438914+0000","last_change":"2024-04-23T18:16:33.994860+0000","last_active":"2024-04-23T18:16:35.438914+0000","last_peered":"2024-04-23T18:16:35.438914+0000","last_clean":"2024-04-23T18:16:35.438914+0000","last_became_active":"2024-04-23T18:16:33.993438+0000","last_became_peered":"2024-04-23T18:16:33.993438+0000","last_unstale":"2024-04-23T18:16:35.438914+0000","last_undegraded":"2024-04-23T18:16:35.438914+0000","last_fullsized":"2024-04-23T18:16:35.438914+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:55.565873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961163+0000","last_change":"2024-04-23T18:16:32.959251+0000","last_active":"2024-04-23T18:16:35.961163+0000","last_peered":"2024-04-23T18:16:35.961163+0000","last_clean":"2024-04-23T18:16:35.961163+0000","last_became_active":"2024-04-23T18:16:32.958937+0000","last_became_peered":"2024-04-23T18:16:32.958937+0000","last_unstale":"2024-04-23T18:16:35.961163+0000","last_undegraded":"2024-04-23T18:16:35.961163+0000","last_fullsized":"2024-04-23T18:16:35.961163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.126634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.989446+0000","last_change":"2024-04-23T18:16:33.995480+0000","last_active":"2024-04-23T18:16:34.989446+0000","last_peered":"2024-04-23T18:16:34.989446+0000","last_clean":"2024-04-23T18:16:34.989446+0000","last_became_active":"2024-04-23T18:16:33.994361+0000","last_became_peered":"2024-04-23T18:16:33.994361+0000","last_unstale":"2024-04-23T18:16:34.989446+0000","last_undegraded":"2024-04-23T18:16:34.989446+0000","last_fullsized":"2024-04-23T18:16:34.989446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:59:35.240562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.979120+0000","last_change":"2024-04-23T18:16:32.967787+0000","last_active":"2024-04-23T18:16:33.979120+0000","last_peered":"2024-04-23T18:16:33.979120+0000","last_clean":"2024-04-23T18:16:33.979120+0000","last_became_active":"2024-04-23T18:16:32.967463+0000","last_became_peered":"2024-04-23T18:16:32.967463+0000","last_unstale":"2024-04-23T18:16:33.979120+0000","last_undegraded":"2024-04-23T18:16:33.979120+0000","last_fullsized":"2024-04-23T18:16:33.979120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:58.743682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966167+0000","last_change":"2024-04-23T18:16:33.980558+0000","last_active":"2024-04-23T18:16:35.966167+0000","last_peered":"2024-04-23T18:16:35.966167+0000","last_clean":"2024-04-23T18:16:35.966167+0000","last_became_active":"2024-04-23T18:16:33.980146+0000","last_became_peered":"2024-04-23T18:16:33.980146+0000","last_unstale":"2024-04-23T18:16:35.966167+0000","last_undegraded":"2024-04-23T18:16:35.966167+0000","last_fullsized":"2024-04-23T18:16:35.966167+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:58.001833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.958741+0000","last_change":"2024-04-23T18:16:32.962139+0000","last_active":"2024-04-23T18:16:35.958741+0000","last_peered":"2024-04-23T18:16:35.958741+0000","last_clean":"2024-04-23T18:16:35.958741+0000","last_became_active":"2024-04-23T18:16:32.961785+0000","last_became_peered":"2024-04-23T18:16:32.961785+0000","last_unstale":"2024-04-23T18:16:35.958741+0000","last_undegraded":"2024-04-23T18:16:35.958741+0000","last_fullsized":"2024-04-23T18:16:35.958741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:09.619414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.993716+0000","last_change":"2024-04-23T18:16:33.992902+0000","last_active":"2024-04-23T18:16:33.993716+0000","last_peered":"2024-04-23T18:16:33.993716+0000","last_clean":"2024-04-23T18:16:33.993716+0000","last_became_active":"2024-04-23T18:16:33.992574+0000","last_became_peered":"2024-04-23T18:16:33.992574+0000","last_unstale":"2024-04-23T18:16:33.993716+0000","last_undegraded":"2024-04-23T18:16:33.993716+0000","last_fullsized":"2024-04-23T18:16:33.993716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:52:02.827360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.962093+0000","last_change":"2024-04-23T18:16:32.956965+0000","last_active":"2024-04-23T18:16:35.962093+0000","last_peered":"2024-04-23T18:16:35.962093+0000","last_clean":"2024-04-23T18:16:35.962093+0000","last_became_active":"2024-04-23T18:16:32.956672+0000","last_became_peered":"2024-04-23T18:16:32.956672+0000","last_unstale":"2024-04-23T18:16:35.962093+0000","last_undegraded":"2024-04-23T18:16:35.962093+0000","last_fullsized":"2024-04-23T18:16:35.962093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:00:44.470809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985573+0000","last_change":"2024-04-23T18:16:34.002711+0000","last_active":"2024-04-23T18:16:34.985573+0000","last_peered":"2024-04-23T18:16:34.985573+0000","last_clean":"2024-04-23T18:16:34.985573+0000","last_became_active":"2024-04-23T18:16:34.002365+0000","last_became_peered":"2024-04-23T18:16:34.002365+0000","last_unstale":"2024-04-23T18:16:34.985573+0000","last_undegraded":"2024-04-23T18:16:34.985573+0000","last_fullsized":"2024-04-23T18:16:34.985573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:52:11.982735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.970897+0000","last_change":"2024-04-23T18:16:32.965121+0000","last_active":"2024-04-23T18:16:36.970897+0000","last_peered":"2024-04-23T18:16:36.970897+0000","last_clean":"2024-04-23T18:16:36.970897+0000","last_became_active":"2024-04-23T18:16:32.964732+0000","last_became_peered":"2024-04-23T18:16:32.964732+0000","last_unstale":"2024-04-23T18:16:36.970897+0000","last_undegraded":"2024-04-23T18:16:36.970897+0000","last_fullsized":"2024-04-23T18:16:36.970897+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:19:34.690245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985896+0000","last_change":"2024-04-23T18:16:33.996357+0000","last_active":"2024-04-23T18:16:34.985896+0000","last_peered":"2024-04-23T18:16:34.985896+0000","last_clean":"2024-04-23T18:16:34.985896+0000","last_became_active":"2024-04-23T18:16:33.996017+0000","last_became_peered":"2024-04-23T18:16:33.996017+0000","last_unstale":"2024-04-23T18:16:34.985896+0000","last_undegraded":"2024-04-23T18:16:34.985896+0000","last_fullsized":"2024-04-23T18:16:34.985896+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:20:57.321228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.004852+0000","last_change":"2024-04-23T18:16:32.961705+0000","last_active":"2024-04-23T18:16:34.004852+0000","last_peered":"2024-04-23T18:16:34.004852+0000","last_clean":"2024-04-23T18:16:34.004852+0000","last_became_active":"2024-04-23T18:16:32.961342+0000","last_became_peered":"2024-04-23T18:16:32.961342+0000","last_unstale":"2024-04-23T18:16:34.004852+0000","last_undegraded":"2024-04-23T18:16:34.004852+0000","last_fullsized":"2024-04-23T18:16:34.004852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:51.340910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985786+0000","last_change":"2024-04-23T18:16:33.992944+0000","last_active":"2024-04-23T18:16:34.985786+0000","last_peered":"2024-04-23T18:16:34.985786+0000","last_clean":"2024-04-23T18:16:34.985786+0000","last_became_active":"2024-04-23T18:16:33.992579+0000","last_became_peered":"2024-04-23T18:16:33.992579+0000","last_unstale":"2024-04-23T18:16:34.985786+0000","last_undegraded":"2024-04-23T18:16:34.985786+0000","last_fullsized":"2024-04-23T18:16:34.985786+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:13:37.829588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.984869+0000","last_change":"2024-04-23T18:16:32.955192+0000","last_active":"2024-04-23T18:16:34.984869+0000","last_peered":"2024-04-23T18:16:34.984869+0000","last_clean":"2024-04-23T18:16:34.984869+0000","last_became_active":"2024-04-23T18:16:32.954860+0000","last_became_peered":"2024-04-23T18:16:32.954860+0000","last_unstale":"2024-04-23T18:16:34.984869+0000","last_undegraded":"2024-04-23T18:16:34.984869+0000","last_fullsized":"2024-04-23T18:16:34.984869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:20.796060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985954+0000","last_change":"2024-04-23T18:16:33.986211+0000","last_active":"2024-04-23T18:16:34.985954+0000","last_peered":"2024-04-23T18:16:34.985954+0000","last_clean":"2024-04-23T18:16:34.985954+0000","last_became_active":"2024-04-23T18:16:33.985679+0000","last_became_peered":"2024-04-23T18:16:33.985679+0000","last_unstale":"2024-04-23T18:16:34.985954+0000","last_undegraded":"2024-04-23T18:16:34.985954+0000","last_fullsized":"2024-04-23T18:16:34.985954+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:03:38.797864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966363+0000","last_change":"2024-04-23T18:16:33.979462+0000","last_active":"2024-04-23T18:16:35.966363+0000","last_peered":"2024-04-23T18:16:35.966363+0000","last_clean":"2024-04-23T18:16:35.966363+0000","last_became_active":"2024-04-23T18:16:33.979114+0000","last_became_peered":"2024-04-23T18:16:33.979114+0000","last_unstale":"2024-04-23T18:16:35.966363+0000","last_undegraded":"2024-04-23T18:16:35.966363+0000","last_fullsized":"2024-04-23T18:16:35.966363+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:16:49.708240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.958490+0000","last_change":"2024-04-23T18:16:33.979255+0000","last_active":"2024-04-23T18:16:35.958490+0000","last_peered":"2024-04-23T18:16:35.958490+0000","last_clean":"2024-04-23T18:16:35.958490+0000","last_became_active":"2024-04-23T18:16:33.978934+0000","last_became_peered":"2024-04-23T18:16:33.978934+0000","last_unstale":"2024-04-23T18:16:35.958490+0000","last_undegraded":"2024-04-23T18:16:35.958490+0000","last_fullsized":"2024-04-23T18:16:35.958490+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:10.104564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.958676+0000","last_change":"2024-04-23T18:16:33.981877+0000","last_active":"2024-04-23T18:16:35.958676+0000","last_peered":"2024-04-23T18:16:35.958676+0000","last_clean":"2024-04-23T18:16:35.958676+0000","last_became_active":"2024-04-23T18:16:33.981536+0000","last_became_peered":"2024-04-23T18:16:33.981536+0000","last_unstale":"2024-04-23T18:16:35.958676+0000","last_undegraded":"2024-04-23T18:16:35.958676+0000","last_fullsized":"2024-04-23T18:16:35.958676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:00:15.056779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.438881+0000","last_change":"2024-04-23T18:16:33.999090+0000","last_active":"2024-04-23T18:16:35.438881+0000","last_peered":"2024-04-23T18:16:35.438881+0000","last_clean":"2024-04-23T18:16:35.438881+0000","last_became_active":"2024-04-23T18:16:33.998513+0000","last_became_peered":"2024-04-23T18:16:33.998513+0000","last_unstale":"2024-04-23T18:16:35.438881+0000","last_undegraded":"2024-04-23T18:16:35.438881+0000","last_fullsized":"2024-04-23T18:16:35.438881+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:14.115406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985020+0000","last_change":"2024-04-23T18:16:33.993375+0000","last_active":"2024-04-23T18:16:34.985020+0000","last_peered":"2024-04-23T18:16:34.985020+0000","last_clean":"2024-04-23T18:16:34.985020+0000","last_became_active":"2024-04-23T18:16:33.991841+0000","last_became_peered":"2024-04-23T18:16:33.991841+0000","last_unstale":"2024-04-23T18:16:34.985020+0000","last_undegraded":"2024-04-23T18:16:34.985020+0000","last_fullsized":"2024-04-23T18:16:34.985020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:12.777495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966416+0000","last_change":"2024-04-23T18:16:33.983621+0000","last_active":"2024-04-23T18:16:35.966416+0000","last_peered":"2024-04-23T18:16:35.966416+0000","last_clean":"2024-04-23T18:16:35.966416+0000","last_became_active":"2024-04-23T18:16:33.983356+0000","last_became_peered":"2024-04-23T18:16:33.983356+0000","last_unstale":"2024-04-23T18:16:35.966416+0000","last_undegraded":"2024-04-23T18:16:35.966416+0000","last_fullsized":"2024-04-23T18:16:35.966416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:27:19.391281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961380+0000","last_change":"2024-04-23T18:16:33.987952+0000","last_active":"2024-04-23T18:16:35.961380+0000","last_peered":"2024-04-23T18:16:35.961380+0000","last_clean":"2024-04-23T18:16:35.961380+0000","last_became_active":"2024-04-23T18:16:33.987334+0000","last_became_peered":"2024-04-23T18:16:33.987334+0000","last_unstale":"2024-04-23T18:16:35.961380+0000","last_undegraded":"2024-04-23T18:16:35.961380+0000","last_fullsized":"2024-04-23T18:16:35.961380+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:41.902779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.985378+0000","last_change":"2024-04-23T18:16:33.983914+0000","last_active":"2024-04-23T18:16:33.985378+0000","last_peered":"2024-04-23T18:16:33.985378+0000","last_clean":"2024-04-23T18:16:33.985378+0000","last_became_active":"2024-04-23T18:16:33.983444+0000","last_became_peered":"2024-04-23T18:16:33.983444+0000","last_unstale":"2024-04-23T18:16:33.985378+0000","last_undegraded":"2024-04-23T18:16:33.985378+0000","last_fullsized":"2024-04-23T18:16:33.985378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:41:29.865551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.993010+0000","last_change":"2024-04-23T18:16:33.992496+0000","last_active":"2024-04-23T18:16:33.993010+0000","last_peered":"2024-04-23T18:16:33.993010+0000","last_clean":"2024-04-23T18:16:33.993010+0000","last_became_active":"2024-04-23T18:16:33.992155+0000","last_became_peered":"2024-04-23T18:16:33.992155+0000","last_unstale":"2024-04-23T18:16:33.993010+0000","last_undegraded":"2024-04-23T18:16:33.993010+0000","last_fullsized":"2024-04-23T18:16:33.993010+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:33:28.503307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.985412+0000","last_change":"2024-04-23T18:16:33.984409+0000","last_active":"2024-04-23T18:16:33.985412+0000","last_peered":"2024-04-23T18:16:33.985412+0000","last_clean":"2024-04-23T18:16:33.985412+0000","last_became_active":"2024-04-23T18:16:33.984073+0000","last_became_peered":"2024-04-23T18:16:33.984073+0000","last_unstale":"2024-04-23T18:16:33.985412+0000","last_undegraded":"2024-04-23T18:16:33.985412+0000","last_fullsized":"2024-04-23T18:16:33.985412+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:49:05.598391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.799737+0000","last_change":"2024-04-23T18:16:33.986376+0000","last_active":"2024-04-23T18:16:35.799737+0000","last_peered":"2024-04-23T18:16:35.799737+0000","last_clean":"2024-04-23T18:16:35.799737+0000","last_became_active":"2024-04-23T18:16:33.986012+0000","last_became_peered":"2024-04-23T18:16:33.986012+0000","last_unstale":"2024-04-23T18:16:35.799737+0000","last_undegraded":"2024-04-23T18:16:35.799737+0000","last_fullsized":"2024-04-23T18:16:35.799737+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:26.940215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.984836+0000","last_change":"2024-04-23T18:16:33.999795+0000","last_active":"2024-04-23T18:16:34.984836+0000","last_peered":"2024-04-23T18:16:34.984836+0000","last_clean":"2024-04-23T18:16:34.984836+0000","last_became_active":"2024-04-23T18:16:33.997034+0000","last_became_peered":"2024-04-23T18:16:33.997034+0000","last_unstale":"2024-04-23T18:16:34.984836+0000","last_undegraded":"2024-04-23T18:16:34.984836+0000","last_fullsized":"2024-04-23T18:16:34.984836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:23.983846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966543+0000","last_change":"2024-04-23T18:16:33.985495+0000","last_active":"2024-04-23T18:16:35.966543+0000","last_peered":"2024-04-23T18:16:35.966543+0000","last_clean":"2024-04-23T18:16:35.966543+0000","last_became_active":"2024-04-23T18:16:33.985178+0000","last_became_peered":"2024-04-23T18:16:33.985178+0000","last_unstale":"2024-04-23T18:16:35.966543+0000","last_undegraded":"2024-04-23T18:16:35.966543+0000","last_fullsized":"2024-04-23T18:16:35.966543+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:22:25.325756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3928,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":101,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318284,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":937984,"data_stored":450534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32712,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712440,"statfs":{"total":95995035648,"available":95961538560,"internally_reserved":0,"allocated":1568768,"data_stored":1043470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776151,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":929792,"data_stored":442546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037788,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":451272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32692,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712460,"statfs":{"total":95995035648,"available":95961559040,"internally_reserved":0,"allocated":1548288,"data_stored":1042694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":450741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":452146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40440,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704712,"statfs":{"total":95995035648,"available":95953625088,"internally_reserved":0,"allocated":966656,"data_stored":452012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":974848,"data_stored":453496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":933888,"data_stored":442068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":942080,"data_stored":450556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1531904,"data_stored":1032514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:16:40.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:16:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.932+0000 7f3e57101640 1 -- 172.21.15.50:0/1687875297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e50102a00 msgr2=0x7f3e50109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.932+0000 7f3e57101640 1 --2- 172.21.15.50:0/1687875297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e50102a00 0x7f3e50109320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009ca0 tx=0x7f3e4802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.932+0000 7f3e57101640 1 -- 172.21.15.50:0/1687875297 shutdown_connections 2024-04-23T18:16:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.932+0000 7f3e57101640 1 --2- 172.21.15.50:0/1687875297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e50102a00 0x7f3e50109320 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.932+0000 7f3e57101640 1 --2- 172.21.15.50:0/1687875297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e50102040 0x7f3e501024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.932+0000 7f3e57101640 1 --2- 172.21.15.50:0/1687875297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50100e60 0x7f3e50101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.932+0000 7f3e57101640 1 -- 172.21.15.50:0/1687875297 >> 172.21.15.50:0/1687875297 conn(0x7f3e500fc440 msgr2=0x7f3e500fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.932+0000 7f3e57101640 1 -- 172.21.15.50:0/1687875297 shutdown_connections 2024-04-23T18:16:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.932+0000 7f3e57101640 1 -- 172.21.15.50:0/1687875297 wait complete. 2024-04-23T18:16:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.932+0000 7f3e57101640 1 Processor -- start 2024-04-23T18:16:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e57101640 1 -- start start 2024-04-23T18:16:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e57101640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e50100e60 0x7f3e50104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e57101640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e50102040 0x7f3e501034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e57101640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50102a00 0x7f3e501039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e57101640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5010ca40 con 0x7f3e50100e60 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e57101640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5010c8c0 con 0x7f3e50102a00 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e57101640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5010cbc0 con 0x7f3e50102040 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e54e9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e50100e60 0x7f3e50104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e54e9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e50100e60 0x7f3e50104e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38882/0 (socket says 172.21.15.50:38882) 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e54e9d640 1 -- 172.21.15.50:0/77841759 learned_addr learned my addr 172.21.15.50:0/77841759 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e5569e640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50102a00 0x7f3e501039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e4ffff640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e50102040 0x7f3e501034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e54e9d640 1 -- 172.21.15.50:0/77841759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e50102040 msgr2=0x7f3e501034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e54e9d640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e50102040 0x7f3e501034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e54e9d640 1 -- 172.21.15.50:0/77841759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50102a00 msgr2=0x7f3e501039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e54e9d640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50102a00 0x7f3e501039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e54e9d640 1 -- 172.21.15.50:0/77841759 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e50104280 con 0x7f3e50100e60 2024-04-23T18:16:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e4ffff640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e50102040 0x7f3e501034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e5569e640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50102a00 0x7f3e501039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e54e9d640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e50100e60 0x7f3e50104e70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400d770 tx=0x7f3e4400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e4dffb640 1 -- 172.21.15.50:0/77841759 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e44017070 con 0x7f3e50100e60 2024-04-23T18:16:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e4dffb640 1 -- 172.21.15.50:0/77841759 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e44009c30 con 0x7f3e50100e60 2024-04-23T18:16:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e57101640 1 -- 172.21.15.50:0/77841759 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e501a2290 con 0x7f3e50100e60 2024-04-23T18:16:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.933+0000 7f3e4dffb640 1 -- 172.21.15.50:0/77841759 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e44003cc0 con 0x7f3e50100e60 2024-04-23T18:16:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.934+0000 7f3e57101640 1 -- 172.21.15.50:0/77841759 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e501a2830 con 0x7f3e50100e60 2024-04-23T18:16:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.935+0000 7f3e57101640 1 -- 172.21.15.50:0/77841759 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e501a2af0 con 0x7f3e50100e60 2024-04-23T18:16:40.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.936+0000 7f3e4dffb640 1 -- 172.21.15.50:0/77841759 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3e44003e60 con 0x7f3e50100e60 2024-04-23T18:16:40.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.936+0000 7f3e4dffb640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3e400a3cc0 0x7f3e400a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:40.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.936+0000 7f3e4ffff640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3e400a3cc0 0x7f3e400a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:40.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.936+0000 7f3e4dffb640 1 -- 172.21.15.50:0/77841759 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f3e440cfda0 con 0x7f3e50100e60 2024-04-23T18:16:40.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.937+0000 7f3e4ffff640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3e400a3cc0 0x7f3e400a6180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3e50104be0 tx=0x7f3e3c005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:40.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:40.941+0000 7f3e4dffb640 1 -- 172.21.15.50:0/77841759 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e44009de0 con 0x7f3e50100e60 2024-04-23T18:16:41.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.124+0000 7f3e57101640 1 -- 172.21.15.50:0/77841759 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3e500fe4f0 con 0x7f3e400a3cc0 2024-04-23T18:16:41.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.131+0000 7f3e4dffb640 1 -- 172.21.15.50:0/77841759 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258772 (secure 0 0 0) 0x7f3e500fe4f0 con 0x7f3e400a3cc0 2024-04-23T18:16:41.127 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:41.129 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-23T18:16:41.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.135+0000 7f3e57101640 1 -- 172.21.15.50:0/77841759 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3e400a3cc0 msgr2=0x7f3e400a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:41.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.135+0000 7f3e57101640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3e400a3cc0 0x7f3e400a6180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3e50104be0 tx=0x7f3e3c005d90 comp rx=0 tx=0).stop 2024-04-23T18:16:41.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.135+0000 7f3e57101640 1 -- 172.21.15.50:0/77841759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e50100e60 msgr2=0x7f3e50104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:41.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.135+0000 7f3e57101640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e50100e60 0x7f3e50104e70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400d770 tx=0x7f3e4400dc30 comp rx=0 tx=0).stop 2024-04-23T18:16:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.135+0000 7f3e57101640 1 -- 172.21.15.50:0/77841759 shutdown_connections 2024-04-23T18:16:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.135+0000 7f3e57101640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3e400a3cc0 0x7f3e400a6180 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.135+0000 7f3e57101640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50102a00 0x7f3e501039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.135+0000 7f3e57101640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e50102040 0x7f3e501034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.135+0000 7f3e57101640 1 --2- 172.21.15.50:0/77841759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e50100e60 0x7f3e50104e70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.135+0000 7f3e57101640 1 -- 172.21.15.50:0/77841759 >> 172.21.15.50:0/77841759 conn(0x7f3e500fc440 msgr2=0x7f3e500fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.135+0000 7f3e57101640 1 -- 172.21.15.50:0/77841759 shutdown_connections 2024-04-23T18:16:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:41.135+0000 7f3e57101640 1 -- 172.21.15.50:0/77841759 wait complete. 2024-04-23T18:16:41.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:41 smithi050 ceph-mon[29187]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:41.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:41 smithi050 ceph-mon[29187]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:41.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:41 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2933436550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:41.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:41 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1690086715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:41.140 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-04-23T18:16:40.501319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594296,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":143,"num_read_kb":126,"num_write":282,"num_write_kb":2972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":206,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":387036,"kb_used_data":12880,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124554788,"statfs":{"total":1151940427776,"available":1151544102912,"internally_reserved":0,"allocated":13189120,"data_stored":7164049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3928,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":101,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.557677"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966580+0000","last_change":"2024-04-23T18:16:33.987102+0000","last_active":"2024-04-23T18:16:35.966580+0000","last_peered":"2024-04-23T18:16:35.966580+0000","last_clean":"2024-04-23T18:16:35.966580+0000","last_became_active":"2024-04-23T18:16:33.986716+0000","last_became_peered":"2024-04-23T18:16:33.986716+0000","last_unstale":"2024-04-23T18:16:35.966580+0000","last_undegraded":"2024-04-23T18:16:35.966580+0000","last_fullsized":"2024-04-23T18:16:35.966580+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:41.895219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.439035+0000","last_change":"2024-04-23T18:16:33.995703+0000","last_active":"2024-04-23T18:16:35.439035+0000","last_peered":"2024-04-23T18:16:35.439035+0000","last_clean":"2024-04-23T18:16:35.439035+0000","last_became_active":"2024-04-23T18:16:33.995242+0000","last_became_peered":"2024-04-23T18:16:33.995242+0000","last_unstale":"2024-04-23T18:16:35.439035+0000","last_undegraded":"2024-04-23T18:16:35.439035+0000","last_fullsized":"2024-04-23T18:16:35.439035+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:57.813822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.438836+0000","last_change":"2024-04-23T18:16:34.000864+0000","last_active":"2024-04-23T18:16:35.438836+0000","last_peered":"2024-04-23T18:16:35.438836+0000","last_clean":"2024-04-23T18:16:35.438836+0000","last_became_active":"2024-04-23T18:16:33.998934+0000","last_became_peered":"2024-04-23T18:16:33.998934+0000","last_unstale":"2024-04-23T18:16:35.438836+0000","last_undegraded":"2024-04-23T18:16:35.438836+0000","last_fullsized":"2024-04-23T18:16:35.438836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:47:29.481697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961511+0000","last_change":"2024-04-23T18:16:33.989351+0000","last_active":"2024-04-23T18:16:35.961511+0000","last_peered":"2024-04-23T18:16:35.961511+0000","last_clean":"2024-04-23T18:16:35.961511+0000","last_became_active":"2024-04-23T18:16:33.989030+0000","last_became_peered":"2024-04-23T18:16:33.989030+0000","last_unstale":"2024-04-23T18:16:35.961511+0000","last_undegraded":"2024-04-23T18:16:35.961511+0000","last_fullsized":"2024-04-23T18:16:35.961511+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:29:03.023675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.989528+0000","last_change":"2024-04-23T18:16:33.996062+0000","last_active":"2024-04-23T18:16:34.989528+0000","last_peered":"2024-04-23T18:16:34.989528+0000","last_clean":"2024-04-23T18:16:34.989528+0000","last_became_active":"2024-04-23T18:16:33.995732+0000","last_became_peered":"2024-04-23T18:16:33.995732+0000","last_unstale":"2024-04-23T18:16:34.989528+0000","last_undegraded":"2024-04-23T18:16:34.989528+0000","last_fullsized":"2024-04-23T18:16:34.989528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:04.175580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961265+0000","last_change":"2024-04-23T18:16:33.989895+0000","last_active":"2024-04-23T18:16:35.961265+0000","last_peered":"2024-04-23T18:16:35.961265+0000","last_clean":"2024-04-23T18:16:35.961265+0000","last_became_active":"2024-04-23T18:16:33.989547+0000","last_became_peered":"2024-04-23T18:16:33.989547+0000","last_unstale":"2024-04-23T18:16:35.961265+0000","last_undegraded":"2024-04-23T18:16:35.961265+0000","last_fullsized":"2024-04-23T18:16:35.961265+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:12:37.317974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985478+0000","last_change":"2024-04-23T18:16:33.999636+0000","last_active":"2024-04-23T18:16:34.985478+0000","last_peered":"2024-04-23T18:16:34.985478+0000","last_clean":"2024-04-23T18:16:34.985478+0000","last_became_active":"2024-04-23T18:16:33.999282+0000","last_became_peered":"2024-04-23T18:16:33.999282+0000","last_unstale":"2024-04-23T18:16:34.985478+0000","last_undegraded":"2024-04-23T18:16:34.985478+0000","last_fullsized":"2024-04-23T18:16:34.985478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:28.841392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.999421+0000","last_change":"2024-04-23T18:16:33.996602+0000","last_active":"2024-04-23T18:16:33.999421+0000","last_peered":"2024-04-23T18:16:33.999421+0000","last_clean":"2024-04-23T18:16:33.999421+0000","last_became_active":"2024-04-23T18:16:33.994401+0000","last_became_peered":"2024-04-23T18:16:33.994401+0000","last_unstale":"2024-04-23T18:16:33.999421+0000","last_undegraded":"2024-04-23T18:16:33.999421+0000","last_fullsized":"2024-04-23T18:16:33.999421+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:58.575037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.800019+0000","last_change":"2024-04-23T18:16:33.994763+0000","last_active":"2024-04-23T18:16:35.800019+0000","last_peered":"2024-04-23T18:16:35.800019+0000","last_clean":"2024-04-23T18:16:35.800019+0000","last_became_active":"2024-04-23T18:16:33.994408+0000","last_became_peered":"2024-04-23T18:16:33.994408+0000","last_unstale":"2024-04-23T18:16:35.800019+0000","last_undegraded":"2024-04-23T18:16:35.800019+0000","last_fullsized":"2024-04-23T18:16:35.800019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:02.706160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961665+0000","last_change":"2024-04-23T18:16:33.993435+0000","last_active":"2024-04-23T18:16:35.961665+0000","last_peered":"2024-04-23T18:16:35.961665+0000","last_clean":"2024-04-23T18:16:35.961665+0000","last_became_active":"2024-04-23T18:16:33.993108+0000","last_became_peered":"2024-04-23T18:16:33.993108+0000","last_unstale":"2024-04-23T18:16:35.961665+0000","last_undegraded":"2024-04-23T18:16:35.961665+0000","last_fullsized":"2024-04-23T18:16:35.961665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:28:32.584896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985884+0000","last_change":"2024-04-23T18:16:33.997526+0000","last_active":"2024-04-23T18:16:34.985884+0000","last_peered":"2024-04-23T18:16:34.985884+0000","last_clean":"2024-04-23T18:16:34.985884+0000","last_became_active":"2024-04-23T18:16:33.997189+0000","last_became_peered":"2024-04-23T18:16:33.997189+0000","last_unstale":"2024-04-23T18:16:34.985884+0000","last_undegraded":"2024-04-23T18:16:34.985884+0000","last_fullsized":"2024-04-23T18:16:34.985884+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:11:41.208261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961359+0000","last_change":"2024-04-23T18:16:33.989775+0000","last_active":"2024-04-23T18:16:35.961359+0000","last_peered":"2024-04-23T18:16:35.961359+0000","last_clean":"2024-04-23T18:16:35.961359+0000","last_became_active":"2024-04-23T18:16:33.989382+0000","last_became_peered":"2024-04-23T18:16:33.989382+0000","last_unstale":"2024-04-23T18:16:35.961359+0000","last_undegraded":"2024-04-23T18:16:35.961359+0000","last_fullsized":"2024-04-23T18:16:35.961359+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:20:17.756258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.984796+0000","last_change":"2024-04-23T18:16:34.005871+0000","last_active":"2024-04-23T18:16:34.984796+0000","last_peered":"2024-04-23T18:16:34.984796+0000","last_clean":"2024-04-23T18:16:34.984796+0000","last_became_active":"2024-04-23T18:16:34.005218+0000","last_became_peered":"2024-04-23T18:16:34.005218+0000","last_unstale":"2024-04-23T18:16:34.984796+0000","last_undegraded":"2024-04-23T18:16:34.984796+0000","last_fullsized":"2024-04-23T18:16:34.984796+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:30:03.864306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.438680+0000","last_change":"2024-04-23T18:16:33.995054+0000","last_active":"2024-04-23T18:16:35.438680+0000","last_peered":"2024-04-23T18:16:35.438680+0000","last_clean":"2024-04-23T18:16:35.438680+0000","last_became_active":"2024-04-23T18:16:33.994389+0000","last_became_peered":"2024-04-23T18:16:33.994389+0000","last_unstale":"2024-04-23T18:16:35.438680+0000","last_undegraded":"2024-04-23T18:16:35.438680+0000","last_fullsized":"2024-04-23T18:16:35.438680+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:27.344616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.438661+0000","last_change":"2024-04-23T18:16:33.984490+0000","last_active":"2024-04-23T18:16:35.438661+0000","last_peered":"2024-04-23T18:16:35.438661+0000","last_clean":"2024-04-23T18:16:35.438661+0000","last_became_active":"2024-04-23T18:16:33.984071+0000","last_became_peered":"2024-04-23T18:16:33.984071+0000","last_unstale":"2024-04-23T18:16:35.438661+0000","last_undegraded":"2024-04-23T18:16:35.438661+0000","last_fullsized":"2024-04-23T18:16:35.438661+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:23.589290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.799608+0000","last_change":"2024-04-23T18:16:34.002163+0000","last_active":"2024-04-23T18:16:35.799608+0000","last_peered":"2024-04-23T18:16:35.799608+0000","last_clean":"2024-04-23T18:16:35.799608+0000","last_became_active":"2024-04-23T18:16:34.001764+0000","last_became_peered":"2024-04-23T18:16:34.001764+0000","last_unstale":"2024-04-23T18:16:35.799608+0000","last_undegraded":"2024-04-23T18:16:35.799608+0000","last_fullsized":"2024-04-23T18:16:35.799608+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:16:03.616358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966605+0000","last_change":"2024-04-23T18:16:33.987586+0000","last_active":"2024-04-23T18:16:35.966605+0000","last_peered":"2024-04-23T18:16:35.966605+0000","last_clean":"2024-04-23T18:16:35.966605+0000","last_became_active":"2024-04-23T18:16:33.987266+0000","last_became_peered":"2024-04-23T18:16:33.987266+0000","last_unstale":"2024-04-23T18:16:35.966605+0000","last_undegraded":"2024-04-23T18:16:35.966605+0000","last_fullsized":"2024-04-23T18:16:35.966605+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:41.610529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.993997+0000","last_change":"2024-04-23T18:16:33.992938+0000","last_active":"2024-04-23T18:16:33.993997+0000","last_peered":"2024-04-23T18:16:33.993997+0000","last_clean":"2024-04-23T18:16:33.993997+0000","last_became_active":"2024-04-23T18:16:33.990441+0000","last_became_peered":"2024-04-23T18:16:33.990441+0000","last_unstale":"2024-04-23T18:16:33.993997+0000","last_undegraded":"2024-04-23T18:16:33.993997+0000","last_fullsized":"2024-04-23T18:16:33.993997+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:54.914022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.959747+0000","last_change":"2024-04-23T18:16:33.994844+0000","last_active":"2024-04-23T18:16:35.959747+0000","last_peered":"2024-04-23T18:16:35.959747+0000","last_clean":"2024-04-23T18:16:35.959747+0000","last_became_active":"2024-04-23T18:16:33.994505+0000","last_became_peered":"2024-04-23T18:16:33.994505+0000","last_unstale":"2024-04-23T18:16:35.959747+0000","last_undegraded":"2024-04-23T18:16:35.959747+0000","last_fullsized":"2024-04-23T18:16:35.959747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:32:33.445266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.000893+0000","last_change":"2024-04-23T18:16:32.958841+0000","last_active":"2024-04-23T18:16:34.000893+0000","last_peered":"2024-04-23T18:16:34.000893+0000","last_clean":"2024-04-23T18:16:34.000893+0000","last_became_active":"2024-04-23T18:16:32.958529+0000","last_became_peered":"2024-04-23T18:16:32.958529+0000","last_unstale":"2024-04-23T18:16:34.000893+0000","last_undegraded":"2024-04-23T18:16:34.000893+0000","last_fullsized":"2024-04-23T18:16:34.000893+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:47:09.933786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.993360+0000","last_change":"2024-04-23T18:16:33.992108+0000","last_active":"2024-04-23T18:16:33.993360+0000","last_peered":"2024-04-23T18:16:33.993360+0000","last_clean":"2024-04-23T18:16:33.993360+0000","last_became_active":"2024-04-23T18:16:33.991751+0000","last_became_peered":"2024-04-23T18:16:33.991751+0000","last_unstale":"2024-04-23T18:16:33.993360+0000","last_undegraded":"2024-04-23T18:16:33.993360+0000","last_fullsized":"2024-04-23T18:16:33.993360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:48.631561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.003714+0000","last_change":"2024-04-23T18:16:32.960461+0000","last_active":"2024-04-23T18:16:34.003714+0000","last_peered":"2024-04-23T18:16:34.003714+0000","last_clean":"2024-04-23T18:16:34.003714+0000","last_became_active":"2024-04-23T18:16:32.960115+0000","last_became_peered":"2024-04-23T18:16:32.960115+0000","last_unstale":"2024-04-23T18:16:34.003714+0000","last_undegraded":"2024-04-23T18:16:34.003714+0000","last_fullsized":"2024-04-23T18:16:34.003714+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:30.533043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.439004+0000","last_change":"2024-04-23T18:16:34.001831+0000","last_active":"2024-04-23T18:16:35.439004+0000","last_peered":"2024-04-23T18:16:35.439004+0000","last_clean":"2024-04-23T18:16:35.439004+0000","last_became_active":"2024-04-23T18:16:34.001486+0000","last_became_peered":"2024-04-23T18:16:34.001486+0000","last_unstale":"2024-04-23T18:16:35.439004+0000","last_undegraded":"2024-04-23T18:16:35.439004+0000","last_fullsized":"2024-04-23T18:16:35.439004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:06:40.452366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.969781+0000","last_change":"2024-04-23T18:16:32.960387+0000","last_active":"2024-04-23T18:16:36.969781+0000","last_peered":"2024-04-23T18:16:36.969781+0000","last_clean":"2024-04-23T18:16:36.969781+0000","last_became_active":"2024-04-23T18:16:32.960046+0000","last_became_peered":"2024-04-23T18:16:32.960046+0000","last_unstale":"2024-04-23T18:16:36.969781+0000","last_undegraded":"2024-04-23T18:16:36.969781+0000","last_fullsized":"2024-04-23T18:16:36.969781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:03.296653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.799689+0000","last_change":"2024-04-23T18:16:33.996520+0000","last_active":"2024-04-23T18:16:35.799689+0000","last_peered":"2024-04-23T18:16:35.799689+0000","last_clean":"2024-04-23T18:16:35.799689+0000","last_became_active":"2024-04-23T18:16:33.995636+0000","last_became_peered":"2024-04-23T18:16:33.995636+0000","last_unstale":"2024-04-23T18:16:35.799689+0000","last_undegraded":"2024-04-23T18:16:35.799689+0000","last_fullsized":"2024-04-23T18:16:35.799689+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:29.665407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.989840+0000","last_change":"2024-04-23T18:16:32.956813+0000","last_active":"2024-04-23T18:16:34.989840+0000","last_peered":"2024-04-23T18:16:34.989840+0000","last_clean":"2024-04-23T18:16:34.989840+0000","last_became_active":"2024-04-23T18:16:32.956505+0000","last_became_peered":"2024-04-23T18:16:32.956505+0000","last_unstale":"2024-04-23T18:16:34.989840+0000","last_undegraded":"2024-04-23T18:16:34.989840+0000","last_fullsized":"2024-04-23T18:16:34.989840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:36:27.330353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.000115+0000","last_change":"2024-04-23T18:16:33.999754+0000","last_active":"2024-04-23T18:16:34.000115+0000","last_peered":"2024-04-23T18:16:34.000115+0000","last_clean":"2024-04-23T18:16:34.000115+0000","last_became_active":"2024-04-23T18:16:33.999034+0000","last_became_peered":"2024-04-23T18:16:33.999034+0000","last_unstale":"2024-04-23T18:16:34.000115+0000","last_undegraded":"2024-04-23T18:16:34.000115+0000","last_fullsized":"2024-04-23T18:16:34.000115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:20:04.819520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.001496+0000","last_change":"2024-04-23T18:16:32.961347+0000","last_active":"2024-04-23T18:16:34.001496+0000","last_peered":"2024-04-23T18:16:34.001496+0000","last_clean":"2024-04-23T18:16:34.001496+0000","last_became_active":"2024-04-23T18:16:32.961045+0000","last_became_peered":"2024-04-23T18:16:32.961045+0000","last_unstale":"2024-04-23T18:16:34.001496+0000","last_undegraded":"2024-04-23T18:16:34.001496+0000","last_fullsized":"2024-04-23T18:16:34.001496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:47:39.662705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.989516+0000","last_change":"2024-04-23T18:16:33.995166+0000","last_active":"2024-04-23T18:16:34.989516+0000","last_peered":"2024-04-23T18:16:34.989516+0000","last_clean":"2024-04-23T18:16:34.989516+0000","last_became_active":"2024-04-23T18:16:33.994347+0000","last_became_peered":"2024-04-23T18:16:33.994347+0000","last_unstale":"2024-04-23T18:16:34.989516+0000","last_undegraded":"2024-04-23T18:16:34.989516+0000","last_fullsized":"2024-04-23T18:16:34.989516+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.305837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961159+0000","last_change":"2024-04-23T18:16:32.959358+0000","last_active":"2024-04-23T18:16:35.961159+0000","last_peered":"2024-04-23T18:16:35.961159+0000","last_clean":"2024-04-23T18:16:35.961159+0000","last_became_active":"2024-04-23T18:16:32.959036+0000","last_became_peered":"2024-04-23T18:16:32.959036+0000","last_unstale":"2024-04-23T18:16:35.961159+0000","last_undegraded":"2024-04-23T18:16:35.961159+0000","last_fullsized":"2024-04-23T18:16:35.961159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:03:15.734754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.989070+0000","last_change":"2024-04-23T18:16:33.988578+0000","last_active":"2024-04-23T18:16:33.989070+0000","last_peered":"2024-04-23T18:16:33.989070+0000","last_clean":"2024-04-23T18:16:33.989070+0000","last_became_active":"2024-04-23T18:16:33.988248+0000","last_became_peered":"2024-04-23T18:16:33.988248+0000","last_unstale":"2024-04-23T18:16:33.989070+0000","last_undegraded":"2024-04-23T18:16:33.989070+0000","last_fullsized":"2024-04-23T18:16:33.989070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:50.122995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.969393+0000","last_change":"2024-04-23T18:16:32.952945+0000","last_active":"2024-04-23T18:16:36.969393+0000","last_peered":"2024-04-23T18:16:36.969393+0000","last_clean":"2024-04-23T18:16:36.969393+0000","last_became_active":"2024-04-23T18:16:32.952602+0000","last_became_peered":"2024-04-23T18:16:32.952602+0000","last_unstale":"2024-04-23T18:16:36.969393+0000","last_undegraded":"2024-04-23T18:16:36.969393+0000","last_fullsized":"2024-04-23T18:16:36.969393+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:20.037431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.999446+0000","last_change":"2024-04-23T18:16:33.996897+0000","last_active":"2024-04-23T18:16:33.999446+0000","last_peered":"2024-04-23T18:16:33.999446+0000","last_clean":"2024-04-23T18:16:33.999446+0000","last_became_active":"2024-04-23T18:16:33.996214+0000","last_became_peered":"2024-04-23T18:16:33.996214+0000","last_unstale":"2024-04-23T18:16:33.999446+0000","last_undegraded":"2024-04-23T18:16:33.999446+0000","last_fullsized":"2024-04-23T18:16:33.999446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:10:05.662144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.975990+0000","last_change":"2024-04-23T18:16:32.959637+0000","last_active":"2024-04-23T18:16:33.975990+0000","last_peered":"2024-04-23T18:16:33.975990+0000","last_clean":"2024-04-23T18:16:33.975990+0000","last_became_active":"2024-04-23T18:16:32.959332+0000","last_became_peered":"2024-04-23T18:16:32.959332+0000","last_unstale":"2024-04-23T18:16:33.975990+0000","last_undegraded":"2024-04-23T18:16:33.975990+0000","last_fullsized":"2024-04-23T18:16:33.975990+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:30:49.089101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.986793+0000","last_change":"2024-04-23T18:16:33.986288+0000","last_active":"2024-04-23T18:16:33.986793+0000","last_peered":"2024-04-23T18:16:33.986793+0000","last_clean":"2024-04-23T18:16:33.986793+0000","last_became_active":"2024-04-23T18:16:33.985866+0000","last_became_peered":"2024-04-23T18:16:33.985866+0000","last_unstale":"2024-04-23T18:16:33.986793+0000","last_undegraded":"2024-04-23T18:16:33.986793+0000","last_fullsized":"2024-04-23T18:16:33.986793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:22:10.864070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.965205+0000","last_change":"2024-04-23T18:16:32.958466+0000","last_active":"2024-04-23T18:16:35.965205+0000","last_peered":"2024-04-23T18:16:35.965205+0000","last_clean":"2024-04-23T18:16:35.965205+0000","last_became_active":"2024-04-23T18:16:32.958155+0000","last_became_peered":"2024-04-23T18:16:32.958155+0000","last_unstale":"2024-04-23T18:16:35.965205+0000","last_undegraded":"2024-04-23T18:16:35.965205+0000","last_fullsized":"2024-04-23T18:16:35.965205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:15:49.073934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.799958+0000","last_change":"2024-04-23T18:16:33.996156+0000","last_active":"2024-04-23T18:16:35.799958+0000","last_peered":"2024-04-23T18:16:35.799958+0000","last_clean":"2024-04-23T18:16:35.799958+0000","last_became_active":"2024-04-23T18:16:33.995711+0000","last_became_peered":"2024-04-23T18:16:33.995711+0000","last_unstale":"2024-04-23T18:16:35.799958+0000","last_undegraded":"2024-04-23T18:16:35.799958+0000","last_fullsized":"2024-04-23T18:16:35.799958+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:45.272598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.962041+0000","last_change":"2024-04-23T18:16:32.959395+0000","last_active":"2024-04-23T18:16:35.962041+0000","last_peered":"2024-04-23T18:16:35.962041+0000","last_clean":"2024-04-23T18:16:35.962041+0000","last_became_active":"2024-04-23T18:16:32.959096+0000","last_became_peered":"2024-04-23T18:16:32.959096+0000","last_unstale":"2024-04-23T18:16:35.962041+0000","last_undegraded":"2024-04-23T18:16:35.962041+0000","last_fullsized":"2024-04-23T18:16:35.962041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:24:46.910385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.438960+0000","last_change":"2024-04-23T18:16:33.995800+0000","last_active":"2024-04-23T18:16:35.438960+0000","last_peered":"2024-04-23T18:16:35.438960+0000","last_clean":"2024-04-23T18:16:35.438960+0000","last_became_active":"2024-04-23T18:16:33.995306+0000","last_became_peered":"2024-04-23T18:16:33.995306+0000","last_unstale":"2024-04-23T18:16:35.438960+0000","last_undegraded":"2024-04-23T18:16:35.438960+0000","last_fullsized":"2024-04-23T18:16:35.438960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:13:40.097771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.002649+0000","last_change":"2024-04-23T18:16:32.966725+0000","last_active":"2024-04-23T18:16:34.002649+0000","last_peered":"2024-04-23T18:16:34.002649+0000","last_clean":"2024-04-23T18:16:34.002649+0000","last_became_active":"2024-04-23T18:16:32.966304+0000","last_became_peered":"2024-04-23T18:16:32.966304+0000","last_unstale":"2024-04-23T18:16:34.002649+0000","last_undegraded":"2024-04-23T18:16:34.002649+0000","last_fullsized":"2024-04-23T18:16:34.002649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:45.191051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.799873+0000","last_change":"2024-04-23T18:16:33.994667+0000","last_active":"2024-04-23T18:16:35.799873+0000","last_peered":"2024-04-23T18:16:35.799873+0000","last_clean":"2024-04-23T18:16:35.799873+0000","last_became_active":"2024-04-23T18:16:33.994305+0000","last_became_peered":"2024-04-23T18:16:33.994305+0000","last_unstale":"2024-04-23T18:16:35.799873+0000","last_undegraded":"2024-04-23T18:16:35.799873+0000","last_fullsized":"2024-04-23T18:16:35.799873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:23.807075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'8","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.965773+0000","last_change":"2024-04-23T18:16:32.957946+0000","last_active":"2024-04-23T18:16:35.965773+0000","last_peered":"2024-04-23T18:16:35.965773+0000","last_clean":"2024-04-23T18:16:35.965773+0000","last_became_active":"2024-04-23T18:16:32.957478+0000","last_became_peered":"2024-04-23T18:16:32.957478+0000","last_unstale":"2024-04-23T18:16:35.965773+0000","last_undegraded":"2024-04-23T18:16:35.965773+0000","last_fullsized":"2024-04-23T18:16:35.965773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:16.850604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.989797+0000","last_change":"2024-04-23T18:16:33.989537+0000","last_active":"2024-04-23T18:16:34.989797+0000","last_peered":"2024-04-23T18:16:34.989797+0000","last_clean":"2024-04-23T18:16:34.989797+0000","last_became_active":"2024-04-23T18:16:33.989180+0000","last_became_peered":"2024-04-23T18:16:33.989180+0000","last_unstale":"2024-04-23T18:16:34.989797+0000","last_undegraded":"2024-04-23T18:16:34.989797+0000","last_fullsized":"2024-04-23T18:16:34.989797+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:53:18.700347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.969039+0000","last_change":"2024-04-23T18:16:32.967314+0000","last_active":"2024-04-23T18:16:36.969039+0000","last_peered":"2024-04-23T18:16:36.969039+0000","last_clean":"2024-04-23T18:16:36.969039+0000","last_became_active":"2024-04-23T18:16:32.966861+0000","last_became_peered":"2024-04-23T18:16:32.966861+0000","last_unstale":"2024-04-23T18:16:36.969039+0000","last_undegraded":"2024-04-23T18:16:36.969039+0000","last_fullsized":"2024-04-23T18:16:36.969039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:00:25.606402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.984593+0000","last_change":"2024-04-23T18:16:33.992593+0000","last_active":"2024-04-23T18:16:34.984593+0000","last_peered":"2024-04-23T18:16:34.984593+0000","last_clean":"2024-04-23T18:16:34.984593+0000","last_became_active":"2024-04-23T18:16:33.992228+0000","last_became_peered":"2024-04-23T18:16:33.992228+0000","last_unstale":"2024-04-23T18:16:34.984593+0000","last_undegraded":"2024-04-23T18:16:34.984593+0000","last_fullsized":"2024-04-23T18:16:34.984593+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:12:23.836829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.976815+0000","last_change":"2024-04-23T18:16:32.961740+0000","last_active":"2024-04-23T18:16:33.976815+0000","last_peered":"2024-04-23T18:16:33.976815+0000","last_clean":"2024-04-23T18:16:33.976815+0000","last_became_active":"2024-04-23T18:16:32.961331+0000","last_became_peered":"2024-04-23T18:16:32.961331+0000","last_unstale":"2024-04-23T18:16:33.976815+0000","last_undegraded":"2024-04-23T18:16:33.976815+0000","last_fullsized":"2024-04-23T18:16:33.976815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:59:17.665288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985806+0000","last_change":"2024-04-23T18:16:33.996572+0000","last_active":"2024-04-23T18:16:34.985806+0000","last_peered":"2024-04-23T18:16:34.985806+0000","last_clean":"2024-04-23T18:16:34.985806+0000","last_became_active":"2024-04-23T18:16:33.994839+0000","last_became_peered":"2024-04-23T18:16:33.994839+0000","last_unstale":"2024-04-23T18:16:34.985806+0000","last_undegraded":"2024-04-23T18:16:34.985806+0000","last_fullsized":"2024-04-23T18:16:34.985806+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:40:02.176521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.968358+0000","last_change":"2024-04-23T18:16:32.954935+0000","last_active":"2024-04-23T18:16:36.968358+0000","last_peered":"2024-04-23T18:16:36.968358+0000","last_clean":"2024-04-23T18:16:36.968358+0000","last_became_active":"2024-04-23T18:16:32.954595+0000","last_became_peered":"2024-04-23T18:16:32.954595+0000","last_unstale":"2024-04-23T18:16:36.968358+0000","last_undegraded":"2024-04-23T18:16:36.968358+0000","last_fullsized":"2024-04-23T18:16:36.968358+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:22.954101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985307+0000","last_change":"2024-04-23T18:16:34.000215+0000","last_active":"2024-04-23T18:16:34.985307+0000","last_peered":"2024-04-23T18:16:34.985307+0000","last_clean":"2024-04-23T18:16:34.985307+0000","last_became_active":"2024-04-23T18:16:33.999855+0000","last_became_peered":"2024-04-23T18:16:33.999855+0000","last_unstale":"2024-04-23T18:16:34.985307+0000","last_undegraded":"2024-04-23T18:16:34.985307+0000","last_fullsized":"2024-04-23T18:16:34.985307+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:14.686916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.971055+0000","last_change":"2024-04-23T18:16:32.966907+0000","last_active":"2024-04-23T18:16:36.971055+0000","last_peered":"2024-04-23T18:16:36.971055+0000","last_clean":"2024-04-23T18:16:36.971055+0000","last_became_active":"2024-04-23T18:16:32.966530+0000","last_became_peered":"2024-04-23T18:16:32.966530+0000","last_unstale":"2024-04-23T18:16:36.971055+0000","last_undegraded":"2024-04-23T18:16:36.971055+0000","last_fullsized":"2024-04-23T18:16:36.971055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:30.631851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.999398+0000","last_change":"2024-04-23T18:16:33.998817+0000","last_active":"2024-04-23T18:16:33.999398+0000","last_peered":"2024-04-23T18:16:33.999398+0000","last_clean":"2024-04-23T18:16:33.999398+0000","last_became_active":"2024-04-23T18:16:33.997982+0000","last_became_peered":"2024-04-23T18:16:33.997982+0000","last_unstale":"2024-04-23T18:16:33.999398+0000","last_undegraded":"2024-04-23T18:16:33.999398+0000","last_fullsized":"2024-04-23T18:16:33.999398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:10:32.635610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:37.964985+0000","last_change":"2024-04-23T18:16:32.959253+0000","last_active":"2024-04-23T18:16:37.964985+0000","last_peered":"2024-04-23T18:16:37.964985+0000","last_clean":"2024-04-23T18:16:37.964985+0000","last_became_active":"2024-04-23T18:16:32.958929+0000","last_became_peered":"2024-04-23T18:16:32.958929+0000","last_unstale":"2024-04-23T18:16:37.964985+0000","last_undegraded":"2024-04-23T18:16:37.964985+0000","last_fullsized":"2024-04-23T18:16:37.964985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:36.317516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966233+0000","last_change":"2024-04-23T18:16:33.979464+0000","last_active":"2024-04-23T18:16:35.966233+0000","last_peered":"2024-04-23T18:16:35.966233+0000","last_clean":"2024-04-23T18:16:35.966233+0000","last_became_active":"2024-04-23T18:16:33.979106+0000","last_became_peered":"2024-04-23T18:16:33.979106+0000","last_unstale":"2024-04-23T18:16:35.966233+0000","last_undegraded":"2024-04-23T18:16:35.966233+0000","last_fullsized":"2024-04-23T18:16:35.966233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:00.125514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.969236+0000","last_change":"2024-04-23T18:16:32.960812+0000","last_active":"2024-04-23T18:16:33.969236+0000","last_peered":"2024-04-23T18:16:33.969236+0000","last_clean":"2024-04-23T18:16:33.969236+0000","last_became_active":"2024-04-23T18:16:32.959807+0000","last_became_peered":"2024-04-23T18:16:32.959807+0000","last_unstale":"2024-04-23T18:16:33.969236+0000","last_undegraded":"2024-04-23T18:16:33.969236+0000","last_fullsized":"2024-04-23T18:16:33.969236+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:45.097443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961994+0000","last_change":"2024-04-23T18:16:33.992891+0000","last_active":"2024-04-23T18:16:35.961994+0000","last_peered":"2024-04-23T18:16:35.961994+0000","last_clean":"2024-04-23T18:16:35.961994+0000","last_became_active":"2024-04-23T18:16:33.992559+0000","last_became_peered":"2024-04-23T18:16:33.992559+0000","last_unstale":"2024-04-23T18:16:35.961994+0000","last_undegraded":"2024-04-23T18:16:35.961994+0000","last_fullsized":"2024-04-23T18:16:35.961994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:48.615158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985377+0000","last_change":"2024-04-23T18:16:32.957015+0000","last_active":"2024-04-23T18:16:34.985377+0000","last_peered":"2024-04-23T18:16:34.985377+0000","last_clean":"2024-04-23T18:16:34.985377+0000","last_became_active":"2024-04-23T18:16:32.956690+0000","last_became_peered":"2024-04-23T18:16:32.956690+0000","last_unstale":"2024-04-23T18:16:34.985377+0000","last_undegraded":"2024-04-23T18:16:34.985377+0000","last_fullsized":"2024-04-23T18:16:34.985377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:20:42.087521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961653+0000","last_change":"2024-04-23T18:16:33.998906+0000","last_active":"2024-04-23T18:16:35.961653+0000","last_peered":"2024-04-23T18:16:35.961653+0000","last_clean":"2024-04-23T18:16:35.961653+0000","last_became_active":"2024-04-23T18:16:33.997297+0000","last_became_peered":"2024-04-23T18:16:33.997297+0000","last_unstale":"2024-04-23T18:16:35.961653+0000","last_undegraded":"2024-04-23T18:16:35.961653+0000","last_fullsized":"2024-04-23T18:16:35.961653+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:06.668830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.987932+0000","last_change":"2024-04-23T18:16:33.987529+0000","last_active":"2024-04-23T18:16:33.987932+0000","last_peered":"2024-04-23T18:16:33.987932+0000","last_clean":"2024-04-23T18:16:33.987932+0000","last_became_active":"2024-04-23T18:16:33.987215+0000","last_became_peered":"2024-04-23T18:16:33.987215+0000","last_unstale":"2024-04-23T18:16:33.987932+0000","last_undegraded":"2024-04-23T18:16:33.987932+0000","last_fullsized":"2024-04-23T18:16:33.987932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:48.580489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:37.972347+0000","last_change":"2024-04-23T18:16:32.961423+0000","last_active":"2024-04-23T18:16:37.972347+0000","last_peered":"2024-04-23T18:16:37.972347+0000","last_clean":"2024-04-23T18:16:37.972347+0000","last_became_active":"2024-04-23T18:16:32.961116+0000","last_became_peered":"2024-04-23T18:16:32.961116+0000","last_unstale":"2024-04-23T18:16:37.972347+0000","last_undegraded":"2024-04-23T18:16:37.972347+0000","last_fullsized":"2024-04-23T18:16:37.972347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:28.525642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.993245+0000","last_change":"2024-04-23T18:16:33.990747+0000","last_active":"2024-04-23T18:16:33.993245+0000","last_peered":"2024-04-23T18:16:33.993245+0000","last_clean":"2024-04-23T18:16:33.993245+0000","last_became_active":"2024-04-23T18:16:33.989972+0000","last_became_peered":"2024-04-23T18:16:33.989972+0000","last_unstale":"2024-04-23T18:16:33.993245+0000","last_undegraded":"2024-04-23T18:16:33.993245+0000","last_fullsized":"2024-04-23T18:16:33.993245+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:15:03.047353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.965328+0000","last_change":"2024-04-23T18:16:32.952821+0000","last_active":"2024-04-23T18:16:35.965328+0000","last_peered":"2024-04-23T18:16:35.965328+0000","last_clean":"2024-04-23T18:16:35.965328+0000","last_became_active":"2024-04-23T18:16:32.952470+0000","last_became_peered":"2024-04-23T18:16:32.952470+0000","last_unstale":"2024-04-23T18:16:35.965328+0000","last_undegraded":"2024-04-23T18:16:35.965328+0000","last_fullsized":"2024-04-23T18:16:35.965328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:19.392670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":349,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985221+0000","last_change":"2024-04-23T18:15:25.867370+0000","last_active":"2024-04-23T18:16:34.985221+0000","last_peered":"2024-04-23T18:16:34.985221+0000","last_clean":"2024-04-23T18:16:34.985221+0000","last_became_active":"2024-04-23T18:15:25.866815+0000","last_became_peered":"2024-04-23T18:15:25.866815+0000","last_unstale":"2024-04-23T18:16:34.985221+0000","last_undegraded":"2024-04-23T18:16:34.985221+0000","last_fullsized":"2024-04-23T18:16:34.985221+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_clean_scrub_stamp":"2024-04-23T18:12:02.913350+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:42.482422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.984495+0000","last_change":"2024-04-23T18:16:33.992988+0000","last_active":"2024-04-23T18:16:34.984495+0000","last_peered":"2024-04-23T18:16:34.984495+0000","last_clean":"2024-04-23T18:16:34.984495+0000","last_became_active":"2024-04-23T18:16:33.992650+0000","last_became_peered":"2024-04-23T18:16:33.992650+0000","last_unstale":"2024-04-23T18:16:34.984495+0000","last_undegraded":"2024-04-23T18:16:34.984495+0000","last_fullsized":"2024-04-23T18:16:34.984495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:54.054708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.968944+0000","last_change":"2024-04-23T18:16:32.957318+0000","last_active":"2024-04-23T18:16:36.968944+0000","last_peered":"2024-04-23T18:16:36.968944+0000","last_clean":"2024-04-23T18:16:36.968944+0000","last_became_active":"2024-04-23T18:16:32.956973+0000","last_became_peered":"2024-04-23T18:16:32.956973+0000","last_unstale":"2024-04-23T18:16:36.968944+0000","last_undegraded":"2024-04-23T18:16:36.968944+0000","last_fullsized":"2024-04-23T18:16:36.968944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:35.684825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.958607+0000","last_change":"2024-04-23T18:16:32.966814+0000","last_active":"2024-04-23T18:16:35.958607+0000","last_peered":"2024-04-23T18:16:35.958607+0000","last_clean":"2024-04-23T18:16:35.958607+0000","last_became_active":"2024-04-23T18:16:32.966451+0000","last_became_peered":"2024-04-23T18:16:32.966451+0000","last_unstale":"2024-04-23T18:16:35.958607+0000","last_undegraded":"2024-04-23T18:16:35.958607+0000","last_fullsized":"2024-04-23T18:16:35.958607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:08:22.325576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.984938+0000","last_change":"2024-04-23T18:16:33.996345+0000","last_active":"2024-04-23T18:16:34.984938+0000","last_peered":"2024-04-23T18:16:34.984938+0000","last_clean":"2024-04-23T18:16:34.984938+0000","last_became_active":"2024-04-23T18:16:33.993802+0000","last_became_peered":"2024-04-23T18:16:33.993802+0000","last_unstale":"2024-04-23T18:16:34.984938+0000","last_undegraded":"2024-04-23T18:16:34.984938+0000","last_fullsized":"2024-04-23T18:16:34.984938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:32.410330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.969822+0000","last_change":"2024-04-23T18:16:32.957953+0000","last_active":"2024-04-23T18:16:33.969822+0000","last_peered":"2024-04-23T18:16:33.969822+0000","last_clean":"2024-04-23T18:16:33.969822+0000","last_became_active":"2024-04-23T18:16:32.957616+0000","last_became_peered":"2024-04-23T18:16:32.957616+0000","last_unstale":"2024-04-23T18:16:33.969822+0000","last_undegraded":"2024-04-23T18:16:33.969822+0000","last_fullsized":"2024-04-23T18:16:33.969822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:15.485373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.986335+0000","last_change":"2024-04-23T18:16:33.985685+0000","last_active":"2024-04-23T18:16:33.986335+0000","last_peered":"2024-04-23T18:16:33.986335+0000","last_clean":"2024-04-23T18:16:33.986335+0000","last_became_active":"2024-04-23T18:16:33.984631+0000","last_became_peered":"2024-04-23T18:16:33.984631+0000","last_unstale":"2024-04-23T18:16:33.986335+0000","last_undegraded":"2024-04-23T18:16:33.986335+0000","last_fullsized":"2024-04-23T18:16:33.986335+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:24.236851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.965256+0000","last_change":"2024-04-23T18:16:32.954281+0000","last_active":"2024-04-23T18:16:35.965256+0000","last_peered":"2024-04-23T18:16:35.965256+0000","last_clean":"2024-04-23T18:16:35.965256+0000","last_became_active":"2024-04-23T18:16:32.953988+0000","last_became_peered":"2024-04-23T18:16:32.953988+0000","last_unstale":"2024-04-23T18:16:35.965256+0000","last_undegraded":"2024-04-23T18:16:35.965256+0000","last_fullsized":"2024-04-23T18:16:35.965256+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:09.049861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.438914+0000","last_change":"2024-04-23T18:16:33.994860+0000","last_active":"2024-04-23T18:16:35.438914+0000","last_peered":"2024-04-23T18:16:35.438914+0000","last_clean":"2024-04-23T18:16:35.438914+0000","last_became_active":"2024-04-23T18:16:33.993438+0000","last_became_peered":"2024-04-23T18:16:33.993438+0000","last_unstale":"2024-04-23T18:16:35.438914+0000","last_undegraded":"2024-04-23T18:16:35.438914+0000","last_fullsized":"2024-04-23T18:16:35.438914+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:55.565873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961163+0000","last_change":"2024-04-23T18:16:32.959251+0000","last_active":"2024-04-23T18:16:35.961163+0000","last_peered":"2024-04-23T18:16:35.961163+0000","last_clean":"2024-04-23T18:16:35.961163+0000","last_became_active":"2024-04-23T18:16:32.958937+0000","last_became_peered":"2024-04-23T18:16:32.958937+0000","last_unstale":"2024-04-23T18:16:35.961163+0000","last_undegraded":"2024-04-23T18:16:35.961163+0000","last_fullsized":"2024-04-23T18:16:35.961163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.126634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.989446+0000","last_change":"2024-04-23T18:16:33.995480+0000","last_active":"2024-04-23T18:16:34.989446+0000","last_peered":"2024-04-23T18:16:34.989446+0000","last_clean":"2024-04-23T18:16:34.989446+0000","last_became_active":"2024-04-23T18:16:33.994361+0000","last_became_peered":"2024-04-23T18:16:33.994361+0000","last_unstale":"2024-04-23T18:16:34.989446+0000","last_undegraded":"2024-04-23T18:16:34.989446+0000","last_fullsized":"2024-04-23T18:16:34.989446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:59:35.240562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.979120+0000","last_change":"2024-04-23T18:16:32.967787+0000","last_active":"2024-04-23T18:16:33.979120+0000","last_peered":"2024-04-23T18:16:33.979120+0000","last_clean":"2024-04-23T18:16:33.979120+0000","last_became_active":"2024-04-23T18:16:32.967463+0000","last_became_peered":"2024-04-23T18:16:32.967463+0000","last_unstale":"2024-04-23T18:16:33.979120+0000","last_undegraded":"2024-04-23T18:16:33.979120+0000","last_fullsized":"2024-04-23T18:16:33.979120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:58.743682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966167+0000","last_change":"2024-04-23T18:16:33.980558+0000","last_active":"2024-04-23T18:16:35.966167+0000","last_peered":"2024-04-23T18:16:35.966167+0000","last_clean":"2024-04-23T18:16:35.966167+0000","last_became_active":"2024-04-23T18:16:33.980146+0000","last_became_peered":"2024-04-23T18:16:33.980146+0000","last_unstale":"2024-04-23T18:16:35.966167+0000","last_undegraded":"2024-04-23T18:16:35.966167+0000","last_fullsized":"2024-04-23T18:16:35.966167+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:58.001833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.958741+0000","last_change":"2024-04-23T18:16:32.962139+0000","last_active":"2024-04-23T18:16:35.958741+0000","last_peered":"2024-04-23T18:16:35.958741+0000","last_clean":"2024-04-23T18:16:35.958741+0000","last_became_active":"2024-04-23T18:16:32.961785+0000","last_became_peered":"2024-04-23T18:16:32.961785+0000","last_unstale":"2024-04-23T18:16:35.958741+0000","last_undegraded":"2024-04-23T18:16:35.958741+0000","last_fullsized":"2024-04-23T18:16:35.958741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:09.619414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.993716+0000","last_change":"2024-04-23T18:16:33.992902+0000","last_active":"2024-04-23T18:16:33.993716+0000","last_peered":"2024-04-23T18:16:33.993716+0000","last_clean":"2024-04-23T18:16:33.993716+0000","last_became_active":"2024-04-23T18:16:33.992574+0000","last_became_peered":"2024-04-23T18:16:33.992574+0000","last_unstale":"2024-04-23T18:16:33.993716+0000","last_undegraded":"2024-04-23T18:16:33.993716+0000","last_fullsized":"2024-04-23T18:16:33.993716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:52:02.827360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.962093+0000","last_change":"2024-04-23T18:16:32.956965+0000","last_active":"2024-04-23T18:16:35.962093+0000","last_peered":"2024-04-23T18:16:35.962093+0000","last_clean":"2024-04-23T18:16:35.962093+0000","last_became_active":"2024-04-23T18:16:32.956672+0000","last_became_peered":"2024-04-23T18:16:32.956672+0000","last_unstale":"2024-04-23T18:16:35.962093+0000","last_undegraded":"2024-04-23T18:16:35.962093+0000","last_fullsized":"2024-04-23T18:16:35.962093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:00:44.470809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985573+0000","last_change":"2024-04-23T18:16:34.002711+0000","last_active":"2024-04-23T18:16:34.985573+0000","last_peered":"2024-04-23T18:16:34.985573+0000","last_clean":"2024-04-23T18:16:34.985573+0000","last_became_active":"2024-04-23T18:16:34.002365+0000","last_became_peered":"2024-04-23T18:16:34.002365+0000","last_unstale":"2024-04-23T18:16:34.985573+0000","last_undegraded":"2024-04-23T18:16:34.985573+0000","last_fullsized":"2024-04-23T18:16:34.985573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:52:11.982735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:36.970897+0000","last_change":"2024-04-23T18:16:32.965121+0000","last_active":"2024-04-23T18:16:36.970897+0000","last_peered":"2024-04-23T18:16:36.970897+0000","last_clean":"2024-04-23T18:16:36.970897+0000","last_became_active":"2024-04-23T18:16:32.964732+0000","last_became_peered":"2024-04-23T18:16:32.964732+0000","last_unstale":"2024-04-23T18:16:36.970897+0000","last_undegraded":"2024-04-23T18:16:36.970897+0000","last_fullsized":"2024-04-23T18:16:36.970897+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:19:34.690245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985896+0000","last_change":"2024-04-23T18:16:33.996357+0000","last_active":"2024-04-23T18:16:34.985896+0000","last_peered":"2024-04-23T18:16:34.985896+0000","last_clean":"2024-04-23T18:16:34.985896+0000","last_became_active":"2024-04-23T18:16:33.996017+0000","last_became_peered":"2024-04-23T18:16:33.996017+0000","last_unstale":"2024-04-23T18:16:34.985896+0000","last_undegraded":"2024-04-23T18:16:34.985896+0000","last_fullsized":"2024-04-23T18:16:34.985896+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:20:57.321228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.004852+0000","last_change":"2024-04-23T18:16:32.961705+0000","last_active":"2024-04-23T18:16:34.004852+0000","last_peered":"2024-04-23T18:16:34.004852+0000","last_clean":"2024-04-23T18:16:34.004852+0000","last_became_active":"2024-04-23T18:16:32.961342+0000","last_became_peered":"2024-04-23T18:16:32.961342+0000","last_unstale":"2024-04-23T18:16:34.004852+0000","last_undegraded":"2024-04-23T18:16:34.004852+0000","last_fullsized":"2024-04-23T18:16:34.004852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:51.340910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985786+0000","last_change":"2024-04-23T18:16:33.992944+0000","last_active":"2024-04-23T18:16:34.985786+0000","last_peered":"2024-04-23T18:16:34.985786+0000","last_clean":"2024-04-23T18:16:34.985786+0000","last_became_active":"2024-04-23T18:16:33.992579+0000","last_became_peered":"2024-04-23T18:16:33.992579+0000","last_unstale":"2024-04-23T18:16:34.985786+0000","last_undegraded":"2024-04-23T18:16:34.985786+0000","last_fullsized":"2024-04-23T18:16:34.985786+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:13:37.829588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.984869+0000","last_change":"2024-04-23T18:16:32.955192+0000","last_active":"2024-04-23T18:16:34.984869+0000","last_peered":"2024-04-23T18:16:34.984869+0000","last_clean":"2024-04-23T18:16:34.984869+0000","last_became_active":"2024-04-23T18:16:32.954860+0000","last_became_peered":"2024-04-23T18:16:32.954860+0000","last_unstale":"2024-04-23T18:16:34.984869+0000","last_undegraded":"2024-04-23T18:16:34.984869+0000","last_fullsized":"2024-04-23T18:16:34.984869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:20.796060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985954+0000","last_change":"2024-04-23T18:16:33.986211+0000","last_active":"2024-04-23T18:16:34.985954+0000","last_peered":"2024-04-23T18:16:34.985954+0000","last_clean":"2024-04-23T18:16:34.985954+0000","last_became_active":"2024-04-23T18:16:33.985679+0000","last_became_peered":"2024-04-23T18:16:33.985679+0000","last_unstale":"2024-04-23T18:16:34.985954+0000","last_undegraded":"2024-04-23T18:16:34.985954+0000","last_fullsized":"2024-04-23T18:16:34.985954+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:03:38.797864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966363+0000","last_change":"2024-04-23T18:16:33.979462+0000","last_active":"2024-04-23T18:16:35.966363+0000","last_peered":"2024-04-23T18:16:35.966363+0000","last_clean":"2024-04-23T18:16:35.966363+0000","last_became_active":"2024-04-23T18:16:33.979114+0000","last_became_peered":"2024-04-23T18:16:33.979114+0000","last_unstale":"2024-04-23T18:16:35.966363+0000","last_undegraded":"2024-04-23T18:16:35.966363+0000","last_fullsized":"2024-04-23T18:16:35.966363+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:16:49.708240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.958490+0000","last_change":"2024-04-23T18:16:33.979255+0000","last_active":"2024-04-23T18:16:35.958490+0000","last_peered":"2024-04-23T18:16:35.958490+0000","last_clean":"2024-04-23T18:16:35.958490+0000","last_became_active":"2024-04-23T18:16:33.978934+0000","last_became_peered":"2024-04-23T18:16:33.978934+0000","last_unstale":"2024-04-23T18:16:35.958490+0000","last_undegraded":"2024-04-23T18:16:35.958490+0000","last_fullsized":"2024-04-23T18:16:35.958490+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:10.104564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.958676+0000","last_change":"2024-04-23T18:16:33.981877+0000","last_active":"2024-04-23T18:16:35.958676+0000","last_peered":"2024-04-23T18:16:35.958676+0000","last_clean":"2024-04-23T18:16:35.958676+0000","last_became_active":"2024-04-23T18:16:33.981536+0000","last_became_peered":"2024-04-23T18:16:33.981536+0000","last_unstale":"2024-04-23T18:16:35.958676+0000","last_undegraded":"2024-04-23T18:16:35.958676+0000","last_fullsized":"2024-04-23T18:16:35.958676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:00:15.056779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.438881+0000","last_change":"2024-04-23T18:16:33.999090+0000","last_active":"2024-04-23T18:16:35.438881+0000","last_peered":"2024-04-23T18:16:35.438881+0000","last_clean":"2024-04-23T18:16:35.438881+0000","last_became_active":"2024-04-23T18:16:33.998513+0000","last_became_peered":"2024-04-23T18:16:33.998513+0000","last_unstale":"2024-04-23T18:16:35.438881+0000","last_undegraded":"2024-04-23T18:16:35.438881+0000","last_fullsized":"2024-04-23T18:16:35.438881+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:14.115406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.985020+0000","last_change":"2024-04-23T18:16:33.993375+0000","last_active":"2024-04-23T18:16:34.985020+0000","last_peered":"2024-04-23T18:16:34.985020+0000","last_clean":"2024-04-23T18:16:34.985020+0000","last_became_active":"2024-04-23T18:16:33.991841+0000","last_became_peered":"2024-04-23T18:16:33.991841+0000","last_unstale":"2024-04-23T18:16:34.985020+0000","last_undegraded":"2024-04-23T18:16:34.985020+0000","last_fullsized":"2024-04-23T18:16:34.985020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:12.777495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966416+0000","last_change":"2024-04-23T18:16:33.983621+0000","last_active":"2024-04-23T18:16:35.966416+0000","last_peered":"2024-04-23T18:16:35.966416+0000","last_clean":"2024-04-23T18:16:35.966416+0000","last_became_active":"2024-04-23T18:16:33.983356+0000","last_became_peered":"2024-04-23T18:16:33.983356+0000","last_unstale":"2024-04-23T18:16:35.966416+0000","last_undegraded":"2024-04-23T18:16:35.966416+0000","last_fullsized":"2024-04-23T18:16:35.966416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:27:19.391281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.961380+0000","last_change":"2024-04-23T18:16:33.987952+0000","last_active":"2024-04-23T18:16:35.961380+0000","last_peered":"2024-04-23T18:16:35.961380+0000","last_clean":"2024-04-23T18:16:35.961380+0000","last_became_active":"2024-04-23T18:16:33.987334+0000","last_became_peered":"2024-04-23T18:16:33.987334+0000","last_unstale":"2024-04-23T18:16:35.961380+0000","last_undegraded":"2024-04-23T18:16:35.961380+0000","last_fullsized":"2024-04-23T18:16:35.961380+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:41.902779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.985378+0000","last_change":"2024-04-23T18:16:33.983914+0000","last_active":"2024-04-23T18:16:33.985378+0000","last_peered":"2024-04-23T18:16:33.985378+0000","last_clean":"2024-04-23T18:16:33.985378+0000","last_became_active":"2024-04-23T18:16:33.983444+0000","last_became_peered":"2024-04-23T18:16:33.983444+0000","last_unstale":"2024-04-23T18:16:33.985378+0000","last_undegraded":"2024-04-23T18:16:33.985378+0000","last_fullsized":"2024-04-23T18:16:33.985378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:41:29.865551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.993010+0000","last_change":"2024-04-23T18:16:33.992496+0000","last_active":"2024-04-23T18:16:33.993010+0000","last_peered":"2024-04-23T18:16:33.993010+0000","last_clean":"2024-04-23T18:16:33.993010+0000","last_became_active":"2024-04-23T18:16:33.992155+0000","last_became_peered":"2024-04-23T18:16:33.992155+0000","last_unstale":"2024-04-23T18:16:33.993010+0000","last_undegraded":"2024-04-23T18:16:33.993010+0000","last_fullsized":"2024-04-23T18:16:33.993010+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:33:28.503307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T18:16:33.985412+0000","last_change":"2024-04-23T18:16:33.984409+0000","last_active":"2024-04-23T18:16:33.985412+0000","last_peered":"2024-04-23T18:16:33.985412+0000","last_clean":"2024-04-23T18:16:33.985412+0000","last_became_active":"2024-04-23T18:16:33.984073+0000","last_became_peered":"2024-04-23T18:16:33.984073+0000","last_unstale":"2024-04-23T18:16:33.985412+0000","last_undegraded":"2024-04-23T18:16:33.985412+0000","last_fullsized":"2024-04-23T18:16:33.985412+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:49:05.598391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.799737+0000","last_change":"2024-04-23T18:16:33.986376+0000","last_active":"2024-04-23T18:16:35.799737+0000","last_peered":"2024-04-23T18:16:35.799737+0000","last_clean":"2024-04-23T18:16:35.799737+0000","last_became_active":"2024-04-23T18:16:33.986012+0000","last_became_peered":"2024-04-23T18:16:33.986012+0000","last_unstale":"2024-04-23T18:16:35.799737+0000","last_undegraded":"2024-04-23T18:16:35.799737+0000","last_fullsized":"2024-04-23T18:16:35.799737+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:26.940215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:34.984836+0000","last_change":"2024-04-23T18:16:33.999795+0000","last_active":"2024-04-23T18:16:34.984836+0000","last_peered":"2024-04-23T18:16:34.984836+0000","last_clean":"2024-04-23T18:16:34.984836+0000","last_became_active":"2024-04-23T18:16:33.997034+0000","last_became_peered":"2024-04-23T18:16:33.997034+0000","last_unstale":"2024-04-23T18:16:34.984836+0000","last_undegraded":"2024-04-23T18:16:34.984836+0000","last_fullsized":"2024-04-23T18:16:34.984836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:23.983846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T18:16:35.966543+0000","last_change":"2024-04-23T18:16:33.985495+0000","last_active":"2024-04-23T18:16:35.966543+0000","last_peered":"2024-04-23T18:16:35.966543+0000","last_clean":"2024-04-23T18:16:35.966543+0000","last_became_active":"2024-04-23T18:16:33.985178+0000","last_became_peered":"2024-04-23T18:16:33.985178+0000","last_unstale":"2024-04-23T18:16:35.966543+0000","last_undegraded":"2024-04-23T18:16:35.966543+0000","last_fullsized":"2024-04-23T18:16:35.966543+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:22:25.325756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3928,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":101,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318284,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":937984,"data_stored":450534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32712,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712440,"statfs":{"total":95995035648,"available":95961538560,"internally_reserved":0,"allocated":1568768,"data_stored":1043470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776151,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":929792,"data_stored":442546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037788,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":451272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32692,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712460,"statfs":{"total":95995035648,"available":95961559040,"internally_reserved":0,"allocated":1548288,"data_stored":1042694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":450741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":452146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40440,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704712,"statfs":{"total":95995035648,"available":95953625088,"internally_reserved":0,"allocated":966656,"data_stored":452012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":974848,"data_stored":453496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":933888,"data_stored":442068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":942080,"data_stored":450556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1531904,"data_stored":1032514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:16:41.143 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-23T18:16:41.144 INFO:tasks.cephadm:Setting up client nodes... 2024-04-23T18:16:41.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-23T18:16:41.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:41 smithi177 ceph-mon[32308]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:41.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:41 smithi177 ceph-mon[32308]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:41.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:41 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2933436550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:41.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:41 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1690086715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:41.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:41 smithi188 ceph-mon[32694]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:41.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:41 smithi188 ceph-mon[32694]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:16:41.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:41 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2933436550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:16:41.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:41 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1690086715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:41.425 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:42.175 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:42 smithi050 ceph-mon[29187]: pgmap v290: 97 pgs: 97 active+clean; 580 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 948 B/s rd, 4.5 KiB/s wr, 14 op/s 2024-04-23T18:16:42.175 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:42 smithi050 ceph-mon[29187]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:16:42.175 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:42 smithi050 ceph-mon[29187]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:16:42.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:42 smithi177 ceph-mon[32308]: pgmap v290: 97 pgs: 97 active+clean; 580 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 948 B/s rd, 4.5 KiB/s wr, 14 op/s 2024-04-23T18:16:42.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:42 smithi177 ceph-mon[32308]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:16:42.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:42 smithi177 ceph-mon[32308]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:16:42.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:42 smithi188 ceph-mon[32694]: pgmap v290: 97 pgs: 97 active+clean; 580 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 948 B/s rd, 4.5 KiB/s wr, 14 op/s 2024-04-23T18:16:42.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:42 smithi188 ceph-mon[32694]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:16:42.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:42 smithi188 ceph-mon[32694]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:16:43.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.037+0000 7f98fd1ec700 1 -- 172.21.15.50:0/2867928854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98f8103560 msgr2=0x7f98f8109e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.037+0000 7f98fd1ec700 1 --2- 172.21.15.50:0/2867928854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98f8103560 0x7f98f8109e10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f98f000b0d0 tx=0x7f98f000b3e0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.037+0000 7f98fd1ec700 1 -- 172.21.15.50:0/2867928854 shutdown_connections 2024-04-23T18:16:43.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.037+0000 7f98fd1ec700 1 --2- 172.21.15.50:0/2867928854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98f8103560 0x7f98f8109e10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.037+0000 7f98fd1ec700 1 --2- 172.21.15.50:0/2867928854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98f81019c0 0x7f98f8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.037+0000 7f98fd1ec700 1 --2- 172.21.15.50:0/2867928854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98f8102bc0 0x7f98f8103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.037+0000 7f98fd1ec700 1 -- 172.21.15.50:0/2867928854 >> 172.21.15.50:0/2867928854 conn(0x7f98f80fcf60 msgr2=0x7f98f80ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:43.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.037+0000 7f98fd1ec700 1 -- 172.21.15.50:0/2867928854 shutdown_connections 2024-04-23T18:16:43.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.037+0000 7f98fd1ec700 1 -- 172.21.15.50:0/2867928854 wait complete. 2024-04-23T18:16:43.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98fd1ec700 1 Processor -- start 2024-04-23T18:16:43.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98fd1ec700 1 -- start start 2024-04-23T18:16:43.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98fd1ec700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98f81019c0 0x7f98f8198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98fd1ec700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98f8102bc0 0x7f98f8199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98fd1ec700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98f8103560 0x7f98f81a0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98fd1ec700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f98f810d460 con 0x7f98f8102bc0 2024-04-23T18:16:43.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98fd1ec700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f98f810d2e0 con 0x7f98f81019c0 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98fd1ec700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f98f810d5e0 con 0x7f98f8103560 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98f659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98f8102bc0 0x7f98f8199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98f659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98f8102bc0 0x7f98f8199010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45370/0 (socket says 172.21.15.50:45370) 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98f659c700 1 -- 172.21.15.50:0/912729040 learned_addr learned my addr 172.21.15.50:0/912729040 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98f759e700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98f8103560 0x7f98f81a0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.038+0000 7f98f6d9d700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98f81019c0 0x7f98f8198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.039+0000 7f98f659c700 1 -- 172.21.15.50:0/912729040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98f8103560 msgr2=0x7f98f81a0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.039+0000 7f98f659c700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98f8103560 0x7f98f81a0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.039+0000 7f98f659c700 1 -- 172.21.15.50:0/912729040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98f81019c0 msgr2=0x7f98f8198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.039+0000 7f98f659c700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98f81019c0 0x7f98f8198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.039+0000 7f98f659c700 1 -- 172.21.15.50:0/912729040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f81a0730 con 0x7f98f8102bc0 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.039+0000 7f98f659c700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98f8102bc0 0x7f98f8199010 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00d980 tx=0x7f98ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:43.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.039+0000 7f98dffff700 1 -- 172.21.15.50:0/912729040 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec016070 con 0x7f98f8102bc0 2024-04-23T18:16:43.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.039+0000 7f98fd1ec700 1 -- 172.21.15.50:0/912729040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98f81a0a20 con 0x7f98f8102bc0 2024-04-23T18:16:43.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.039+0000 7f98dffff700 1 -- 172.21.15.50:0/912729040 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ec009d80 con 0x7f98f8102bc0 2024-04-23T18:16:43.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.039+0000 7f98dffff700 1 -- 172.21.15.50:0/912729040 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec00f3c0 con 0x7f98f8102bc0 2024-04-23T18:16:43.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.039+0000 7f98fd1ec700 1 -- 172.21.15.50:0/912729040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98f81a0fb0 con 0x7f98f8102bc0 2024-04-23T18:16:43.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.041+0000 7f98fd1ec700 1 -- 172.21.15.50:0/912729040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98f8050f80 con 0x7f98f8102bc0 2024-04-23T18:16:43.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.041+0000 7f98dffff700 1 -- 172.21.15.50:0/912729040 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f98ec019070 con 0x7f98f8102bc0 2024-04-23T18:16:43.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.042+0000 7f98dffff700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f98e00a3d80 0x7f98e00a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:43.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.042+0000 7f98f6d9d700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f98e00a3d80 0x7f98e00a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:43.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.042+0000 7f98dffff700 1 -- 172.21.15.50:0/912729040 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f98ec0d0210 con 0x7f98f8102bc0 2024-04-23T18:16:43.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.042+0000 7f98f6d9d700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f98e00a3d80 0x7f98e00a6240 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f98f8102a20 tx=0x7f98e800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:43.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.047+0000 7f98dffff700 1 -- 172.21.15.50:0/912729040 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98ec09cc40 con 0x7f98f8102bc0 2024-04-23T18:16:43.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.347+0000 7f98fd1ec700 1 -- 172.21.15.50:0/912729040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f98f81a1320 con 0x7f98f8102bc0 2024-04-23T18:16:43.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.349+0000 7f98dffff700 1 -- 172.21.15.50:0/912729040 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f98ec09c540 con 0x7f98f8102bc0 2024-04-23T18:16:43.343 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2024-04-23T18:16:43.343 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAL+ydmgbrIFBAAAsnpxNUMAgyHSfyv6Jnmlw== 2024-04-23T18:16:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.351+0000 7f98fd1ec700 1 -- 172.21.15.50:0/912729040 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f98e00a3d80 msgr2=0x7f98e00a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.351+0000 7f98fd1ec700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f98e00a3d80 0x7f98e00a6240 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f98f8102a20 tx=0x7f98e800b410 comp rx=0 tx=0).stop 2024-04-23T18:16:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.351+0000 7f98fd1ec700 1 -- 172.21.15.50:0/912729040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98f8102bc0 msgr2=0x7f98f8199010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.351+0000 7f98fd1ec700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98f8102bc0 0x7f98f8199010 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00d980 tx=0x7f98ec00dc90 comp rx=0 tx=0).stop 2024-04-23T18:16:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.351+0000 7f98fd1ec700 1 -- 172.21.15.50:0/912729040 shutdown_connections 2024-04-23T18:16:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.351+0000 7f98fd1ec700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98f8103560 0x7f98f81a0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.351+0000 7f98fd1ec700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f98e00a3d80 0x7f98e00a6240 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.351+0000 7f98fd1ec700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98f81019c0 0x7f98f8198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.351+0000 7f98fd1ec700 1 --2- 172.21.15.50:0/912729040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98f8102bc0 0x7f98f8199010 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:43.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.351+0000 7f98fd1ec700 1 -- 172.21.15.50:0/912729040 >> 172.21.15.50:0/912729040 conn(0x7f98f80fcf60 msgr2=0x7f98f80fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:43.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.351+0000 7f98fd1ec700 1 -- 172.21.15.50:0/912729040 shutdown_connections 2024-04-23T18:16:43.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:43.351+0000 7f98fd1ec700 1 -- 172.21.15.50:0/912729040 wait complete. 2024-04-23T18:16:43.946 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:16:43.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-23T18:16:43.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-23T18:16:43.986 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-23T18:16:43.986 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-23T18:16:43.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-23T18:16:44.274 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:44.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:44 smithi177 ceph-mon[32308]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 9.7 KiB/s wr, 22 op/s 2024-04-23T18:16:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:44 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/912729040' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T18:16:44.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:44 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/912729040' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T18:16:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:44 smithi188 ceph-mon[32694]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 9.7 KiB/s wr, 22 op/s 2024-04-23T18:16:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:44 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/912729040' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T18:16:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:44 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/912729040' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T18:16:44.396 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:44 smithi050 ceph-mon[29187]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 9.7 KiB/s wr, 22 op/s 2024-04-23T18:16:44.396 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:44 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/912729040' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T18:16:44.396 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:44 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/912729040' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T18:16:45.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.921+0000 7fcb004c4700 1 -- 172.21.15.50:0/2835147169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8101970 msgr2=0x7fcaf8101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:45.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.921+0000 7fcb004c4700 1 --2- 172.21.15.50:0/2835147169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8101970 0x7fcaf8101df0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcae8009990 tx=0x7fcae8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:45.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 -- 172.21.15.50:0/2835147169 shutdown_connections 2024-04-23T18:16:45.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 --2- 172.21.15.50:0/2835147169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaf8102330 0x7fcaf810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:45.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 --2- 172.21.15.50:0/2835147169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf8107920 0x7fcaf8107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:45.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 --2- 172.21.15.50:0/2835147169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8101970 0x7fcaf8101df0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:45.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 -- 172.21.15.50:0/2835147169 >> 172.21.15.50:0/2835147169 conn(0x7fcaf80788f0 msgr2=0x7fcaf8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:45.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 -- 172.21.15.50:0/2835147169 shutdown_connections 2024-04-23T18:16:45.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 -- 172.21.15.50:0/2835147169 wait complete. 2024-04-23T18:16:45.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 Processor -- start 2024-04-23T18:16:45.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 -- start start 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaf8101970 0x7fcaf8199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf8102330 0x7fcaf8199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107920 0x7fcaf819da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf8111b60 con 0x7fcaf8107920 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf81119e0 con 0x7fcaf8102330 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.922+0000 7fcb004c4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf8111ce0 con 0x7fcaf8101970 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcafea61700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107920 0x7fcaf819da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcafea61700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107920 0x7fcaf819da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45378/0 (socket says 172.21.15.50:45378) 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcafea61700 1 -- 172.21.15.50:0/3417442517 learned_addr learned my addr 172.21.15.50:0/3417442517 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcafe260700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaf8101970 0x7fcaf8199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcafda5f700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf8102330 0x7fcaf8199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcafea61700 1 -- 172.21.15.50:0/3417442517 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaf8101970 msgr2=0x7fcaf8199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcafea61700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaf8101970 0x7fcaf8199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcafea61700 1 -- 172.21.15.50:0/3417442517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf8102330 msgr2=0x7fcaf8199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcafea61700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf8102330 0x7fcaf8199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcafea61700 1 -- 172.21.15.50:0/3417442517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf819e160 con 0x7fcaf8107920 2024-04-23T18:16:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcafea61700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107920 0x7fcaf819da20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0007d70 tx=0x7fcaf000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcaef7fe700 1 -- 172.21.15.50:0/3417442517 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0016020 con 0x7fcaf8107920 2024-04-23T18:16:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcb004c4700 1 -- 172.21.15.50:0/3417442517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaf819e450 con 0x7fcaf8107920 2024-04-23T18:16:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcaef7fe700 1 -- 172.21.15.50:0/3417442517 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcaf000ad40 con 0x7fcaf8107920 2024-04-23T18:16:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcb004c4700 1 -- 172.21.15.50:0/3417442517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaf81a5cd0 con 0x7fcaf8107920 2024-04-23T18:16:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.923+0000 7fcaef7fe700 1 -- 172.21.15.50:0/3417442517 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf001b600 con 0x7fcaf8107920 2024-04-23T18:16:45.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.924+0000 7fcb004c4700 1 -- 172.21.15.50:0/3417442517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaf8069730 con 0x7fcaf8107920 2024-04-23T18:16:45.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.925+0000 7fcaef7fe700 1 -- 172.21.15.50:0/3417442517 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcaf0019020 con 0x7fcaf8107920 2024-04-23T18:16:45.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.926+0000 7fcaef7fe700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcae40a3cd0 0x7fcae40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:45.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.926+0000 7fcafe260700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcae40a3cd0 0x7fcae40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:45.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.926+0000 7fcaef7fe700 1 -- 172.21.15.50:0/3417442517 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fcaf00d08c0 con 0x7fcaf8107920 2024-04-23T18:16:45.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.927+0000 7fcafe260700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcae40a3cd0 0x7fcae40a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009a10 tx=0x7fcaf4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:45.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:45.930+0000 7fcaef7fe700 1 -- 172.21.15.50:0/3417442517 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcaf0020020 con 0x7fcaf8107920 2024-04-23T18:16:46.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.200+0000 7fcb004c4700 1 -- 172.21.15.50:0/3417442517 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fcaf802df80 con 0x7fcaf8107920 2024-04-23T18:16:46.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.205+0000 7fcaef7fe700 1 -- 172.21.15.50:0/3417442517 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264997 (secure 0 0 0) 0x7fcaf009d300 con 0x7fcaf8107920 2024-04-23T18:16:46.203 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:46.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.208+0000 7fcb004c4700 1 -- 172.21.15.50:0/3417442517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcae40a3cd0 msgr2=0x7fcae40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:46.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.208+0000 7fcb004c4700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcae40a3cd0 0x7fcae40a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009a10 tx=0x7fcaf4009190 comp rx=0 tx=0).stop 2024-04-23T18:16:46.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.208+0000 7fcb004c4700 1 -- 172.21.15.50:0/3417442517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107920 msgr2=0x7fcaf819da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:46.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.208+0000 7fcb004c4700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107920 0x7fcaf819da20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0007d70 tx=0x7fcaf000a340 comp rx=0 tx=0).stop 2024-04-23T18:16:46.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.208+0000 7fcb004c4700 1 -- 172.21.15.50:0/3417442517 shutdown_connections 2024-04-23T18:16:46.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.208+0000 7fcb004c4700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107920 0x7fcaf819da20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.208+0000 7fcb004c4700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcaf8101970 0x7fcaf8199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.208+0000 7fcb004c4700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcae40a3cd0 0x7fcae40a6190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.208+0000 7fcb004c4700 1 --2- 172.21.15.50:0/3417442517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf8102330 0x7fcaf8199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:46.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.208+0000 7fcb004c4700 1 -- 172.21.15.50:0/3417442517 >> 172.21.15.50:0/3417442517 conn(0x7fcaf80788f0 msgr2=0x7fcaf806b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:46.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.208+0000 7fcb004c4700 1 -- 172.21.15.50:0/3417442517 shutdown_connections 2024-04-23T18:16:46.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:46.208+0000 7fcb004c4700 1 -- 172.21.15.50:0/3417442517 wait complete. 2024-04-23T18:16:46.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:46 smithi177 ceph-mon[32308]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 8.3 KiB/s wr, 19 op/s 2024-04-23T18:16:46.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:46 smithi188 ceph-mon[32694]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 8.3 KiB/s wr, 19 op/s 2024-04-23T18:16:46.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:46 smithi050 ceph-mon[29187]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 8.3 KiB/s wr, 19 op/s 2024-04-23T18:16:46.885 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1004018784},{"type":"v1","addr":"172.21.15.50:6801","nonce":1004018784}]},"active_addr":"172.21.15.50:6801/1004018784","active_change":"2024-04-23T18:09:34.403699+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.50:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":444338023}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":1479623929}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":2484340243}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":3248139242}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":1490938794}]}]} 2024-04-23T18:16:46.891 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-23T18:16:46.891 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T18:16:46.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T18:16:47.143 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:47.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:47 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3417442517' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T18:16:47.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:47 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3417442517' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T18:16:47.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:47 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3417442517' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T18:16:48.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:48 smithi177 ceph-mon[32308]: pgmap v293: 97 pgs: 97 active+clean; 618 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-04-23T18:16:48.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:48 smithi050 ceph-mon[29187]: pgmap v293: 97 pgs: 97 active+clean; 618 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-04-23T18:16:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:48 smithi188 ceph-mon[32694]: pgmap v293: 97 pgs: 97 active+clean; 618 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-04-23T18:16:48.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.773+0000 7f506ecf7700 1 -- 172.21.15.50:0/2993553060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5068102ac0 msgr2=0x7f5068102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:48.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.773+0000 7f506ecf7700 1 --2- 172.21.15.50:0/2993553060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5068102ac0 0x7f5068102f40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5054009990 tx=0x7f5054009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:48.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.773+0000 7f506ecf7700 1 -- 172.21.15.50:0/2993553060 shutdown_connections 2024-04-23T18:16:48.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.773+0000 7f506ecf7700 1 --2- 172.21.15.50:0/2993553060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5068103480 0x7f506810f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:48.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.773+0000 7f506ecf7700 1 --2- 172.21.15.50:0/2993553060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5068108a70 0x7f5068108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:48.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.773+0000 7f506ecf7700 1 --2- 172.21.15.50:0/2993553060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5068102ac0 0x7f5068102f40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:48.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.773+0000 7f506ecf7700 1 -- 172.21.15.50:0/2993553060 >> 172.21.15.50:0/2993553060 conn(0x7f5068079bb0 msgr2=0x7f5068079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:48.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.773+0000 7f506ecf7700 1 -- 172.21.15.50:0/2993553060 shutdown_connections 2024-04-23T18:16:48.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.773+0000 7f506ecf7700 1 -- 172.21.15.50:0/2993553060 wait complete. 2024-04-23T18:16:48.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506ecf7700 1 Processor -- start 2024-04-23T18:16:48.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506ecf7700 1 -- start start 2024-04-23T18:16:48.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506ecf7700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5068102ac0 0x7f5068074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:48.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506ecf7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5068103480 0x7f5068075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:48.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506ecf7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5068108a70 0x7f50680757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:48.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506ecf7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5068112d00 con 0x7f5068108a70 2024-04-23T18:16:48.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506ecf7700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5068112b80 con 0x7f5068102ac0 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506ecf7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5068112e80 con 0x7f5068103480 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506d294700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5068108a70 0x7f50680757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506d294700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5068108a70 0x7f50680757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45396/0 (socket says 172.21.15.50:45396) 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506d294700 1 -- 172.21.15.50:0/4075960462 learned_addr learned my addr 172.21.15.50:0/4075960462 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506ca93700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5068102ac0 0x7f5068074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f5067fff700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5068103480 0x7f5068075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506d294700 1 -- 172.21.15.50:0/4075960462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5068103480 msgr2=0x7f5068075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506d294700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5068103480 0x7f5068075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506d294700 1 -- 172.21.15.50:0/4075960462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5068102ac0 msgr2=0x7f5068074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506d294700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5068102ac0 0x7f5068074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.774+0000 7f506d294700 1 -- 172.21.15.50:0/4075960462 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50681a77c0 con 0x7f5068108a70 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.775+0000 7f506d294700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5068108a70 0x7f50680757c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f506000b970 tx=0x7f506000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.775+0000 7f5065ffb700 1 -- 172.21.15.50:0/4075960462 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5060019070 con 0x7f5068108a70 2024-04-23T18:16:48.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.775+0000 7f5065ffb700 1 -- 172.21.15.50:0/4075960462 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5060014440 con 0x7f5068108a70 2024-04-23T18:16:48.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.775+0000 7f506ecf7700 1 -- 172.21.15.50:0/4075960462 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50681a7ab0 con 0x7f5068108a70 2024-04-23T18:16:48.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.775+0000 7f5065ffb700 1 -- 172.21.15.50:0/4075960462 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506000ce60 con 0x7f5068108a70 2024-04-23T18:16:48.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.775+0000 7f506ecf7700 1 -- 172.21.15.50:0/4075960462 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50681a7ff0 con 0x7f5068108a70 2024-04-23T18:16:48.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.776+0000 7f506ecf7700 1 -- 172.21.15.50:0/4075960462 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5068069730 con 0x7f5068108a70 2024-04-23T18:16:48.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.777+0000 7f5065ffb700 1 -- 172.21.15.50:0/4075960462 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f506001c070 con 0x7f5068108a70 2024-04-23T18:16:48.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.778+0000 7f5065ffb700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f50580a3cd0 0x7f50580a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:48.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.778+0000 7f506ca93700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f50580a3cd0 0x7f50580a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:48.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.778+0000 7f5065ffb700 1 -- 172.21.15.50:0/4075960462 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f50600cfbc0 con 0x7f5068108a70 2024-04-23T18:16:48.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.779+0000 7f506ca93700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f50580a3cd0 0x7f50580a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f505c006fd0 tx=0x7f505c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:48.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.782+0000 7f5065ffb700 1 -- 172.21.15.50:0/4075960462 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f506009c700 con 0x7f5068108a70 2024-04-23T18:16:48.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.992+0000 7f506ecf7700 1 -- 172.21.15.50:0/4075960462 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f506802df80 con 0x7f5068108a70 2024-04-23T18:16:48.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.993+0000 7f5065ffb700 1 -- 172.21.15.50:0/4075960462 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22181 (secure 0 0 0) 0x7f506009c000 con 0x7f5068108a70 2024-04-23T18:16:48.987 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:48.988 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":87,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:16:34.977786+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:16:48.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.996+0000 7f506ecf7700 1 -- 172.21.15.50:0/4075960462 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f50580a3cd0 msgr2=0x7f50580a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:48.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.996+0000 7f506ecf7700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f50580a3cd0 0x7f50580a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f505c006fd0 tx=0x7f505c008040 comp rx=0 tx=0).stop 2024-04-23T18:16:48.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.996+0000 7f506ecf7700 1 -- 172.21.15.50:0/4075960462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5068108a70 msgr2=0x7f50680757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:48.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.996+0000 7f506ecf7700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5068108a70 0x7f50680757c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f506000b970 tx=0x7f506000bd30 comp rx=0 tx=0).stop 2024-04-23T18:16:48.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.996+0000 7f506ecf7700 1 -- 172.21.15.50:0/4075960462 shutdown_connections 2024-04-23T18:16:48.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.996+0000 7f506ecf7700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5068108a70 0x7f50680757c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:48.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.996+0000 7f506ecf7700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f50580a3cd0 0x7f50580a6190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:48.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.996+0000 7f506ecf7700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5068102ac0 0x7f5068074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:48.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.996+0000 7f506ecf7700 1 --2- 172.21.15.50:0/4075960462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5068103480 0x7f5068075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:48.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.996+0000 7f506ecf7700 1 -- 172.21.15.50:0/4075960462 >> 172.21.15.50:0/4075960462 conn(0x7f5068079bb0 msgr2=0x7f50681019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:48.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.996+0000 7f506ecf7700 1 -- 172.21.15.50:0/4075960462 shutdown_connections 2024-04-23T18:16:48.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:48.996+0000 7f506ecf7700 1 -- 172.21.15.50:0/4075960462 wait complete. 2024-04-23T18:16:49.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:49 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4075960462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:49.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:49 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4075960462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:49.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:49 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4075960462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:49.626 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T18:16:49.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T18:16:49.875 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:50.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:50 smithi050 ceph-mon[29187]: pgmap v294: 97 pgs: 97 active+clean; 627 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.7 KiB/s wr, 16 op/s 2024-04-23T18:16:50.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:50 smithi177 ceph-mon[32308]: pgmap v294: 97 pgs: 97 active+clean; 627 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.7 KiB/s wr, 16 op/s 2024-04-23T18:16:50.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:50 smithi188 ceph-mon[32694]: pgmap v294: 97 pgs: 97 active+clean; 627 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.7 KiB/s wr, 16 op/s 2024-04-23T18:16:51.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.518+0000 7efc3d6c9700 1 -- 172.21.15.50:0/4014813089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc38107990 msgr2=0x7efc38107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:51.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.518+0000 7efc3d6c9700 1 --2- 172.21.15.50:0/4014813089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc38107990 0x7efc38107d70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efc28009990 tx=0x7efc28009ca0 comp rx=0 tx=0).stop 2024-04-23T18:16:51.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.518+0000 7efc3d6c9700 1 -- 172.21.15.50:0/4014813089 shutdown_connections 2024-04-23T18:16:51.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.518+0000 7efc3d6c9700 1 --2- 172.21.15.50:0/4014813089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc38102300 0x7efc3810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:51.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.518+0000 7efc3d6c9700 1 --2- 172.21.15.50:0/4014813089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc38107990 0x7efc38107d70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:51.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.518+0000 7efc3d6c9700 1 --2- 172.21.15.50:0/4014813089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc38101940 0x7efc38101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:51.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.518+0000 7efc3d6c9700 1 -- 172.21.15.50:0/4014813089 >> 172.21.15.50:0/4014813089 conn(0x7efc380788f0 msgr2=0x7efc38078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:51.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.518+0000 7efc3d6c9700 1 -- 172.21.15.50:0/4014813089 shutdown_connections 2024-04-23T18:16:51.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.518+0000 7efc3d6c9700 1 -- 172.21.15.50:0/4014813089 wait complete. 2024-04-23T18:16:51.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.518+0000 7efc3d6c9700 1 Processor -- start 2024-04-23T18:16:51.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc3d6c9700 1 -- start start 2024-04-23T18:16:51.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc3d6c9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc38101940 0x7efc38199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:51.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc3d6c9700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc38102300 0x7efc3819a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:51.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc3d6c9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc38107990 0x7efc381a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc3d6c9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efc38111ae0 con 0x7efc38101940 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc3d6c9700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efc38111960 con 0x7efc38102300 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc3d6c9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efc38111c60 con 0x7efc38107990 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc36ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc38101940 0x7efc38199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc36ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc38101940 0x7efc38199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45406/0 (socket says 172.21.15.50:45406) 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc36ffd700 1 -- 172.21.15.50:0/3531022169 learned_addr learned my addr 172.21.15.50:0/3531022169 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc367fc700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc38102300 0x7efc3819a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc377fe700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc38107990 0x7efc381a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc36ffd700 1 -- 172.21.15.50:0/3531022169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc38107990 msgr2=0x7efc381a4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc36ffd700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc38107990 0x7efc381a4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc36ffd700 1 -- 172.21.15.50:0/3531022169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc38102300 msgr2=0x7efc3819a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc36ffd700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc38102300 0x7efc3819a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.519+0000 7efc36ffd700 1 -- 172.21.15.50:0/3531022169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc381a54e0 con 0x7efc38101940 2024-04-23T18:16:51.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.520+0000 7efc377fe700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc38107990 0x7efc381a4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:16:51.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.520+0000 7efc36ffd700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc38101940 0x7efc38199f20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efc28000c00 tx=0x7efc280047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:51.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.520+0000 7efc2ffff700 1 -- 172.21.15.50:0/3531022169 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc28027070 con 0x7efc38101940 2024-04-23T18:16:51.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.520+0000 7efc3d6c9700 1 -- 172.21.15.50:0/3531022169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc381a56b0 con 0x7efc38101940 2024-04-23T18:16:51.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.520+0000 7efc2ffff700 1 -- 172.21.15.50:0/3531022169 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc28003f20 con 0x7efc38101940 2024-04-23T18:16:51.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.520+0000 7efc2ffff700 1 -- 172.21.15.50:0/3531022169 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc28022cf0 con 0x7efc38101940 2024-04-23T18:16:51.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.520+0000 7efc3d6c9700 1 -- 172.21.15.50:0/3531022169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc381a5b00 con 0x7efc38101940 2024-04-23T18:16:51.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.521+0000 7efc3d6c9700 1 -- 172.21.15.50:0/3531022169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc38109eb0 con 0x7efc38101940 2024-04-23T18:16:51.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.522+0000 7efc2ffff700 1 -- 172.21.15.50:0/3531022169 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7efc2802a070 con 0x7efc38101940 2024-04-23T18:16:51.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.523+0000 7efc2ffff700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7efc240a3cd0 0x7efc240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:51.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.523+0000 7efc367fc700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7efc240a3cd0 0x7efc240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:51.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.523+0000 7efc2ffff700 1 -- 172.21.15.50:0/3531022169 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7efc280d11c0 con 0x7efc38101940 2024-04-23T18:16:51.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.523+0000 7efc367fc700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7efc240a3cd0 0x7efc240a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efc380792d0 tx=0x7efc2000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:51.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.528+0000 7efc2ffff700 1 -- 172.21.15.50:0/3531022169 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc28013070 con 0x7efc38101940 2024-04-23T18:16:51.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.774+0000 7efc3d6c9700 1 -- 172.21.15.50:0/3531022169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc38109890 con 0x7efc38101940 2024-04-23T18:16:51.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.776+0000 7efc2ffff700 1 -- 172.21.15.50:0/3531022169 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22181 (secure 0 0 0) 0x7efc2809dc80 con 0x7efc38101940 2024-04-23T18:16:51.770 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:16:51.770 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":87,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:16:34.977786+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:16:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.779+0000 7efc3d6c9700 1 -- 172.21.15.50:0/3531022169 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7efc240a3cd0 msgr2=0x7efc240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.779+0000 7efc3d6c9700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7efc240a3cd0 0x7efc240a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efc380792d0 tx=0x7efc2000b410 comp rx=0 tx=0).stop 2024-04-23T18:16:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.779+0000 7efc3d6c9700 1 -- 172.21.15.50:0/3531022169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc38101940 msgr2=0x7efc38199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.779+0000 7efc3d6c9700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc38101940 0x7efc38199f20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efc28000c00 tx=0x7efc280047b0 comp rx=0 tx=0).stop 2024-04-23T18:16:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.779+0000 7efc3d6c9700 1 -- 172.21.15.50:0/3531022169 shutdown_connections 2024-04-23T18:16:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.779+0000 7efc3d6c9700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc38107990 0x7efc381a4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.779+0000 7efc3d6c9700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc38101940 0x7efc38199f20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.779+0000 7efc3d6c9700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7efc240a3cd0 0x7efc240a6190 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.779+0000 7efc3d6c9700 1 --2- 172.21.15.50:0/3531022169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc38102300 0x7efc3819a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.779+0000 7efc3d6c9700 1 -- 172.21.15.50:0/3531022169 >> 172.21.15.50:0/3531022169 conn(0x7efc380788f0 msgr2=0x7efc38105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.780+0000 7efc3d6c9700 1 -- 172.21.15.50:0/3531022169 shutdown_connections 2024-04-23T18:16:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:51.780+0000 7efc3d6c9700 1 -- 172.21.15.50:0/3531022169 wait complete. 2024-04-23T18:16:52.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:52 smithi177 ceph-mon[32308]: pgmap v295: 97 pgs: 97 active+clean; 627 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-04-23T18:16:52.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:52 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3531022169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:52.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:52 smithi188 ceph-mon[32694]: pgmap v295: 97 pgs: 97 active+clean; 627 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-04-23T18:16:52.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:52 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3531022169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:52.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:52 smithi050 ceph-mon[29187]: pgmap v295: 97 pgs: 97 active+clean; 627 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-04-23T18:16:52.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:52 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3531022169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:16:52.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-23T18:16:52.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-23T18:16:52.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-23T18:16:52.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-23T18:16:52.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-23T18:16:52.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-23T18:16:52.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-23T18:16:52.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-23T18:16:52.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-23T18:16:52.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-23T18:16:52.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-23T18:16:52.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-23T18:16:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:53.188 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:53.195 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:54.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:54 smithi177 ceph-mon[32308]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-04-23T18:16:54.327 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:54 smithi050 ceph-mon[29187]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-04-23T18:16:54.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:54 smithi188 ceph-mon[32694]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-04-23T18:16:54.598 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:54.836 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:54.837 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:54.837 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:55.368 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:55.368 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:55.369 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:16:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.134+0000 7fc14f8e0700 1 -- 172.21.15.50:0/2349725050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc148107c30 msgr2=0x7fc148108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.134+0000 7fc14f8e0700 1 --2- 172.21.15.50:0/2349725050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc148107c30 0x7fc148108010 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc1440099c0 tx=0x7fc144009cd0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.135+0000 7fc14f8e0700 1 -- 172.21.15.50:0/2349725050 shutdown_connections 2024-04-23T18:16:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.135+0000 7fc14f8e0700 1 --2- 172.21.15.50:0/2349725050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc148102ed0 0x7fc14810e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.135+0000 7fc14f8e0700 1 --2- 172.21.15.50:0/2349725050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc148107c30 0x7fc148108010 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.135+0000 7fc14f8e0700 1 --2- 172.21.15.50:0/2349725050 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc148102510 0x7fc148102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.135+0000 7fc14f8e0700 1 -- 172.21.15.50:0/2349725050 >> 172.21.15.50:0/2349725050 conn(0x7fc148076c60 msgr2=0x7fc148077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.135+0000 7fc14f8e0700 1 -- 172.21.15.50:0/2349725050 shutdown_connections 2024-04-23T18:16:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.135+0000 7fc14f8e0700 1 -- 172.21.15.50:0/2349725050 wait complete. 2024-04-23T18:16:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.135+0000 7fc14f8e0700 1 Processor -- start 2024-04-23T18:16:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.135+0000 7fc14f8e0700 1 -- start start 2024-04-23T18:16:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.135+0000 7fc14f8e0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc148102510 0x7fc14819b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.135+0000 7fc14f8e0700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc148102ed0 0x7fc14819b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14f8e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc148107c30 0x7fc14819fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14f8e0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc148111c70 con 0x7fc148107c30 2024-04-23T18:16:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14f8e0700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc148111af0 con 0x7fc148102ed0 2024-04-23T18:16:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14f8e0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc148111df0 con 0x7fc148102510 2024-04-23T18:16:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14de7d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc148107c30 0x7fc14819fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14de7d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc148107c30 0x7fc14819fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43614/0 (socket says 172.21.15.50:43614) 2024-04-23T18:16:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14de7d700 1 -- 172.21.15.50:0/1235152359 learned_addr learned my addr 172.21.15.50:0/1235152359 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:16:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14ce7b700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc148102ed0 0x7fc14819b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14d67c700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc148102510 0x7fc14819b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14de7d700 1 -- 172.21.15.50:0/1235152359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc148102510 msgr2=0x7fc14819b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14de7d700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc148102510 0x7fc14819b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14de7d700 1 -- 172.21.15.50:0/1235152359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc148102ed0 msgr2=0x7fc14819b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14de7d700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc148102ed0 0x7fc14819b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14de7d700 1 -- 172.21.15.50:0/1235152359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1481a0460 con 0x7fc148107c30 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14d67c700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc148102510 0x7fc14819b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14de7d700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc148107c30 0x7fc14819fd20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc140007ec0 tx=0x7fc14000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc13e7fc700 1 -- 172.21.15.50:0/1235152359 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14000dd70 con 0x7fc148107c30 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc13e7fc700 1 -- 172.21.15.50:0/1235152359 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc140018070 con 0x7fc148107c30 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc13e7fc700 1 -- 172.21.15.50:0/1235152359 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140012780 con 0x7fc148107c30 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14f8e0700 1 -- 172.21.15.50:0/1235152359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1481a0750 con 0x7fc148107c30 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.136+0000 7fc14f8e0700 1 -- 172.21.15.50:0/1235152359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1481a7fd0 con 0x7fc148107c30 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.137+0000 7fc14f8e0700 1 -- 172.21.15.50:0/1235152359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc148063110 con 0x7fc148107c30 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.138+0000 7fc13e7fc700 1 -- 172.21.15.50:0/1235152359 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc1400133f0 con 0x7fc148107c30 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.139+0000 7fc13e7fc700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc1340a3cd0 0x7fc1340a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.139+0000 7fc14d67c700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc1340a3cd0 0x7fc1340a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.139+0000 7fc13e7fc700 1 -- 172.21.15.50:0/1235152359 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fc1400d0840 con 0x7fc148107c30 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.139+0000 7fc13e7fc700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303] conn(0x7fc1340ad4b0 0x7fc1340af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.139+0000 7fc13e7fc700 1 -- 172.21.15.50:0/1235152359 --> [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc140024e50 con 0x7fc1340ad4b0 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.139+0000 7fc13e7fc700 1 -- 172.21.15.50:0/1235152359 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fc1400d0d20 con 0x7fc148107c30 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.139+0000 7fc14ce7b700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303] conn(0x7fc1340ad4b0 0x7fc1340af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.140+0000 7fc14d67c700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc1340a3cd0 0x7fc1340a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc144003800 tx=0x7fc144009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.140+0000 7fc14ce7b700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303] conn(0x7fc1340ad4b0 0x7fc1340af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.140+0000 7fc13e7fc700 1 -- 172.21.15.50:0/1235152359 <== osd.3 v2:172.21.15.50:6810/3121149303 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc140024e50 con 0x7fc1340ad4b0 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.157+0000 7fc14f8e0700 1 -- 172.21.15.50:0/1235152359 --> [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc148063510 con 0x7fc1340ad4b0 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.157+0000 7fc13e7fc700 1 -- 172.21.15.50:0/1235152359 <== osd.3 v2:172.21.15.50:6810/3121149303 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc148063510 con 0x7fc1340ad4b0 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.157+0000 7fc14f8e0700 1 -- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303] conn(0x7fc1340ad4b0 msgr2=0x7fc1340af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.157+0000 7fc14f8e0700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303] conn(0x7fc1340ad4b0 0x7fc1340af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.158+0000 7fc14f8e0700 1 -- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc1340a3cd0 msgr2=0x7fc1340a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.158+0000 7fc14f8e0700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc1340a3cd0 0x7fc1340a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc144003800 tx=0x7fc144009f10 comp rx=0 tx=0).stop 2024-04-23T18:16:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.158+0000 7fc14f8e0700 1 -- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc148107c30 msgr2=0x7fc14819fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:16:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.158+0000 7fc14f8e0700 1 --2- 172.21.15.50:0/1235152359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc148107c30 0x7fc14819fd20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc140007ec0 tx=0x7fc14000d3b0 comp rx=0 tx=0).stop 2024-04-23T18:16:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.158+0000 7fc14de7d700 1 -- 172.21.15.50:0/1235152359 reap_dead start 2024-04-23T18:16:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.158+0000 7fc14f8e0700 1 -- 172.21.15.50:0/1235152359 shutdown_connections 2024-04-23T18:16:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.158+0000 7fc14f8e0700 1 -- 172.21.15.50:0/1235152359 >> 172.21.15.50:0/1235152359 conn(0x7fc148076c60 msgr2=0x7fc14810ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:16:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.158+0000 7fc14f8e0700 1 -- 172.21.15.50:0/1235152359 shutdown_connections 2024-04-23T18:16:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:16:56.158+0000 7fc14f8e0700 1 -- 172.21.15.50:0/1235152359 wait complete. 2024-04-23T18:16:56.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:56 smithi177 ceph-mon[32308]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-23T18:16:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:56 smithi188 ceph-mon[32694]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-23T18:16:56.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:56 smithi050 ceph-mon[29187]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-23T18:16:58.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:16:58 smithi050 ceph-mon[29187]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-23T18:16:58.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:16:58 smithi177 ceph-mon[32308]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-23T18:16:58.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:16:58 smithi188 ceph-mon[32694]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-23T18:17:00.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:00 smithi177 ceph-mon[32308]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-23T18:17:00.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:00 smithi188 ceph-mon[32694]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-23T18:17:00.424 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:00 smithi050 ceph-mon[29187]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-23T18:17:01.111 INFO:teuthology.orchestra.run.smithi050.stdout:107374182457 2024-04-23T18:17:01.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-23T18:17:02.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:02 smithi050 ceph-mon[29187]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-23T18:17:02.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:02 smithi177 ceph-mon[32308]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-23T18:17:02.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:02 smithi188 ceph-mon[32694]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-23T18:17:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:02.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.878+0000 7faa98935700 1 -- 172.21.15.50:0/3739963045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa941026e0 msgr2=0x7faa9410e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.878+0000 7faa98935700 1 --2- 172.21.15.50:0/3739963045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa941026e0 0x7faa9410e770 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00b0a0 tx=0x7faa8c00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.879+0000 7faa98935700 1 -- 172.21.15.50:0/3739963045 shutdown_connections 2024-04-23T18:17:02.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.879+0000 7faa98935700 1 --2- 172.21.15.50:0/3739963045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa941026e0 0x7faa9410e770 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.879+0000 7faa98935700 1 --2- 172.21.15.50:0/3739963045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa94106990 0x7faa94106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.879+0000 7faa98935700 1 --2- 172.21.15.50:0/3739963045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa94101d20 0x7faa941021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.879+0000 7faa98935700 1 -- 172.21.15.50:0/3739963045 >> 172.21.15.50:0/3739963045 conn(0x7faa940788f0 msgr2=0x7faa94078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:02.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.879+0000 7faa98935700 1 -- 172.21.15.50:0/3739963045 shutdown_connections 2024-04-23T18:17:02.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.879+0000 7faa98935700 1 -- 172.21.15.50:0/3739963045 wait complete. 2024-04-23T18:17:02.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.879+0000 7faa98935700 1 Processor -- start 2024-04-23T18:17:02.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.879+0000 7faa98935700 1 -- start start 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa98935700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa94101d20 0x7faa94198fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa98935700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa941026e0 0x7faa94199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa98935700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa94106990 0x7faa9419d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa98935700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faa94111ae0 con 0x7faa941026e0 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa98935700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faa94111960 con 0x7faa94106990 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa98935700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faa94111c60 con 0x7faa94101d20 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa91d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa941026e0 0x7faa94199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa91d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa941026e0 0x7faa94199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44612/0 (socket says 172.21.15.50:44612) 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa91d9b700 1 -- 172.21.15.50:0/2894650311 learned_addr learned my addr 172.21.15.50:0/2894650311 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa9259c700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa94101d20 0x7faa94198fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa92d9d700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa94106990 0x7faa9419d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa91d9b700 1 -- 172.21.15.50:0/2894650311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa94101d20 msgr2=0x7faa94198fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa91d9b700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa94101d20 0x7faa94198fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa91d9b700 1 -- 172.21.15.50:0/2894650311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa94106990 msgr2=0x7faa9419d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa91d9b700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa94106990 0x7faa9419d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa91d9b700 1 -- 172.21.15.50:0/2894650311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9419df80 con 0x7faa941026e0 2024-04-23T18:17:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.880+0000 7faa91d9b700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa941026e0 0x7faa94199500 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7faa8800d980 tx=0x7faa8800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.881+0000 7faa837fe700 1 -- 172.21.15.50:0/2894650311 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa88016070 con 0x7faa941026e0 2024-04-23T18:17:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.881+0000 7faa98935700 1 -- 172.21.15.50:0/2894650311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa9419e270 con 0x7faa941026e0 2024-04-23T18:17:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.881+0000 7faa837fe700 1 -- 172.21.15.50:0/2894650311 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa88004830 con 0x7faa941026e0 2024-04-23T18:17:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.881+0000 7faa837fe700 1 -- 172.21.15.50:0/2894650311 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa880054c0 con 0x7faa941026e0 2024-04-23T18:17:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.881+0000 7faa98935700 1 -- 172.21.15.50:0/2894650311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa941a5af0 con 0x7faa941026e0 2024-04-23T18:17:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.881+0000 7faa98935700 1 -- 172.21.15.50:0/2894650311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faa940631d0 con 0x7faa941026e0 2024-04-23T18:17:02.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.883+0000 7faa837fe700 1 -- 172.21.15.50:0/2894650311 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faa88004220 con 0x7faa941026e0 2024-04-23T18:17:02.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.883+0000 7faa837fe700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faa7c0a3cd0 0x7faa7c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.883+0000 7faa9259c700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faa7c0a3cd0 0x7faa7c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.883+0000 7faa837fe700 1 -- 172.21.15.50:0/2894650311 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7faa880d0930 con 0x7faa941026e0 2024-04-23T18:17:02.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.884+0000 7faa837fe700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948] conn(0x7faa7c0ad4b0 0x7faa7c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.884+0000 7faa837fe700 1 -- 172.21.15.50:0/2894650311 --> [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa88015ec0 con 0x7faa7c0ad4b0 2024-04-23T18:17:02.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.884+0000 7faa837fe700 1 -- 172.21.15.50:0/2894650311 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7faa880d0e10 con 0x7faa941026e0 2024-04-23T18:17:02.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.884+0000 7faa9259c700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faa7c0a3cd0 0x7faa7c0a6190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faa8400abd0 tx=0x7faa8400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:02.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.884+0000 7faa92d9d700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948] conn(0x7faa7c0ad4b0 0x7faa7c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.884+0000 7faa92d9d700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948] conn(0x7faa7c0ad4b0 0x7faa7c0af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:02.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.885+0000 7faa837fe700 1 -- 172.21.15.50:0/2894650311 <== osd.0 v2:172.21.15.50:6802/4125602948 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faa88015ec0 con 0x7faa7c0ad4b0 2024-04-23T18:17:02.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.901+0000 7faa98935700 1 -- 172.21.15.50:0/2894650311 --> [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faa940635d0 con 0x7faa7c0ad4b0 2024-04-23T18:17:02.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.901+0000 7faa837fe700 1 -- 172.21.15.50:0/2894650311 <== osd.0 v2:172.21.15.50:6802/4125602948 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faa940635d0 con 0x7faa7c0ad4b0 2024-04-23T18:17:02.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.902+0000 7faa98935700 1 -- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948] conn(0x7faa7c0ad4b0 msgr2=0x7faa7c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.902+0000 7faa98935700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948] conn(0x7faa7c0ad4b0 0x7faa7c0af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.902+0000 7faa98935700 1 -- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faa7c0a3cd0 msgr2=0x7faa7c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.902+0000 7faa98935700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faa7c0a3cd0 0x7faa7c0a6190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faa8400abd0 tx=0x7faa8400a380 comp rx=0 tx=0).stop 2024-04-23T18:17:02.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.902+0000 7faa98935700 1 -- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa941026e0 msgr2=0x7faa94199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.902+0000 7faa98935700 1 --2- 172.21.15.50:0/2894650311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa941026e0 0x7faa94199500 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7faa8800d980 tx=0x7faa8800dc90 comp rx=0 tx=0).stop 2024-04-23T18:17:02.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.902+0000 7faa92d9d700 1 -- 172.21.15.50:0/2894650311 reap_dead start 2024-04-23T18:17:02.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.902+0000 7faa98935700 1 -- 172.21.15.50:0/2894650311 shutdown_connections 2024-04-23T18:17:02.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.902+0000 7faa98935700 1 -- 172.21.15.50:0/2894650311 >> 172.21.15.50:0/2894650311 conn(0x7faa940788f0 msgr2=0x7faa9406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:02.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.902+0000 7faa98935700 1 -- 172.21.15.50:0/2894650311 shutdown_connections 2024-04-23T18:17:02.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.902+0000 7faa98935700 1 -- 172.21.15.50:0/2894650311 wait complete. 2024-04-23T18:17:02.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.799+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/930000939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ae0103480 msgr2=0x7f3ae010f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.799+0000 7f3ae7bd7700 1 --2- 172.21.15.50:0/930000939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ae0103480 0x7f3ae010f980 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3ad800b0a0 tx=0x7f3ad800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.799+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/930000939 shutdown_connections 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.799+0000 7f3ae7bd7700 1 --2- 172.21.15.50:0/930000939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ae0103480 0x7f3ae010f980 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.799+0000 7f3ae7bd7700 1 --2- 172.21.15.50:0/930000939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ae0108ab0 0x7f3ae0108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.799+0000 7f3ae7bd7700 1 --2- 172.21.15.50:0/930000939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ae0102ac0 0x7f3ae0102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.799+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/930000939 >> 172.21.15.50:0/930000939 conn(0x7f3ae0079bf0 msgr2=0x7f3ae007a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.800+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/930000939 shutdown_connections 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.800+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/930000939 wait complete. 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.800+0000 7f3ae7bd7700 1 Processor -- start 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.800+0000 7f3ae7bd7700 1 -- start start 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae7bd7700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ae0102ac0 0x7f3ae0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae7bd7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ae0103480 0x7f3ae0075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae7bd7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ae0108ab0 0x7f3ae00757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae7bd7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae0112cb0 con 0x7f3ae0103480 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae7bd7700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae0112b30 con 0x7f3ae0102ac0 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae7bd7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae0112e30 con 0x7f3ae0108ab0 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae5172700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ae0103480 0x7f3ae0075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae5172700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ae0103480 0x7f3ae0075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44598/0 (socket says 172.21.15.50:44598) 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae5172700 1 -- 172.21.15.50:0/1906884902 learned_addr learned my addr 172.21.15.50:0/1906884902 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae5973700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ae0102ac0 0x7f3ae0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae6174700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ae0108ab0 0x7f3ae00757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae5172700 1 -- 172.21.15.50:0/1906884902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ae0108ab0 msgr2=0x7f3ae00757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae5172700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ae0108ab0 0x7f3ae00757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae5172700 1 -- 172.21.15.50:0/1906884902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ae0102ac0 msgr2=0x7f3ae0074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae5172700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ae0102ac0 0x7f3ae0074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae5172700 1 -- 172.21.15.50:0/1906884902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae01a7610 con 0x7f3ae0103480 2024-04-23T18:17:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae6174700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ae0108ab0 0x7f3ae00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae5172700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ae0103480 0x7f3ae0075280 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3ad400e9c0 tx=0x7f3ad400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ad2ffd700 1 -- 172.21.15.50:0/1906884902 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad400cbd0 con 0x7f3ae0103480 2024-04-23T18:17:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ad2ffd700 1 -- 172.21.15.50:0/1906884902 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ad400cd70 con 0x7f3ae0103480 2024-04-23T18:17:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/1906884902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae01a7900 con 0x7f3ae0103480 2024-04-23T18:17:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.801+0000 7f3ad2ffd700 1 -- 172.21.15.50:0/1906884902 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4005610 con 0x7f3ae0103480 2024-04-23T18:17:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.802+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/1906884902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ae01a7e40 con 0x7f3ae0103480 2024-04-23T18:17:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.802+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/1906884902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3ae0063110 con 0x7f3ae0103480 2024-04-23T18:17:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.804+0000 7f3ad2ffd700 1 -- 172.21.15.50:0/1906884902 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3ad4004750 con 0x7f3ae0103480 2024-04-23T18:17:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.804+0000 7f3ad2ffd700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3acc0a3d50 0x7f3acc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.804+0000 7f3ae5973700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3acc0a3d50 0x7f3acc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.804+0000 7f3ad2ffd700 1 -- 172.21.15.50:0/1906884902 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f3ad40d0c90 con 0x7f3ae0103480 2024-04-23T18:17:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.804+0000 7f3ad2ffd700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415] conn(0x7f3acc0ad530 0x7f3acc0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.804+0000 7f3ad2ffd700 1 -- 172.21.15.50:0/1906884902 --> [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3acc0b0000 con 0x7f3acc0ad530 2024-04-23T18:17:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.804+0000 7f3ad2ffd700 1 -- 172.21.15.50:0/1906884902 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f3ad40d1170 con 0x7f3ae0103480 2024-04-23T18:17:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.805+0000 7f3ae6174700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415] conn(0x7f3acc0ad530 0x7f3acc0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.805+0000 7f3ae5973700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3acc0a3d50 0x7f3acc0a6210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3adc007b60 tx=0x7f3adc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.805+0000 7f3ae6174700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415] conn(0x7f3acc0ad530 0x7f3acc0af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.806+0000 7f3ad2ffd700 1 -- 172.21.15.50:0/1906884902 <== osd.6 v2:172.21.15.50:6818/4290265415 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3acc0b0000 con 0x7f3acc0ad530 2024-04-23T18:17:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.822+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/1906884902 --> [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3ae0063510 con 0x7f3acc0ad530 2024-04-23T18:17:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.822+0000 7f3ad2ffd700 1 -- 172.21.15.50:0/1906884902 <== osd.6 v2:172.21.15.50:6818/4290265415 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3ae0063510 con 0x7f3acc0ad530 2024-04-23T18:17:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.823+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415] conn(0x7f3acc0ad530 msgr2=0x7f3acc0af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.823+0000 7f3ae7bd7700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415] conn(0x7f3acc0ad530 0x7f3acc0af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.823+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3acc0a3d50 msgr2=0x7f3acc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.823+0000 7f3ae7bd7700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3acc0a3d50 0x7f3acc0a6210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3adc007b60 tx=0x7f3adc0073d0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.823+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ae0103480 msgr2=0x7f3ae0075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.823+0000 7f3ae7bd7700 1 --2- 172.21.15.50:0/1906884902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ae0103480 0x7f3ae0075280 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3ad400e9c0 tx=0x7f3ad400ecd0 comp rx=0 tx=0).stop 2024-04-23T18:17:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.823+0000 7f3ae6174700 1 -- 172.21.15.50:0/1906884902 reap_dead start 2024-04-23T18:17:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.823+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/1906884902 shutdown_connections 2024-04-23T18:17:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.823+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/1906884902 >> 172.21.15.50:0/1906884902 conn(0x7f3ae0079bf0 msgr2=0x7f3ae0100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.823+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/1906884902 shutdown_connections 2024-04-23T18:17:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:02.823+0000 7f3ae7bd7700 1 -- 172.21.15.50:0/1906884902 wait complete. 2024-04-23T18:17:03.583 INFO:teuthology.orchestra.run.smithi050.stdout:197568495659 2024-04-23T18:17:03.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-23T18:17:04.106 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:04 smithi050 ceph-mon[29187]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-23T18:17:04.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:04 smithi177 ceph-mon[32308]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-23T18:17:04.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:04 smithi188 ceph-mon[32694]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-23T18:17:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.167+0000 7f455a497700 1 -- 172.21.15.50:0/1637984230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f455406b980 msgr2=0x7f455410f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.167+0000 7f455a497700 1 --2- 172.21.15.50:0/1637984230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f455406b980 0x7f455410f8f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f454c00b0a0 tx=0x7f454c00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.167+0000 7f455a497700 1 -- 172.21.15.50:0/1637984230 shutdown_connections 2024-04-23T18:17:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.167+0000 7f455a497700 1 --2- 172.21.15.50:0/1637984230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f455406b980 0x7f455410f8f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.167+0000 7f455a497700 1 --2- 172.21.15.50:0/1637984230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4554108a20 0x7f4554108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.167+0000 7f455a497700 1 --2- 172.21.15.50:0/1637984230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f455406afe0 0x7f455406b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.167+0000 7f455a497700 1 -- 172.21.15.50:0/1637984230 >> 172.21.15.50:0/1637984230 conn(0x7f45540788b0 msgr2=0x7f4554078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.167+0000 7f455a497700 1 -- 172.21.15.50:0/1637984230 shutdown_connections 2024-04-23T18:17:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.167+0000 7f455a497700 1 -- 172.21.15.50:0/1637984230 wait complete. 2024-04-23T18:17:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.168+0000 7f455a497700 1 Processor -- start 2024-04-23T18:17:05.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.168+0000 7f455a497700 1 -- start start 2024-04-23T18:17:05.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.168+0000 7f455a497700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f455406afe0 0x7f45541990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:05.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.168+0000 7f455a497700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f455406b980 0x7f4554199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:05.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.168+0000 7f455a497700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4554108a20 0x7f455419d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.168+0000 7f455a497700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4554112bc0 con 0x7f4554108a20 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.168+0000 7f455a497700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4554112a40 con 0x7f455406b980 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.168+0000 7f455a497700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4554112d40 con 0x7f455406afe0 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f4558a34700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4554108a20 0x7f455419d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f4558a34700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4554108a20 0x7f455419d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44636/0 (socket says 172.21.15.50:44636) 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f4558a34700 1 -- 172.21.15.50:0/3033167186 learned_addr learned my addr 172.21.15.50:0/3033167186 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f45537fe700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f455406b980 0x7f4554199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f4553fff700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f455406afe0 0x7f45541990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f4558a34700 1 -- 172.21.15.50:0/3033167186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f455406afe0 msgr2=0x7f45541990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f4558a34700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f455406afe0 0x7f45541990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f4558a34700 1 -- 172.21.15.50:0/3033167186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f455406b980 msgr2=0x7f4554199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f4558a34700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f455406b980 0x7f4554199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f4558a34700 1 -- 172.21.15.50:0/3033167186 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455419e0b0 con 0x7f4554108a20 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f4558a34700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4554108a20 0x7f455419d970 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f454c00b830 tx=0x7f454c004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f45517fa700 1 -- 172.21.15.50:0/3033167186 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c021070 con 0x7f4554108a20 2024-04-23T18:17:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f455a497700 1 -- 172.21.15.50:0/3033167186 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f455419e340 con 0x7f4554108a20 2024-04-23T18:17:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f45517fa700 1 -- 172.21.15.50:0/3033167186 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f454c007e30 con 0x7f4554108a20 2024-04-23T18:17:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f45517fa700 1 -- 172.21.15.50:0/3033167186 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c0047d0 con 0x7f4554108a20 2024-04-23T18:17:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.169+0000 7f455a497700 1 -- 172.21.15.50:0/3033167186 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45541a5c20 con 0x7f4554108a20 2024-04-23T18:17:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.170+0000 7f455a497700 1 -- 172.21.15.50:0/3033167186 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f45540631d0 con 0x7f4554108a20 2024-04-23T18:17:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.171+0000 7f45517fa700 1 -- 172.21.15.50:0/3033167186 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f454c007980 con 0x7f4554108a20 2024-04-23T18:17:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.172+0000 7f45517fa700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f453c0a3d50 0x7f453c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.172+0000 7f45517fa700 1 -- 172.21.15.50:0/3033167186 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f454c00f020 con 0x7f4554108a20 2024-04-23T18:17:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.172+0000 7f4553fff700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f453c0a3d50 0x7f453c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.172+0000 7f45517fa700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768] conn(0x7f453c0ad400 0x7f453c0af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.172+0000 7f45517fa700 1 -- 172.21.15.50:0/3033167186 --> [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f454c01de50 con 0x7f453c0ad400 2024-04-23T18:17:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.172+0000 7f45517fa700 1 -- 172.21.15.50:0/3033167186 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f454c0d1f90 con 0x7f4554108a20 2024-04-23T18:17:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.172+0000 7f4553fff700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f453c0a3d50 0x7f453c0a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4544005fd0 tx=0x7f4544005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:05.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.172+0000 7f45537fe700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768] conn(0x7f453c0ad400 0x7f453c0af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:05.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.173+0000 7f45537fe700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768] conn(0x7f453c0ad400 0x7f453c0af820 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:05.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.174+0000 7f45517fa700 1 -- 172.21.15.50:0/3033167186 <== osd.5 v2:172.21.15.188:6808/56520768 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f454c01de50 con 0x7f453c0ad400 2024-04-23T18:17:05.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.190+0000 7f455a497700 1 -- 172.21.15.50:0/3033167186 --> [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f45540635d0 con 0x7f453c0ad400 2024-04-23T18:17:05.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.190+0000 7f45517fa700 1 -- 172.21.15.50:0/3033167186 <== osd.5 v2:172.21.15.188:6808/56520768 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f45540635d0 con 0x7f453c0ad400 2024-04-23T18:17:05.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.191+0000 7f455a497700 1 -- 172.21.15.50:0/3033167186 >> [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768] conn(0x7f453c0ad400 msgr2=0x7f453c0af820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:05.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.191+0000 7f455a497700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768] conn(0x7f453c0ad400 0x7f453c0af820 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.191+0000 7f455a497700 1 -- 172.21.15.50:0/3033167186 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f453c0a3d50 msgr2=0x7f453c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:05.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.191+0000 7f455a497700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f453c0a3d50 0x7f453c0a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4544005fd0 tx=0x7f4544005d90 comp rx=0 tx=0).stop 2024-04-23T18:17:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.191+0000 7f455a497700 1 -- 172.21.15.50:0/3033167186 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4554108a20 msgr2=0x7f455419d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.191+0000 7f455a497700 1 --2- 172.21.15.50:0/3033167186 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4554108a20 0x7f455419d970 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f454c00b830 tx=0x7f454c004ce0 comp rx=0 tx=0).stop 2024-04-23T18:17:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.191+0000 7f4558a34700 1 -- 172.21.15.50:0/3033167186 reap_dead start 2024-04-23T18:17:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.191+0000 7f455a497700 1 -- 172.21.15.50:0/3033167186 shutdown_connections 2024-04-23T18:17:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.191+0000 7f455a497700 1 -- 172.21.15.50:0/3033167186 >> 172.21.15.50:0/3033167186 conn(0x7f45540788b0 msgr2=0x7f45540ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.191+0000 7f455a497700 1 -- 172.21.15.50:0/3033167186 shutdown_connections 2024-04-23T18:17:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:05.191+0000 7f455a497700 1 -- 172.21.15.50:0/3033167186 wait complete. 2024-04-23T18:17:06.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:06 smithi177 ceph-mon[32308]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-23T18:17:06.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:06 smithi188 ceph-mon[32694]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-23T18:17:06.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:06 smithi050 ceph-mon[29187]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-23T18:17:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.302+0000 7faacb176700 1 -- 172.21.15.50:0/1755336920 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faac4107960 msgr2=0x7faac4107d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.302+0000 7faacb176700 1 --2- 172.21.15.50:0/1755336920 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faac4107960 0x7faac4107d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faab8009990 tx=0x7faab8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.302+0000 7faacb176700 1 -- 172.21.15.50:0/1755336920 shutdown_connections 2024-04-23T18:17:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.302+0000 7faacb176700 1 --2- 172.21.15.50:0/1755336920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faac4102320 0x7faac410e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.302+0000 7faacb176700 1 --2- 172.21.15.50:0/1755336920 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faac4107960 0x7faac4107d40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.302+0000 7faacb176700 1 --2- 172.21.15.50:0/1755336920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac4101960 0x7faac4101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.302+0000 7faacb176700 1 -- 172.21.15.50:0/1755336920 >> 172.21.15.50:0/1755336920 conn(0x7faac40789e0 msgr2=0x7faac4078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.302+0000 7faacb176700 1 -- 172.21.15.50:0/1755336920 shutdown_connections 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.302+0000 7faacb176700 1 -- 172.21.15.50:0/1755336920 wait complete. 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.303+0000 7faacb176700 1 Processor -- start 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.303+0000 7faacb176700 1 -- start start 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.303+0000 7faacb176700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faac4101960 0x7faac4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.303+0000 7faacb176700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faac4102320 0x7faac41984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.303+0000 7faacb176700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac4107960 0x7faac419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.303+0000 7faacb176700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faac4111b40 con 0x7faac4102320 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.303+0000 7faacb176700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faac41119c0 con 0x7faac4107960 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.303+0000 7faacb176700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faac4111cc0 con 0x7faac4101960 2024-04-23T18:17:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.303+0000 7faac3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faac4102320 0x7faac41984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faac4102320 0x7faac41984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44656/0 (socket says 172.21.15.50:44656) 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac3fff700 1 -- 172.21.15.50:0/3668331597 learned_addr learned my addr 172.21.15.50:0/3668331597 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac8f12700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faac4101960 0x7faac4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac9713700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac4107960 0x7faac419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac3fff700 1 -- 172.21.15.50:0/3668331597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faac4101960 msgr2=0x7faac4197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac3fff700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faac4101960 0x7faac4197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac3fff700 1 -- 172.21.15.50:0/3668331597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac4107960 msgr2=0x7faac419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac3fff700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac4107960 0x7faac419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac3fff700 1 -- 172.21.15.50:0/3668331597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac419cf70 con 0x7faac4102320 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac8f12700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faac4101960 0x7faac4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac9713700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faac4107960 0x7faac419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac3fff700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faac4102320 0x7faac41984f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faab000c9b0 tx=0x7faab000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac1ffb700 1 -- 172.21.15.50:0/3668331597 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0007b00 con 0x7faac4102320 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac1ffb700 1 -- 172.21.15.50:0/3668331597 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faab0007ca0 con 0x7faac4102320 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faac1ffb700 1 -- 172.21.15.50:0/3668331597 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0018730 con 0x7faac4102320 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faacb176700 1 -- 172.21.15.50:0/3668331597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faac419d260 con 0x7faac4102320 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.304+0000 7faacb176700 1 -- 172.21.15.50:0/3668331597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faac41a4ae0 con 0x7faac4102320 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.305+0000 7faacb176700 1 -- 172.21.15.50:0/3668331597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faac40631d0 con 0x7faac4102320 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.306+0000 7faac1ffb700 1 -- 172.21.15.50:0/3668331597 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faab000f450 con 0x7faac4102320 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.307+0000 7faac1ffb700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faab40a3cd0 0x7faab40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.307+0000 7faac8f12700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faab40a3cd0 0x7faab40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.307+0000 7faac1ffb700 1 -- 172.21.15.50:0/3668331597 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7faab00d08f0 con 0x7faac4102320 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.307+0000 7faac1ffb700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369] conn(0x7faab40ad4b0 0x7faab40af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.307+0000 7faac1ffb700 1 -- 172.21.15.50:0/3668331597 --> [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faab0004500 con 0x7faab40ad4b0 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.307+0000 7faac1ffb700 1 -- 172.21.15.50:0/3668331597 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7faab00d0dd0 con 0x7faac4102320 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.307+0000 7faac8f12700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faab40a3cd0 0x7faab40a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faab8006010 tx=0x7faab800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.308+0000 7faac9713700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369] conn(0x7faab40ad4b0 0x7faab40af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.308+0000 7faac9713700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369] conn(0x7faab40ad4b0 0x7faab40af8d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.309+0000 7faac1ffb700 1 -- 172.21.15.50:0/3668331597 <== osd.2 v2:172.21.15.188:6800/2182491369 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faab0004500 con 0x7faab40ad4b0 2024-04-23T18:17:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.326+0000 7faacb176700 1 -- 172.21.15.50:0/3668331597 --> [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faac4063650 con 0x7faab40ad4b0 2024-04-23T18:17:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.327+0000 7faac1ffb700 1 -- 172.21.15.50:0/3668331597 <== osd.2 v2:172.21.15.188:6800/2182491369 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faac4063650 con 0x7faab40ad4b0 2024-04-23T18:17:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.328+0000 7faacb176700 1 -- 172.21.15.50:0/3668331597 >> [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369] conn(0x7faab40ad4b0 msgr2=0x7faab40af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.328+0000 7faacb176700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369] conn(0x7faab40ad4b0 0x7faab40af8d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.328+0000 7faacb176700 1 -- 172.21.15.50:0/3668331597 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faab40a3cd0 msgr2=0x7faab40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.328+0000 7faacb176700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faab40a3cd0 0x7faab40a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faab8006010 tx=0x7faab800b540 comp rx=0 tx=0).stop 2024-04-23T18:17:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.328+0000 7faacb176700 1 -- 172.21.15.50:0/3668331597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faac4102320 msgr2=0x7faac41984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.328+0000 7faacb176700 1 --2- 172.21.15.50:0/3668331597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faac4102320 0x7faac41984f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faab000c9b0 tx=0x7faab000cd70 comp rx=0 tx=0).stop 2024-04-23T18:17:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.328+0000 7faac9713700 1 -- 172.21.15.50:0/3668331597 reap_dead start 2024-04-23T18:17:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.328+0000 7faacb176700 1 -- 172.21.15.50:0/3668331597 shutdown_connections 2024-04-23T18:17:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.328+0000 7faacb176700 1 -- 172.21.15.50:0/3668331597 >> 172.21.15.50:0/3668331597 conn(0x7faac40789e0 msgr2=0x7faac41044d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.328+0000 7faacb176700 1 -- 172.21.15.50:0/3668331597 shutdown_connections 2024-04-23T18:17:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:07.328+0000 7faacb176700 1 -- 172.21.15.50:0/3668331597 wait complete. 2024-04-23T18:17:07.676 INFO:teuthology.orchestra.run.smithi050.stdout:163208757297 2024-04-23T18:17:07.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-23T18:17:08.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:08 smithi050 ceph-mon[29187]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:17:08.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:08 smithi188 ceph-mon[32694]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:17:08.535 INFO:teuthology.orchestra.run.smithi050.stdout:34359738441 2024-04-23T18:17:08.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-23T18:17:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:08 smithi177 ceph-mon[32308]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T18:17:10.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:10 smithi188 ceph-mon[32694]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:17:10.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:10 smithi050 ceph-mon[29187]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:17:10.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:10 smithi177 ceph-mon[32308]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:17:10.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.000+0000 7f5a08cce700 1 -- 172.21.15.50:0/145152517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a04101970 msgr2=0x7f5a04101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:10.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.000+0000 7f5a08cce700 1 --2- 172.21.15.50:0/145152517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a04101970 0x7f5a04101df0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009990 tx=0x7f59f8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.000+0000 7f5a08cce700 1 -- 172.21.15.50:0/145152517 shutdown_connections 2024-04-23T18:17:10.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.000+0000 7f5a08cce700 1 --2- 172.21.15.50:0/145152517 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a04102330 0x7f5a0410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.000+0000 7f5a08cce700 1 --2- 172.21.15.50:0/145152517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a04107920 0x7f5a04107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.000+0000 7f5a08cce700 1 --2- 172.21.15.50:0/145152517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a04101970 0x7f5a04101df0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.000+0000 7f5a08cce700 1 -- 172.21.15.50:0/145152517 >> 172.21.15.50:0/145152517 conn(0x7f5a040788f0 msgr2=0x7f5a04078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:10.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.000+0000 7f5a08cce700 1 -- 172.21.15.50:0/145152517 shutdown_connections 2024-04-23T18:17:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.000+0000 7f5a08cce700 1 -- 172.21.15.50:0/145152517 wait complete. 2024-04-23T18:17:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a08cce700 1 Processor -- start 2024-04-23T18:17:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a08cce700 1 -- start start 2024-04-23T18:17:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a08cce700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a04101970 0x7f5a041034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a08cce700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a04102330 0x7f5a04103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a08cce700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a04107920 0x7f5a04103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a08cce700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5a04111b00 con 0x7f5a04101970 2024-04-23T18:17:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a08cce700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5a04111980 con 0x7f5a04102330 2024-04-23T18:17:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a08cce700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5a04111c80 con 0x7f5a04107920 2024-04-23T18:17:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a0259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a04101970 0x7f5a041034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a0259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a04101970 0x7f5a041034f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44674/0 (socket says 172.21.15.50:44674) 2024-04-23T18:17:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a0259c700 1 -- 172.21.15.50:0/1186132214 learned_addr learned my addr 172.21.15.50:0/1186132214 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a02d9d700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a04107920 0x7f5a04103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.001+0000 7f5a01d9b700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a04102330 0x7f5a04103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.002+0000 7f5a0259c700 1 -- 172.21.15.50:0/1186132214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a04107920 msgr2=0x7f5a04103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.002+0000 7f5a0259c700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a04107920 0x7f5a04103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.002+0000 7f5a0259c700 1 -- 172.21.15.50:0/1186132214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a04102330 msgr2=0x7f5a04103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.002+0000 7f5a0259c700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a04102330 0x7f5a04103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.002+0000 7f5a0259c700 1 -- 172.21.15.50:0/1186132214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a041a7620 con 0x7f5a04101970 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.002+0000 7f5a0259c700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a04101970 0x7f5a041034f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f59f400d980 tx=0x7f59f400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.002+0000 7f59f37fe700 1 -- 172.21.15.50:0/1186132214 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f4016070 con 0x7f5a04101970 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.002+0000 7f59f37fe700 1 -- 172.21.15.50:0/1186132214 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59f4004830 con 0x7f5a04101970 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.002+0000 7f5a08cce700 1 -- 172.21.15.50:0/1186132214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a041a7910 con 0x7f5a04101970 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.002+0000 7f59f37fe700 1 -- 172.21.15.50:0/1186132214 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f4005540 con 0x7f5a04101970 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.002+0000 7f5a08cce700 1 -- 172.21.15.50:0/1186132214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a041a7e20 con 0x7f5a04101970 2024-04-23T18:17:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.003+0000 7f5a08cce700 1 -- 172.21.15.50:0/1186132214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5a040630d0 con 0x7f5a04101970 2024-04-23T18:17:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.004+0000 7f59f37fe700 1 -- 172.21.15.50:0/1186132214 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f59f4004220 con 0x7f5a04101970 2024-04-23T18:17:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.005+0000 7f59f37fe700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f59ec0a3cd0 0x7f59ec0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.005+0000 7f5a01d9b700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f59ec0a3cd0 0x7f59ec0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.005+0000 7f59f37fe700 1 -- 172.21.15.50:0/1186132214 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f59f40d08d0 con 0x7f5a04101970 2024-04-23T18:17:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.005+0000 7f59f37fe700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686] conn(0x7f59ec0ad4b0 0x7f59ec0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.005+0000 7f59f37fe700 1 -- 172.21.15.50:0/1186132214 --> [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59f4015ec0 con 0x7f59ec0ad4b0 2024-04-23T18:17:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.005+0000 7f59f37fe700 1 -- 172.21.15.50:0/1186132214 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f59f40d0c80 con 0x7f5a04101970 2024-04-23T18:17:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.005+0000 7f5a02d9d700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686] conn(0x7f59ec0ad4b0 0x7f59ec0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.005+0000 7f5a01d9b700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f59ec0a3cd0 0x7f59ec0a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f59f800b5c0 tx=0x7f59f8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.006+0000 7f5a02d9d700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686] conn(0x7f59ec0ad4b0 0x7f59ec0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.007+0000 7f59f37fe700 1 -- 172.21.15.50:0/1186132214 <== osd.4 v2:172.21.15.177:6808/1367742686 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f59f4015ec0 con 0x7f59ec0ad4b0 2024-04-23T18:17:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.026+0000 7f5a08cce700 1 -- 172.21.15.50:0/1186132214 --> [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5a04063550 con 0x7f59ec0ad4b0 2024-04-23T18:17:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.027+0000 7f59f37fe700 1 -- 172.21.15.50:0/1186132214 <== osd.4 v2:172.21.15.177:6808/1367742686 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5a04063550 con 0x7f59ec0ad4b0 2024-04-23T18:17:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.027+0000 7f5a08cce700 1 -- 172.21.15.50:0/1186132214 >> [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686] conn(0x7f59ec0ad4b0 msgr2=0x7f59ec0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.027+0000 7f5a08cce700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686] conn(0x7f59ec0ad4b0 0x7f59ec0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.027+0000 7f5a08cce700 1 -- 172.21.15.50:0/1186132214 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f59ec0a3cd0 msgr2=0x7f59ec0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.027+0000 7f5a08cce700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f59ec0a3cd0 0x7f59ec0a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f59f800b5c0 tx=0x7f59f8005960 comp rx=0 tx=0).stop 2024-04-23T18:17:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.027+0000 7f5a08cce700 1 -- 172.21.15.50:0/1186132214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a04101970 msgr2=0x7f5a041034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.027+0000 7f5a08cce700 1 --2- 172.21.15.50:0/1186132214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a04101970 0x7f5a041034f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f59f400d980 tx=0x7f59f400dc90 comp rx=0 tx=0).stop 2024-04-23T18:17:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.027+0000 7f5a02d9d700 1 -- 172.21.15.50:0/1186132214 reap_dead start 2024-04-23T18:17:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.027+0000 7f5a08cce700 1 -- 172.21.15.50:0/1186132214 shutdown_connections 2024-04-23T18:17:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.027+0000 7f5a08cce700 1 -- 172.21.15.50:0/1186132214 >> 172.21.15.50:0/1186132214 conn(0x7f5a040788f0 msgr2=0x7f5a0406b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.027+0000 7f5a08cce700 1 -- 172.21.15.50:0/1186132214 shutdown_connections 2024-04-23T18:17:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.027+0000 7f5a08cce700 1 -- 172.21.15.50:0/1186132214 wait complete. 2024-04-23T18:17:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.234+0000 7fddbf5e0700 1 -- 172.21.15.50:0/961420211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb806b0f0 msgr2=0x7fddb810f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.234+0000 7fddbf5e0700 1 --2- 172.21.15.50:0/961420211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb806b0f0 0x7fddb810f900 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fddb000b0a0 tx=0x7fddb000b3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.234+0000 7fddbf5e0700 1 -- 172.21.15.50:0/961420211 shutdown_connections 2024-04-23T18:17:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.234+0000 7fddbf5e0700 1 --2- 172.21.15.50:0/961420211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb806b0f0 0x7fddb810f900 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.234+0000 7fddbf5e0700 1 --2- 172.21.15.50:0/961420211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb8108a30 0x7fddb8108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.234+0000 7fddbf5e0700 1 --2- 172.21.15.50:0/961420211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddb806a750 0x7fddb806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.234+0000 7fddbf5e0700 1 -- 172.21.15.50:0/961420211 >> 172.21.15.50:0/961420211 conn(0x7fddb80788f0 msgr2=0x7fddb8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:11.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.234+0000 7fddbf5e0700 1 -- 172.21.15.50:0/961420211 shutdown_connections 2024-04-23T18:17:11.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.234+0000 7fddbf5e0700 1 -- 172.21.15.50:0/961420211 wait complete. 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbf5e0700 1 Processor -- start 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbf5e0700 1 -- start start 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbf5e0700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb806a750 0x7fddb81990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbf5e0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddb806b0f0 0x7fddb8199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbf5e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb8108a30 0x7fddb819d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbf5e0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fddb8112c20 con 0x7fddb8108a30 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbf5e0700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fddb8112aa0 con 0x7fddb806a750 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbf5e0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fddb8112da0 con 0x7fddb806b0f0 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbdb7d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb8108a30 0x7fddb819d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbdb7d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb8108a30 0x7fddb819d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44688/0 (socket says 172.21.15.50:44688) 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbdb7d700 1 -- 172.21.15.50:0/3787817339 learned_addr learned my addr 172.21.15.50:0/3787817339 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbd37c700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb806a750 0x7fddb81990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbcb7b700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddb806b0f0 0x7fddb8199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbdb7d700 1 -- 172.21.15.50:0/3787817339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddb806b0f0 msgr2=0x7fddb8199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbdb7d700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddb806b0f0 0x7fddb8199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbdb7d700 1 -- 172.21.15.50:0/3787817339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb806a750 msgr2=0x7fddb81990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbdb7d700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb806a750 0x7fddb81990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.235+0000 7fddbdb7d700 1 -- 172.21.15.50:0/3787817339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb819e100 con 0x7fddb8108a30 2024-04-23T18:17:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.236+0000 7fddbdb7d700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb8108a30 0x7fddb819d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fddb000b070 tx=0x7fddb0004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.236+0000 7fddae7fc700 1 -- 172.21.15.50:0/3787817339 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb0021070 con 0x7fddb8108a30 2024-04-23T18:17:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.236+0000 7fddbf5e0700 1 -- 172.21.15.50:0/3787817339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddb819e390 con 0x7fddb8108a30 2024-04-23T18:17:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.236+0000 7fddae7fc700 1 -- 172.21.15.50:0/3787817339 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddb0007e30 con 0x7fddb8108a30 2024-04-23T18:17:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.236+0000 7fddae7fc700 1 -- 172.21.15.50:0/3787817339 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb00046d0 con 0x7fddb8108a30 2024-04-23T18:17:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.236+0000 7fddbf5e0700 1 -- 172.21.15.50:0/3787817339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddb81a5c70 con 0x7fddb8108a30 2024-04-23T18:17:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.237+0000 7fddbf5e0700 1 -- 172.21.15.50:0/3787817339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fddb80631d0 con 0x7fddb8108a30 2024-04-23T18:17:11.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.238+0000 7fddae7fc700 1 -- 172.21.15.50:0/3787817339 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fddb0007980 con 0x7fddb8108a30 2024-04-23T18:17:11.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.239+0000 7fddae7fc700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fdda40a3cd0 0x7fdda40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.239+0000 7fddbd37c700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fdda40a3cd0 0x7fdda40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.239+0000 7fddae7fc700 1 -- 172.21.15.50:0/3787817339 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fddb000f020 con 0x7fddb8108a30 2024-04-23T18:17:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.239+0000 7fddae7fc700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853] conn(0x7fdda40ad4b0 0x7fdda40af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.239+0000 7fddae7fc700 1 -- 172.21.15.50:0/3787817339 --> [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdda40aff80 con 0x7fdda40ad4b0 2024-04-23T18:17:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.239+0000 7fddae7fc700 1 -- 172.21.15.50:0/3787817339 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fddb00d1e10 con 0x7fddb8108a30 2024-04-23T18:17:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.239+0000 7fddbd37c700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fdda40a3cd0 0x7fdda40a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009be0 tx=0x7fddb4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.239+0000 7fddbcb7b700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853] conn(0x7fdda40ad4b0 0x7fdda40af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.240+0000 7fddbcb7b700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853] conn(0x7fdda40ad4b0 0x7fdda40af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.241+0000 7fddae7fc700 1 -- 172.21.15.50:0/3787817339 <== osd.11 v2:172.21.15.188:6824/854484853 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdda40aff80 con 0x7fdda40ad4b0 2024-04-23T18:17:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.257+0000 7fddbf5e0700 1 -- 172.21.15.50:0/3787817339 --> [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fddb8063600 con 0x7fdda40ad4b0 2024-04-23T18:17:11.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.257+0000 7fddae7fc700 1 -- 172.21.15.50:0/3787817339 <== osd.11 v2:172.21.15.188:6824/854484853 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fddb8063600 con 0x7fdda40ad4b0 2024-04-23T18:17:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.258+0000 7fddbf5e0700 1 -- 172.21.15.50:0/3787817339 >> [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853] conn(0x7fdda40ad4b0 msgr2=0x7fdda40af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.258+0000 7fddbf5e0700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853] conn(0x7fdda40ad4b0 0x7fdda40af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.258+0000 7fddbf5e0700 1 -- 172.21.15.50:0/3787817339 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fdda40a3cd0 msgr2=0x7fdda40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.258+0000 7fddbf5e0700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fdda40a3cd0 0x7fdda40a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009be0 tx=0x7fddb4009260 comp rx=0 tx=0).stop 2024-04-23T18:17:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.258+0000 7fddbf5e0700 1 -- 172.21.15.50:0/3787817339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb8108a30 msgr2=0x7fddb819d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.258+0000 7fddbf5e0700 1 --2- 172.21.15.50:0/3787817339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb8108a30 0x7fddb819d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fddb000b070 tx=0x7fddb0004cf0 comp rx=0 tx=0).stop 2024-04-23T18:17:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.258+0000 7fddbdb7d700 1 -- 172.21.15.50:0/3787817339 reap_dead start 2024-04-23T18:17:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.258+0000 7fddbf5e0700 1 -- 172.21.15.50:0/3787817339 shutdown_connections 2024-04-23T18:17:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.258+0000 7fddbf5e0700 1 -- 172.21.15.50:0/3787817339 >> 172.21.15.50:0/3787817339 conn(0x7fddb80788f0 msgr2=0x7fddb80ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.258+0000 7fddbf5e0700 1 -- 172.21.15.50:0/3787817339 shutdown_connections 2024-04-23T18:17:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:11.258+0000 7fddbf5e0700 1 -- 172.21.15.50:0/3787817339 wait complete. 2024-04-23T18:17:12.096 INFO:teuthology.orchestra.run.smithi050.stdout:352187318291 2024-04-23T18:17:12.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-23T18:17:12.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:12 smithi050 ceph-mon[29187]: pgmap v305: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:17:12.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:12 smithi188 ceph-mon[32694]: pgmap v305: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:17:12.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:12 smithi177 ceph-mon[32308]: pgmap v305: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:17:12.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.356+0000 7f82a35c4700 1 -- 172.21.15.50:0/696078095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f829c106990 msgr2=0x7f829c106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.356+0000 7f82a35c4700 1 --2- 172.21.15.50:0/696078095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f829c106990 0x7f829c106d70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8298009990 tx=0x7f8298009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.356+0000 7f82a35c4700 1 -- 172.21.15.50:0/696078095 shutdown_connections 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.356+0000 7f82a35c4700 1 --2- 172.21.15.50:0/696078095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f829c1026e0 0x7f829c10e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.356+0000 7f82a35c4700 1 --2- 172.21.15.50:0/696078095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f829c106990 0x7f829c106d70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.356+0000 7f82a35c4700 1 --2- 172.21.15.50:0/696078095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c101d20 0x7f829c1021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.356+0000 7f82a35c4700 1 -- 172.21.15.50:0/696078095 >> 172.21.15.50:0/696078095 conn(0x7f829c0788f0 msgr2=0x7f829c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.356+0000 7f82a35c4700 1 -- 172.21.15.50:0/696078095 shutdown_connections 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.356+0000 7f82a35c4700 1 -- 172.21.15.50:0/696078095 wait complete. 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.357+0000 7f82a35c4700 1 Processor -- start 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.357+0000 7f82a35c4700 1 -- start start 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.357+0000 7f82a35c4700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c101d20 0x7f829c1035e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.357+0000 7f82a35c4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f829c1026e0 0x7f829c103b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.357+0000 7f82a35c4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f829c106990 0x7f829c104060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.357+0000 7f82a35c4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f829c111b20 con 0x7f829c106990 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.357+0000 7f82a35c4700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1119a0 con 0x7f829c101d20 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.357+0000 7f82a35c4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f829c111ca0 con 0x7f829c1026e0 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.357+0000 7f82a1b61700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f829c106990 0x7f829c104060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.357+0000 7f82a1b61700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f829c106990 0x7f829c104060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57548/0 (socket says 172.21.15.50:57548) 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.357+0000 7f82a1b61700 1 -- 172.21.15.50:0/3724100231 learned_addr learned my addr 172.21.15.50:0/3724100231 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.357+0000 7f82a0b5f700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f829c1026e0 0x7f829c103b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82a1360700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c101d20 0x7f829c1035e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82a1b61700 1 -- 172.21.15.50:0/3724100231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f829c1026e0 msgr2=0x7f829c103b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82a1b61700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f829c1026e0 0x7f829c103b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82a1b61700 1 -- 172.21.15.50:0/3724100231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c101d20 msgr2=0x7f829c1035e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82a1b61700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c101d20 0x7f829c1035e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82a1b61700 1 -- 172.21.15.50:0/3724100231 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829c1a7570 con 0x7f829c106990 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82a1360700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c101d20 0x7f829c1035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82a1b61700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f829c106990 0x7f829c104060 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8294007ee0 tx=0x7f829400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82927fc700 1 -- 172.21.15.50:0/3724100231 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829400dbd0 con 0x7f829c106990 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82927fc700 1 -- 172.21.15.50:0/3724100231 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f829400dd70 con 0x7f829c106990 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82a35c4700 1 -- 172.21.15.50:0/3724100231 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829c1a7860 con 0x7f829c106990 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82927fc700 1 -- 172.21.15.50:0/3724100231 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294005990 con 0x7f829c106990 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.358+0000 7f82a35c4700 1 -- 172.21.15.50:0/3724100231 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f829c1a7df0 con 0x7f829c106990 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.359+0000 7f82a35c4700 1 -- 172.21.15.50:0/3724100231 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f829c063150 con 0x7f829c106990 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.360+0000 7f82927fc700 1 -- 172.21.15.50:0/3724100231 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f829401a020 con 0x7f829c106990 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.361+0000 7f82927fc700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f82880a3d50 0x7f82880a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.361+0000 7f82a1360700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f82880a3d50 0x7f82880a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.361+0000 7f82927fc700 1 -- 172.21.15.50:0/3724100231 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f82940d0170 con 0x7f829c106990 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.361+0000 7f82927fc700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360] conn(0x7f82880ad530 0x7f82880af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.361+0000 7f82927fc700 1 -- 172.21.15.50:0/3724100231 --> [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f829401be50 con 0x7f82880ad530 2024-04-23T18:17:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.361+0000 7f82927fc700 1 -- 172.21.15.50:0/3724100231 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f82940d0650 con 0x7f829c106990 2024-04-23T18:17:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.362+0000 7f82a1360700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f82880a3d50 0x7f82880a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f829800b5c0 tx=0x7f8298005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.362+0000 7f82a0b5f700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360] conn(0x7f82880ad530 0x7f82880af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.362+0000 7f82a0b5f700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360] conn(0x7f82880ad530 0x7f82880af950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.363+0000 7f82927fc700 1 -- 172.21.15.50:0/3724100231 <== osd.8 v2:172.21.15.188:6816/4982360 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f829401be50 con 0x7f82880ad530 2024-04-23T18:17:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.379+0000 7f82a35c4700 1 -- 172.21.15.50:0/3724100231 --> [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f829c063550 con 0x7f82880ad530 2024-04-23T18:17:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.380+0000 7f82927fc700 1 -- 172.21.15.50:0/3724100231 <== osd.8 v2:172.21.15.188:6816/4982360 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f829c063550 con 0x7f82880ad530 2024-04-23T18:17:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.380+0000 7f82a35c4700 1 -- 172.21.15.50:0/3724100231 >> [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360] conn(0x7f82880ad530 msgr2=0x7f82880af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.380+0000 7f82a35c4700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360] conn(0x7f82880ad530 0x7f82880af950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.380+0000 7f82a35c4700 1 -- 172.21.15.50:0/3724100231 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f82880a3d50 msgr2=0x7f82880a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.381+0000 7f82a35c4700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f82880a3d50 0x7f82880a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f829800b5c0 tx=0x7f8298005fd0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.381+0000 7f82a35c4700 1 -- 172.21.15.50:0/3724100231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f829c106990 msgr2=0x7f829c104060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.381+0000 7f82a35c4700 1 --2- 172.21.15.50:0/3724100231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f829c106990 0x7f829c104060 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8294007ee0 tx=0x7f829400d3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.381+0000 7f82a1b61700 1 -- 172.21.15.50:0/3724100231 reap_dead start 2024-04-23T18:17:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.381+0000 7f82a35c4700 1 -- 172.21.15.50:0/3724100231 shutdown_connections 2024-04-23T18:17:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.381+0000 7f82a35c4700 1 -- 172.21.15.50:0/3724100231 >> 172.21.15.50:0/3724100231 conn(0x7f829c0788f0 msgr2=0x7f829c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.381+0000 7f82a35c4700 1 -- 172.21.15.50:0/3724100231 shutdown_connections 2024-04-23T18:17:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:12.381+0000 7f82a35c4700 1 -- 172.21.15.50:0/3724100231 wait complete. 2024-04-23T18:17:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.360+0000 7f6f05d0b700 1 -- 172.21.15.50:0/459350081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f0006b1f0 msgr2=0x7f6f0006b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.360+0000 7f6f05d0b700 1 --2- 172.21.15.50:0/459350081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f0006b1f0 0x7f6f0006b670 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009990 tx=0x7f6ef4009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.360+0000 7f6f05d0b700 1 -- 172.21.15.50:0/459350081 shutdown_connections 2024-04-23T18:17:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.360+0000 7f6f05d0b700 1 --2- 172.21.15.50:0/459350081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f0006bbb0 0x7f6f0010f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.360+0000 7f6f05d0b700 1 --2- 172.21.15.50:0/459350081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f00108a60 0x7f6f00108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.360+0000 7f6f05d0b700 1 --2- 172.21.15.50:0/459350081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f0006b1f0 0x7f6f0006b670 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.360+0000 7f6f05d0b700 1 -- 172.21.15.50:0/459350081 >> 172.21.15.50:0/459350081 conn(0x7f6f000789e0 msgr2=0x7f6f00078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.360+0000 7f6f05d0b700 1 -- 172.21.15.50:0/459350081 shutdown_connections 2024-04-23T18:17:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.360+0000 7f6f05d0b700 1 -- 172.21.15.50:0/459350081 wait complete. 2024-04-23T18:17:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.361+0000 7f6f05d0b700 1 Processor -- start 2024-04-23T18:17:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.361+0000 7f6f05d0b700 1 -- start start 2024-04-23T18:17:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.361+0000 7f6f05d0b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f0006b1f0 0x7f6f0019b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.361+0000 7f6f05d0b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f0006bbb0 0x7f6f0019b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.361+0000 7f6f05d0b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f00108a60 0x7f6f0019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.361+0000 7f6f05d0b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6f00112c60 con 0x7f6f0006bbb0 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.361+0000 7f6f05d0b700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6f00112ae0 con 0x7f6f00108a60 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.361+0000 7f6f05d0b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6f00112de0 con 0x7f6f0006b1f0 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.361+0000 7f6efeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f0006bbb0 0x7f6f0019b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6efeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f0006bbb0 0x7f6f0019b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57572/0 (socket says 172.21.15.50:57572) 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6efeffd700 1 -- 172.21.15.50:0/3002754207 learned_addr learned my addr 172.21.15.50:0/3002754207 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6eff7fe700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f0006b1f0 0x7f6f0019b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6efffff700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f00108a60 0x7f6f0019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6efeffd700 1 -- 172.21.15.50:0/3002754207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f0006b1f0 msgr2=0x7f6f0019b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6efeffd700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f0006b1f0 0x7f6f0019b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6efeffd700 1 -- 172.21.15.50:0/3002754207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f00108a60 msgr2=0x7f6f0019fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6efeffd700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f00108a60 0x7f6f0019fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6efeffd700 1 -- 172.21.15.50:0/3002754207 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f001a02a0 con 0x7f6f0006bbb0 2024-04-23T18:17:13.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6efeffd700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f0006bbb0 0x7f6f0019b7d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4000c00 tx=0x7f6ef40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:13.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6efcff9700 1 -- 172.21.15.50:0/3002754207 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4027070 con 0x7f6f0006bbb0 2024-04-23T18:17:13.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6efcff9700 1 -- 172.21.15.50:0/3002754207 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ef4003f20 con 0x7f6f0006bbb0 2024-04-23T18:17:13.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6f05d0b700 1 -- 172.21.15.50:0/3002754207 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f001a0530 con 0x7f6f0006bbb0 2024-04-23T18:17:13.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6efcff9700 1 -- 172.21.15.50:0/3002754207 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4022cf0 con 0x7f6f0006bbb0 2024-04-23T18:17:13.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.362+0000 7f6f05d0b700 1 -- 172.21.15.50:0/3002754207 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f001a7e10 con 0x7f6f0006bbb0 2024-04-23T18:17:13.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.363+0000 7f6f05d0b700 1 -- 172.21.15.50:0/3002754207 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6f000631d0 con 0x7f6f0006bbb0 2024-04-23T18:17:13.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.364+0000 7f6efcff9700 1 -- 172.21.15.50:0/3002754207 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6ef402a070 con 0x7f6f0006bbb0 2024-04-23T18:17:13.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.365+0000 7f6efcff9700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ee80a3d50 0x7f6ee80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.365+0000 7f6eff7fe700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ee80a3d50 0x7f6ee80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.365+0000 7f6efcff9700 1 -- 172.21.15.50:0/3002754207 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f6ef40d1750 con 0x7f6f0006bbb0 2024-04-23T18:17:13.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.365+0000 7f6efcff9700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791] conn(0x7f6ee80ad530 0x7f6ee80af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:13.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.365+0000 7f6efcff9700 1 -- 172.21.15.50:0/3002754207 --> [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ef40095d0 con 0x7f6ee80ad530 2024-04-23T18:17:13.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.365+0000 7f6efcff9700 1 -- 172.21.15.50:0/3002754207 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f6ef40d1c30 con 0x7f6f0006bbb0 2024-04-23T18:17:13.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.365+0000 7f6efffff700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791] conn(0x7f6ee80ad530 0x7f6ee80af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:13.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.366+0000 7f6eff7fe700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ee80a3d50 0x7f6ee80a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0007fd0 tx=0x7f6ef00079e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:13.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.366+0000 7f6efffff700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791] conn(0x7f6ee80ad530 0x7f6ee80af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:13.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.367+0000 7f6efcff9700 1 -- 172.21.15.50:0/3002754207 <== osd.9 v2:172.21.15.50:6826/1497125791 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ef40095d0 con 0x7f6ee80ad530 2024-04-23T18:17:13.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.383+0000 7f6f05d0b700 1 -- 172.21.15.50:0/3002754207 --> [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6f000635d0 con 0x7f6ee80ad530 2024-04-23T18:17:13.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.383+0000 7f6efcff9700 1 -- 172.21.15.50:0/3002754207 <== osd.9 v2:172.21.15.50:6826/1497125791 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6f000635d0 con 0x7f6ee80ad530 2024-04-23T18:17:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.384+0000 7f6f05d0b700 1 -- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791] conn(0x7f6ee80ad530 msgr2=0x7f6ee80af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.384+0000 7f6f05d0b700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791] conn(0x7f6ee80ad530 0x7f6ee80af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.384+0000 7f6f05d0b700 1 -- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ee80a3d50 msgr2=0x7f6ee80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.384+0000 7f6f05d0b700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ee80a3d50 0x7f6ee80a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0007fd0 tx=0x7f6ef00079e0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.384+0000 7f6f05d0b700 1 -- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f0006bbb0 msgr2=0x7f6f0019b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.384+0000 7f6f05d0b700 1 --2- 172.21.15.50:0/3002754207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f0006bbb0 0x7f6f0019b7d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4000c00 tx=0x7f6ef40047b0 comp rx=0 tx=0).stop 2024-04-23T18:17:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.384+0000 7f6efffff700 1 -- 172.21.15.50:0/3002754207 reap_dead start 2024-04-23T18:17:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.384+0000 7f6f05d0b700 1 -- 172.21.15.50:0/3002754207 shutdown_connections 2024-04-23T18:17:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.384+0000 7f6f05d0b700 1 -- 172.21.15.50:0/3002754207 >> 172.21.15.50:0/3002754207 conn(0x7f6f000789e0 msgr2=0x7f6f0010de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.384+0000 7f6f05d0b700 1 -- 172.21.15.50:0/3002754207 shutdown_connections 2024-04-23T18:17:13.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:13.384+0000 7f6f05d0b700 1 -- 172.21.15.50:0/3002754207 wait complete. 2024-04-23T18:17:13.746 INFO:teuthology.orchestra.run.smithi050.stdout:77309411393 2024-04-23T18:17:13.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-23T18:17:14.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:14 smithi050 ceph-mon[29187]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-23T18:17:14.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:14 smithi188 ceph-mon[32694]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-23T18:17:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.341+0000 7f168f93d700 1 -- 172.21.15.50:0/2110166090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f168806bbb0 msgr2=0x7f168810f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.341+0000 7f168f93d700 1 --2- 172.21.15.50:0/2110166090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f168806bbb0 0x7f168810f930 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f168000b0a0 tx=0x7f168000b3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.341+0000 7f168f93d700 1 -- 172.21.15.50:0/2110166090 shutdown_connections 2024-04-23T18:17:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.341+0000 7f168f93d700 1 --2- 172.21.15.50:0/2110166090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f168806bbb0 0x7f168810f930 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.341+0000 7f168f93d700 1 --2- 172.21.15.50:0/2110166090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1688108a60 0x7f1688108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.341+0000 7f168f93d700 1 --2- 172.21.15.50:0/2110166090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f168806b1f0 0x7f168806b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.341+0000 7f168f93d700 1 -- 172.21.15.50:0/2110166090 >> 172.21.15.50:0/2110166090 conn(0x7f16880789e0 msgr2=0x7f1688078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.342+0000 7f168f93d700 1 -- 172.21.15.50:0/2110166090 shutdown_connections 2024-04-23T18:17:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.342+0000 7f168f93d700 1 -- 172.21.15.50:0/2110166090 wait complete. 2024-04-23T18:17:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.342+0000 7f168f93d700 1 Processor -- start 2024-04-23T18:17:14.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.342+0000 7f168f93d700 1 -- start start 2024-04-23T18:17:14.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.342+0000 7f168f93d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f168806b1f0 0x7f16881019b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.342+0000 7f168f93d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f168806bbb0 0x7f1688101ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.342+0000 7f168f93d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1688108a60 0x7f1688102430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.342+0000 7f168f93d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1688112ca0 con 0x7f1688108a60 2024-04-23T18:17:14.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.342+0000 7f168f93d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1688112b20 con 0x7f168806bbb0 2024-04-23T18:17:14.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.342+0000 7f168f93d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1688112e20 con 0x7f168806b1f0 2024-04-23T18:17:14.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168deda700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1688108a60 0x7f1688102430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168deda700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1688108a60 0x7f1688102430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57602/0 (socket says 172.21.15.50:57602) 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168deda700 1 -- 172.21.15.50:0/2164287580 learned_addr learned my addr 172.21.15.50:0/2164287580 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168ced8700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f168806bbb0 0x7f1688101ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168d6d9700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f168806b1f0 0x7f16881019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168deda700 1 -- 172.21.15.50:0/2164287580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f168806b1f0 msgr2=0x7f16881019b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168deda700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f168806b1f0 0x7f16881019b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168deda700 1 -- 172.21.15.50:0/2164287580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f168806bbb0 msgr2=0x7f1688101ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168deda700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f168806bbb0 0x7f1688101ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168deda700 1 -- 172.21.15.50:0/2164287580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16881a7630 con 0x7f1688108a60 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168deda700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1688108a60 0x7f1688102430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f168000b830 tx=0x7f1680004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f167e7fc700 1 -- 172.21.15.50:0/2164287580 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1680021070 con 0x7f1688108a60 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168f93d700 1 -- 172.21.15.50:0/2164287580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16881a78c0 con 0x7f1688108a60 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f167e7fc700 1 -- 172.21.15.50:0/2164287580 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1680007e30 con 0x7f1688108a60 2024-04-23T18:17:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f167e7fc700 1 -- 172.21.15.50:0/2164287580 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1680004700 con 0x7f1688108a60 2024-04-23T18:17:14.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.343+0000 7f168f93d700 1 -- 172.21.15.50:0/2164287580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16881a7dc0 con 0x7f1688108a60 2024-04-23T18:17:14.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.344+0000 7f168f93d700 1 -- 172.21.15.50:0/2164287580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1688063150 con 0x7f1688108a60 2024-04-23T18:17:14.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.346+0000 7f167e7fc700 1 -- 172.21.15.50:0/2164287580 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1680003d70 con 0x7f1688108a60 2024-04-23T18:17:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.346+0000 7f167e7fc700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f16740a3cd0 0x7f16740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.346+0000 7f168d6d9700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f16740a3cd0 0x7f16740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.346+0000 7f167e7fc700 1 -- 172.21.15.50:0/2164287580 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f168000f020 con 0x7f1688108a60 2024-04-23T18:17:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.346+0000 7f167e7fc700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435] conn(0x7f16740ad4b0 0x7f16740af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.347+0000 7f167e7fc700 1 -- 172.21.15.50:0/2164287580 --> [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f168001de50 con 0x7f16740ad4b0 2024-04-23T18:17:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.347+0000 7f167e7fc700 1 -- 172.21.15.50:0/2164287580 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f16800d1f70 con 0x7f1688108a60 2024-04-23T18:17:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.347+0000 7f168d6d9700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f16740a3cd0 0x7f16740a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1684009a10 tx=0x7f1684009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.347+0000 7f168ced8700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435] conn(0x7f16740ad4b0 0x7f16740af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.347+0000 7f168ced8700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435] conn(0x7f16740ad4b0 0x7f16740af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:14.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.348+0000 7f167e7fc700 1 -- 172.21.15.50:0/2164287580 <== osd.1 v2:172.21.15.177:6800/3726952435 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f168001de50 con 0x7f16740ad4b0 2024-04-23T18:17:14.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.364+0000 7f168f93d700 1 -- 172.21.15.50:0/2164287580 --> [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f16880635d0 con 0x7f16740ad4b0 2024-04-23T18:17:14.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.365+0000 7f167e7fc700 1 -- 172.21.15.50:0/2164287580 <== osd.1 v2:172.21.15.177:6800/3726952435 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f16880635d0 con 0x7f16740ad4b0 2024-04-23T18:17:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.365+0000 7f168f93d700 1 -- 172.21.15.50:0/2164287580 >> [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435] conn(0x7f16740ad4b0 msgr2=0x7f16740af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.365+0000 7f168f93d700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435] conn(0x7f16740ad4b0 0x7f16740af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.365+0000 7f168f93d700 1 -- 172.21.15.50:0/2164287580 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f16740a3cd0 msgr2=0x7f16740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.365+0000 7f168f93d700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f16740a3cd0 0x7f16740a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1684009a10 tx=0x7f1684009190 comp rx=0 tx=0).stop 2024-04-23T18:17:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.365+0000 7f168f93d700 1 -- 172.21.15.50:0/2164287580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1688108a60 msgr2=0x7f1688102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.365+0000 7f168f93d700 1 --2- 172.21.15.50:0/2164287580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1688108a60 0x7f1688102430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f168000b830 tx=0x7f1680004ce0 comp rx=0 tx=0).stop 2024-04-23T18:17:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.365+0000 7f168deda700 1 -- 172.21.15.50:0/2164287580 reap_dead start 2024-04-23T18:17:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.365+0000 7f168f93d700 1 -- 172.21.15.50:0/2164287580 shutdown_connections 2024-04-23T18:17:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.365+0000 7f168f93d700 1 -- 172.21.15.50:0/2164287580 >> 172.21.15.50:0/2164287580 conn(0x7f16880789e0 msgr2=0x7f16880ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.365+0000 7f168f93d700 1 -- 172.21.15.50:0/2164287580 shutdown_connections 2024-04-23T18:17:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:14.365+0000 7f168f93d700 1 -- 172.21.15.50:0/2164287580 wait complete. 2024-04-23T18:17:14.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:14 smithi177 ceph-mon[32308]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-23T18:17:14.879 INFO:teuthology.orchestra.run.smithi050.stdout:292057776159 2024-04-23T18:17:14.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-23T18:17:15.347 INFO:teuthology.orchestra.run.smithi050.stdout:257698037795 2024-04-23T18:17:15.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-23T18:17:16.149 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:16 smithi050 ceph-mon[29187]: pgmap v307: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:17:16.180 INFO:teuthology.orchestra.run.smithi050.stdout:141733920823 2024-04-23T18:17:16.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-23T18:17:16.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:16 smithi188 ceph-mon[32694]: pgmap v307: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:17:16.339 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:16 smithi177 ceph-mon[32308]: pgmap v307: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:17:16.407 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:16.857 INFO:teuthology.orchestra.run.smithi050.stdout:55834574918 2024-04-23T18:17:16.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-23T18:17:17.312 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:17.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.596+0000 7fa902df9700 1 -- 172.21.15.50:0/1953593435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8fc107990 msgr2=0x7fa8fc107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.596+0000 7fa902df9700 1 --2- 172.21.15.50:0/1953593435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8fc107990 0x7fa8fc107d70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009990 tx=0x7fa8ec009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.596+0000 7fa902df9700 1 -- 172.21.15.50:0/1953593435 shutdown_connections 2024-04-23T18:17:17.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.596+0000 7fa902df9700 1 --2- 172.21.15.50:0/1953593435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8fc102300 0x7fa8fc10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.596+0000 7fa902df9700 1 --2- 172.21.15.50:0/1953593435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8fc107990 0x7fa8fc107d70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.596+0000 7fa902df9700 1 --2- 172.21.15.50:0/1953593435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8fc101940 0x7fa8fc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.596+0000 7fa902df9700 1 -- 172.21.15.50:0/1953593435 >> 172.21.15.50:0/1953593435 conn(0x7fa8fc0788f0 msgr2=0x7fa8fc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.596+0000 7fa902df9700 1 -- 172.21.15.50:0/1953593435 shutdown_connections 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.596+0000 7fa902df9700 1 -- 172.21.15.50:0/1953593435 wait complete. 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa902df9700 1 Processor -- start 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa902df9700 1 -- start start 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa902df9700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8fc101940 0x7fa8fc199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa902df9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8fc102300 0x7fa8fc19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa902df9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8fc107990 0x7fa8fc1a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa902df9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa8fc111ae0 con 0x7fa8fc107990 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa902df9700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa8fc111960 con 0x7fa8fc101940 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa902df9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa8fc111c60 con 0x7fa8fc102300 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa900b95700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8fc101940 0x7fa8fc199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa900b95700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8fc101940 0x7fa8fc199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:38290/0 (socket says 172.21.15.50:38290) 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa900b95700 1 -- 172.21.15.50:0/816185650 learned_addr learned my addr 172.21.15.50:0/816185650 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa901396700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8fc107990 0x7fa8fc1a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa8fbfff700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8fc102300 0x7fa8fc19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa900b95700 1 -- 172.21.15.50:0/816185650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8fc102300 msgr2=0x7fa8fc19a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa900b95700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8fc102300 0x7fa8fc19a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa900b95700 1 -- 172.21.15.50:0/816185650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8fc107990 msgr2=0x7fa8fc1a5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa900b95700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8fc107990 0x7fa8fc1a5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa900b95700 1 -- 172.21.15.50:0/816185650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8fc1a5540 con 0x7fa8fc101940 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.597+0000 7fa901396700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8fc107990 0x7fa8fc1a5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.598+0000 7fa900b95700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8fc101940 0x7fa8fc199f80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0057c0 tx=0x7fa8ec009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.598+0000 7fa8f9ffb700 1 -- 172.21.15.50:0/816185650 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec026070 con 0x7fa8fc101940 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.598+0000 7fa8f9ffb700 1 -- 172.21.15.50:0/816185650 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8ec00fbf0 con 0x7fa8fc101940 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.598+0000 7fa902df9700 1 -- 172.21.15.50:0/816185650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8fc1a5710 con 0x7fa8fc101940 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.598+0000 7fa8f9ffb700 1 -- 172.21.15.50:0/816185650 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec0173f0 con 0x7fa8fc101940 2024-04-23T18:17:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.598+0000 7fa902df9700 1 -- 172.21.15.50:0/816185650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8fc1a5bc0 con 0x7fa8fc101940 2024-04-23T18:17:17.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.599+0000 7fa902df9700 1 -- 172.21.15.50:0/816185650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa8fc063110 con 0x7fa8fc101940 2024-04-23T18:17:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.600+0000 7fa8f9ffb700 1 -- 172.21.15.50:0/816185650 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa8ec029070 con 0x7fa8fc101940 2024-04-23T18:17:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.601+0000 7fa8f9ffb700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa8e40a3cd0 0x7fa8e40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.601+0000 7fa8f9ffb700 1 -- 172.21.15.50:0/816185650 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fa8ec0d16b0 con 0x7fa8fc101940 2024-04-23T18:17:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.601+0000 7fa8fbfff700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa8e40a3cd0 0x7fa8e40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.601+0000 7fa8f9ffb700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913] conn(0x7fa8e40ad380 0x7fa8e40af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.601+0000 7fa8f9ffb700 1 -- 172.21.15.50:0/816185650 --> [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa8ec004e50 con 0x7fa8e40ad380 2024-04-23T18:17:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.601+0000 7fa8f9ffb700 1 -- 172.21.15.50:0/816185650 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fa8ec0d1a60 con 0x7fa8fc101940 2024-04-23T18:17:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.602+0000 7fa8fbfff700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa8e40a3cd0 0x7fa8e40a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0005ea0 tx=0x7fa8f0005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.602+0000 7fa901396700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913] conn(0x7fa8e40ad380 0x7fa8e40af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.602+0000 7fa901396700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913] conn(0x7fa8e40ad380 0x7fa8e40af7a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.603+0000 7fa8f9ffb700 1 -- 172.21.15.50:0/816185650 <== osd.7 v2:172.21.15.177:6816/1787147913 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa8ec004e50 con 0x7fa8e40ad380 2024-04-23T18:17:17.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.620+0000 7fa902df9700 1 -- 172.21.15.50:0/816185650 --> [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa8fc063540 con 0x7fa8e40ad380 2024-04-23T18:17:17.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.621+0000 7fa8f9ffb700 1 -- 172.21.15.50:0/816185650 <== osd.7 v2:172.21.15.177:6816/1787147913 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa8fc063540 con 0x7fa8e40ad380 2024-04-23T18:17:17.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.621+0000 7fa902df9700 1 -- 172.21.15.50:0/816185650 >> [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913] conn(0x7fa8e40ad380 msgr2=0x7fa8e40af7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.621+0000 7fa902df9700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913] conn(0x7fa8e40ad380 0x7fa8e40af7a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.621+0000 7fa902df9700 1 -- 172.21.15.50:0/816185650 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa8e40a3cd0 msgr2=0x7fa8e40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.621+0000 7fa902df9700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa8e40a3cd0 0x7fa8e40a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0005ea0 tx=0x7fa8f0005e30 comp rx=0 tx=0).stop 2024-04-23T18:17:17.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.621+0000 7fa902df9700 1 -- 172.21.15.50:0/816185650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8fc101940 msgr2=0x7fa8fc199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.621+0000 7fa902df9700 1 --2- 172.21.15.50:0/816185650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8fc101940 0x7fa8fc199f80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0057c0 tx=0x7fa8ec009580 comp rx=0 tx=0).stop 2024-04-23T18:17:17.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.621+0000 7fa901396700 1 -- 172.21.15.50:0/816185650 reap_dead start 2024-04-23T18:17:17.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.621+0000 7fa902df9700 1 -- 172.21.15.50:0/816185650 shutdown_connections 2024-04-23T18:17:17.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.621+0000 7fa902df9700 1 -- 172.21.15.50:0/816185650 >> 172.21.15.50:0/816185650 conn(0x7fa8fc0788f0 msgr2=0x7fa8fc1007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:17.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.621+0000 7fa902df9700 1 -- 172.21.15.50:0/816185650 shutdown_connections 2024-04-23T18:17:17.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.622+0000 7fa902df9700 1 -- 172.21.15.50:0/816185650 wait complete. 2024-04-23T18:17:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.901+0000 7f5609061700 1 -- 172.21.15.50:0/3507630702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5604108a20 msgr2=0x7f5604108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.901+0000 7f5609061700 1 --2- 172.21.15.50:0/3507630702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5604108a20 0x7f5604108e00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009990 tx=0x7f55f4009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.901+0000 7f5609061700 1 -- 172.21.15.50:0/3507630702 shutdown_connections 2024-04-23T18:17:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.901+0000 7f5609061700 1 --2- 172.21.15.50:0/3507630702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f560406b980 0x7f560410f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.901+0000 7f5609061700 1 --2- 172.21.15.50:0/3507630702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5604108a20 0x7f5604108e00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.901+0000 7f5609061700 1 --2- 172.21.15.50:0/3507630702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f560406afe0 0x7f560406b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.901+0000 7f5609061700 1 -- 172.21.15.50:0/3507630702 >> 172.21.15.50:0/3507630702 conn(0x7f56040788b0 msgr2=0x7f5604078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:17.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.901+0000 7f5609061700 1 -- 172.21.15.50:0/3507630702 shutdown_connections 2024-04-23T18:17:17.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.901+0000 7f5609061700 1 -- 172.21.15.50:0/3507630702 wait complete. 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f5609061700 1 Processor -- start 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f5609061700 1 -- start start 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f5609061700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f560406afe0 0x7f5604101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f5609061700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f560406b980 0x7f5604101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f5609061700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5604108a20 0x7f5604102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f5609061700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5604112bd0 con 0x7f560406afe0 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f5609061700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5604112a50 con 0x7f5604108a20 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f5609061700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5604112d50 con 0x7f560406b980 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f5602d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f560406afe0 0x7f5604101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f5602d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f560406afe0 0x7f5604101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57634/0 (socket says 172.21.15.50:57634) 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f5602d9d700 1 -- 172.21.15.50:0/1510710276 learned_addr learned my addr 172.21.15.50:0/1510710276 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f560359e700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5604108a20 0x7f5604102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.902+0000 7f560259c700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f560406b980 0x7f5604101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.903+0000 7f5602d9d700 1 -- 172.21.15.50:0/1510710276 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f560406b980 msgr2=0x7f5604101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.903+0000 7f5602d9d700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f560406b980 0x7f5604101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.903+0000 7f5602d9d700 1 -- 172.21.15.50:0/1510710276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5604108a20 msgr2=0x7f5604102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.903+0000 7f5602d9d700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5604108a20 0x7f5604102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.903+0000 7f5602d9d700 1 -- 172.21.15.50:0/1510710276 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56041a7650 con 0x7f560406afe0 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.903+0000 7f560359e700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5604108a20 0x7f5604102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.903+0000 7f5602d9d700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f560406afe0 0x7f5604101a00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f55f4000c00 tx=0x7f55f400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.903+0000 7f55ebfff700 1 -- 172.21.15.50:0/1510710276 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f4027070 con 0x7f560406afe0 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.903+0000 7f5609061700 1 -- 172.21.15.50:0/1510710276 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56041a78e0 con 0x7f560406afe0 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.903+0000 7f55ebfff700 1 -- 172.21.15.50:0/1510710276 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55f4022470 con 0x7f560406afe0 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.903+0000 7f55ebfff700 1 -- 172.21.15.50:0/1510710276 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f4017650 con 0x7f560406afe0 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.903+0000 7f5609061700 1 -- 172.21.15.50:0/1510710276 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56041a7d90 con 0x7f560406afe0 2024-04-23T18:17:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.904+0000 7f5609061700 1 -- 172.21.15.50:0/1510710276 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5604063110 con 0x7f560406afe0 2024-04-23T18:17:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.905+0000 7f55ebfff700 1 -- 172.21.15.50:0/1510710276 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f55f402a070 con 0x7f560406afe0 2024-04-23T18:17:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.906+0000 7f55ebfff700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f55ec0a3cd0 0x7f55ec0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.906+0000 7f560259c700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f55ec0a3cd0 0x7f55ec0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.906+0000 7f55ebfff700 1 -- 172.21.15.50:0/1510710276 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f55f40d14c0 con 0x7f560406afe0 2024-04-23T18:17:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.906+0000 7f55ebfff700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803] conn(0x7f55ec0ad4b0 0x7f55ec0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.906+0000 7f55ebfff700 1 -- 172.21.15.50:0/1510710276 --> [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f55f400eea0 con 0x7f55ec0ad4b0 2024-04-23T18:17:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.906+0000 7f55ebfff700 1 -- 172.21.15.50:0/1510710276 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f55f40d19a0 con 0x7f560406afe0 2024-04-23T18:17:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.906+0000 7f560359e700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803] conn(0x7f55ec0ad4b0 0x7f55ec0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.906+0000 7f560259c700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f55ec0a3cd0 0x7f55ec0a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5604078ff0 tx=0x7f55f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.907+0000 7f560359e700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803] conn(0x7f55ec0ad4b0 0x7f55ec0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.907+0000 7f55ebfff700 1 -- 172.21.15.50:0/1510710276 <== osd.10 v2:172.21.15.177:6824/1988770803 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f55f400eea0 con 0x7f55ec0ad4b0 2024-04-23T18:17:17.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.924+0000 7f5609061700 1 -- 172.21.15.50:0/1510710276 --> [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5604063540 con 0x7f55ec0ad4b0 2024-04-23T18:17:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.924+0000 7f55ebfff700 1 -- 172.21.15.50:0/1510710276 <== osd.10 v2:172.21.15.177:6824/1988770803 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5604063540 con 0x7f55ec0ad4b0 2024-04-23T18:17:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.924+0000 7f5609061700 1 -- 172.21.15.50:0/1510710276 >> [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803] conn(0x7f55ec0ad4b0 msgr2=0x7f55ec0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.924+0000 7f5609061700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803] conn(0x7f55ec0ad4b0 0x7f55ec0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.924+0000 7f5609061700 1 -- 172.21.15.50:0/1510710276 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f55ec0a3cd0 msgr2=0x7f55ec0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.924+0000 7f5609061700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f55ec0a3cd0 0x7f55ec0a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5604078ff0 tx=0x7f55f8009260 comp rx=0 tx=0).stop 2024-04-23T18:17:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.924+0000 7f5609061700 1 -- 172.21.15.50:0/1510710276 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f560406afe0 msgr2=0x7f5604101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.924+0000 7f5609061700 1 --2- 172.21.15.50:0/1510710276 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f560406afe0 0x7f5604101a00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f55f4000c00 tx=0x7f55f400bfd0 comp rx=0 tx=0).stop 2024-04-23T18:17:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.924+0000 7f560359e700 1 -- 172.21.15.50:0/1510710276 reap_dead start 2024-04-23T18:17:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.925+0000 7f5609061700 1 -- 172.21.15.50:0/1510710276 shutdown_connections 2024-04-23T18:17:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.925+0000 7f5609061700 1 -- 172.21.15.50:0/1510710276 >> 172.21.15.50:0/1510710276 conn(0x7f56040788b0 msgr2=0x7f56040ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.925+0000 7f5609061700 1 -- 172.21.15.50:0/1510710276 shutdown_connections 2024-04-23T18:17:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:17.925+0000 7f5609061700 1 -- 172.21.15.50:0/1510710276 wait complete. 2024-04-23T18:17:18.113 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:18 smithi050 ceph-mon[29187]: pgmap v308: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:17:18.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:18 smithi188 ceph-mon[32694]: pgmap v308: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:17:18.533 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:18 smithi177 ceph-mon[32308]: pgmap v308: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T18:17:20.051 INFO:teuthology.orchestra.run.smithi050.stdout:223338299435 2024-04-23T18:17:20.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-23T18:17:20.067 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:20.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:20 smithi188 ceph-mon[32694]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-23T18:17:20.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:20 smithi050 ceph-mon[29187]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-23T18:17:20.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:20 smithi177 ceph-mon[32308]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-23T18:17:20.626 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:20.628 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:20.628 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:20.629 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:21.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.763+0000 7fd5c185d700 1 -- 172.21.15.50:0/3043612779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5bc103450 msgr2=0x7fd5bc10f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:21.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.763+0000 7fd5c185d700 1 --2- 172.21.15.50:0/3043612779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5bc103450 0x7fd5bc10f8b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400b0a0 tx=0x7fd5b400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:21.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.763+0000 7fd5c185d700 1 -- 172.21.15.50:0/3043612779 shutdown_connections 2024-04-23T18:17:21.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.763+0000 7fd5c185d700 1 --2- 172.21.15.50:0/3043612779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5bc103450 0x7fd5bc10f8b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:21.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.763+0000 7fd5c185d700 1 --2- 172.21.15.50:0/3043612779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc108a50 0x7fd5bc108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:21.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.763+0000 7fd5c185d700 1 --2- 172.21.15.50:0/3043612779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5bc102a90 0x7fd5bc102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:21.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.763+0000 7fd5c185d700 1 -- 172.21.15.50:0/3043612779 >> 172.21.15.50:0/3043612779 conn(0x7fd5bc079ba0 msgr2=0x7fd5bc079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:21.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.763+0000 7fd5c185d700 1 -- 172.21.15.50:0/3043612779 shutdown_connections 2024-04-23T18:17:21.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.763+0000 7fd5c185d700 1 -- 172.21.15.50:0/3043612779 wait complete. 2024-04-23T18:17:21.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.764+0000 7fd5c185d700 1 Processor -- start 2024-04-23T18:17:21.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.764+0000 7fd5c185d700 1 -- start start 2024-04-23T18:17:21.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.764+0000 7fd5c185d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5bc102a90 0x7fd5bc199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:21.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.764+0000 7fd5c185d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc103450 0x7fd5bc199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:21.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.764+0000 7fd5c185d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5bc108a50 0x7fd5bc19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:21.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.764+0000 7fd5c185d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc112c50 con 0x7fd5bc108a50 2024-04-23T18:17:21.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.764+0000 7fd5c185d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc112ad0 con 0x7fd5bc102a90 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.764+0000 7fd5c185d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc112dd0 con 0x7fd5bc103450 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.764+0000 7fd5bb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5bc108a50 0x7fd5bc19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5bb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5bc108a50 0x7fd5bc19d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47474/0 (socket says 172.21.15.50:47474) 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5baffd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5bc102a90 0x7fd5bc199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5bb7fe700 1 -- 172.21.15.50:0/2151788960 learned_addr learned my addr 172.21.15.50:0/2151788960 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5ba7fc700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc103450 0x7fd5bc199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5bb7fe700 1 -- 172.21.15.50:0/2151788960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc103450 msgr2=0x7fd5bc199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5bb7fe700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc103450 0x7fd5bc199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5bb7fe700 1 -- 172.21.15.50:0/2151788960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5bc102a90 msgr2=0x7fd5bc199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5bb7fe700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5bc102a90 0x7fd5bc199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5bb7fe700 1 -- 172.21.15.50:0/2151788960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5bc19e0e0 con 0x7fd5bc108a50 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5bb7fe700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5bc108a50 0x7fd5bc19d9a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400b070 tx=0x7fd5b4004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5c085b700 1 -- 172.21.15.50:0/2151788960 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4021070 con 0x7fd5bc108a50 2024-04-23T18:17:21.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5c085b700 1 -- 172.21.15.50:0/2151788960 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5b4007e30 con 0x7fd5bc108a50 2024-04-23T18:17:21.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5c085b700 1 -- 172.21.15.50:0/2151788960 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b40046d0 con 0x7fd5bc108a50 2024-04-23T18:17:21.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5c185d700 1 -- 172.21.15.50:0/2151788960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5bc19e370 con 0x7fd5bc108a50 2024-04-23T18:17:21.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.765+0000 7fd5c185d700 1 -- 172.21.15.50:0/2151788960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5bc1a5c50 con 0x7fd5bc108a50 2024-04-23T18:17:21.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.767+0000 7fd5c185d700 1 -- 172.21.15.50:0/2151788960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5bc069730 con 0x7fd5bc108a50 2024-04-23T18:17:21.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.768+0000 7fd5c085b700 1 -- 172.21.15.50:0/2151788960 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd5b4007980 con 0x7fd5bc108a50 2024-04-23T18:17:21.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.768+0000 7fd5c085b700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd5a40a3d50 0x7fd5a40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:21.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.769+0000 7fd5baffd700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd5a40a3d50 0x7fd5a40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:21.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.769+0000 7fd5c085b700 1 -- 172.21.15.50:0/2151788960 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fd5b40d12b0 con 0x7fd5bc108a50 2024-04-23T18:17:21.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.769+0000 7fd5baffd700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd5a40a3d50 0x7fd5a40a6210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009be0 tx=0x7fd5ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:21.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.772+0000 7fd5c085b700 1 -- 172.21.15.50:0/2151788960 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd5b409dc70 con 0x7fd5bc108a50 2024-04-23T18:17:21.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.983+0000 7fd5c185d700 1 -- 172.21.15.50:0/2151788960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fd5bc02df80 con 0x7fd5bc108a50 2024-04-23T18:17:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.983+0000 7fd5c085b700 1 -- 172.21.15.50:0/2151788960 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd5b4004870 con 0x7fd5bc108a50 2024-04-23T18:17:21.978 INFO:teuthology.orchestra.run.smithi050.stdout:197568495663 2024-04-23T18:17:21.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.986+0000 7fd5c185d700 1 -- 172.21.15.50:0/2151788960 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd5a40a3d50 msgr2=0x7fd5a40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:21.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.986+0000 7fd5c185d700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd5a40a3d50 0x7fd5a40a6210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009be0 tx=0x7fd5ac009260 comp rx=0 tx=0).stop 2024-04-23T18:17:21.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.986+0000 7fd5c185d700 1 -- 172.21.15.50:0/2151788960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5bc108a50 msgr2=0x7fd5bc19d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.986+0000 7fd5c185d700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5bc108a50 0x7fd5bc19d9a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400b070 tx=0x7fd5b4004cf0 comp rx=0 tx=0).stop 2024-04-23T18:17:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.986+0000 7fd5c185d700 1 -- 172.21.15.50:0/2151788960 shutdown_connections 2024-04-23T18:17:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.986+0000 7fd5c185d700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5bc108a50 0x7fd5bc19d9a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.986+0000 7fd5c185d700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd5a40a3d50 0x7fd5a40a6210 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.986+0000 7fd5c185d700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5bc102a90 0x7fd5bc199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.986+0000 7fd5c185d700 1 --2- 172.21.15.50:0/2151788960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc103450 0x7fd5bc199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.986+0000 7fd5c185d700 1 -- 172.21.15.50:0/2151788960 >> 172.21.15.50:0/2151788960 conn(0x7fd5bc079ba0 msgr2=0x7fd5bc1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.986+0000 7fd5c185d700 1 -- 172.21.15.50:0/2151788960 shutdown_connections 2024-04-23T18:17:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:21.986+0000 7fd5c185d700 1 -- 172.21.15.50:0/2151788960 wait complete. 2024-04-23T18:17:22.200 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:22.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:22 smithi050 ceph-mon[29187]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-23T18:17:22.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:22 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2151788960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T18:17:22.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:22 smithi188 ceph-mon[32694]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-23T18:17:22.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:22 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2151788960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T18:17:22.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:22 smithi177 ceph-mon[32308]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-23T18:17:22.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:22 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2151788960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T18:17:23.169 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495659 got 197568495663 for osd.6 2024-04-23T18:17:23.169 DEBUG:teuthology.parallel:result is None 2024-04-23T18:17:23.641 INFO:teuthology.orchestra.run.smithi050.stdout:317827579931 2024-04-23T18:17:23.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-23T18:17:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.853+0000 7f52527d6700 1 -- 172.21.15.50:0/2338023372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f524c06afe0 msgr2=0x7f524c06b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.853+0000 7f52527d6700 1 --2- 172.21.15.50:0/2338023372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f524c06afe0 0x7f524c06b440 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5240009990 tx=0x7f5240009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.853+0000 7f52527d6700 1 -- 172.21.15.50:0/2338023372 shutdown_connections 2024-04-23T18:17:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.853+0000 7f52527d6700 1 --2- 172.21.15.50:0/2338023372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f524c06b980 0x7f524c10f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.853+0000 7f52527d6700 1 --2- 172.21.15.50:0/2338023372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f524c108a20 0x7f524c108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.853+0000 7f52527d6700 1 --2- 172.21.15.50:0/2338023372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f524c06afe0 0x7f524c06b440 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.853+0000 7f52527d6700 1 -- 172.21.15.50:0/2338023372 >> 172.21.15.50:0/2338023372 conn(0x7f524c0788b0 msgr2=0x7f524c078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.853+0000 7f52527d6700 1 -- 172.21.15.50:0/2338023372 shutdown_connections 2024-04-23T18:17:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.853+0000 7f52527d6700 1 -- 172.21.15.50:0/2338023372 wait complete. 2024-04-23T18:17:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.854+0000 7f52527d6700 1 Processor -- start 2024-04-23T18:17:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.854+0000 7f52527d6700 1 -- start start 2024-04-23T18:17:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.855+0000 7f52527d6700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f524c06afe0 0x7f524c1990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.855+0000 7f52527d6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f524c06b980 0x7f524c199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.855+0000 7f52527d6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f524c108a20 0x7f524c19d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.855+0000 7f52527d6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f524c112be0 con 0x7f524c108a20 2024-04-23T18:17:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.855+0000 7f52527d6700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f524c112a60 con 0x7f524c06afe0 2024-04-23T18:17:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.855+0000 7f52527d6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f524c112d60 con 0x7f524c06b980 2024-04-23T18:17:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.855+0000 7f524bfff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f524c06afe0 0x7f524c1990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:23.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.855+0000 7f5250d73700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f524c108a20 0x7f524c19d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:23.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.855+0000 7f524bfff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f524c06afe0 0x7f524c1990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:35356/0 (socket says 172.21.15.50:35356) 2024-04-23T18:17:23.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.855+0000 7f5250d73700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f524c108a20 0x7f524c19d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47498/0 (socket says 172.21.15.50:47498) 2024-04-23T18:17:23.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.855+0000 7f524bfff700 1 -- 172.21.15.50:0/4269024586 learned_addr learned my addr 172.21.15.50:0/4269024586 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:23.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.855+0000 7f524b7fe700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f524c06b980 0x7f524c199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.856+0000 7f524bfff700 1 -- 172.21.15.50:0/4269024586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f524c06b980 msgr2=0x7f524c199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.856+0000 7f524bfff700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f524c06b980 0x7f524c199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.856+0000 7f524bfff700 1 -- 172.21.15.50:0/4269024586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f524c108a20 msgr2=0x7f524c19d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.856+0000 7f524bfff700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f524c108a20 0x7f524c19d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.856+0000 7f524bfff700 1 -- 172.21.15.50:0/4269024586 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524c19e100 con 0x7f524c06afe0 2024-04-23T18:17:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.856+0000 7f5250d73700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f524c108a20 0x7f524c19d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.856+0000 7f524bfff700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f524c06afe0 0x7f524c1990f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f523c00e9a0 tx=0x7f523c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.856+0000 7f52497fa700 1 -- 172.21.15.50:0/4269024586 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c00ca10 con 0x7f524c06afe0 2024-04-23T18:17:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.856+0000 7f52497fa700 1 -- 172.21.15.50:0/4269024586 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f523c00cbb0 con 0x7f524c06afe0 2024-04-23T18:17:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.856+0000 7f52527d6700 1 -- 172.21.15.50:0/4269024586 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f524c19e3f0 con 0x7f524c06afe0 2024-04-23T18:17:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.856+0000 7f52497fa700 1 -- 172.21.15.50:0/4269024586 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c018680 con 0x7f524c06afe0 2024-04-23T18:17:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.856+0000 7f52527d6700 1 -- 172.21.15.50:0/4269024586 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f524c1a5c70 con 0x7f524c06afe0 2024-04-23T18:17:23.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.857+0000 7f52527d6700 1 -- 172.21.15.50:0/4269024586 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f524c069730 con 0x7f524c06afe0 2024-04-23T18:17:23.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.859+0000 7f52497fa700 1 -- 172.21.15.50:0/4269024586 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f523c010430 con 0x7f524c06afe0 2024-04-23T18:17:23.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.859+0000 7f52497fa700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52340a3d50 0x7f52340a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:23.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.860+0000 7f524b7fe700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52340a3d50 0x7f52340a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:23.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.860+0000 7f52497fa700 1 -- 172.21.15.50:0/4269024586 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f523c0d0840 con 0x7f524c06afe0 2024-04-23T18:17:23.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.860+0000 7f524b7fe700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52340a3d50 0x7f52340a6210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f524000b2e0 tx=0x7f5240005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:23.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:23.863+0000 7f52497fa700 1 -- 172.21.15.50:0/4269024586 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f523c09d170 con 0x7f524c06afe0 2024-04-23T18:17:24.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.086+0000 7f52527d6700 1 -- 172.21.15.50:0/4269024586 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f524c19a450 con 0x7f524c06afe0 2024-04-23T18:17:24.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.087+0000 7f52497fa700 1 -- 172.21.15.50:0/4269024586 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f523c09ca70 con 0x7f524c06afe0 2024-04-23T18:17:24.081 INFO:teuthology.orchestra.run.smithi050.stdout:107374182462 2024-04-23T18:17:24.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.089+0000 7f52527d6700 1 -- 172.21.15.50:0/4269024586 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52340a3d50 msgr2=0x7f52340a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:24.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.089+0000 7f52527d6700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52340a3d50 0x7f52340a6210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f524000b2e0 tx=0x7f5240005960 comp rx=0 tx=0).stop 2024-04-23T18:17:24.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.089+0000 7f52527d6700 1 -- 172.21.15.50:0/4269024586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f524c06afe0 msgr2=0x7f524c1990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:24.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.089+0000 7f52527d6700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f524c06afe0 0x7f524c1990f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f523c00e9a0 tx=0x7f523c00ecb0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.089+0000 7f52527d6700 1 -- 172.21.15.50:0/4269024586 shutdown_connections 2024-04-23T18:17:24.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.089+0000 7f52527d6700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f524c108a20 0x7f524c19d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.089+0000 7f52527d6700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f52340a3d50 0x7f52340a6210 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.089+0000 7f52527d6700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f524c06afe0 0x7f524c1990f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.089+0000 7f52527d6700 1 --2- 172.21.15.50:0/4269024586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f524c06b980 0x7f524c199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:24.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.089+0000 7f52527d6700 1 -- 172.21.15.50:0/4269024586 >> 172.21.15.50:0/4269024586 conn(0x7f524c0788b0 msgr2=0x7f524c0ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:24.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.089+0000 7f52527d6700 1 -- 172.21.15.50:0/4269024586 shutdown_connections 2024-04-23T18:17:24.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:24.089+0000 7f52527d6700 1 -- 172.21.15.50:0/4269024586 wait complete. 2024-04-23T18:17:24.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:24 smithi050 ceph-mon[29187]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-23T18:17:24.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:24 smithi188 ceph-mon[32694]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-23T18:17:24.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:24 smithi177 ceph-mon[32308]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-23T18:17:25.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:25 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4269024586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T18:17:25.338 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:25 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4269024586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T18:17:25.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:25 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4269024586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T18:17:26.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:26 smithi188 ceph-mon[32694]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:17:26.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:26 smithi050 ceph-mon[29187]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:17:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:26.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:26 smithi177 ceph-mon[32308]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:17:28.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:28 smithi188 ceph-mon[32694]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:17:28.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:28 smithi050 ceph-mon[29187]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:17:28.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:28 smithi177 ceph-mon[32308]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:17:30.140 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:30 smithi050 ceph-mon[29187]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:17:30.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:30 smithi188 ceph-mon[32694]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:17:30.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.494+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/3477363088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac4103480 msgr2=0x7f2ac410f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:30.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.494+0000 7f2ac9e0d700 1 --2- 172.21.15.50:0/3477363088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac4103480 0x7f2ac410f8d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00b0a0 tx=0x7f2abc00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.494+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/3477363088 shutdown_connections 2024-04-23T18:17:30.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.494+0000 7f2ac9e0d700 1 --2- 172.21.15.50:0/3477363088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac4103480 0x7f2ac410f8d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.494+0000 7f2ac9e0d700 1 --2- 172.21.15.50:0/3477363088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ac4108a70 0x7f2ac4108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.494+0000 7f2ac9e0d700 1 --2- 172.21.15.50:0/3477363088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac4102ac0 0x7f2ac4102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.494+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/3477363088 >> 172.21.15.50:0/3477363088 conn(0x7f2ac4079bb0 msgr2=0x7f2ac4079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:30.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.494+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/3477363088 shutdown_connections 2024-04-23T18:17:30.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.494+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/3477363088 wait complete. 2024-04-23T18:17:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.495+0000 7f2ac9e0d700 1 Processor -- start 2024-04-23T18:17:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.495+0000 7f2ac9e0d700 1 -- start start 2024-04-23T18:17:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.495+0000 7f2ac9e0d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ac4102ac0 0x7f2ac4074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.495+0000 7f2ac9e0d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac4103480 0x7f2ac4075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.495+0000 7f2ac9e0d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac4108a70 0x7f2ac40757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.495+0000 7f2ac9e0d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac4112c80 con 0x7f2ac4108a70 2024-04-23T18:17:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.495+0000 7f2ac9e0d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac4112b00 con 0x7f2ac4102ac0 2024-04-23T18:17:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.495+0000 7f2ac9e0d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac4112e00 con 0x7f2ac4103480 2024-04-23T18:17:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac4108a70 0x7f2ac40757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac4108a70 0x7f2ac40757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47514/0 (socket says 172.21.15.50:47514) 2024-04-23T18:17:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac3fff700 1 -- 172.21.15.50:0/1019913843 learned_addr learned my addr 172.21.15.50:0/1019913843 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac2ffd700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac4103480 0x7f2ac4075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac37fe700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ac4102ac0 0x7f2ac4074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac3fff700 1 -- 172.21.15.50:0/1019913843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac4103480 msgr2=0x7f2ac4075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac3fff700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac4103480 0x7f2ac4075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac3fff700 1 -- 172.21.15.50:0/1019913843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ac4102ac0 msgr2=0x7f2ac4074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac3fff700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ac4102ac0 0x7f2ac4074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac3fff700 1 -- 172.21.15.50:0/1019913843 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac41a7620 con 0x7f2ac4108a70 2024-04-23T18:17:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac3fff700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac4108a70 0x7f2ac40757c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0062a0 tx=0x7f2abc004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac0ff9700 1 -- 172.21.15.50:0/1019913843 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc021070 con 0x7f2ac4108a70 2024-04-23T18:17:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac0ff9700 1 -- 172.21.15.50:0/1019913843 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2abc007d00 con 0x7f2ac4108a70 2024-04-23T18:17:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac0ff9700 1 -- 172.21.15.50:0/1019913843 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc01d450 con 0x7f2ac4108a70 2024-04-23T18:17:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/1019913843 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac41a78b0 con 0x7f2ac4108a70 2024-04-23T18:17:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.496+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/1019913843 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ac41a7df0 con 0x7f2ac4108a70 2024-04-23T18:17:30.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.497+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/1019913843 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ac4069730 con 0x7f2ac4108a70 2024-04-23T18:17:30.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.498+0000 7f2ac0ff9700 1 -- 172.21.15.50:0/1019913843 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2abc004020 con 0x7f2ac4108a70 2024-04-23T18:17:30.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.499+0000 7f2ac0ff9700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2aac0a3cd0 0x7f2aac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:30.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.499+0000 7f2ac37fe700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2aac0a3cd0 0x7f2aac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:30.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.499+0000 7f2ac0ff9700 1 -- 172.21.15.50:0/1019913843 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f2abc00f020 con 0x7f2ac4108a70 2024-04-23T18:17:30.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.500+0000 7f2ac37fe700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2aac0a3cd0 0x7f2aac0a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4006fd0 tx=0x7f2ab4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:30.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.504+0000 7f2ac0ff9700 1 -- 172.21.15.50:0/1019913843 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2abc02a020 con 0x7f2ac4108a70 2024-04-23T18:17:30.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:30 smithi177 ceph-mon[32308]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:17:30.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.702+0000 7fad0ea13700 1 -- 172.21.15.50:0/793305113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad08107980 msgr2=0x7fad08107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:30.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.702+0000 7fad0ea13700 1 --2- 172.21.15.50:0/793305113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad08107980 0x7fad08107d60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7facf8009990 tx=0x7facf8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.702+0000 7fad0ea13700 1 -- 172.21.15.50:0/793305113 shutdown_connections 2024-04-23T18:17:30.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.702+0000 7fad0ea13700 1 --2- 172.21.15.50:0/793305113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad081022f0 0x7fad0810e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.702+0000 7fad0ea13700 1 --2- 172.21.15.50:0/793305113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad08107980 0x7fad08107d60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.702+0000 7fad0ea13700 1 --2- 172.21.15.50:0/793305113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad08101930 0x7fad08101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.702+0000 7fad0ea13700 1 -- 172.21.15.50:0/793305113 >> 172.21.15.50:0/793305113 conn(0x7fad080788b0 msgr2=0x7fad08078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:30.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.702+0000 7fad0ea13700 1 -- 172.21.15.50:0/793305113 shutdown_connections 2024-04-23T18:17:30.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.702+0000 7fad0ea13700 1 -- 172.21.15.50:0/793305113 wait complete. 2024-04-23T18:17:30.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.703+0000 7fad0ea13700 1 Processor -- start 2024-04-23T18:17:30.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.703+0000 7fad0ea13700 1 -- start start 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0ea13700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad08101930 0x7fad08199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0ea13700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad081022f0 0x7fad0819a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0ea13700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad08107980 0x7fad081a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0ea13700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fad08111ad0 con 0x7fad08107980 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0ea13700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fad08111950 con 0x7fad08101930 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0ea13700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fad08111c50 con 0x7fad081022f0 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0cfb0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad08107980 0x7fad081a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0cfb0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad08107980 0x7fad081a4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47534/0 (socket says 172.21.15.50:47534) 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0cfb0700 1 -- 172.21.15.50:0/41269077 learned_addr learned my addr 172.21.15.50:0/41269077 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad07fff700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad08101930 0x7fad08199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad077fe700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad081022f0 0x7fad0819a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0cfb0700 1 -- 172.21.15.50:0/41269077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad081022f0 msgr2=0x7fad0819a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0cfb0700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad081022f0 0x7fad0819a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0cfb0700 1 -- 172.21.15.50:0/41269077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad08101930 msgr2=0x7fad08199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:30.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0cfb0700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad08101930 0x7fad08199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0cfb0700 1 -- 172.21.15.50:0/41269077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad081a54f0 con 0x7fad08107980 2024-04-23T18:17:30.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.704+0000 7fad0cfb0700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad08107980 0x7fad081a4fb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fad0000ada0 tx=0x7fad0000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:30.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.705+0000 7fad057fa700 1 -- 172.21.15.50:0/41269077 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad00017020 con 0x7fad08107980 2024-04-23T18:17:30.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.705+0000 7fad057fa700 1 -- 172.21.15.50:0/41269077 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad0000ce30 con 0x7fad08107980 2024-04-23T18:17:30.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.705+0000 7fad0ea13700 1 -- 172.21.15.50:0/41269077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad081a5780 con 0x7fad08107980 2024-04-23T18:17:30.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.705+0000 7fad057fa700 1 -- 172.21.15.50:0/41269077 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad00005350 con 0x7fad08107980 2024-04-23T18:17:30.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.705+0000 7fad0ea13700 1 -- 172.21.15.50:0/41269077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad081a5c90 con 0x7fad08107980 2024-04-23T18:17:30.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.706+0000 7fad0ea13700 1 -- 172.21.15.50:0/41269077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad08109ea0 con 0x7fad08107980 2024-04-23T18:17:30.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.708+0000 7fad057fa700 1 -- 172.21.15.50:0/41269077 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fad0000c980 con 0x7fad08107980 2024-04-23T18:17:30.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.709+0000 7fad057fa700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7facf00a3d50 0x7facf00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:30.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.709+0000 7fad07fff700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7facf00a3d50 0x7facf00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:30.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.709+0000 7fad057fa700 1 -- 172.21.15.50:0/41269077 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fad000d00f0 con 0x7fad08107980 2024-04-23T18:17:30.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.709+0000 7fad07fff700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7facf00a3d50 0x7facf00a6210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7facf800b5c0 tx=0x7facf8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:30.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.713+0000 7fad057fa700 1 -- 172.21.15.50:0/41269077 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad00021020 con 0x7fad08107980 2024-04-23T18:17:30.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.719+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/1019913843 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f2ac402e140 con 0x7f2ac4108a70 2024-04-23T18:17:30.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.719+0000 7f2ac0ff9700 1 -- 172.21.15.50:0/1019913843 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f2abc09e5d0 con 0x7f2ac4108a70 2024-04-23T18:17:30.713 INFO:teuthology.orchestra.run.smithi050.stdout:352187318295 2024-04-23T18:17:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.721+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/1019913843 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2aac0a3cd0 msgr2=0x7f2aac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.721+0000 7f2ac9e0d700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2aac0a3cd0 0x7f2aac0a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4006fd0 tx=0x7f2ab4008040 comp rx=0 tx=0).stop 2024-04-23T18:17:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.721+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/1019913843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac4108a70 msgr2=0x7f2ac40757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.721+0000 7f2ac9e0d700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac4108a70 0x7f2ac40757c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0062a0 tx=0x7f2abc004ed0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.721+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/1019913843 shutdown_connections 2024-04-23T18:17:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.721+0000 7f2ac9e0d700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac4108a70 0x7f2ac40757c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.721+0000 7f2ac9e0d700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f2aac0a3cd0 0x7f2aac0a6190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.721+0000 7f2ac9e0d700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ac4102ac0 0x7f2ac4074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.721+0000 7f2ac9e0d700 1 --2- 172.21.15.50:0/1019913843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ac4103480 0x7f2ac4075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.721+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/1019913843 >> 172.21.15.50:0/1019913843 conn(0x7f2ac4079bb0 msgr2=0x7f2ac4101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:30.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.722+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/1019913843 shutdown_connections 2024-04-23T18:17:30.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.722+0000 7f2ac9e0d700 1 -- 172.21.15.50:0/1019913843 wait complete. 2024-04-23T18:17:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.924+0000 7fad0ea13700 1 -- 172.21.15.50:0/41269077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fad08109880 con 0x7fad08107980 2024-04-23T18:17:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.924+0000 7fad057fa700 1 -- 172.21.15.50:0/41269077 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fad0001a310 con 0x7fad08107980 2024-04-23T18:17:30.918 INFO:teuthology.orchestra.run.smithi050.stdout:163208757302 2024-04-23T18:17:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.926+0000 7fad0ea13700 1 -- 172.21.15.50:0/41269077 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7facf00a3d50 msgr2=0x7facf00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.926+0000 7fad0ea13700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7facf00a3d50 0x7facf00a6210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7facf800b5c0 tx=0x7facf8005960 comp rx=0 tx=0).stop 2024-04-23T18:17:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.926+0000 7fad0ea13700 1 -- 172.21.15.50:0/41269077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad08107980 msgr2=0x7fad081a4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.926+0000 7fad0ea13700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad08107980 0x7fad081a4fb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fad0000ada0 tx=0x7fad0000c330 comp rx=0 tx=0).stop 2024-04-23T18:17:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.926+0000 7fad0ea13700 1 -- 172.21.15.50:0/41269077 shutdown_connections 2024-04-23T18:17:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.926+0000 7fad0ea13700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad08107980 0x7fad081a4fb0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.926+0000 7fad0ea13700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7facf00a3d50 0x7facf00a6210 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.926+0000 7fad0ea13700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad08101930 0x7fad08199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.926+0000 7fad0ea13700 1 --2- 172.21.15.50:0/41269077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad081022f0 0x7fad0819a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.926+0000 7fad0ea13700 1 -- 172.21.15.50:0/41269077 >> 172.21.15.50:0/41269077 conn(0x7fad080788b0 msgr2=0x7fad081007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.927+0000 7fad0ea13700 1 -- 172.21.15.50:0/41269077 shutdown_connections 2024-04-23T18:17:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:30.927+0000 7fad0ea13700 1 -- 172.21.15.50:0/41269077 wait complete. 2024-04-23T18:17:31.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:31 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1019913843' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T18:17:31.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:31 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/41269077' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T18:17:31.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:31 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1019913843' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T18:17:31.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:31 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/41269077' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T18:17:31.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:31 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1019913843' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T18:17:31.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:31 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/41269077' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T18:17:32.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:32 smithi050 ceph-mon[29187]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-23T18:17:32.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:32 smithi177 ceph-mon[32308]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-23T18:17:32.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:32 smithi188 ceph-mon[32694]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-23T18:17:32.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.693+0000 7fcfe362a700 1 -- 172.21.15.50:0/1914993223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfdc108ad0 msgr2=0x7fcfdc108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.693+0000 7fcfe362a700 1 --2- 172.21.15.50:0/1914993223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfdc108ad0 0x7fcfdc108eb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009990 tx=0x7fcfd8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.694+0000 7fcfe362a700 1 -- 172.21.15.50:0/1914993223 shutdown_connections 2024-04-23T18:17:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.694+0000 7fcfe362a700 1 --2- 172.21.15.50:0/1914993223 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfdc1019d0 0x7fcfdc10f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.694+0000 7fcfe362a700 1 --2- 172.21.15.50:0/1914993223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfdc108ad0 0x7fcfdc108eb0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.694+0000 7fcfe362a700 1 --2- 172.21.15.50:0/1914993223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc101030 0x7fcfdc101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.694+0000 7fcfe362a700 1 -- 172.21.15.50:0/1914993223 >> 172.21.15.50:0/1914993223 conn(0x7fcfdc077fb0 msgr2=0x7fcfdc0ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.694+0000 7fcfe362a700 1 -- 172.21.15.50:0/1914993223 shutdown_connections 2024-04-23T18:17:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.694+0000 7fcfe362a700 1 -- 172.21.15.50:0/1914993223 wait complete. 2024-04-23T18:17:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.694+0000 7fcfe362a700 1 Processor -- start 2024-04-23T18:17:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.694+0000 7fcfe362a700 1 -- start start 2024-04-23T18:17:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe362a700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc101030 0x7fcfdc1990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe362a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfdc1019d0 0x7fcfdc199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe362a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfdc108ad0 0x7fcfdc19d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe362a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc112be0 con 0x7fcfdc1019d0 2024-04-23T18:17:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe362a700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc112a60 con 0x7fcfdc101030 2024-04-23T18:17:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe362a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc112d60 con 0x7fcfdc108ad0 2024-04-23T18:17:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe0bc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfdc1019d0 0x7fcfdc199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe0bc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfdc1019d0 0x7fcfdc199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51338/0 (socket says 172.21.15.50:51338) 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe0bc5700 1 -- 172.21.15.50:0/3672555439 learned_addr learned my addr 172.21.15.50:0/3672555439 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe1bc7700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfdc108ad0 0x7fcfdc19d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe13c6700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc101030 0x7fcfdc1990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe0bc5700 1 -- 172.21.15.50:0/3672555439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfdc108ad0 msgr2=0x7fcfdc19d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe0bc5700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfdc108ad0 0x7fcfdc19d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe0bc5700 1 -- 172.21.15.50:0/3672555439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc101030 msgr2=0x7fcfdc1990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe0bc5700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc101030 0x7fcfdc1990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe0bc5700 1 -- 172.21.15.50:0/3672555439 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc19e100 con 0x7fcfdc1019d0 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe13c6700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc101030 0x7fcfdc1990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfe0bc5700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfdc1019d0 0x7fcfdc199630 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00e9a0 tx=0x7fcfcc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.695+0000 7fcfd27fc700 1 -- 172.21.15.50:0/3672555439 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc00cbd0 con 0x7fcfdc1019d0 2024-04-23T18:17:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.696+0000 7fcfd27fc700 1 -- 172.21.15.50:0/3672555439 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfcc00cd70 con 0x7fcfdc1019d0 2024-04-23T18:17:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.696+0000 7fcfe362a700 1 -- 172.21.15.50:0/3672555439 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfdc19e3f0 con 0x7fcfdc1019d0 2024-04-23T18:17:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.696+0000 7fcfd27fc700 1 -- 172.21.15.50:0/3672555439 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc018680 con 0x7fcfdc1019d0 2024-04-23T18:17:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.696+0000 7fcfe362a700 1 -- 172.21.15.50:0/3672555439 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfdc1a5c70 con 0x7fcfdc1019d0 2024-04-23T18:17:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.697+0000 7fcfe362a700 1 -- 172.21.15.50:0/3672555439 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfdc104350 con 0x7fcfdc1019d0 2024-04-23T18:17:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.698+0000 7fcfd27fc700 1 -- 172.21.15.50:0/3672555439 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcfcc010970 con 0x7fcfdc1019d0 2024-04-23T18:17:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.698+0000 7fcfd27fc700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcfc80a3d50 0x7fcfc80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.699+0000 7fcfe13c6700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcfc80a3d50 0x7fcfc80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.699+0000 7fcfd27fc700 1 -- 172.21.15.50:0/3672555439 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fcfcc0d0b60 con 0x7fcfdc1019d0 2024-04-23T18:17:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.699+0000 7fcfe13c6700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcfc80a3d50 0x7fcfc80a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800b2e0 tx=0x7fcfd8005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:32.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.702+0000 7fcfd27fc700 1 -- 172.21.15.50:0/3672555439 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcfcc09d590 con 0x7fcfdc1019d0 2024-04-23T18:17:32.816 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757297 got 163208757302 for osd.5 2024-04-23T18:17:32.816 DEBUG:teuthology.parallel:result is None 2024-04-23T18:17:32.891 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182457 got 107374182462 for osd.3 2024-04-23T18:17:32.892 DEBUG:teuthology.parallel:result is None 2024-04-23T18:17:32.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.914+0000 7fcfe362a700 1 -- 172.21.15.50:0/3672555439 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fcfdc02df80 con 0x7fcfdc1019d0 2024-04-23T18:17:32.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.914+0000 7fcfd27fc700 1 -- 172.21.15.50:0/3672555439 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcfcc09ce90 con 0x7fcfdc1019d0 2024-04-23T18:17:32.908 INFO:teuthology.orchestra.run.smithi050.stdout:34359738447 2024-04-23T18:17:32.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.916+0000 7fcfe362a700 1 -- 172.21.15.50:0/3672555439 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcfc80a3d50 msgr2=0x7fcfc80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:32.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.916+0000 7fcfe362a700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcfc80a3d50 0x7fcfc80a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800b2e0 tx=0x7fcfd8005e80 comp rx=0 tx=0).stop 2024-04-23T18:17:32.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.916+0000 7fcfe362a700 1 -- 172.21.15.50:0/3672555439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfdc1019d0 msgr2=0x7fcfdc199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:32.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.916+0000 7fcfe362a700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfdc1019d0 0x7fcfdc199630 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00e9a0 tx=0x7fcfcc00ed60 comp rx=0 tx=0).stop 2024-04-23T18:17:32.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.916+0000 7fcfe362a700 1 -- 172.21.15.50:0/3672555439 shutdown_connections 2024-04-23T18:17:32.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.916+0000 7fcfe362a700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfdc108ad0 0x7fcfdc19d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:32.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.916+0000 7fcfe362a700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcfc80a3d50 0x7fcfc80a6210 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:32.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.916+0000 7fcfe362a700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc101030 0x7fcfdc1990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:32.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.916+0000 7fcfe362a700 1 --2- 172.21.15.50:0/3672555439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfdc1019d0 0x7fcfdc199630 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:32.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.916+0000 7fcfe362a700 1 -- 172.21.15.50:0/3672555439 >> 172.21.15.50:0/3672555439 conn(0x7fcfdc077fb0 msgr2=0x7fcfdc0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:32.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.916+0000 7fcfe362a700 1 -- 172.21.15.50:0/3672555439 shutdown_connections 2024-04-23T18:17:32.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:32.916+0000 7fcfe362a700 1 -- 172.21.15.50:0/3672555439 wait complete. 2024-04-23T18:17:33.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:33 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3672555439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T18:17:33.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:33 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:17:33.524 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:33.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:33 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3672555439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T18:17:33.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:33 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:17:33.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:33 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3672555439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T18:17:33.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:33 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:17:34.103 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:34 smithi177 ceph-mon[32308]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-23T18:17:34.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.246+0000 7f590bbe2700 1 -- 172.21.15.50:0/3857392592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5904103480 msgr2=0x7f590410f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.246+0000 7f590bbe2700 1 --2- 172.21.15.50:0/3857392592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5904103480 0x7f590410f8d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00b0a0 tx=0x7f58fc00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.246+0000 7f590bbe2700 1 -- 172.21.15.50:0/3857392592 shutdown_connections 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.246+0000 7f590bbe2700 1 --2- 172.21.15.50:0/3857392592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5904103480 0x7f590410f8d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.246+0000 7f590bbe2700 1 --2- 172.21.15.50:0/3857392592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5904108a70 0x7f5904108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.246+0000 7f590bbe2700 1 --2- 172.21.15.50:0/3857392592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5904102ac0 0x7f5904102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.246+0000 7f590bbe2700 1 -- 172.21.15.50:0/3857392592 >> 172.21.15.50:0/3857392592 conn(0x7f5904079bb0 msgr2=0x7f5904079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.246+0000 7f590bbe2700 1 -- 172.21.15.50:0/3857392592 shutdown_connections 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590bbe2700 1 -- 172.21.15.50:0/3857392592 wait complete. 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590bbe2700 1 Processor -- start 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590bbe2700 1 -- start start 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590bbe2700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5904102ac0 0x7f59041990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590bbe2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5904103480 0x7f5904199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590bbe2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5904108a70 0x7f590419d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590bbe2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5904112c00 con 0x7f5904108a70 2024-04-23T18:17:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590bbe2700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5904112a80 con 0x7f5904102ac0 2024-04-23T18:17:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590bbe2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5904112d80 con 0x7f5904103480 2024-04-23T18:17:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590a17f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5904108a70 0x7f590419d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590917d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5904103480 0x7f5904199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590a17f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5904108a70 0x7f590419d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51370/0 (socket says 172.21.15.50:51370) 2024-04-23T18:17:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.247+0000 7f590a17f700 1 -- 172.21.15.50:0/267654082 learned_addr learned my addr 172.21.15.50:0/267654082 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.248+0000 7f590997e700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5904102ac0 0x7f59041990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.248+0000 7f590a17f700 1 -- 172.21.15.50:0/267654082 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5904103480 msgr2=0x7f5904199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.248+0000 7f590a17f700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5904103480 0x7f5904199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.248+0000 7f590a17f700 1 -- 172.21.15.50:0/267654082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5904102ac0 msgr2=0x7f59041990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.248+0000 7f590a17f700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5904102ac0 0x7f59041990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.248+0000 7f590a17f700 1 -- 172.21.15.50:0/267654082 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590419e100 con 0x7f5904108a70 2024-04-23T18:17:34.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.248+0000 7f590a17f700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5904108a70 0x7f590419d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00b830 tx=0x7f58fc004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:34.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.248+0000 7f58faffd700 1 -- 172.21.15.50:0/267654082 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc021070 con 0x7f5904108a70 2024-04-23T18:17:34.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.248+0000 7f58faffd700 1 -- 172.21.15.50:0/267654082 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58fc007e30 con 0x7f5904108a70 2024-04-23T18:17:34.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.248+0000 7f590bbe2700 1 -- 172.21.15.50:0/267654082 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f590419e390 con 0x7f5904108a70 2024-04-23T18:17:34.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.248+0000 7f58faffd700 1 -- 172.21.15.50:0/267654082 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc004700 con 0x7f5904108a70 2024-04-23T18:17:34.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.248+0000 7f590bbe2700 1 -- 172.21.15.50:0/267654082 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59041a5c70 con 0x7f5904108a70 2024-04-23T18:17:34.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.249+0000 7f590bbe2700 1 -- 172.21.15.50:0/267654082 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5904069730 con 0x7f5904108a70 2024-04-23T18:17:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.250+0000 7f58faffd700 1 -- 172.21.15.50:0/267654082 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f58fc007980 con 0x7f5904108a70 2024-04-23T18:17:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.251+0000 7f58faffd700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58f00a3cd0 0x7f58f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.251+0000 7f590997e700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58f00a3cd0 0x7f58f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.251+0000 7f58faffd700 1 -- 172.21.15.50:0/267654082 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f58fc00f020 con 0x7f5904108a70 2024-04-23T18:17:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.252+0000 7f590997e700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58f00a3cd0 0x7f58f00a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5900005950 tx=0x7f590000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:34.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.257+0000 7f58faffd700 1 -- 172.21.15.50:0/267654082 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f58fc02a020 con 0x7f5904108a70 2024-04-23T18:17:34.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:34 smithi188 ceph-mon[32694]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-23T18:17:34.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:34 smithi050 ceph-mon[29187]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-23T18:17:34.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.470+0000 7f590bbe2700 1 -- 172.21.15.50:0/267654082 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f590402df80 con 0x7f5904108a70 2024-04-23T18:17:34.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.471+0000 7f58faffd700 1 -- 172.21.15.50:0/267654082 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f58fc09e3a0 con 0x7f5904108a70 2024-04-23T18:17:34.465 INFO:teuthology.orchestra.run.smithi050.stdout:292057776163 2024-04-23T18:17:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.474+0000 7f590bbe2700 1 -- 172.21.15.50:0/267654082 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58f00a3cd0 msgr2=0x7f58f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.474+0000 7f590bbe2700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58f00a3cd0 0x7f58f00a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5900005950 tx=0x7f590000b410 comp rx=0 tx=0).stop 2024-04-23T18:17:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.474+0000 7f590bbe2700 1 -- 172.21.15.50:0/267654082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5904108a70 msgr2=0x7f590419d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.474+0000 7f590bbe2700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5904108a70 0x7f590419d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00b830 tx=0x7f58fc004ce0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.474+0000 7f590bbe2700 1 -- 172.21.15.50:0/267654082 shutdown_connections 2024-04-23T18:17:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.474+0000 7f590bbe2700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5904108a70 0x7f590419d9c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.474+0000 7f590bbe2700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58f00a3cd0 0x7f58f00a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.474+0000 7f590bbe2700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5904102ac0 0x7f59041990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.474+0000 7f590bbe2700 1 --2- 172.21.15.50:0/267654082 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5904103480 0x7f5904199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.474+0000 7f590bbe2700 1 -- 172.21.15.50:0/267654082 >> 172.21.15.50:0/267654082 conn(0x7f5904079bb0 msgr2=0x7f5904101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.474+0000 7f590bbe2700 1 -- 172.21.15.50:0/267654082 shutdown_connections 2024-04-23T18:17:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:34.474+0000 7f590bbe2700 1 -- 172.21.15.50:0/267654082 wait complete. 2024-04-23T18:17:34.834 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318291 got 352187318295 for osd.11 2024-04-23T18:17:34.834 DEBUG:teuthology.parallel:result is None 2024-04-23T18:17:35.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-23T18:17:35.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-23T18:17:35.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-23T18:17:35.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-04-23T18:17:35.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-04-23T18:17:35.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-04-23T18:17:35.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-23T18:17:35.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-23T18:17:35.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-23T18:17:35.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/267654082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T18:17:35.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:35.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:35.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:35.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:35 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-23T18:17:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-23T18:17:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-23T18:17:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-04-23T18:17:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-04-23T18:17:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-04-23T18:17:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-23T18:17:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-23T18:17:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-23T18:17:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/267654082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T18:17:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:35 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:35.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-23T18:17:35.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-23T18:17:35.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-23T18:17:35.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-04-23T18:17:35.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-04-23T18:17:35.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-04-23T18:17:35.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-23T18:17:35.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-23T18:17:35.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-23T18:17:35.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/267654082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T18:17:35.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:35.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:35.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:35.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:35 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:35.666 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776159 got 292057776163 for osd.9 2024-04-23T18:17:35.666 DEBUG:teuthology.parallel:result is None 2024-04-23T18:17:35.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.826+0000 7fd41807e700 1 -- 172.21.15.50:0/3304448922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd410107990 msgr2=0x7fd410107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:35.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.826+0000 7fd41807e700 1 --2- 172.21.15.50:0/3304448922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd410107990 0x7fd410107d70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd40800b0a0 tx=0x7fd40800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:35.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.826+0000 7fd41807e700 1 -- 172.21.15.50:0/3304448922 shutdown_connections 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.826+0000 7fd41807e700 1 --2- 172.21.15.50:0/3304448922 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd410102300 0x7fd41010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.826+0000 7fd41807e700 1 --2- 172.21.15.50:0/3304448922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd410107990 0x7fd410107d70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.826+0000 7fd41807e700 1 --2- 172.21.15.50:0/3304448922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd410101940 0x7fd410101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.826+0000 7fd41807e700 1 -- 172.21.15.50:0/3304448922 >> 172.21.15.50:0/3304448922 conn(0x7fd4100788f0 msgr2=0x7fd410078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.826+0000 7fd41807e700 1 -- 172.21.15.50:0/3304448922 shutdown_connections 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.826+0000 7fd41807e700 1 -- 172.21.15.50:0/3304448922 wait complete. 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd41807e700 1 Processor -- start 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd41807e700 1 -- start start 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd41807e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd410101940 0x7fd41019b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd41807e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd410102300 0x7fd41019b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd41807e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd410107990 0x7fd41019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd41807e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd410112c40 con 0x7fd410107990 2024-04-23T18:17:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd41807e700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd410112ac0 con 0x7fd410102300 2024-04-23T18:17:35.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd41807e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd410112dc0 con 0x7fd410101940 2024-04-23T18:17:35.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd415e1a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd410101940 0x7fd41019b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:35.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd41661b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd410107990 0x7fd41019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:35.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd415619700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd410102300 0x7fd41019b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:35.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd415e1a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd410101940 0x7fd41019b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:40840/0 (socket says 172.21.15.50:40840) 2024-04-23T18:17:35.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd41661b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd410107990 0x7fd41019fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51378/0 (socket says 172.21.15.50:51378) 2024-04-23T18:17:35.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd415e1a700 1 -- 172.21.15.50:0/2400938818 learned_addr learned my addr 172.21.15.50:0/2400938818 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:35.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd415e1a700 1 -- 172.21.15.50:0/2400938818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd410102300 msgr2=0x7fd41019b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:35.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd415e1a700 1 --2- 172.21.15.50:0/2400938818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd410102300 0x7fd41019b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:35.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd415e1a700 1 -- 172.21.15.50:0/2400938818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd410107990 msgr2=0x7fd41019fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:35.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.827+0000 7fd415e1a700 1 --2- 172.21.15.50:0/2400938818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd410107990 0x7fd41019fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:35.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.828+0000 7fd415e1a700 1 -- 172.21.15.50:0/2400938818 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4101a02e0 con 0x7fd410101940 2024-04-23T18:17:35.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.828+0000 7fd415e1a700 1 --2- 172.21.15.50:0/2400938818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd410101940 0x7fd41019b2d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd40800b070 tx=0x7fd408004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:35.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.828+0000 7fd40effd700 1 -- 172.21.15.50:0/2400938818 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd408021070 con 0x7fd410101940 2024-04-23T18:17:35.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.828+0000 7fd41807e700 1 -- 172.21.15.50:0/2400938818 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4101a0570 con 0x7fd410101940 2024-04-23T18:17:35.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.828+0000 7fd40effd700 1 -- 172.21.15.50:0/2400938818 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd408007e30 con 0x7fd410101940 2024-04-23T18:17:35.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.828+0000 7fd40effd700 1 -- 172.21.15.50:0/2400938818 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4080046c0 con 0x7fd410101940 2024-04-23T18:17:35.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.828+0000 7fd41807e700 1 -- 172.21.15.50:0/2400938818 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4101a7e50 con 0x7fd410101940 2024-04-23T18:17:35.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.829+0000 7fd41807e700 1 -- 172.21.15.50:0/2400938818 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd410069730 con 0x7fd410101940 2024-04-23T18:17:35.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.830+0000 7fd40effd700 1 -- 172.21.15.50:0/2400938818 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd408007980 con 0x7fd410101940 2024-04-23T18:17:35.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.831+0000 7fd40effd700 1 --2- 172.21.15.50:0/2400938818 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd3f80a3cd0 0x7fd3f80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:35.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.831+0000 7fd40effd700 1 -- 172.21.15.50:0/2400938818 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fd40800f020 con 0x7fd410101940 2024-04-23T18:17:35.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.831+0000 7fd415619700 1 --2- 172.21.15.50:0/2400938818 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd3f80a3cd0 0x7fd3f80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:35.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.831+0000 7fd415619700 1 --2- 172.21.15.50:0/2400938818 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd3f80a3cd0 0x7fd3f80a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd41010ec30 tx=0x7fd40400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:35.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:35.835+0000 7fd40effd700 1 -- 172.21.15.50:0/2400938818 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd40802a020 con 0x7fd410101940 2024-04-23T18:17:36.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.043+0000 7fd41807e700 1 -- 172.21.15.50:0/2400938818 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fd4101a8020 con 0x7fd410101940 2024-04-23T18:17:36.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.044+0000 7fd40effd700 1 -- 172.21.15.50:0/2400938818 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd40809e6f0 con 0x7fd410101940 2024-04-23T18:17:36.038 INFO:teuthology.orchestra.run.smithi050.stdout:257698037799 2024-04-23T18:17:36.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.046+0000 7fd41807e700 1 -- 172.21.15.50:0/2400938818 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd3f80a3cd0 msgr2=0x7fd3f80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:36.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.046+0000 7fd41807e700 1 --2- 172.21.15.50:0/2400938818 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd3f80a3cd0 0x7fd3f80a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd41010ec30 tx=0x7fd40400b410 comp rx=0 tx=0).stop 2024-04-23T18:17:36.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.046+0000 7fd41807e700 1 -- 172.21.15.50:0/2400938818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd410101940 msgr2=0x7fd41019b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:36.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.046+0000 7fd41807e700 1 --2- 172.21.15.50:0/2400938818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd410101940 0x7fd41019b2d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd40800b070 tx=0x7fd408004d00 comp rx=0 tx=0).stop 2024-04-23T18:17:36.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.046+0000 7fd41807e700 1 -- 172.21.15.50:0/2400938818 shutdown_connections 2024-04-23T18:17:36.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.046+0000 7fd41807e700 1 --2- 172.21.15.50:0/2400938818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd410107990 0x7fd41019fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.046+0000 7fd41807e700 1 --2- 172.21.15.50:0/2400938818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd410101940 0x7fd41019b2d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.046+0000 7fd41807e700 1 --2- 172.21.15.50:0/2400938818 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd3f80a3cd0 0x7fd3f80a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.046+0000 7fd41807e700 1 --2- 172.21.15.50:0/2400938818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd410102300 0x7fd41019b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:36.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.046+0000 7fd41807e700 1 -- 172.21.15.50:0/2400938818 >> 172.21.15.50:0/2400938818 conn(0x7fd4100788f0 msgr2=0x7fd410100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:36.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.046+0000 7fd41807e700 1 -- 172.21.15.50:0/2400938818 shutdown_connections 2024-04-23T18:17:36.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:36.046+0000 7fd41807e700 1 -- 172.21.15.50:0/2400938818 wait complete. 2024-04-23T18:17:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:36 smithi050 ceph-mon[29187]: pgmap v317: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:17:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:36 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-23T18:17:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:36 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-23T18:17:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:36 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-23T18:17:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:36 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-04-23T18:17:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:36 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-04-23T18:17:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:36 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-04-23T18:17:36.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:36 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-23T18:17:36.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:36 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-23T18:17:36.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:36 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-23T18:17:36.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:36 smithi050 ceph-mon[29187]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T18:17:36.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:36 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2400938818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T18:17:36.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:36 smithi177 ceph-mon[32308]: pgmap v317: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:17:36.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:36 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-23T18:17:36.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:36 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-23T18:17:36.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:36 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-23T18:17:36.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:36 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-04-23T18:17:36.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:36 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-04-23T18:17:36.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:36 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-04-23T18:17:36.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:36 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-23T18:17:36.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:36 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-23T18:17:36.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:36 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-23T18:17:36.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:36 smithi177 ceph-mon[32308]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T18:17:36.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:36 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2400938818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T18:17:36.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:36 smithi188 ceph-mon[32694]: pgmap v317: 97 pgs: 97 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:17:36.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:36 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-23T18:17:36.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:36 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-23T18:17:36.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:36 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-23T18:17:36.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:36 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-04-23T18:17:36.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:36 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-04-23T18:17:36.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:36 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-04-23T18:17:36.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:36 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-23T18:17:36.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:36 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-23T18:17:36.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:36 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-23T18:17:36.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:36 smithi188 ceph-mon[32694]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T18:17:36.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:36 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2400938818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T18:17:37.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.160+0000 7f63e0e90700 1 -- 172.21.15.50:0/616268528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63dc103480 msgr2=0x7f63dc10f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:37.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.160+0000 7f63e0e90700 1 --2- 172.21.15.50:0/616268528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63dc103480 0x7f63dc10f8d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f63d400b0a0 tx=0x7f63d400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:37.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.160+0000 7f63e0e90700 1 -- 172.21.15.50:0/616268528 shutdown_connections 2024-04-23T18:17:37.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.160+0000 7f63e0e90700 1 --2- 172.21.15.50:0/616268528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63dc103480 0x7f63dc10f8d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:37.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.160+0000 7f63e0e90700 1 --2- 172.21.15.50:0/616268528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63dc108a70 0x7f63dc108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:37.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.160+0000 7f63e0e90700 1 --2- 172.21.15.50:0/616268528 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63dc102ac0 0x7f63dc102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:37.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.160+0000 7f63e0e90700 1 -- 172.21.15.50:0/616268528 >> 172.21.15.50:0/616268528 conn(0x7f63dc079bb0 msgr2=0x7f63dc079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:37.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.160+0000 7f63e0e90700 1 -- 172.21.15.50:0/616268528 shutdown_connections 2024-04-23T18:17:37.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.160+0000 7f63e0e90700 1 -- 172.21.15.50:0/616268528 wait complete. 2024-04-23T18:17:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.161+0000 7f63e0e90700 1 Processor -- start 2024-04-23T18:17:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.161+0000 7f63e0e90700 1 -- start start 2024-04-23T18:17:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.161+0000 7f63e0e90700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63dc102ac0 0x7f63dc1990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.161+0000 7f63e0e90700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63dc103480 0x7f63dc1995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.161+0000 7f63e0e90700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63dc108a70 0x7f63dc19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.161+0000 7f63e0e90700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc112ca0 con 0x7f63dc103480 2024-04-23T18:17:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.161+0000 7f63e0e90700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc112b20 con 0x7f63dc102ac0 2024-04-23T18:17:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.161+0000 7f63e0e90700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc112e20 con 0x7f63dc108a70 2024-04-23T18:17:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63d9d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63dc103480 0x7f63dc1995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63d9d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63dc103480 0x7f63dc1995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51384/0 (socket says 172.21.15.50:51384) 2024-04-23T18:17:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63d9d9b700 1 -- 172.21.15.50:0/267290690 learned_addr learned my addr 172.21.15.50:0/267290690 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63dad9d700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63dc108a70 0x7f63dc19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63da59c700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63dc102ac0 0x7f63dc1990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63d9d9b700 1 -- 172.21.15.50:0/267290690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63dc108a70 msgr2=0x7f63dc19d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63d9d9b700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63dc108a70 0x7f63dc19d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63d9d9b700 1 -- 172.21.15.50:0/267290690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63dc102ac0 msgr2=0x7f63dc1990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63d9d9b700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63dc102ac0 0x7f63dc1990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63d9d9b700 1 -- 172.21.15.50:0/267290690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc19e070 con 0x7f63dc103480 2024-04-23T18:17:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63d9d9b700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63dc103480 0x7f63dc1995f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f63d000b530 tx=0x7f63d000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63cb7fe700 1 -- 172.21.15.50:0/267290690 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0015020 con 0x7f63dc103480 2024-04-23T18:17:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63e0e90700 1 -- 172.21.15.50:0/267290690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63dc19e360 con 0x7f63dc103480 2024-04-23T18:17:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63cb7fe700 1 -- 172.21.15.50:0/267290690 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d0011ce0 con 0x7f63dc103480 2024-04-23T18:17:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63cb7fe700 1 -- 172.21.15.50:0/267290690 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0010e40 con 0x7f63dc103480 2024-04-23T18:17:37.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.162+0000 7f63e0e90700 1 -- 172.21.15.50:0/267290690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63dc1a5be0 con 0x7f63dc103480 2024-04-23T18:17:37.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.163+0000 7f63e0e90700 1 -- 172.21.15.50:0/267290690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63dc069730 con 0x7f63dc103480 2024-04-23T18:17:37.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.165+0000 7f63cb7fe700 1 -- 172.21.15.50:0/267290690 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f63d0018020 con 0x7f63dc103480 2024-04-23T18:17:37.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.165+0000 7f63cb7fe700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f63c40a3cd0 0x7f63c40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:37.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.165+0000 7f63da59c700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f63c40a3cd0 0x7f63c40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:37.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.165+0000 7f63cb7fe700 1 -- 172.21.15.50:0/267290690 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f63d00d0230 con 0x7f63dc103480 2024-04-23T18:17:37.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.166+0000 7f63da59c700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f63c40a3cd0 0x7f63c40a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f63cc005ea0 tx=0x7f63cc005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:37.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.169+0000 7f63cb7fe700 1 -- 172.21.15.50:0/267290690 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63d001f020 con 0x7f63dc103480 2024-04-23T18:17:37.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.376+0000 7f63e0e90700 1 -- 172.21.15.50:0/267290690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f63dc02df80 con 0x7f63dc103480 2024-04-23T18:17:37.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:37 smithi050 ceph-mon[29187]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T18:17:37.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.377+0000 7f63cb7fe700 1 -- 172.21.15.50:0/267290690 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f63d009cbc0 con 0x7f63dc103480 2024-04-23T18:17:37.371 INFO:teuthology.orchestra.run.smithi050.stdout:55834574922 2024-04-23T18:17:37.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.379+0000 7f63e0e90700 1 -- 172.21.15.50:0/267290690 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f63c40a3cd0 msgr2=0x7f63c40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:37.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.379+0000 7f63e0e90700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f63c40a3cd0 0x7f63c40a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f63cc005ea0 tx=0x7f63cc005e30 comp rx=0 tx=0).stop 2024-04-23T18:17:37.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.379+0000 7f63e0e90700 1 -- 172.21.15.50:0/267290690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63dc103480 msgr2=0x7f63dc1995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:37.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.379+0000 7f63e0e90700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63dc103480 0x7f63dc1995f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f63d000b530 tx=0x7f63d000b840 comp rx=0 tx=0).stop 2024-04-23T18:17:37.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.379+0000 7f63e0e90700 1 -- 172.21.15.50:0/267290690 shutdown_connections 2024-04-23T18:17:37.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.379+0000 7f63e0e90700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63dc108a70 0x7f63dc19d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:37.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.379+0000 7f63e0e90700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f63c40a3cd0 0x7f63c40a6190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:37.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.379+0000 7f63e0e90700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63dc102ac0 0x7f63dc1990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:37.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.379+0000 7f63e0e90700 1 --2- 172.21.15.50:0/267290690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63dc103480 0x7f63dc1995f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:37.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.379+0000 7f63e0e90700 1 -- 172.21.15.50:0/267290690 >> 172.21.15.50:0/267290690 conn(0x7f63dc079bb0 msgr2=0x7f63dc101600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:37.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.379+0000 7f63e0e90700 1 -- 172.21.15.50:0/267290690 shutdown_connections 2024-04-23T18:17:37.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:37.380+0000 7f63e0e90700 1 -- 172.21.15.50:0/267290690 wait complete. 2024-04-23T18:17:37.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:37 smithi177 ceph-mon[32308]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T18:17:37.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:37 smithi188 ceph-mon[32694]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T18:17:37.967 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574918 got 55834574922 for osd.1 2024-04-23T18:17:37.967 DEBUG:teuthology.parallel:result is None 2024-04-23T18:17:38.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:38 smithi050 ceph-mon[29187]: pgmap v320: 97 pgs: 4 peering, 93 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:17:38.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:38 smithi050 ceph-mon[29187]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-23T18:17:38.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:38 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/267290690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T18:17:38.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:38 smithi177 ceph-mon[32308]: pgmap v320: 97 pgs: 4 peering, 93 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:17:38.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:38 smithi177 ceph-mon[32308]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-23T18:17:38.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:38 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/267290690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T18:17:38.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:38 smithi188 ceph-mon[32694]: pgmap v320: 97 pgs: 4 peering, 93 active+clean; 640 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:17:38.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:38 smithi188 ceph-mon[32694]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-23T18:17:38.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:38 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/267290690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T18:17:39.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.449+0000 7fa3e3131700 1 -- 172.21.15.50:0/2664402256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3dc108ab0 msgr2=0x7fa3dc108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:39.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.449+0000 7fa3e3131700 1 --2- 172.21.15.50:0/2664402256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3dc108ab0 0x7fa3dc108e90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009990 tx=0x7fa3d0009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:39.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.449+0000 7fa3e3131700 1 -- 172.21.15.50:0/2664402256 shutdown_connections 2024-04-23T18:17:39.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.449+0000 7fa3e3131700 1 --2- 172.21.15.50:0/2664402256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3dc103480 0x7fa3dc10f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:39.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.449+0000 7fa3e3131700 1 --2- 172.21.15.50:0/2664402256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3dc108ab0 0x7fa3dc108e90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:39.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.449+0000 7fa3e3131700 1 --2- 172.21.15.50:0/2664402256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc102ac0 0x7fa3dc102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:39.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.449+0000 7fa3e3131700 1 -- 172.21.15.50:0/2664402256 >> 172.21.15.50:0/2664402256 conn(0x7fa3dc079bf0 msgr2=0x7fa3dc07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:39.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.449+0000 7fa3e3131700 1 -- 172.21.15.50:0/2664402256 shutdown_connections 2024-04-23T18:17:39.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.449+0000 7fa3e3131700 1 -- 172.21.15.50:0/2664402256 wait complete. 2024-04-23T18:17:39.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.450+0000 7fa3e3131700 1 Processor -- start 2024-04-23T18:17:39.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.450+0000 7fa3e3131700 1 -- start start 2024-04-23T18:17:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.450+0000 7fa3e3131700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3dc102ac0 0x7fa3dc199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.450+0000 7fa3e3131700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3dc103480 0x7fa3dc1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.450+0000 7fa3e3131700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc108ab0 0x7fa3dc19da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.450+0000 7fa3e3131700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa3dc112ca0 con 0x7fa3dc102ac0 2024-04-23T18:17:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.450+0000 7fa3e3131700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa3dc112b20 con 0x7fa3dc108ab0 2024-04-23T18:17:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.450+0000 7fa3e3131700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa3dc112e20 con 0x7fa3dc103480 2024-04-23T18:17:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.450+0000 7fa3e0ecd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3dc102ac0 0x7fa3dc199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.450+0000 7fa3e0ecd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3dc102ac0 0x7fa3dc199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51400/0 (socket says 172.21.15.50:51400) 2024-04-23T18:17:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.450+0000 7fa3e0ecd700 1 -- 172.21.15.50:0/3844843687 learned_addr learned my addr 172.21.15.50:0/3844843687 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3dbfff700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3dc103480 0x7fa3dc1996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3e16ce700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc108ab0 0x7fa3dc19da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3e0ecd700 1 -- 172.21.15.50:0/3844843687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3dc103480 msgr2=0x7fa3dc1996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3e0ecd700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3dc103480 0x7fa3dc1996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3e0ecd700 1 -- 172.21.15.50:0/3844843687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc108ab0 msgr2=0x7fa3dc19da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3e0ecd700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc108ab0 0x7fa3dc19da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3e0ecd700 1 -- 172.21.15.50:0/3844843687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3dc19e180 con 0x7fa3dc102ac0 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3e16ce700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc108ab0 0x7fa3dc19da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3dbfff700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3dc103480 0x7fa3dc1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3e0ecd700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3dc102ac0 0x7fa3dc199170 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0000c00 tx=0x7fa3d00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3d9ffb700 1 -- 172.21.15.50:0/3844843687 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0027070 con 0x7fa3dc102ac0 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3e3131700 1 -- 172.21.15.50:0/3844843687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3dc19e410 con 0x7fa3dc102ac0 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3d9ffb700 1 -- 172.21.15.50:0/3844843687 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3d0003fa0 con 0x7fa3dc102ac0 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3d9ffb700 1 -- 172.21.15.50:0/3844843687 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0022db0 con 0x7fa3dc102ac0 2024-04-23T18:17:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.451+0000 7fa3e3131700 1 -- 172.21.15.50:0/3844843687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3dc1a5cf0 con 0x7fa3dc102ac0 2024-04-23T18:17:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.453+0000 7fa3e3131700 1 -- 172.21.15.50:0/3844843687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3dc1041f0 con 0x7fa3dc102ac0 2024-04-23T18:17:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.455+0000 7fa3d9ffb700 1 -- 172.21.15.50:0/3844843687 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa3d002a070 con 0x7fa3dc102ac0 2024-04-23T18:17:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.456+0000 7fa3d9ffb700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa3cc0a3cd0 0x7fa3cc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:39.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.456+0000 7fa3dbfff700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa3cc0a3cd0 0x7fa3cc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:39.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.456+0000 7fa3d9ffb700 1 -- 172.21.15.50:0/3844843687 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fa3d00d1ad0 con 0x7fa3dc102ac0 2024-04-23T18:17:39.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.456+0000 7fa3dbfff700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa3cc0a3cd0 0x7fa3cc0a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8005950 tx=0x7fa3c80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:39.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.461+0000 7fa3d9ffb700 1 -- 172.21.15.50:0/3844843687 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa3d0031070 con 0x7fa3dc102ac0 2024-04-23T18:17:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.672+0000 7fa3e3131700 1 -- 172.21.15.50:0/3844843687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa3dc02df80 con 0x7fa3dc102ac0 2024-04-23T18:17:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.672+0000 7fa3d9ffb700 1 -- 172.21.15.50:0/3844843687 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa3d0013070 con 0x7fa3dc102ac0 2024-04-23T18:17:39.666 INFO:teuthology.orchestra.run.smithi050.stdout:77309411400 2024-04-23T18:17:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.675+0000 7fa3e3131700 1 -- 172.21.15.50:0/3844843687 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa3cc0a3cd0 msgr2=0x7fa3cc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.675+0000 7fa3e3131700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa3cc0a3cd0 0x7fa3cc0a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8005950 tx=0x7fa3c80058e0 comp rx=0 tx=0).stop 2024-04-23T18:17:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.675+0000 7fa3e3131700 1 -- 172.21.15.50:0/3844843687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3dc102ac0 msgr2=0x7fa3dc199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.675+0000 7fa3e3131700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3dc102ac0 0x7fa3dc199170 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0000c00 tx=0x7fa3d00047b0 comp rx=0 tx=0).stop 2024-04-23T18:17:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.675+0000 7fa3e3131700 1 -- 172.21.15.50:0/3844843687 shutdown_connections 2024-04-23T18:17:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.675+0000 7fa3e3131700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa3cc0a3cd0 0x7fa3cc0a6190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.675+0000 7fa3e3131700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc108ab0 0x7fa3dc19da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.675+0000 7fa3e3131700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3dc102ac0 0x7fa3dc199170 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.675+0000 7fa3e3131700 1 --2- 172.21.15.50:0/3844843687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3dc103480 0x7fa3dc1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.675+0000 7fa3e3131700 1 -- 172.21.15.50:0/3844843687 >> 172.21.15.50:0/3844843687 conn(0x7fa3dc079bf0 msgr2=0x7fa3dc100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.675+0000 7fa3e3131700 1 -- 172.21.15.50:0/3844843687 shutdown_connections 2024-04-23T18:17:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:39.675+0000 7fa3e3131700 1 -- 172.21.15.50:0/3844843687 wait complete. 2024-04-23T18:17:40.190 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:40 smithi050 ceph-mon[29187]: pgmap v321: 97 pgs: 4 peering, 93 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 16 B/s, 1 objects/s recovering 2024-04-23T18:17:40.191 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:40 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3844843687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T18:17:40.505 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411393 got 77309411400 for osd.2 2024-04-23T18:17:40.505 DEBUG:teuthology.parallel:result is None 2024-04-23T18:17:40.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:40 smithi177 ceph-mon[32308]: pgmap v321: 97 pgs: 4 peering, 93 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 16 B/s, 1 objects/s recovering 2024-04-23T18:17:40.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:40 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3844843687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T18:17:40.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:40 smithi188 ceph-mon[32694]: pgmap v321: 97 pgs: 4 peering, 93 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 16 B/s, 1 objects/s recovering 2024-04-23T18:17:40.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:40 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3844843687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T18:17:40.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.930+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2147631992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bf8108a70 msgr2=0x7f5bf8108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.930+0000 7f5bfe8db700 1 --2- 172.21.15.50:0/2147631992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bf8108a70 0x7f5bf8108e50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009990 tx=0x7f5be8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.930+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2147631992 shutdown_connections 2024-04-23T18:17:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.930+0000 7f5bfe8db700 1 --2- 172.21.15.50:0/2147631992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8103480 0x7f5bf810f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.930+0000 7f5bfe8db700 1 --2- 172.21.15.50:0/2147631992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bf8108a70 0x7f5bf8108e50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.930+0000 7f5bfe8db700 1 --2- 172.21.15.50:0/2147631992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf8102ac0 0x7f5bf8102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.930+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2147631992 >> 172.21.15.50:0/2147631992 conn(0x7f5bf8079bb0 msgr2=0x7f5bf8079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.930+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2147631992 shutdown_connections 2024-04-23T18:17:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.930+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2147631992 wait complete. 2024-04-23T18:17:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bfe8db700 1 Processor -- start 2024-04-23T18:17:40.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bfe8db700 1 -- start start 2024-04-23T18:17:40.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bfe8db700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8102ac0 0x7f5bf8199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:40.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bfe8db700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf8103480 0x7f5bf81997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:40.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bfe8db700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bf8108a70 0x7f5bf819db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bfe8db700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf8112ca0 con 0x7f5bf8108a70 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bfe8db700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf8112b20 con 0x7f5bf8103480 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bfe8db700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf8112e20 con 0x7f5bf8102ac0 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bf7fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8102ac0 0x7f5bf8199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bfce78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bf8108a70 0x7f5bf819db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bf7fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8102ac0 0x7f5bf8199270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:40906/0 (socket says 172.21.15.50:40906) 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bf7fff700 1 -- 172.21.15.50:0/2013139631 learned_addr learned my addr 172.21.15.50:0/2013139631 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.931+0000 7f5bfce78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bf8108a70 0x7f5bf819db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51430/0 (socket says 172.21.15.50:51430) 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.932+0000 7f5bf77fe700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf8103480 0x7f5bf81997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.932+0000 7f5bfce78700 1 -- 172.21.15.50:0/2013139631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8102ac0 msgr2=0x7f5bf8199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.932+0000 7f5bfce78700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8102ac0 0x7f5bf8199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.932+0000 7f5bfce78700 1 -- 172.21.15.50:0/2013139631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf8103480 msgr2=0x7f5bf81997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.932+0000 7f5bfce78700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf8103480 0x7f5bf81997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.932+0000 7f5bfce78700 1 -- 172.21.15.50:0/2013139631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf819e280 con 0x7f5bf8108a70 2024-04-23T18:17:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.932+0000 7f5bfce78700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bf8108a70 0x7f5bf819db40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5bf000ab40 tx=0x7f5bf000ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.932+0000 7f5bf57fa700 1 -- 172.21.15.50:0/2013139631 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0017020 con 0x7f5bf8108a70 2024-04-23T18:17:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.932+0000 7f5bf57fa700 1 -- 172.21.15.50:0/2013139631 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf0004bb0 con 0x7f5bf8108a70 2024-04-23T18:17:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.932+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2013139631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf819e570 con 0x7f5bf8108a70 2024-04-23T18:17:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.932+0000 7f5bf57fa700 1 -- 172.21.15.50:0/2013139631 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0005280 con 0x7f5bf8108a70 2024-04-23T18:17:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.932+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2013139631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bf81a5df0 con 0x7f5bf8108a70 2024-04-23T18:17:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.933+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2013139631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bf8069730 con 0x7f5bf8108a70 2024-04-23T18:17:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.934+0000 7f5bf57fa700 1 -- 172.21.15.50:0/2013139631 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5bf000c640 con 0x7f5bf8108a70 2024-04-23T18:17:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.935+0000 7f5bf57fa700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f5be00a3cd0 0x7f5be00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.935+0000 7f5bf7fff700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f5be00a3cd0 0x7f5be00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.935+0000 7f5bf57fa700 1 -- 172.21.15.50:0/2013139631 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f5bf00d0190 con 0x7f5bf8108a70 2024-04-23T18:17:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.935+0000 7f5bf7fff700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f5be00a3cd0 0x7f5be00a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009750 tx=0x7f5be8005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:40.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:40.939+0000 7f5bf57fa700 1 -- 172.21.15.50:0/2013139631 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5bf0021020 con 0x7f5bf8108a70 2024-04-23T18:17:41.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.150+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2013139631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f5bf802df80 con 0x7f5bf8108a70 2024-04-23T18:17:41.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.150+0000 7f5bf57fa700 1 -- 172.21.15.50:0/2013139631 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5bf009caa0 con 0x7f5bf8108a70 2024-04-23T18:17:41.144 INFO:teuthology.orchestra.run.smithi050.stdout:223338299439 2024-04-23T18:17:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.152+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2013139631 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f5be00a3cd0 msgr2=0x7f5be00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.152+0000 7f5bfe8db700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f5be00a3cd0 0x7f5be00a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009750 tx=0x7f5be8005c20 comp rx=0 tx=0).stop 2024-04-23T18:17:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.152+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2013139631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bf8108a70 msgr2=0x7f5bf819db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.152+0000 7f5bfe8db700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bf8108a70 0x7f5bf819db40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5bf000ab40 tx=0x7f5bf000ae50 comp rx=0 tx=0).stop 2024-04-23T18:17:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.152+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2013139631 shutdown_connections 2024-04-23T18:17:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.152+0000 7f5bfe8db700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bf8108a70 0x7f5bf819db40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.152+0000 7f5bfe8db700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8102ac0 0x7f5bf8199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.152+0000 7f5bfe8db700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f5be00a3cd0 0x7f5be00a6190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.152+0000 7f5bfe8db700 1 --2- 172.21.15.50:0/2013139631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf8103480 0x7f5bf81997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.152+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2013139631 >> 172.21.15.50:0/2013139631 conn(0x7f5bf8079bb0 msgr2=0x7f5bf81018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.152+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2013139631 shutdown_connections 2024-04-23T18:17:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:41.152+0000 7f5bfe8db700 1 -- 172.21.15.50:0/2013139631 wait complete. 2024-04-23T18:17:41.325 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037795 got 257698037799 for osd.8 2024-04-23T18:17:41.325 DEBUG:teuthology.parallel:result is None 2024-04-23T18:17:42.062 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299435 got 223338299439 for osd.7 2024-04-23T18:17:42.062 DEBUG:teuthology.parallel:result is None 2024-04-23T18:17:42.414 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:42 smithi050 ceph-mon[29187]: pgmap v322: 97 pgs: 4 peering, 93 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 16 B/s, 1 objects/s recovering 2024-04-23T18:17:42.414 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:42 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2013139631' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T18:17:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:42 smithi177 ceph-mon[32308]: pgmap v322: 97 pgs: 4 peering, 93 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 16 B/s, 1 objects/s recovering 2024-04-23T18:17:42.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:42 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2013139631' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T18:17:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:42 smithi188 ceph-mon[32694]: pgmap v322: 97 pgs: 4 peering, 93 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 16 B/s, 1 objects/s recovering 2024-04-23T18:17:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:42 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2013139631' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T18:17:42.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.693+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1340534052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4108a70 msgr2=0x7fbac4108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:42.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.693+0000 7fbac9e3e700 1 --2- 172.21.15.50:0/1340534052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4108a70 0x7fbac4108e50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009990 tx=0x7fbab4009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:42.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.693+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1340534052 shutdown_connections 2024-04-23T18:17:42.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.693+0000 7fbac9e3e700 1 --2- 172.21.15.50:0/1340534052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac406bb30 0x7fbac410f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:42.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.693+0000 7fbac9e3e700 1 --2- 172.21.15.50:0/1340534052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4108a70 0x7fbac4108e50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:42.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.693+0000 7fbac9e3e700 1 --2- 172.21.15.50:0/1340534052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbac406b170 0x7fbac406b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:42.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.693+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1340534052 >> 172.21.15.50:0/1340534052 conn(0x7fbac40777b0 msgr2=0x7fbac4077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:42.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.694+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1340534052 shutdown_connections 2024-04-23T18:17:42.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.694+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1340534052 wait complete. 2024-04-23T18:17:42.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.694+0000 7fbac9e3e700 1 Processor -- start 2024-04-23T18:17:42.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.694+0000 7fbac9e3e700 1 -- start start 2024-04-23T18:17:42.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.694+0000 7fbac9e3e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac406b170 0x7fbac4199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:42.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.694+0000 7fbac9e3e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac406bb30 0x7fbac4199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:42.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.694+0000 7fbac9e3e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbac4108a70 0x7fbac419d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.694+0000 7fbac9e3e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbac4112c40 con 0x7fbac406bb30 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.694+0000 7fbac9e3e700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbac4112ac0 con 0x7fbac406b170 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.694+0000 7fbac9e3e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbac4112dc0 con 0x7fbac4108a70 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac406bb30 0x7fbac4199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac406bb30 0x7fbac4199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46462/0 (socket says 172.21.15.50:46462) 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac2ffd700 1 -- 172.21.15.50:0/1495505298 learned_addr learned my addr 172.21.15.50:0/1495505298 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac3fff700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbac4108a70 0x7fbac419d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac37fe700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac406b170 0x7fbac4199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac2ffd700 1 -- 172.21.15.50:0/1495505298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbac4108a70 msgr2=0x7fbac419d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac2ffd700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbac4108a70 0x7fbac419d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac2ffd700 1 -- 172.21.15.50:0/1495505298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac406b170 msgr2=0x7fbac4199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac2ffd700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac406b170 0x7fbac4199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac2ffd700 1 -- 172.21.15.50:0/1495505298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac419e0f0 con 0x7fbac406bb30 2024-04-23T18:17:42.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac2ffd700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac406bb30 0x7fbac4199670 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbab800d950 tx=0x7fbab800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:42.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac0ff9700 1 -- 172.21.15.50:0/1495505298 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab8016070 con 0x7fbac406bb30 2024-04-23T18:17:42.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac0ff9700 1 -- 172.21.15.50:0/1495505298 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbab800fd40 con 0x7fbac406bb30 2024-04-23T18:17:42.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1495505298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac419e3e0 con 0x7fbac406bb30 2024-04-23T18:17:42.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac0ff9700 1 -- 172.21.15.50:0/1495505298 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab800e3b0 con 0x7fbac406bb30 2024-04-23T18:17:42.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.695+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1495505298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac41a5c60 con 0x7fbac406bb30 2024-04-23T18:17:42.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.696+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1495505298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac41041a0 con 0x7fbac406bb30 2024-04-23T18:17:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.698+0000 7fbac0ff9700 1 -- 172.21.15.50:0/1495505298 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbab8019070 con 0x7fbac406bb30 2024-04-23T18:17:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.698+0000 7fbac0ff9700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fbaac0a3cd0 0x7fbaac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.698+0000 7fbac37fe700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fbaac0a3cd0 0x7fbaac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.699+0000 7fbac0ff9700 1 -- 172.21.15.50:0/1495505298 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fbab80cffc0 con 0x7fbac406bb30 2024-04-23T18:17:42.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.699+0000 7fbac37fe700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fbaac0a3cd0 0x7fbaac0a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbab400b5c0 tx=0x7fbab4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:42.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.702+0000 7fbac0ff9700 1 -- 172.21.15.50:0/1495505298 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbab809c840 con 0x7fbac406bb30 2024-04-23T18:17:42.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.915+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1495505298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fbac402df80 con 0x7fbac406bb30 2024-04-23T18:17:42.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.915+0000 7fbac0ff9700 1 -- 172.21.15.50:0/1495505298 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbab809c140 con 0x7fbac406bb30 2024-04-23T18:17:42.909 INFO:teuthology.orchestra.run.smithi050.stdout:141733920829 2024-04-23T18:17:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.917+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1495505298 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fbaac0a3cd0 msgr2=0x7fbaac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.917+0000 7fbac9e3e700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fbaac0a3cd0 0x7fbaac0a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbab400b5c0 tx=0x7fbab4005960 comp rx=0 tx=0).stop 2024-04-23T18:17:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.917+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1495505298 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac406bb30 msgr2=0x7fbac4199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.917+0000 7fbac9e3e700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac406bb30 0x7fbac4199670 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbab800d950 tx=0x7fbab800dc60 comp rx=0 tx=0).stop 2024-04-23T18:17:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.918+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1495505298 shutdown_connections 2024-04-23T18:17:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.918+0000 7fbac9e3e700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbac4108a70 0x7fbac419d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.918+0000 7fbac9e3e700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fbaac0a3cd0 0x7fbaac0a6190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.918+0000 7fbac9e3e700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac406b170 0x7fbac4199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.918+0000 7fbac9e3e700 1 --2- 172.21.15.50:0/1495505298 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac406bb30 0x7fbac4199670 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.918+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1495505298 >> 172.21.15.50:0/1495505298 conn(0x7fbac40777b0 msgr2=0x7fbac4100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.918+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1495505298 shutdown_connections 2024-04-23T18:17:42.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:42.918+0000 7fbac9e3e700 1 -- 172.21.15.50:0/1495505298 wait complete. 2024-04-23T18:17:43.070 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738441 got 34359738447 for osd.0 2024-04-23T18:17:43.070 DEBUG:teuthology.parallel:result is None 2024-04-23T18:17:43.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:43 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1495505298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T18:17:43.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:43 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1495505298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T18:17:43.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:43 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1495505298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T18:17:43.712 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920823 got 141733920829 for osd.4 2024-04-23T18:17:43.712 DEBUG:teuthology.parallel:result is None 2024-04-23T18:17:44.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.112+0000 7f850e77f700 1 -- 172.21.15.50:0/1339467629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85081078a0 msgr2=0x7f8508107c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:44.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.112+0000 7f850e77f700 1 --2- 172.21.15.50:0/1339467629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85081078a0 0x7f8508107c80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009990 tx=0x7f84f8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:44.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.112+0000 7f850e77f700 1 -- 172.21.15.50:0/1339467629 shutdown_connections 2024-04-23T18:17:44.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.112+0000 7f850e77f700 1 --2- 172.21.15.50:0/1339467629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8508106c80 0x7f850810e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:44.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.112+0000 7f850e77f700 1 --2- 172.21.15.50:0/1339467629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85081078a0 0x7f8508107c80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:44.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.112+0000 7f850e77f700 1 --2- 172.21.15.50:0/1339467629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8508108250 0x7f85081065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:44.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.112+0000 7f850e77f700 1 -- 172.21.15.50:0/1339467629 >> 172.21.15.50:0/1339467629 conn(0x7f8508077b40 msgr2=0x7f85080feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:44.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.112+0000 7f850e77f700 1 -- 172.21.15.50:0/1339467629 shutdown_connections 2024-04-23T18:17:44.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.112+0000 7f850e77f700 1 -- 172.21.15.50:0/1339467629 wait complete. 2024-04-23T18:17:44.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.113+0000 7f850e77f700 1 Processor -- start 2024-04-23T18:17:44.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.113+0000 7f850e77f700 1 -- start start 2024-04-23T18:17:44.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.113+0000 7f850e77f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8508106c80 0x7f8508102630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:44.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.113+0000 7f850e77f700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85081078a0 0x7f8508102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.113+0000 7f850e77f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8508108250 0x7f85081030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.113+0000 7f850e77f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8508111b20 con 0x7f8508108250 2024-04-23T18:17:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.113+0000 7f850e77f700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f85081119a0 con 0x7f85081078a0 2024-04-23T18:17:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.113+0000 7f850e77f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8508111ca0 con 0x7f8508106c80 2024-04-23T18:17:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f850cd1c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8508108250 0x7f85081030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f8507fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8508106c80 0x7f8508102630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f850cd1c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8508108250 0x7f85081030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46476/0 (socket says 172.21.15.50:46476) 2024-04-23T18:17:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f850cd1c700 1 -- 172.21.15.50:0/4013972484 learned_addr learned my addr 172.21.15.50:0/4013972484 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f85077fe700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85081078a0 0x7f8508102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f850cd1c700 1 -- 172.21.15.50:0/4013972484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8508106c80 msgr2=0x7f8508102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f850cd1c700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8508106c80 0x7f8508102630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f850cd1c700 1 -- 172.21.15.50:0/4013972484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85081078a0 msgr2=0x7f8508102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f850cd1c700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85081078a0 0x7f8508102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f850cd1c700 1 -- 172.21.15.50:0/4013972484 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85081a76b0 con 0x7f8508108250 2024-04-23T18:17:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f850cd1c700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8508108250 0x7f85081030b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8500007ee0 tx=0x7f850000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f85057fa700 1 -- 172.21.15.50:0/4013972484 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8500004bb0 con 0x7f8508108250 2024-04-23T18:17:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f850e77f700 1 -- 172.21.15.50:0/4013972484 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85081a79a0 con 0x7f8508108250 2024-04-23T18:17:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f85057fa700 1 -- 172.21.15.50:0/4013972484 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8500004d50 con 0x7f8508108250 2024-04-23T18:17:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f85057fa700 1 -- 172.21.15.50:0/4013972484 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8500005920 con 0x7f8508108250 2024-04-23T18:17:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.114+0000 7f850e77f700 1 -- 172.21.15.50:0/4013972484 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85081a7e90 con 0x7f8508108250 2024-04-23T18:17:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.115+0000 7f850e77f700 1 -- 172.21.15.50:0/4013972484 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8508100f40 con 0x7f8508108250 2024-04-23T18:17:44.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.116+0000 7f85057fa700 1 -- 172.21.15.50:0/4013972484 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f850001a020 con 0x7f8508108250 2024-04-23T18:17:44.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.117+0000 7f85057fa700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f84f00a3cd0 0x7f84f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:44.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.117+0000 7f8507fff700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f84f00a3cd0 0x7f84f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:44.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.117+0000 7f85057fa700 1 -- 172.21.15.50:0/4013972484 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f85000cfc40 con 0x7f8508108250 2024-04-23T18:17:44.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.117+0000 7f8507fff700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f84f00a3cd0 0x7f84f00a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f84f80098e0 tx=0x7f84f8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:44.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.122+0000 7f85057fa700 1 -- 172.21.15.50:0/4013972484 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f85000d6080 con 0x7f8508108250 2024-04-23T18:17:44.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:44 smithi050 ceph-mon[29187]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s; 143 B/s, 1 objects/s recovering 2024-04-23T18:17:44.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:44 smithi050 ceph-mon[29187]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-04-23T18:17:44.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:44 smithi050 ceph-mon[29187]: Cluster is now healthy 2024-04-23T18:17:44.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.333+0000 7f850e77f700 1 -- 172.21.15.50:0/4013972484 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f85081a8180 con 0x7f8508108250 2024-04-23T18:17:44.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.334+0000 7f85057fa700 1 -- 172.21.15.50:0/4013972484 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f8500010070 con 0x7f8508108250 2024-04-23T18:17:44.328 INFO:teuthology.orchestra.run.smithi050.stdout:317827579936 2024-04-23T18:17:44.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.336+0000 7f850e77f700 1 -- 172.21.15.50:0/4013972484 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f84f00a3cd0 msgr2=0x7f84f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:44.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.336+0000 7f850e77f700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f84f00a3cd0 0x7f84f00a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f84f80098e0 tx=0x7f84f8005960 comp rx=0 tx=0).stop 2024-04-23T18:17:44.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.336+0000 7f850e77f700 1 -- 172.21.15.50:0/4013972484 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8508108250 msgr2=0x7f85081030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:44.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.336+0000 7f850e77f700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8508108250 0x7f85081030b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8500007ee0 tx=0x7f850000d3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:44.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.336+0000 7f850e77f700 1 -- 172.21.15.50:0/4013972484 shutdown_connections 2024-04-23T18:17:44.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.336+0000 7f850e77f700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8508108250 0x7f85081030b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:44.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.336+0000 7f850e77f700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8508106c80 0x7f8508102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:44.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.336+0000 7f850e77f700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f84f00a3cd0 0x7f84f00a6190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:44.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.336+0000 7f850e77f700 1 --2- 172.21.15.50:0/4013972484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85081078a0 0x7f8508102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:44.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.336+0000 7f850e77f700 1 -- 172.21.15.50:0/4013972484 >> 172.21.15.50:0/4013972484 conn(0x7f8508077b40 msgr2=0x7f8508101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:44.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.336+0000 7f850e77f700 1 -- 172.21.15.50:0/4013972484 shutdown_connections 2024-04-23T18:17:44.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:44.336+0000 7f850e77f700 1 -- 172.21.15.50:0/4013972484 wait complete. 2024-04-23T18:17:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:44 smithi177 ceph-mon[32308]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s; 143 B/s, 1 objects/s recovering 2024-04-23T18:17:44.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:44 smithi177 ceph-mon[32308]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-04-23T18:17:44.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:44 smithi177 ceph-mon[32308]: Cluster is now healthy 2024-04-23T18:17:44.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:44 smithi188 ceph-mon[32694]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s; 143 B/s, 1 objects/s recovering 2024-04-23T18:17:44.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:44 smithi188 ceph-mon[32694]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-04-23T18:17:44.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:44 smithi188 ceph-mon[32694]: Cluster is now healthy 2024-04-23T18:17:45.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:45 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4013972484' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T18:17:45.411 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579931 got 317827579936 for osd.10 2024-04-23T18:17:45.412 DEBUG:teuthology.parallel:result is None 2024-04-23T18:17:45.412 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T18:17:45.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T18:17:45.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:45 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4013972484' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T18:17:45.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:45 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4013972484' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T18:17:45.680 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:46.439 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:46 smithi177 ceph-mon[32308]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 121 B/s, 1 objects/s recovering 2024-04-23T18:17:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:46 smithi050 ceph-mon[29187]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 121 B/s, 1 objects/s recovering 2024-04-23T18:17:46.481 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:46 smithi188 ceph-mon[32694]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 121 B/s, 1 objects/s recovering 2024-04-23T18:17:47.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.265+0000 7f58facfc700 1 -- 172.21.15.50:0/522465811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58f4101940 msgr2=0x7f58f4101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:47.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.265+0000 7f58facfc700 1 --2- 172.21.15.50:0/522465811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58f4101940 0x7f58f4101dc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009990 tx=0x7f58e8009ca0 comp rx=0 tx=0).stop 2024-04-23T18:17:47.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.265+0000 7f58facfc700 1 -- 172.21.15.50:0/522465811 shutdown_connections 2024-04-23T18:17:47.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.265+0000 7f58facfc700 1 --2- 172.21.15.50:0/522465811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58f4102300 0x7f58f410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:47.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.265+0000 7f58facfc700 1 --2- 172.21.15.50:0/522465811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58f4107990 0x7f58f4107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:47.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.265+0000 7f58facfc700 1 --2- 172.21.15.50:0/522465811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58f4101940 0x7f58f4101dc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:47.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.265+0000 7f58facfc700 1 -- 172.21.15.50:0/522465811 >> 172.21.15.50:0/522465811 conn(0x7f58f40788f0 msgr2=0x7f58f4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:47.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.265+0000 7f58facfc700 1 -- 172.21.15.50:0/522465811 shutdown_connections 2024-04-23T18:17:47.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.265+0000 7f58facfc700 1 -- 172.21.15.50:0/522465811 wait complete. 2024-04-23T18:17:47.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.266+0000 7f58facfc700 1 Processor -- start 2024-04-23T18:17:47.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.266+0000 7f58facfc700 1 -- start start 2024-04-23T18:17:47.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.266+0000 7f58facfc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58f4101940 0x7f58f419b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:47.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.266+0000 7f58facfc700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58f4102300 0x7f58f419b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:47.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.266+0000 7f58facfc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58f4107990 0x7f58f419fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:47.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.266+0000 7f58facfc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f58f4111b40 con 0x7f58f4101940 2024-04-23T18:17:47.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.266+0000 7f58facfc700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f58f41119c0 con 0x7f58f4102300 2024-04-23T18:17:47.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.266+0000 7f58facfc700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f58f4111cc0 con 0x7f58f4107990 2024-04-23T18:17:47.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.266+0000 7f58f8a98700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58f4101940 0x7f58f419b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:47.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.266+0000 7f58f8a98700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58f4101940 0x7f58f419b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46492/0 (socket says 172.21.15.50:46492) 2024-04-23T18:17:47.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.266+0000 7f58f8a98700 1 -- 172.21.15.50:0/3508290678 learned_addr learned my addr 172.21.15.50:0/3508290678 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:47.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.266+0000 7f58f3fff700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58f4102300 0x7f58f419b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58f8a98700 1 -- 172.21.15.50:0/3508290678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58f4107990 msgr2=0x7f58f419fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58f9299700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58f4107990 0x7f58f419fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58f8a98700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58f4107990 0x7f58f419fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58f8a98700 1 -- 172.21.15.50:0/3508290678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58f4102300 msgr2=0x7f58f419b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58f8a98700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58f4102300 0x7f58f419b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58f8a98700 1 -- 172.21.15.50:0/3508290678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f41a0300 con 0x7f58f4101940 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58f3fff700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58f4102300 0x7f58f419b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58f8a98700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58f4101940 0x7f58f419b2f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f58e400c9b0 tx=0x7f58e400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58f1ffb700 1 -- 172.21.15.50:0/3508290678 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e4007b00 con 0x7f58f4101940 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58facfc700 1 -- 172.21.15.50:0/3508290678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f41a05f0 con 0x7f58f4101940 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58f1ffb700 1 -- 172.21.15.50:0/3508290678 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58e4007ca0 con 0x7f58f4101940 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58f1ffb700 1 -- 172.21.15.50:0/3508290678 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e4018730 con 0x7f58f4101940 2024-04-23T18:17:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.267+0000 7f58facfc700 1 -- 172.21.15.50:0/3508290678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f41a7e70 con 0x7f58f4101940 2024-04-23T18:17:47.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.269+0000 7f58facfc700 1 -- 172.21.15.50:0/3508290678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f4069730 con 0x7f58f4101940 2024-04-23T18:17:47.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.269+0000 7f58f1ffb700 1 -- 172.21.15.50:0/3508290678 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f58e400f450 con 0x7f58f4101940 2024-04-23T18:17:47.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.270+0000 7f58f1ffb700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58dc0a3d50 0x7f58dc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:47.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.270+0000 7f58f3fff700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58dc0a3d50 0x7f58dc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:47.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.270+0000 7f58f1ffb700 1 -- 172.21.15.50:0/3508290678 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f58e40d04f0 con 0x7f58f4101940 2024-04-23T18:17:47.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.271+0000 7f58f3fff700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58dc0a3d50 0x7f58dc0a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009750 tx=0x7f58e8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:47.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.275+0000 7f58f1ffb700 1 -- 172.21.15.50:0/3508290678 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f58e4014030 con 0x7f58f4101940 2024-04-23T18:17:47.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.476+0000 7f58facfc700 1 -- 172.21.15.50:0/3508290678 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f58f406ae30 con 0x7f58dc0a3d50 2024-04-23T18:17:47.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.484+0000 7f58f1ffb700 1 -- 172.21.15.50:0/3508290678 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258728 (secure 0 0 0) 0x7f58f406ae30 con 0x7f58dc0a3d50 2024-04-23T18:17:47.484 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:17:47.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.489+0000 7f58facfc700 1 -- 172.21.15.50:0/3508290678 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58dc0a3d50 msgr2=0x7f58dc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.489+0000 7f58facfc700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58dc0a3d50 0x7f58dc0a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009750 tx=0x7f58e8005fd0 comp rx=0 tx=0).stop 2024-04-23T18:17:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.489+0000 7f58facfc700 1 -- 172.21.15.50:0/3508290678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58f4101940 msgr2=0x7f58f419b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.489+0000 7f58facfc700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58f4101940 0x7f58f419b2f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f58e400c9b0 tx=0x7f58e400cd70 comp rx=0 tx=0).stop 2024-04-23T18:17:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.489+0000 7f58facfc700 1 -- 172.21.15.50:0/3508290678 shutdown_connections 2024-04-23T18:17:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.489+0000 7f58facfc700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58f4107990 0x7f58f419fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.489+0000 7f58facfc700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58f4101940 0x7f58f419b2f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.489+0000 7f58facfc700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f58dc0a3d50 0x7f58dc0a6210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.489+0000 7f58facfc700 1 --2- 172.21.15.50:0/3508290678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58f4102300 0x7f58f419b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.489+0000 7f58facfc700 1 -- 172.21.15.50:0/3508290678 >> 172.21.15.50:0/3508290678 conn(0x7f58f40788f0 msgr2=0x7f58f406a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.489+0000 7f58facfc700 1 -- 172.21.15.50:0/3508290678 shutdown_connections 2024-04-23T18:17:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:47.489+0000 7f58facfc700 1 -- 172.21.15.50:0/3508290678 wait complete. 2024-04-23T18:17:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-23T18:17:48.101 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-04-23T18:17:46.519040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655017,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":547,"num_read_kb":394,"num_write":341,"num_write_kb":3050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181883,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":276,"ondisk_log_size":276,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":391892,"kb_used_data":13548,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549932,"statfs":{"total":1151940427776,"available":1151539130368,"internally_reserved":0,"allocated":13873152,"data_stored":7577631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":387233082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":28,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.414396"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108834+0000","last_change":"2024-04-23T18:16:33.987102+0000","last_active":"2024-04-23T18:17:36.108834+0000","last_peered":"2024-04-23T18:17:36.108834+0000","last_clean":"2024-04-23T18:17:36.108834+0000","last_became_active":"2024-04-23T18:16:33.986716+0000","last_became_peered":"2024-04-23T18:16:33.986716+0000","last_unstale":"2024-04-23T18:17:36.108834+0000","last_undegraded":"2024-04-23T18:17:36.108834+0000","last_fullsized":"2024-04-23T18:17:36.108834+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:41.895219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106901+0000","last_change":"2024-04-23T18:16:33.995703+0000","last_active":"2024-04-23T18:17:36.106901+0000","last_peered":"2024-04-23T18:17:36.106901+0000","last_clean":"2024-04-23T18:17:36.106901+0000","last_became_active":"2024-04-23T18:16:33.995242+0000","last_became_peered":"2024-04-23T18:16:33.995242+0000","last_unstale":"2024-04-23T18:17:36.106901+0000","last_undegraded":"2024-04-23T18:17:36.106901+0000","last_fullsized":"2024-04-23T18:17:36.106901+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:57.813822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106713+0000","last_change":"2024-04-23T18:16:34.000864+0000","last_active":"2024-04-23T18:17:36.106713+0000","last_peered":"2024-04-23T18:17:36.106713+0000","last_clean":"2024-04-23T18:17:36.106713+0000","last_became_active":"2024-04-23T18:16:33.998934+0000","last_became_peered":"2024-04-23T18:16:33.998934+0000","last_unstale":"2024-04-23T18:17:36.106713+0000","last_undegraded":"2024-04-23T18:17:36.106713+0000","last_fullsized":"2024-04-23T18:17:36.106713+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:47:29.481697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212783+0000","last_change":"2024-04-23T18:16:33.989351+0000","last_active":"2024-04-23T18:17:36.212783+0000","last_peered":"2024-04-23T18:17:36.212783+0000","last_clean":"2024-04-23T18:17:36.212783+0000","last_became_active":"2024-04-23T18:16:33.989030+0000","last_became_peered":"2024-04-23T18:16:33.989030+0000","last_unstale":"2024-04-23T18:17:36.212783+0000","last_undegraded":"2024-04-23T18:17:36.212783+0000","last_fullsized":"2024-04-23T18:17:36.212783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:29:03.023675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.112341+0000","last_change":"2024-04-23T18:16:33.996062+0000","last_active":"2024-04-23T18:17:36.112341+0000","last_peered":"2024-04-23T18:17:36.112341+0000","last_clean":"2024-04-23T18:17:36.112341+0000","last_became_active":"2024-04-23T18:16:33.995732+0000","last_became_peered":"2024-04-23T18:16:33.995732+0000","last_unstale":"2024-04-23T18:17:36.112341+0000","last_undegraded":"2024-04-23T18:17:36.112341+0000","last_fullsized":"2024-04-23T18:17:36.112341+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:04.175580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212478+0000","last_change":"2024-04-23T18:16:33.989895+0000","last_active":"2024-04-23T18:17:36.212478+0000","last_peered":"2024-04-23T18:17:36.212478+0000","last_clean":"2024-04-23T18:17:36.212478+0000","last_became_active":"2024-04-23T18:16:33.989547+0000","last_became_peered":"2024-04-23T18:16:33.989547+0000","last_unstale":"2024-04-23T18:17:36.212478+0000","last_undegraded":"2024-04-23T18:17:36.212478+0000","last_fullsized":"2024-04-23T18:17:36.212478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:12:37.317974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108649+0000","last_change":"2024-04-23T18:16:33.999636+0000","last_active":"2024-04-23T18:17:36.108649+0000","last_peered":"2024-04-23T18:17:36.108649+0000","last_clean":"2024-04-23T18:17:36.108649+0000","last_became_active":"2024-04-23T18:16:33.999282+0000","last_became_peered":"2024-04-23T18:16:33.999282+0000","last_unstale":"2024-04-23T18:17:36.108649+0000","last_undegraded":"2024-04-23T18:17:36.108649+0000","last_fullsized":"2024-04-23T18:17:36.108649+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:28.841392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234476+0000","last_change":"2024-04-23T18:16:33.996602+0000","last_active":"2024-04-23T18:17:36.234476+0000","last_peered":"2024-04-23T18:17:36.234476+0000","last_clean":"2024-04-23T18:17:36.234476+0000","last_became_active":"2024-04-23T18:16:33.994401+0000","last_became_peered":"2024-04-23T18:16:33.994401+0000","last_unstale":"2024-04-23T18:17:36.234476+0000","last_undegraded":"2024-04-23T18:17:36.234476+0000","last_fullsized":"2024-04-23T18:17:36.234476+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:31:54.553537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109862+0000","last_change":"2024-04-23T18:16:33.994763+0000","last_active":"2024-04-23T18:17:36.109862+0000","last_peered":"2024-04-23T18:17:36.109862+0000","last_clean":"2024-04-23T18:17:36.109862+0000","last_became_active":"2024-04-23T18:16:33.994408+0000","last_became_peered":"2024-04-23T18:16:33.994408+0000","last_unstale":"2024-04-23T18:17:36.109862+0000","last_undegraded":"2024-04-23T18:17:36.109862+0000","last_fullsized":"2024-04-23T18:17:36.109862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:02.706160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.213087+0000","last_change":"2024-04-23T18:16:33.993435+0000","last_active":"2024-04-23T18:17:36.213087+0000","last_peered":"2024-04-23T18:17:36.213087+0000","last_clean":"2024-04-23T18:17:36.213087+0000","last_became_active":"2024-04-23T18:16:33.993108+0000","last_became_peered":"2024-04-23T18:16:33.993108+0000","last_unstale":"2024-04-23T18:17:36.213087+0000","last_undegraded":"2024-04-23T18:17:36.213087+0000","last_fullsized":"2024-04-23T18:17:36.213087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:28:32.584896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108998+0000","last_change":"2024-04-23T18:16:33.997526+0000","last_active":"2024-04-23T18:17:36.108998+0000","last_peered":"2024-04-23T18:17:36.108998+0000","last_clean":"2024-04-23T18:17:36.108998+0000","last_became_active":"2024-04-23T18:16:33.997189+0000","last_became_peered":"2024-04-23T18:16:33.997189+0000","last_unstale":"2024-04-23T18:17:36.108998+0000","last_undegraded":"2024-04-23T18:17:36.108998+0000","last_fullsized":"2024-04-23T18:17:36.108998+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:11:41.208261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212256+0000","last_change":"2024-04-23T18:16:33.989775+0000","last_active":"2024-04-23T18:17:36.212256+0000","last_peered":"2024-04-23T18:17:36.212256+0000","last_clean":"2024-04-23T18:17:36.212256+0000","last_became_active":"2024-04-23T18:16:33.989382+0000","last_became_peered":"2024-04-23T18:16:33.989382+0000","last_unstale":"2024-04-23T18:17:36.212256+0000","last_undegraded":"2024-04-23T18:17:36.212256+0000","last_fullsized":"2024-04-23T18:17:36.212256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:20:17.756258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.105971+0000","last_change":"2024-04-23T18:16:34.005871+0000","last_active":"2024-04-23T18:17:36.105971+0000","last_peered":"2024-04-23T18:17:36.105971+0000","last_clean":"2024-04-23T18:17:36.105971+0000","last_became_active":"2024-04-23T18:16:34.005218+0000","last_became_peered":"2024-04-23T18:16:34.005218+0000","last_unstale":"2024-04-23T18:17:36.105971+0000","last_undegraded":"2024-04-23T18:17:36.105971+0000","last_fullsized":"2024-04-23T18:17:36.105971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:30:03.864306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106475+0000","last_change":"2024-04-23T18:16:33.995054+0000","last_active":"2024-04-23T18:17:36.106475+0000","last_peered":"2024-04-23T18:17:36.106475+0000","last_clean":"2024-04-23T18:17:36.106475+0000","last_became_active":"2024-04-23T18:16:33.994389+0000","last_became_peered":"2024-04-23T18:16:33.994389+0000","last_unstale":"2024-04-23T18:17:36.106475+0000","last_undegraded":"2024-04-23T18:17:36.106475+0000","last_fullsized":"2024-04-23T18:17:36.106475+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:27.344616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106378+0000","last_change":"2024-04-23T18:16:33.984490+0000","last_active":"2024-04-23T18:17:36.106378+0000","last_peered":"2024-04-23T18:17:36.106378+0000","last_clean":"2024-04-23T18:17:36.106378+0000","last_became_active":"2024-04-23T18:16:33.984071+0000","last_became_peered":"2024-04-23T18:16:33.984071+0000","last_unstale":"2024-04-23T18:17:36.106378+0000","last_undegraded":"2024-04-23T18:17:36.106378+0000","last_fullsized":"2024-04-23T18:17:36.106378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:23.589290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109075+0000","last_change":"2024-04-23T18:16:34.002163+0000","last_active":"2024-04-23T18:17:36.109075+0000","last_peered":"2024-04-23T18:17:36.109075+0000","last_clean":"2024-04-23T18:17:36.109075+0000","last_became_active":"2024-04-23T18:16:34.001764+0000","last_became_peered":"2024-04-23T18:16:34.001764+0000","last_unstale":"2024-04-23T18:17:36.109075+0000","last_undegraded":"2024-04-23T18:17:36.109075+0000","last_fullsized":"2024-04-23T18:17:36.109075+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:16:03.616358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108906+0000","last_change":"2024-04-23T18:16:33.987586+0000","last_active":"2024-04-23T18:17:36.108906+0000","last_peered":"2024-04-23T18:17:36.108906+0000","last_clean":"2024-04-23T18:17:36.108906+0000","last_became_active":"2024-04-23T18:16:33.987266+0000","last_became_peered":"2024-04-23T18:16:33.987266+0000","last_unstale":"2024-04-23T18:17:36.108906+0000","last_undegraded":"2024-04-23T18:17:36.108906+0000","last_fullsized":"2024-04-23T18:17:36.108906+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:41.610529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108628+0000","last_change":"2024-04-23T18:16:33.992938+0000","last_active":"2024-04-23T18:17:36.108628+0000","last_peered":"2024-04-23T18:17:36.108628+0000","last_clean":"2024-04-23T18:17:36.108628+0000","last_became_active":"2024-04-23T18:16:33.990441+0000","last_became_peered":"2024-04-23T18:16:33.990441+0000","last_unstale":"2024-04-23T18:17:36.108628+0000","last_undegraded":"2024-04-23T18:17:36.108628+0000","last_fullsized":"2024-04-23T18:17:36.108628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:10.689333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111521+0000","last_change":"2024-04-23T18:16:33.994844+0000","last_active":"2024-04-23T18:17:36.111521+0000","last_peered":"2024-04-23T18:17:36.111521+0000","last_clean":"2024-04-23T18:17:36.111521+0000","last_became_active":"2024-04-23T18:16:33.994505+0000","last_became_peered":"2024-04-23T18:16:33.994505+0000","last_unstale":"2024-04-23T18:17:36.111521+0000","last_undegraded":"2024-04-23T18:17:36.111521+0000","last_fullsized":"2024-04-23T18:17:36.111521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:32:33.445266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108539+0000","last_change":"2024-04-23T18:16:32.958841+0000","last_active":"2024-04-23T18:17:36.108539+0000","last_peered":"2024-04-23T18:17:36.108539+0000","last_clean":"2024-04-23T18:17:36.108539+0000","last_became_active":"2024-04-23T18:16:32.958529+0000","last_became_peered":"2024-04-23T18:16:32.958529+0000","last_unstale":"2024-04-23T18:17:36.108539+0000","last_undegraded":"2024-04-23T18:17:36.108539+0000","last_fullsized":"2024-04-23T18:17:36.108539+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:47:09.933786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234746+0000","last_change":"2024-04-23T18:16:33.992108+0000","last_active":"2024-04-23T18:17:36.234746+0000","last_peered":"2024-04-23T18:17:36.234746+0000","last_clean":"2024-04-23T18:17:36.234746+0000","last_became_active":"2024-04-23T18:16:33.991751+0000","last_became_peered":"2024-04-23T18:16:33.991751+0000","last_unstale":"2024-04-23T18:17:36.234746+0000","last_undegraded":"2024-04-23T18:17:36.234746+0000","last_fullsized":"2024-04-23T18:17:36.234746+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:07:57.416263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234975+0000","last_change":"2024-04-23T18:16:32.960461+0000","last_active":"2024-04-23T18:17:36.234975+0000","last_peered":"2024-04-23T18:17:36.234975+0000","last_clean":"2024-04-23T18:17:36.234975+0000","last_became_active":"2024-04-23T18:16:32.960115+0000","last_became_peered":"2024-04-23T18:16:32.960115+0000","last_unstale":"2024-04-23T18:17:36.234975+0000","last_undegraded":"2024-04-23T18:17:36.234975+0000","last_fullsized":"2024-04-23T18:17:36.234975+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:30.533043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106850+0000","last_change":"2024-04-23T18:16:34.001831+0000","last_active":"2024-04-23T18:17:36.106850+0000","last_peered":"2024-04-23T18:17:36.106850+0000","last_clean":"2024-04-23T18:17:36.106850+0000","last_became_active":"2024-04-23T18:16:34.001486+0000","last_became_peered":"2024-04-23T18:16:34.001486+0000","last_unstale":"2024-04-23T18:17:36.106850+0000","last_undegraded":"2024-04-23T18:17:36.106850+0000","last_fullsized":"2024-04-23T18:17:36.106850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:06:40.452366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.113098+0000","last_change":"2024-04-23T18:16:32.960387+0000","last_active":"2024-04-23T18:17:36.113098+0000","last_peered":"2024-04-23T18:17:36.113098+0000","last_clean":"2024-04-23T18:17:36.113098+0000","last_became_active":"2024-04-23T18:16:32.960046+0000","last_became_peered":"2024-04-23T18:16:32.960046+0000","last_unstale":"2024-04-23T18:17:36.113098+0000","last_undegraded":"2024-04-23T18:17:36.113098+0000","last_fullsized":"2024-04-23T18:17:36.113098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:03.296653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109327+0000","last_change":"2024-04-23T18:16:33.996520+0000","last_active":"2024-04-23T18:17:36.109327+0000","last_peered":"2024-04-23T18:17:36.109327+0000","last_clean":"2024-04-23T18:17:36.109327+0000","last_became_active":"2024-04-23T18:16:33.995636+0000","last_became_peered":"2024-04-23T18:16:33.995636+0000","last_unstale":"2024-04-23T18:17:36.109327+0000","last_undegraded":"2024-04-23T18:17:36.109327+0000","last_fullsized":"2024-04-23T18:17:36.109327+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:29.665407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.112783+0000","last_change":"2024-04-23T18:16:32.956813+0000","last_active":"2024-04-23T18:17:36.112783+0000","last_peered":"2024-04-23T18:17:36.112783+0000","last_clean":"2024-04-23T18:17:36.112783+0000","last_became_active":"2024-04-23T18:16:32.956505+0000","last_became_peered":"2024-04-23T18:16:32.956505+0000","last_unstale":"2024-04-23T18:17:36.112783+0000","last_undegraded":"2024-04-23T18:17:36.112783+0000","last_fullsized":"2024-04-23T18:17:36.112783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:36:27.330353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110391+0000","last_change":"2024-04-23T18:16:33.999754+0000","last_active":"2024-04-23T18:17:36.110391+0000","last_peered":"2024-04-23T18:17:36.110391+0000","last_clean":"2024-04-23T18:17:36.110391+0000","last_became_active":"2024-04-23T18:16:33.999034+0000","last_became_peered":"2024-04-23T18:16:33.999034+0000","last_unstale":"2024-04-23T18:17:36.110391+0000","last_undegraded":"2024-04-23T18:17:36.110391+0000","last_fullsized":"2024-04-23T18:17:36.110391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:26:08.595745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":108,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:44.127571+0000","last_change":"2024-04-23T18:17:36.120371+0000","last_active":"2024-04-23T18:17:44.127571+0000","last_peered":"2024-04-23T18:17:44.127571+0000","last_clean":"2024-04-23T18:17:44.127571+0000","last_became_active":"2024-04-23T18:17:36.116149+0000","last_became_peered":"2024-04-23T18:17:36.116149+0000","last_unstale":"2024-04-23T18:17:44.127571+0000","last_undegraded":"2024-04-23T18:17:44.127571+0000","last_fullsized":"2024-04-23T18:17:44.127571+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:54.994254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.112158+0000","last_change":"2024-04-23T18:16:33.995166+0000","last_active":"2024-04-23T18:17:36.112158+0000","last_peered":"2024-04-23T18:17:36.112158+0000","last_clean":"2024-04-23T18:17:36.112158+0000","last_became_active":"2024-04-23T18:16:33.994347+0000","last_became_peered":"2024-04-23T18:16:33.994347+0000","last_unstale":"2024-04-23T18:17:36.112158+0000","last_undegraded":"2024-04-23T18:17:36.112158+0000","last_fullsized":"2024-04-23T18:17:36.112158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.305837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212357+0000","last_change":"2024-04-23T18:16:32.959358+0000","last_active":"2024-04-23T18:17:36.212357+0000","last_peered":"2024-04-23T18:17:36.212357+0000","last_clean":"2024-04-23T18:17:36.212357+0000","last_became_active":"2024-04-23T18:16:32.959036+0000","last_became_peered":"2024-04-23T18:16:32.959036+0000","last_unstale":"2024-04-23T18:17:36.212357+0000","last_undegraded":"2024-04-23T18:17:36.212357+0000","last_fullsized":"2024-04-23T18:17:36.212357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:03:15.734754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111598+0000","last_change":"2024-04-23T18:16:33.988578+0000","last_active":"2024-04-23T18:17:36.111598+0000","last_peered":"2024-04-23T18:17:36.111598+0000","last_clean":"2024-04-23T18:17:36.111598+0000","last_became_active":"2024-04-23T18:16:33.988248+0000","last_became_peered":"2024-04-23T18:16:33.988248+0000","last_unstale":"2024-04-23T18:17:36.111598+0000","last_undegraded":"2024-04-23T18:17:36.111598+0000","last_fullsized":"2024-04-23T18:17:36.111598+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:45:15.538971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.105727+0000","last_change":"2024-04-23T18:16:32.952945+0000","last_active":"2024-04-23T18:17:36.105727+0000","last_peered":"2024-04-23T18:17:36.105727+0000","last_clean":"2024-04-23T18:17:36.105727+0000","last_became_active":"2024-04-23T18:16:32.952602+0000","last_became_peered":"2024-04-23T18:16:32.952602+0000","last_unstale":"2024-04-23T18:17:36.105727+0000","last_undegraded":"2024-04-23T18:17:36.105727+0000","last_fullsized":"2024-04-23T18:17:36.105727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:20.037431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234542+0000","last_change":"2024-04-23T18:16:33.996897+0000","last_active":"2024-04-23T18:17:36.234542+0000","last_peered":"2024-04-23T18:17:36.234542+0000","last_clean":"2024-04-23T18:17:36.234542+0000","last_became_active":"2024-04-23T18:16:33.996214+0000","last_became_peered":"2024-04-23T18:16:33.996214+0000","last_unstale":"2024-04-23T18:17:36.234542+0000","last_undegraded":"2024-04-23T18:17:36.234542+0000","last_fullsized":"2024-04-23T18:17:36.234542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:34.931697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109970+0000","last_change":"2024-04-23T18:16:32.959637+0000","last_active":"2024-04-23T18:17:36.109970+0000","last_peered":"2024-04-23T18:17:36.109970+0000","last_clean":"2024-04-23T18:17:36.109970+0000","last_became_active":"2024-04-23T18:16:32.959332+0000","last_became_peered":"2024-04-23T18:16:32.959332+0000","last_unstale":"2024-04-23T18:17:36.109970+0000","last_undegraded":"2024-04-23T18:17:36.109970+0000","last_fullsized":"2024-04-23T18:17:36.109970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:30:49.089101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108255+0000","last_change":"2024-04-23T18:16:33.986288+0000","last_active":"2024-04-23T18:17:36.108255+0000","last_peered":"2024-04-23T18:17:36.108255+0000","last_clean":"2024-04-23T18:17:36.108255+0000","last_became_active":"2024-04-23T18:16:33.985866+0000","last_became_peered":"2024-04-23T18:16:33.985866+0000","last_unstale":"2024-04-23T18:17:36.108255+0000","last_undegraded":"2024-04-23T18:17:36.108255+0000","last_fullsized":"2024-04-23T18:17:36.108255+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:30:17.675952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:40.122776+0000","last_change":"2024-04-23T18:16:32.958466+0000","last_active":"2024-04-23T18:17:40.122776+0000","last_peered":"2024-04-23T18:17:40.122776+0000","last_clean":"2024-04-23T18:17:40.122776+0000","last_became_active":"2024-04-23T18:16:32.958155+0000","last_became_peered":"2024-04-23T18:16:32.958155+0000","last_unstale":"2024-04-23T18:17:40.122776+0000","last_undegraded":"2024-04-23T18:17:40.122776+0000","last_fullsized":"2024-04-23T18:17:40.122776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:15:49.073934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":64,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109772+0000","last_change":"2024-04-23T18:16:33.996156+0000","last_active":"2024-04-23T18:17:36.109772+0000","last_peered":"2024-04-23T18:17:36.109772+0000","last_clean":"2024-04-23T18:17:36.109772+0000","last_became_active":"2024-04-23T18:16:33.995711+0000","last_became_peered":"2024-04-23T18:16:33.995711+0000","last_unstale":"2024-04-23T18:17:36.109772+0000","last_undegraded":"2024-04-23T18:17:36.109772+0000","last_fullsized":"2024-04-23T18:17:36.109772+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:45.272598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.217460+0000","last_change":"2024-04-23T18:17:36.217460+0000","last_active":"2024-04-23T18:17:36.217460+0000","last_peered":"2024-04-23T18:17:36.217460+0000","last_clean":"2024-04-23T18:17:36.217460+0000","last_became_active":"2024-04-23T18:17:36.214196+0000","last_became_peered":"2024-04-23T18:17:36.214196+0000","last_unstale":"2024-04-23T18:17:36.217460+0000","last_undegraded":"2024-04-23T18:17:36.217460+0000","last_fullsized":"2024-04-23T18:17:36.217460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:34:55.557943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.114238+0000","last_change":"2024-04-23T18:17:36.114238+0000","last_active":"2024-04-23T18:17:36.114238+0000","last_peered":"2024-04-23T18:17:36.114238+0000","last_clean":"2024-04-23T18:17:36.114238+0000","last_became_active":"2024-04-23T18:17:36.113810+0000","last_became_peered":"2024-04-23T18:17:36.113810+0000","last_unstale":"2024-04-23T18:17:36.114238+0000","last_undegraded":"2024-04-23T18:17:36.114238+0000","last_fullsized":"2024-04-23T18:17:36.114238+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:53.854596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110496+0000","last_change":"2024-04-23T18:16:32.966725+0000","last_active":"2024-04-23T18:17:36.110496+0000","last_peered":"2024-04-23T18:17:36.110496+0000","last_clean":"2024-04-23T18:17:36.110496+0000","last_became_active":"2024-04-23T18:16:32.966304+0000","last_became_peered":"2024-04-23T18:16:32.966304+0000","last_unstale":"2024-04-23T18:17:36.110496+0000","last_undegraded":"2024-04-23T18:17:36.110496+0000","last_fullsized":"2024-04-23T18:17:36.110496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:45.191051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109512+0000","last_change":"2024-04-23T18:16:33.994667+0000","last_active":"2024-04-23T18:17:36.109512+0000","last_peered":"2024-04-23T18:17:36.109512+0000","last_clean":"2024-04-23T18:17:36.109512+0000","last_became_active":"2024-04-23T18:16:33.994305+0000","last_became_peered":"2024-04-23T18:16:33.994305+0000","last_unstale":"2024-04-23T18:17:36.109512+0000","last_undegraded":"2024-04-23T18:17:36.109512+0000","last_fullsized":"2024-04-23T18:17:36.109512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:23.807075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'29","reported_seq":184,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:41.122181+0000","last_change":"2024-04-23T18:16:32.957946+0000","last_active":"2024-04-23T18:17:41.122181+0000","last_peered":"2024-04-23T18:17:41.122181+0000","last_clean":"2024-04-23T18:17:41.122181+0000","last_became_active":"2024-04-23T18:16:32.957478+0000","last_became_peered":"2024-04-23T18:16:32.957478+0000","last_unstale":"2024-04-23T18:17:41.122181+0000","last_undegraded":"2024-04-23T18:17:41.122181+0000","last_fullsized":"2024-04-23T18:17:41.122181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:16.850604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60023,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":136,"num_read_kb":126,"num_write":30,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.112890+0000","last_change":"2024-04-23T18:16:33.989537+0000","last_active":"2024-04-23T18:17:36.112890+0000","last_peered":"2024-04-23T18:17:36.112890+0000","last_clean":"2024-04-23T18:17:36.112890+0000","last_became_active":"2024-04-23T18:16:33.989180+0000","last_became_peered":"2024-04-23T18:16:33.989180+0000","last_unstale":"2024-04-23T18:17:36.112890+0000","last_undegraded":"2024-04-23T18:17:36.112890+0000","last_fullsized":"2024-04-23T18:17:36.112890+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:53:18.700347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111503+0000","last_change":"2024-04-23T18:16:32.967314+0000","last_active":"2024-04-23T18:17:36.111503+0000","last_peered":"2024-04-23T18:17:36.111503+0000","last_clean":"2024-04-23T18:17:36.111503+0000","last_became_active":"2024-04-23T18:16:32.966861+0000","last_became_peered":"2024-04-23T18:16:32.966861+0000","last_unstale":"2024-04-23T18:17:36.111503+0000","last_undegraded":"2024-04-23T18:17:36.111503+0000","last_fullsized":"2024-04-23T18:17:36.111503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:00:25.606402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.105698+0000","last_change":"2024-04-23T18:16:33.992593+0000","last_active":"2024-04-23T18:17:36.105698+0000","last_peered":"2024-04-23T18:17:36.105698+0000","last_clean":"2024-04-23T18:17:36.105698+0000","last_became_active":"2024-04-23T18:16:33.992228+0000","last_became_peered":"2024-04-23T18:16:33.992228+0000","last_unstale":"2024-04-23T18:17:36.105698+0000","last_undegraded":"2024-04-23T18:17:36.105698+0000","last_fullsized":"2024-04-23T18:17:36.105698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:12:23.836829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110368+0000","last_change":"2024-04-23T18:16:32.961740+0000","last_active":"2024-04-23T18:17:36.110368+0000","last_peered":"2024-04-23T18:17:36.110368+0000","last_clean":"2024-04-23T18:17:36.110368+0000","last_became_active":"2024-04-23T18:16:32.961331+0000","last_became_peered":"2024-04-23T18:16:32.961331+0000","last_unstale":"2024-04-23T18:17:36.110368+0000","last_undegraded":"2024-04-23T18:17:36.110368+0000","last_fullsized":"2024-04-23T18:17:36.110368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:59:17.665288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111151+0000","last_change":"2024-04-23T18:17:36.111151+0000","last_active":"2024-04-23T18:17:36.111151+0000","last_peered":"2024-04-23T18:17:36.111151+0000","last_clean":"2024-04-23T18:17:36.111151+0000","last_became_active":"2024-04-23T18:17:36.110770+0000","last_became_peered":"2024-04-23T18:17:36.110770+0000","last_unstale":"2024-04-23T18:17:36.111151+0000","last_undegraded":"2024-04-23T18:17:36.111151+0000","last_fullsized":"2024-04-23T18:17:36.111151+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:01:03.456738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'4","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:41.061089+0000","last_change":"2024-04-23T18:16:32.954935+0000","last_active":"2024-04-23T18:17:41.061089+0000","last_peered":"2024-04-23T18:17:41.061089+0000","last_clean":"2024-04-23T18:17:41.061089+0000","last_became_active":"2024-04-23T18:16:32.954595+0000","last_became_peered":"2024-04-23T18:16:32.954595+0000","last_unstale":"2024-04-23T18:17:41.061089+0000","last_undegraded":"2024-04-23T18:17:41.061089+0000","last_fullsized":"2024-04-23T18:17:41.061089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:22.954101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108474+0000","last_change":"2024-04-23T18:16:34.000215+0000","last_active":"2024-04-23T18:17:36.108474+0000","last_peered":"2024-04-23T18:17:36.108474+0000","last_clean":"2024-04-23T18:17:36.108474+0000","last_became_active":"2024-04-23T18:16:33.999855+0000","last_became_peered":"2024-04-23T18:16:33.999855+0000","last_unstale":"2024-04-23T18:17:36.108474+0000","last_undegraded":"2024-04-23T18:17:36.108474+0000","last_fullsized":"2024-04-23T18:17:36.108474+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:14.686916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.128101+0000","last_change":"2024-04-23T18:17:36.127721+0000","last_active":"2024-04-23T18:17:36.128101+0000","last_peered":"2024-04-23T18:17:36.128101+0000","last_clean":"2024-04-23T18:17:36.128101+0000","last_became_active":"2024-04-23T18:17:36.113735+0000","last_became_peered":"2024-04-23T18:17:36.113735+0000","last_unstale":"2024-04-23T18:17:36.128101+0000","last_undegraded":"2024-04-23T18:17:36.128101+0000","last_fullsized":"2024-04-23T18:17:36.128101+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:56:41.395925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.214244+0000","last_change":"2024-04-23T18:17:36.214244+0000","last_active":"2024-04-23T18:17:36.214244+0000","last_peered":"2024-04-23T18:17:36.214244+0000","last_clean":"2024-04-23T18:17:36.214244+0000","last_became_active":"2024-04-23T18:17:36.213897+0000","last_became_peered":"2024-04-23T18:17:36.213897+0000","last_unstale":"2024-04-23T18:17:36.214244+0000","last_undegraded":"2024-04-23T18:17:36.214244+0000","last_fullsized":"2024-04-23T18:17:36.214244+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:44:24.935747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111781+0000","last_change":"2024-04-23T18:16:32.959253+0000","last_active":"2024-04-23T18:17:36.111781+0000","last_peered":"2024-04-23T18:17:36.111781+0000","last_clean":"2024-04-23T18:17:36.111781+0000","last_became_active":"2024-04-23T18:16:32.958929+0000","last_became_peered":"2024-04-23T18:16:32.958929+0000","last_unstale":"2024-04-23T18:17:36.111781+0000","last_undegraded":"2024-04-23T18:17:36.111781+0000","last_fullsized":"2024-04-23T18:17:36.111781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:36.317516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108748+0000","last_change":"2024-04-23T18:16:33.979464+0000","last_active":"2024-04-23T18:17:36.108748+0000","last_peered":"2024-04-23T18:17:36.108748+0000","last_clean":"2024-04-23T18:17:36.108748+0000","last_became_active":"2024-04-23T18:16:33.979106+0000","last_became_peered":"2024-04-23T18:16:33.979106+0000","last_unstale":"2024-04-23T18:17:36.108748+0000","last_undegraded":"2024-04-23T18:17:36.108748+0000","last_fullsized":"2024-04-23T18:17:36.108748+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:00.125514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108469+0000","last_change":"2024-04-23T18:16:32.960812+0000","last_active":"2024-04-23T18:17:36.108469+0000","last_peered":"2024-04-23T18:17:36.108469+0000","last_clean":"2024-04-23T18:17:36.108469+0000","last_became_active":"2024-04-23T18:16:32.959807+0000","last_became_peered":"2024-04-23T18:16:32.959807+0000","last_unstale":"2024-04-23T18:17:36.108469+0000","last_undegraded":"2024-04-23T18:17:36.108469+0000","last_fullsized":"2024-04-23T18:17:36.108469+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:45.097443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212762+0000","last_change":"2024-04-23T18:16:33.992891+0000","last_active":"2024-04-23T18:17:36.212762+0000","last_peered":"2024-04-23T18:17:36.212762+0000","last_clean":"2024-04-23T18:17:36.212762+0000","last_became_active":"2024-04-23T18:16:33.992559+0000","last_became_peered":"2024-04-23T18:16:33.992559+0000","last_unstale":"2024-04-23T18:17:36.212762+0000","last_undegraded":"2024-04-23T18:17:36.212762+0000","last_fullsized":"2024-04-23T18:17:36.212762+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:48.615158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108792+0000","last_change":"2024-04-23T18:16:32.957015+0000","last_active":"2024-04-23T18:17:36.108792+0000","last_peered":"2024-04-23T18:17:36.108792+0000","last_clean":"2024-04-23T18:17:36.108792+0000","last_became_active":"2024-04-23T18:16:32.956690+0000","last_became_peered":"2024-04-23T18:16:32.956690+0000","last_unstale":"2024-04-23T18:17:36.108792+0000","last_undegraded":"2024-04-23T18:17:36.108792+0000","last_fullsized":"2024-04-23T18:17:36.108792+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:20:42.087521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212425+0000","last_change":"2024-04-23T18:16:33.998906+0000","last_active":"2024-04-23T18:17:36.212425+0000","last_peered":"2024-04-23T18:17:36.212425+0000","last_clean":"2024-04-23T18:17:36.212425+0000","last_became_active":"2024-04-23T18:16:33.997297+0000","last_became_peered":"2024-04-23T18:16:33.997297+0000","last_unstale":"2024-04-23T18:17:36.212425+0000","last_undegraded":"2024-04-23T18:17:36.212425+0000","last_fullsized":"2024-04-23T18:17:36.212425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:06.668830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.113962+0000","last_change":"2024-04-23T18:17:36.113962+0000","last_active":"2024-04-23T18:17:36.113962+0000","last_peered":"2024-04-23T18:17:36.113962+0000","last_clean":"2024-04-23T18:17:36.113962+0000","last_became_active":"2024-04-23T18:17:36.113573+0000","last_became_peered":"2024-04-23T18:17:36.113573+0000","last_unstale":"2024-04-23T18:17:36.113962+0000","last_undegraded":"2024-04-23T18:17:36.113962+0000","last_fullsized":"2024-04-23T18:17:36.113962+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:47:25.408144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":90,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:43.063217+0000","last_change":"2024-04-23T18:16:32.961423+0000","last_active":"2024-04-23T18:17:43.063217+0000","last_peered":"2024-04-23T18:17:43.063217+0000","last_clean":"2024-04-23T18:17:43.063217+0000","last_became_active":"2024-04-23T18:16:32.961116+0000","last_became_peered":"2024-04-23T18:16:32.961116+0000","last_unstale":"2024-04-23T18:17:43.063217+0000","last_undegraded":"2024-04-23T18:17:43.063217+0000","last_fullsized":"2024-04-23T18:17:43.063217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:28.525642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234845+0000","last_change":"2024-04-23T18:16:33.990747+0000","last_active":"2024-04-23T18:17:36.234845+0000","last_peered":"2024-04-23T18:17:36.234845+0000","last_clean":"2024-04-23T18:17:36.234845+0000","last_became_active":"2024-04-23T18:16:33.989972+0000","last_became_peered":"2024-04-23T18:16:33.989972+0000","last_unstale":"2024-04-23T18:17:36.234845+0000","last_undegraded":"2024-04-23T18:17:36.234845+0000","last_fullsized":"2024-04-23T18:17:36.234845+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:04.667446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212337+0000","last_change":"2024-04-23T18:16:32.952821+0000","last_active":"2024-04-23T18:17:36.212337+0000","last_peered":"2024-04-23T18:17:36.212337+0000","last_clean":"2024-04-23T18:17:36.212337+0000","last_became_active":"2024-04-23T18:16:32.952470+0000","last_became_peered":"2024-04-23T18:16:32.952470+0000","last_unstale":"2024-04-23T18:17:36.212337+0000","last_undegraded":"2024-04-23T18:17:36.212337+0000","last_fullsized":"2024-04-23T18:17:36.212337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:19.392670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":353,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108520+0000","last_change":"2024-04-23T18:15:25.867370+0000","last_active":"2024-04-23T18:17:36.108520+0000","last_peered":"2024-04-23T18:17:36.108520+0000","last_clean":"2024-04-23T18:17:36.108520+0000","last_became_active":"2024-04-23T18:15:25.866815+0000","last_became_peered":"2024-04-23T18:15:25.866815+0000","last_unstale":"2024-04-23T18:17:36.108520+0000","last_undegraded":"2024-04-23T18:17:36.108520+0000","last_fullsized":"2024-04-23T18:17:36.108520+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_clean_scrub_stamp":"2024-04-23T18:12:02.913350+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:42.482422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.105597+0000","last_change":"2024-04-23T18:16:33.992988+0000","last_active":"2024-04-23T18:17:36.105597+0000","last_peered":"2024-04-23T18:17:36.105597+0000","last_clean":"2024-04-23T18:17:36.105597+0000","last_became_active":"2024-04-23T18:16:33.992650+0000","last_became_peered":"2024-04-23T18:16:33.992650+0000","last_unstale":"2024-04-23T18:17:36.105597+0000","last_undegraded":"2024-04-23T18:17:36.105597+0000","last_fullsized":"2024-04-23T18:17:36.105597+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:54.054708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108693+0000","last_change":"2024-04-23T18:16:32.957318+0000","last_active":"2024-04-23T18:17:36.108693+0000","last_peered":"2024-04-23T18:17:36.108693+0000","last_clean":"2024-04-23T18:17:36.108693+0000","last_became_active":"2024-04-23T18:16:32.956973+0000","last_became_peered":"2024-04-23T18:16:32.956973+0000","last_unstale":"2024-04-23T18:17:36.108693+0000","last_undegraded":"2024-04-23T18:17:36.108693+0000","last_fullsized":"2024-04-23T18:17:36.108693+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:35.684825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.117673+0000","last_change":"2024-04-23T18:17:36.117253+0000","last_active":"2024-04-23T18:17:36.117673+0000","last_peered":"2024-04-23T18:17:36.117673+0000","last_clean":"2024-04-23T18:17:36.117673+0000","last_became_active":"2024-04-23T18:17:36.111210+0000","last_became_peered":"2024-04-23T18:17:36.111210+0000","last_unstale":"2024-04-23T18:17:36.117673+0000","last_undegraded":"2024-04-23T18:17:36.117673+0000","last_fullsized":"2024-04-23T18:17:36.117673+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:31.599001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106159+0000","last_change":"2024-04-23T18:16:33.996345+0000","last_active":"2024-04-23T18:17:36.106159+0000","last_peered":"2024-04-23T18:17:36.106159+0000","last_clean":"2024-04-23T18:17:36.106159+0000","last_became_active":"2024-04-23T18:16:33.993802+0000","last_became_peered":"2024-04-23T18:16:33.993802+0000","last_unstale":"2024-04-23T18:17:36.106159+0000","last_undegraded":"2024-04-23T18:17:36.106159+0000","last_fullsized":"2024-04-23T18:17:36.106159+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:32.410330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108694+0000","last_change":"2024-04-23T18:16:32.957953+0000","last_active":"2024-04-23T18:17:36.108694+0000","last_peered":"2024-04-23T18:17:36.108694+0000","last_clean":"2024-04-23T18:17:36.108694+0000","last_became_active":"2024-04-23T18:16:32.957616+0000","last_became_peered":"2024-04-23T18:16:32.957616+0000","last_unstale":"2024-04-23T18:17:36.108694+0000","last_undegraded":"2024-04-23T18:17:36.108694+0000","last_fullsized":"2024-04-23T18:17:36.108694+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:15.485373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234995+0000","last_change":"2024-04-23T18:16:33.985685+0000","last_active":"2024-04-23T18:17:36.234995+0000","last_peered":"2024-04-23T18:17:36.234995+0000","last_clean":"2024-04-23T18:17:36.234995+0000","last_became_active":"2024-04-23T18:16:33.984631+0000","last_became_peered":"2024-04-23T18:16:33.984631+0000","last_unstale":"2024-04-23T18:17:36.234995+0000","last_undegraded":"2024-04-23T18:17:36.234995+0000","last_fullsized":"2024-04-23T18:17:36.234995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:45.893285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.213198+0000","last_change":"2024-04-23T18:16:32.954281+0000","last_active":"2024-04-23T18:17:36.213198+0000","last_peered":"2024-04-23T18:17:36.213198+0000","last_clean":"2024-04-23T18:17:36.213198+0000","last_became_active":"2024-04-23T18:16:32.953988+0000","last_became_peered":"2024-04-23T18:16:32.953988+0000","last_unstale":"2024-04-23T18:17:36.213198+0000","last_undegraded":"2024-04-23T18:17:36.213198+0000","last_fullsized":"2024-04-23T18:17:36.213198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:09.049861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106785+0000","last_change":"2024-04-23T18:16:33.994860+0000","last_active":"2024-04-23T18:17:36.106785+0000","last_peered":"2024-04-23T18:17:36.106785+0000","last_clean":"2024-04-23T18:17:36.106785+0000","last_became_active":"2024-04-23T18:16:33.993438+0000","last_became_peered":"2024-04-23T18:16:33.993438+0000","last_unstale":"2024-04-23T18:17:36.106785+0000","last_undegraded":"2024-04-23T18:17:36.106785+0000","last_fullsized":"2024-04-23T18:17:36.106785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:55.565873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108913+0000","last_change":"2024-04-23T18:16:32.959251+0000","last_active":"2024-04-23T18:17:36.108913+0000","last_peered":"2024-04-23T18:17:36.108913+0000","last_clean":"2024-04-23T18:17:36.108913+0000","last_became_active":"2024-04-23T18:16:32.958937+0000","last_became_peered":"2024-04-23T18:16:32.958937+0000","last_unstale":"2024-04-23T18:17:36.108913+0000","last_undegraded":"2024-04-23T18:17:36.108913+0000","last_fullsized":"2024-04-23T18:17:36.108913+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.126634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111905+0000","last_change":"2024-04-23T18:16:33.995480+0000","last_active":"2024-04-23T18:17:36.111905+0000","last_peered":"2024-04-23T18:17:36.111905+0000","last_clean":"2024-04-23T18:17:36.111905+0000","last_became_active":"2024-04-23T18:16:33.994361+0000","last_became_peered":"2024-04-23T18:16:33.994361+0000","last_unstale":"2024-04-23T18:17:36.111905+0000","last_undegraded":"2024-04-23T18:17:36.111905+0000","last_fullsized":"2024-04-23T18:17:36.111905+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:59:35.240562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234665+0000","last_change":"2024-04-23T18:16:32.967787+0000","last_active":"2024-04-23T18:17:36.234665+0000","last_peered":"2024-04-23T18:17:36.234665+0000","last_clean":"2024-04-23T18:17:36.234665+0000","last_became_active":"2024-04-23T18:16:32.967463+0000","last_became_peered":"2024-04-23T18:16:32.967463+0000","last_unstale":"2024-04-23T18:17:36.234665+0000","last_undegraded":"2024-04-23T18:17:36.234665+0000","last_fullsized":"2024-04-23T18:17:36.234665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:58.743682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109014+0000","last_change":"2024-04-23T18:16:33.980558+0000","last_active":"2024-04-23T18:17:36.109014+0000","last_peered":"2024-04-23T18:17:36.109014+0000","last_clean":"2024-04-23T18:17:36.109014+0000","last_became_active":"2024-04-23T18:16:33.980146+0000","last_became_peered":"2024-04-23T18:16:33.980146+0000","last_unstale":"2024-04-23T18:17:36.109014+0000","last_undegraded":"2024-04-23T18:17:36.109014+0000","last_fullsized":"2024-04-23T18:17:36.109014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:58.001833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110646+0000","last_change":"2024-04-23T18:16:32.962139+0000","last_active":"2024-04-23T18:17:36.110646+0000","last_peered":"2024-04-23T18:17:36.110646+0000","last_clean":"2024-04-23T18:17:36.110646+0000","last_became_active":"2024-04-23T18:16:32.961785+0000","last_became_peered":"2024-04-23T18:16:32.961785+0000","last_unstale":"2024-04-23T18:17:36.110646+0000","last_undegraded":"2024-04-23T18:17:36.110646+0000","last_fullsized":"2024-04-23T18:17:36.110646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:09.619414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.214255+0000","last_change":"2024-04-23T18:17:36.214255+0000","last_active":"2024-04-23T18:17:36.214255+0000","last_peered":"2024-04-23T18:17:36.214255+0000","last_clean":"2024-04-23T18:17:36.214255+0000","last_became_active":"2024-04-23T18:17:36.213832+0000","last_became_peered":"2024-04-23T18:17:36.213832+0000","last_unstale":"2024-04-23T18:17:36.214255+0000","last_undegraded":"2024-04-23T18:17:36.214255+0000","last_fullsized":"2024-04-23T18:17:36.214255+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:59:35.497710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.112439+0000","last_change":"2024-04-23T18:16:32.956965+0000","last_active":"2024-04-23T18:17:36.112439+0000","last_peered":"2024-04-23T18:17:36.112439+0000","last_clean":"2024-04-23T18:17:36.112439+0000","last_became_active":"2024-04-23T18:16:32.956672+0000","last_became_peered":"2024-04-23T18:16:32.956672+0000","last_unstale":"2024-04-23T18:17:36.112439+0000","last_undegraded":"2024-04-23T18:17:36.112439+0000","last_fullsized":"2024-04-23T18:17:36.112439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:00:44.470809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109231+0000","last_change":"2024-04-23T18:16:34.002711+0000","last_active":"2024-04-23T18:17:36.109231+0000","last_peered":"2024-04-23T18:17:36.109231+0000","last_clean":"2024-04-23T18:17:36.109231+0000","last_became_active":"2024-04-23T18:16:34.002365+0000","last_became_peered":"2024-04-23T18:16:34.002365+0000","last_unstale":"2024-04-23T18:17:36.109231+0000","last_undegraded":"2024-04-23T18:17:36.109231+0000","last_fullsized":"2024-04-23T18:17:36.109231+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:52:11.982735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111252+0000","last_change":"2024-04-23T18:16:32.965121+0000","last_active":"2024-04-23T18:17:36.111252+0000","last_peered":"2024-04-23T18:17:36.111252+0000","last_clean":"2024-04-23T18:17:36.111252+0000","last_became_active":"2024-04-23T18:16:32.964732+0000","last_became_peered":"2024-04-23T18:16:32.964732+0000","last_unstale":"2024-04-23T18:17:36.111252+0000","last_undegraded":"2024-04-23T18:17:36.111252+0000","last_fullsized":"2024-04-23T18:17:36.111252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:19:34.690245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109037+0000","last_change":"2024-04-23T18:16:33.996357+0000","last_active":"2024-04-23T18:17:36.109037+0000","last_peered":"2024-04-23T18:17:36.109037+0000","last_clean":"2024-04-23T18:17:36.109037+0000","last_became_active":"2024-04-23T18:16:33.996017+0000","last_became_peered":"2024-04-23T18:16:33.996017+0000","last_unstale":"2024-04-23T18:17:36.109037+0000","last_undegraded":"2024-04-23T18:17:36.109037+0000","last_fullsized":"2024-04-23T18:17:36.109037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:20:57.321228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110284+0000","last_change":"2024-04-23T18:16:32.961705+0000","last_active":"2024-04-23T18:17:36.110284+0000","last_peered":"2024-04-23T18:17:36.110284+0000","last_clean":"2024-04-23T18:17:36.110284+0000","last_became_active":"2024-04-23T18:16:32.961342+0000","last_became_peered":"2024-04-23T18:16:32.961342+0000","last_unstale":"2024-04-23T18:17:36.110284+0000","last_undegraded":"2024-04-23T18:17:36.110284+0000","last_fullsized":"2024-04-23T18:17:36.110284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:51.340910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108878+0000","last_change":"2024-04-23T18:16:33.992944+0000","last_active":"2024-04-23T18:17:36.108878+0000","last_peered":"2024-04-23T18:17:36.108878+0000","last_clean":"2024-04-23T18:17:36.108878+0000","last_became_active":"2024-04-23T18:16:33.992579+0000","last_became_peered":"2024-04-23T18:16:33.992579+0000","last_unstale":"2024-04-23T18:17:36.108878+0000","last_undegraded":"2024-04-23T18:17:36.108878+0000","last_fullsized":"2024-04-23T18:17:36.108878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:13:37.829588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106027+0000","last_change":"2024-04-23T18:16:32.955192+0000","last_active":"2024-04-23T18:17:36.106027+0000","last_peered":"2024-04-23T18:17:36.106027+0000","last_clean":"2024-04-23T18:17:36.106027+0000","last_became_active":"2024-04-23T18:16:32.954860+0000","last_became_peered":"2024-04-23T18:16:32.954860+0000","last_unstale":"2024-04-23T18:17:36.106027+0000","last_undegraded":"2024-04-23T18:17:36.106027+0000","last_fullsized":"2024-04-23T18:17:36.106027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:20.796060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108864+0000","last_change":"2024-04-23T18:16:33.986211+0000","last_active":"2024-04-23T18:17:36.108864+0000","last_peered":"2024-04-23T18:17:36.108864+0000","last_clean":"2024-04-23T18:17:36.108864+0000","last_became_active":"2024-04-23T18:16:33.985679+0000","last_became_peered":"2024-04-23T18:16:33.985679+0000","last_unstale":"2024-04-23T18:17:36.108864+0000","last_undegraded":"2024-04-23T18:17:36.108864+0000","last_fullsized":"2024-04-23T18:17:36.108864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:03:38.797864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109118+0000","last_change":"2024-04-23T18:16:33.979462+0000","last_active":"2024-04-23T18:17:36.109118+0000","last_peered":"2024-04-23T18:17:36.109118+0000","last_clean":"2024-04-23T18:17:36.109118+0000","last_became_active":"2024-04-23T18:16:33.979114+0000","last_became_peered":"2024-04-23T18:16:33.979114+0000","last_unstale":"2024-04-23T18:17:36.109118+0000","last_undegraded":"2024-04-23T18:17:36.109118+0000","last_fullsized":"2024-04-23T18:17:36.109118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:16:49.708240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111077+0000","last_change":"2024-04-23T18:16:33.979255+0000","last_active":"2024-04-23T18:17:36.111077+0000","last_peered":"2024-04-23T18:17:36.111077+0000","last_clean":"2024-04-23T18:17:36.111077+0000","last_became_active":"2024-04-23T18:16:33.978934+0000","last_became_peered":"2024-04-23T18:16:33.978934+0000","last_unstale":"2024-04-23T18:17:36.111077+0000","last_undegraded":"2024-04-23T18:17:36.111077+0000","last_fullsized":"2024-04-23T18:17:36.111077+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:10.104564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110531+0000","last_change":"2024-04-23T18:16:33.981877+0000","last_active":"2024-04-23T18:17:36.110531+0000","last_peered":"2024-04-23T18:17:36.110531+0000","last_clean":"2024-04-23T18:17:36.110531+0000","last_became_active":"2024-04-23T18:16:33.981536+0000","last_became_peered":"2024-04-23T18:16:33.981536+0000","last_unstale":"2024-04-23T18:17:36.110531+0000","last_undegraded":"2024-04-23T18:17:36.110531+0000","last_fullsized":"2024-04-23T18:17:36.110531+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:00:15.056779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106669+0000","last_change":"2024-04-23T18:16:33.999090+0000","last_active":"2024-04-23T18:17:36.106669+0000","last_peered":"2024-04-23T18:17:36.106669+0000","last_clean":"2024-04-23T18:17:36.106669+0000","last_became_active":"2024-04-23T18:16:33.998513+0000","last_became_peered":"2024-04-23T18:16:33.998513+0000","last_unstale":"2024-04-23T18:17:36.106669+0000","last_undegraded":"2024-04-23T18:17:36.106669+0000","last_fullsized":"2024-04-23T18:17:36.106669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:14.115406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106114+0000","last_change":"2024-04-23T18:16:33.993375+0000","last_active":"2024-04-23T18:17:36.106114+0000","last_peered":"2024-04-23T18:17:36.106114+0000","last_clean":"2024-04-23T18:17:36.106114+0000","last_became_active":"2024-04-23T18:16:33.991841+0000","last_became_peered":"2024-04-23T18:16:33.991841+0000","last_unstale":"2024-04-23T18:17:36.106114+0000","last_undegraded":"2024-04-23T18:17:36.106114+0000","last_fullsized":"2024-04-23T18:17:36.106114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:12.777495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109706+0000","last_change":"2024-04-23T18:16:33.983621+0000","last_active":"2024-04-23T18:17:36.109706+0000","last_peered":"2024-04-23T18:17:36.109706+0000","last_clean":"2024-04-23T18:17:36.109706+0000","last_became_active":"2024-04-23T18:16:33.983356+0000","last_became_peered":"2024-04-23T18:16:33.983356+0000","last_unstale":"2024-04-23T18:17:36.109706+0000","last_undegraded":"2024-04-23T18:17:36.109706+0000","last_fullsized":"2024-04-23T18:17:36.109706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:27:19.391281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.112107+0000","last_change":"2024-04-23T18:16:33.987952+0000","last_active":"2024-04-23T18:17:36.112107+0000","last_peered":"2024-04-23T18:17:36.112107+0000","last_clean":"2024-04-23T18:17:36.112107+0000","last_became_active":"2024-04-23T18:16:33.987334+0000","last_became_peered":"2024-04-23T18:16:33.987334+0000","last_unstale":"2024-04-23T18:17:36.112107+0000","last_undegraded":"2024-04-23T18:17:36.112107+0000","last_fullsized":"2024-04-23T18:17:36.112107+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:41.902779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234875+0000","last_change":"2024-04-23T18:16:33.983914+0000","last_active":"2024-04-23T18:17:36.234875+0000","last_peered":"2024-04-23T18:17:36.234875+0000","last_clean":"2024-04-23T18:17:36.234875+0000","last_became_active":"2024-04-23T18:16:33.983444+0000","last_became_peered":"2024-04-23T18:16:33.983444+0000","last_unstale":"2024-04-23T18:17:36.234875+0000","last_undegraded":"2024-04-23T18:17:36.234875+0000","last_fullsized":"2024-04-23T18:17:36.234875+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:10:17.630049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109958+0000","last_change":"2024-04-23T18:16:33.992496+0000","last_active":"2024-04-23T18:17:36.109958+0000","last_peered":"2024-04-23T18:17:36.109958+0000","last_clean":"2024-04-23T18:17:36.109958+0000","last_became_active":"2024-04-23T18:16:33.992155+0000","last_became_peered":"2024-04-23T18:16:33.992155+0000","last_unstale":"2024-04-23T18:17:36.109958+0000","last_undegraded":"2024-04-23T18:17:36.109958+0000","last_fullsized":"2024-04-23T18:17:36.109958+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:48.835992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234557+0000","last_change":"2024-04-23T18:16:33.984409+0000","last_active":"2024-04-23T18:17:36.234557+0000","last_peered":"2024-04-23T18:17:36.234557+0000","last_clean":"2024-04-23T18:17:36.234557+0000","last_became_active":"2024-04-23T18:16:33.984073+0000","last_became_peered":"2024-04-23T18:16:33.984073+0000","last_unstale":"2024-04-23T18:17:36.234557+0000","last_undegraded":"2024-04-23T18:17:36.234557+0000","last_fullsized":"2024-04-23T18:17:36.234557+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:45:37.796783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110072+0000","last_change":"2024-04-23T18:16:33.986376+0000","last_active":"2024-04-23T18:17:36.110072+0000","last_peered":"2024-04-23T18:17:36.110072+0000","last_clean":"2024-04-23T18:17:36.110072+0000","last_became_active":"2024-04-23T18:16:33.986012+0000","last_became_peered":"2024-04-23T18:16:33.986012+0000","last_unstale":"2024-04-23T18:17:36.110072+0000","last_undegraded":"2024-04-23T18:17:36.110072+0000","last_fullsized":"2024-04-23T18:17:36.110072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:26.940215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106072+0000","last_change":"2024-04-23T18:16:33.999795+0000","last_active":"2024-04-23T18:17:36.106072+0000","last_peered":"2024-04-23T18:17:36.106072+0000","last_clean":"2024-04-23T18:17:36.106072+0000","last_became_active":"2024-04-23T18:16:33.997034+0000","last_became_peered":"2024-04-23T18:16:33.997034+0000","last_unstale":"2024-04-23T18:17:36.106072+0000","last_undegraded":"2024-04-23T18:17:36.106072+0000","last_fullsized":"2024-04-23T18:17:36.106072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:23.983846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108549+0000","last_change":"2024-04-23T18:16:33.985495+0000","last_active":"2024-04-23T18:17:36.108549+0000","last_peered":"2024-04-23T18:17:36.108549+0000","last_clean":"2024-04-23T18:17:36.108549+0000","last_became_active":"2024-04-23T18:16:33.985178+0000","last_became_peered":"2024-04-23T18:16:33.985178+0000","last_unstale":"2024-04-23T18:17:36.108549+0000","last_undegraded":"2024-04-23T18:17:36.108549+0000","last_fullsized":"2024-04-23T18:17:36.108549+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:22:25.325756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64415,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":411,"num_read_kb":275,"num_write":158,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1147,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":193245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318298,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":468431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579936,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":1671168,"data_stored":1120192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776166,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":468533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037802,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":468319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32732,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712420,"statfs":{"total":95995035648,"available":95961518080,"internally_reserved":0,"allocated":1585152,"data_stored":1060774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495668,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":467922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":528751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40476,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704676,"statfs":{"total":95995035648,"available":95953588224,"internally_reserved":0,"allocated":999424,"data_stored":469193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":528466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182467,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":995328,"data_stored":468233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920830,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":983040,"data_stored":468750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757306,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1597440,"data_stored":1060067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":60751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:17:48.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T18:17:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:48 smithi188 ceph-mon[32694]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 110 B/s, 1 objects/s recovering 2024-04-23T18:17:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:17:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:17:48.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:48 smithi188 ceph-mon[32694]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:48.351 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:48.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:48 smithi050 ceph-mon[29187]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 110 B/s, 1 objects/s recovering 2024-04-23T18:17:48.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:48.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:48.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:17:48.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:17:48.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:48 smithi050 ceph-mon[29187]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:48.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:48 smithi177 ceph-mon[32308]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 110 B/s, 1 objects/s recovering 2024-04-23T18:17:48.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:48.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:48.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:17:48.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:17:48.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:48 smithi177 ceph-mon[32308]: from='mgr.14152 172.21.15.50:0/3285015640' entity='mgr.x' 2024-04-23T18:17:49.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:49 smithi188 ceph-mon[32694]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:49.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:49 smithi050 ceph-mon[29187]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:49.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:49 smithi177 ceph-mon[32308]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:49.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.922+0000 7f3ed2510700 1 -- 172.21.15.50:0/4000720656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ecc1064e0 msgr2=0x7f3ecc10e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.922+0000 7f3ed2510700 1 --2- 172.21.15.50:0/4000720656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ecc1064e0 0x7f3ecc10e020 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400b0a0 tx=0x7f3ec400b3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.922+0000 7f3ed2510700 1 -- 172.21.15.50:0/4000720656 shutdown_connections 2024-04-23T18:17:49.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.922+0000 7f3ed2510700 1 --2- 172.21.15.50:0/4000720656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ecc1064e0 0x7f3ecc10e020 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.922+0000 7f3ed2510700 1 --2- 172.21.15.50:0/4000720656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ecc101640 0x7f3ecc101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.922+0000 7f3ed2510700 1 --2- 172.21.15.50:0/4000720656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ecc101ff0 0x7f3ecc105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.922+0000 7f3ed2510700 1 -- 172.21.15.50:0/4000720656 >> 172.21.15.50:0/4000720656 conn(0x7f3ecc076c60 msgr2=0x7f3ecc077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.922+0000 7f3ed2510700 1 -- 172.21.15.50:0/4000720656 shutdown_connections 2024-04-23T18:17:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.922+0000 7f3ed2510700 1 -- 172.21.15.50:0/4000720656 wait complete. 2024-04-23T18:17:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.922+0000 7f3ed2510700 1 Processor -- start 2024-04-23T18:17:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.922+0000 7f3ed2510700 1 -- start start 2024-04-23T18:17:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ed2510700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ecc101640 0x7f3ecc102de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ed2510700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ecc101ff0 0x7f3ecc103320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ed2510700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ecc1064e0 0x7f3ecc103860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ed2510700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc111ba0 con 0x7f3ecc101ff0 2024-04-23T18:17:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ed2510700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc111a20 con 0x7f3ecc1064e0 2024-04-23T18:17:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ed2510700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc111d20 con 0x7f3ecc101640 2024-04-23T18:17:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ecb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ecc101ff0 0x7f3ecc103320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ecb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ecc101ff0 0x7f3ecc103320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46504/0 (socket says 172.21.15.50:46504) 2024-04-23T18:17:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ecb7fe700 1 -- 172.21.15.50:0/524763647 learned_addr learned my addr 172.21.15.50:0/524763647 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ecbfff700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ecc101640 0x7f3ecc102de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ed0aad700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ecc1064e0 0x7f3ecc103860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ecb7fe700 1 -- 172.21.15.50:0/524763647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ecc101640 msgr2=0x7f3ecc102de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ecb7fe700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ecc101640 0x7f3ecc102de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ecb7fe700 1 -- 172.21.15.50:0/524763647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ecc1064e0 msgr2=0x7f3ecc103860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ecb7fe700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ecc1064e0 0x7f3ecc103860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ecb7fe700 1 -- 172.21.15.50:0/524763647 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ecc1a7560 con 0x7f3ecc101ff0 2024-04-23T18:17:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ecb7fe700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ecc101ff0 0x7f3ecc103320 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000c9b0 tx=0x7f3ec000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.923+0000 7f3ec97fa700 1 -- 172.21.15.50:0/524763647 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0007aa0 con 0x7f3ecc101ff0 2024-04-23T18:17:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.924+0000 7f3ec97fa700 1 -- 172.21.15.50:0/524763647 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ec000ce50 con 0x7f3ecc101ff0 2024-04-23T18:17:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.924+0000 7f3ed2510700 1 -- 172.21.15.50:0/524763647 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ecc1a7850 con 0x7f3ecc101ff0 2024-04-23T18:17:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.924+0000 7f3ec97fa700 1 -- 172.21.15.50:0/524763647 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0018660 con 0x7f3ecc101ff0 2024-04-23T18:17:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.924+0000 7f3ed2510700 1 -- 172.21.15.50:0/524763647 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ecc1a7de0 con 0x7f3ecc101ff0 2024-04-23T18:17:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.925+0000 7f3ed2510700 1 -- 172.21.15.50:0/524763647 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ecc069730 con 0x7f3ecc101ff0 2024-04-23T18:17:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.926+0000 7f3ec97fa700 1 -- 172.21.15.50:0/524763647 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3ec000f450 con 0x7f3ecc101ff0 2024-04-23T18:17:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.926+0000 7f3ec97fa700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3eb40a3d50 0x7f3eb40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.926+0000 7f3ecbfff700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3eb40a3d50 0x7f3eb40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.926+0000 7f3ec97fa700 1 -- 172.21.15.50:0/524763647 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f3ec00d0290 con 0x7f3ecc101ff0 2024-04-23T18:17:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.927+0000 7f3ecbfff700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3eb40a3d50 0x7f3eb40a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc005950 tx=0x7f3ebc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:49.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:49.931+0000 7f3ec97fa700 1 -- 172.21.15.50:0/524763647 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3ec0014030 con 0x7f3ecc101ff0 2024-04-23T18:17:50.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.134+0000 7f3ed2510700 1 -- 172.21.15.50:0/524763647 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3ecc062e80 con 0x7f3eb40a3d50 2024-04-23T18:17:50.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.141+0000 7f3ec97fa700 1 -- 172.21.15.50:0/524763647 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258728 (secure 0 0 0) 0x7f3ecc062e80 con 0x7f3eb40a3d50 2024-04-23T18:17:50.139 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.145+0000 7f3ed2510700 1 -- 172.21.15.50:0/524763647 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3eb40a3d50 msgr2=0x7f3eb40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.145+0000 7f3ed2510700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3eb40a3d50 0x7f3eb40a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc005950 tx=0x7f3ebc00b410 comp rx=0 tx=0).stop 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.145+0000 7f3ed2510700 1 -- 172.21.15.50:0/524763647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ecc101ff0 msgr2=0x7f3ecc103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.145+0000 7f3ed2510700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ecc101ff0 0x7f3ecc103320 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000c9b0 tx=0x7f3ec000ccc0 comp rx=0 tx=0).stop 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.145+0000 7f3ed2510700 1 -- 172.21.15.50:0/524763647 shutdown_connections 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.145+0000 7f3ed2510700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f3eb40a3d50 0x7f3eb40a6210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.145+0000 7f3ed2510700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ecc1064e0 0x7f3ecc103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.145+0000 7f3ed2510700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ecc101640 0x7f3ecc102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.145+0000 7f3ed2510700 1 --2- 172.21.15.50:0/524763647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ecc101ff0 0x7f3ecc103320 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.145+0000 7f3ed2510700 1 -- 172.21.15.50:0/524763647 >> 172.21.15.50:0/524763647 conn(0x7f3ecc076c60 msgr2=0x7f3ecc10eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.145+0000 7f3ed2510700 1 -- 172.21.15.50:0/524763647 shutdown_connections 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:50.145+0000 7f3ed2510700 1 -- 172.21.15.50:0/524763647 wait complete. 2024-04-23T18:17:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-23T18:17:50.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:50 smithi188 ceph-mon[32694]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 95 B/s, 0 objects/s recovering 2024-04-23T18:17:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:50 smithi050 ceph-mon[29187]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 95 B/s, 0 objects/s recovering 2024-04-23T18:17:50.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:50 smithi177 ceph-mon[32308]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 95 B/s, 0 objects/s recovering 2024-04-23T18:17:50.912 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-04-23T18:17:48.519552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655017,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":567,"num_read_kb":404,"num_write":341,"num_write_kb":3050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181883,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":276,"ondisk_log_size":276,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":391892,"kb_used_data":13548,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549932,"statfs":{"total":1151940427776,"available":1151539130368,"internally_reserved":0,"allocated":13873152,"data_stored":7577631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":387233082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":78,"num_read_kb":38,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003058"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108834+0000","last_change":"2024-04-23T18:16:33.987102+0000","last_active":"2024-04-23T18:17:36.108834+0000","last_peered":"2024-04-23T18:17:36.108834+0000","last_clean":"2024-04-23T18:17:36.108834+0000","last_became_active":"2024-04-23T18:16:33.986716+0000","last_became_peered":"2024-04-23T18:16:33.986716+0000","last_unstale":"2024-04-23T18:17:36.108834+0000","last_undegraded":"2024-04-23T18:17:36.108834+0000","last_fullsized":"2024-04-23T18:17:36.108834+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:41.895219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106901+0000","last_change":"2024-04-23T18:16:33.995703+0000","last_active":"2024-04-23T18:17:36.106901+0000","last_peered":"2024-04-23T18:17:36.106901+0000","last_clean":"2024-04-23T18:17:36.106901+0000","last_became_active":"2024-04-23T18:16:33.995242+0000","last_became_peered":"2024-04-23T18:16:33.995242+0000","last_unstale":"2024-04-23T18:17:36.106901+0000","last_undegraded":"2024-04-23T18:17:36.106901+0000","last_fullsized":"2024-04-23T18:17:36.106901+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:57.813822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106713+0000","last_change":"2024-04-23T18:16:34.000864+0000","last_active":"2024-04-23T18:17:36.106713+0000","last_peered":"2024-04-23T18:17:36.106713+0000","last_clean":"2024-04-23T18:17:36.106713+0000","last_became_active":"2024-04-23T18:16:33.998934+0000","last_became_peered":"2024-04-23T18:16:33.998934+0000","last_unstale":"2024-04-23T18:17:36.106713+0000","last_undegraded":"2024-04-23T18:17:36.106713+0000","last_fullsized":"2024-04-23T18:17:36.106713+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:47:29.481697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212783+0000","last_change":"2024-04-23T18:16:33.989351+0000","last_active":"2024-04-23T18:17:36.212783+0000","last_peered":"2024-04-23T18:17:36.212783+0000","last_clean":"2024-04-23T18:17:36.212783+0000","last_became_active":"2024-04-23T18:16:33.989030+0000","last_became_peered":"2024-04-23T18:16:33.989030+0000","last_unstale":"2024-04-23T18:17:36.212783+0000","last_undegraded":"2024-04-23T18:17:36.212783+0000","last_fullsized":"2024-04-23T18:17:36.212783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:29:03.023675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.112341+0000","last_change":"2024-04-23T18:16:33.996062+0000","last_active":"2024-04-23T18:17:36.112341+0000","last_peered":"2024-04-23T18:17:36.112341+0000","last_clean":"2024-04-23T18:17:36.112341+0000","last_became_active":"2024-04-23T18:16:33.995732+0000","last_became_peered":"2024-04-23T18:16:33.995732+0000","last_unstale":"2024-04-23T18:17:36.112341+0000","last_undegraded":"2024-04-23T18:17:36.112341+0000","last_fullsized":"2024-04-23T18:17:36.112341+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:04.175580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212478+0000","last_change":"2024-04-23T18:16:33.989895+0000","last_active":"2024-04-23T18:17:36.212478+0000","last_peered":"2024-04-23T18:17:36.212478+0000","last_clean":"2024-04-23T18:17:36.212478+0000","last_became_active":"2024-04-23T18:16:33.989547+0000","last_became_peered":"2024-04-23T18:16:33.989547+0000","last_unstale":"2024-04-23T18:17:36.212478+0000","last_undegraded":"2024-04-23T18:17:36.212478+0000","last_fullsized":"2024-04-23T18:17:36.212478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:12:37.317974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108649+0000","last_change":"2024-04-23T18:16:33.999636+0000","last_active":"2024-04-23T18:17:36.108649+0000","last_peered":"2024-04-23T18:17:36.108649+0000","last_clean":"2024-04-23T18:17:36.108649+0000","last_became_active":"2024-04-23T18:16:33.999282+0000","last_became_peered":"2024-04-23T18:16:33.999282+0000","last_unstale":"2024-04-23T18:17:36.108649+0000","last_undegraded":"2024-04-23T18:17:36.108649+0000","last_fullsized":"2024-04-23T18:17:36.108649+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:28.841392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234476+0000","last_change":"2024-04-23T18:16:33.996602+0000","last_active":"2024-04-23T18:17:36.234476+0000","last_peered":"2024-04-23T18:17:36.234476+0000","last_clean":"2024-04-23T18:17:36.234476+0000","last_became_active":"2024-04-23T18:16:33.994401+0000","last_became_peered":"2024-04-23T18:16:33.994401+0000","last_unstale":"2024-04-23T18:17:36.234476+0000","last_undegraded":"2024-04-23T18:17:36.234476+0000","last_fullsized":"2024-04-23T18:17:36.234476+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:31:54.553537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109862+0000","last_change":"2024-04-23T18:16:33.994763+0000","last_active":"2024-04-23T18:17:36.109862+0000","last_peered":"2024-04-23T18:17:36.109862+0000","last_clean":"2024-04-23T18:17:36.109862+0000","last_became_active":"2024-04-23T18:16:33.994408+0000","last_became_peered":"2024-04-23T18:16:33.994408+0000","last_unstale":"2024-04-23T18:17:36.109862+0000","last_undegraded":"2024-04-23T18:17:36.109862+0000","last_fullsized":"2024-04-23T18:17:36.109862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:02.706160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.213087+0000","last_change":"2024-04-23T18:16:33.993435+0000","last_active":"2024-04-23T18:17:36.213087+0000","last_peered":"2024-04-23T18:17:36.213087+0000","last_clean":"2024-04-23T18:17:36.213087+0000","last_became_active":"2024-04-23T18:16:33.993108+0000","last_became_peered":"2024-04-23T18:16:33.993108+0000","last_unstale":"2024-04-23T18:17:36.213087+0000","last_undegraded":"2024-04-23T18:17:36.213087+0000","last_fullsized":"2024-04-23T18:17:36.213087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:28:32.584896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108998+0000","last_change":"2024-04-23T18:16:33.997526+0000","last_active":"2024-04-23T18:17:36.108998+0000","last_peered":"2024-04-23T18:17:36.108998+0000","last_clean":"2024-04-23T18:17:36.108998+0000","last_became_active":"2024-04-23T18:16:33.997189+0000","last_became_peered":"2024-04-23T18:16:33.997189+0000","last_unstale":"2024-04-23T18:17:36.108998+0000","last_undegraded":"2024-04-23T18:17:36.108998+0000","last_fullsized":"2024-04-23T18:17:36.108998+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:11:41.208261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212256+0000","last_change":"2024-04-23T18:16:33.989775+0000","last_active":"2024-04-23T18:17:36.212256+0000","last_peered":"2024-04-23T18:17:36.212256+0000","last_clean":"2024-04-23T18:17:36.212256+0000","last_became_active":"2024-04-23T18:16:33.989382+0000","last_became_peered":"2024-04-23T18:16:33.989382+0000","last_unstale":"2024-04-23T18:17:36.212256+0000","last_undegraded":"2024-04-23T18:17:36.212256+0000","last_fullsized":"2024-04-23T18:17:36.212256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:20:17.756258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.105971+0000","last_change":"2024-04-23T18:16:34.005871+0000","last_active":"2024-04-23T18:17:36.105971+0000","last_peered":"2024-04-23T18:17:36.105971+0000","last_clean":"2024-04-23T18:17:36.105971+0000","last_became_active":"2024-04-23T18:16:34.005218+0000","last_became_peered":"2024-04-23T18:16:34.005218+0000","last_unstale":"2024-04-23T18:17:36.105971+0000","last_undegraded":"2024-04-23T18:17:36.105971+0000","last_fullsized":"2024-04-23T18:17:36.105971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:30:03.864306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106475+0000","last_change":"2024-04-23T18:16:33.995054+0000","last_active":"2024-04-23T18:17:36.106475+0000","last_peered":"2024-04-23T18:17:36.106475+0000","last_clean":"2024-04-23T18:17:36.106475+0000","last_became_active":"2024-04-23T18:16:33.994389+0000","last_became_peered":"2024-04-23T18:16:33.994389+0000","last_unstale":"2024-04-23T18:17:36.106475+0000","last_undegraded":"2024-04-23T18:17:36.106475+0000","last_fullsized":"2024-04-23T18:17:36.106475+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:27.344616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106378+0000","last_change":"2024-04-23T18:16:33.984490+0000","last_active":"2024-04-23T18:17:36.106378+0000","last_peered":"2024-04-23T18:17:36.106378+0000","last_clean":"2024-04-23T18:17:36.106378+0000","last_became_active":"2024-04-23T18:16:33.984071+0000","last_became_peered":"2024-04-23T18:16:33.984071+0000","last_unstale":"2024-04-23T18:17:36.106378+0000","last_undegraded":"2024-04-23T18:17:36.106378+0000","last_fullsized":"2024-04-23T18:17:36.106378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:23.589290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109075+0000","last_change":"2024-04-23T18:16:34.002163+0000","last_active":"2024-04-23T18:17:36.109075+0000","last_peered":"2024-04-23T18:17:36.109075+0000","last_clean":"2024-04-23T18:17:36.109075+0000","last_became_active":"2024-04-23T18:16:34.001764+0000","last_became_peered":"2024-04-23T18:16:34.001764+0000","last_unstale":"2024-04-23T18:17:36.109075+0000","last_undegraded":"2024-04-23T18:17:36.109075+0000","last_fullsized":"2024-04-23T18:17:36.109075+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:16:03.616358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108906+0000","last_change":"2024-04-23T18:16:33.987586+0000","last_active":"2024-04-23T18:17:36.108906+0000","last_peered":"2024-04-23T18:17:36.108906+0000","last_clean":"2024-04-23T18:17:36.108906+0000","last_became_active":"2024-04-23T18:16:33.987266+0000","last_became_peered":"2024-04-23T18:16:33.987266+0000","last_unstale":"2024-04-23T18:17:36.108906+0000","last_undegraded":"2024-04-23T18:17:36.108906+0000","last_fullsized":"2024-04-23T18:17:36.108906+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:41.610529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108628+0000","last_change":"2024-04-23T18:16:33.992938+0000","last_active":"2024-04-23T18:17:36.108628+0000","last_peered":"2024-04-23T18:17:36.108628+0000","last_clean":"2024-04-23T18:17:36.108628+0000","last_became_active":"2024-04-23T18:16:33.990441+0000","last_became_peered":"2024-04-23T18:16:33.990441+0000","last_unstale":"2024-04-23T18:17:36.108628+0000","last_undegraded":"2024-04-23T18:17:36.108628+0000","last_fullsized":"2024-04-23T18:17:36.108628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:10.689333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111521+0000","last_change":"2024-04-23T18:16:33.994844+0000","last_active":"2024-04-23T18:17:36.111521+0000","last_peered":"2024-04-23T18:17:36.111521+0000","last_clean":"2024-04-23T18:17:36.111521+0000","last_became_active":"2024-04-23T18:16:33.994505+0000","last_became_peered":"2024-04-23T18:16:33.994505+0000","last_unstale":"2024-04-23T18:17:36.111521+0000","last_undegraded":"2024-04-23T18:17:36.111521+0000","last_fullsized":"2024-04-23T18:17:36.111521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:32:33.445266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108539+0000","last_change":"2024-04-23T18:16:32.958841+0000","last_active":"2024-04-23T18:17:36.108539+0000","last_peered":"2024-04-23T18:17:36.108539+0000","last_clean":"2024-04-23T18:17:36.108539+0000","last_became_active":"2024-04-23T18:16:32.958529+0000","last_became_peered":"2024-04-23T18:16:32.958529+0000","last_unstale":"2024-04-23T18:17:36.108539+0000","last_undegraded":"2024-04-23T18:17:36.108539+0000","last_fullsized":"2024-04-23T18:17:36.108539+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:47:09.933786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234746+0000","last_change":"2024-04-23T18:16:33.992108+0000","last_active":"2024-04-23T18:17:36.234746+0000","last_peered":"2024-04-23T18:17:36.234746+0000","last_clean":"2024-04-23T18:17:36.234746+0000","last_became_active":"2024-04-23T18:16:33.991751+0000","last_became_peered":"2024-04-23T18:16:33.991751+0000","last_unstale":"2024-04-23T18:17:36.234746+0000","last_undegraded":"2024-04-23T18:17:36.234746+0000","last_fullsized":"2024-04-23T18:17:36.234746+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:07:57.416263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234975+0000","last_change":"2024-04-23T18:16:32.960461+0000","last_active":"2024-04-23T18:17:36.234975+0000","last_peered":"2024-04-23T18:17:36.234975+0000","last_clean":"2024-04-23T18:17:36.234975+0000","last_became_active":"2024-04-23T18:16:32.960115+0000","last_became_peered":"2024-04-23T18:16:32.960115+0000","last_unstale":"2024-04-23T18:17:36.234975+0000","last_undegraded":"2024-04-23T18:17:36.234975+0000","last_fullsized":"2024-04-23T18:17:36.234975+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:30.533043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106850+0000","last_change":"2024-04-23T18:16:34.001831+0000","last_active":"2024-04-23T18:17:36.106850+0000","last_peered":"2024-04-23T18:17:36.106850+0000","last_clean":"2024-04-23T18:17:36.106850+0000","last_became_active":"2024-04-23T18:16:34.001486+0000","last_became_peered":"2024-04-23T18:16:34.001486+0000","last_unstale":"2024-04-23T18:17:36.106850+0000","last_undegraded":"2024-04-23T18:17:36.106850+0000","last_fullsized":"2024-04-23T18:17:36.106850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:06:40.452366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.113098+0000","last_change":"2024-04-23T18:16:32.960387+0000","last_active":"2024-04-23T18:17:36.113098+0000","last_peered":"2024-04-23T18:17:36.113098+0000","last_clean":"2024-04-23T18:17:36.113098+0000","last_became_active":"2024-04-23T18:16:32.960046+0000","last_became_peered":"2024-04-23T18:16:32.960046+0000","last_unstale":"2024-04-23T18:17:36.113098+0000","last_undegraded":"2024-04-23T18:17:36.113098+0000","last_fullsized":"2024-04-23T18:17:36.113098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:03.296653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109327+0000","last_change":"2024-04-23T18:16:33.996520+0000","last_active":"2024-04-23T18:17:36.109327+0000","last_peered":"2024-04-23T18:17:36.109327+0000","last_clean":"2024-04-23T18:17:36.109327+0000","last_became_active":"2024-04-23T18:16:33.995636+0000","last_became_peered":"2024-04-23T18:16:33.995636+0000","last_unstale":"2024-04-23T18:17:36.109327+0000","last_undegraded":"2024-04-23T18:17:36.109327+0000","last_fullsized":"2024-04-23T18:17:36.109327+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:29.665407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.112783+0000","last_change":"2024-04-23T18:16:32.956813+0000","last_active":"2024-04-23T18:17:36.112783+0000","last_peered":"2024-04-23T18:17:36.112783+0000","last_clean":"2024-04-23T18:17:36.112783+0000","last_became_active":"2024-04-23T18:16:32.956505+0000","last_became_peered":"2024-04-23T18:16:32.956505+0000","last_unstale":"2024-04-23T18:17:36.112783+0000","last_undegraded":"2024-04-23T18:17:36.112783+0000","last_fullsized":"2024-04-23T18:17:36.112783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:36:27.330353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110391+0000","last_change":"2024-04-23T18:16:33.999754+0000","last_active":"2024-04-23T18:17:36.110391+0000","last_peered":"2024-04-23T18:17:36.110391+0000","last_clean":"2024-04-23T18:17:36.110391+0000","last_became_active":"2024-04-23T18:16:33.999034+0000","last_became_peered":"2024-04-23T18:16:33.999034+0000","last_unstale":"2024-04-23T18:17:36.110391+0000","last_undegraded":"2024-04-23T18:17:36.110391+0000","last_fullsized":"2024-04-23T18:17:36.110391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:26:08.595745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":108,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:44.127571+0000","last_change":"2024-04-23T18:17:36.120371+0000","last_active":"2024-04-23T18:17:44.127571+0000","last_peered":"2024-04-23T18:17:44.127571+0000","last_clean":"2024-04-23T18:17:44.127571+0000","last_became_active":"2024-04-23T18:17:36.116149+0000","last_became_peered":"2024-04-23T18:17:36.116149+0000","last_unstale":"2024-04-23T18:17:44.127571+0000","last_undegraded":"2024-04-23T18:17:44.127571+0000","last_fullsized":"2024-04-23T18:17:44.127571+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:54.994254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.112158+0000","last_change":"2024-04-23T18:16:33.995166+0000","last_active":"2024-04-23T18:17:36.112158+0000","last_peered":"2024-04-23T18:17:36.112158+0000","last_clean":"2024-04-23T18:17:36.112158+0000","last_became_active":"2024-04-23T18:16:33.994347+0000","last_became_peered":"2024-04-23T18:16:33.994347+0000","last_unstale":"2024-04-23T18:17:36.112158+0000","last_undegraded":"2024-04-23T18:17:36.112158+0000","last_fullsized":"2024-04-23T18:17:36.112158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.305837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212357+0000","last_change":"2024-04-23T18:16:32.959358+0000","last_active":"2024-04-23T18:17:36.212357+0000","last_peered":"2024-04-23T18:17:36.212357+0000","last_clean":"2024-04-23T18:17:36.212357+0000","last_became_active":"2024-04-23T18:16:32.959036+0000","last_became_peered":"2024-04-23T18:16:32.959036+0000","last_unstale":"2024-04-23T18:17:36.212357+0000","last_undegraded":"2024-04-23T18:17:36.212357+0000","last_fullsized":"2024-04-23T18:17:36.212357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:03:15.734754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111598+0000","last_change":"2024-04-23T18:16:33.988578+0000","last_active":"2024-04-23T18:17:36.111598+0000","last_peered":"2024-04-23T18:17:36.111598+0000","last_clean":"2024-04-23T18:17:36.111598+0000","last_became_active":"2024-04-23T18:16:33.988248+0000","last_became_peered":"2024-04-23T18:16:33.988248+0000","last_unstale":"2024-04-23T18:17:36.111598+0000","last_undegraded":"2024-04-23T18:17:36.111598+0000","last_fullsized":"2024-04-23T18:17:36.111598+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:45:15.538971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.105727+0000","last_change":"2024-04-23T18:16:32.952945+0000","last_active":"2024-04-23T18:17:36.105727+0000","last_peered":"2024-04-23T18:17:36.105727+0000","last_clean":"2024-04-23T18:17:36.105727+0000","last_became_active":"2024-04-23T18:16:32.952602+0000","last_became_peered":"2024-04-23T18:16:32.952602+0000","last_unstale":"2024-04-23T18:17:36.105727+0000","last_undegraded":"2024-04-23T18:17:36.105727+0000","last_fullsized":"2024-04-23T18:17:36.105727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:20.037431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234542+0000","last_change":"2024-04-23T18:16:33.996897+0000","last_active":"2024-04-23T18:17:36.234542+0000","last_peered":"2024-04-23T18:17:36.234542+0000","last_clean":"2024-04-23T18:17:36.234542+0000","last_became_active":"2024-04-23T18:16:33.996214+0000","last_became_peered":"2024-04-23T18:16:33.996214+0000","last_unstale":"2024-04-23T18:17:36.234542+0000","last_undegraded":"2024-04-23T18:17:36.234542+0000","last_fullsized":"2024-04-23T18:17:36.234542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:34.931697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109970+0000","last_change":"2024-04-23T18:16:32.959637+0000","last_active":"2024-04-23T18:17:36.109970+0000","last_peered":"2024-04-23T18:17:36.109970+0000","last_clean":"2024-04-23T18:17:36.109970+0000","last_became_active":"2024-04-23T18:16:32.959332+0000","last_became_peered":"2024-04-23T18:16:32.959332+0000","last_unstale":"2024-04-23T18:17:36.109970+0000","last_undegraded":"2024-04-23T18:17:36.109970+0000","last_fullsized":"2024-04-23T18:17:36.109970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:30:49.089101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108255+0000","last_change":"2024-04-23T18:16:33.986288+0000","last_active":"2024-04-23T18:17:36.108255+0000","last_peered":"2024-04-23T18:17:36.108255+0000","last_clean":"2024-04-23T18:17:36.108255+0000","last_became_active":"2024-04-23T18:16:33.985866+0000","last_became_peered":"2024-04-23T18:16:33.985866+0000","last_unstale":"2024-04-23T18:17:36.108255+0000","last_undegraded":"2024-04-23T18:17:36.108255+0000","last_fullsized":"2024-04-23T18:17:36.108255+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:30:17.675952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:45.128206+0000","last_change":"2024-04-23T18:16:32.958466+0000","last_active":"2024-04-23T18:17:45.128206+0000","last_peered":"2024-04-23T18:17:45.128206+0000","last_clean":"2024-04-23T18:17:45.128206+0000","last_became_active":"2024-04-23T18:16:32.958155+0000","last_became_peered":"2024-04-23T18:16:32.958155+0000","last_unstale":"2024-04-23T18:17:45.128206+0000","last_undegraded":"2024-04-23T18:17:45.128206+0000","last_fullsized":"2024-04-23T18:17:45.128206+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:15:49.073934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109772+0000","last_change":"2024-04-23T18:16:33.996156+0000","last_active":"2024-04-23T18:17:36.109772+0000","last_peered":"2024-04-23T18:17:36.109772+0000","last_clean":"2024-04-23T18:17:36.109772+0000","last_became_active":"2024-04-23T18:16:33.995711+0000","last_became_peered":"2024-04-23T18:16:33.995711+0000","last_unstale":"2024-04-23T18:17:36.109772+0000","last_undegraded":"2024-04-23T18:17:36.109772+0000","last_fullsized":"2024-04-23T18:17:36.109772+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:45.272598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.217460+0000","last_change":"2024-04-23T18:17:36.217460+0000","last_active":"2024-04-23T18:17:36.217460+0000","last_peered":"2024-04-23T18:17:36.217460+0000","last_clean":"2024-04-23T18:17:36.217460+0000","last_became_active":"2024-04-23T18:17:36.214196+0000","last_became_peered":"2024-04-23T18:17:36.214196+0000","last_unstale":"2024-04-23T18:17:36.217460+0000","last_undegraded":"2024-04-23T18:17:36.217460+0000","last_fullsized":"2024-04-23T18:17:36.217460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:34:55.557943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.114238+0000","last_change":"2024-04-23T18:17:36.114238+0000","last_active":"2024-04-23T18:17:36.114238+0000","last_peered":"2024-04-23T18:17:36.114238+0000","last_clean":"2024-04-23T18:17:36.114238+0000","last_became_active":"2024-04-23T18:17:36.113810+0000","last_became_peered":"2024-04-23T18:17:36.113810+0000","last_unstale":"2024-04-23T18:17:36.114238+0000","last_undegraded":"2024-04-23T18:17:36.114238+0000","last_fullsized":"2024-04-23T18:17:36.114238+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:53.854596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110496+0000","last_change":"2024-04-23T18:16:32.966725+0000","last_active":"2024-04-23T18:17:36.110496+0000","last_peered":"2024-04-23T18:17:36.110496+0000","last_clean":"2024-04-23T18:17:36.110496+0000","last_became_active":"2024-04-23T18:16:32.966304+0000","last_became_peered":"2024-04-23T18:16:32.966304+0000","last_unstale":"2024-04-23T18:17:36.110496+0000","last_undegraded":"2024-04-23T18:17:36.110496+0000","last_fullsized":"2024-04-23T18:17:36.110496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:45.191051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109512+0000","last_change":"2024-04-23T18:16:33.994667+0000","last_active":"2024-04-23T18:17:36.109512+0000","last_peered":"2024-04-23T18:17:36.109512+0000","last_clean":"2024-04-23T18:17:36.109512+0000","last_became_active":"2024-04-23T18:16:33.994305+0000","last_became_peered":"2024-04-23T18:16:33.994305+0000","last_unstale":"2024-04-23T18:17:36.109512+0000","last_undegraded":"2024-04-23T18:17:36.109512+0000","last_fullsized":"2024-04-23T18:17:36.109512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:23.807075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'29","reported_seq":194,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:46.128448+0000","last_change":"2024-04-23T18:16:32.957946+0000","last_active":"2024-04-23T18:17:46.128448+0000","last_peered":"2024-04-23T18:17:46.128448+0000","last_clean":"2024-04-23T18:17:46.128448+0000","last_became_active":"2024-04-23T18:16:32.957478+0000","last_became_peered":"2024-04-23T18:16:32.957478+0000","last_unstale":"2024-04-23T18:17:46.128448+0000","last_undegraded":"2024-04-23T18:17:46.128448+0000","last_fullsized":"2024-04-23T18:17:46.128448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:16.850604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60023,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":131,"num_write":30,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.112890+0000","last_change":"2024-04-23T18:16:33.989537+0000","last_active":"2024-04-23T18:17:36.112890+0000","last_peered":"2024-04-23T18:17:36.112890+0000","last_clean":"2024-04-23T18:17:36.112890+0000","last_became_active":"2024-04-23T18:16:33.989180+0000","last_became_peered":"2024-04-23T18:16:33.989180+0000","last_unstale":"2024-04-23T18:17:36.112890+0000","last_undegraded":"2024-04-23T18:17:36.112890+0000","last_fullsized":"2024-04-23T18:17:36.112890+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:53:18.700347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111503+0000","last_change":"2024-04-23T18:16:32.967314+0000","last_active":"2024-04-23T18:17:36.111503+0000","last_peered":"2024-04-23T18:17:36.111503+0000","last_clean":"2024-04-23T18:17:36.111503+0000","last_became_active":"2024-04-23T18:16:32.966861+0000","last_became_peered":"2024-04-23T18:16:32.966861+0000","last_unstale":"2024-04-23T18:17:36.111503+0000","last_undegraded":"2024-04-23T18:17:36.111503+0000","last_fullsized":"2024-04-23T18:17:36.111503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:00:25.606402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.105698+0000","last_change":"2024-04-23T18:16:33.992593+0000","last_active":"2024-04-23T18:17:36.105698+0000","last_peered":"2024-04-23T18:17:36.105698+0000","last_clean":"2024-04-23T18:17:36.105698+0000","last_became_active":"2024-04-23T18:16:33.992228+0000","last_became_peered":"2024-04-23T18:16:33.992228+0000","last_unstale":"2024-04-23T18:17:36.105698+0000","last_undegraded":"2024-04-23T18:17:36.105698+0000","last_fullsized":"2024-04-23T18:17:36.105698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:12:23.836829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110368+0000","last_change":"2024-04-23T18:16:32.961740+0000","last_active":"2024-04-23T18:17:36.110368+0000","last_peered":"2024-04-23T18:17:36.110368+0000","last_clean":"2024-04-23T18:17:36.110368+0000","last_became_active":"2024-04-23T18:16:32.961331+0000","last_became_peered":"2024-04-23T18:16:32.961331+0000","last_unstale":"2024-04-23T18:17:36.110368+0000","last_undegraded":"2024-04-23T18:17:36.110368+0000","last_fullsized":"2024-04-23T18:17:36.110368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:59:17.665288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111151+0000","last_change":"2024-04-23T18:17:36.111151+0000","last_active":"2024-04-23T18:17:36.111151+0000","last_peered":"2024-04-23T18:17:36.111151+0000","last_clean":"2024-04-23T18:17:36.111151+0000","last_became_active":"2024-04-23T18:17:36.110770+0000","last_became_peered":"2024-04-23T18:17:36.110770+0000","last_unstale":"2024-04-23T18:17:36.111151+0000","last_undegraded":"2024-04-23T18:17:36.111151+0000","last_fullsized":"2024-04-23T18:17:36.111151+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:01:03.456738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'4","reported_seq":96,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:46.068677+0000","last_change":"2024-04-23T18:16:32.954935+0000","last_active":"2024-04-23T18:17:46.068677+0000","last_peered":"2024-04-23T18:17:46.068677+0000","last_clean":"2024-04-23T18:17:46.068677+0000","last_became_active":"2024-04-23T18:16:32.954595+0000","last_became_peered":"2024-04-23T18:16:32.954595+0000","last_unstale":"2024-04-23T18:17:46.068677+0000","last_undegraded":"2024-04-23T18:17:46.068677+0000","last_fullsized":"2024-04-23T18:17:46.068677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:22.954101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108474+0000","last_change":"2024-04-23T18:16:34.000215+0000","last_active":"2024-04-23T18:17:36.108474+0000","last_peered":"2024-04-23T18:17:36.108474+0000","last_clean":"2024-04-23T18:17:36.108474+0000","last_became_active":"2024-04-23T18:16:33.999855+0000","last_became_peered":"2024-04-23T18:16:33.999855+0000","last_unstale":"2024-04-23T18:17:36.108474+0000","last_undegraded":"2024-04-23T18:17:36.108474+0000","last_fullsized":"2024-04-23T18:17:36.108474+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:14.686916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.128101+0000","last_change":"2024-04-23T18:17:36.127721+0000","last_active":"2024-04-23T18:17:36.128101+0000","last_peered":"2024-04-23T18:17:36.128101+0000","last_clean":"2024-04-23T18:17:36.128101+0000","last_became_active":"2024-04-23T18:17:36.113735+0000","last_became_peered":"2024-04-23T18:17:36.113735+0000","last_unstale":"2024-04-23T18:17:36.128101+0000","last_undegraded":"2024-04-23T18:17:36.128101+0000","last_fullsized":"2024-04-23T18:17:36.128101+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:56:41.395925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.214244+0000","last_change":"2024-04-23T18:17:36.214244+0000","last_active":"2024-04-23T18:17:36.214244+0000","last_peered":"2024-04-23T18:17:36.214244+0000","last_clean":"2024-04-23T18:17:36.214244+0000","last_became_active":"2024-04-23T18:17:36.213897+0000","last_became_peered":"2024-04-23T18:17:36.213897+0000","last_unstale":"2024-04-23T18:17:36.214244+0000","last_undegraded":"2024-04-23T18:17:36.214244+0000","last_fullsized":"2024-04-23T18:17:36.214244+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:44:24.935747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111781+0000","last_change":"2024-04-23T18:16:32.959253+0000","last_active":"2024-04-23T18:17:36.111781+0000","last_peered":"2024-04-23T18:17:36.111781+0000","last_clean":"2024-04-23T18:17:36.111781+0000","last_became_active":"2024-04-23T18:16:32.958929+0000","last_became_peered":"2024-04-23T18:16:32.958929+0000","last_unstale":"2024-04-23T18:17:36.111781+0000","last_undegraded":"2024-04-23T18:17:36.111781+0000","last_fullsized":"2024-04-23T18:17:36.111781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:36.317516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108748+0000","last_change":"2024-04-23T18:16:33.979464+0000","last_active":"2024-04-23T18:17:36.108748+0000","last_peered":"2024-04-23T18:17:36.108748+0000","last_clean":"2024-04-23T18:17:36.108748+0000","last_became_active":"2024-04-23T18:16:33.979106+0000","last_became_peered":"2024-04-23T18:16:33.979106+0000","last_unstale":"2024-04-23T18:17:36.108748+0000","last_undegraded":"2024-04-23T18:17:36.108748+0000","last_fullsized":"2024-04-23T18:17:36.108748+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:00.125514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108469+0000","last_change":"2024-04-23T18:16:32.960812+0000","last_active":"2024-04-23T18:17:36.108469+0000","last_peered":"2024-04-23T18:17:36.108469+0000","last_clean":"2024-04-23T18:17:36.108469+0000","last_became_active":"2024-04-23T18:16:32.959807+0000","last_became_peered":"2024-04-23T18:16:32.959807+0000","last_unstale":"2024-04-23T18:17:36.108469+0000","last_undegraded":"2024-04-23T18:17:36.108469+0000","last_fullsized":"2024-04-23T18:17:36.108469+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:45.097443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212762+0000","last_change":"2024-04-23T18:16:33.992891+0000","last_active":"2024-04-23T18:17:36.212762+0000","last_peered":"2024-04-23T18:17:36.212762+0000","last_clean":"2024-04-23T18:17:36.212762+0000","last_became_active":"2024-04-23T18:16:33.992559+0000","last_became_peered":"2024-04-23T18:16:33.992559+0000","last_unstale":"2024-04-23T18:17:36.212762+0000","last_undegraded":"2024-04-23T18:17:36.212762+0000","last_fullsized":"2024-04-23T18:17:36.212762+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:48.615158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108792+0000","last_change":"2024-04-23T18:16:32.957015+0000","last_active":"2024-04-23T18:17:36.108792+0000","last_peered":"2024-04-23T18:17:36.108792+0000","last_clean":"2024-04-23T18:17:36.108792+0000","last_became_active":"2024-04-23T18:16:32.956690+0000","last_became_peered":"2024-04-23T18:16:32.956690+0000","last_unstale":"2024-04-23T18:17:36.108792+0000","last_undegraded":"2024-04-23T18:17:36.108792+0000","last_fullsized":"2024-04-23T18:17:36.108792+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:20:42.087521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212425+0000","last_change":"2024-04-23T18:16:33.998906+0000","last_active":"2024-04-23T18:17:36.212425+0000","last_peered":"2024-04-23T18:17:36.212425+0000","last_clean":"2024-04-23T18:17:36.212425+0000","last_became_active":"2024-04-23T18:16:33.997297+0000","last_became_peered":"2024-04-23T18:16:33.997297+0000","last_unstale":"2024-04-23T18:17:36.212425+0000","last_undegraded":"2024-04-23T18:17:36.212425+0000","last_fullsized":"2024-04-23T18:17:36.212425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:06.668830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.113962+0000","last_change":"2024-04-23T18:17:36.113962+0000","last_active":"2024-04-23T18:17:36.113962+0000","last_peered":"2024-04-23T18:17:36.113962+0000","last_clean":"2024-04-23T18:17:36.113962+0000","last_became_active":"2024-04-23T18:17:36.113573+0000","last_became_peered":"2024-04-23T18:17:36.113573+0000","last_unstale":"2024-04-23T18:17:36.113962+0000","last_undegraded":"2024-04-23T18:17:36.113962+0000","last_fullsized":"2024-04-23T18:17:36.113962+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:47:25.408144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":90,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:43.063217+0000","last_change":"2024-04-23T18:16:32.961423+0000","last_active":"2024-04-23T18:17:43.063217+0000","last_peered":"2024-04-23T18:17:43.063217+0000","last_clean":"2024-04-23T18:17:43.063217+0000","last_became_active":"2024-04-23T18:16:32.961116+0000","last_became_peered":"2024-04-23T18:16:32.961116+0000","last_unstale":"2024-04-23T18:17:43.063217+0000","last_undegraded":"2024-04-23T18:17:43.063217+0000","last_fullsized":"2024-04-23T18:17:43.063217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:28.525642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234845+0000","last_change":"2024-04-23T18:16:33.990747+0000","last_active":"2024-04-23T18:17:36.234845+0000","last_peered":"2024-04-23T18:17:36.234845+0000","last_clean":"2024-04-23T18:17:36.234845+0000","last_became_active":"2024-04-23T18:16:33.989972+0000","last_became_peered":"2024-04-23T18:16:33.989972+0000","last_unstale":"2024-04-23T18:17:36.234845+0000","last_undegraded":"2024-04-23T18:17:36.234845+0000","last_fullsized":"2024-04-23T18:17:36.234845+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:04.667446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.212337+0000","last_change":"2024-04-23T18:16:32.952821+0000","last_active":"2024-04-23T18:17:36.212337+0000","last_peered":"2024-04-23T18:17:36.212337+0000","last_clean":"2024-04-23T18:17:36.212337+0000","last_became_active":"2024-04-23T18:16:32.952470+0000","last_became_peered":"2024-04-23T18:16:32.952470+0000","last_unstale":"2024-04-23T18:17:36.212337+0000","last_undegraded":"2024-04-23T18:17:36.212337+0000","last_fullsized":"2024-04-23T18:17:36.212337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:19.392670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":353,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108520+0000","last_change":"2024-04-23T18:15:25.867370+0000","last_active":"2024-04-23T18:17:36.108520+0000","last_peered":"2024-04-23T18:17:36.108520+0000","last_clean":"2024-04-23T18:17:36.108520+0000","last_became_active":"2024-04-23T18:15:25.866815+0000","last_became_peered":"2024-04-23T18:15:25.866815+0000","last_unstale":"2024-04-23T18:17:36.108520+0000","last_undegraded":"2024-04-23T18:17:36.108520+0000","last_fullsized":"2024-04-23T18:17:36.108520+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_clean_scrub_stamp":"2024-04-23T18:12:02.913350+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:42.482422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.105597+0000","last_change":"2024-04-23T18:16:33.992988+0000","last_active":"2024-04-23T18:17:36.105597+0000","last_peered":"2024-04-23T18:17:36.105597+0000","last_clean":"2024-04-23T18:17:36.105597+0000","last_became_active":"2024-04-23T18:16:33.992650+0000","last_became_peered":"2024-04-23T18:16:33.992650+0000","last_unstale":"2024-04-23T18:17:36.105597+0000","last_undegraded":"2024-04-23T18:17:36.105597+0000","last_fullsized":"2024-04-23T18:17:36.105597+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:54.054708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108693+0000","last_change":"2024-04-23T18:16:32.957318+0000","last_active":"2024-04-23T18:17:36.108693+0000","last_peered":"2024-04-23T18:17:36.108693+0000","last_clean":"2024-04-23T18:17:36.108693+0000","last_became_active":"2024-04-23T18:16:32.956973+0000","last_became_peered":"2024-04-23T18:16:32.956973+0000","last_unstale":"2024-04-23T18:17:36.108693+0000","last_undegraded":"2024-04-23T18:17:36.108693+0000","last_fullsized":"2024-04-23T18:17:36.108693+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:35.684825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.117673+0000","last_change":"2024-04-23T18:17:36.117253+0000","last_active":"2024-04-23T18:17:36.117673+0000","last_peered":"2024-04-23T18:17:36.117673+0000","last_clean":"2024-04-23T18:17:36.117673+0000","last_became_active":"2024-04-23T18:17:36.111210+0000","last_became_peered":"2024-04-23T18:17:36.111210+0000","last_unstale":"2024-04-23T18:17:36.117673+0000","last_undegraded":"2024-04-23T18:17:36.117673+0000","last_fullsized":"2024-04-23T18:17:36.117673+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:31.599001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106159+0000","last_change":"2024-04-23T18:16:33.996345+0000","last_active":"2024-04-23T18:17:36.106159+0000","last_peered":"2024-04-23T18:17:36.106159+0000","last_clean":"2024-04-23T18:17:36.106159+0000","last_became_active":"2024-04-23T18:16:33.993802+0000","last_became_peered":"2024-04-23T18:16:33.993802+0000","last_unstale":"2024-04-23T18:17:36.106159+0000","last_undegraded":"2024-04-23T18:17:36.106159+0000","last_fullsized":"2024-04-23T18:17:36.106159+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:32.410330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108694+0000","last_change":"2024-04-23T18:16:32.957953+0000","last_active":"2024-04-23T18:17:36.108694+0000","last_peered":"2024-04-23T18:17:36.108694+0000","last_clean":"2024-04-23T18:17:36.108694+0000","last_became_active":"2024-04-23T18:16:32.957616+0000","last_became_peered":"2024-04-23T18:16:32.957616+0000","last_unstale":"2024-04-23T18:17:36.108694+0000","last_undegraded":"2024-04-23T18:17:36.108694+0000","last_fullsized":"2024-04-23T18:17:36.108694+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:15.485373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234995+0000","last_change":"2024-04-23T18:16:33.985685+0000","last_active":"2024-04-23T18:17:36.234995+0000","last_peered":"2024-04-23T18:17:36.234995+0000","last_clean":"2024-04-23T18:17:36.234995+0000","last_became_active":"2024-04-23T18:16:33.984631+0000","last_became_peered":"2024-04-23T18:16:33.984631+0000","last_unstale":"2024-04-23T18:17:36.234995+0000","last_undegraded":"2024-04-23T18:17:36.234995+0000","last_fullsized":"2024-04-23T18:17:36.234995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:45.893285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.213198+0000","last_change":"2024-04-23T18:16:32.954281+0000","last_active":"2024-04-23T18:17:36.213198+0000","last_peered":"2024-04-23T18:17:36.213198+0000","last_clean":"2024-04-23T18:17:36.213198+0000","last_became_active":"2024-04-23T18:16:32.953988+0000","last_became_peered":"2024-04-23T18:16:32.953988+0000","last_unstale":"2024-04-23T18:17:36.213198+0000","last_undegraded":"2024-04-23T18:17:36.213198+0000","last_fullsized":"2024-04-23T18:17:36.213198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:09.049861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106785+0000","last_change":"2024-04-23T18:16:33.994860+0000","last_active":"2024-04-23T18:17:36.106785+0000","last_peered":"2024-04-23T18:17:36.106785+0000","last_clean":"2024-04-23T18:17:36.106785+0000","last_became_active":"2024-04-23T18:16:33.993438+0000","last_became_peered":"2024-04-23T18:16:33.993438+0000","last_unstale":"2024-04-23T18:17:36.106785+0000","last_undegraded":"2024-04-23T18:17:36.106785+0000","last_fullsized":"2024-04-23T18:17:36.106785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:55.565873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108913+0000","last_change":"2024-04-23T18:16:32.959251+0000","last_active":"2024-04-23T18:17:36.108913+0000","last_peered":"2024-04-23T18:17:36.108913+0000","last_clean":"2024-04-23T18:17:36.108913+0000","last_became_active":"2024-04-23T18:16:32.958937+0000","last_became_peered":"2024-04-23T18:16:32.958937+0000","last_unstale":"2024-04-23T18:17:36.108913+0000","last_undegraded":"2024-04-23T18:17:36.108913+0000","last_fullsized":"2024-04-23T18:17:36.108913+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.126634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111905+0000","last_change":"2024-04-23T18:16:33.995480+0000","last_active":"2024-04-23T18:17:36.111905+0000","last_peered":"2024-04-23T18:17:36.111905+0000","last_clean":"2024-04-23T18:17:36.111905+0000","last_became_active":"2024-04-23T18:16:33.994361+0000","last_became_peered":"2024-04-23T18:16:33.994361+0000","last_unstale":"2024-04-23T18:17:36.111905+0000","last_undegraded":"2024-04-23T18:17:36.111905+0000","last_fullsized":"2024-04-23T18:17:36.111905+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:59:35.240562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234665+0000","last_change":"2024-04-23T18:16:32.967787+0000","last_active":"2024-04-23T18:17:36.234665+0000","last_peered":"2024-04-23T18:17:36.234665+0000","last_clean":"2024-04-23T18:17:36.234665+0000","last_became_active":"2024-04-23T18:16:32.967463+0000","last_became_peered":"2024-04-23T18:16:32.967463+0000","last_unstale":"2024-04-23T18:17:36.234665+0000","last_undegraded":"2024-04-23T18:17:36.234665+0000","last_fullsized":"2024-04-23T18:17:36.234665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:58.743682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109014+0000","last_change":"2024-04-23T18:16:33.980558+0000","last_active":"2024-04-23T18:17:36.109014+0000","last_peered":"2024-04-23T18:17:36.109014+0000","last_clean":"2024-04-23T18:17:36.109014+0000","last_became_active":"2024-04-23T18:16:33.980146+0000","last_became_peered":"2024-04-23T18:16:33.980146+0000","last_unstale":"2024-04-23T18:17:36.109014+0000","last_undegraded":"2024-04-23T18:17:36.109014+0000","last_fullsized":"2024-04-23T18:17:36.109014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:58.001833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110646+0000","last_change":"2024-04-23T18:16:32.962139+0000","last_active":"2024-04-23T18:17:36.110646+0000","last_peered":"2024-04-23T18:17:36.110646+0000","last_clean":"2024-04-23T18:17:36.110646+0000","last_became_active":"2024-04-23T18:16:32.961785+0000","last_became_peered":"2024-04-23T18:16:32.961785+0000","last_unstale":"2024-04-23T18:17:36.110646+0000","last_undegraded":"2024-04-23T18:17:36.110646+0000","last_fullsized":"2024-04-23T18:17:36.110646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:09.619414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.214255+0000","last_change":"2024-04-23T18:17:36.214255+0000","last_active":"2024-04-23T18:17:36.214255+0000","last_peered":"2024-04-23T18:17:36.214255+0000","last_clean":"2024-04-23T18:17:36.214255+0000","last_became_active":"2024-04-23T18:17:36.213832+0000","last_became_peered":"2024-04-23T18:17:36.213832+0000","last_unstale":"2024-04-23T18:17:36.214255+0000","last_undegraded":"2024-04-23T18:17:36.214255+0000","last_fullsized":"2024-04-23T18:17:36.214255+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:59:35.497710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.112439+0000","last_change":"2024-04-23T18:16:32.956965+0000","last_active":"2024-04-23T18:17:36.112439+0000","last_peered":"2024-04-23T18:17:36.112439+0000","last_clean":"2024-04-23T18:17:36.112439+0000","last_became_active":"2024-04-23T18:16:32.956672+0000","last_became_peered":"2024-04-23T18:16:32.956672+0000","last_unstale":"2024-04-23T18:17:36.112439+0000","last_undegraded":"2024-04-23T18:17:36.112439+0000","last_fullsized":"2024-04-23T18:17:36.112439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:00:44.470809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109231+0000","last_change":"2024-04-23T18:16:34.002711+0000","last_active":"2024-04-23T18:17:36.109231+0000","last_peered":"2024-04-23T18:17:36.109231+0000","last_clean":"2024-04-23T18:17:36.109231+0000","last_became_active":"2024-04-23T18:16:34.002365+0000","last_became_peered":"2024-04-23T18:16:34.002365+0000","last_unstale":"2024-04-23T18:17:36.109231+0000","last_undegraded":"2024-04-23T18:17:36.109231+0000","last_fullsized":"2024-04-23T18:17:36.109231+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:52:11.982735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111252+0000","last_change":"2024-04-23T18:16:32.965121+0000","last_active":"2024-04-23T18:17:36.111252+0000","last_peered":"2024-04-23T18:17:36.111252+0000","last_clean":"2024-04-23T18:17:36.111252+0000","last_became_active":"2024-04-23T18:16:32.964732+0000","last_became_peered":"2024-04-23T18:16:32.964732+0000","last_unstale":"2024-04-23T18:17:36.111252+0000","last_undegraded":"2024-04-23T18:17:36.111252+0000","last_fullsized":"2024-04-23T18:17:36.111252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:19:34.690245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109037+0000","last_change":"2024-04-23T18:16:33.996357+0000","last_active":"2024-04-23T18:17:36.109037+0000","last_peered":"2024-04-23T18:17:36.109037+0000","last_clean":"2024-04-23T18:17:36.109037+0000","last_became_active":"2024-04-23T18:16:33.996017+0000","last_became_peered":"2024-04-23T18:16:33.996017+0000","last_unstale":"2024-04-23T18:17:36.109037+0000","last_undegraded":"2024-04-23T18:17:36.109037+0000","last_fullsized":"2024-04-23T18:17:36.109037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:20:57.321228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110284+0000","last_change":"2024-04-23T18:16:32.961705+0000","last_active":"2024-04-23T18:17:36.110284+0000","last_peered":"2024-04-23T18:17:36.110284+0000","last_clean":"2024-04-23T18:17:36.110284+0000","last_became_active":"2024-04-23T18:16:32.961342+0000","last_became_peered":"2024-04-23T18:16:32.961342+0000","last_unstale":"2024-04-23T18:17:36.110284+0000","last_undegraded":"2024-04-23T18:17:36.110284+0000","last_fullsized":"2024-04-23T18:17:36.110284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:51.340910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108878+0000","last_change":"2024-04-23T18:16:33.992944+0000","last_active":"2024-04-23T18:17:36.108878+0000","last_peered":"2024-04-23T18:17:36.108878+0000","last_clean":"2024-04-23T18:17:36.108878+0000","last_became_active":"2024-04-23T18:16:33.992579+0000","last_became_peered":"2024-04-23T18:16:33.992579+0000","last_unstale":"2024-04-23T18:17:36.108878+0000","last_undegraded":"2024-04-23T18:17:36.108878+0000","last_fullsized":"2024-04-23T18:17:36.108878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:13:37.829588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106027+0000","last_change":"2024-04-23T18:16:32.955192+0000","last_active":"2024-04-23T18:17:36.106027+0000","last_peered":"2024-04-23T18:17:36.106027+0000","last_clean":"2024-04-23T18:17:36.106027+0000","last_became_active":"2024-04-23T18:16:32.954860+0000","last_became_peered":"2024-04-23T18:16:32.954860+0000","last_unstale":"2024-04-23T18:17:36.106027+0000","last_undegraded":"2024-04-23T18:17:36.106027+0000","last_fullsized":"2024-04-23T18:17:36.106027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:20.796060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108864+0000","last_change":"2024-04-23T18:16:33.986211+0000","last_active":"2024-04-23T18:17:36.108864+0000","last_peered":"2024-04-23T18:17:36.108864+0000","last_clean":"2024-04-23T18:17:36.108864+0000","last_became_active":"2024-04-23T18:16:33.985679+0000","last_became_peered":"2024-04-23T18:16:33.985679+0000","last_unstale":"2024-04-23T18:17:36.108864+0000","last_undegraded":"2024-04-23T18:17:36.108864+0000","last_fullsized":"2024-04-23T18:17:36.108864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:03:38.797864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109118+0000","last_change":"2024-04-23T18:16:33.979462+0000","last_active":"2024-04-23T18:17:36.109118+0000","last_peered":"2024-04-23T18:17:36.109118+0000","last_clean":"2024-04-23T18:17:36.109118+0000","last_became_active":"2024-04-23T18:16:33.979114+0000","last_became_peered":"2024-04-23T18:16:33.979114+0000","last_unstale":"2024-04-23T18:17:36.109118+0000","last_undegraded":"2024-04-23T18:17:36.109118+0000","last_fullsized":"2024-04-23T18:17:36.109118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:16:49.708240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.111077+0000","last_change":"2024-04-23T18:16:33.979255+0000","last_active":"2024-04-23T18:17:36.111077+0000","last_peered":"2024-04-23T18:17:36.111077+0000","last_clean":"2024-04-23T18:17:36.111077+0000","last_became_active":"2024-04-23T18:16:33.978934+0000","last_became_peered":"2024-04-23T18:16:33.978934+0000","last_unstale":"2024-04-23T18:17:36.111077+0000","last_undegraded":"2024-04-23T18:17:36.111077+0000","last_fullsized":"2024-04-23T18:17:36.111077+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:10.104564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110531+0000","last_change":"2024-04-23T18:16:33.981877+0000","last_active":"2024-04-23T18:17:36.110531+0000","last_peered":"2024-04-23T18:17:36.110531+0000","last_clean":"2024-04-23T18:17:36.110531+0000","last_became_active":"2024-04-23T18:16:33.981536+0000","last_became_peered":"2024-04-23T18:16:33.981536+0000","last_unstale":"2024-04-23T18:17:36.110531+0000","last_undegraded":"2024-04-23T18:17:36.110531+0000","last_fullsized":"2024-04-23T18:17:36.110531+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:00:15.056779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106669+0000","last_change":"2024-04-23T18:16:33.999090+0000","last_active":"2024-04-23T18:17:36.106669+0000","last_peered":"2024-04-23T18:17:36.106669+0000","last_clean":"2024-04-23T18:17:36.106669+0000","last_became_active":"2024-04-23T18:16:33.998513+0000","last_became_peered":"2024-04-23T18:16:33.998513+0000","last_unstale":"2024-04-23T18:17:36.106669+0000","last_undegraded":"2024-04-23T18:17:36.106669+0000","last_fullsized":"2024-04-23T18:17:36.106669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:14.115406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106114+0000","last_change":"2024-04-23T18:16:33.993375+0000","last_active":"2024-04-23T18:17:36.106114+0000","last_peered":"2024-04-23T18:17:36.106114+0000","last_clean":"2024-04-23T18:17:36.106114+0000","last_became_active":"2024-04-23T18:16:33.991841+0000","last_became_peered":"2024-04-23T18:16:33.991841+0000","last_unstale":"2024-04-23T18:17:36.106114+0000","last_undegraded":"2024-04-23T18:17:36.106114+0000","last_fullsized":"2024-04-23T18:17:36.106114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:12.777495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109706+0000","last_change":"2024-04-23T18:16:33.983621+0000","last_active":"2024-04-23T18:17:36.109706+0000","last_peered":"2024-04-23T18:17:36.109706+0000","last_clean":"2024-04-23T18:17:36.109706+0000","last_became_active":"2024-04-23T18:16:33.983356+0000","last_became_peered":"2024-04-23T18:16:33.983356+0000","last_unstale":"2024-04-23T18:17:36.109706+0000","last_undegraded":"2024-04-23T18:17:36.109706+0000","last_fullsized":"2024-04-23T18:17:36.109706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:27:19.391281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.112107+0000","last_change":"2024-04-23T18:16:33.987952+0000","last_active":"2024-04-23T18:17:36.112107+0000","last_peered":"2024-04-23T18:17:36.112107+0000","last_clean":"2024-04-23T18:17:36.112107+0000","last_became_active":"2024-04-23T18:16:33.987334+0000","last_became_peered":"2024-04-23T18:16:33.987334+0000","last_unstale":"2024-04-23T18:17:36.112107+0000","last_undegraded":"2024-04-23T18:17:36.112107+0000","last_fullsized":"2024-04-23T18:17:36.112107+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:41.902779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234875+0000","last_change":"2024-04-23T18:16:33.983914+0000","last_active":"2024-04-23T18:17:36.234875+0000","last_peered":"2024-04-23T18:17:36.234875+0000","last_clean":"2024-04-23T18:17:36.234875+0000","last_became_active":"2024-04-23T18:16:33.983444+0000","last_became_peered":"2024-04-23T18:16:33.983444+0000","last_unstale":"2024-04-23T18:17:36.234875+0000","last_undegraded":"2024-04-23T18:17:36.234875+0000","last_fullsized":"2024-04-23T18:17:36.234875+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:10:17.630049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.109958+0000","last_change":"2024-04-23T18:16:33.992496+0000","last_active":"2024-04-23T18:17:36.109958+0000","last_peered":"2024-04-23T18:17:36.109958+0000","last_clean":"2024-04-23T18:17:36.109958+0000","last_became_active":"2024-04-23T18:16:33.992155+0000","last_became_peered":"2024-04-23T18:16:33.992155+0000","last_unstale":"2024-04-23T18:17:36.109958+0000","last_undegraded":"2024-04-23T18:17:36.109958+0000","last_fullsized":"2024-04-23T18:17:36.109958+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:48.835992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.234557+0000","last_change":"2024-04-23T18:16:33.984409+0000","last_active":"2024-04-23T18:17:36.234557+0000","last_peered":"2024-04-23T18:17:36.234557+0000","last_clean":"2024-04-23T18:17:36.234557+0000","last_became_active":"2024-04-23T18:16:33.984073+0000","last_became_peered":"2024-04-23T18:16:33.984073+0000","last_unstale":"2024-04-23T18:17:36.234557+0000","last_undegraded":"2024-04-23T18:17:36.234557+0000","last_fullsized":"2024-04-23T18:17:36.234557+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:45:37.796783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.110072+0000","last_change":"2024-04-23T18:16:33.986376+0000","last_active":"2024-04-23T18:17:36.110072+0000","last_peered":"2024-04-23T18:17:36.110072+0000","last_clean":"2024-04-23T18:17:36.110072+0000","last_became_active":"2024-04-23T18:16:33.986012+0000","last_became_peered":"2024-04-23T18:16:33.986012+0000","last_unstale":"2024-04-23T18:17:36.110072+0000","last_undegraded":"2024-04-23T18:17:36.110072+0000","last_fullsized":"2024-04-23T18:17:36.110072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:26.940215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.106072+0000","last_change":"2024-04-23T18:16:33.999795+0000","last_active":"2024-04-23T18:17:36.106072+0000","last_peered":"2024-04-23T18:17:36.106072+0000","last_clean":"2024-04-23T18:17:36.106072+0000","last_became_active":"2024-04-23T18:16:33.997034+0000","last_became_peered":"2024-04-23T18:16:33.997034+0000","last_unstale":"2024-04-23T18:17:36.106072+0000","last_undegraded":"2024-04-23T18:17:36.106072+0000","last_fullsized":"2024-04-23T18:17:36.106072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:23.983846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T18:17:36.108549+0000","last_change":"2024-04-23T18:16:33.985495+0000","last_active":"2024-04-23T18:17:36.108549+0000","last_peered":"2024-04-23T18:17:36.108549+0000","last_clean":"2024-04-23T18:17:36.108549+0000","last_became_active":"2024-04-23T18:16:33.985178+0000","last_became_peered":"2024-04-23T18:16:33.985178+0000","last_unstale":"2024-04-23T18:17:36.108549+0000","last_undegraded":"2024-04-23T18:17:36.108549+0000","last_fullsized":"2024-04-23T18:17:36.108549+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:22:25.325756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64415,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":431,"num_read_kb":285,"num_write":158,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1147,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":193245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318299,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":468431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579937,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":1671168,"data_stored":1120192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776166,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":468533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037802,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":468319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32732,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712420,"statfs":{"total":95995035648,"available":95961518080,"internally_reserved":0,"allocated":1585152,"data_stored":1060774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495668,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":467922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":528751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40476,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704676,"statfs":{"total":95995035648,"available":95953588224,"internally_reserved":0,"allocated":999424,"data_stored":469193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":528466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182467,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":995328,"data_stored":468233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920831,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":983040,"data_stored":468750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757306,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1597440,"data_stored":1060067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":60751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:17:50.915 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T18:17:50.915 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-23T18:17:50.916 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-23T18:17:50.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-23T18:17:51.168 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:51.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:51 smithi050 ceph-mon[29187]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:51.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:51 smithi188 ceph-mon[32694]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:51.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:51 smithi177 ceph-mon[32308]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:17:52.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:52 smithi188 ceph-mon[32694]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 84 B/s, 0 objects/s recovering 2024-04-23T18:17:52.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:52 smithi050 ceph-mon[29187]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 84 B/s, 0 objects/s recovering 2024-04-23T18:17:52.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:52 smithi177 ceph-mon[32308]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 84 B/s, 0 objects/s recovering 2024-04-23T18:17:52.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.770+0000 7f817db6e700 1 -- 172.21.15.50:0/3438324951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817806b370 msgr2=0x7f817806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:52.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.770+0000 7f817db6e700 1 --2- 172.21.15.50:0/3438324951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817806b370 0x7f817806b750 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f81680099c0 tx=0x7f8168009cd0 comp rx=0 tx=0).stop 2024-04-23T18:17:52.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.770+0000 7f817db6e700 1 -- 172.21.15.50:0/3438324951 shutdown_connections 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.770+0000 7f817db6e700 1 --2- 172.21.15.50:0/3438324951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8178107e10 0x7f817810f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.770+0000 7f817db6e700 1 --2- 172.21.15.50:0/3438324951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817806b370 0x7f817806b750 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.770+0000 7f817db6e700 1 --2- 172.21.15.50:0/3438324951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817806bd20 0x7f81781076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.770+0000 7f817db6e700 1 -- 172.21.15.50:0/3438324951 >> 172.21.15.50:0/3438324951 conn(0x7f81780788f0 msgr2=0x7f8178078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.770+0000 7f817db6e700 1 -- 172.21.15.50:0/3438324951 shutdown_connections 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.770+0000 7f817db6e700 1 -- 172.21.15.50:0/3438324951 wait complete. 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.771+0000 7f817db6e700 1 Processor -- start 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.771+0000 7f817db6e700 1 -- start start 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.771+0000 7f817db6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817806b370 0x7f8178101df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.771+0000 7f817db6e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817806bd20 0x7f8178102330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.771+0000 7f817db6e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8178107e10 0x7f8178102870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.771+0000 7f817db6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8178112c00 con 0x7f817806b370 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.771+0000 7f817db6e700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8178112a80 con 0x7f817806bd20 2024-04-23T18:17:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.771+0000 7f817db6e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8178112d80 con 0x7f8178107e10 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f81777fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817806b370 0x7f8178101df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f81777fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817806b370 0x7f8178101df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39650/0 (socket says 172.21.15.50:39650) 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f81777fe700 1 -- 172.21.15.50:0/739406481 learned_addr learned my addr 172.21.15.50:0/739406481 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f8177fff700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8178107e10 0x7f8178102870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f8176ffd700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817806bd20 0x7f8178102330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f81777fe700 1 -- 172.21.15.50:0/739406481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8178107e10 msgr2=0x7f8178102870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f81777fe700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8178107e10 0x7f8178102870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f81777fe700 1 -- 172.21.15.50:0/739406481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817806bd20 msgr2=0x7f8178102330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f81777fe700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817806bd20 0x7f8178102330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f81777fe700 1 -- 172.21.15.50:0/739406481 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81781a7600 con 0x7f817806b370 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f81777fe700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817806b370 0x7f8178101df0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8168006010 tx=0x7f81680045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f8174ff9700 1 -- 172.21.15.50:0/739406481 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168026070 con 0x7f817806b370 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f817db6e700 1 -- 172.21.15.50:0/739406481 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81781a7890 con 0x7f817806b370 2024-04-23T18:17:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f8174ff9700 1 -- 172.21.15.50:0/739406481 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f816800f460 con 0x7f817806b370 2024-04-23T18:17:52.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f8174ff9700 1 -- 172.21.15.50:0/739406481 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168009e60 con 0x7f817806b370 2024-04-23T18:17:52.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.772+0000 7f817db6e700 1 -- 172.21.15.50:0/739406481 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81781a7ce0 con 0x7f817806b370 2024-04-23T18:17:52.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.774+0000 7f817db6e700 1 -- 172.21.15.50:0/739406481 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f817810b050 con 0x7f817806b370 2024-04-23T18:17:52.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.775+0000 7f8174ff9700 1 -- 172.21.15.50:0/739406481 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8168029070 con 0x7f817806b370 2024-04-23T18:17:52.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.776+0000 7f8174ff9700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f81600a3cd0 0x7f81600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:52.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.776+0000 7f8176ffd700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f81600a3cd0 0x7f81600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:52.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.776+0000 7f8174ff9700 1 -- 172.21.15.50:0/739406481 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f81680d0de0 con 0x7f817806b370 2024-04-23T18:17:52.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.776+0000 7f8176ffd700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f81600a3cd0 0x7f81600a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f81781018f0 tx=0x7f816c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:52.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:52.781+0000 7f8174ff9700 1 -- 172.21.15.50:0/739406481 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f816805dff0 con 0x7f817806b370 2024-04-23T18:17:53.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.043+0000 7f817db6e700 1 -- 172.21.15.50:0/739406481 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8178104190 con 0x7f817806b370 2024-04-23T18:17:53.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.043+0000 7f8174ff9700 1 -- 172.21.15.50:0/739406481 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f8168013070 con 0x7f817806b370 2024-04-23T18:17:53.037 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:17:53.037 INFO:teuthology.orchestra.run.smithi050.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-23T18:17:53.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.046+0000 7f817db6e700 1 -- 172.21.15.50:0/739406481 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f81600a3cd0 msgr2=0x7f81600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:53.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.046+0000 7f817db6e700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f81600a3cd0 0x7f81600a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f81781018f0 tx=0x7f816c009260 comp rx=0 tx=0).stop 2024-04-23T18:17:53.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.046+0000 7f817db6e700 1 -- 172.21.15.50:0/739406481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817806b370 msgr2=0x7f8178101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:53.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.046+0000 7f817db6e700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817806b370 0x7f8178101df0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8168006010 tx=0x7f81680045e0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.046+0000 7f817db6e700 1 -- 172.21.15.50:0/739406481 shutdown_connections 2024-04-23T18:17:53.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.046+0000 7f817db6e700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8178107e10 0x7f8178102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.046+0000 7f817db6e700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817806b370 0x7f8178101df0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.046+0000 7f817db6e700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f81600a3cd0 0x7f81600a6190 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.046+0000 7f817db6e700 1 --2- 172.21.15.50:0/739406481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817806bd20 0x7f8178102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.046+0000 7f817db6e700 1 -- 172.21.15.50:0/739406481 >> 172.21.15.50:0/739406481 conn(0x7f81780788f0 msgr2=0x7f81780fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:53.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.047+0000 7f817db6e700 1 -- 172.21.15.50:0/739406481 shutdown_connections 2024-04-23T18:17:53.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.047+0000 7f817db6e700 1 -- 172.21.15.50:0/739406481 wait complete. 2024-04-23T18:17:53.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:53 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/739406481' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T18:17:53.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:53 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/739406481' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T18:17:53.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:53 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/739406481' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T18:17:53.684 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-23T18:17:53.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:17:53.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.796+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3680044423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa758102a70 msgr2=0x7fa758104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:53.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.796+0000 7fa75e6e1640 1 --2- 172.21.15.50:0/3680044423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa758102a70 0x7fa758104e60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa748009900 tx=0x7fa74802a410 comp rx=0 tx=0).stop 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.796+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3680044423 shutdown_connections 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.796+0000 7fa75e6e1640 1 --2- 172.21.15.50:0/3680044423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa758107cd0 0x7fa75810a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.796+0000 7fa75e6e1640 1 --2- 172.21.15.50:0/3680044423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7581053a0 0x7fa758107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.796+0000 7fa75e6e1640 1 --2- 172.21.15.50:0/3680044423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa758102a70 0x7fa758104e60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.796+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3680044423 >> 172.21.15.50:0/3680044423 conn(0x7fa7580fc400 msgr2=0x7fa7580fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.796+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3680044423 shutdown_connections 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.796+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3680044423 wait complete. 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa75e6e1640 1 Processor -- start 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa75e6e1640 1 -- start start 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa75e6e1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa758102a70 0x7fa758198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa75e6e1640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7581053a0 0x7fa758198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa75e6e1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa758107cd0 0x7fa75819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa75e6e1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa75810dac0 con 0x7fa758102a70 2024-04-23T18:17:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa75e6e1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa75810d940 con 0x7fa758107cd0 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa75e6e1640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa75810dc40 con 0x7fa7581053a0 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa757fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa758102a70 0x7fa758198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa757fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa758102a70 0x7fa758198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39676/0 (socket says 172.21.15.50:39676) 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa757fff640 1 -- 172.21.15.50:0/3577995910 learned_addr learned my addr 172.21.15.50:0/3577995910 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa7577fe640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7581053a0 0x7fa758198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.797+0000 7fa75cc7e640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa758107cd0 0x7fa75819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.798+0000 7fa757fff640 1 -- 172.21.15.50:0/3577995910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7581053a0 msgr2=0x7fa758198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.798+0000 7fa757fff640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7581053a0 0x7fa758198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.798+0000 7fa757fff640 1 -- 172.21.15.50:0/3577995910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa758107cd0 msgr2=0x7fa75819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.798+0000 7fa757fff640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa758107cd0 0x7fa75819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.798+0000 7fa757fff640 1 -- 172.21.15.50:0/3577995910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75819fe50 con 0x7fa758102a70 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.798+0000 7fa757fff640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa758102a70 0x7fa758198230 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa748005940 tx=0x7fa748004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.798+0000 7fa7557fa640 1 -- 172.21.15.50:0/3577995910 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74803f030 con 0x7fa758102a70 2024-04-23T18:17:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.798+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3577995910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7581a00e0 con 0x7fa758102a70 2024-04-23T18:17:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.798+0000 7fa7557fa640 1 -- 172.21.15.50:0/3577995910 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa748003cb0 con 0x7fa758102a70 2024-04-23T18:17:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.798+0000 7fa7557fa640 1 -- 172.21.15.50:0/3577995910 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa748035070 con 0x7fa758102a70 2024-04-23T18:17:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.798+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3577995910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7581a05d0 con 0x7fa758102a70 2024-04-23T18:17:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.799+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3577995910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7580fdef0 con 0x7fa758102a70 2024-04-23T18:17:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.800+0000 7fa7557fa640 1 -- 172.21.15.50:0/3577995910 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa748042030 con 0x7fa758102a70 2024-04-23T18:17:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.801+0000 7fa7557fa640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa7400a3c40 0x7fa7400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.801+0000 7fa7557fa640 1 -- 172.21.15.50:0/3577995910 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fa7480f1160 con 0x7fa758102a70 2024-04-23T18:17:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.801+0000 7fa7577fe640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa7400a3c40 0x7fa7400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.801+0000 7fa7577fe640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa7400a3c40 0x7fa7400a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa758199740 tx=0x7fa74c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:53.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.806+0000 7fa7557fa640 1 -- 172.21.15.50:0/3577995910 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa7480bd9e0 con 0x7fa758102a70 2024-04-23T18:17:53.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.997+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3577995910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa758062b50 con 0x7fa758102a70 2024-04-23T18:17:53.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:53.999+0000 7fa7557fa640 1 -- 172.21.15.50:0/3577995910 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22661 (secure 0 0 0) 0x7fa748003e60 con 0x7fa758102a70 2024-04-23T18:17:53.992 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:17:53.992 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":89,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:17:36.102736+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:17:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.001+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3577995910 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa7400a3c40 msgr2=0x7fa7400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.001+0000 7fa75e6e1640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa7400a3c40 0x7fa7400a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa758199740 tx=0x7fa74c00b410 comp rx=0 tx=0).stop 2024-04-23T18:17:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.001+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3577995910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa758102a70 msgr2=0x7fa758198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.001+0000 7fa75e6e1640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa758102a70 0x7fa758198230 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa748005940 tx=0x7fa748004430 comp rx=0 tx=0).stop 2024-04-23T18:17:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.001+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3577995910 shutdown_connections 2024-04-23T18:17:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.001+0000 7fa75e6e1640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fa7400a3c40 0x7fa7400a6100 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.001+0000 7fa75e6e1640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa758107cd0 0x7fa75819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.001+0000 7fa75e6e1640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7581053a0 0x7fa758198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.001+0000 7fa75e6e1640 1 --2- 172.21.15.50:0/3577995910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa758102a70 0x7fa758198230 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.001+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3577995910 >> 172.21.15.50:0/3577995910 conn(0x7fa7580fc400 msgr2=0x7fa758104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.001+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3577995910 shutdown_connections 2024-04-23T18:17:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.002+0000 7fa75e6e1640 1 -- 172.21.15.50:0/3577995910 wait complete. 2024-04-23T18:17:54.006 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:12:02.493647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:16:31.679148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:16:32.269478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:17:54.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:17:54.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.119+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3440335301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04102a10 msgr2=0x7f1b04104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.119+0000 7f1b0be7c640 1 --2- 172.21.15.50:0/3440335301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04102a10 0x7f1b04104e00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009ca0 tx=0x7f1b0002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3440335301 shutdown_connections 2024-04-23T18:17:54.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 --2- 172.21.15.50:0/3440335301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b04107c70 0x7f1b0410a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 --2- 172.21.15.50:0/3440335301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b04105340 0x7f1b04107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 --2- 172.21.15.50:0/3440335301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04102a10 0x7f1b04104e00 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3440335301 >> 172.21.15.50:0/3440335301 conn(0x7f1b040fc400 msgr2=0x7f1b040fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3440335301 shutdown_connections 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3440335301 wait complete. 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 Processor -- start 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 -- start start 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04102a10 0x7f1b04198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b04105340 0x7f1b041988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b04107c70 0x7f1b0419f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0410dbb0 con 0x7f1b04102a10 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0410da30 con 0x7f1b04107c70 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b0be7c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0410dd30 con 0x7f1b04105340 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b09c18640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04102a10 0x7f1b04198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b09c18640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04102a10 0x7f1b04198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39688/0 (socket says 172.21.15.50:39688) 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.120+0000 7f1b09c18640 1 -- 172.21.15.50:0/3229359612 learned_addr learned my addr 172.21.15.50:0/3229359612 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1b0a419640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b04107c70 0x7f1b0419f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1b09417640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b04105340 0x7f1b041988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1b09c18640 1 -- 172.21.15.50:0/3229359612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b04105340 msgr2=0x7f1b041988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1b09c18640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b04105340 0x7f1b041988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1b09c18640 1 -- 172.21.15.50:0/3229359612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b04107c70 msgr2=0x7f1b0419f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1b09c18640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b04107c70 0x7f1b0419f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1b09c18640 1 -- 172.21.15.50:0/3229359612 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0419ffa0 con 0x7f1b04102a10 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1b0a419640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b04107c70 0x7f1b0419f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1b09c18640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04102a10 0x7f1b04198380 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1b00006010 tx=0x7f1b00003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1afaffd640 1 -- 172.21.15.50:0/3229359612 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b00004040 con 0x7f1b04102a10 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1afaffd640 1 -- 172.21.15.50:0/3229359612 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b00004690 con 0x7f1b04102a10 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3229359612 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b041a0230 con 0x7f1b04102a10 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1afaffd640 1 -- 172.21.15.50:0/3229359612 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0002fed0 con 0x7f1b04102a10 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.121+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3229359612 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b041a0850 con 0x7f1b04102a10 2024-04-23T18:17:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.122+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3229359612 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b040fdef0 con 0x7f1b04102a10 2024-04-23T18:17:54.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.124+0000 7f1afaffd640 1 -- 172.21.15.50:0/3229359612 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1b0002f660 con 0x7f1b04102a10 2024-04-23T18:17:54.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.124+0000 7f1afaffd640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1af00a3cc0 0x7f1af00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:54.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.125+0000 7f1b09417640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1af00a3cc0 0x7f1af00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:54.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.125+0000 7f1afaffd640 1 -- 172.21.15.50:0/3229359612 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f1b000f1df0 con 0x7f1b04102a10 2024-04-23T18:17:54.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.125+0000 7f1b09417640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1af00a3cc0 0x7f1af00a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b04199890 tx=0x7f1af4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:54.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.128+0000 7f1afaffd640 1 -- 172.21.15.50:0/3229359612 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b000be810 con 0x7f1b04102a10 2024-04-23T18:17:54.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.313+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3229359612 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1b04062b50 con 0x7f1b04102a10 2024-04-23T18:17:54.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.314+0000 7f1afaffd640 1 -- 172.21.15.50:0/3229359612 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f1b000be110 con 0x7f1b04102a10 2024-04-23T18:17:54.307 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-23T18:17:54.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.316+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3229359612 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1af00a3cc0 msgr2=0x7f1af00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.316+0000 7f1b0be7c640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1af00a3cc0 0x7f1af00a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b04199890 tx=0x7f1af4008040 comp rx=0 tx=0).stop 2024-04-23T18:17:54.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.316+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3229359612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04102a10 msgr2=0x7f1b04198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.316+0000 7f1b0be7c640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04102a10 0x7f1b04198380 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1b00006010 tx=0x7f1b00003ba0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.316+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3229359612 shutdown_connections 2024-04-23T18:17:54.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.316+0000 7f1b0be7c640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1af00a3cc0 0x7f1af00a6180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.316+0000 7f1b0be7c640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b04107c70 0x7f1b0419f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.316+0000 7f1b0be7c640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b04105340 0x7f1b041988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.316+0000 7f1b0be7c640 1 --2- 172.21.15.50:0/3229359612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04102a10 0x7f1b04198380 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.316+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3229359612 >> 172.21.15.50:0/3229359612 conn(0x7f1b040fc400 msgr2=0x7f1b04103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:54.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.316+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3229359612 shutdown_connections 2024-04-23T18:17:54.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.316+0000 7f1b0be7c640 1 -- 172.21.15.50:0/3229359612 wait complete. 2024-04-23T18:17:54.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:54 smithi050 ceph-mon[29187]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s; 84 B/s, 0 objects/s recovering 2024-04-23T18:17:54.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:54 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3577995910' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:54.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:17:54.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:54 smithi188 ceph-mon[32694]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s; 84 B/s, 0 objects/s recovering 2024-04-23T18:17:54.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:54 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3577995910' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:54.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.432+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/1150041610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6a8101fc0 msgr2=0x7ff6a8102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.432+0000 7ff6ae5fe640 1 --2- 172.21.15.50:0/1150041610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6a8101fc0 0x7ff6a8102440 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009900 tx=0x7ff69c02a410 comp rx=0 tx=0).stop 2024-04-23T18:17:54.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.432+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/1150041610 shutdown_connections 2024-04-23T18:17:54.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.432+0000 7ff6ae5fe640 1 --2- 172.21.15.50:0/1150041610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6a8102980 0x7ff6a81092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.432+0000 7ff6ae5fe640 1 --2- 172.21.15.50:0/1150041610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6a8101fc0 0x7ff6a8102440 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.432+0000 7ff6ae5fe640 1 --2- 172.21.15.50:0/1150041610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6a8100e70 0x7ff6a8101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.432+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/1150041610 >> 172.21.15.50:0/1150041610 conn(0x7ff6a80fc3e0 msgr2=0x7ff6a80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.432+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/1150041610 shutdown_connections 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.432+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/1150041610 wait complete. 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.432+0000 7ff6ae5fe640 1 Processor -- start 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6ae5fe640 1 -- start start 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6ae5fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6a8100e70 0x7ff6a8104df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6ae5fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6a8101fc0 0x7ff6a8103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6ae5fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6a8102980 0x7ff6a8103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6ae5fe640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a810c9a0 con 0x7ff6a8101fc0 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6ae5fe640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a810c820 con 0x7ff6a8100e70 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6ae5fe640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a810cb20 con 0x7ff6a8102980 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6a77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6a8101fc0 0x7ff6a8103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6a77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6a8101fc0 0x7ff6a8103430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39694/0 (socket says 172.21.15.50:39694) 2024-04-23T18:17:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6a77fe640 1 -- 172.21.15.50:0/3313071711 learned_addr learned my addr 172.21.15.50:0/3313071711 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6a7fff640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6a8100e70 0x7ff6a8104df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6acb9b640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6a8102980 0x7ff6a8103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6a77fe640 1 -- 172.21.15.50:0/3313071711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6a8102980 msgr2=0x7ff6a8103970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6a77fe640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6a8102980 0x7ff6a8103970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6a77fe640 1 -- 172.21.15.50:0/3313071711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6a8100e70 msgr2=0x7ff6a8104df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6a77fe640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6a8100e70 0x7ff6a8104df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.433+0000 7ff6a77fe640 1 -- 172.21.15.50:0/3313071711 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a8104200 con 0x7ff6a8101fc0 2024-04-23T18:17:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.434+0000 7ff6a77fe640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6a8101fc0 0x7ff6a8103430 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff69c006010 tx=0x7ff69c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.434+0000 7ff6a57fa640 1 -- 172.21.15.50:0/3313071711 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c03f030 con 0x7ff6a8101fc0 2024-04-23T18:17:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.434+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/3313071711 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a81a22b0 con 0x7ff6a8101fc0 2024-04-23T18:17:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.434+0000 7ff6a57fa640 1 -- 172.21.15.50:0/3313071711 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff69c003cb0 con 0x7ff6a8101fc0 2024-04-23T18:17:54.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.434+0000 7ff6a57fa640 1 -- 172.21.15.50:0/3313071711 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c035070 con 0x7ff6a8101fc0 2024-04-23T18:17:54.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.434+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/3313071711 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a81a2800 con 0x7ff6a8101fc0 2024-04-23T18:17:54.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.435+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/3313071711 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a81a2a60 con 0x7ff6a8101fc0 2024-04-23T18:17:54.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.436+0000 7ff6a57fa640 1 -- 172.21.15.50:0/3313071711 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff69c042030 con 0x7ff6a8101fc0 2024-04-23T18:17:54.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.437+0000 7ff6a57fa640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6900a3c40 0x7ff6900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:54.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.437+0000 7ff6a7fff640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6900a3c40 0x7ff6900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:54.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.437+0000 7ff6a57fa640 1 -- 172.21.15.50:0/3313071711 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7ff69c0f0c10 con 0x7ff6a8101fc0 2024-04-23T18:17:54.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.437+0000 7ff6a7fff640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6900a3c40 0x7ff6900a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff69800bbe0 tx=0x7ff69800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:54.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.441+0000 7ff6a57fa640 1 -- 172.21.15.50:0/3313071711 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff69c0bd450 con 0x7ff6a8101fc0 2024-04-23T18:17:54.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:54 smithi177 ceph-mon[32308]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s; 84 B/s, 0 objects/s recovering 2024-04-23T18:17:54.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:54 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3577995910' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:54.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.629+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/3313071711 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff6a81a2d50 con 0x7ff6a8101fc0 2024-04-23T18:17:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.630+0000 7ff6a57fa640 1 -- 172.21.15.50:0/3313071711 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7ff69c0bcd50 con 0x7ff6a8101fc0 2024-04-23T18:17:54.626 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2024-04-23T18:17:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.632+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/3313071711 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6900a3c40 msgr2=0x7ff6900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.632+0000 7ff6ae5fe640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6900a3c40 0x7ff6900a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff69800bbe0 tx=0x7ff69800b390 comp rx=0 tx=0).stop 2024-04-23T18:17:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.632+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/3313071711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6a8101fc0 msgr2=0x7ff6a8103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.632+0000 7ff6ae5fe640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6a8101fc0 0x7ff6a8103430 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff69c006010 tx=0x7ff69c0042f0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.632+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/3313071711 shutdown_connections 2024-04-23T18:17:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.632+0000 7ff6ae5fe640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7ff6900a3c40 0x7ff6900a6100 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.632+0000 7ff6ae5fe640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6a8102980 0x7ff6a8103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.632+0000 7ff6ae5fe640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6a8101fc0 0x7ff6a8103430 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.632+0000 7ff6ae5fe640 1 --2- 172.21.15.50:0/3313071711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6a8100e70 0x7ff6a8104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.632+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/3313071711 >> 172.21.15.50:0/3313071711 conn(0x7ff6a80fc3e0 msgr2=0x7ff6a80691d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:54.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.632+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/3313071711 shutdown_connections 2024-04-23T18:17:54.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.632+0000 7ff6ae5fe640 1 -- 172.21.15.50:0/3313071711 wait complete. 2024-04-23T18:17:54.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:17:54.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.748+0000 7f0dbdedc640 1 -- 172.21.15.50:0/2159328022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db8107c90 msgr2=0x7f0db810a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.748+0000 7f0dbdedc640 1 --2- 172.21.15.50:0/2159328022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db8107c90 0x7f0db810a0e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0db0009ca0 tx=0x7f0db002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.748+0000 7f0dbdedc640 1 -- 172.21.15.50:0/2159328022 shutdown_connections 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.748+0000 7f0dbdedc640 1 --2- 172.21.15.50:0/2159328022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db8107c90 0x7f0db810a0e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.748+0000 7f0dbdedc640 1 --2- 172.21.15.50:0/2159328022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0db8105360 0x7f0db8107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.748+0000 7f0dbdedc640 1 --2- 172.21.15.50:0/2159328022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db8102a30 0x7f0db8104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.748+0000 7f0dbdedc640 1 -- 172.21.15.50:0/2159328022 >> 172.21.15.50:0/2159328022 conn(0x7f0db80fc440 msgr2=0x7f0db80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.748+0000 7f0dbdedc640 1 -- 172.21.15.50:0/2159328022 shutdown_connections 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.748+0000 7f0dbdedc640 1 -- 172.21.15.50:0/2159328022 wait complete. 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.749+0000 7f0dbdedc640 1 Processor -- start 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.749+0000 7f0dbdedc640 1 -- start start 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.749+0000 7f0dbdedc640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db8102a30 0x7f0db81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.749+0000 7f0dbdedc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db8105360 0x7f0db81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.749+0000 7f0dbdedc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0db8107c90 0x7f0db819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.749+0000 7f0dbdedc640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0db810daf0 con 0x7f0db8105360 2024-04-23T18:17:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.749+0000 7f0dbdedc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0db810d970 con 0x7f0db8107c90 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.749+0000 7f0dbdedc640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0db810dc70 con 0x7f0db8102a30 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.749+0000 7f0db6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db8105360 0x7f0db81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.749+0000 7f0db6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db8105360 0x7f0db81986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39706/0 (socket says 172.21.15.50:39706) 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.749+0000 7f0db6ffd640 1 -- 172.21.15.50:0/1921183032 learned_addr learned my addr 172.21.15.50:0/1921183032 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db77fe640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db8102a30 0x7f0db81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db7fff640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0db8107c90 0x7f0db819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db6ffd640 1 -- 172.21.15.50:0/1921183032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db8102a30 msgr2=0x7f0db81981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db6ffd640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db8102a30 0x7f0db81981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db6ffd640 1 -- 172.21.15.50:0/1921183032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0db8107c90 msgr2=0x7f0db819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db6ffd640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0db8107c90 0x7f0db819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db6ffd640 1 -- 172.21.15.50:0/1921183032 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db819fdd0 con 0x7f0db8105360 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db7fff640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0db8107c90 0x7f0db819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db77fe640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db8102a30 0x7f0db81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db6ffd640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db8105360 0x7f0db81986f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0dac00d770 tx=0x7f0dac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db4ff9640 1 -- 172.21.15.50:0/1921183032 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac017070 con 0x7f0db8105360 2024-04-23T18:17:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0dbdedc640 1 -- 172.21.15.50:0/1921183032 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0db81a00c0 con 0x7f0db8105360 2024-04-23T18:17:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db4ff9640 1 -- 172.21.15.50:0/1921183032 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dac009c30 con 0x7f0db8105360 2024-04-23T18:17:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0db4ff9640 1 -- 172.21.15.50:0/1921183032 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac00f620 con 0x7f0db8105360 2024-04-23T18:17:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.750+0000 7f0dbdedc640 1 -- 172.21.15.50:0/1921183032 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0db81a0660 con 0x7f0db8105360 2024-04-23T18:17:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.751+0000 7f0dbdedc640 1 -- 172.21.15.50:0/1921183032 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0db80fdf30 con 0x7f0db8105360 2024-04-23T18:17:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.752+0000 7f0db4ff9640 1 -- 172.21.15.50:0/1921183032 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0dac01a070 con 0x7f0db8105360 2024-04-23T18:17:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.753+0000 7f0db4ff9640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f0da00a3c40 0x7f0da00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.753+0000 7f0db77fe640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f0da00a3c40 0x7f0da00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.753+0000 7f0db4ff9640 1 -- 172.21.15.50:0/1921183032 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f0dac0d01b0 con 0x7f0db8105360 2024-04-23T18:17:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.753+0000 7f0db77fe640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f0da00a3c40 0x7f0da00a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009c20 tx=0x7f0da8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:54.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.756+0000 7f0db4ff9640 1 -- 172.21.15.50:0/1921183032 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0dac009de0 con 0x7f0db8105360 2024-04-23T18:17:54.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.940+0000 7f0dbdedc640 1 -- 172.21.15.50:0/1921183032 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0db802d7e0 con 0x7f0db8105360 2024-04-23T18:17:54.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.940+0000 7f0db4ff9640 1 -- 172.21.15.50:0/1921183032 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f0dac09c9f0 con 0x7f0db8105360 2024-04-23T18:17:54.933 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2024-04-23T18:17:54.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.942+0000 7f0dbdedc640 1 -- 172.21.15.50:0/1921183032 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f0da00a3c40 msgr2=0x7f0da00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.942+0000 7f0dbdedc640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f0da00a3c40 0x7f0da00a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009c20 tx=0x7f0da8005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:54.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.942+0000 7f0dbdedc640 1 -- 172.21.15.50:0/1921183032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db8105360 msgr2=0x7f0db81986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:54.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.942+0000 7f0dbdedc640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db8105360 0x7f0db81986f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0dac00d770 tx=0x7f0dac00dc30 comp rx=0 tx=0).stop 2024-04-23T18:17:54.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.942+0000 7f0dbdedc640 1 -- 172.21.15.50:0/1921183032 shutdown_connections 2024-04-23T18:17:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.942+0000 7f0dbdedc640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f0da00a3c40 0x7f0da00a6100 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.942+0000 7f0dbdedc640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0db8107c90 0x7f0db819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.942+0000 7f0dbdedc640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db8105360 0x7f0db81986f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.942+0000 7f0dbdedc640 1 --2- 172.21.15.50:0/1921183032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db8102a30 0x7f0db81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.942+0000 7f0dbdedc640 1 -- 172.21.15.50:0/1921183032 >> 172.21.15.50:0/1921183032 conn(0x7f0db80fc440 msgr2=0x7f0db81032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.942+0000 7f0dbdedc640 1 -- 172.21.15.50:0/1921183032 shutdown_connections 2024-04-23T18:17:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:54.942+0000 7f0dbdedc640 1 -- 172.21.15.50:0/1921183032 wait complete. 2024-04-23T18:17:54.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:17:55.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.056+0000 7f75392a0640 1 -- 172.21.15.50:0/1175910467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7534105360 msgr2=0x7f7534107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.056+0000 7f75392a0640 1 --2- 172.21.15.50:0/1175910467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7534105360 0x7f7534107750 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f751c009900 tx=0x7f751c02a410 comp rx=0 tx=0).stop 2024-04-23T18:17:55.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.056+0000 7f75392a0640 1 -- 172.21.15.50:0/1175910467 shutdown_connections 2024-04-23T18:17:55.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.056+0000 7f75392a0640 1 --2- 172.21.15.50:0/1175910467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7534107c90 0x7f753410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.056+0000 7f75392a0640 1 --2- 172.21.15.50:0/1175910467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7534105360 0x7f7534107750 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.056+0000 7f75392a0640 1 --2- 172.21.15.50:0/1175910467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7534102a30 0x7f7534104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.056+0000 7f75392a0640 1 -- 172.21.15.50:0/1175910467 >> 172.21.15.50:0/1175910467 conn(0x7f75340fc440 msgr2=0x7f75340fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.056+0000 7f75392a0640 1 -- 172.21.15.50:0/1175910467 shutdown_connections 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.056+0000 7f75392a0640 1 -- 172.21.15.50:0/1175910467 wait complete. 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75392a0640 1 Processor -- start 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75392a0640 1 -- start start 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75392a0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7534102a30 0x7f7534198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75392a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7534105360 0x7f75341986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75392a0640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7534107c90 0x7f753419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75392a0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f753410daf0 con 0x7f7534105360 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75392a0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f753410d970 con 0x7f7534102a30 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75392a0640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f753410dc70 con 0x7f7534107c90 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75327fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7534105360 0x7f75341986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75327fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7534105360 0x7f75341986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39724/0 (socket says 172.21.15.50:39724) 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75327fc640 1 -- 172.21.15.50:0/730884162 learned_addr learned my addr 172.21.15.50:0/730884162 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75337fe640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7534107c90 0x7f753419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:55.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f7532ffd640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7534102a30 0x7f7534198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75327fc640 1 -- 172.21.15.50:0/730884162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7534107c90 msgr2=0x7f753419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75327fc640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7534107c90 0x7f753419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75327fc640 1 -- 172.21.15.50:0/730884162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7534102a30 msgr2=0x7f7534198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75327fc640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7534102a30 0x7f7534198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75327fc640 1 -- 172.21.15.50:0/730884162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753419fda0 con 0x7f7534105360 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f75337fe640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7534107c90 0x7f753419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.057+0000 7f7532ffd640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7534102a30 0x7f7534198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.058+0000 7f75327fc640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7534105360 0x7f75341986c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f751c006010 tx=0x7f751c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.058+0000 7f752bfff640 1 -- 172.21.15.50:0/730884162 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c03f030 con 0x7f7534105360 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.058+0000 7f752bfff640 1 -- 172.21.15.50:0/730884162 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f751c003cb0 con 0x7f7534105360 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.058+0000 7f75392a0640 1 -- 172.21.15.50:0/730884162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75341a0030 con 0x7f7534105360 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.058+0000 7f752bfff640 1 -- 172.21.15.50:0/730884162 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c035070 con 0x7f7534105360 2024-04-23T18:17:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.058+0000 7f75392a0640 1 -- 172.21.15.50:0/730884162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75341a0520 con 0x7f7534105360 2024-04-23T18:17:55.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.059+0000 7f75392a0640 1 -- 172.21.15.50:0/730884162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75340fdf30 con 0x7f7534105360 2024-04-23T18:17:55.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.060+0000 7f752bfff640 1 -- 172.21.15.50:0/730884162 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f751c042030 con 0x7f7534105360 2024-04-23T18:17:55.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.061+0000 7f752bfff640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f75200a3c40 0x7f75200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:55.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.061+0000 7f7532ffd640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f75200a3c40 0x7f75200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:55.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.061+0000 7f752bfff640 1 -- 172.21.15.50:0/730884162 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f751c0f0cc0 con 0x7f7534105360 2024-04-23T18:17:55.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.061+0000 7f7532ffd640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f75200a3c40 0x7f75200a6100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7524007e70 tx=0x7f7524007850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:55.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.065+0000 7f752bfff640 1 -- 172.21.15.50:0/730884162 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f751c0f7080 con 0x7f7534105360 2024-04-23T18:17:55.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.293+0000 7f75392a0640 1 -- 172.21.15.50:0/730884162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f753402d810 con 0x7f7534105360 2024-04-23T18:17:55.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.294+0000 7f752bfff640 1 -- 172.21.15.50:0/730884162 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11875 (secure 0 0 0) 0x7f751c0bd500 con 0x7f7534105360 2024-04-23T18:17:55.287 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:17:55.287 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:16:37.965071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:17:55.287 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 20 2024-04-23T18:17:55.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.296+0000 7f75392a0640 1 -- 172.21.15.50:0/730884162 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f75200a3c40 msgr2=0x7f75200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.296+0000 7f75392a0640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f75200a3c40 0x7f75200a6100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7524007e70 tx=0x7f7524007850 comp rx=0 tx=0).stop 2024-04-23T18:17:55.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.296+0000 7f75392a0640 1 -- 172.21.15.50:0/730884162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7534105360 msgr2=0x7f75341986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.296+0000 7f75392a0640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7534105360 0x7f75341986c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f751c006010 tx=0x7f751c0042f0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.296+0000 7f75392a0640 1 -- 172.21.15.50:0/730884162 shutdown_connections 2024-04-23T18:17:55.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.296+0000 7f75392a0640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f75200a3c40 0x7f75200a6100 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.296+0000 7f75392a0640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7534107c90 0x7f753419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.296+0000 7f75392a0640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7534105360 0x7f75341986c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.296+0000 7f75392a0640 1 --2- 172.21.15.50:0/730884162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7534102a30 0x7f7534198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.296+0000 7f75392a0640 1 -- 172.21.15.50:0/730884162 >> 172.21.15.50:0/730884162 conn(0x7f75340fc440 msgr2=0x7f75341032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:55.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.296+0000 7f75392a0640 1 -- 172.21.15.50:0/730884162 shutdown_connections 2024-04-23T18:17:55.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.296+0000 7f75392a0640 1 -- 172.21.15.50:0/730884162 wait complete. 2024-04-23T18:17:55.299 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:55 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3229359612' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:55.299 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:55 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3313071711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:55.299 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:55 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1921183032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:55.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:17:55.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:55 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3229359612' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:55.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:55 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3313071711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:55.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:55 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1921183032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.409+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2196584587 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fcc107ce0 msgr2=0x7f0fcc10a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.409+0000 7f0fd1fcd640 1 --2- 172.21.15.50:0/2196584587 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fcc107ce0 0x7f0fcc10a0d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009ca0 tx=0x7f0fc402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.409+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2196584587 shutdown_connections 2024-04-23T18:17:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.409+0000 7f0fd1fcd640 1 --2- 172.21.15.50:0/2196584587 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fcc107ce0 0x7f0fcc10a0d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.409+0000 7f0fd1fcd640 1 --2- 172.21.15.50:0/2196584587 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fcc1053b0 0x7f0fcc1077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.409+0000 7f0fd1fcd640 1 --2- 172.21.15.50:0/2196584587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fcc102a80 0x7f0fcc104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.409+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2196584587 >> 172.21.15.50:0/2196584587 conn(0x7f0fcc0fc430 msgr2=0x7f0fcc0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.409+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2196584587 shutdown_connections 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.409+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2196584587 wait complete. 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fd1fcd640 1 Processor -- start 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fd1fcd640 1 -- start start 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fd1fcd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fcc102a80 0x7f0fcc1023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fd1fcd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fcc1053b0 0x7f0fcc100a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fd1fcd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fcc107ce0 0x7f0fcc100f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fd1fcd640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc10dae0 con 0x7f0fcc1053b0 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fd1fcd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc10d960 con 0x7f0fcc102a80 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fd1fcd640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc10dc60 con 0x7f0fcc107ce0 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fcaffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fcc1053b0 0x7f0fcc100a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fcaffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fcc1053b0 0x7f0fcc100a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39752/0 (socket says 172.21.15.50:39752) 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fcaffd640 1 -- 172.21.15.50:0/2790744641 learned_addr learned my addr 172.21.15.50:0/2790744641 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fcb7fe640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fcc102a80 0x7f0fcc1023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.410+0000 7f0fcbfff640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fcc107ce0 0x7f0fcc100f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.411+0000 7f0fcaffd640 1 -- 172.21.15.50:0/2790744641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fcc107ce0 msgr2=0x7f0fcc100f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.411+0000 7f0fcaffd640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fcc107ce0 0x7f0fcc100f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.411+0000 7f0fcaffd640 1 -- 172.21.15.50:0/2790744641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fcc102a80 msgr2=0x7f0fcc1023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.411+0000 7f0fcaffd640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fcc102a80 0x7f0fcc1023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.411+0000 7f0fcaffd640 1 -- 172.21.15.50:0/2790744641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fcc101800 con 0x7f0fcc1053b0 2024-04-23T18:17:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.411+0000 7f0fcaffd640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fcc1053b0 0x7f0fcc100a30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c970 tx=0x7f0fc000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.411+0000 7f0fc8ff9640 1 -- 172.21.15.50:0/2790744641 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0014070 con 0x7f0fcc1053b0 2024-04-23T18:17:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.411+0000 7f0fc8ff9640 1 -- 172.21.15.50:0/2790744641 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fc0004440 con 0x7f0fcc1053b0 2024-04-23T18:17:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.411+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2790744641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fcc1a2250 con 0x7f0fcc1053b0 2024-04-23T18:17:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.411+0000 7f0fc8ff9640 1 -- 172.21.15.50:0/2790744641 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0003e50 con 0x7f0fcc1053b0 2024-04-23T18:17:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.411+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2790744641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fcc1a27a0 con 0x7f0fcc1053b0 2024-04-23T18:17:55.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.412+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2790744641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fcc0fdf20 con 0x7f0fcc1053b0 2024-04-23T18:17:55.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.413+0000 7f0fc8ff9640 1 -- 172.21.15.50:0/2790744641 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0fc0017070 con 0x7f0fcc1053b0 2024-04-23T18:17:55.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.414+0000 7f0fc8ff9640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f0fb40a3de0 0x7f0fb40a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:55.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.414+0000 7f0fcb7fe640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f0fb40a3de0 0x7f0fb40a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:55.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.414+0000 7f0fc8ff9640 1 -- 172.21.15.50:0/2790744641 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f0fc00d08c0 con 0x7f0fcc1053b0 2024-04-23T18:17:55.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.414+0000 7f0fcb7fe640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f0fb40a3de0 0x7f0fb40a62a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009bd0 tx=0x7f0fbc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:55.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.418+0000 7f0fc8ff9640 1 -- 172.21.15.50:0/2790744641 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0fc001e020 con 0x7f0fcc1053b0 2024-04-23T18:17:55.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:55 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3229359612' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:55.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:55 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3313071711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:55.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:55 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1921183032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:55.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.605+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2790744641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fcc02d810 con 0x7f0fcc1053b0 2024-04-23T18:17:55.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.607+0000 7f0fc8ff9640 1 -- 172.21.15.50:0/2790744641 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22661 (secure 0 0 0) 0x7f0fc009d100 con 0x7f0fcc1053b0 2024-04-23T18:17:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:17:55.600 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":89,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:17:36.102736+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:17:55.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.608+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2790744641 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f0fb40a3de0 msgr2=0x7f0fb40a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.608+0000 7f0fd1fcd640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f0fb40a3de0 0x7f0fb40a62a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009bd0 tx=0x7f0fbc005f90 comp rx=0 tx=0).stop 2024-04-23T18:17:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.608+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2790744641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fcc1053b0 msgr2=0x7f0fcc100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.608+0000 7f0fd1fcd640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fcc1053b0 0x7f0fcc100a30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c970 tx=0x7f0fc000ce30 comp rx=0 tx=0).stop 2024-04-23T18:17:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.609+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2790744641 shutdown_connections 2024-04-23T18:17:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.609+0000 7f0fd1fcd640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f0fb40a3de0 0x7f0fb40a62a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.609+0000 7f0fd1fcd640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fcc107ce0 0x7f0fcc100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.609+0000 7f0fd1fcd640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fcc1053b0 0x7f0fcc100a30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.609+0000 7f0fd1fcd640 1 --2- 172.21.15.50:0/2790744641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fcc102a80 0x7f0fcc1023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.609+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2790744641 >> 172.21.15.50:0/2790744641 conn(0x7f0fcc0fc430 msgr2=0x7f0fcc1033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.609+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2790744641 shutdown_connections 2024-04-23T18:17:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.609+0000 7f0fd1fcd640 1 -- 172.21.15.50:0/2790744641 wait complete. 2024-04-23T18:17:55.613 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:12:02.493647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:16:31.679148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:16:32.269478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:17:55.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:17:55.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.725+0000 7eff46b4a640 1 -- 172.21.15.50:0/4014713911 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff40100e90 msgr2=0x7eff401012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.725+0000 7eff46b4a640 1 --2- 172.21.15.50:0/4014713911 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff40100e90 0x7eff401012b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7eff34009900 tx=0x7eff3402a3f0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.725+0000 7eff46b4a640 1 -- 172.21.15.50:0/4014713911 shutdown_connections 2024-04-23T18:17:55.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.725+0000 7eff46b4a640 1 --2- 172.21.15.50:0/4014713911 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff401029a0 0x7eff401092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.725+0000 7eff46b4a640 1 --2- 172.21.15.50:0/4014713911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff40101fe0 0x7eff40102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.725+0000 7eff46b4a640 1 --2- 172.21.15.50:0/4014713911 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff40100e90 0x7eff401012b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.725+0000 7eff46b4a640 1 -- 172.21.15.50:0/4014713911 >> 172.21.15.50:0/4014713911 conn(0x7eff400fc400 msgr2=0x7eff400fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.725+0000 7eff46b4a640 1 -- 172.21.15.50:0/4014713911 shutdown_connections 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.725+0000 7eff46b4a640 1 -- 172.21.15.50:0/4014713911 wait complete. 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff46b4a640 1 Processor -- start 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff46b4a640 1 -- start start 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff46b4a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff40100e90 0x7eff401981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff46b4a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff40101fe0 0x7eff401986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff46b4a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff401029a0 0x7eff4019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff46b4a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7eff4010c9e0 con 0x7eff40101fe0 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff46b4a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7eff4010c860 con 0x7eff40100e90 2024-04-23T18:17:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff46b4a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7eff4010cb60 con 0x7eff401029a0 2024-04-23T18:17:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff3ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff40101fe0 0x7eff401986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff3ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff40101fe0 0x7eff401986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39778/0 (socket says 172.21.15.50:39778) 2024-04-23T18:17:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff3ffff640 1 -- 172.21.15.50:0/1802596943 learned_addr learned my addr 172.21.15.50:0/1802596943 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff450e7640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff401029a0 0x7eff4019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff448e6640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff40100e90 0x7eff401981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff3ffff640 1 -- 172.21.15.50:0/1802596943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff401029a0 msgr2=0x7eff4019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff3ffff640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff401029a0 0x7eff4019f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff3ffff640 1 -- 172.21.15.50:0/1802596943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff40100e90 msgr2=0x7eff401981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.726+0000 7eff3ffff640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff40100e90 0x7eff401981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.727+0000 7eff3ffff640 1 -- 172.21.15.50:0/1802596943 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4019fdd0 con 0x7eff40101fe0 2024-04-23T18:17:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.727+0000 7eff448e6640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff40100e90 0x7eff401981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.727+0000 7eff3ffff640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff40101fe0 0x7eff401986f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00c970 tx=0x7eff2c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:55.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.727+0000 7eff3dffb640 1 -- 172.21.15.50:0/1802596943 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c007c10 con 0x7eff40101fe0 2024-04-23T18:17:55.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.727+0000 7eff3dffb640 1 -- 172.21.15.50:0/1802596943 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff2c007db0 con 0x7eff40101fe0 2024-04-23T18:17:55.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.727+0000 7eff46b4a640 1 -- 172.21.15.50:0/1802596943 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff401a00c0 con 0x7eff40101fe0 2024-04-23T18:17:55.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.727+0000 7eff3dffb640 1 -- 172.21.15.50:0/1802596943 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c00f870 con 0x7eff40101fe0 2024-04-23T18:17:55.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.727+0000 7eff46b4a640 1 -- 172.21.15.50:0/1802596943 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff401a0610 con 0x7eff40101fe0 2024-04-23T18:17:55.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.728+0000 7eff46b4a640 1 -- 172.21.15.50:0/1802596943 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff401a08d0 con 0x7eff40101fe0 2024-04-23T18:17:55.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.729+0000 7eff3dffb640 1 -- 172.21.15.50:0/1802596943 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7eff2c00fa10 con 0x7eff40101fe0 2024-04-23T18:17:55.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.730+0000 7eff3dffb640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7eff300a3c40 0x7eff300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:55.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.730+0000 7eff448e6640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7eff300a3c40 0x7eff300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:55.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.730+0000 7eff3dffb640 1 -- 172.21.15.50:0/1802596943 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7eff2c0d0980 con 0x7eff40101fe0 2024-04-23T18:17:55.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.730+0000 7eff448e6640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7eff300a3c40 0x7eff300a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7eff34006010 tx=0x7eff34000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:55.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.734+0000 7eff3dffb640 1 -- 172.21.15.50:0/1802596943 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff2c09d250 con 0x7eff40101fe0 2024-04-23T18:17:55.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.918+0000 7eff46b4a640 1 -- 172.21.15.50:0/1802596943 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7eff401a0aa0 con 0x7eff40101fe0 2024-04-23T18:17:55.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.918+0000 7eff3dffb640 1 -- 172.21.15.50:0/1802596943 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7eff2c09cb50 con 0x7eff40101fe0 2024-04-23T18:17:55.911 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-23T18:17:55.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.920+0000 7eff46b4a640 1 -- 172.21.15.50:0/1802596943 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7eff300a3c40 msgr2=0x7eff300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.920+0000 7eff46b4a640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7eff300a3c40 0x7eff300a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7eff34006010 tx=0x7eff34000bc0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.920+0000 7eff46b4a640 1 -- 172.21.15.50:0/1802596943 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff40101fe0 msgr2=0x7eff401986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.920+0000 7eff46b4a640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff40101fe0 0x7eff401986f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00c970 tx=0x7eff2c00ce30 comp rx=0 tx=0).stop 2024-04-23T18:17:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.920+0000 7eff46b4a640 1 -- 172.21.15.50:0/1802596943 shutdown_connections 2024-04-23T18:17:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.920+0000 7eff46b4a640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7eff300a3c40 0x7eff300a6100 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.920+0000 7eff46b4a640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff401029a0 0x7eff4019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.920+0000 7eff46b4a640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff40101fe0 0x7eff401986f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.920+0000 7eff46b4a640 1 --2- 172.21.15.50:0/1802596943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff40100e90 0x7eff401981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.920+0000 7eff46b4a640 1 -- 172.21.15.50:0/1802596943 >> 172.21.15.50:0/1802596943 conn(0x7eff400fc400 msgr2=0x7eff400fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.921+0000 7eff46b4a640 1 -- 172.21.15.50:0/1802596943 shutdown_connections 2024-04-23T18:17:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:55.921+0000 7eff46b4a640 1 -- 172.21.15.50:0/1802596943 wait complete. 2024-04-23T18:17:55.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:17:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.037+0000 7fda125af640 1 -- 172.21.15.50:0/511136093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0c1053c0 msgr2=0x7fda0c1077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.037+0000 7fda125af640 1 --2- 172.21.15.50:0/511136093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0c1053c0 0x7fda0c1077b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fda00009900 tx=0x7fda0002a410 comp rx=0 tx=0).stop 2024-04-23T18:17:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.037+0000 7fda125af640 1 -- 172.21.15.50:0/511136093 shutdown_connections 2024-04-23T18:17:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.037+0000 7fda125af640 1 --2- 172.21.15.50:0/511136093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda0c107cf0 0x7fda0c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.037+0000 7fda125af640 1 --2- 172.21.15.50:0/511136093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0c1053c0 0x7fda0c1077b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.037+0000 7fda125af640 1 --2- 172.21.15.50:0/511136093 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda0c102a90 0x7fda0c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.037+0000 7fda125af640 1 -- 172.21.15.50:0/511136093 >> 172.21.15.50:0/511136093 conn(0x7fda0c0fc3e0 msgr2=0x7fda0c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.037+0000 7fda125af640 1 -- 172.21.15.50:0/511136093 shutdown_connections 2024-04-23T18:17:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.037+0000 7fda125af640 1 -- 172.21.15.50:0/511136093 wait complete. 2024-04-23T18:17:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.037+0000 7fda125af640 1 Processor -- start 2024-04-23T18:17:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.037+0000 7fda125af640 1 -- start start 2024-04-23T18:17:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda125af640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda0c102a90 0x7fda0c198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda125af640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0c1053c0 0x7fda0c198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda125af640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda0c107cf0 0x7fda0c19f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda125af640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fda0c10db10 con 0x7fda0c1053c0 2024-04-23T18:17:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda125af640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fda0c10d990 con 0x7fda0c107cf0 2024-04-23T18:17:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda125af640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fda0c10dc90 con 0x7fda0c102a90 2024-04-23T18:17:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda0b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0c1053c0 0x7fda0c198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda0b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0c1053c0 0x7fda0c198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39800/0 (socket says 172.21.15.50:39800) 2024-04-23T18:17:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda0b7fe640 1 -- 172.21.15.50:0/1413690542 learned_addr learned my addr 172.21.15.50:0/1413690542 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda0bfff640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda0c102a90 0x7fda0c198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda10b4c640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda0c107cf0 0x7fda0c19f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda0b7fe640 1 -- 172.21.15.50:0/1413690542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda0c102a90 msgr2=0x7fda0c198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda0b7fe640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda0c102a90 0x7fda0c198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda0b7fe640 1 -- 172.21.15.50:0/1413690542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda0c107cf0 msgr2=0x7fda0c19f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda0b7fe640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda0c107cf0 0x7fda0c19f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda0b7fe640 1 -- 172.21.15.50:0/1413690542 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda0c19fe60 con 0x7fda0c1053c0 2024-04-23T18:17:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda10b4c640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda0c107cf0 0x7fda0c19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda0b7fe640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0c1053c0 0x7fda0c198780 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fda00006010 tx=0x7fda000042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda097fa640 1 -- 172.21.15.50:0/1413690542 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0003f030 con 0x7fda0c1053c0 2024-04-23T18:17:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda125af640 1 -- 172.21.15.50:0/1413690542 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda0c1a00f0 con 0x7fda0c1053c0 2024-04-23T18:17:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda097fa640 1 -- 172.21.15.50:0/1413690542 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda00003cb0 con 0x7fda0c1053c0 2024-04-23T18:17:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda097fa640 1 -- 172.21.15.50:0/1413690542 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00035070 con 0x7fda0c1053c0 2024-04-23T18:17:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.038+0000 7fda125af640 1 -- 172.21.15.50:0/1413690542 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda0c1a0660 con 0x7fda0c1053c0 2024-04-23T18:17:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.039+0000 7fda125af640 1 -- 172.21.15.50:0/1413690542 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda0c0fded0 con 0x7fda0c1053c0 2024-04-23T18:17:56.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.041+0000 7fda097fa640 1 -- 172.21.15.50:0/1413690542 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fda00042030 con 0x7fda0c1053c0 2024-04-23T18:17:56.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.041+0000 7fda097fa640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd9f40a3c40 0x7fd9f40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:56.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.041+0000 7fda0bfff640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd9f40a3c40 0x7fd9f40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:56.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.041+0000 7fda097fa640 1 -- 172.21.15.50:0/1413690542 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fda000f0fb0 con 0x7fda0c1053c0 2024-04-23T18:17:56.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.042+0000 7fda0bfff640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd9f40a3c40 0x7fd9f40a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc005fd0 tx=0x7fd9fc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.045+0000 7fda097fa640 1 -- 172.21.15.50:0/1413690542 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda000bd8f0 con 0x7fda0c1053c0 2024-04-23T18:17:56.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:56 smithi050 ceph-mon[29187]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:17:56.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:56 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/730884162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:56.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:56 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2790744641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:56.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:56 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1802596943' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.231+0000 7fda125af640 1 -- 172.21.15.50:0/1413690542 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fda0c02d810 con 0x7fda0c1053c0 2024-04-23T18:17:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.231+0000 7fda097fa640 1 -- 172.21.15.50:0/1413690542 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fda000bd1f0 con 0x7fda0c1053c0 2024-04-23T18:17:56.224 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2024-04-23T18:17:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.233+0000 7fda125af640 1 -- 172.21.15.50:0/1413690542 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd9f40a3c40 msgr2=0x7fd9f40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.233+0000 7fda125af640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd9f40a3c40 0x7fd9f40a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc005fd0 tx=0x7fd9fc005850 comp rx=0 tx=0).stop 2024-04-23T18:17:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.233+0000 7fda125af640 1 -- 172.21.15.50:0/1413690542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0c1053c0 msgr2=0x7fda0c198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.233+0000 7fda125af640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0c1053c0 0x7fda0c198780 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fda00006010 tx=0x7fda000042f0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.233+0000 7fda125af640 1 -- 172.21.15.50:0/1413690542 shutdown_connections 2024-04-23T18:17:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.233+0000 7fda125af640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd9f40a3c40 0x7fd9f40a6100 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.233+0000 7fda125af640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda0c107cf0 0x7fda0c19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.233+0000 7fda125af640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0c1053c0 0x7fda0c198780 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.233+0000 7fda125af640 1 --2- 172.21.15.50:0/1413690542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda0c102a90 0x7fda0c198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.233+0000 7fda125af640 1 -- 172.21.15.50:0/1413690542 >> 172.21.15.50:0/1413690542 conn(0x7fda0c0fc3e0 msgr2=0x7fda0c1034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:56.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.233+0000 7fda125af640 1 -- 172.21.15.50:0/1413690542 shutdown_connections 2024-04-23T18:17:56.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.233+0000 7fda125af640 1 -- 172.21.15.50:0/1413690542 wait complete. 2024-04-23T18:17:56.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:17:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:56 smithi188 ceph-mon[32694]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:17:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:56 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/730884162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:56 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2790744641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:56 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1802596943' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:56.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.348+0000 7fdb3b449640 1 -- 172.21.15.50:0/1910112852 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb34103b60 msgr2=0x7fdb3410a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.348+0000 7fdb3b449640 1 --2- 172.21.15.50:0/1910112852 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb34103b60 0x7fdb3410a3f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c009900 tx=0x7fdb2c02a410 comp rx=0 tx=0).stop 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.348+0000 7fdb3b449640 1 -- 172.21.15.50:0/1910112852 shutdown_connections 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.348+0000 7fdb3b449640 1 --2- 172.21.15.50:0/1910112852 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb34103b60 0x7fdb3410a3f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.348+0000 7fdb3b449640 1 --2- 172.21.15.50:0/1910112852 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb341031a0 0x7fdb34103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.348+0000 7fdb3b449640 1 --2- 172.21.15.50:0/1910112852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb34101fb0 0x7fdb341023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.348+0000 7fdb3b449640 1 -- 172.21.15.50:0/1910112852 >> 172.21.15.50:0/1910112852 conn(0x7fdb340fd520 msgr2=0x7fdb340ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.348+0000 7fdb3b449640 1 -- 172.21.15.50:0/1910112852 shutdown_connections 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.348+0000 7fdb3b449640 1 -- 172.21.15.50:0/1910112852 wait complete. 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb3b449640 1 Processor -- start 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb3b449640 1 -- start start 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb3b449640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb34101fb0 0x7fdb34075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb3b449640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb341031a0 0x7fdb34074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb3b449640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb34103b60 0x7fdb34074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb3b449640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3410daf0 con 0x7fdb341031a0 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb3b449640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3410d970 con 0x7fdb34103b60 2024-04-23T18:17:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb3b449640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3410dc70 con 0x7fdb34101fb0 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb389e4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb341031a0 0x7fdb34074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb389e4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb341031a0 0x7fdb34074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39814/0 (socket says 172.21.15.50:39814) 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb389e4640 1 -- 172.21.15.50:0/3120558255 learned_addr learned my addr 172.21.15.50:0/3120558255 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb391e5640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb34101fb0 0x7fdb34075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb399e6640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb34103b60 0x7fdb34074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb389e4640 1 -- 172.21.15.50:0/3120558255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb34101fb0 msgr2=0x7fdb34075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb389e4640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb34101fb0 0x7fdb34075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb389e4640 1 -- 172.21.15.50:0/3120558255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb34103b60 msgr2=0x7fdb34074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.349+0000 7fdb389e4640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb34103b60 0x7fdb34074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.350+0000 7fdb389e4640 1 -- 172.21.15.50:0/3120558255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb34075110 con 0x7fdb341031a0 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.350+0000 7fdb399e6640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb34103b60 0x7fdb34074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.350+0000 7fdb389e4640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb341031a0 0x7fdb34074340 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400d770 tx=0x7fdb2400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.350+0000 7fdb2a7fc640 1 -- 172.21.15.50:0/3120558255 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb24017070 con 0x7fdb341031a0 2024-04-23T18:17:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.350+0000 7fdb2a7fc640 1 -- 172.21.15.50:0/3120558255 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb240054d0 con 0x7fdb341031a0 2024-04-23T18:17:56.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.350+0000 7fdb2a7fc640 1 -- 172.21.15.50:0/3120558255 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb24004c30 con 0x7fdb341031a0 2024-04-23T18:17:56.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.350+0000 7fdb3b449640 1 -- 172.21.15.50:0/3120558255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb341a22c0 con 0x7fdb341031a0 2024-04-23T18:17:56.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.350+0000 7fdb3b449640 1 -- 172.21.15.50:0/3120558255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb341a2810 con 0x7fdb341031a0 2024-04-23T18:17:56.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.351+0000 7fdb3b449640 1 -- 172.21.15.50:0/3120558255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb341092f0 con 0x7fdb341031a0 2024-04-23T18:17:56.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.352+0000 7fdb2a7fc640 1 -- 172.21.15.50:0/3120558255 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdb24004df0 con 0x7fdb341031a0 2024-04-23T18:17:56.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.353+0000 7fdb2a7fc640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fdb200a3cc0 0x7fdb200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:56.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.353+0000 7fdb391e5640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fdb200a3cc0 0x7fdb200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:56.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.353+0000 7fdb2a7fc640 1 -- 172.21.15.50:0/3120558255 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fdb240d04e0 con 0x7fdb341031a0 2024-04-23T18:17:56.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.353+0000 7fdb391e5640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fdb200a3cc0 0x7fdb200a6180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000a300 tx=0x7fdb3000a290 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:56.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.356+0000 7fdb2a7fc640 1 -- 172.21.15.50:0/3120558255 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb2409cda0 con 0x7fdb341031a0 2024-04-23T18:17:56.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.543+0000 7fdb3b449640 1 -- 172.21.15.50:0/3120558255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdb3402d830 con 0x7fdb341031a0 2024-04-23T18:17:56.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.543+0000 7fdb2a7fc640 1 -- 172.21.15.50:0/3120558255 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fdb2409c6a0 con 0x7fdb341031a0 2024-04-23T18:17:56.536 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2024-04-23T18:17:56.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:56 smithi177 ceph-mon[32308]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:17:56.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:56 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/730884162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:56.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:56 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2790744641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:56.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:56 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1802596943' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:17:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.545+0000 7fdb3b449640 1 -- 172.21.15.50:0/3120558255 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fdb200a3cc0 msgr2=0x7fdb200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.545+0000 7fdb3b449640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fdb200a3cc0 0x7fdb200a6180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000a300 tx=0x7fdb3000a290 comp rx=0 tx=0).stop 2024-04-23T18:17:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.545+0000 7fdb3b449640 1 -- 172.21.15.50:0/3120558255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb341031a0 msgr2=0x7fdb34074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.545+0000 7fdb3b449640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb341031a0 0x7fdb34074340 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400d770 tx=0x7fdb2400dc30 comp rx=0 tx=0).stop 2024-04-23T18:17:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.545+0000 7fdb3b449640 1 -- 172.21.15.50:0/3120558255 shutdown_connections 2024-04-23T18:17:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.545+0000 7fdb3b449640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fdb200a3cc0 0x7fdb200a6180 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.545+0000 7fdb3b449640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb34103b60 0x7fdb34074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.545+0000 7fdb3b449640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb341031a0 0x7fdb34074340 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.545+0000 7fdb3b449640 1 --2- 172.21.15.50:0/3120558255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb34101fb0 0x7fdb34075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.545+0000 7fdb3b449640 1 -- 172.21.15.50:0/3120558255 >> 172.21.15.50:0/3120558255 conn(0x7fdb340fd520 msgr2=0x7fdb341064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.545+0000 7fdb3b449640 1 -- 172.21.15.50:0/3120558255 shutdown_connections 2024-04-23T18:17:56.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.545+0000 7fdb3b449640 1 -- 172.21.15.50:0/3120558255 wait complete. 2024-04-23T18:17:56.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:17:56.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.657+0000 7f93ba144640 1 -- 172.21.15.50:0/3819455002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b4101fc0 msgr2=0x7f93b41023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.657+0000 7f93ba144640 1 --2- 172.21.15.50:0/3819455002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b4101fc0 0x7f93b41023e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009900 tx=0x7f93a402a410 comp rx=0 tx=0).stop 2024-04-23T18:17:56.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 -- 172.21.15.50:0/3819455002 shutdown_connections 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 --2- 172.21.15.50:0/3819455002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93b4103b70 0x7f93b410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 --2- 172.21.15.50:0/3819455002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93b41031b0 0x7f93b4103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 --2- 172.21.15.50:0/3819455002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b4101fc0 0x7f93b41023e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 -- 172.21.15.50:0/3819455002 >> 172.21.15.50:0/3819455002 conn(0x7f93b40fd550 msgr2=0x7f93b40ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 -- 172.21.15.50:0/3819455002 shutdown_connections 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 -- 172.21.15.50:0/3819455002 wait complete. 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 Processor -- start 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 -- start start 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93b4101fc0 0x7f93b4198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b41031b0 0x7f93b41987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93b4103b70 0x7f93b419f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f93b410db50 con 0x7f93b41031b0 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f93b410d9d0 con 0x7f93b4103b70 2024-04-23T18:17:56.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.658+0000 7f93ba144640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f93b410dcd0 con 0x7f93b4101fc0 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b41031b0 0x7f93b41987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b41031b0 0x7f93b41987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39828/0 (socket says 172.21.15.50:39828) 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b2ffd640 1 -- 172.21.15.50:0/228500902 learned_addr learned my addr 172.21.15.50:0/228500902 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b37fe640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93b4101fc0 0x7f93b4198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b3fff640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93b4103b70 0x7f93b419f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b2ffd640 1 -- 172.21.15.50:0/228500902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93b4101fc0 msgr2=0x7f93b4198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b2ffd640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93b4101fc0 0x7f93b4198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b2ffd640 1 -- 172.21.15.50:0/228500902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93b4103b70 msgr2=0x7f93b419f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b2ffd640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93b4103b70 0x7f93b419f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b2ffd640 1 -- 172.21.15.50:0/228500902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b419fea0 con 0x7f93b41031b0 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b2ffd640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b41031b0 0x7f93b41987c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f93a800c970 tx=0x7f93a800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b0ff9640 1 -- 172.21.15.50:0/228500902 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a8007bb0 con 0x7f93b41031b0 2024-04-23T18:17:56.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b0ff9640 1 -- 172.21.15.50:0/228500902 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93a8007d50 con 0x7f93b41031b0 2024-04-23T18:17:56.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93b0ff9640 1 -- 172.21.15.50:0/228500902 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a800f450 con 0x7f93b41031b0 2024-04-23T18:17:56.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93ba144640 1 -- 172.21.15.50:0/228500902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b41a0190 con 0x7f93b41031b0 2024-04-23T18:17:56.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.659+0000 7f93ba144640 1 -- 172.21.15.50:0/228500902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b41a0690 con 0x7f93b41031b0 2024-04-23T18:17:56.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.660+0000 7f93ba144640 1 -- 172.21.15.50:0/228500902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b41093a0 con 0x7f93b41031b0 2024-04-23T18:17:56.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.661+0000 7f93b0ff9640 1 -- 172.21.15.50:0/228500902 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f93a8004f20 con 0x7f93b41031b0 2024-04-23T18:17:56.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.662+0000 7f93b0ff9640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f939c0a3c40 0x7f939c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:56.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.662+0000 7f93b37fe640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f939c0a3c40 0x7f939c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:56.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.662+0000 7f93b0ff9640 1 -- 172.21.15.50:0/228500902 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f93a80d0300 con 0x7f93b41031b0 2024-04-23T18:17:56.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.663+0000 7f93b37fe640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f939c0a3c40 0x7f939c0a6100 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f93a4005ab0 tx=0x7f93a4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:56.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.667+0000 7f93b0ff9640 1 -- 172.21.15.50:0/228500902 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f93a8014030 con 0x7f93b41031b0 2024-04-23T18:17:56.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.893+0000 7f93ba144640 1 -- 172.21.15.50:0/228500902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93b402d770 con 0x7f93b41031b0 2024-04-23T18:17:56.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.893+0000 7f93b0ff9640 1 -- 172.21.15.50:0/228500902 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11875 (secure 0 0 0) 0x7f93a809cb80 con 0x7f93b41031b0 2024-04-23T18:17:56.886 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:17:56.887 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:16:37.965071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:17:56.887 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 20 2024-04-23T18:17:56.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.896+0000 7f93ba144640 1 -- 172.21.15.50:0/228500902 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f939c0a3c40 msgr2=0x7f939c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.896+0000 7f93ba144640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f939c0a3c40 0x7f939c0a6100 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f93a4005ab0 tx=0x7f93a4005a20 comp rx=0 tx=0).stop 2024-04-23T18:17:56.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.896+0000 7f93ba144640 1 -- 172.21.15.50:0/228500902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b41031b0 msgr2=0x7f93b41987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:56.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.896+0000 7f93ba144640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b41031b0 0x7f93b41987c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f93a800c970 tx=0x7f93a800ce30 comp rx=0 tx=0).stop 2024-04-23T18:17:56.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.896+0000 7f93ba144640 1 -- 172.21.15.50:0/228500902 shutdown_connections 2024-04-23T18:17:56.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.896+0000 7f93ba144640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f939c0a3c40 0x7f939c0a6100 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.896+0000 7f93ba144640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93b4103b70 0x7f93b419f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.896+0000 7f93ba144640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93b41031b0 0x7f93b41987c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.896+0000 7f93ba144640 1 --2- 172.21.15.50:0/228500902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93b4101fc0 0x7f93b4198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:56.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.896+0000 7f93ba144640 1 -- 172.21.15.50:0/228500902 >> 172.21.15.50:0/228500902 conn(0x7f93b40fd550 msgr2=0x7f93b4106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:56.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.896+0000 7f93ba144640 1 -- 172.21.15.50:0/228500902 shutdown_connections 2024-04-23T18:17:56.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:56.896+0000 7f93ba144640 1 -- 172.21.15.50:0/228500902 wait complete. 2024-04-23T18:17:56.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:17:57.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.009+0000 7f40888a9640 1 -- 172.21.15.50:0/390474300 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4084102040 msgr2=0x7f40841024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.009+0000 7f40888a9640 1 --2- 172.21.15.50:0/390474300 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4084102040 0x7f40841024c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4074009900 tx=0x7f407402a410 comp rx=0 tx=0).stop 2024-04-23T18:17:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.009+0000 7f40888a9640 1 -- 172.21.15.50:0/390474300 shutdown_connections 2024-04-23T18:17:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.009+0000 7f40888a9640 1 --2- 172.21.15.50:0/390474300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4084102a00 0x7f4084109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.009+0000 7f40888a9640 1 --2- 172.21.15.50:0/390474300 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4084102040 0x7f40841024c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.009+0000 7f40888a9640 1 --2- 172.21.15.50:0/390474300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4084100e60 0x7f4084101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.009+0000 7f40888a9640 1 -- 172.21.15.50:0/390474300 >> 172.21.15.50:0/390474300 conn(0x7f40840fc440 msgr2=0x7f40840fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.009+0000 7f40888a9640 1 -- 172.21.15.50:0/390474300 shutdown_connections 2024-04-23T18:17:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.009+0000 7f40888a9640 1 -- 172.21.15.50:0/390474300 wait complete. 2024-04-23T18:17:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f40888a9640 1 Processor -- start 2024-04-23T18:17:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f40888a9640 1 -- start start 2024-04-23T18:17:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f40888a9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4084100e60 0x7f4084198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f40888a9640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4084102040 0x7f4084198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f40888a9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4084102a00 0x7f408419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f40888a9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f408410ca00 con 0x7f4084100e60 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f40888a9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f408410c880 con 0x7f4084102a00 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f40888a9640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f408410cb80 con 0x7f4084102040 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f408259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4084100e60 0x7f4084198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f408259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4084100e60 0x7f4084198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39844/0 (socket says 172.21.15.50:39844) 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f408259c640 1 -- 172.21.15.50:0/3506216456 learned_addr learned my addr 172.21.15.50:0/3506216456 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f4081d9b640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4084102040 0x7f4084198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f408259c640 1 -- 172.21.15.50:0/3506216456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4084102040 msgr2=0x7f4084198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f408259c640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4084102040 0x7f4084198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f408259c640 1 -- 172.21.15.50:0/3506216456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4084102a00 msgr2=0x7f408419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f408259c640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4084102a00 0x7f408419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f408259c640 1 -- 172.21.15.50:0/3506216456 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408419fdf0 con 0x7f4084100e60 2024-04-23T18:17:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f408259c640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4084100e60 0x7f4084198220 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f407800e9c0 tx=0x7f407800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f40737fe640 1 -- 172.21.15.50:0/3506216456 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407800cd80 con 0x7f4084100e60 2024-04-23T18:17:57.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f40737fe640 1 -- 172.21.15.50:0/3506216456 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40780035f0 con 0x7f4084100e60 2024-04-23T18:17:57.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f40888a9640 1 -- 172.21.15.50:0/3506216456 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40841a00e0 con 0x7f4084100e60 2024-04-23T18:17:57.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.010+0000 7f40737fe640 1 -- 172.21.15.50:0/3506216456 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4078010690 con 0x7f4084100e60 2024-04-23T18:17:57.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.011+0000 7f40888a9640 1 -- 172.21.15.50:0/3506216456 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40841a0680 con 0x7f4084100e60 2024-04-23T18:17:57.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.011+0000 7f40888a9640 1 -- 172.21.15.50:0/3506216456 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40841a0940 con 0x7f4084100e60 2024-04-23T18:17:57.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.013+0000 7f40737fe640 1 -- 172.21.15.50:0/3506216456 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f40780037a0 con 0x7f4084100e60 2024-04-23T18:17:57.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.013+0000 7f40737fe640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f406c0a3cc0 0x7f406c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.013+0000 7f4081d9b640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f406c0a3cc0 0x7f406c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.014+0000 7f40737fe640 1 -- 172.21.15.50:0/3506216456 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f40780d09f0 con 0x7f4084100e60 2024-04-23T18:17:57.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.014+0000 7f4081d9b640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f406c0a3cc0 0x7f406c0a6180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f407402a930 tx=0x7f4074000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.018+0000 7f40737fe640 1 -- 172.21.15.50:0/3506216456 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f407809d270 con 0x7f4084100e60 2024-04-23T18:17:57.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:57 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1413690542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:57.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:57 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3120558255' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:57.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:57 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/228500902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:57.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.210+0000 7f40888a9640 1 -- 172.21.15.50:0/3506216456 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40841a0b10 con 0x7f4084100e60 2024-04-23T18:17:57.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.212+0000 7f40737fe640 1 -- 172.21.15.50:0/3506216456 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22661 (secure 0 0 0) 0x7f407809cb70 con 0x7f4084100e60 2024-04-23T18:17:57.205 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:17:57.205 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":89,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:17:36.102736+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:17:57.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.214+0000 7f40888a9640 1 -- 172.21.15.50:0/3506216456 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f406c0a3cc0 msgr2=0x7f406c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.214+0000 7f40888a9640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f406c0a3cc0 0x7f406c0a6180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f407402a930 tx=0x7f4074000bc0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.214+0000 7f40888a9640 1 -- 172.21.15.50:0/3506216456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4084100e60 msgr2=0x7f4084198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.214+0000 7f40888a9640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4084100e60 0x7f4084198220 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f407800e9c0 tx=0x7f407800ee80 comp rx=0 tx=0).stop 2024-04-23T18:17:57.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.214+0000 7f40888a9640 1 -- 172.21.15.50:0/3506216456 shutdown_connections 2024-04-23T18:17:57.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.214+0000 7f40888a9640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f406c0a3cc0 0x7f406c0a6180 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.214+0000 7f40888a9640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4084102a00 0x7f408419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.214+0000 7f40888a9640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4084102040 0x7f4084198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.214+0000 7f40888a9640 1 --2- 172.21.15.50:0/3506216456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4084100e60 0x7f4084198220 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.214+0000 7f40888a9640 1 -- 172.21.15.50:0/3506216456 >> 172.21.15.50:0/3506216456 conn(0x7f40840fc440 msgr2=0x7f40840fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.214+0000 7f40888a9640 1 -- 172.21.15.50:0/3506216456 shutdown_connections 2024-04-23T18:17:57.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.214+0000 7f40888a9640 1 -- 172.21.15.50:0/3506216456 wait complete. 2024-04-23T18:17:57.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:17:57.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.327+0000 7f33ba04e640 1 -- 172.21.15.50:0/693567492 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4102a00 msgr2=0x7f33b4109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.327+0000 7f33ba04e640 1 --2- 172.21.15.50:0/693567492 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4102a00 0x7f33b4109320 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f33ac009ca0 tx=0x7f33ac02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.327+0000 7f33ba04e640 1 -- 172.21.15.50:0/693567492 shutdown_connections 2024-04-23T18:17:57.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.327+0000 7f33ba04e640 1 --2- 172.21.15.50:0/693567492 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4102a00 0x7f33b4109320 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.327+0000 7f33ba04e640 1 --2- 172.21.15.50:0/693567492 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b4102040 0x7f33b41024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.327+0000 7f33ba04e640 1 --2- 172.21.15.50:0/693567492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b4100e60 0x7f33b4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.327+0000 7f33ba04e640 1 -- 172.21.15.50:0/693567492 >> 172.21.15.50:0/693567492 conn(0x7f33b40fc440 msgr2=0x7f33b40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.327+0000 7f33ba04e640 1 -- 172.21.15.50:0/693567492 shutdown_connections 2024-04-23T18:17:57.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.327+0000 7f33ba04e640 1 -- 172.21.15.50:0/693567492 wait complete. 2024-04-23T18:17:57.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.328+0000 7f33ba04e640 1 Processor -- start 2024-04-23T18:17:57.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.328+0000 7f33ba04e640 1 -- start start 2024-04-23T18:17:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.328+0000 7f33ba04e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4100e60 0x7f33b4198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.328+0000 7f33ba04e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b4102040 0x7f33b4198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.328+0000 7f33ba04e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b4102a00 0x7f33b419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.328+0000 7f33ba04e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f33b410ca00 con 0x7f33b4100e60 2024-04-23T18:17:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.328+0000 7f33ba04e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f33b410c880 con 0x7f33b4102a00 2024-04-23T18:17:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.328+0000 7f33ba04e640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f33b410cb80 con 0x7f33b4102040 2024-04-23T18:17:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.328+0000 7f33b37fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4100e60 0x7f33b4198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.328+0000 7f33b37fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4100e60 0x7f33b4198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39868/0 (socket says 172.21.15.50:39868) 2024-04-23T18:17:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.328+0000 7f33b37fe640 1 -- 172.21.15.50:0/1552693384 learned_addr learned my addr 172.21.15.50:0/1552693384 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.329+0000 7f33b2ffd640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b4102040 0x7f33b4198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.329+0000 7f33b37fe640 1 -- 172.21.15.50:0/1552693384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b4102040 msgr2=0x7f33b4198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.329+0000 7f33b37fe640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b4102040 0x7f33b4198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.329+0000 7f33b37fe640 1 -- 172.21.15.50:0/1552693384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b4102a00 msgr2=0x7f33b419f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.329+0000 7f33b37fe640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b4102a00 0x7f33b419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.329+0000 7f33b37fe640 1 -- 172.21.15.50:0/1552693384 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b419fd30 con 0x7f33b4100e60 2024-04-23T18:17:57.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.329+0000 7f33b37fe640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4100e60 0x7f33b4198110 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f33a400c970 tx=0x7f33a400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.329+0000 7f33b0ff9640 1 -- 172.21.15.50:0/1552693384 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4007c10 con 0x7f33b4100e60 2024-04-23T18:17:57.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.329+0000 7f33b0ff9640 1 -- 172.21.15.50:0/1552693384 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33a4007db0 con 0x7f33b4100e60 2024-04-23T18:17:57.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.329+0000 7f33ba04e640 1 -- 172.21.15.50:0/1552693384 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33b419ffc0 con 0x7f33b4100e60 2024-04-23T18:17:57.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.329+0000 7f33b0ff9640 1 -- 172.21.15.50:0/1552693384 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a400f870 con 0x7f33b4100e60 2024-04-23T18:17:57.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.329+0000 7f33ba04e640 1 -- 172.21.15.50:0/1552693384 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33b41a04b0 con 0x7f33b4100e60 2024-04-23T18:17:57.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.330+0000 7f33ba04e640 1 -- 172.21.15.50:0/1552693384 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33b41a0770 con 0x7f33b4100e60 2024-04-23T18:17:57.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.331+0000 7f33b0ff9640 1 -- 172.21.15.50:0/1552693384 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f33a400fa10 con 0x7f33b4100e60 2024-04-23T18:17:57.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.332+0000 7f33b0ff9640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f339c0a3c40 0x7f339c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.332+0000 7f33b2ffd640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f339c0a3c40 0x7f339c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.332+0000 7f33b0ff9640 1 -- 172.21.15.50:0/1552693384 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f33a40d0c80 con 0x7f33b4100e60 2024-04-23T18:17:57.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.332+0000 7f33b2ffd640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f339c0a3c40 0x7f339c0a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f33a8005fd0 tx=0x7f33a8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.335+0000 7f33b0ff9640 1 -- 172.21.15.50:0/1552693384 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f33a409d5e0 con 0x7f33b4100e60 2024-04-23T18:17:57.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:57 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1413690542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:57.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:57 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3120558255' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:57.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:57 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/228500902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:57.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.561+0000 7f33ba04e640 1 -- 172.21.15.50:0/1552693384 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33b41a0b00 con 0x7f33b4100e60 2024-04-23T18:17:57.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.561+0000 7f33b0ff9640 1 -- 172.21.15.50:0/1552693384 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11875 (secure 0 0 0) 0x7f33a409cee0 con 0x7f33b4100e60 2024-04-23T18:17:57.554 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:17:57.554 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:16:37.965071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:17:57.555 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 20 2024-04-23T18:17:57.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.563+0000 7f33ba04e640 1 -- 172.21.15.50:0/1552693384 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f339c0a3c40 msgr2=0x7f339c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.563+0000 7f33ba04e640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f339c0a3c40 0x7f339c0a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f33a8005fd0 tx=0x7f33a8005850 comp rx=0 tx=0).stop 2024-04-23T18:17:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.563+0000 7f33ba04e640 1 -- 172.21.15.50:0/1552693384 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4100e60 msgr2=0x7f33b4198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.563+0000 7f33ba04e640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4100e60 0x7f33b4198110 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f33a400c970 tx=0x7f33a400ce30 comp rx=0 tx=0).stop 2024-04-23T18:17:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.563+0000 7f33ba04e640 1 -- 172.21.15.50:0/1552693384 shutdown_connections 2024-04-23T18:17:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.563+0000 7f33ba04e640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f339c0a3c40 0x7f339c0a6100 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.563+0000 7f33ba04e640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b4102a00 0x7f33b419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.563+0000 7f33ba04e640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33b4102040 0x7f33b4198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.563+0000 7f33ba04e640 1 --2- 172.21.15.50:0/1552693384 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4100e60 0x7f33b4198110 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.563+0000 7f33ba04e640 1 -- 172.21.15.50:0/1552693384 >> 172.21.15.50:0/1552693384 conn(0x7f33b40fc440 msgr2=0x7f33b40fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.564+0000 7f33ba04e640 1 -- 172.21.15.50:0/1552693384 shutdown_connections 2024-04-23T18:17:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.564+0000 7f33ba04e640 1 -- 172.21.15.50:0/1552693384 wait complete. 2024-04-23T18:17:57.568 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T18:16:33.316468+0000', 'modified': '2024-04-23T18:16:37.965071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24506, 'mds_1': 24499, 'mds_2': 24475}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24440': {'gid': 24440, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6835/475975073', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6834', 'nonce': 475975073}, {'type': 'v1', 'addr': '172.21.15.177:6835', 'nonce': 475975073}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6835/1248812622', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6834', 'nonce': 1248812622}, {'type': 'v1', 'addr': '172.21.15.188:6835', 'nonce': 1248812622}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6837/1048893636', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6836', 'nonce': 1048893636}, {'type': 'v1', 'addr': '172.21.15.177:6837', 'nonce': 1048893636}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.188:6837/2190889779', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6836', 'nonce': 2190889779}, {'type': 'v1', 'addr': '172.21.15.188:6837', 'nonce': 2190889779}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.188:6839/2202634301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6838', 'nonce': 2202634301}, {'type': 'v1', 'addr': '172.21.15.188:6839', 'nonce': 2202634301}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.177:6839/2936205355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 2936205355}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 2936205355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24506, 'qdb_cluster': [24475, 24506, 24499]} 2024-04-23T18:17:57.568 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-23T18:17:57.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-23T18:17:57.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:57 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1413690542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:17:57.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:57 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3120558255' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:17:57.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:57 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/228500902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.678+0000 7faf0d1cd640 1 -- 172.21.15.50:0/203064978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf08102a00 msgr2=0x7faf08109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.678+0000 7faf0d1cd640 1 --2- 172.21.15.50:0/203064978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf08102a00 0x7faf08109320 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faf00009ca0 tx=0x7faf0002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.678+0000 7faf0d1cd640 1 -- 172.21.15.50:0/203064978 shutdown_connections 2024-04-23T18:17:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.678+0000 7faf0d1cd640 1 --2- 172.21.15.50:0/203064978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf08102a00 0x7faf08109320 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.678+0000 7faf0d1cd640 1 --2- 172.21.15.50:0/203064978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf08102040 0x7faf081024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.678+0000 7faf0d1cd640 1 --2- 172.21.15.50:0/203064978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf08100e60 0x7faf08101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.678+0000 7faf0d1cd640 1 -- 172.21.15.50:0/203064978 >> 172.21.15.50:0/203064978 conn(0x7faf080fc440 msgr2=0x7faf080fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.678+0000 7faf0d1cd640 1 -- 172.21.15.50:0/203064978 shutdown_connections 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.678+0000 7faf0d1cd640 1 -- 172.21.15.50:0/203064978 wait complete. 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.678+0000 7faf0d1cd640 1 Processor -- start 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0d1cd640 1 -- start start 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0d1cd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf08100e60 0x7faf08104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0d1cd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf08102040 0x7faf081034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0d1cd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf08102a00 0x7faf081039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0d1cd640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faf0810c9f0 con 0x7faf08102a00 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0d1cd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faf0810c870 con 0x7faf08102040 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0d1cd640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faf0810cb70 con 0x7faf08100e60 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0759e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf08102a00 0x7faf081039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0759e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf08102a00 0x7faf081039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39888/0 (socket says 172.21.15.50:39888) 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0759e640 1 -- 172.21.15.50:0/3376196728 learned_addr learned my addr 172.21.15.50:0/3376196728 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0659c640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf08102040 0x7faf081034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf06d9d640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf08100e60 0x7faf08104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0759e640 1 -- 172.21.15.50:0/3376196728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf08100e60 msgr2=0x7faf08104e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0759e640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf08100e60 0x7faf08104e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0759e640 1 -- 172.21.15.50:0/3376196728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf08102040 msgr2=0x7faf081034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0759e640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf08102040 0x7faf081034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0759e640 1 -- 172.21.15.50:0/3376196728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf08104280 con 0x7faf08102a00 2024-04-23T18:17:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf06d9d640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf08100e60 0x7faf08104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.679+0000 7faf0759e640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf08102a00 0x7faf081039f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faf000097b0 tx=0x7faf00003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.680+0000 7faeeffff640 1 -- 172.21.15.50:0/3376196728 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00004240 con 0x7faf08102a00 2024-04-23T18:17:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.680+0000 7faeeffff640 1 -- 172.21.15.50:0/3376196728 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf000317c0 con 0x7faf08102a00 2024-04-23T18:17:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.680+0000 7faf0d1cd640 1 -- 172.21.15.50:0/3376196728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf081a2190 con 0x7faf08102a00 2024-04-23T18:17:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.680+0000 7faeeffff640 1 -- 172.21.15.50:0/3376196728 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0002fc70 con 0x7faf08102a00 2024-04-23T18:17:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.680+0000 7faf0d1cd640 1 -- 172.21.15.50:0/3376196728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf081a26e0 con 0x7faf08102a00 2024-04-23T18:17:57.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.682+0000 7faeeffff640 1 -- 172.21.15.50:0/3376196728 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faf000043e0 con 0x7faf08102a00 2024-04-23T18:17:57.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf0ffc3740 1 Processor -- start 2024-04-23T18:17:57.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf0ffc3740 1 -- start start 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf0ffc3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb2b80 0x55f011fb2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf0ffc3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 0x55f011fb41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf0ffc3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb4710 0x55f011fbafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf0ffc3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f011f17c50 con 0x55f011fb3d50 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf0ffc3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f011eb4830 con 0x55f011fb4710 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf0ffc3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f011f13c10 con 0x55f011fb2b80 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf06d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 0x55f011fb41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faeeffff640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faef00a3c40 0x7faef00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf06d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 0x55f011fb41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39900/0 (socket says 172.21.15.50:39900) 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf06d9d640 1 -- 172.21.15.50:0/3392792510 learned_addr learned my addr 172.21.15.50:0/3392792510 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf0659c640 1 --2- 172.21.15.50:0/3392792510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb4710 0x55f011fbafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf0659c640 1 --2- 172.21.15.50:0/3392792510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb2b80 0x55f011fb2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf0759e640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faef00a3c40 0x7faef00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf06d9d640 1 -- 172.21.15.50:0/3392792510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb2b80 msgr2=0x55f011fb2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf06d9d640 1 --2- 172.21.15.50:0/3392792510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb2b80 0x55f011fb2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faeeffff640 1 -- 172.21.15.50:0/3376196728 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7faf000f1ea0 con 0x7faf08102a00 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf06d9d640 1 -- 172.21.15.50:0/3392792510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb4710 msgr2=0x55f011fbafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf06d9d640 1 --2- 172.21.15.50:0/3392792510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb4710 0x55f011fbafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf06d9d640 1 -- 172.21.15.50:0/3392792510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f011c2a2d0 con 0x55f011fb3d50 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.683+0000 7faf0659c640 1 --2- 172.21.15.50:0/3392792510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb4710 0x55f011fbafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.684+0000 7faf06d9d640 1 --2- 172.21.15.50:0/3392792510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 0x55f011fb41d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7faef800bd30 tx=0x7faef800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2a39c573c4e6a2a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.684+0000 7faf0759e640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faef00a3c40 0x7faef00a6100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7faf000c6950 tx=0x7faf00090e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.684+0000 7faeed7fa640 1 -- 172.21.15.50:0/3392792510 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8015070 con 0x55f011fb3d50 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.684+0000 7faeed7fa640 1 -- 172.21.15.50:0/3392792510 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef800be20 con 0x55f011fb3d50 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.684+0000 7faeed7fa640 1 -- 172.21.15.50:0/3392792510 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef80106c0 con 0x55f011fb3d50 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.684+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3392792510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 msgr2=0x55f011fb41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.684+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/3392792510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 0x55f011fb41d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7faef800bd30 tx=0x7faef800f730 comp rx=0 tx=0).stop 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3392792510 shutdown_connections 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/3392792510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb4710 0x55f011fbafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/3392792510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 0x55f011fb41d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/3392792510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb2b80 0x55f011fb2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3392792510 >> 172.21.15.50:0/3392792510 conn(0x55f011fae070 msgr2=0x55f011fae470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3392792510 shutdown_connections 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3392792510 wait complete. 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 -- start start 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 Processor -- start 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 0x55f011fbd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb3d50 0x55f011fbd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 0x55f011fc4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f011fbb720 con 0x55f011fb2b80 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f011eb2470 con 0x55f011fb3d50 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f011fbb8a0 con 0x55f011fb4710 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0ffc3740 10 client.0 _ll_register_callbacks cb 0x55f011ed7f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 0x55f011fbd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 0x55f011fbd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39912/0 (socket says 172.21.15.50:39912) 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0659c640 1 -- 172.21.15.50:0/3419184019 learned_addr learned my addr 172.21.15.50:0/3419184019 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf06d9d640 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb3d50 0x55f011fbd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0659c640 1 -- 172.21.15.50:0/3419184019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 msgr2=0x55f011fc4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0659c640 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 0x55f011fc4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0659c640 1 -- 172.21.15.50:0/3419184019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb3d50 msgr2=0x55f011fbd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.685+0000 7faf0659c640 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb3d50 0x55f011fbd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faf0659c640 1 -- 172.21.15.50:0/3419184019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f011cf46b0 con 0x55f011fb2b80 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faf06d9d640 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb3d50 0x55f011fbd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faf0659c640 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 0x55f011fbd010 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faefc011ed0 tx=0x7faefc00f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c660e8bcf14c57 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faeeaff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faeeaff5640 1 -- 172.21.15.50:0/3419184019 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc005220 con 0x55f011fb2b80 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faeeaff5640 1 -- 172.21.15.50:0/3419184019 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7faefc0053c0 con 0x55f011fb2b80 2024-04-23T18:17:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3419184019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faef800be20 con 0x55f011fb2b80 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faeeaff5640 1 -- 172.21.15.50:0/3419184019 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7faefc0055c0 con 0x55f011fb2b80 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faf0ffc3740 10 client.15033 fetch_fsmap learned FSMap version 20 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3419184019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f011fc7970 con 0x55f011fb2b80 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faeeaff5640 1 -- 172.21.15.50:0/3419184019 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef800be20 con 0x55f011fb2b80 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faeeaff5640 1 -- 172.21.15.50:0/3419184019 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7faefc00e9b0 con 0x55f011fb2b80 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faf0ffc3740 10 client.15033 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faf0ffc3740 10 client.15033 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faf0ffc3740 20 client.15033 populate_metadata read hostname 'smithi050' 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55f011fca010 0x55f011fcb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faf0ffc3740 4 client.15033 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.686+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3419184019 --> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f011f25670 con 0x55f011fca010 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.687+0000 7faf06d9d640 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55f011fca010 0x55f011fcb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.687+0000 7faf06d9d640 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55f011fca010 0x55f011fcb480 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e9adc3e9f77f2179 server_cookie=26551891202bf788 in_seq=0 out_seq=0 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.687+0000 7faeeaff5640 10 client.15033 ms_handle_connect on v2:172.21.15.188:6836/2190889779 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faeeaff5640 1 -- 172.21.15.50:0/3419184019 <== mds.2 v2:172.21.15.188:6836/2190889779 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f011f25670 con 0x55f011fca010 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faeeaff5640 10 client.15033 handle_command_reply: tid=0 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faf0ffc3740 1 client.15033 shutdown 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3419184019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 msgr2=0x55f011fbd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 0x55f011fbd010 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faefc011ed0 tx=0x7faefc00f7f0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3419184019 shutdown_connections 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55f011fca010 0x55f011fcb480 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 0x55f011fc4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb3d50 0x55f011fbd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faeeaff5640 0 client.15033 ms_handle_reset on v2:172.21.15.188:6836/2190889779 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/3419184019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 0x55f011fbd010 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3419184019 >> 172.21.15.50:0/3419184019 conn(0x55f011fae070 msgr2=0x55f011fad1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3419184019 shutdown_connections 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faf0ffc3740 1 -- 172.21.15.50:0/3419184019 wait complete. 2024-04-23T18:17:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.688+0000 7faf0ffc3740 20 client.15033 trim_cache size 0 max 16384 2024-04-23T18:17:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.706+0000 7faf0ffc3740 1 Processor -- start 2024-04-23T18:17:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.706+0000 7faf0ffc3740 1 -- start start 2024-04-23T18:17:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf0ffc3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb2b80 0x55f011fbbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf0ffc3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 0x55f011fc4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf0ffc3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 0x55f011fb77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf0ffc3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f011fc97c0 con 0x55f011fb3d50 2024-04-23T18:17:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf0ffc3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f011fc9e90 con 0x55f011fb2b80 2024-04-23T18:17:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf0ffc3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f011fca560 con 0x55f011fb4710 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf06d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 0x55f011fc4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf06d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 0x55f011fc4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39926/0 (socket says 172.21.15.50:39926) 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf06d9d640 1 -- 172.21.15.50:0/1593710117 learned_addr learned my addr 172.21.15.50:0/1593710117 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf0659c640 1 --2- 172.21.15.50:0/1593710117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 0x55f011fb77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf06d9d640 1 -- 172.21.15.50:0/1593710117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 msgr2=0x55f011fb77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf06d9d640 1 --2- 172.21.15.50:0/1593710117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 0x55f011fb77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf0659c640 1 --2- 172.21.15.50:0/1593710117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb2b80 0x55f011fbbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf06d9d640 1 -- 172.21.15.50:0/1593710117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb2b80 msgr2=0x55f011fbbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf06d9d640 1 --2- 172.21.15.50:0/1593710117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb2b80 0x55f011fbbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf06d9d640 1 -- 172.21.15.50:0/1593710117 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f011fbaae0 con 0x55f011fb3d50 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faf06d9d640 1 --2- 172.21.15.50:0/1593710117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 0x55f011fc4520 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x55f011faef40 tx=0x7faef8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c35e5025c1469 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faeecff9640 1 -- 172.21.15.50:0/1593710117 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8015040 con 0x55f011fb3d50 2024-04-23T18:17:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faeecff9640 1 -- 172.21.15.50:0/1593710117 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef8005480 con 0x55f011fb3d50 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.707+0000 7faeecff9640 1 -- 172.21.15.50:0/1593710117 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef80297c0 con 0x55f011fb3d50 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1593710117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 msgr2=0x55f011fc4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/1593710117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 0x55f011fc4520 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x55f011faef40 tx=0x7faef8005850 comp rx=0 tx=0).stop 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1593710117 shutdown_connections 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/1593710117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 0x55f011fb77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/1593710117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb3d50 0x55f011fc4520 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/1593710117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb2b80 0x55f011fbbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1593710117 >> 172.21.15.50:0/1593710117 conn(0x55f011fae070 msgr2=0x55f011fa90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1593710117 shutdown_connections 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1593710117 wait complete. 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 -- start start 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 Processor -- start 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 0x55f011fbfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb3d50 0x55f011fbfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 0x55f011fc2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f011fcb300 con 0x55f011fb2b80 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f011fcac30 con 0x55f011fb3d50 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f011fcb9d0 con 0x55f011fb4710 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0ffc3740 10 client.0 _ll_register_callbacks cb 0x7faefc00acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.708+0000 7faf0659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 0x55f011fbfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf0659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 0x55f011fbfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39938/0 (socket says 172.21.15.50:39938) 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf0659c640 1 -- 172.21.15.50:0/1020075893 learned_addr learned my addr 172.21.15.50:0/1020075893 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf0659c640 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 0x55f011fc2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf06d9d640 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb3d50 0x55f011fbfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf0659c640 1 -- 172.21.15.50:0/1020075893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 msgr2=0x55f011fc2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf0659c640 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 0x55f011fc2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf0659c640 1 -- 172.21.15.50:0/1020075893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb3d50 msgr2=0x55f011fbfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf0659c640 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb3d50 0x55f011fbfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf0659c640 1 -- 172.21.15.50:0/1020075893 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f011fc3590 con 0x55f011fb2b80 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf0659c640 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 0x55f011fbfab0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7faefc00ada0 tx=0x7faefc013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c42c39bb589528 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faeeaff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faeeaff5640 1 -- 172.21.15.50:0/1020075893 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc00e850 con 0x55f011fb2b80 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faeeaff5640 1 -- 172.21.15.50:0/1020075893 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7faefc00e9f0 con 0x55f011fb2b80 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1020075893 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faef8005480 con 0x55f011fb2b80 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faeeaff5640 1 -- 172.21.15.50:0/1020075893 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7faefc0227c0 con 0x55f011fb2b80 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf0ffc3740 10 client.15039 fetch_fsmap learned FSMap version 20 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1020075893 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f01202ffc0 con 0x55f011fb2b80 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.709+0000 7faeeaff5640 1 -- 172.21.15.50:0/1020075893 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef8005480 con 0x55f011fb2b80 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.710+0000 7faeeaff5640 1 -- 172.21.15.50:0/1020075893 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7faefc03a760 con 0x55f011fb2b80 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.710+0000 7faf0ffc3740 10 client.15039 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.710+0000 7faf0ffc3740 10 client.15039 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-23T18:17:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.710+0000 7faf0ffc3740 20 client.15039 populate_metadata read hostname 'smithi050' 2024-04-23T18:17:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.710+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55f012034ae0 0x55f0120337f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.710+0000 7faf0ffc3740 4 client.15039 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:17:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.710+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1020075893 --> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] -- command(tid 0: {"prefix": "status"}) -- 0x55f011f1f4e0 con 0x55f012034ae0 2024-04-23T18:17:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.710+0000 7faf06d9d640 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55f012034ae0 0x55f0120337f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf06d9d640 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55f012034ae0 0x55f0120337f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ddfd4e534a50d2d5 server_cookie=17f8bf9b37f44fb1 in_seq=0 out_seq=0 2024-04-23T18:17:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faeeaff5640 10 client.15039 ms_handle_connect on v2:172.21.15.188:6836/2190889779 2024-04-23T18:17:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faeeaff5640 1 -- 172.21.15.50:0/1020075893 <== mds.2 v2:172.21.15.188:6836/2190889779 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x55f011f1f4e0 con 0x55f012034ae0 2024-04-23T18:17:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faeeaff5640 10 client.15039 handle_command_reply: tid=0 2024-04-23T18:17:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf0ffc3740 1 client.15039 shutdown 2024-04-23T18:17:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1020075893 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 msgr2=0x55f011fbfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 0x55f011fbfab0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7faefc00ada0 tx=0x7faefc013040 comp rx=0 tx=0).stop 2024-04-23T18:17:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1020075893 shutdown_connections 2024-04-23T18:17:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55f012034ae0 0x55f0120337f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f011fb4710 0x55f011fc2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f011fb3d50 0x55f011fbfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faeeaff5640 0 client.15039 ms_handle_reset on v2:172.21.15.188:6836/2190889779 2024-04-23T18:17:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf0ffc3740 1 --2- 172.21.15.50:0/1020075893 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f011fb2b80 0x55f011fbfab0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1020075893 >> 172.21.15.50:0/1020075893 conn(0x55f011fae070 msgr2=0x55f011fb77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1020075893 shutdown_connections 2024-04-23T18:17:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf0ffc3740 1 -- 172.21.15.50:0/1020075893 wait complete. 2024-04-23T18:17:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.711+0000 7faf0ffc3740 20 client.15039 trim_cache size 0 max 16384 2024-04-23T18:17:57.711 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:17:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:17:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 2, 2024-04-23T18:17:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 24475, 2024-04-23T18:17:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2024-04-23T18:17:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2024-04-23T18:17:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2024-04-23T18:17:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 80.750722173, 2024-04-23T18:17:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 20, 2024-04-23T18:17:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 89, 2024-04-23T18:17:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 87, 2024-04-23T18:17:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 102.786131544 2024-04-23T18:17:57.713 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:17:57.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.712+0000 7faf0d1cd640 1 -- 172.21.15.50:0/3376196728 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faef00a3c40 msgr2=0x7faef00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.712+0000 7faf0d1cd640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faef00a3c40 0x7faef00a6100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7faf000c6950 tx=0x7faf00090e70 comp rx=0 tx=0).stop 2024-04-23T18:17:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.712+0000 7faf0d1cd640 1 -- 172.21.15.50:0/3376196728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf08102a00 msgr2=0x7faf081039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.712+0000 7faf0d1cd640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf08102a00 0x7faf081039f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faf000097b0 tx=0x7faf00003c20 comp rx=0 tx=0).stop 2024-04-23T18:17:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.712+0000 7faf0d1cd640 1 -- 172.21.15.50:0/3376196728 shutdown_connections 2024-04-23T18:17:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.712+0000 7faf0d1cd640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faef00a3c40 0x7faef00a6100 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.712+0000 7faf0d1cd640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf08102a00 0x7faf081039f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.712+0000 7faf0d1cd640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf08102040 0x7faf081034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.712+0000 7faf0d1cd640 1 --2- 172.21.15.50:0/3376196728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf08100e60 0x7faf08104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.712+0000 7faf0d1cd640 1 -- 172.21.15.50:0/3376196728 >> 172.21.15.50:0/3376196728 conn(0x7faf080fc440 msgr2=0x7faf080fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.712+0000 7faf0d1cd640 1 -- 172.21.15.50:0/3376196728 shutdown_connections 2024-04-23T18:17:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.712+0000 7faf0d1cd640 1 -- 172.21.15.50:0/3376196728 wait complete. 2024-04-23T18:17:57.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-23T18:17:57.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.825+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/2234644288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc4102040 msgr2=0x7f7fc41024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.825+0000 7f7fcb1ad640 1 --2- 172.21.15.50:0/2234644288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc4102040 0x7f7fc41024c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009900 tx=0x7f7fb002a410 comp rx=0 tx=0).stop 2024-04-23T18:17:57.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.825+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/2234644288 shutdown_connections 2024-04-23T18:17:57.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.825+0000 7f7fcb1ad640 1 --2- 172.21.15.50:0/2234644288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fc4102a00 0x7f7fc4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.825+0000 7f7fcb1ad640 1 --2- 172.21.15.50:0/2234644288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc4102040 0x7f7fc41024c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.825+0000 7f7fcb1ad640 1 --2- 172.21.15.50:0/2234644288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fc4100e60 0x7f7fc4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.825+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/2234644288 >> 172.21.15.50:0/2234644288 conn(0x7f7fc40fc440 msgr2=0x7f7fc40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.825+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/2234644288 shutdown_connections 2024-04-23T18:17:57.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.825+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/2234644288 wait complete. 2024-04-23T18:17:57.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fcb1ad640 1 Processor -- start 2024-04-23T18:17:57.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fcb1ad640 1 -- start start 2024-04-23T18:17:57.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fcb1ad640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fc4100e60 0x7f7fc406b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fcb1ad640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fc4102040 0x7f7fc419cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fcb1ad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc4102a00 0x7f7fc419f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fcb1ad640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc410ca10 con 0x7f7fc4102a00 2024-04-23T18:17:57.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fcb1ad640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc410c890 con 0x7f7fc4100e60 2024-04-23T18:17:57.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fcb1ad640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc410cb90 con 0x7f7fc4102040 2024-04-23T18:17:57.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fc974a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc4102a00 0x7f7fc419f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fc974a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc4102a00 0x7f7fc419f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39948/0 (socket says 172.21.15.50:39948) 2024-04-23T18:17:57.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fc974a640 1 -- 172.21.15.50:0/3633324784 learned_addr learned my addr 172.21.15.50:0/3633324784 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fc8f49640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fc4100e60 0x7f7fc406b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fc3fff640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fc4102040 0x7f7fc419cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fc974a640 1 -- 172.21.15.50:0/3633324784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fc4102040 msgr2=0x7f7fc419cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fc974a640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fc4102040 0x7f7fc419cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fc974a640 1 -- 172.21.15.50:0/3633324784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fc4100e60 msgr2=0x7f7fc406b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fc974a640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fc4100e60 0x7f7fc406b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fc974a640 1 -- 172.21.15.50:0/3633324784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc419fe60 con 0x7f7fc4102a00 2024-04-23T18:17:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fc8f49640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fc4100e60 0x7f7fc406b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.826+0000 7f7fc974a640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc4102a00 0x7f7fc419f550 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00eaa0 tx=0x7f7fbc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.827+0000 7f7fc1ffb640 1 -- 172.21.15.50:0/3633324784 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc00cdb0 con 0x7f7fc4102a00 2024-04-23T18:17:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.827+0000 7f7fc1ffb640 1 -- 172.21.15.50:0/3633324784 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fbc005020 con 0x7f7fc4102a00 2024-04-23T18:17:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.827+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/3633324784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc41a00f0 con 0x7f7fc4102a00 2024-04-23T18:17:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.827+0000 7f7fc1ffb640 1 -- 172.21.15.50:0/3633324784 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc0116d0 con 0x7f7fc4102a00 2024-04-23T18:17:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.827+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/3633324784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc41a0640 con 0x7f7fc4102a00 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.829+0000 7f7fc1ffb640 1 -- 172.21.15.50:0/3633324784 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7fbc0051d0 con 0x7f7fc4102a00 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.829+0000 7f7fcdfa1740 1 Processor -- start 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.829+0000 7f7fcdfa1740 1 -- start start 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.829+0000 7f7fcdfa1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023ab80 0x55ffd023afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.829+0000 7f7fcdfa1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd023c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.829+0000 7f7fcdfa1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 0x55ffd0242fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.829+0000 7f7fc1ffb640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7fb40a3cc0 0x7f7fb40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.829+0000 7f7fcdfa1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ffd019fc50 con 0x55ffd023c710 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.829+0000 7f7fcdfa1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ffd013c830 con 0x55ffd023ab80 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.829+0000 7f7fcdfa1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ffd019bc10 con 0x55ffd023bd50 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc974a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 0x55ffd0242fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc3fff640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7fb40a3cc0 0x7f7fb40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc974a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 0x55ffd0242fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39954/0 (socket says 172.21.15.50:39954) 2024-04-23T18:17:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc974a640 1 -- 172.21.15.50:0/2839353960 learned_addr learned my addr 172.21.15.50:0/2839353960 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc1ffb640 1 -- 172.21.15.50:0/3633324784 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f7fbc0d0810 con 0x7f7fc4102a00 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc8f49640 1 --2- 172.21.15.50:0/2839353960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd023c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc3fff640 1 --2- 172.21.15.50:0/2839353960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023ab80 0x55ffd023afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc974a640 1 -- 172.21.15.50:0/2839353960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 msgr2=0x55ffd023c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc974a640 1 --2- 172.21.15.50:0/2839353960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd023c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc974a640 1 -- 172.21.15.50:0/2839353960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023ab80 msgr2=0x55ffd023afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc974a640 1 --2- 172.21.15.50:0/2839353960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023ab80 0x55ffd023afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc974a640 1 -- 172.21.15.50:0/2839353960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffcfeb22d0 con 0x55ffd023c710 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc974a640 1 --2- 172.21.15.50:0/2839353960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 0x55ffd0242fa0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0ce1b0 tx=0x7f7fbc06c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=328bcde926c8777f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7fc3fff640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7fb40a3cc0 0x7f7fb40a6180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0031500 tx=0x7f7fb0032d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7faeffd640 1 -- 172.21.15.50:0/2839353960 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc01f030 con 0x55ffd023c710 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7faeffd640 1 -- 172.21.15.50:0/2839353960 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fbc09d050 con 0x55ffd023c710 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.830+0000 7f7faeffd640 1 -- 172.21.15.50:0/2839353960 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc0ce3d0 con 0x55ffd023c710 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.831+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2839353960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 msgr2=0x55ffd0242fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.831+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/2839353960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 0x55ffd0242fa0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0ce1b0 tx=0x7f7fbc06c280 comp rx=0 tx=0).stop 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.831+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2839353960 shutdown_connections 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.831+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/2839353960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 0x55ffd0242fa0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.831+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/2839353960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd023c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.831+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/2839353960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023ab80 0x55ffd023afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.831+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2839353960 >> 172.21.15.50:0/2839353960 conn(0x55ffd0236070 msgr2=0x55ffd0236470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.831+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2839353960 shutdown_connections 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.831+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2839353960 wait complete. 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.831+0000 7f7fcdfa1740 1 -- start start 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.831+0000 7f7fcdfa1740 1 Processor -- start 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fcdfa1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 0x55ffd0244f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fcdfa1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd0245490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fcdfa1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023c710 0x55ffd024c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fcdfa1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ffd0243660 con 0x55ffd023ab80 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fcdfa1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ffd013a470 con 0x55ffd023c710 2024-04-23T18:17:57.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fcdfa1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ffd02437e0 con 0x55ffd023bd50 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fcdfa1740 10 client.0 _ll_register_callbacks cb 0x55ffd015ff30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fc8f49640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 0x55ffd0244f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fc8f49640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 0x55ffd0244f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39964/0 (socket says 172.21.15.50:39964) 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fc8f49640 1 -- 172.21.15.50:0/3183228402 learned_addr learned my addr 172.21.15.50:0/3183228402 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fc3fff640 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd0245490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fc974a640 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023c710 0x55ffd024c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fc8f49640 1 -- 172.21.15.50:0/3183228402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 msgr2=0x55ffd0245490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fc8f49640 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd0245490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fc8f49640 1 -- 172.21.15.50:0/3183228402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023c710 msgr2=0x55ffd024c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fc8f49640 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023c710 0x55ffd024c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fc8f49640 1 -- 172.21.15.50:0/3183228402 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffcff7c6b0 con 0x55ffd023ab80 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fc974a640 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023c710 0x55ffd024c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fc8f49640 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 0x55ffd0244f50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800be60 tx=0x7f7fb80117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d2143d3918cf88 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fa3fff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fa3fff640 1 -- 172.21.15.50:0/3183228402 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8004440 con 0x55ffd023ab80 2024-04-23T18:17:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fa3fff640 1 -- 172.21.15.50:0/3183228402 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f7fb80035f0 con 0x55ffd023ab80 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.832+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/3183228402 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fbc09d050 con 0x55ffd023ab80 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.833+0000 7f7fa3fff640 1 -- 172.21.15.50:0/3183228402 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7fb80037f0 con 0x55ffd023ab80 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.833+0000 7f7fcdfa1740 10 client.15051 fetch_fsmap learned FSMap version 20 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.833+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/3183228402 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55ffd024f910 con 0x55ffd023ab80 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.833+0000 7f7fa3fff640 1 -- 172.21.15.50:0/3183228402 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fbc09d050 con 0x55ffd023ab80 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.833+0000 7f7fa3fff640 1 -- 172.21.15.50:0/3183228402 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f7fb801ba90 con 0x55ffd023ab80 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.833+0000 7f7fcdfa1740 10 client.15051 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.833+0000 7f7fcdfa1740 10 client.15051 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.833+0000 7f7fcdfa1740 20 client.15051 populate_metadata read hostname 'smithi050' 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.833+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55ffd02526d0 0x55ffd0253470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.833+0000 7f7fcdfa1740 4 client.15051 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.833+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/3183228402 --> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffd01ad670 con 0x55ffd02526d0 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.833+0000 7f7fc3fff640 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55ffd02526d0 0x55ffd0253470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.834+0000 7f7fc3fff640 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55ffd02526d0 0x55ffd0253470 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dafae7d3b12bf064 server_cookie=c07307627e0a9e38 in_seq=0 out_seq=0 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.834+0000 7f7fa3fff640 10 client.15051 ms_handle_connect on v2:172.21.15.188:6838/2202634301 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fa3fff640 1 -- 172.21.15.50:0/3183228402 <== mds.1 v2:172.21.15.188:6838/2202634301 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ffd01ad670 con 0x55ffd02526d0 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fa3fff640 10 client.15051 handle_command_reply: tid=0 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fcdfa1740 1 client.15051 shutdown 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/3183228402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 msgr2=0x55ffd0244f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 0x55ffd0244f50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800be60 tx=0x7f7fb80117b0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/3183228402 shutdown_connections 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55ffd02526d0 0x55ffd0253470 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023c710 0x55ffd024c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd0245490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fa3fff640 0 client.15051 ms_handle_reset on v2:172.21.15.188:6838/2202634301 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/3183228402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 0x55ffd0244f50 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/3183228402 >> 172.21.15.50:0/3183228402 conn(0x55ffd0236070 msgr2=0x55ffd02351d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/3183228402 shutdown_connections 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/3183228402 wait complete. 2024-04-23T18:17:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.835+0000 7f7fcdfa1740 20 client.15051 trim_cache size 0 max 16384 2024-04-23T18:17:57.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.853+0000 7f7fcdfa1740 1 Processor -- start 2024-04-23T18:17:57.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.853+0000 7f7fcdfa1740 1 -- start start 2024-04-23T18:17:57.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.853+0000 7f7fcdfa1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023ab80 0x55ffd02310b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.853+0000 7f7fcdfa1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd024c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.853+0000 7f7fcdfa1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 0x55ffd0234940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.853+0000 7f7fcdfa1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ffd0251a90 con 0x55ffd023c710 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.853+0000 7f7fcdfa1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ffd0251e80 con 0x55ffd023ab80 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.853+0000 7f7fcdfa1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ffd0252550 con 0x55ffd023bd50 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.853+0000 7f7fc974a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 0x55ffd0234940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.853+0000 7f7fc974a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 0x55ffd0234940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39976/0 (socket says 172.21.15.50:39976) 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.853+0000 7f7fc974a640 1 -- 172.21.15.50:0/528982966 learned_addr learned my addr 172.21.15.50:0/528982966 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fc3fff640 1 --2- 172.21.15.50:0/528982966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd024c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fc8f49640 1 --2- 172.21.15.50:0/528982966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023ab80 0x55ffd02310b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fc974a640 1 -- 172.21.15.50:0/528982966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 msgr2=0x55ffd024c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fc974a640 1 --2- 172.21.15.50:0/528982966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd024c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fc974a640 1 -- 172.21.15.50:0/528982966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023ab80 msgr2=0x55ffd02310b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fc974a640 1 --2- 172.21.15.50:0/528982966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023ab80 0x55ffd02310b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fc974a640 1 -- 172.21.15.50:0/528982966 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffd0242ae0 con 0x55ffd023c710 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fc974a640 1 --2- 172.21.15.50:0/528982966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 0x55ffd0234940 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc09c650 tx=0x7f7fbc0ce0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f6ccde7c2deb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fae7fc640 1 -- 172.21.15.50:0/528982966 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc01f040 con 0x55ffd023c710 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fae7fc640 1 -- 172.21.15.50:0/528982966 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fbc0a0d00 con 0x55ffd023c710 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fae7fc640 1 -- 172.21.15.50:0/528982966 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc0054f0 con 0x55ffd023c710 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/528982966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 msgr2=0x55ffd0234940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/528982966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 0x55ffd0234940 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc09c650 tx=0x7f7fbc0ce0c0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/528982966 shutdown_connections 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/528982966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023c710 0x55ffd0234940 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/528982966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd024c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/528982966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023ab80 0x55ffd02310b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.854+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/528982966 >> 172.21.15.50:0/528982966 conn(0x55ffd0236070 msgr2=0x55ffd0236450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/528982966 shutdown_connections 2024-04-23T18:17:57.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/528982966 wait complete. 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fcdfa1740 1 -- start start 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fcdfa1740 1 Processor -- start 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fcdfa1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 0x55ffd02479e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fcdfa1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd0247f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fcdfa1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023c710 0x55ffd024af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fcdfa1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ffd02532f0 con 0x55ffd023ab80 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fcdfa1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ffd0252c20 con 0x55ffd023c710 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fcdfa1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ffd02539c0 con 0x55ffd023bd50 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fcdfa1740 10 client.0 _ll_register_callbacks cb 0x7f7fb800b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fc8f49640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 0x55ffd02479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fc8f49640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 0x55ffd02479e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39978/0 (socket says 172.21.15.50:39978) 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.855+0000 7f7fc8f49640 1 -- 172.21.15.50:0/2514439331 learned_addr learned my addr 172.21.15.50:0/2514439331 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fc8f49640 1 -- 172.21.15.50:0/2514439331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 msgr2=0x55ffd0247f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fc974a640 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023c710 0x55ffd024af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fc8f49640 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd0247f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fc8f49640 1 -- 172.21.15.50:0/2514439331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023c710 msgr2=0x55ffd024af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fc8f49640 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023c710 0x55ffd024af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fc8f49640 1 -- 172.21.15.50:0/2514439331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffd024b530 con 0x55ffd023ab80 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fc974a640 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023c710 0x55ffd024af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fc8f49640 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 0x55ffd02479e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800b930 tx=0x7f7fb8003c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb79c5bae1659cbb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fa3fff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fa3fff640 1 -- 172.21.15.50:0/2514439331 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8034060 con 0x55ffd023ab80 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fa3fff640 1 -- 172.21.15.50:0/2514439331 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f7fb8003e30 con 0x55ffd023ab80 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2514439331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ffd01a74e0 con 0x55ffd023ab80 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fa3fff640 1 -- 172.21.15.50:0/2514439331 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7fb802b040 con 0x55ffd023ab80 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fcdfa1740 10 client.15057 fetch_fsmap learned FSMap version 20 2024-04-23T18:17:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2514439331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55ffd02b7fb0 con 0x55ffd023ab80 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fa3fff640 1 -- 172.21.15.50:0/2514439331 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ffd01a74e0 con 0x55ffd023ab80 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fa3fff640 1 -- 172.21.15.50:0/2514439331 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f7fb8039070 con 0x55ffd023ab80 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fcdfa1740 10 client.15057 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fcdfa1740 10 client.15057 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.856+0000 7f7fcdfa1740 20 client.15057 populate_metadata read hostname 'smithi050' 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.857+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55ffd02bcc10 0x55ffd02bb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.857+0000 7f7fcdfa1740 4 client.15057 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.857+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2514439331 --> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] -- command(tid 0: {"prefix": "status"}) -- 0x55ffd01239a0 con 0x55ffd02bcc10 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.857+0000 7f7fc3fff640 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55ffd02bcc10 0x55ffd02bb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.857+0000 7f7fc3fff640 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55ffd02bcc10 0x55ffd02bb920 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c85934558f943f36 server_cookie=a1f33f2949a4f029 in_seq=0 out_seq=0 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.857+0000 7f7fa3fff640 10 client.15057 ms_handle_connect on v2:172.21.15.188:6838/2202634301 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fa3fff640 1 -- 172.21.15.50:0/2514439331 <== mds.1 v2:172.21.15.188:6838/2202634301 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ffd01239a0 con 0x55ffd02bcc10 2024-04-23T18:17:57.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fa3fff640 10 client.15057 handle_command_reply: tid=0 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcdfa1740 1 client.15057 shutdown 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2514439331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 msgr2=0x55ffd02479e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 0x55ffd02479e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800b930 tx=0x7f7fb8003c40 comp rx=0 tx=0).stop 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2514439331 shutdown_connections 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55ffd02bcc10 0x55ffd02bb920 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ffd023c710 0x55ffd024af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ffd023bd50 0x55ffd0247f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fa3fff640 0 client.15057 ms_handle_reset on v2:172.21.15.188:6838/2202634301 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcdfa1740 1 --2- 172.21.15.50:0/2514439331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ffd023ab80 0x55ffd02479e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2514439331 >> 172.21.15.50:0/2514439331 conn(0x55ffd0236070 msgr2=0x55ffd02310b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2514439331 shutdown_connections 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcdfa1740 1 -- 172.21.15.50:0/2514439331 wait complete. 2024-04-23T18:17:57.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcdfa1740 20 client.15057 trim_cache size 0 max 16384 2024-04-23T18:17:57.858 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:17:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:17:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 1, 2024-04-23T18:17:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 24499, 2024-04-23T18:17:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2024-04-23T18:17:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2024-04-23T18:17:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2024-04-23T18:17:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 81.908805979999997, 2024-04-23T18:17:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 20, 2024-04-23T18:17:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 89, 2024-04-23T18:17:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 87, 2024-04-23T18:17:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 97.397154842999996 2024-04-23T18:17:57.859 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:17:57.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/3633324784 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7fb40a3cc0 msgr2=0x7f7fb40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcb1ad640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7fb40a3cc0 0x7f7fb40a6180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0031500 tx=0x7f7fb0032d50 comp rx=0 tx=0).stop 2024-04-23T18:17:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/3633324784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc4102a00 msgr2=0x7f7fc419f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcb1ad640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc4102a00 0x7f7fc419f550 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00eaa0 tx=0x7f7fbc00ef60 comp rx=0 tx=0).stop 2024-04-23T18:17:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.858+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/3633324784 shutdown_connections 2024-04-23T18:17:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.859+0000 7f7fcb1ad640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f7fb40a3cc0 0x7f7fb40a6180 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.859+0000 7f7fcb1ad640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc4102a00 0x7f7fc419f550 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.859+0000 7f7fcb1ad640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fc4102040 0x7f7fc419cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.859+0000 7f7fcb1ad640 1 --2- 172.21.15.50:0/3633324784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fc4100e60 0x7f7fc406b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.859+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/3633324784 >> 172.21.15.50:0/3633324784 conn(0x7f7fc40fc440 msgr2=0x7f7fc4105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.859+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/3633324784 shutdown_connections 2024-04-23T18:17:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.859+0000 7f7fcb1ad640 1 -- 172.21.15.50:0/3633324784 wait complete. 2024-04-23T18:17:57.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T18:17:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.971+0000 7f8465031640 1 -- 172.21.15.50:0/826031370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8460102ab0 msgr2=0x7f8460104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.971+0000 7f8465031640 1 --2- 172.21.15.50:0/826031370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8460102ab0 0x7f8460104ea0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8458009900 tx=0x7f845802a3f0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.971+0000 7f8465031640 1 -- 172.21.15.50:0/826031370 shutdown_connections 2024-04-23T18:17:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.971+0000 7f8465031640 1 --2- 172.21.15.50:0/826031370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8460107d10 0x7f846010a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.971+0000 7f8465031640 1 --2- 172.21.15.50:0/826031370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84601053e0 0x7f84601077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.971+0000 7f8465031640 1 --2- 172.21.15.50:0/826031370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8460102ab0 0x7f8460104ea0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.971+0000 7f8465031640 1 -- 172.21.15.50:0/826031370 >> 172.21.15.50:0/826031370 conn(0x7f84600fc400 msgr2=0x7f84600fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.971+0000 7f8465031640 1 -- 172.21.15.50:0/826031370 shutdown_connections 2024-04-23T18:17:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.971+0000 7f8465031640 1 -- 172.21.15.50:0/826031370 wait complete. 2024-04-23T18:17:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.972+0000 7f8465031640 1 Processor -- start 2024-04-23T18:17:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.972+0000 7f8465031640 1 -- start start 2024-04-23T18:17:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.972+0000 7f8465031640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8460102ab0 0x7f84601980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.972+0000 7f8465031640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84601053e0 0x7f8460198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.972+0000 7f8465031640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8460107d10 0x7f846019f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.972+0000 7f8465031640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f846010daf0 con 0x7f84601053e0 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.972+0000 7f8465031640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f846010d970 con 0x7f8460102ab0 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.972+0000 7f8465031640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f846010dc70 con 0x7f8460107d10 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.972+0000 7f845e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84601053e0 0x7f8460198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.972+0000 7f845e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84601053e0 0x7f8460198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39994/0 (socket says 172.21.15.50:39994) 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.972+0000 7f845e59c640 1 -- 172.21.15.50:0/1394115283 learned_addr learned my addr 172.21.15.50:0/1394115283 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.972+0000 7f845ed9d640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8460102ab0 0x7f84601980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.973+0000 7f845f59e640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8460107d10 0x7f846019f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.973+0000 7f845e59c640 1 -- 172.21.15.50:0/1394115283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8460107d10 msgr2=0x7f846019f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.973+0000 7f845e59c640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8460107d10 0x7f846019f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.973+0000 7f845e59c640 1 -- 172.21.15.50:0/1394115283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8460102ab0 msgr2=0x7f84601980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.973+0000 7f845e59c640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8460102ab0 0x7f84601980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.973+0000 7f845e59c640 1 -- 172.21.15.50:0/1394115283 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f846019fce0 con 0x7f84601053e0 2024-04-23T18:17:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.973+0000 7f845e59c640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84601053e0 0x7f8460198600 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f844800c970 tx=0x7f844800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.973+0000 7f8457fff640 1 -- 172.21.15.50:0/1394115283 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8448014070 con 0x7f84601053e0 2024-04-23T18:17:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.973+0000 7f8465031640 1 -- 172.21.15.50:0/1394115283 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f846019ffd0 con 0x7f84601053e0 2024-04-23T18:17:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.973+0000 7f8457fff640 1 -- 172.21.15.50:0/1394115283 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8448004440 con 0x7f84601053e0 2024-04-23T18:17:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.973+0000 7f8457fff640 1 -- 172.21.15.50:0/1394115283 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8448003e50 con 0x7f84601053e0 2024-04-23T18:17:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.973+0000 7f8465031640 1 -- 172.21.15.50:0/1394115283 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84601a0490 con 0x7f84601053e0 2024-04-23T18:17:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.975+0000 7f8457fff640 1 -- 172.21.15.50:0/1394115283 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8448017070 con 0x7f84601053e0 2024-04-23T18:17:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f8467e22740 1 Processor -- start 2024-04-23T18:17:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f8467e22740 1 -- start start 2024-04-23T18:17:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f8467e22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87f0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f8467e22740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 0x55a5c87f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f8467e22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f2710 0x55a5c87f8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f8467e22740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a5c8755c50 con 0x55a5c87f1d50 2024-04-23T18:17:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f8467e22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a5c86f2830 con 0x55a5c87f2710 2024-04-23T18:17:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f8467e22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a5c8751c10 con 0x55a5c87f0b80 2024-04-23T18:17:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f8457fff640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f844c0a3de0 0x7f844c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f845ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 0x55a5c87f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f845ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 0x55a5c87f21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40010/0 (socket says 172.21.15.50:40010) 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f845ed9d640 1 -- 172.21.15.50:0/3336666170 learned_addr learned my addr 172.21.15.50:0/3336666170 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.976+0000 7f845f59e640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f844c0a3de0 0x7f844c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f845f59e640 1 --2- 172.21.15.50:0/3336666170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87f0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f8457fff640 1 -- 172.21.15.50:0/1394115283 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f84480d0320 con 0x7f84601053e0 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f845e59c640 1 --2- 172.21.15.50:0/3336666170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f2710 0x55a5c87f8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f845ed9d640 1 -- 172.21.15.50:0/3336666170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 msgr2=0x55a5c87f0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f845ed9d640 1 --2- 172.21.15.50:0/3336666170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87f0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f845ed9d640 1 -- 172.21.15.50:0/3336666170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f2710 msgr2=0x55a5c87f8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f845ed9d640 1 --2- 172.21.15.50:0/3336666170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f2710 0x55a5c87f8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f845ed9d640 1 -- 172.21.15.50:0/3336666170 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5c84682d0 con 0x55a5c87f1d50 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f845ed9d640 1 --2- 172.21.15.50:0/3336666170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 0x55a5c87f21d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8458009900 tx=0x7f8458031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b300d0e72928f3b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f845f59e640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f844c0a3de0 0x7f844c0a62a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55a5c87f1c50 tx=0x7f84500072a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f84557fa640 1 -- 172.21.15.50:0/3336666170 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8458035030 con 0x55a5c87f1d50 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f84557fa640 1 -- 172.21.15.50:0/3336666170 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8458040070 con 0x55a5c87f1d50 2024-04-23T18:17:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.977+0000 7f84557fa640 1 -- 172.21.15.50:0/3336666170 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8458003c60 con 0x55a5c87f1d50 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 -- 172.21.15.50:0/3336666170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 msgr2=0x55a5c87f21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 --2- 172.21.15.50:0/3336666170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 0x55a5c87f21d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8458009900 tx=0x7f8458031e60 comp rx=0 tx=0).stop 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 -- 172.21.15.50:0/3336666170 shutdown_connections 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 --2- 172.21.15.50:0/3336666170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f2710 0x55a5c87f8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 --2- 172.21.15.50:0/3336666170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 0x55a5c87f21d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 --2- 172.21.15.50:0/3336666170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87f0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 -- 172.21.15.50:0/3336666170 >> 172.21.15.50:0/3336666170 conn(0x55a5c87ec070 msgr2=0x55a5c87ec470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 -- 172.21.15.50:0/3336666170 shutdown_connections 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 -- 172.21.15.50:0/3336666170 wait complete. 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 -- start start 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 Processor -- start 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87fafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 0x55a5c87fb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f2710 0x55a5c8802550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a5c87f96c0 con 0x55a5c87f1d50 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a5c86f0470 con 0x55a5c87f2710 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.978+0000 7f8467e22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a5c87f9840 con 0x55a5c87f0b80 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f8467e22740 10 client.0 _ll_register_callbacks cb 0x55a5c8715f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f845e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 0x55a5c87fb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f845e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 0x55a5c87fb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40022/0 (socket says 172.21.15.50:40022) 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f845e59c640 1 -- 172.21.15.50:0/2781862099 learned_addr learned my addr 172.21.15.50:0/2781862099 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f845ed9d640 1 --2- 172.21.15.50:0/2781862099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87fafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f845e59c640 1 -- 172.21.15.50:0/2781862099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 msgr2=0x55a5c87fafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f845e59c640 1 --2- 172.21.15.50:0/2781862099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87fafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f845e59c640 1 -- 172.21.15.50:0/2781862099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f2710 msgr2=0x55a5c8802550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f845e59c640 1 --2- 172.21.15.50:0/2781862099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f2710 0x55a5c8802550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f845e59c640 1 -- 172.21.15.50:0/2781862099 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5c85326b0 con 0x55a5c87f1d50 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f845e59c640 1 --2- 172.21.15.50:0/2781862099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 0x55a5c87fb510 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8448098e70 tx=0x7f844809c0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c35c3b48ab029556 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f84467fc640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f84467fc640 1 -- 172.21.15.50:0/2781862099 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84480a0850 con 0x55a5c87f1d50 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f84467fc640 1 -- 172.21.15.50:0/2781862099 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f84480035f0 con 0x55a5c87f1d50 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f8467e22740 1 -- 172.21.15.50:0/2781862099 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8458040070 con 0x55a5c87f1d50 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f84467fc640 1 -- 172.21.15.50:0/2781862099 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f84480a5110 con 0x55a5c87f1d50 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f8467e22740 10 client.15069 fetch_fsmap learned FSMap version 20 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f8467e22740 1 -- 172.21.15.50:0/2781862099 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55a5c88058a0 con 0x55a5c87f1d50 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.979+0000 7f84467fc640 1 -- 172.21.15.50:0/2781862099 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8458040070 con 0x55a5c87f1d50 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.980+0000 7f84467fc640 1 -- 172.21.15.50:0/2781862099 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f84480983f0 con 0x55a5c87f1d50 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.980+0000 7f8467e22740 10 client.15069 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.980+0000 7f8467e22740 10 client.15069 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.980+0000 7f8467e22740 20 client.15069 populate_metadata read hostname 'smithi050' 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.980+0000 7f8467e22740 1 --2- 172.21.15.50:0/2781862099 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a5c8807ea0 0x55a5c8809310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.980+0000 7f8467e22740 4 client.15069 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.980+0000 7f8467e22740 1 -- 172.21.15.50:0/2781862099 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5c8763670 con 0x55a5c8807ea0 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.980+0000 7f845ed9d640 1 --2- 172.21.15.50:0/2781862099 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a5c8807ea0 0x55a5c8809310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.980+0000 7f845ed9d640 1 --2- 172.21.15.50:0/2781862099 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a5c8807ea0 0x55a5c8809310 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0f6f14564dbc5d6 server_cookie=b89d2346467c1070 in_seq=0 out_seq=0 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.980+0000 7f84467fc640 10 client.15069 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f84467fc640 1 -- 172.21.15.50:0/2781862099 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a5c8763670 con 0x55a5c8807ea0 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f84467fc640 10 client.15069 handle_command_reply: tid=0 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f8467e22740 1 client.15069 shutdown 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f8467e22740 1 -- 172.21.15.50:0/2781862099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 msgr2=0x55a5c87fb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f8467e22740 1 --2- 172.21.15.50:0/2781862099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 0x55a5c87fb510 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8448098e70 tx=0x7f844809c0d0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f8467e22740 1 -- 172.21.15.50:0/2781862099 shutdown_connections 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f8467e22740 1 --2- 172.21.15.50:0/2781862099 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a5c8807ea0 0x55a5c8809310 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f8467e22740 1 --2- 172.21.15.50:0/2781862099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f2710 0x55a5c8802550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f8467e22740 1 --2- 172.21.15.50:0/2781862099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f1d50 0x55a5c87fb510 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f84467fc640 0 client.15069 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:17:57.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f8467e22740 1 --2- 172.21.15.50:0/2781862099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87fafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f8467e22740 1 -- 172.21.15.50:0/2781862099 >> 172.21.15.50:0/2781862099 conn(0x55a5c87ec070 msgr2=0x55a5c87eb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:57.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.981+0000 7f8467e22740 1 -- 172.21.15.50:0/2781862099 shutdown_connections 2024-04-23T18:17:57.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.982+0000 7f8467e22740 1 -- 172.21.15.50:0/2781862099 wait complete. 2024-04-23T18:17:57.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:57.982+0000 7f8467e22740 20 client.15069 trim_cache size 0 max 16384 2024-04-23T18:17:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f8467e22740 1 Processor -- start 2024-04-23T18:17:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f8467e22740 1 -- start start 2024-04-23T18:17:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f8467e22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f0b80 0x55a5c87f9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f8467e22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f1d50 0x55a5c8802510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f8467e22740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 0x55a5c87fafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f8467e22740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a5c8807650 con 0x55a5c87f2710 2024-04-23T18:17:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f8467e22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a5c8807d20 con 0x55a5c87f0b80 2024-04-23T18:17:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f8467e22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a5c88083f0 con 0x55a5c87f1d50 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f845f59e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 0x55a5c87fafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f845f59e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 0x55a5c87fafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40034/0 (socket says 172.21.15.50:40034) 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f845f59e640 1 -- 172.21.15.50:0/2899597830 learned_addr learned my addr 172.21.15.50:0/2899597830 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f845e59c640 1 --2- 172.21.15.50:0/2899597830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f1d50 0x55a5c8802510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.000+0000 7f845ed9d640 1 --2- 172.21.15.50:0/2899597830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f0b80 0x55a5c87f9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.001+0000 7f845f59e640 1 -- 172.21.15.50:0/2899597830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f1d50 msgr2=0x55a5c8802510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.001+0000 7f845f59e640 1 --2- 172.21.15.50:0/2899597830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f1d50 0x55a5c8802510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.001+0000 7f845f59e640 1 -- 172.21.15.50:0/2899597830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f0b80 msgr2=0x55a5c87f9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.001+0000 7f845f59e640 1 --2- 172.21.15.50:0/2899597830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f0b80 0x55a5c87f9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.001+0000 7f845f59e640 1 -- 172.21.15.50:0/2899597830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5c87f17d0 con 0x55a5c87f2710 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.001+0000 7f845e59c640 1 --2- 172.21.15.50:0/2899597830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f1d50 0x55a5c8802510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.001+0000 7f845f59e640 1 --2- 172.21.15.50:0/2899597830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 0x55a5c87fafd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8450007960 tx=0x7f8450004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502bba198ce030b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.001+0000 7f8454ff9640 1 -- 172.21.15.50:0/2899597830 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8450024030 con 0x55a5c87f2710 2024-04-23T18:17:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.001+0000 7f8454ff9640 1 -- 172.21.15.50:0/2899597830 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8450005020 con 0x55a5c87f2710 2024-04-23T18:17:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.001+0000 7f8454ff9640 1 -- 172.21.15.50:0/2899597830 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84500206e0 con 0x55a5c87f2710 2024-04-23T18:17:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 -- 172.21.15.50:0/2899597830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 msgr2=0x55a5c87fafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 --2- 172.21.15.50:0/2899597830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 0x55a5c87fafd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8450007960 tx=0x7f8450004ab0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 -- 172.21.15.50:0/2899597830 shutdown_connections 2024-04-23T18:17:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 --2- 172.21.15.50:0/2899597830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 0x55a5c87fafd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 --2- 172.21.15.50:0/2899597830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f1d50 0x55a5c8802510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 --2- 172.21.15.50:0/2899597830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f0b80 0x55a5c87f9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 -- 172.21.15.50:0/2899597830 >> 172.21.15.50:0/2899597830 conn(0x55a5c87ec070 msgr2=0x55a5c87e70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 -- 172.21.15.50:0/2899597830 shutdown_connections 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 -- 172.21.15.50:0/2899597830 wait complete. 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 -- start start 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 Processor -- start 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87fdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f1d50 0x55a5c87fe1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 0x55a5c88011c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a5c8809190 con 0x55a5c87f2710 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a5c8808ac0 con 0x55a5c87f1d50 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.002+0000 7f8467e22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a5c8809860 con 0x55a5c87f0b80 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f8467e22740 10 client.0 _ll_register_callbacks cb 0x7f84480987f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f845f59e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 0x55a5c88011c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f845f59e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 0x55a5c88011c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40042/0 (socket says 172.21.15.50:40042) 2024-04-23T18:17:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f845f59e640 1 -- 172.21.15.50:0/2184239898 learned_addr learned my addr 172.21.15.50:0/2184239898 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f845e59c640 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f1d50 0x55a5c87fe1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f845ed9d640 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87fdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f845f59e640 1 -- 172.21.15.50:0/2184239898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 msgr2=0x55a5c87fdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f845f59e640 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87fdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f845f59e640 1 -- 172.21.15.50:0/2184239898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f1d50 msgr2=0x55a5c87fe1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:17:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f845f59e640 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f1d50 0x55a5c87fe1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f845f59e640 1 -- 172.21.15.50:0/2184239898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5c88017b0 con 0x55a5c87f2710 2024-04-23T18:17:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f845ed9d640 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87fdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:17:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f845f59e640 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 0x55a5c88011c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8450005c40 tx=0x7f8450005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f3f44ab08a157f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f84467fc640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:17:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f84467fc640 1 -- 172.21.15.50:0/2184239898 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8450024040 con 0x55a5c87f2710 2024-04-23T18:17:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f84467fc640 1 -- 172.21.15.50:0/2184239898 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f8450020680 con 0x55a5c87f2710 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f8467e22740 1 -- 172.21.15.50:0/2184239898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8450005020 con 0x55a5c87f2710 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f84467fc640 1 -- 172.21.15.50:0/2184239898 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f8450020880 con 0x55a5c87f2710 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f8467e22740 10 client.15075 fetch_fsmap learned FSMap version 20 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.003+0000 7f8467e22740 1 -- 172.21.15.50:0/2184239898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55a5c886deb0 con 0x55a5c87f2710 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.004+0000 7f84467fc640 1 -- 172.21.15.50:0/2184239898 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8450005020 con 0x55a5c87f2710 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.004+0000 7f84467fc640 1 -- 172.21.15.50:0/2184239898 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f845001d050 con 0x55a5c87f2710 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.004+0000 7f8467e22740 10 client.15075 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.004+0000 7f8467e22740 10 client.15075 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.004+0000 7f8467e22740 20 client.15075 populate_metadata read hostname 'smithi050' 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.004+0000 7f8467e22740 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a5c8872a80 0x55a5c8871790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.004+0000 7f8467e22740 4 client.15075 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.004+0000 7f8467e22740 1 -- 172.21.15.50:0/2184239898 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "status"}) -- 0x55a5c875d4e0 con 0x55a5c8872a80 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.004+0000 7f845ed9d640 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a5c8872a80 0x55a5c8871790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.004+0000 7f845ed9d640 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a5c8872a80 0x55a5c8871790 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1adbfbe4d57d5c66 server_cookie=c6a93c8d86eb45ab in_seq=0 out_seq=0 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.004+0000 7f84467fc640 10 client.15075 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f84467fc640 1 -- 172.21.15.50:0/2184239898 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a5c875d4e0 con 0x55a5c8872a80 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f84467fc640 10 client.15075 handle_command_reply: tid=0 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8467e22740 1 client.15075 shutdown 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8467e22740 1 -- 172.21.15.50:0/2184239898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 msgr2=0x55a5c88011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8467e22740 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 0x55a5c88011c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8450005c40 tx=0x7f8450005480 comp rx=0 tx=0).stop 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8467e22740 1 -- 172.21.15.50:0/2184239898 shutdown_connections 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8467e22740 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a5c8872a80 0x55a5c8871790 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8467e22740 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5c87f2710 0x55a5c88011c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f84467fc640 0 client.15075 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8467e22740 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5c87f1d50 0x55a5c87fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8467e22740 1 --2- 172.21.15.50:0/2184239898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5c87f0b80 0x55a5c87fdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8467e22740 1 -- 172.21.15.50:0/2184239898 >> 172.21.15.50:0/2184239898 conn(0x55a5c87ec070 msgr2=0x55a5c87f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8467e22740 1 -- 172.21.15.50:0/2184239898 shutdown_connections 2024-04-23T18:17:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8467e22740 1 -- 172.21.15.50:0/2184239898 wait complete. 2024-04-23T18:17:58.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8467e22740 20 client.15075 trim_cache size 0 max 16384 2024-04-23T18:17:58.005 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:17:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:17:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2024-04-23T18:17:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 24506, 2024-04-23T18:17:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2024-04-23T18:17:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2024-04-23T18:17:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2024-04-23T18:17:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 84.036904202000002, 2024-04-23T18:17:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 20, 2024-04-23T18:17:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 89, 2024-04-23T18:17:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 87, 2024-04-23T18:17:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 89.206023154999997 2024-04-23T18:17:58.006 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:17:58.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8465031640 1 -- 172.21.15.50:0/1394115283 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f844c0a3de0 msgr2=0x7f844c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8465031640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f844c0a3de0 0x7f844c0a62a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55a5c87f1c50 tx=0x7f84500072a0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8465031640 1 -- 172.21.15.50:0/1394115283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84601053e0 msgr2=0x7f8460198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8465031640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84601053e0 0x7f8460198600 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f844800c970 tx=0x7f844800ce30 comp rx=0 tx=0).stop 2024-04-23T18:17:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8465031640 1 -- 172.21.15.50:0/1394115283 shutdown_connections 2024-04-23T18:17:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8465031640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f844c0a3de0 0x7f844c0a62a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8465031640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8460107d10 0x7f846019f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8465031640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84601053e0 0x7f8460198600 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8465031640 1 --2- 172.21.15.50:0/1394115283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8460102ab0 0x7f84601980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.005+0000 7f8465031640 1 -- 172.21.15.50:0/1394115283 >> 172.21.15.50:0/1394115283 conn(0x7f84600fc400 msgr2=0x7f84601033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.006+0000 7f8465031640 1 -- 172.21.15.50:0/1394115283 shutdown_connections 2024-04-23T18:17:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:58.006+0000 7f8465031640 1 -- 172.21.15.50:0/1394115283 wait complete. 2024-04-23T18:17:58.009 INFO:tasks.cephadm:Setup complete, yielding 2024-04-23T18:17:58.010 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T18:17:58.022 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi050.front.sepia.ceph.com 2024-04-23T18:17:58.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-23T18:17:58.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:58 smithi050 ceph-mon[29187]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:17:58.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:58 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3506216456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:58.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:17:58 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1552693384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:58.276 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:17:58.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:58 smithi177 ceph-mon[32308]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:17:58.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:58 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3506216456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:58.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:17:58 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1552693384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:58.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:58 smithi188 ceph-mon[32694]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:17:58.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:58 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3506216456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:17:58.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:17:58 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1552693384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:17:59.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.943+0000 7fb7f7df5700 1 -- 172.21.15.50:0/3221110922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7f01087b0 msgr2=0x7fb7f010ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:59.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.943+0000 7fb7f7df5700 1 --2- 172.21.15.50:0/3221110922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7f01087b0 0x7fb7f010ac20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800b0a0 tx=0x7fb7e800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:17:59.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.943+0000 7fb7f7df5700 1 -- 172.21.15.50:0/3221110922 shutdown_connections 2024-04-23T18:17:59.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.943+0000 7fb7f7df5700 1 --2- 172.21.15.50:0/3221110922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7f01087b0 0x7fb7f010ac20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:59.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.943+0000 7fb7f7df5700 1 --2- 172.21.15.50:0/3221110922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7f0103550 0x7fb7f0105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:59.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.943+0000 7fb7f7df5700 1 --2- 172.21.15.50:0/3221110922 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7f0105e80 0x7fb7f0108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:59.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.943+0000 7fb7f7df5700 1 -- 172.21.15.50:0/3221110922 >> 172.21.15.50:0/3221110922 conn(0x7fb7f00fcf60 msgr2=0x7fb7f00ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:17:59.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.943+0000 7fb7f7df5700 1 -- 172.21.15.50:0/3221110922 shutdown_connections 2024-04-23T18:17:59.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.943+0000 7fb7f7df5700 1 -- 172.21.15.50:0/3221110922 wait complete. 2024-04-23T18:17:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f7df5700 1 Processor -- start 2024-04-23T18:17:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f7df5700 1 -- start start 2024-04-23T18:17:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f7df5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7f0103550 0x7fb7f0198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f7df5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7f0105e80 0x7fb7f0199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f7df5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7f01087b0 0x7fb7f01a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f7df5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f010e520 con 0x7fb7f01087b0 2024-04-23T18:17:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f7df5700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f010e3a0 con 0x7fb7f0105e80 2024-04-23T18:17:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f7df5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f010e6a0 con 0x7fb7f0103550 2024-04-23T18:17:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f5390700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7f0105e80 0x7fb7f0199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f6392700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7f01087b0 0x7fb7f01a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f5390700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7f0105e80 0x7fb7f0199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:48852/0 (socket says 172.21.15.50:48852) 2024-04-23T18:17:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f5390700 1 -- 172.21.15.50:0/248038377 learned_addr learned my addr 172.21.15.50:0/248038377 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f6392700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7f01087b0 0x7fb7f01a0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40054/0 (socket says 172.21.15.50:40054) 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.944+0000 7fb7f5b91700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7f0103550 0x7fb7f0198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.945+0000 7fb7f6392700 1 -- 172.21.15.50:0/248038377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7f0103550 msgr2=0x7fb7f0198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.945+0000 7fb7f6392700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7f0103550 0x7fb7f0198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.945+0000 7fb7f6392700 1 -- 172.21.15.50:0/248038377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7f0105e80 msgr2=0x7fb7f0199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.945+0000 7fb7f6392700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7f0105e80 0x7fb7f0199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.945+0000 7fb7f6392700 1 -- 172.21.15.50:0/248038377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f01a07b0 con 0x7fb7f01087b0 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.945+0000 7fb7f6392700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7f01087b0 0x7fb7f01a0110 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800b0a0 tx=0x7fb7e8004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.945+0000 7fb7e6ffd700 1 -- 172.21.15.50:0/248038377 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8021070 con 0x7fb7f01087b0 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.945+0000 7fb7f7df5700 1 -- 172.21.15.50:0/248038377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f01a0a40 con 0x7fb7f01087b0 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.945+0000 7fb7e6ffd700 1 -- 172.21.15.50:0/248038377 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7e8007b80 con 0x7fb7f01087b0 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.945+0000 7fb7e6ffd700 1 -- 172.21.15.50:0/248038377 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e80052a0 con 0x7fb7f01087b0 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.945+0000 7fb7f7df5700 1 -- 172.21.15.50:0/248038377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f01a0fd0 con 0x7fb7f01087b0 2024-04-23T18:17:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.946+0000 7fb7f7df5700 1 -- 172.21.15.50:0/248038377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7f0069730 con 0x7fb7f01087b0 2024-04-23T18:17:59.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.948+0000 7fb7e6ffd700 1 -- 172.21.15.50:0/248038377 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb7e8024070 con 0x7fb7f01087b0 2024-04-23T18:17:59.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.948+0000 7fb7e6ffd700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb7dc0a3cd0 0x7fb7dc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:17:59.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.949+0000 7fb7f5b91700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb7dc0a3cd0 0x7fb7dc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:17:59.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.949+0000 7fb7e6ffd700 1 -- 172.21.15.50:0/248038377 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fb7e800f020 con 0x7fb7f01087b0 2024-04-23T18:17:59.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.949+0000 7fb7f5b91700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb7dc0a3cd0 0x7fb7dc0a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec005950 tx=0x7fb7ec00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:17:59.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:17:59.952+0000 7fb7e6ffd700 1 -- 172.21.15.50:0/248038377 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7e802b020 con 0x7fb7f01087b0 2024-04-23T18:18:00.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.179+0000 7fb7f7df5700 1 -- 172.21.15.50:0/248038377 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fb7f0062e80 con 0x7fb7dc0a3cd0 2024-04-23T18:18:00.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.180+0000 7fb7e6ffd700 1 -- 172.21.15.50:0/248038377 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fb7f0062e80 con 0x7fb7dc0a3cd0 2024-04-23T18:18:00.174 INFO:teuthology.orchestra.run.smithi050.stdout:Backend: cephadm 2024-04-23T18:18:00.174 INFO:teuthology.orchestra.run.smithi050.stdout:Available: Yes 2024-04-23T18:18:00.174 INFO:teuthology.orchestra.run.smithi050.stdout:Paused: No 2024-04-23T18:18:00.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.182+0000 7fb7f7df5700 1 -- 172.21.15.50:0/248038377 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb7dc0a3cd0 msgr2=0x7fb7dc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:00.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.182+0000 7fb7f7df5700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb7dc0a3cd0 0x7fb7dc0a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec005950 tx=0x7fb7ec00b410 comp rx=0 tx=0).stop 2024-04-23T18:18:00.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.182+0000 7fb7f7df5700 1 -- 172.21.15.50:0/248038377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7f01087b0 msgr2=0x7fb7f01a0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:00.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.182+0000 7fb7f7df5700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7f01087b0 0x7fb7f01a0110 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800b0a0 tx=0x7fb7e8004dc0 comp rx=0 tx=0).stop 2024-04-23T18:18:00.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.182+0000 7fb7f7df5700 1 -- 172.21.15.50:0/248038377 shutdown_connections 2024-04-23T18:18:00.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.182+0000 7fb7f7df5700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7f01087b0 0x7fb7f01a0110 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:00.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.182+0000 7fb7f7df5700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7f0103550 0x7fb7f0198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:00.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.182+0000 7fb7f7df5700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb7dc0a3cd0 0x7fb7dc0a6190 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:00.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.182+0000 7fb7f7df5700 1 --2- 172.21.15.50:0/248038377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7f0105e80 0x7fb7f0199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:00.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.182+0000 7fb7f7df5700 1 -- 172.21.15.50:0/248038377 >> 172.21.15.50:0/248038377 conn(0x7fb7f00fcf60 msgr2=0x7fb7f0103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:00.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.182+0000 7fb7f7df5700 1 -- 172.21.15.50:0/248038377 shutdown_connections 2024-04-23T18:18:00.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:00.182+0000 7fb7f7df5700 1 -- 172.21.15.50:0/248038377 wait complete. 2024-04-23T18:18:00.187 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:00 smithi050 ceph-mon[29187]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:18:00.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:00 smithi177 ceph-mon[32308]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:18:00.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:00 smithi188 ceph-mon[32694]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T18:18:00.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-23T18:18:01.134 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:18:01.393 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:01 smithi050 ceph-mon[29187]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:01.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:01 smithi177 ceph-mon[32308]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:01.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:01 smithi188 ceph-mon[32694]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:02.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:02 smithi050 ceph-mon[29187]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:18:02.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:02 smithi177 ceph-mon[32308]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:18:02.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:02 smithi188 ceph-mon[32694]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:18:02.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.762+0000 7fcf55a57700 1 -- 172.21.15.50:0/4104100774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf50076a70 msgr2=0x7fcf50074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.762+0000 7fcf55a57700 1 --2- 172.21.15.50:0/4104100774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf50076a70 0x7fcf50074ed0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fcf40009990 tx=0x7fcf40009ca0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.762+0000 7fcf55a57700 1 -- 172.21.15.50:0/4104100774 shutdown_connections 2024-04-23T18:18:02.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.762+0000 7fcf55a57700 1 --2- 172.21.15.50:0/4104100774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf5010f0a0 0x7fcf50111490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.762+0000 7fcf55a57700 1 --2- 172.21.15.50:0/4104100774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf50076a70 0x7fcf50074ed0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.762+0000 7fcf55a57700 1 --2- 172.21.15.50:0/4104100774 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf500754a0 0x7fcf50075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.762+0000 7fcf55a57700 1 -- 172.21.15.50:0/4104100774 >> 172.21.15.50:0/4104100774 conn(0x7fcf500fe060 msgr2=0x7fcf501004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:02.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.762+0000 7fcf55a57700 1 -- 172.21.15.50:0/4104100774 shutdown_connections 2024-04-23T18:18:02.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.762+0000 7fcf55a57700 1 -- 172.21.15.50:0/4104100774 wait complete. 2024-04-23T18:18:02.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf55a57700 1 Processor -- start 2024-04-23T18:18:02.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf55a57700 1 -- start start 2024-04-23T18:18:02.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf55a57700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf500754a0 0x7fcf5019f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf55a57700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf50076a70 0x7fcf5019f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf55a57700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf5010f0a0 0x7fcf501a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf55a57700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcf50114b50 con 0x7fcf50076a70 2024-04-23T18:18:02.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf55a57700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcf501149d0 con 0x7fcf500754a0 2024-04-23T18:18:02.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf55a57700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcf50114cd0 con 0x7fcf5010f0a0 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf4e7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf50076a70 0x7fcf5019f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf4e7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf50076a70 0x7fcf5019f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38902/0 (socket says 172.21.15.50:38902) 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf4e7fc700 1 -- 172.21.15.50:0/2845577785 learned_addr learned my addr 172.21.15.50:0/2845577785 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf4effd700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf500754a0 0x7fcf5019f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf4f7fe700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf5010f0a0 0x7fcf501a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.763+0000 7fcf4e7fc700 1 -- 172.21.15.50:0/2845577785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf5010f0a0 msgr2=0x7fcf501a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.764+0000 7fcf4e7fc700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf5010f0a0 0x7fcf501a6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.764+0000 7fcf4e7fc700 1 -- 172.21.15.50:0/2845577785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf500754a0 msgr2=0x7fcf5019f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.764+0000 7fcf4e7fc700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf500754a0 0x7fcf5019f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.764+0000 7fcf4e7fc700 1 -- 172.21.15.50:0/2845577785 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf501a6e20 con 0x7fcf50076a70 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.764+0000 7fcf4e7fc700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf50076a70 0x7fcf5019f700 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400d9b0 tx=0x7fcf4400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.764+0000 7fcf54a55700 1 -- 172.21.15.50:0/2845577785 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44017070 con 0x7fcf50076a70 2024-04-23T18:18:02.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.764+0000 7fcf54a55700 1 -- 172.21.15.50:0/2845577785 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf44009de0 con 0x7fcf50076a70 2024-04-23T18:18:02.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.764+0000 7fcf54a55700 1 -- 172.21.15.50:0/2845577785 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4400f3c0 con 0x7fcf50076a70 2024-04-23T18:18:02.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.764+0000 7fcf55a57700 1 -- 172.21.15.50:0/2845577785 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf501a7110 con 0x7fcf50076a70 2024-04-23T18:18:02.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.764+0000 7fcf55a57700 1 -- 172.21.15.50:0/2845577785 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf501a7650 con 0x7fcf50076a70 2024-04-23T18:18:02.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.765+0000 7fcf55a57700 1 -- 172.21.15.50:0/2845577785 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf50069730 con 0x7fcf50076a70 2024-04-23T18:18:02.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.766+0000 7fcf54a55700 1 -- 172.21.15.50:0/2845577785 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcf44009930 con 0x7fcf50076a70 2024-04-23T18:18:02.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.767+0000 7fcf54a55700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcf380a3d50 0x7fcf380a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:02.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.767+0000 7fcf4effd700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcf380a3d50 0x7fcf380a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:02.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.767+0000 7fcf54a55700 1 -- 172.21.15.50:0/2845577785 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fcf440cffc0 con 0x7fcf50076a70 2024-04-23T18:18:02.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.767+0000 7fcf4effd700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcf380a3d50 0x7fcf380a6210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcf400038a0 tx=0x7fcf40005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:02.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.771+0000 7fcf54a55700 1 -- 172.21.15.50:0/2845577785 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcf4409c900 con 0x7fcf50076a70 2024-04-23T18:18:02.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:02.987+0000 7fcf55a57700 1 -- 172.21.15.50:0/2845577785 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fcf50062e80 con 0x7fcf380a3d50 2024-04-23T18:18:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.001+0000 7fcf54a55700 1 -- 172.21.15.50:0/2845577785 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7fcf50062e80 con 0x7fcf380a3d50 2024-04-23T18:18:02.995 INFO:teuthology.orchestra.run.smithi050.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T18:18:02.995 INFO:teuthology.orchestra.run.smithi050.stdout:mds.a smithi050 running (97s) 15s ago 96s 16.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 48c5cf119a73 2024-04-23T18:18:02.995 INFO:teuthology.orchestra.run.smithi050.stdout:mds.b smithi177 running (94s) 28s ago 94s 28.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 87e740c0e35f 2024-04-23T18:18:02.995 INFO:teuthology.orchestra.run.smithi050.stdout:mds.c smithi188 running (108s) 28s ago 108s 19.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 9ced8839238f 2024-04-23T18:18:02.995 INFO:teuthology.orchestra.run.smithi050.stdout:mds.d smithi050 running (116s) 15s ago 115s 15.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 637b26b1f203 2024-04-23T18:18:02.995 INFO:teuthology.orchestra.run.smithi050.stdout:mds.e smithi177 running (113s) 28s ago 113s 19.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 18dff365af23 2024-04-23T18:18:02.995 INFO:teuthology.orchestra.run.smithi050.stdout:mds.f smithi188 running (111s) 28s ago 110s 18.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 3323d250daaa 2024-04-23T18:18:02.995 INFO:teuthology.orchestra.run.smithi050.stdout:mds.g smithi050 running (100s) 15s ago 99s 12.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 ba154573e027 2024-04-23T18:18:02.995 INFO:teuthology.orchestra.run.smithi050.stdout:mds.h smithi177 running (105s) 28s ago 105s 20.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 9b09d288c4b2 2024-04-23T18:18:02.995 INFO:teuthology.orchestra.run.smithi050.stdout:mds.i smithi188 running (102s) 28s ago 102s 19.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 d24dc11dcb0a 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:mds.j smithi050 running (119s) 15s ago 118s 14.1M - 19.0.0-3156-gd69f13d8 da4b004c09e8 cea48831c114 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:mds.k smithi177 running (2m) 28s ago 2m 16.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 6e3b8564481e 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:mds.l smithi188 running (2m) 28s ago 2m 14.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 80cd05b82f1c 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:mgr.x smithi050 *:9283,8765 running (9m) 15s ago 9m 515M - 19.0.0-3156-gd69f13d8 da4b004c09e8 6cdd679662e2 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:mgr.y smithi177 *:8443,8765 running (7m) 28s ago 7m 453M - 19.0.0-3156-gd69f13d8 da4b004c09e8 0f041e72f6ee 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:mgr.z smithi188 *:8443,8765 running (7m) 28s ago 7m 454M - 19.0.0-3156-gd69f13d8 da4b004c09e8 315dbf4c4ee4 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:mon.a smithi050 running (9m) 15s ago 9m 49.4M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 b09dd59349ea 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:mon.b smithi177 running (7m) 28s ago 7m 43.7M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 c0b85f066c10 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:mon.c smithi188 running (7m) 28s ago 7m 40.5M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 df051151edb9 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0 smithi050 running (6m) 15s ago 6m 54.5M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 186e4caf556a 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1 smithi177 running (6m) 28s ago 6m 48.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 c84cc7089d29 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2 smithi188 running (6m) 28s ago 6m 56.6M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 a2f6df90dae5 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3 smithi050 running (5m) 15s ago 5m 54.7M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 5a43518c9d04 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4 smithi177 running (5m) 28s ago 5m 52.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 be28b11658c5 2024-04-23T18:18:02.996 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5 smithi188 running (4m) 28s ago 4m 49.2M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 debc47de7003 2024-04-23T18:18:02.997 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6 smithi050 running (4m) 15s ago 4m 45.2M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 e126837afdae 2024-04-23T18:18:02.997 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7 smithi177 running (4m) 28s ago 4m 56.0M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 b7b3ad0105d1 2024-04-23T18:18:02.997 INFO:teuthology.orchestra.run.smithi050.stdout:osd.8 smithi188 running (3m) 28s ago 3m 46.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 24e4f3519493 2024-04-23T18:18:02.997 INFO:teuthology.orchestra.run.smithi050.stdout:osd.9 smithi050 running (3m) 15s ago 3m 52.6M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 2d03b62f9fc1 2024-04-23T18:18:02.997 INFO:teuthology.orchestra.run.smithi050.stdout:osd.10 smithi177 running (2m) 28s ago 2m 57.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 1391ad4173de 2024-04-23T18:18:02.997 INFO:teuthology.orchestra.run.smithi050.stdout:osd.11 smithi188 running (2m) 28s ago 2m 43.4M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 a1e0911f2bb0 2024-04-23T18:18:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.003+0000 7fcf55a57700 1 -- 172.21.15.50:0/2845577785 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcf380a3d50 msgr2=0x7fcf380a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.003+0000 7fcf55a57700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcf380a3d50 0x7fcf380a6210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcf400038a0 tx=0x7fcf40005fb0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.003+0000 7fcf55a57700 1 -- 172.21.15.50:0/2845577785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf50076a70 msgr2=0x7fcf5019f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.003+0000 7fcf55a57700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf50076a70 0x7fcf5019f700 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400d9b0 tx=0x7fcf4400dcc0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.003+0000 7fcf55a57700 1 -- 172.21.15.50:0/2845577785 shutdown_connections 2024-04-23T18:18:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.003+0000 7fcf55a57700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf5010f0a0 0x7fcf501a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.003+0000 7fcf55a57700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcf380a3d50 0x7fcf380a6210 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.003+0000 7fcf55a57700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf500754a0 0x7fcf5019f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.003+0000 7fcf55a57700 1 --2- 172.21.15.50:0/2845577785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcf50076a70 0x7fcf5019f700 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.003+0000 7fcf55a57700 1 -- 172.21.15.50:0/2845577785 >> 172.21.15.50:0/2845577785 conn(0x7fcf500fe060 msgr2=0x7fcf50102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.003+0000 7fcf55a57700 1 -- 172.21.15.50:0/2845577785 shutdown_connections 2024-04-23T18:18:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:03.003+0000 7fcf55a57700 1 -- 172.21.15.50:0/2845577785 wait complete. 2024-04-23T18:18:03.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-23T18:18:03.944 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:18:04.391 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:04 smithi050 ceph-mon[29187]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-23T18:18:04.391 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:04 smithi050 ceph-mon[29187]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:04.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:04 smithi177 ceph-mon[32308]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-23T18:18:04.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:04 smithi177 ceph-mon[32308]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:04.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:04 smithi188 ceph-mon[32694]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-23T18:18:04.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:04 smithi188 ceph-mon[32694]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.585+0000 7f1f2ead8700 1 -- 172.21.15.50:0/1191163003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f28101170 msgr2=0x7f1f28101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.585+0000 7f1f2ead8700 1 --2- 172.21.15.50:0/1191163003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f28101170 0x7f1f28101590 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0099c0 tx=0x7f1f1c009cd0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.585+0000 7f1f2ead8700 1 -- 172.21.15.50:0/1191163003 shutdown_connections 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.585+0000 7f1f2ead8700 1 --2- 172.21.15.50:0/1191163003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f28102d10 0x7f1f281095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.585+0000 7f1f2ead8700 1 --2- 172.21.15.50:0/1191163003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f28101170 0x7f1f28101590 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.585+0000 7f1f2ead8700 1 --2- 172.21.15.50:0/1191163003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f28102370 0x7f1f281027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.586+0000 7f1f2ead8700 1 -- 172.21.15.50:0/1191163003 >> 172.21.15.50:0/1191163003 conn(0x7f1f280fc710 msgr2=0x7f1f280feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.586+0000 7f1f2ead8700 1 -- 172.21.15.50:0/1191163003 shutdown_connections 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.586+0000 7f1f2ead8700 1 -- 172.21.15.50:0/1191163003 wait complete. 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.586+0000 7f1f2ead8700 1 Processor -- start 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.586+0000 7f1f2ead8700 1 -- start start 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.586+0000 7f1f2ead8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f28101170 0x7f1f28105100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.586+0000 7f1f2ead8700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f28102370 0x7f1f28103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.586+0000 7f1f2ead8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f28102d10 0x7f1f28103c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.586+0000 7f1f2ead8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1f2810d4a0 con 0x7f1f28102d10 2024-04-23T18:18:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.586+0000 7f1f2ead8700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1f2810d320 con 0x7f1f28102370 2024-04-23T18:18:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.586+0000 7f1f2ead8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1f2810d620 con 0x7f1f28101170 2024-04-23T18:18:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f2d075700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f28102d10 0x7f1f28103c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f2d075700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f28102d10 0x7f1f28103c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38926/0 (socket says 172.21.15.50:38926) 2024-04-23T18:18:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f2d075700 1 -- 172.21.15.50:0/338030446 learned_addr learned my addr 172.21.15.50:0/338030446 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f2c874700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f28101170 0x7f1f28105100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f27fff700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f28102370 0x7f1f28103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f2d075700 1 -- 172.21.15.50:0/338030446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f28101170 msgr2=0x7f1f28105100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f2d075700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f28101170 0x7f1f28105100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f2d075700 1 -- 172.21.15.50:0/338030446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f28102370 msgr2=0x7f1f28103750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f2d075700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f28102370 0x7f1f28103750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f2d075700 1 -- 172.21.15.50:0/338030446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f28104540 con 0x7f1f28102d10 2024-04-23T18:18:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f2d075700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f28102d10 0x7f1f28103c90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1f2000ada0 tx=0x7f1f2000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f25ffb700 1 -- 172.21.15.50:0/338030446 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f20017020 con 0x7f1f28102d10 2024-04-23T18:18:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f25ffb700 1 -- 172.21.15.50:0/338030446 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f2000ce30 con 0x7f1f28102d10 2024-04-23T18:18:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f25ffb700 1 -- 172.21.15.50:0/338030446 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f20005350 con 0x7f1f28102d10 2024-04-23T18:18:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.587+0000 7f1f2ead8700 1 -- 172.21.15.50:0/338030446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f281a2d50 con 0x7f1f28102d10 2024-04-23T18:18:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.588+0000 7f1f2ead8700 1 -- 172.21.15.50:0/338030446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f281a31b0 con 0x7f1f28102d10 2024-04-23T18:18:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.588+0000 7f1f2ead8700 1 -- 172.21.15.50:0/338030446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f281a33e0 con 0x7f1f28102d10 2024-04-23T18:18:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.589+0000 7f1f25ffb700 1 -- 172.21.15.50:0/338030446 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1f2000c980 con 0x7f1f28102d10 2024-04-23T18:18:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.590+0000 7f1f25ffb700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1f180a3cc0 0x7f1f180a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.590+0000 7f1f2c874700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1f180a3cc0 0x7f1f180a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.590+0000 7f1f25ffb700 1 -- 172.21.15.50:0/338030446 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f1f200cfae0 con 0x7f1f28102d10 2024-04-23T18:18:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.591+0000 7f1f2c874700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1f180a3cc0 0x7f1f180a6180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0038c0 tx=0x7f1f1c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:05.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.595+0000 7f1f25ffb700 1 -- 172.21.15.50:0/338030446 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f20021020 con 0x7f1f28102d10 2024-04-23T18:18:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.810+0000 7f1f2ead8700 1 -- 172.21.15.50:0/338030446 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f1f280fe8c0 con 0x7f1f180a3cc0 2024-04-23T18:18:05.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.816+0000 7f1f25ffb700 1 -- 172.21.15.50:0/338030446 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f1f280fe8c0 con 0x7f1f180a3cc0 2024-04-23T18:18:05.810 INFO:teuthology.orchestra.run.smithi050.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-23T18:18:05.811 INFO:teuthology.orchestra.run.smithi050.stdout:mds.all 12/12 31s ago 2m smithi050=a;smithi050=d;smithi050=g;smithi050=j;smithi177=b;smithi177=e;smithi177=h;smithi177=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2024-04-23T18:18:05.811 INFO:teuthology.orchestra.run.smithi050.stdout:mgr 3/3 31s ago 7m smithi050=x;smithi177=y;smithi188=z;count:3 2024-04-23T18:18:05.811 INFO:teuthology.orchestra.run.smithi050.stdout:mon 3/3 31s ago 7m smithi050:172.21.15.50=a;smithi177:172.21.15.177=b;smithi188:172.21.15.188=c;count:3 2024-04-23T18:18:05.811 INFO:teuthology.orchestra.run.smithi050.stdout:osd 12 31s ago - 2024-04-23T18:18:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.819+0000 7f1f2ead8700 1 -- 172.21.15.50:0/338030446 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1f180a3cc0 msgr2=0x7f1f180a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.819+0000 7f1f2ead8700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1f180a3cc0 0x7f1f180a6180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0038c0 tx=0x7f1f1c00b540 comp rx=0 tx=0).stop 2024-04-23T18:18:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.819+0000 7f1f2ead8700 1 -- 172.21.15.50:0/338030446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f28102d10 msgr2=0x7f1f28103c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.819+0000 7f1f2ead8700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f28102d10 0x7f1f28103c90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1f2000ada0 tx=0x7f1f2000c330 comp rx=0 tx=0).stop 2024-04-23T18:18:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.819+0000 7f1f2ead8700 1 -- 172.21.15.50:0/338030446 shutdown_connections 2024-04-23T18:18:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.819+0000 7f1f2ead8700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f28102d10 0x7f1f28103c90 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.819+0000 7f1f2ead8700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f28101170 0x7f1f28105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.819+0000 7f1f2ead8700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1f180a3cc0 0x7f1f180a6180 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.819+0000 7f1f2ead8700 1 --2- 172.21.15.50:0/338030446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f28102370 0x7f1f28103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.819+0000 7f1f2ead8700 1 -- 172.21.15.50:0/338030446 >> 172.21.15.50:0/338030446 conn(0x7f1f280fc710 msgr2=0x7f1f280fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.819+0000 7f1f2ead8700 1 -- 172.21.15.50:0/338030446 shutdown_connections 2024-04-23T18:18:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:05.819+0000 7f1f2ead8700 1 -- 172.21.15.50:0/338030446 wait complete. 2024-04-23T18:18:06.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:06 smithi050 ceph-mon[29187]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:18:06.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:06 smithi050 ceph-mon[29187]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-23T18:18:06.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-23T18:18:06.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:06 smithi177 ceph-mon[32308]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:18:06.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:06 smithi177 ceph-mon[32308]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-23T18:18:06.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:06 smithi188 ceph-mon[32694]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:18:06.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:06 smithi188 ceph-mon[32694]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-23T18:18:06.656 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:18:07.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:07 smithi050 ceph-mon[29187]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:07.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:07 smithi177 ceph-mon[32308]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:07.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:07 smithi188 ceph-mon[32694]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:08.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.227+0000 7f135a9a0700 1 -- 172.21.15.50:0/4094082188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354102660 msgr2=0x7f1354102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:08.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.227+0000 7f135a9a0700 1 --2- 172.21.15.50:0/4094082188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354102660 0x7f1354102ae0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f134c00b0a0 tx=0x7f134c00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:18:08.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.227+0000 7f135a9a0700 1 -- 172.21.15.50:0/4094082188 shutdown_connections 2024-04-23T18:18:08.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.227+0000 7f135a9a0700 1 --2- 172.21.15.50:0/4094082188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354102660 0x7f1354102ae0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:08.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.227+0000 7f135a9a0700 1 --2- 172.21.15.50:0/4094082188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1354101340 0x7f1354101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:08.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.227+0000 7f135a9a0700 1 --2- 172.21.15.50:0/4094082188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1354101ca0 0x7f1354102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:08.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.227+0000 7f135a9a0700 1 -- 172.21.15.50:0/4094082188 >> 172.21.15.50:0/4094082188 conn(0x7f13540fcf60 msgr2=0x7f13540ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:08.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.227+0000 7f135a9a0700 1 -- 172.21.15.50:0/4094082188 shutdown_connections 2024-04-23T18:18:08.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.227+0000 7f135a9a0700 1 -- 172.21.15.50:0/4094082188 wait complete. 2024-04-23T18:18:08.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.227+0000 7f135a9a0700 1 Processor -- start 2024-04-23T18:18:08.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.227+0000 7f135a9a0700 1 -- start start 2024-04-23T18:18:08.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f135a9a0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354101340 0x7f135419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:08.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f135a9a0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1354101ca0 0x7f135419b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:08.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f135a9a0700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1354102660 0x7f13541a2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f135a9a0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1354110720 con 0x7f1354101340 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f135a9a0700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f13541105a0 con 0x7f1354102660 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f135a9a0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f13541108a0 con 0x7f1354101ca0 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f1353fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354101340 0x7f135419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f1353fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354101340 0x7f135419aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38956/0 (socket says 172.21.15.50:38956) 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f1353fff700 1 -- 172.21.15.50:0/1432374697 learned_addr learned my addr 172.21.15.50:0/1432374697 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f1358f3d700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1354102660 0x7f13541a2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f13537fe700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1354101ca0 0x7f135419b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f1353fff700 1 -- 172.21.15.50:0/1432374697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1354101ca0 msgr2=0x7f135419b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f1353fff700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1354101ca0 0x7f135419b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f1353fff700 1 -- 172.21.15.50:0/1432374697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1354102660 msgr2=0x7f13541a2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f1353fff700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1354102660 0x7f13541a2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f1353fff700 1 -- 172.21.15.50:0/1432374697 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13541a2920 con 0x7f1354101340 2024-04-23T18:18:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.228+0000 7f1353fff700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354101340 0x7f135419aca0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f134400c9b0 tx=0x7f134400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.229+0000 7f13517fa700 1 -- 172.21.15.50:0/1432374697 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344007b00 con 0x7f1354101340 2024-04-23T18:18:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.229+0000 7f13517fa700 1 -- 172.21.15.50:0/1432374697 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f134400ce50 con 0x7f1354101340 2024-04-23T18:18:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.229+0000 7f13517fa700 1 -- 172.21.15.50:0/1432374697 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344018660 con 0x7f1354101340 2024-04-23T18:18:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.229+0000 7f135a9a0700 1 -- 172.21.15.50:0/1432374697 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13541a2c10 con 0x7f1354101340 2024-04-23T18:18:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.229+0000 7f135a9a0700 1 -- 172.21.15.50:0/1432374697 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13541a3100 con 0x7f1354101340 2024-04-23T18:18:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.230+0000 7f135a9a0700 1 -- 172.21.15.50:0/1432374697 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1354069730 con 0x7f1354101340 2024-04-23T18:18:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.231+0000 7f13517fa700 1 -- 172.21.15.50:0/1432374697 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f134400f450 con 0x7f1354101340 2024-04-23T18:18:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.231+0000 7f13517fa700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f133c0a3c40 0x7f133c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.232+0000 7f13537fe700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f133c0a3c40 0x7f133c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.232+0000 7f13517fa700 1 -- 172.21.15.50:0/1432374697 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f13440d05a0 con 0x7f1354101340 2024-04-23T18:18:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.232+0000 7f13537fe700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f133c0a3c40 0x7f133c0a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f135419c1b0 tx=0x7f134800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.235+0000 7f13517fa700 1 -- 172.21.15.50:0/1432374697 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1344014030 con 0x7f1354101340 2024-04-23T18:18:08.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.450+0000 7f135a9a0700 1 -- 172.21.15.50:0/1432374697 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f1354062e80 con 0x7f133c0a3c40 2024-04-23T18:18:08.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:08 smithi050 ceph-mon[29187]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T18:18:08.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.451+0000 7f13517fa700 1 -- 172.21.15.50:0/1432374697 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f1354062e80 con 0x7f133c0a3c40 2024-04-23T18:18:08.445 INFO:teuthology.orchestra.run.smithi050.stdout:HOST ADDR LABELS STATUS 2024-04-23T18:18:08.445 INFO:teuthology.orchestra.run.smithi050.stdout:smithi050 172.21.15.50 2024-04-23T18:18:08.445 INFO:teuthology.orchestra.run.smithi050.stdout:smithi177 172.21.15.177 2024-04-23T18:18:08.445 INFO:teuthology.orchestra.run.smithi050.stdout:smithi188 172.21.15.188 2024-04-23T18:18:08.445 INFO:teuthology.orchestra.run.smithi050.stdout:3 hosts in cluster 2024-04-23T18:18:08.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.453+0000 7f135a9a0700 1 -- 172.21.15.50:0/1432374697 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f133c0a3c40 msgr2=0x7f133c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.453+0000 7f135a9a0700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f133c0a3c40 0x7f133c0a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f135419c1b0 tx=0x7f134800a300 comp rx=0 tx=0).stop 2024-04-23T18:18:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.453+0000 7f135a9a0700 1 -- 172.21.15.50:0/1432374697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354101340 msgr2=0x7f135419aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.453+0000 7f135a9a0700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354101340 0x7f135419aca0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f134400c9b0 tx=0x7f134400ccc0 comp rx=0 tx=0).stop 2024-04-23T18:18:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.453+0000 7f135a9a0700 1 -- 172.21.15.50:0/1432374697 shutdown_connections 2024-04-23T18:18:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.453+0000 7f135a9a0700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f133c0a3c40 0x7f133c0a6100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.453+0000 7f135a9a0700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1354102660 0x7f13541a2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.453+0000 7f135a9a0700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354101340 0x7f135419aca0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.454+0000 7f135a9a0700 1 --2- 172.21.15.50:0/1432374697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1354101ca0 0x7f135419b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.454+0000 7f135a9a0700 1 -- 172.21.15.50:0/1432374697 >> 172.21.15.50:0/1432374697 conn(0x7f13540fcf60 msgr2=0x7f13540fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.454+0000 7f135a9a0700 1 -- 172.21.15.50:0/1432374697 shutdown_connections 2024-04-23T18:18:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:08.454+0000 7f135a9a0700 1 -- 172.21.15.50:0/1432374697 wait complete. 2024-04-23T18:18:08.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:08 smithi177 ceph-mon[32308]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T18:18:08.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:08 smithi188 ceph-mon[32694]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T18:18:09.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-23T18:18:09.305 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:18:10.401 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:10 smithi050 ceph-mon[29187]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:10.401 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:10 smithi050 ceph-mon[29187]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:18:10.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:10 smithi177 ceph-mon[32308]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:10.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:10 smithi177 ceph-mon[32308]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:18:10.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:10 smithi188 ceph-mon[32694]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:10.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:10 smithi188 ceph-mon[32694]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T18:18:10.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.916+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3387405010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a28076a70 msgr2=0x7f1a28074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:10.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.916+0000 7f1a2f2d9700 1 --2- 172.21.15.50:0/3387405010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a28076a70 0x7f1a28074ed0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009990 tx=0x7f1a24009ca0 comp rx=0 tx=0).stop 2024-04-23T18:18:10.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.916+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3387405010 shutdown_connections 2024-04-23T18:18:10.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.916+0000 7f1a2f2d9700 1 --2- 172.21.15.50:0/3387405010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a2810f0b0 0x7f1a281114a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:10.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.916+0000 7f1a2f2d9700 1 --2- 172.21.15.50:0/3387405010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a28076a70 0x7f1a28074ed0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:10.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.916+0000 7f1a2f2d9700 1 --2- 172.21.15.50:0/3387405010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a280754a0 0x7f1a28075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:10.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.916+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3387405010 >> 172.21.15.50:0/3387405010 conn(0x7f1a280fe090 msgr2=0x7f1a281004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:10.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.916+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3387405010 shutdown_connections 2024-04-23T18:18:10.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.916+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3387405010 wait complete. 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.916+0000 7f1a2f2d9700 1 Processor -- start 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.916+0000 7f1a2f2d9700 1 -- start start 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2f2d9700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a280754a0 0x7f1a2819f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2f2d9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a28076a70 0x7f1a2819f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2f2d9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a2810f0b0 0x7f1a281a6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2f2d9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1a28114c60 con 0x7f1a2810f0b0 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2f2d9700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1a28114ae0 con 0x7f1a280754a0 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2f2d9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1a28114de0 con 0x7f1a28076a70 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2d876700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a2810f0b0 0x7f1a281a6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2d876700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a2810f0b0 0x7f1a281a6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38968/0 (socket says 172.21.15.50:38968) 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2d876700 1 -- 172.21.15.50:0/3547728720 learned_addr learned my addr 172.21.15.50:0/3547728720 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2d075700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a280754a0 0x7f1a2819f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:10.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2c874700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a28076a70 0x7f1a2819f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2d876700 1 -- 172.21.15.50:0/3547728720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a28076a70 msgr2=0x7f1a2819f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2d876700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a28076a70 0x7f1a2819f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2d876700 1 -- 172.21.15.50:0/3547728720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a280754a0 msgr2=0x7f1a2819f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2d876700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a280754a0 0x7f1a2819f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2d876700 1 -- 172.21.15.50:0/3547728720 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a281a6ff0 con 0x7f1a2810f0b0 2024-04-23T18:18:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2c874700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a28076a70 0x7f1a2819f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.917+0000 7f1a2d876700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a2810f0b0 0x7f1a281a6950 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1a2000b9a0 tx=0x7f1a2000bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.918+0000 7f1a1e7fc700 1 -- 172.21.15.50:0/3547728720 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20019070 con 0x7f1a2810f0b0 2024-04-23T18:18:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.918+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3547728720 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a281a72e0 con 0x7f1a2810f0b0 2024-04-23T18:18:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.918+0000 7f1a1e7fc700 1 -- 172.21.15.50:0/3547728720 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a2000cde0 con 0x7f1a2810f0b0 2024-04-23T18:18:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.918+0000 7f1a1e7fc700 1 -- 172.21.15.50:0/3547728720 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20007370 con 0x7f1a2810f0b0 2024-04-23T18:18:10.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.918+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3547728720 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a281a7850 con 0x7f1a2810f0b0 2024-04-23T18:18:10.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.919+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3547728720 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a28069730 con 0x7f1a2810f0b0 2024-04-23T18:18:10.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.920+0000 7f1a1e7fc700 1 -- 172.21.15.50:0/3547728720 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1a2001c070 con 0x7f1a2810f0b0 2024-04-23T18:18:10.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.920+0000 7f1a1e7fc700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1a140a3c40 0x7f1a140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:10.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.920+0000 7f1a2d075700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1a140a3c40 0x7f1a140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:10.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.920+0000 7f1a1e7fc700 1 -- 172.21.15.50:0/3547728720 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f1a200cf8e0 con 0x7f1a2810f0b0 2024-04-23T18:18:10.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.921+0000 7f1a2d075700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1a140a3c40 0x7f1a140a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1a240038a0 tx=0x7f1a24005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:10.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:10.925+0000 7f1a1e7fc700 1 -- 172.21.15.50:0/3547728720 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a2009c2c0 con 0x7f1a2810f0b0 2024-04-23T18:18:11.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.142+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3547728720 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f1a28062e80 con 0x7f1a140a3c40 2024-04-23T18:18:11.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.143+0000 7f1a1e7fc700 1 -- 172.21.15.50:0/3547728720 <== mgr.14152 v2:172.21.15.50:6800/1004018784 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f1a28062e80 con 0x7f1a140a3c40 2024-04-23T18:18:11.137 INFO:teuthology.orchestra.run.smithi050.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-23T18:18:11.137 INFO:teuthology.orchestra.run.smithi050.stdout:smithi050 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513001F480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-23T18:18:11.137 INFO:teuthology.orchestra.run.smithi050.stdout:smithi177 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233007S400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-23T18:18:11.138 INFO:teuthology.orchestra.run.smithi050.stdout:smithi188 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500MP400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-23T18:18:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.146+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3547728720 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1a140a3c40 msgr2=0x7f1a140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.146+0000 7f1a2f2d9700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1a140a3c40 0x7f1a140a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1a240038a0 tx=0x7f1a24005fb0 comp rx=0 tx=0).stop 2024-04-23T18:18:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.146+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3547728720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a2810f0b0 msgr2=0x7f1a281a6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.146+0000 7f1a2f2d9700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a2810f0b0 0x7f1a281a6950 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1a2000b9a0 tx=0x7f1a2000bcb0 comp rx=0 tx=0).stop 2024-04-23T18:18:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.146+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3547728720 shutdown_connections 2024-04-23T18:18:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.146+0000 7f1a2f2d9700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a2810f0b0 0x7f1a281a6950 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.146+0000 7f1a2f2d9700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f1a140a3c40 0x7f1a140a6100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.146+0000 7f1a2f2d9700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a280754a0 0x7f1a2819f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.146+0000 7f1a2f2d9700 1 --2- 172.21.15.50:0/3547728720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a28076a70 0x7f1a2819f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:11.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.146+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3547728720 >> 172.21.15.50:0/3547728720 conn(0x7f1a280fe090 msgr2=0x7f1a28102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:11.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.146+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3547728720 shutdown_connections 2024-04-23T18:18:11.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:11.146+0000 7f1a2f2d9700 1 -- 172.21.15.50:0/3547728720 wait complete. 2024-04-23T18:18:11.771 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T18:18:11.779 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi050.front.sepia.ceph.com 2024-04-23T18:18:11.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-23T18:18:12.028 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:18:12.395 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:12 smithi050 ceph-mon[29187]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-23T18:18:12.395 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:12 smithi050 ceph-mon[29187]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:12.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:12 smithi177 ceph-mon[32308]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-23T18:18:12.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:12 smithi177 ceph-mon[32308]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:12.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:12 smithi188 ceph-mon[32694]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-23T18:18:12.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:12 smithi188 ceph-mon[32694]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.821+0000 7f1712bd1700 1 -- 172.21.15.50:0/3285735380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f170c105e50 msgr2=0x7f170c108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.821+0000 7f1712bd1700 1 --2- 172.21.15.50:0/3285735380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f170c105e50 0x7f170c108240 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00b0a0 tx=0x7f16fc00b3b0 comp rx=0 tx=0).stop 2024-04-23T18:18:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.821+0000 7f1712bd1700 1 -- 172.21.15.50:0/3285735380 shutdown_connections 2024-04-23T18:18:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.821+0000 7f1712bd1700 1 --2- 172.21.15.50:0/3285735380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f170c108780 0x7f170c10abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.821+0000 7f1712bd1700 1 --2- 172.21.15.50:0/3285735380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f170c103520 0x7f170c105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.821+0000 7f1712bd1700 1 --2- 172.21.15.50:0/3285735380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f170c105e50 0x7f170c108240 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.821+0000 7f1712bd1700 1 -- 172.21.15.50:0/3285735380 >> 172.21.15.50:0/3285735380 conn(0x7f170c0fcf30 msgr2=0x7f170c0ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.821+0000 7f1712bd1700 1 -- 172.21.15.50:0/3285735380 shutdown_connections 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.821+0000 7f1712bd1700 1 -- 172.21.15.50:0/3285735380 wait complete. 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f1712bd1700 1 Processor -- start 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f1712bd1700 1 -- start start 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f1712bd1700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f170c103520 0x7f170c102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f1712bd1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f170c105e50 0x7f170c1014d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f1712bd1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f170c108780 0x7f170c101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f1712bd1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f170c10e570 con 0x7f170c108780 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f1712bd1700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f170c10e3f0 con 0x7f170c103520 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f1712bd1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f170c10e6f0 con 0x7f170c105e50 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f171116e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f170c108780 0x7f170c101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f171116e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f170c108780 0x7f170c101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40198/0 (socket says 172.21.15.50:40198) 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f171116e700 1 -- 172.21.15.50:0/598107474 learned_addr learned my addr 172.21.15.50:0/598107474 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f170bfff700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f170c105e50 0x7f170c1014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.822+0000 7f171096d700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f170c103520 0x7f170c102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.823+0000 7f171116e700 1 -- 172.21.15.50:0/598107474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f170c105e50 msgr2=0x7f170c1014d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.823+0000 7f171116e700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f170c105e50 0x7f170c1014d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.823+0000 7f171116e700 1 -- 172.21.15.50:0/598107474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f170c103520 msgr2=0x7f170c102e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.823+0000 7f171116e700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f170c103520 0x7f170c102e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.823+0000 7f171116e700 1 -- 172.21.15.50:0/598107474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170c1022c0 con 0x7f170c108780 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.823+0000 7f171096d700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f170c103520 0x7f170c102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.823+0000 7f171116e700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f170c108780 0x7f170c101a10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f170400d740 tx=0x7f170400da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.823+0000 7f1709ffb700 1 -- 172.21.15.50:0/598107474 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704016070 con 0x7f170c108780 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.823+0000 7f1709ffb700 1 -- 172.21.15.50:0/598107474 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f170400f690 con 0x7f170c108780 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.823+0000 7f1712bd1700 1 -- 172.21.15.50:0/598107474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170c1a2dd0 con 0x7f170c108780 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.823+0000 7f1709ffb700 1 -- 172.21.15.50:0/598107474 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704003a80 con 0x7f170c108780 2024-04-23T18:18:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.823+0000 7f1712bd1700 1 -- 172.21.15.50:0/598107474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f170c1a3230 con 0x7f170c108780 2024-04-23T18:18:13.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.824+0000 7f1712bd1700 1 -- 172.21.15.50:0/598107474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f170c069730 con 0x7f170c108780 2024-04-23T18:18:13.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.825+0000 7f1709ffb700 1 -- 172.21.15.50:0/598107474 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1704003c20 con 0x7f170c108780 2024-04-23T18:18:13.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.826+0000 7f1709ffb700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f16f80a3c40 0x7f16f80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:13.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.826+0000 7f171096d700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f16f80a3c40 0x7f16f80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:13.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.826+0000 7f1709ffb700 1 -- 172.21.15.50:0/598107474 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f17040cfea0 con 0x7f170c108780 2024-04-23T18:18:13.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.826+0000 7f171096d700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f16f80a3c40 0x7f16f80a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005950 tx=0x7f16f40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:13.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:13.830+0000 7f1709ffb700 1 -- 172.21.15.50:0/598107474 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f170409c880 con 0x7f170c108780 2024-04-23T18:18:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.089+0000 7f1712bd1700 1 -- 172.21.15.50:0/598107474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f170c063650 con 0x7f170c108780 2024-04-23T18:18:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.089+0000 7f1709ffb700 1 -- 172.21.15.50:0/598107474 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) ==== 76+0+3052 (secure 0 0 0) 0x7f170409c180 con 0x7f170c108780 2024-04-23T18:18:14.083 INFO:teuthology.orchestra.run.smithi050.stdout:e20 2024-04-23T18:18:14.083 INFO:teuthology.orchestra.run.smithi050.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T18:18:14.083 INFO:teuthology.orchestra.run.smithi050.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T18:18:14.083 INFO:teuthology.orchestra.run.smithi050.stdout:legacy client fscid: 1 2024-04-23T18:18:14.083 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:Filesystem 'cephfs' (1) 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:fs_name cephfs 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 20 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:created 2024-04-23T18:16:33.316468+0000 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:modified 2024-04-23T18:16:37.965071+0000 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:tableserver 0 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:root 0 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:session_timeout 300 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:session_autoclose 300 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:max_file_size 1099511627776 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:max_xattr_size 65536 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:required_client_features {} 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:last_failure 0 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:last_failure_osd_epoch 0 2024-04-23T18:18:14.084 INFO:teuthology.orchestra.run.smithi050.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:max_mds 3 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:in 0,1,2 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:up {0=24506,1=24499,2=24475} 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:failed 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:damaged 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:stopped 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:data_pools [3] 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:metadata_pool 2 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:inline_data disabled 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:balancer 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:bal_rank_mask -1 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:standby_count_wanted 0 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:qdb_cluster leader: 24506 members: 24475,24506,24499 2024-04-23T18:18:14.085 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.b{0:24506} state up:active seq 3 addr [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.h{0:24467} state up:standby-replay seq 1 addr [v2:172.21.15.177:6836/1048893636,v1:172.21.15.177:6837/1048893636] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.i{1:24499} state up:active seq 5 addr [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.f{1:24463} state up:standby-replay seq 1 addr [v2:172.21.15.188:6834/1248812622,v1:172.21.15.188:6835/1248812622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.c{2:24475} state up:active seq 7 addr [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.e{2:24440} state up:standby-replay seq 1 addr [v2:172.21.15.177:6834/475975073,v1:172.21.15.177:6835/475975073] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout:Standby daemons: 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.j{-1:14526} state up:standby seq 1 addr [v2:172.21.15.50:6834/878281924,v1:172.21.15.50:6835/878281924] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.d{-1:14538} state up:standby seq 1 addr [v2:172.21.15.50:6836/1480144576,v1:172.21.15.50:6837/1480144576] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.g{-1:14604} state up:standby seq 1 addr [v2:172.21.15.50:6838/2730935833,v1:172.21.15.50:6839/2730935833] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.a{-1:14616} state up:standby seq 1 addr [v2:172.21.15.50:6840/3198542689,v1:172.21.15.50:6841/3198542689] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:18:14.086 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.k{-1:24404} state up:standby seq 1 addr [v2:172.21.15.177:6832/878222815,v1:172.21.15.177:6833/878222815] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:18:14.087 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.l{-1:24412} state up:standby seq 1 addr [v2:172.21.15.188:6832/3630824399,v1:172.21.15.188:6833/3630824399] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T18:18:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.091+0000 7f1712bd1700 1 -- 172.21.15.50:0/598107474 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f16f80a3c40 msgr2=0x7f16f80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.091+0000 7f1712bd1700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f16f80a3c40 0x7f16f80a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005950 tx=0x7f16f40058e0 comp rx=0 tx=0).stop 2024-04-23T18:18:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.091+0000 7f1712bd1700 1 -- 172.21.15.50:0/598107474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f170c108780 msgr2=0x7f170c101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.091+0000 7f1712bd1700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f170c108780 0x7f170c101a10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f170400d740 tx=0x7f170400da50 comp rx=0 tx=0).stop 2024-04-23T18:18:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.091+0000 7f1712bd1700 1 -- 172.21.15.50:0/598107474 shutdown_connections 2024-04-23T18:18:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.091+0000 7f1712bd1700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f170c108780 0x7f170c101a10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.091+0000 7f1712bd1700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f16f80a3c40 0x7f16f80a6100 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.091+0000 7f1712bd1700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f170c103520 0x7f170c102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.091+0000 7f1712bd1700 1 --2- 172.21.15.50:0/598107474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f170c105e50 0x7f170c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.091+0000 7f1712bd1700 1 -- 172.21.15.50:0/598107474 >> 172.21.15.50:0/598107474 conn(0x7f170c0fcf30 msgr2=0x7f170c103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.091+0000 7f1712bd1700 1 -- 172.21.15.50:0/598107474 shutdown_connections 2024-04-23T18:18:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:14.091+0000 7f1712bd1700 1 -- 172.21.15.50:0/598107474 wait complete. 2024-04-23T18:18:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 20 2024-04-23T18:18:14.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:14 smithi050 ceph-mon[29187]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-23T18:18:14.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:14 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/598107474' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T18:18:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:14 smithi177 ceph-mon[32308]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-23T18:18:14.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:14 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/598107474' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T18:18:14.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:14 smithi188 ceph-mon[32694]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-23T18:18:14.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:14 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/598107474' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T18:18:14.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-23T18:18:15.011 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:18:16.381 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:16 smithi188 ceph-mon[32694]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T18:18:16.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:16 smithi050 ceph-mon[29187]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T18:18:16.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:16 smithi177 ceph-mon[32308]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T18:18:16.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.802+0000 7f1077bd3700 1 -- 172.21.15.50:0/1821051578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10701087b0 msgr2=0x7f107010ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.802+0000 7f1077bd3700 1 --2- 172.21.15.50:0/1821051578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10701087b0 0x7f107010ac20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f106800b0a0 tx=0x7f106800b3b0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.802+0000 7f1077bd3700 1 -- 172.21.15.50:0/1821051578 shutdown_connections 2024-04-23T18:18:16.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.802+0000 7f1077bd3700 1 --2- 172.21.15.50:0/1821051578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10701087b0 0x7f107010ac20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.802+0000 7f1077bd3700 1 --2- 172.21.15.50:0/1821051578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1070103550 0x7f1070105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.802+0000 7f1077bd3700 1 --2- 172.21.15.50:0/1821051578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1070105e80 0x7f1070108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.802+0000 7f1077bd3700 1 -- 172.21.15.50:0/1821051578 >> 172.21.15.50:0/1821051578 conn(0x7f10700fcf60 msgr2=0x7f10700ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.802+0000 7f1077bd3700 1 -- 172.21.15.50:0/1821051578 shutdown_connections 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.802+0000 7f1077bd3700 1 -- 172.21.15.50:0/1821051578 wait complete. 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.802+0000 7f1077bd3700 1 Processor -- start 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f1077bd3700 1 -- start start 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f1077bd3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1070103550 0x7f1070198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f1077bd3700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1070105e80 0x7f10701990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f1077bd3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10701087b0 0x7f10701a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f1077bd3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f107010e550 con 0x7f1070103550 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f1077bd3700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f107010e3d0 con 0x7f1070105e80 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f1077bd3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f107010e6d0 con 0x7f10701087b0 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f107596f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1070103550 0x7f1070198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f107596f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1070103550 0x7f1070198b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40220/0 (socket says 172.21.15.50:40220) 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f107596f700 1 -- 172.21.15.50:0/3902521188 learned_addr learned my addr 172.21.15.50:0/3902521188 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f1076170700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10701087b0 0x7f10701a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f107516e700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1070105e80 0x7f10701990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f107596f700 1 -- 172.21.15.50:0/3902521188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10701087b0 msgr2=0x7f10701a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f107596f700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10701087b0 0x7f10701a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f107596f700 1 -- 172.21.15.50:0/3902521188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1070105e80 msgr2=0x7f10701990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f107596f700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1070105e80 0x7f10701990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f107596f700 1 -- 172.21.15.50:0/3902521188 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10701a07c0 con 0x7f1070103550 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f1076170700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10701087b0 0x7f10701a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f107516e700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1070105e80 0x7f10701990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.803+0000 7f107596f700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1070103550 0x7f1070198b60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f106400d980 tx=0x7f106400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.804+0000 7f1062ffd700 1 -- 172.21.15.50:0/3902521188 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1064016070 con 0x7f1070103550 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.804+0000 7f1062ffd700 1 -- 172.21.15.50:0/3902521188 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1064010460 con 0x7f1070103550 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.804+0000 7f1062ffd700 1 -- 172.21.15.50:0/3902521188 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106400f620 con 0x7f1070103550 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.804+0000 7f1077bd3700 1 -- 172.21.15.50:0/3902521188 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10701a0ab0 con 0x7f1070103550 2024-04-23T18:18:16.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.804+0000 7f1077bd3700 1 -- 172.21.15.50:0/3902521188 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10701a0ff0 con 0x7f1070103550 2024-04-23T18:18:16.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.805+0000 7f1077bd3700 1 -- 172.21.15.50:0/3902521188 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1070069730 con 0x7f1070103550 2024-04-23T18:18:16.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.806+0000 7f1062ffd700 1 -- 172.21.15.50:0/3902521188 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1064019070 con 0x7f1070103550 2024-04-23T18:18:16.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.806+0000 7f1062ffd700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f105c0a3cc0 0x7f105c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:16.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.807+0000 7f107516e700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f105c0a3cc0 0x7f105c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:16.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.807+0000 7f1062ffd700 1 -- 172.21.15.50:0/3902521188 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f10640cff90 con 0x7f1070103550 2024-04-23T18:18:16.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.807+0000 7f107516e700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f105c0a3cc0 0x7f105c0a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f106c009be0 tx=0x7f106c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:16.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:16.811+0000 7f1062ffd700 1 -- 172.21.15.50:0/3902521188 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f106409c930 con 0x7f1070103550 2024-04-23T18:18:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.021+0000 7f1077bd3700 1 -- 172.21.15.50:0/3902521188 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f1070063650 con 0x7f1070103550 2024-04-23T18:18:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.022+0000 7f1062ffd700 1 -- 172.21.15.50:0/3902521188 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v89) ==== 56+0+5295 (secure 0 0 0) 0x7f106409c230 con 0x7f1070103550 2024-04-23T18:18:17.016 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 89 2024-04-23T18:18:17.016 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:18:17.016 INFO:teuthology.orchestra.run.smithi050.stdout:created 2024-04-23T18:08:37.569945+0000 2024-04-23T18:18:17.016 INFO:teuthology.orchestra.run.smithi050.stdout:modified 2024-04-23T18:17:36.102736+0000 2024-04-23T18:18:17.016 INFO:teuthology.orchestra.run.smithi050.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-23T18:18:17.016 INFO:teuthology.orchestra.run.smithi050.stdout:crush_version 26 2024-04-23T18:18:17.016 INFO:teuthology.orchestra.run.smithi050.stdout:full_ratio 0.95 2024-04-23T18:18:17.016 INFO:teuthology.orchestra.run.smithi050.stdout:backfillfull_ratio 0.9 2024-04-23T18:18:17.016 INFO:teuthology.orchestra.run.smithi050.stdout:nearfull_ratio 0.85 2024-04-23T18:18:17.016 INFO:teuthology.orchestra.run.smithi050.stdout:require_min_compat_client luminous 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:min_compat_client luminous 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:require_osd_release squid 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:stretch_mode_enabled false 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:max_osd 12 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0 up in weight 1 up_from 8 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6802/4125602948,v1:172.21.15.50:6803/4125602948] [v2:172.21.15.50:6804/4125602948,v1:172.21.15.50:6805/4125602948] exists,up c3cb7067-1f57-4e79-a884-bfae449d5508 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6800/3726952435,v1:172.21.15.177:6801/3726952435] [v2:172.21.15.177:6802/3726952435,v1:172.21.15.177:6803/3726952435] exists,up 1b334ea9-1d7a-49b5-92bc-01416987b3ba 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2 up in weight 1 up_from 18 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6800/2182491369,v1:172.21.15.188:6801/2182491369] [v2:172.21.15.188:6802/2182491369,v1:172.21.15.188:6803/2182491369] exists,up 3a46cc56-b2f7-431d-87c3-5160a3db721d 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3 up in weight 1 up_from 25 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6810/3121149303,v1:172.21.15.50:6811/3121149303] [v2:172.21.15.50:6812/3121149303,v1:172.21.15.50:6813/3121149303] exists,up 02828390-a4c5-468a-8425-092eb1513bdd 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4 up in weight 1 up_from 33 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6808/1367742686,v1:172.21.15.177:6809/1367742686] [v2:172.21.15.177:6810/1367742686,v1:172.21.15.177:6811/1367742686] exists,up 88f4c3e4-72bc-459f-a052-6a41dd0a0e73 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5 up in weight 1 up_from 38 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6808/56520768,v1:172.21.15.188:6809/56520768] [v2:172.21.15.188:6810/56520768,v1:172.21.15.188:6811/56520768] exists,up 1fee2f23-6eec-465e-a935-970aa8ded943 2024-04-23T18:18:17.017 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6 up in weight 1 up_from 46 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6818/4290265415,v1:172.21.15.50:6819/4290265415] [v2:172.21.15.50:6820/4290265415,v1:172.21.15.50:6821/4290265415] exists,up 4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e 2024-04-23T18:18:17.018 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7 up in weight 1 up_from 52 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6816/1787147913,v1:172.21.15.177:6817/1787147913] [v2:172.21.15.177:6818/1787147913,v1:172.21.15.177:6819/1787147913] exists,up b6d24ee9-5db8-439b-907e-c5f2c7c4658a 2024-04-23T18:18:17.018 INFO:teuthology.orchestra.run.smithi050.stdout:osd.8 up in weight 1 up_from 60 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6816/4982360,v1:172.21.15.188:6817/4982360] [v2:172.21.15.188:6818/4982360,v1:172.21.15.188:6819/4982360] exists,up cb3e8deb-e2b5-479b-aed3-5decbd2b9044 2024-04-23T18:18:17.018 INFO:teuthology.orchestra.run.smithi050.stdout:osd.9 up in weight 1 up_from 68 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6826/1497125791,v1:172.21.15.50:6827/1497125791] [v2:172.21.15.50:6828/1497125791,v1:172.21.15.50:6829/1497125791] exists,up 46facc79-031d-4199-a0c0-e551f82182fa 2024-04-23T18:18:17.018 INFO:teuthology.orchestra.run.smithi050.stdout:osd.10 up in weight 1 up_from 74 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6824/1988770803,v1:172.21.15.177:6825/1988770803] [v2:172.21.15.177:6826/1988770803,v1:172.21.15.177:6827/1988770803] exists,up b7af2e9f-fa4b-43d7-9567-013393024176 2024-04-23T18:18:17.018 INFO:teuthology.orchestra.run.smithi050.stdout:osd.11 up in weight 1 up_from 82 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6824/854484853,v1:172.21.15.188:6825/854484853] [v2:172.21.15.188:6826/854484853,v1:172.21.15.188:6827/854484853] exists,up 40d6c591-e063-4b2f-ae02-58b45f6935cd 2024-04-23T18:18:17.018 INFO:teuthology.orchestra.run.smithi050.stdout:pg_upmap_items 2.5 [8,11] 2024-04-23T18:18:17.018 INFO:teuthology.orchestra.run.smithi050.stdout:pg_upmap_items 2.10 [8,11] 2024-04-23T18:18:17.018 INFO:teuthology.orchestra.run.smithi050.stdout:pg_upmap_items 2.16 [2,4] 2024-04-23T18:18:17.018 INFO:teuthology.orchestra.run.smithi050.stdout:pg_upmap_items 2.1b [2,1] 2024-04-23T18:18:17.018 INFO:teuthology.orchestra.run.smithi050.stdout:pg_upmap_items 3.0 [4,6] 2024-04-23T18:18:17.018 INFO:teuthology.orchestra.run.smithi050.stdout:pg_upmap_items 3.b [3,9] 2024-04-23T18:18:17.018 INFO:teuthology.orchestra.run.smithi050.stdout:pg_upmap_items 3.11 [5,9] 2024-04-23T18:18:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:pg_upmap_items 3.13 [4,9] 2024-04-23T18:18:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:pg_upmap_items 3.17 [4,8] 2024-04-23T18:18:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6801/103327972 expires 2024-04-24T18:09:34.403529+0000 2024-04-23T18:18:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6800/103327972 expires 2024-04-24T18:09:34.403529+0000 2024-04-23T18:18:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/1510580864 expires 2024-04-24T18:09:34.403529+0000 2024-04-23T18:18:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/587528851 expires 2024-04-24T18:09:07.548503+0000 2024-04-23T18:18:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/688464416 expires 2024-04-24T18:09:07.548503+0000 2024-04-23T18:18:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6800/1300452320 expires 2024-04-24T18:09:07.548503+0000 2024-04-23T18:18:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/1448760865 expires 2024-04-24T18:09:34.403529+0000 2024-04-23T18:18:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6801/1300452320 expires 2024-04-24T18:09:07.548503+0000 2024-04-23T18:18:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/3023139440 expires 2024-04-24T18:09:07.548503+0000 2024-04-23T18:18:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/2673950234 expires 2024-04-24T18:09:34.403529+0000 2024-04-23T18:18:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.024+0000 7f1077bd3700 1 -- 172.21.15.50:0/3902521188 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f105c0a3cc0 msgr2=0x7f105c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.024+0000 7f1077bd3700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f105c0a3cc0 0x7f105c0a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f106c009be0 tx=0x7f106c009260 comp rx=0 tx=0).stop 2024-04-23T18:18:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.024+0000 7f1077bd3700 1 -- 172.21.15.50:0/3902521188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1070103550 msgr2=0x7f1070198b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.024+0000 7f1077bd3700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1070103550 0x7f1070198b60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f106400d980 tx=0x7f106400dd40 comp rx=0 tx=0).stop 2024-04-23T18:18:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.024+0000 7f1077bd3700 1 -- 172.21.15.50:0/3902521188 shutdown_connections 2024-04-23T18:18:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.024+0000 7f1077bd3700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10701087b0 0x7f10701a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.024+0000 7f1077bd3700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1070103550 0x7f1070198b60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.024+0000 7f1077bd3700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f105c0a3cc0 0x7f105c0a6180 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.024+0000 7f1077bd3700 1 --2- 172.21.15.50:0/3902521188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1070105e80 0x7f10701990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.024+0000 7f1077bd3700 1 -- 172.21.15.50:0/3902521188 >> 172.21.15.50:0/3902521188 conn(0x7f10700fcf60 msgr2=0x7f1070103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:17.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.024+0000 7f1077bd3700 1 -- 172.21.15.50:0/3902521188 shutdown_connections 2024-04-23T18:18:17.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.024+0000 7f1077bd3700 1 -- 172.21.15.50:0/3902521188 wait complete. 2024-04-23T18:18:17.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:17 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3902521188' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T18:18:17.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:17 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3902521188' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T18:18:17.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:17 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3902521188' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T18:18:17.814 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-23T18:18:17.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:18:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 -- 172.21.15.50:0/2195366095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3d8105340 msgr2=0x7fb3d8107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 --2- 172.21.15.50:0/2195366095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3d8105340 0x7fb3d8107730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8009900 tx=0x7fb3c802a410 comp rx=0 tx=0).stop 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 -- 172.21.15.50:0/2195366095 shutdown_connections 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 --2- 172.21.15.50:0/2195366095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d8107c70 0x7fb3d810a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 --2- 172.21.15.50:0/2195366095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3d8105340 0x7fb3d8107730 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 --2- 172.21.15.50:0/2195366095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3d8102a10 0x7fb3d8104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 -- 172.21.15.50:0/2195366095 >> 172.21.15.50:0/2195366095 conn(0x7fb3d80fc400 msgr2=0x7fb3d80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 -- 172.21.15.50:0/2195366095 shutdown_connections 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 -- 172.21.15.50:0/2195366095 wait complete. 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 Processor -- start 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 -- start start 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d8102a10 0x7fb3d8102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3d8105340 0x7fb3d81009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3d8107c70 0x7fb3d8100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d810db10 con 0x7fb3d8107c70 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d810d990 con 0x7fb3d8102a10 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.942+0000 7fb3e0431640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d810dc90 con 0x7fb3d8105340 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3de9ce640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3d8107c70 0x7fb3d8100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3de9ce640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3d8107c70 0x7fb3d8100f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40238/0 (socket says 172.21.15.50:40238) 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3de9ce640 1 -- 172.21.15.50:0/31986755 learned_addr learned my addr 172.21.15.50:0/31986755 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3de1cd640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d8102a10 0x7fb3d8102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3dd9cc640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3d8105340 0x7fb3d81009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3de9ce640 1 -- 172.21.15.50:0/31986755 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3d8105340 msgr2=0x7fb3d81009c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3de9ce640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3d8105340 0x7fb3d81009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3de9ce640 1 -- 172.21.15.50:0/31986755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d8102a10 msgr2=0x7fb3d8102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3de9ce640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d8102a10 0x7fb3d8102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3de9ce640 1 -- 172.21.15.50:0/31986755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d8101790 con 0x7fb3d8107c70 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3de1cd640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d8102a10 0x7fb3d8102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3de9ce640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3d8107c70 0x7fb3d8100f00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb3d000eb10 tx=0x7fb3d000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3cf7fe640 1 -- 172.21.15.50:0/31986755 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0004b20 con 0x7fb3d8107c70 2024-04-23T18:18:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3cf7fe640 1 -- 172.21.15.50:0/31986755 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3d0004cc0 con 0x7fb3d8107c70 2024-04-23T18:18:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3e0431640 1 -- 172.21.15.50:0/31986755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d81a2310 con 0x7fb3d8107c70 2024-04-23T18:18:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3cf7fe640 1 -- 172.21.15.50:0/31986755 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0010670 con 0x7fb3d8107c70 2024-04-23T18:18:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.943+0000 7fb3e0431640 1 -- 172.21.15.50:0/31986755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d81a2810 con 0x7fb3d8107c70 2024-04-23T18:18:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.944+0000 7fb3e0431640 1 -- 172.21.15.50:0/31986755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3d80fdef0 con 0x7fb3d8107c70 2024-04-23T18:18:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.946+0000 7fb3cf7fe640 1 -- 172.21.15.50:0/31986755 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb3d0005020 con 0x7fb3d8107c70 2024-04-23T18:18:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.946+0000 7fb3cf7fe640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3c40a3bb0 0x7fb3c40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.947+0000 7fb3de1cd640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3c40a3bb0 0x7fb3c40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.947+0000 7fb3cf7fe640 1 -- 172.21.15.50:0/31986755 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fb3d00d0b70 con 0x7fb3d8107c70 2024-04-23T18:18:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.947+0000 7fb3de1cd640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3c40a3bb0 0x7fb3c40a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009b50 tx=0x7fb3d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:17.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:17.951+0000 7fb3cf7fe640 1 -- 172.21.15.50:0/31986755 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3d009d550 con 0x7fb3d8107c70 2024-04-23T18:18:18.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.143+0000 7fb3e0431640 1 -- 172.21.15.50:0/31986755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb3d8062b50 con 0x7fb3d8107c70 2024-04-23T18:18:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.145+0000 7fb3cf7fe640 1 -- 172.21.15.50:0/31986755 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22661 (secure 0 0 0) 0x7fb3d009ce50 con 0x7fb3d8107c70 2024-04-23T18:18:18.139 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:18.139 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":89,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:17:36.102736+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:18:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.147+0000 7fb3e0431640 1 -- 172.21.15.50:0/31986755 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3c40a3bb0 msgr2=0x7fb3c40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.147+0000 7fb3e0431640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3c40a3bb0 0x7fb3c40a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009b50 tx=0x7fb3d4005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.147+0000 7fb3e0431640 1 -- 172.21.15.50:0/31986755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3d8107c70 msgr2=0x7fb3d8100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.147+0000 7fb3e0431640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3d8107c70 0x7fb3d8100f00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb3d000eb10 tx=0x7fb3d000c2d0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.147+0000 7fb3e0431640 1 -- 172.21.15.50:0/31986755 shutdown_connections 2024-04-23T18:18:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.147+0000 7fb3e0431640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3c40a3bb0 0x7fb3c40a6070 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.147+0000 7fb3e0431640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3d8107c70 0x7fb3d8100f00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.147+0000 7fb3e0431640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb3d8105340 0x7fb3d81009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.147+0000 7fb3e0431640 1 --2- 172.21.15.50:0/31986755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d8102a10 0x7fb3d8102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.147+0000 7fb3e0431640 1 -- 172.21.15.50:0/31986755 >> 172.21.15.50:0/31986755 conn(0x7fb3d80fc400 msgr2=0x7fb3d81034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.147+0000 7fb3e0431640 1 -- 172.21.15.50:0/31986755 shutdown_connections 2024-04-23T18:18:18.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.147+0000 7fb3e0431640 1 -- 172.21.15.50:0/31986755 wait complete. 2024-04-23T18:18:18.152 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:12:02.493647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:16:31.679148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:16:32.269478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:18:18.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:18:18.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.264+0000 7f7405f7d640 1 -- 172.21.15.50:0/3119087078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74001029b0 msgr2=0x7f74001092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.264+0000 7f7405f7d640 1 --2- 172.21.15.50:0/3119087078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74001029b0 0x7f74001092d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009ca0 tx=0x7f73f802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.264+0000 7f7405f7d640 1 -- 172.21.15.50:0/3119087078 shutdown_connections 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.264+0000 7f7405f7d640 1 --2- 172.21.15.50:0/3119087078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74001029b0 0x7f74001092d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.264+0000 7f7405f7d640 1 --2- 172.21.15.50:0/3119087078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7400101ff0 0x7f7400102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.264+0000 7f7405f7d640 1 --2- 172.21.15.50:0/3119087078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7400100ea0 0x7f74001012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.264+0000 7f7405f7d640 1 -- 172.21.15.50:0/3119087078 >> 172.21.15.50:0/3119087078 conn(0x7f74000fc410 msgr2=0x7f74000fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.264+0000 7f7405f7d640 1 -- 172.21.15.50:0/3119087078 shutdown_connections 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.264+0000 7f7405f7d640 1 -- 172.21.15.50:0/3119087078 wait complete. 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.264+0000 7f7405f7d640 1 Processor -- start 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.264+0000 7f7405f7d640 1 -- start start 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f7405f7d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7400100ea0 0x7f74001981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f7405f7d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400101ff0 0x7f74001986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f7405f7d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74001029b0 0x7f740019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f7405f7d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f740010dba0 con 0x7f7400101ff0 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f7405f7d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f740010da20 con 0x7f74001029b0 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f7405f7d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f740010dd20 con 0x7f7400100ea0 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73feffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400101ff0 0x7f74001986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73feffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400101ff0 0x7f74001986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40264/0 (socket says 172.21.15.50:40264) 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73feffd640 1 -- 172.21.15.50:0/2035986784 learned_addr learned my addr 172.21.15.50:0/2035986784 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73ff7fe640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7400100ea0 0x7f74001981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73fffff640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74001029b0 0x7f740019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73feffd640 1 -- 172.21.15.50:0/2035986784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7400100ea0 msgr2=0x7f74001981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73feffd640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7400100ea0 0x7f74001981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73feffd640 1 -- 172.21.15.50:0/2035986784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74001029b0 msgr2=0x7f740019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73feffd640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74001029b0 0x7f740019f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73feffd640 1 -- 172.21.15.50:0/2035986784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f740019fdd0 con 0x7f7400101ff0 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73feffd640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400101ff0 0x7f74001986f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f73f400d770 tx=0x7f73f400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:18.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73fcff9640 1 -- 172.21.15.50:0/2035986784 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4017070 con 0x7f7400101ff0 2024-04-23T18:18:18.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73fcff9640 1 -- 172.21.15.50:0/2035986784 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73f4009c30 con 0x7f7400101ff0 2024-04-23T18:18:18.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f7405f7d640 1 -- 172.21.15.50:0/2035986784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74001a00c0 con 0x7f7400101ff0 2024-04-23T18:18:18.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f73fcff9640 1 -- 172.21.15.50:0/2035986784 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4003cc0 con 0x7f7400101ff0 2024-04-23T18:18:18.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.265+0000 7f7405f7d640 1 -- 172.21.15.50:0/2035986784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74001a0610 con 0x7f7400101ff0 2024-04-23T18:18:18.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.267+0000 7f7405f7d640 1 -- 172.21.15.50:0/2035986784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7400108230 con 0x7f7400101ff0 2024-04-23T18:18:18.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.268+0000 7f73fcff9640 1 -- 172.21.15.50:0/2035986784 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f73f4003e60 con 0x7f7400101ff0 2024-04-23T18:18:18.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.268+0000 7f73fcff9640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f73e80a3bb0 0x7f73e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.268+0000 7f73ff7fe640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f73e80a3bb0 0x7f73e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.268+0000 7f73fcff9640 1 -- 172.21.15.50:0/2035986784 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f73f40cfe40 con 0x7f7400101ff0 2024-04-23T18:18:18.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.269+0000 7f73ff7fe640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f73e80a3bb0 0x7f73e80a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009b60 tx=0x7f73f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:18.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.273+0000 7f73fcff9640 1 -- 172.21.15.50:0/2035986784 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f73f409c820 con 0x7f7400101ff0 2024-04-23T18:18:18.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:18 smithi050 ceph-mon[29187]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-23T18:18:18.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.459+0000 7f7405f7d640 1 -- 172.21.15.50:0/2035986784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7400062b80 con 0x7f7400101ff0 2024-04-23T18:18:18.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.460+0000 7f73fcff9640 1 -- 172.21.15.50:0/2035986784 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f73f4009de0 con 0x7f7400101ff0 2024-04-23T18:18:18.453 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-23T18:18:18.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.461+0000 7f7405f7d640 1 -- 172.21.15.50:0/2035986784 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f73e80a3bb0 msgr2=0x7f73e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.461+0000 7f7405f7d640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f73e80a3bb0 0x7f73e80a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009b60 tx=0x7f73f0005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:18.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.461+0000 7f7405f7d640 1 -- 172.21.15.50:0/2035986784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400101ff0 msgr2=0x7f74001986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.461+0000 7f7405f7d640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400101ff0 0x7f74001986f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f73f400d770 tx=0x7f73f400dc30 comp rx=0 tx=0).stop 2024-04-23T18:18:18.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.462+0000 7f7405f7d640 1 -- 172.21.15.50:0/2035986784 shutdown_connections 2024-04-23T18:18:18.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.462+0000 7f7405f7d640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f73e80a3bb0 0x7f73e80a6070 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.462+0000 7f7405f7d640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74001029b0 0x7f740019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.462+0000 7f7405f7d640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400101ff0 0x7f74001986f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.462+0000 7f7405f7d640 1 --2- 172.21.15.50:0/2035986784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7400100ea0 0x7f74001981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.462+0000 7f7405f7d640 1 -- 172.21.15.50:0/2035986784 >> 172.21.15.50:0/2035986784 conn(0x7f74000fc410 msgr2=0x7f7400105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:18.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.462+0000 7f7405f7d640 1 -- 172.21.15.50:0/2035986784 shutdown_connections 2024-04-23T18:18:18.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.462+0000 7f7405f7d640 1 -- 172.21.15.50:0/2035986784 wait complete. 2024-04-23T18:18:18.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:18:18.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:18 smithi177 ceph-mon[32308]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-23T18:18:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 -- 172.21.15.50:0/1359739441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc100e70 msgr2=0x7f6adc101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 --2- 172.21.15.50:0/1359739441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc100e70 0x7f6adc101290 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009900 tx=0x7f6ad802a410 comp rx=0 tx=0).stop 2024-04-23T18:18:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 -- 172.21.15.50:0/1359739441 shutdown_connections 2024-04-23T18:18:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 --2- 172.21.15.50:0/1359739441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6adc102980 0x7f6adc1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 --2- 172.21.15.50:0/1359739441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6adc101fc0 0x7f6adc102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 --2- 172.21.15.50:0/1359739441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc100e70 0x7f6adc101290 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 -- 172.21.15.50:0/1359739441 >> 172.21.15.50:0/1359739441 conn(0x7f6adc0fc3e0 msgr2=0x7f6adc0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 -- 172.21.15.50:0/1359739441 shutdown_connections 2024-04-23T18:18:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 -- 172.21.15.50:0/1359739441 wait complete. 2024-04-23T18:18:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 Processor -- start 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 -- start start 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6adc100e70 0x7f6adc1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6adc101fc0 0x7f6adc198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.578+0000 7f6ae4556640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc102980 0x7f6adc19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae4556640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc10db40 con 0x7f6adc102980 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae4556640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc10d9c0 con 0x7f6adc100e70 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae4556640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc10dcc0 con 0x7f6adc101fc0 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae2af3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc102980 0x7f6adc19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae2af3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc102980 0x7f6adc19f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40292/0 (socket says 172.21.15.50:40292) 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae2af3640 1 -- 172.21.15.50:0/1803080668 learned_addr learned my addr 172.21.15.50:0/1803080668 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae22f2640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6adc100e70 0x7f6adc1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae1af1640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6adc101fc0 0x7f6adc198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae2af3640 1 -- 172.21.15.50:0/1803080668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6adc101fc0 msgr2=0x7f6adc198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae2af3640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6adc101fc0 0x7f6adc198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae2af3640 1 -- 172.21.15.50:0/1803080668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6adc100e70 msgr2=0x7f6adc1981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae2af3640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6adc100e70 0x7f6adc1981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae2af3640 1 -- 172.21.15.50:0/1803080668 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6adc19fe00 con 0x7f6adc102980 2024-04-23T18:18:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae22f2640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6adc100e70 0x7f6adc1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae2af3640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc102980 0x7f6adc19f790 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400cce0 tx=0x7f6ad400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ad37fe640 1 -- 172.21.15.50:0/1803080668 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad400f450 con 0x7f6adc102980 2024-04-23T18:18:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ad37fe640 1 -- 172.21.15.50:0/1803080668 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ad400faa0 con 0x7f6adc102980 2024-04-23T18:18:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae4556640 1 -- 172.21.15.50:0/1803080668 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6adc1a00f0 con 0x7f6adc102980 2024-04-23T18:18:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ad37fe640 1 -- 172.21.15.50:0/1803080668 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4018b70 con 0x7f6adc102980 2024-04-23T18:18:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.579+0000 7f6ae4556640 1 -- 172.21.15.50:0/1803080668 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6adc1a0670 con 0x7f6adc102980 2024-04-23T18:18:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.580+0000 7f6ae4556640 1 -- 172.21.15.50:0/1803080668 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6adc108180 con 0x7f6adc102980 2024-04-23T18:18:18.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.581+0000 7f6ad37fe640 1 -- 172.21.15.50:0/1803080668 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6ad4018d10 con 0x7f6adc102980 2024-04-23T18:18:18.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.582+0000 7f6ad37fe640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ac80a3bb0 0x7f6ac80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.582+0000 7f6ae22f2640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ac80a3bb0 0x7f6ac80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.582+0000 7f6ad37fe640 1 -- 172.21.15.50:0/1803080668 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f6ad40d08f0 con 0x7f6adc102980 2024-04-23T18:18:18.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.582+0000 7f6ae22f2640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ac80a3bb0 0x7f6ac80a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8000c00 tx=0x7f6ad80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:18.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:18 smithi188 ceph-mon[32694]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-23T18:18:18.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.587+0000 7f6ad37fe640 1 -- 172.21.15.50:0/1803080668 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ad4014030 con 0x7f6adc102980 2024-04-23T18:18:18.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.772+0000 7f6ae4556640 1 -- 172.21.15.50:0/1803080668 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6adc062b50 con 0x7f6adc102980 2024-04-23T18:18:18.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.773+0000 7f6ad37fe640 1 -- 172.21.15.50:0/1803080668 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f6ad409d2d0 con 0x7f6adc102980 2024-04-23T18:18:18.766 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2024-04-23T18:18:18.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.775+0000 7f6ae4556640 1 -- 172.21.15.50:0/1803080668 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ac80a3bb0 msgr2=0x7f6ac80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.775+0000 7f6ae4556640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ac80a3bb0 0x7f6ac80a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8000c00 tx=0x7f6ad80314a0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.775+0000 7f6ae4556640 1 -- 172.21.15.50:0/1803080668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc102980 msgr2=0x7f6adc19f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.775+0000 7f6ae4556640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc102980 0x7f6adc19f790 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400cce0 tx=0x7f6ad400a620 comp rx=0 tx=0).stop 2024-04-23T18:18:18.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.775+0000 7f6ae4556640 1 -- 172.21.15.50:0/1803080668 shutdown_connections 2024-04-23T18:18:18.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.775+0000 7f6ae4556640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f6ac80a3bb0 0x7f6ac80a6070 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.775+0000 7f6ae4556640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc102980 0x7f6adc19f790 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.775+0000 7f6ae4556640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6adc101fc0 0x7f6adc198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.775+0000 7f6ae4556640 1 --2- 172.21.15.50:0/1803080668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6adc100e70 0x7f6adc1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.775+0000 7f6ae4556640 1 -- 172.21.15.50:0/1803080668 >> 172.21.15.50:0/1803080668 conn(0x7f6adc0fc3e0 msgr2=0x7f6adc105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:18.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.775+0000 7f6ae4556640 1 -- 172.21.15.50:0/1803080668 shutdown_connections 2024-04-23T18:18:18.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.775+0000 7f6ae4556640 1 -- 172.21.15.50:0/1803080668 wait complete. 2024-04-23T18:18:18.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:18:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.890+0000 7faf185f7640 1 -- 172.21.15.50:0/888894184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf10105340 msgr2=0x7faf10107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.890+0000 7faf185f7640 1 --2- 172.21.15.50:0/888894184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf10105340 0x7faf10107730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7faf04009900 tx=0x7faf0402a410 comp rx=0 tx=0).stop 2024-04-23T18:18:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.890+0000 7faf185f7640 1 -- 172.21.15.50:0/888894184 shutdown_connections 2024-04-23T18:18:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.890+0000 7faf185f7640 1 --2- 172.21.15.50:0/888894184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf10107c70 0x7faf1010a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.890+0000 7faf185f7640 1 --2- 172.21.15.50:0/888894184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf10105340 0x7faf10107730 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.890+0000 7faf185f7640 1 --2- 172.21.15.50:0/888894184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf10102a10 0x7faf10104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.890+0000 7faf185f7640 1 -- 172.21.15.50:0/888894184 >> 172.21.15.50:0/888894184 conn(0x7faf100fc400 msgr2=0x7faf100fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.890+0000 7faf185f7640 1 -- 172.21.15.50:0/888894184 shutdown_connections 2024-04-23T18:18:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.890+0000 7faf185f7640 1 -- 172.21.15.50:0/888894184 wait complete. 2024-04-23T18:18:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.891+0000 7faf185f7640 1 Processor -- start 2024-04-23T18:18:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.891+0000 7faf185f7640 1 -- start start 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.891+0000 7faf185f7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf10102a10 0x7faf1019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.891+0000 7faf185f7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf10105340 0x7faf1019a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.891+0000 7faf185f7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf10107c70 0x7faf101a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.891+0000 7faf185f7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faf1010dac0 con 0x7faf10105340 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.891+0000 7faf185f7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faf1010d940 con 0x7faf10102a10 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.891+0000 7faf185f7640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faf1010dc40 con 0x7faf10107c70 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.891+0000 7faf15b92640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf10105340 0x7faf1019a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.891+0000 7faf15b92640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf10105340 0x7faf1019a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40308/0 (socket says 172.21.15.50:40308) 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.891+0000 7faf15b92640 1 -- 172.21.15.50:0/1934850172 learned_addr learned my addr 172.21.15.50:0/1934850172 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.891+0000 7faf16b94640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf10107c70 0x7faf101a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf16393640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf10102a10 0x7faf1019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf15b92640 1 -- 172.21.15.50:0/1934850172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf10107c70 msgr2=0x7faf101a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf15b92640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf10107c70 0x7faf101a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf15b92640 1 -- 172.21.15.50:0/1934850172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf10102a10 msgr2=0x7faf1019a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf15b92640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf10102a10 0x7faf1019a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf15b92640 1 -- 172.21.15.50:0/1934850172 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf101a2000 con 0x7faf10105340 2024-04-23T18:18:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf16b94640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf10107c70 0x7faf101a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf16393640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf10102a10 0x7faf1019a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf15b92640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf10105340 0x7faf1019a920 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faf04006010 tx=0x7faf04004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf0f7fe640 1 -- 172.21.15.50:0/1934850172 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0403f030 con 0x7faf10105340 2024-04-23T18:18:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf0f7fe640 1 -- 172.21.15.50:0/1934850172 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf04003cb0 con 0x7faf10105340 2024-04-23T18:18:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf185f7640 1 -- 172.21.15.50:0/1934850172 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf101a2290 con 0x7faf10105340 2024-04-23T18:18:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf0f7fe640 1 -- 172.21.15.50:0/1934850172 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf04035070 con 0x7faf10105340 2024-04-23T18:18:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.892+0000 7faf185f7640 1 -- 172.21.15.50:0/1934850172 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf101a27d0 con 0x7faf10105340 2024-04-23T18:18:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.893+0000 7faf185f7640 1 -- 172.21.15.50:0/1934850172 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf101097c0 con 0x7faf10105340 2024-04-23T18:18:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.894+0000 7faf0f7fe640 1 -- 172.21.15.50:0/1934850172 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7faf04042030 con 0x7faf10105340 2024-04-23T18:18:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.895+0000 7faf0f7fe640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faef80a3c30 0x7faef80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.895+0000 7faf16393640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faef80a3c30 0x7faef80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.895+0000 7faf0f7fe640 1 -- 172.21.15.50:0/1934850172 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7faf040f0b80 con 0x7faf10105340 2024-04-23T18:18:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.895+0000 7faf16393640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faef80a3c30 0x7faef80a60f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faf080058c0 tx=0x7faf08005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:18.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:18.900+0000 7faf0f7fe640 1 -- 172.21.15.50:0/1934850172 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf04003e60 con 0x7faf10105340 2024-04-23T18:18:19.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.089+0000 7faf185f7640 1 -- 172.21.15.50:0/1934850172 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faf10062b50 con 0x7faf10105340 2024-04-23T18:18:19.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.089+0000 7faf0f7fe640 1 -- 172.21.15.50:0/1934850172 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7faf040bd520 con 0x7faf10105340 2024-04-23T18:18:19.083 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2024-04-23T18:18:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.091+0000 7faf185f7640 1 -- 172.21.15.50:0/1934850172 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faef80a3c30 msgr2=0x7faef80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.091+0000 7faf185f7640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faef80a3c30 0x7faef80a60f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faf080058c0 tx=0x7faf08005d30 comp rx=0 tx=0).stop 2024-04-23T18:18:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.091+0000 7faf185f7640 1 -- 172.21.15.50:0/1934850172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf10105340 msgr2=0x7faf1019a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.091+0000 7faf185f7640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf10105340 0x7faf1019a920 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faf04006010 tx=0x7faf04004430 comp rx=0 tx=0).stop 2024-04-23T18:18:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.091+0000 7faf185f7640 1 -- 172.21.15.50:0/1934850172 shutdown_connections 2024-04-23T18:18:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.092+0000 7faf185f7640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7faef80a3c30 0x7faef80a60f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.092+0000 7faf185f7640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf10107c70 0x7faf101a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.092+0000 7faf185f7640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf10105340 0x7faf1019a920 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.092+0000 7faf185f7640 1 --2- 172.21.15.50:0/1934850172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf10102a10 0x7faf1019a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.092+0000 7faf185f7640 1 -- 172.21.15.50:0/1934850172 >> 172.21.15.50:0/1934850172 conn(0x7faf100fc400 msgr2=0x7faf10102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.092+0000 7faf185f7640 1 -- 172.21.15.50:0/1934850172 shutdown_connections 2024-04-23T18:18:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.092+0000 7faf185f7640 1 -- 172.21.15.50:0/1934850172 wait complete. 2024-04-23T18:18:19.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:18:19.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.205+0000 7f6539bbf640 1 -- 172.21.15.50:0/1757315324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65341053c0 msgr2=0x7f6534105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.205+0000 7f6539bbf640 1 --2- 172.21.15.50:0/1757315324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65341053c0 0x7f6534105840 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f651c009900 tx=0x7f651c02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:19.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.205+0000 7f6539bbf640 1 -- 172.21.15.50:0/1757315324 shutdown_connections 2024-04-23T18:18:19.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.205+0000 7f6539bbf640 1 --2- 172.21.15.50:0/1757315324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6534105d80 0x7f653410c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.205+0000 7f6539bbf640 1 --2- 172.21.15.50:0/1757315324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65341053c0 0x7f6534105840 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.205+0000 7f6539bbf640 1 --2- 172.21.15.50:0/1757315324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65341041d0 0x7f65341045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.205+0000 7f6539bbf640 1 -- 172.21.15.50:0/1757315324 >> 172.21.15.50:0/1757315324 conn(0x7f65340ff740 msgr2=0x7f6534101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.205+0000 7f6539bbf640 1 -- 172.21.15.50:0/1757315324 shutdown_connections 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.205+0000 7f6539bbf640 1 -- 172.21.15.50:0/1757315324 wait complete. 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.206+0000 7f6539bbf640 1 Processor -- start 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.206+0000 7f6539bbf640 1 -- start start 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.206+0000 7f6539bbf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65341041d0 0x7f653419a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.206+0000 7f6539bbf640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65341053c0 0x7f653419a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.206+0000 7f6539bbf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534105d80 0x7f65341a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.206+0000 7f6539bbf640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f653410fd60 con 0x7f6534105d80 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.206+0000 7f6539bbf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f653410fbe0 con 0x7f65341041d0 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.206+0000 7f6539bbf640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f653410fee0 con 0x7f65341053c0 2024-04-23T18:18:19.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.206+0000 7f6533fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534105d80 0x7f65341a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.206+0000 7f6533fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534105d80 0x7f65341a19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40330/0 (socket says 172.21.15.50:40330) 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.206+0000 7f6533fff640 1 -- 172.21.15.50:0/692506020 learned_addr learned my addr 172.21.15.50:0/692506020 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.206+0000 7f6532ffd640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65341053c0 0x7f653419a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.207+0000 7f65337fe640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65341041d0 0x7f653419a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.207+0000 7f6533fff640 1 -- 172.21.15.50:0/692506020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65341053c0 msgr2=0x7f653419a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.207+0000 7f6533fff640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65341053c0 0x7f653419a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.207+0000 7f6533fff640 1 -- 172.21.15.50:0/692506020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65341041d0 msgr2=0x7f653419a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.207+0000 7f6533fff640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65341041d0 0x7f653419a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.207+0000 7f6533fff640 1 -- 172.21.15.50:0/692506020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65341a2010 con 0x7f6534105d80 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.207+0000 7f6533fff640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534105d80 0x7f65341a19a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f652c00e970 tx=0x7f652c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.207+0000 7f6530ff9640 1 -- 172.21.15.50:0/692506020 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c00ccc0 con 0x7f6534105d80 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.207+0000 7f6530ff9640 1 -- 172.21.15.50:0/692506020 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f652c00ce60 con 0x7f6534105d80 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.207+0000 7f6539bbf640 1 -- 172.21.15.50:0/692506020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65341a2300 con 0x7f6534105d80 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.207+0000 7f6530ff9640 1 -- 172.21.15.50:0/692506020 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c010430 con 0x7f6534105d80 2024-04-23T18:18:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.207+0000 7f6539bbf640 1 -- 172.21.15.50:0/692506020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65341a2820 con 0x7f6534105d80 2024-04-23T18:18:19.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.208+0000 7f6539bbf640 1 -- 172.21.15.50:0/692506020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f653410b4f0 con 0x7f6534105d80 2024-04-23T18:18:19.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.209+0000 7f6530ff9640 1 -- 172.21.15.50:0/692506020 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f652c004f70 con 0x7f6534105d80 2024-04-23T18:18:19.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.210+0000 7f6530ff9640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f65200a3bb0 0x7f65200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:19.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.210+0000 7f65337fe640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f65200a3bb0 0x7f65200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:19.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.210+0000 7f6530ff9640 1 -- 172.21.15.50:0/692506020 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f652c0d0a70 con 0x7f6534105d80 2024-04-23T18:18:19.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.210+0000 7f65337fe640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f65200a3bb0 0x7f65200a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6534105220 tx=0x7f6524008230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:19.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.213+0000 7f6530ff9640 1 -- 172.21.15.50:0/692506020 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f652c053ff0 con 0x7f6534105d80 2024-04-23T18:18:19.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.444+0000 7f6539bbf640 1 -- 172.21.15.50:0/692506020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f653402d5c0 con 0x7f6534105d80 2024-04-23T18:18:19.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.445+0000 7f6530ff9640 1 -- 172.21.15.50:0/692506020 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11875 (secure 0 0 0) 0x7f652c09d410 con 0x7f6534105d80 2024-04-23T18:18:19.439 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:19.439 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:16:37.965071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:18:19.439 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 20 2024-04-23T18:18:19.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.447+0000 7f6539bbf640 1 -- 172.21.15.50:0/692506020 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f65200a3bb0 msgr2=0x7f65200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.447+0000 7f6539bbf640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f65200a3bb0 0x7f65200a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6534105220 tx=0x7f6524008230 comp rx=0 tx=0).stop 2024-04-23T18:18:19.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.447+0000 7f6539bbf640 1 -- 172.21.15.50:0/692506020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534105d80 msgr2=0x7f65341a19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.447+0000 7f6539bbf640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534105d80 0x7f65341a19a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f652c00e970 tx=0x7f652c00ed70 comp rx=0 tx=0).stop 2024-04-23T18:18:19.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.447+0000 7f6539bbf640 1 -- 172.21.15.50:0/692506020 shutdown_connections 2024-04-23T18:18:19.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.447+0000 7f6539bbf640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f65200a3bb0 0x7f65200a6070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.447+0000 7f6539bbf640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534105d80 0x7f65341a19a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.447+0000 7f6539bbf640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65341053c0 0x7f653419a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.447+0000 7f6539bbf640 1 --2- 172.21.15.50:0/692506020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65341041d0 0x7f653419a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.447+0000 7f6539bbf640 1 -- 172.21.15.50:0/692506020 >> 172.21.15.50:0/692506020 conn(0x7f65340ff740 msgr2=0x7f65341086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:19.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.447+0000 7f6539bbf640 1 -- 172.21.15.50:0/692506020 shutdown_connections 2024-04-23T18:18:19.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.447+0000 7f6539bbf640 1 -- 172.21.15.50:0/692506020 wait complete. 2024-04-23T18:18:19.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:19 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/31986755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:19.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:19 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2035986784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:19.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:19 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1803080668' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:19.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:19 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1934850172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:19.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:18:19.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:19 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/31986755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:19.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:19 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2035986784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:19.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:19 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1803080668' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:19.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:19 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1934850172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:19.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.560+0000 7fc932efa640 1 -- 172.21.15.50:0/3315403950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92c105360 msgr2=0x7fc92c107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.560+0000 7fc932efa640 1 --2- 172.21.15.50:0/3315403950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92c105360 0x7fc92c107750 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc920009900 tx=0x7fc92002a410 comp rx=0 tx=0).stop 2024-04-23T18:18:19.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.560+0000 7fc932efa640 1 -- 172.21.15.50:0/3315403950 shutdown_connections 2024-04-23T18:18:19.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.560+0000 7fc932efa640 1 --2- 172.21.15.50:0/3315403950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c107c90 0x7fc92c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.560+0000 7fc932efa640 1 --2- 172.21.15.50:0/3315403950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92c105360 0x7fc92c107750 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.560+0000 7fc932efa640 1 --2- 172.21.15.50:0/3315403950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc92c102a30 0x7fc92c104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.560+0000 7fc932efa640 1 -- 172.21.15.50:0/3315403950 >> 172.21.15.50:0/3315403950 conn(0x7fc92c0fc440 msgr2=0x7fc92c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:19.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.560+0000 7fc932efa640 1 -- 172.21.15.50:0/3315403950 shutdown_connections 2024-04-23T18:18:19.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.560+0000 7fc932efa640 1 -- 172.21.15.50:0/3315403950 wait complete. 2024-04-23T18:18:19.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc932efa640 1 Processor -- start 2024-04-23T18:18:19.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc932efa640 1 -- start start 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc932efa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c102a30 0x7fc92c198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc932efa640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc92c105360 0x7fc92c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc932efa640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92c107c90 0x7fc92c19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc932efa640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c10db40 con 0x7fc92c107c90 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc932efa640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c10d9c0 con 0x7fc92c102a30 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc932efa640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c10dcc0 con 0x7fc92c105360 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc931497640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92c107c90 0x7fc92c19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc931497640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92c107c90 0x7fc92c19f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40336/0 (socket says 172.21.15.50:40336) 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc931497640 1 -- 172.21.15.50:0/1673259065 learned_addr learned my addr 172.21.15.50:0/1673259065 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc930c96640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c102a30 0x7fc92c198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc92bfff640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc92c105360 0x7fc92c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc931497640 1 -- 172.21.15.50:0/1673259065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc92c105360 msgr2=0x7fc92c1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc931497640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc92c105360 0x7fc92c1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc931497640 1 -- 172.21.15.50:0/1673259065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c102a30 msgr2=0x7fc92c198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc931497640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c102a30 0x7fc92c198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.561+0000 7fc931497640 1 -- 172.21.15.50:0/1673259065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c19fe90 con 0x7fc92c107c90 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.562+0000 7fc931497640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92c107c90 0x7fc92c19f820 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc924009fd0 tx=0x7fc92400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.562+0000 7fc929ffb640 1 -- 172.21.15.50:0/1673259065 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc924017070 con 0x7fc92c107c90 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.562+0000 7fc929ffb640 1 -- 172.21.15.50:0/1673259065 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc924003e20 con 0x7fc92c107c90 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.562+0000 7fc932efa640 1 -- 172.21.15.50:0/1673259065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc92c1a0180 con 0x7fc92c107c90 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.562+0000 7fc929ffb640 1 -- 172.21.15.50:0/1673259065 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc924010460 con 0x7fc92c107c90 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.562+0000 7fc932efa640 1 -- 172.21.15.50:0/1673259065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc92c1a0650 con 0x7fc92c107c90 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.563+0000 7fc932efa640 1 -- 172.21.15.50:0/1673259065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc92c0fdf30 con 0x7fc92c107c90 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.564+0000 7fc929ffb640 1 -- 172.21.15.50:0/1673259065 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc924003970 con 0x7fc92c107c90 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.564+0000 7fc929ffb640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc9140a3bb0 0x7fc9140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:19.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.565+0000 7fc930c96640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc9140a3bb0 0x7fc9140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:19.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.565+0000 7fc929ffb640 1 -- 172.21.15.50:0/1673259065 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fc9240d0200 con 0x7fc92c107c90 2024-04-23T18:18:19.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.565+0000 7fc930c96640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc9140a3bb0 0x7fc9140a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc91c009bd0 tx=0x7fc91c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:19.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.569+0000 7fc929ffb640 1 -- 172.21.15.50:0/1673259065 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc92409cba0 con 0x7fc92c107c90 2024-04-23T18:18:19.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:19 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/31986755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:19.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:19 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2035986784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:19.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:19 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1803080668' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:19.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:19 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1934850172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.756+0000 7fc932efa640 1 -- 172.21.15.50:0/1673259065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc92c062b50 con 0x7fc92c107c90 2024-04-23T18:18:19.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.758+0000 7fc929ffb640 1 -- 172.21.15.50:0/1673259065 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22661 (secure 0 0 0) 0x7fc92409c4a0 con 0x7fc92c107c90 2024-04-23T18:18:19.752 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:19.752 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":89,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:17:36.102736+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:18:19.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.760+0000 7fc932efa640 1 -- 172.21.15.50:0/1673259065 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc9140a3bb0 msgr2=0x7fc9140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.760+0000 7fc932efa640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc9140a3bb0 0x7fc9140a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc91c009bd0 tx=0x7fc91c005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:19.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.760+0000 7fc932efa640 1 -- 172.21.15.50:0/1673259065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92c107c90 msgr2=0x7fc92c19f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.760+0000 7fc932efa640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92c107c90 0x7fc92c19f820 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc924009fd0 tx=0x7fc92400b260 comp rx=0 tx=0).stop 2024-04-23T18:18:19.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.760+0000 7fc932efa640 1 -- 172.21.15.50:0/1673259065 shutdown_connections 2024-04-23T18:18:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.760+0000 7fc932efa640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc9140a3bb0 0x7fc9140a6070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.760+0000 7fc932efa640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92c107c90 0x7fc92c19f820 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.760+0000 7fc932efa640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc92c105360 0x7fc92c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.760+0000 7fc932efa640 1 --2- 172.21.15.50:0/1673259065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c102a30 0x7fc92c198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.760+0000 7fc932efa640 1 -- 172.21.15.50:0/1673259065 >> 172.21.15.50:0/1673259065 conn(0x7fc92c0fc440 msgr2=0x7fc92c1032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.760+0000 7fc932efa640 1 -- 172.21.15.50:0/1673259065 shutdown_connections 2024-04-23T18:18:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.760+0000 7fc932efa640 1 -- 172.21.15.50:0/1673259065 wait complete. 2024-04-23T18:18:19.765 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:12:02.493647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:16:31.679148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:16:32.269478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:18:19.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:18:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.877+0000 7f73b26e1640 1 -- 172.21.15.50:0/2335371301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73ac103cc0 msgr2=0x7f73ac10a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.877+0000 7f73b26e1640 1 --2- 172.21.15.50:0/2335371301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73ac103cc0 0x7f73ac10a550 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009cd0 tx=0x7f73a402a8e0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.877+0000 7f73b26e1640 1 -- 172.21.15.50:0/2335371301 shutdown_connections 2024-04-23T18:18:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.877+0000 7f73b26e1640 1 --2- 172.21.15.50:0/2335371301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73ac103cc0 0x7f73ac10a550 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.877+0000 7f73b26e1640 1 --2- 172.21.15.50:0/2335371301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73ac103300 0x7f73ac103780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.877+0000 7f73b26e1640 1 --2- 172.21.15.50:0/2335371301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73ac100290 0x7f73ac1006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.877+0000 7f73b26e1640 1 -- 172.21.15.50:0/2335371301 >> 172.21.15.50:0/2335371301 conn(0x7f73ac0fbb80 msgr2=0x7f73ac0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.877+0000 7f73b26e1640 1 -- 172.21.15.50:0/2335371301 shutdown_connections 2024-04-23T18:18:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.877+0000 7f73b26e1640 1 -- 172.21.15.50:0/2335371301 wait complete. 2024-04-23T18:18:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73b26e1640 1 Processor -- start 2024-04-23T18:18:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73b26e1640 1 -- start start 2024-04-23T18:18:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73b26e1640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73ac100290 0x7f73ac198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73b26e1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73ac103300 0x7f73ac198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73b26e1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73ac103cc0 0x7f73ac19f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73b26e1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac10dac0 con 0x7f73ac103300 2024-04-23T18:18:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73b26e1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac10d940 con 0x7f73ac103cc0 2024-04-23T18:18:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73b26e1640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac10dc40 con 0x7f73ac100290 2024-04-23T18:18:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73ab7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73ac103300 0x7f73ac198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73ab7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73ac103300 0x7f73ac198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40358/0 (socket says 172.21.15.50:40358) 2024-04-23T18:18:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73ab7fe640 1 -- 172.21.15.50:0/2727035929 learned_addr learned my addr 172.21.15.50:0/2727035929 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73abfff640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73ac100290 0x7f73ac198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73b0c7e640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73ac103cc0 0x7f73ac19f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73ab7fe640 1 -- 172.21.15.50:0/2727035929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73ac100290 msgr2=0x7f73ac198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73ab7fe640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73ac100290 0x7f73ac198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73ab7fe640 1 -- 172.21.15.50:0/2727035929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73ac103cc0 msgr2=0x7f73ac19f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73ab7fe640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73ac103cc0 0x7f73ac19f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.878+0000 7f73ab7fe640 1 -- 172.21.15.50:0/2727035929 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ac19fd30 con 0x7f73ac103300 2024-04-23T18:18:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.879+0000 7f73ab7fe640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73ac103300 0x7f73ac198650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f739c00c890 tx=0x7f739c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.879+0000 7f73a97fa640 1 -- 172.21.15.50:0/2727035929 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c004220 con 0x7f73ac103300 2024-04-23T18:18:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.879+0000 7f73a97fa640 1 -- 172.21.15.50:0/2727035929 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f739c0043c0 con 0x7f73ac103300 2024-04-23T18:18:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.879+0000 7f73b26e1640 1 -- 172.21.15.50:0/2727035929 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73ac19ffc0 con 0x7f73ac103300 2024-04-23T18:18:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.879+0000 7f73a97fa640 1 -- 172.21.15.50:0/2727035929 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c004a00 con 0x7f73ac103300 2024-04-23T18:18:19.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.879+0000 7f73b26e1640 1 -- 172.21.15.50:0/2727035929 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73ac1a04e0 con 0x7f73ac103300 2024-04-23T18:18:19.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.880+0000 7f73b26e1640 1 -- 172.21.15.50:0/2727035929 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73ac109430 con 0x7f73ac103300 2024-04-23T18:18:19.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.881+0000 7f73a97fa640 1 -- 172.21.15.50:0/2727035929 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f739c0079b0 con 0x7f73ac103300 2024-04-23T18:18:19.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.881+0000 7f73a97fa640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f73940a3bb0 0x7f73940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:19.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.882+0000 7f73abfff640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f73940a3bb0 0x7f73940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:19.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.882+0000 7f73a97fa640 1 -- 172.21.15.50:0/2727035929 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f739c0d0230 con 0x7f73ac103300 2024-04-23T18:18:19.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.882+0000 7f73abfff640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f73940a3bb0 0x7f73940a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f73a00058c0 tx=0x7f73a0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:19.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:19.886+0000 7f73a97fa640 1 -- 172.21.15.50:0/2727035929 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f739c014030 con 0x7f73ac103300 2024-04-23T18:18:20.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.074+0000 7f73b26e1640 1 -- 172.21.15.50:0/2727035929 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f73ac062c50 con 0x7f73ac103300 2024-04-23T18:18:20.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.074+0000 7f73a97fa640 1 -- 172.21.15.50:0/2727035929 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f739c09cbd0 con 0x7f73ac103300 2024-04-23T18:18:20.068 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-23T18:18:20.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.076+0000 7f73b26e1640 1 -- 172.21.15.50:0/2727035929 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f73940a3bb0 msgr2=0x7f73940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.076+0000 7f73b26e1640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f73940a3bb0 0x7f73940a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f73a00058c0 tx=0x7f73a0005850 comp rx=0 tx=0).stop 2024-04-23T18:18:20.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.076+0000 7f73b26e1640 1 -- 172.21.15.50:0/2727035929 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73ac103300 msgr2=0x7f73ac198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.076+0000 7f73b26e1640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73ac103300 0x7f73ac198650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f739c00c890 tx=0x7f739c00cd50 comp rx=0 tx=0).stop 2024-04-23T18:18:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.076+0000 7f73b26e1640 1 -- 172.21.15.50:0/2727035929 shutdown_connections 2024-04-23T18:18:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.076+0000 7f73b26e1640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f73940a3bb0 0x7f73940a6070 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.076+0000 7f73b26e1640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73ac103cc0 0x7f73ac19f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.076+0000 7f73b26e1640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73ac103300 0x7f73ac198650 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.076+0000 7f73b26e1640 1 --2- 172.21.15.50:0/2727035929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73ac100290 0x7f73ac198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.076+0000 7f73b26e1640 1 -- 172.21.15.50:0/2727035929 >> 172.21.15.50:0/2727035929 conn(0x7f73ac0fbb80 msgr2=0x7f73ac106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.076+0000 7f73b26e1640 1 -- 172.21.15.50:0/2727035929 shutdown_connections 2024-04-23T18:18:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.076+0000 7f73b26e1640 1 -- 172.21.15.50:0/2727035929 wait complete. 2024-04-23T18:18:20.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:18:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.197+0000 7f11097c6640 1 -- 172.21.15.50:0/3596747916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11041031a0 msgr2=0x7f1104103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.197+0000 7f11097c6640 1 --2- 172.21.15.50:0/3596747916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11041031a0 0x7f1104103620 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009900 tx=0x7f10f802a410 comp rx=0 tx=0).stop 2024-04-23T18:18:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.197+0000 7f11097c6640 1 -- 172.21.15.50:0/3596747916 shutdown_connections 2024-04-23T18:18:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.197+0000 7f11097c6640 1 --2- 172.21.15.50:0/3596747916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1104103b60 0x7f110410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.197+0000 7f11097c6640 1 --2- 172.21.15.50:0/3596747916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11041031a0 0x7f1104103620 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.197+0000 7f11097c6640 1 --2- 172.21.15.50:0/3596747916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1104101fb0 0x7f11041023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.197+0000 7f11097c6640 1 -- 172.21.15.50:0/3596747916 >> 172.21.15.50:0/3596747916 conn(0x7f11040fd540 msgr2=0x7f11040ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.197+0000 7f11097c6640 1 -- 172.21.15.50:0/3596747916 shutdown_connections 2024-04-23T18:18:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.197+0000 7f11097c6640 1 -- 172.21.15.50:0/3596747916 wait complete. 2024-04-23T18:18:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f11097c6640 1 Processor -- start 2024-04-23T18:18:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f11097c6640 1 -- start start 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f11097c6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1104101fb0 0x7f1104198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f11097c6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11041031a0 0x7f11041986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f11097c6640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1104103b60 0x7f110419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f11097c6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f110410daf0 con 0x7f1104101fb0 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f11097c6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f110410d970 con 0x7f11041031a0 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f11097c6640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f110410dc70 con 0x7f1104103b60 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f1102ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1104101fb0 0x7f1104198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f1102ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1104101fb0 0x7f1104198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40378/0 (socket says 172.21.15.50:40378) 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f1102ffd640 1 -- 172.21.15.50:0/1790591529 learned_addr learned my addr 172.21.15.50:0/1790591529 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f11037fe640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1104103b60 0x7f110419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f1102ffd640 1 -- 172.21.15.50:0/1790591529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1104103b60 msgr2=0x7f110419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f11027fc640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11041031a0 0x7f11041986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f1102ffd640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1104103b60 0x7f110419f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f1102ffd640 1 -- 172.21.15.50:0/1790591529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11041031a0 msgr2=0x7f11041986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f1102ffd640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11041031a0 0x7f11041986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.198+0000 7f1102ffd640 1 -- 172.21.15.50:0/1790591529 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110419fda0 con 0x7f1104101fb0 2024-04-23T18:18:20.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.199+0000 7f1102ffd640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1104101fb0 0x7f1104198180 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f10f400d770 tx=0x7f10f400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:20.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.199+0000 7f10ebfff640 1 -- 172.21.15.50:0/1790591529 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f4017070 con 0x7f1104101fb0 2024-04-23T18:18:20.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.199+0000 7f11097c6640 1 -- 172.21.15.50:0/1790591529 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11041a0090 con 0x7f1104101fb0 2024-04-23T18:18:20.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.199+0000 7f10ebfff640 1 -- 172.21.15.50:0/1790591529 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f4009c30 con 0x7f1104101fb0 2024-04-23T18:18:20.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.199+0000 7f10ebfff640 1 -- 172.21.15.50:0/1790591529 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f4003cc0 con 0x7f1104101fb0 2024-04-23T18:18:20.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.199+0000 7f11097c6640 1 -- 172.21.15.50:0/1790591529 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11041a05e0 con 0x7f1104101fb0 2024-04-23T18:18:20.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.200+0000 7f11097c6640 1 -- 172.21.15.50:0/1790591529 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1104109320 con 0x7f1104101fb0 2024-04-23T18:18:20.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.201+0000 7f10ebfff640 1 -- 172.21.15.50:0/1790591529 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f10f4003e60 con 0x7f1104101fb0 2024-04-23T18:18:20.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.202+0000 7f10ebfff640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f10ec0a3bb0 0x7f10ec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:20.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.202+0000 7f11027fc640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f10ec0a3bb0 0x7f10ec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:20.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.202+0000 7f10ebfff640 1 -- 172.21.15.50:0/1790591529 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f10f40cfe40 con 0x7f1104101fb0 2024-04-23T18:18:20.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.202+0000 7f11027fc640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f10ec0a3bb0 0x7f10ec0a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f10f802a930 tx=0x7f10f8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:20.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.206+0000 7f10ebfff640 1 -- 172.21.15.50:0/1790591529 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f10f409c820 con 0x7f1104101fb0 2024-04-23T18:18:20.364 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:20 smithi050 ceph-mon[29187]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-23T18:18:20.364 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:20 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/692506020' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:20.364 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:20 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1673259065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:20.364 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:20 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2727035929' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:20.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.394+0000 7f11097c6640 1 -- 172.21.15.50:0/1790591529 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1104062b80 con 0x7f1104101fb0 2024-04-23T18:18:20.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.394+0000 7f10ebfff640 1 -- 172.21.15.50:0/1790591529 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f10f4009de0 con 0x7f1104101fb0 2024-04-23T18:18:20.388 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2024-04-23T18:18:20.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.396+0000 7f11097c6640 1 -- 172.21.15.50:0/1790591529 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f10ec0a3bb0 msgr2=0x7f10ec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.396+0000 7f11097c6640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f10ec0a3bb0 0x7f10ec0a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f10f802a930 tx=0x7f10f8000bc0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.396+0000 7f11097c6640 1 -- 172.21.15.50:0/1790591529 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1104101fb0 msgr2=0x7f1104198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.396+0000 7f11097c6640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1104101fb0 0x7f1104198180 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f10f400d770 tx=0x7f10f400dc30 comp rx=0 tx=0).stop 2024-04-23T18:18:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.396+0000 7f11097c6640 1 -- 172.21.15.50:0/1790591529 shutdown_connections 2024-04-23T18:18:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.396+0000 7f11097c6640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f10ec0a3bb0 0x7f10ec0a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.396+0000 7f11097c6640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1104103b60 0x7f110419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.396+0000 7f11097c6640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11041031a0 0x7f11041986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.396+0000 7f11097c6640 1 --2- 172.21.15.50:0/1790591529 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1104101fb0 0x7f1104198180 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.396+0000 7f11097c6640 1 -- 172.21.15.50:0/1790591529 >> 172.21.15.50:0/1790591529 conn(0x7f11040fd540 msgr2=0x7f11041064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.397+0000 7f11097c6640 1 -- 172.21.15.50:0/1790591529 shutdown_connections 2024-04-23T18:18:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.397+0000 7f11097c6640 1 -- 172.21.15.50:0/1790591529 wait complete. 2024-04-23T18:18:20.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:18:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.514+0000 7fcffe530640 1 -- 172.21.15.50:0/3731006903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff8102ad0 msgr2=0x7fcff8104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.514+0000 7fcffe530640 1 --2- 172.21.15.50:0/3731006903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff8102ad0 0x7fcff8104ec0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009900 tx=0x7fcfec02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.515+0000 7fcffe530640 1 -- 172.21.15.50:0/3731006903 shutdown_connections 2024-04-23T18:18:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.515+0000 7fcffe530640 1 --2- 172.21.15.50:0/3731006903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff8107d30 0x7fcff810a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.515+0000 7fcffe530640 1 --2- 172.21.15.50:0/3731006903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcff8105400 0x7fcff81077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.515+0000 7fcffe530640 1 --2- 172.21.15.50:0/3731006903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff8102ad0 0x7fcff8104ec0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.515+0000 7fcffe530640 1 -- 172.21.15.50:0/3731006903 >> 172.21.15.50:0/3731006903 conn(0x7fcff80fc440 msgr2=0x7fcff80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.515+0000 7fcffe530640 1 -- 172.21.15.50:0/3731006903 shutdown_connections 2024-04-23T18:18:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.515+0000 7fcffe530640 1 -- 172.21.15.50:0/3731006903 wait complete. 2024-04-23T18:18:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.515+0000 7fcffe530640 1 Processor -- start 2024-04-23T18:18:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.515+0000 7fcffe530640 1 -- start start 2024-04-23T18:18:20.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcffe530640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff8102ad0 0x7fcff81982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcffe530640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff8105400 0x7fcff8198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcffe530640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcff8107d30 0x7fcff819f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcffe530640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcff810db10 con 0x7fcff8105400 2024-04-23T18:18:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcffe530640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcff810d990 con 0x7fcff8102ad0 2024-04-23T18:18:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcffe530640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcff810dc90 con 0x7fcff8107d30 2024-04-23T18:18:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff8105400 0x7fcff8198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff8105400 0x7fcff8198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40406/0 (socket says 172.21.15.50:40406) 2024-04-23T18:18:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff77fe640 1 -- 172.21.15.50:0/39303029 learned_addr learned my addr 172.21.15.50:0/39303029 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff7fff640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff8102ad0 0x7fcff81982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcffcacd640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcff8107d30 0x7fcff819f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff77fe640 1 -- 172.21.15.50:0/39303029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcff8107d30 msgr2=0x7fcff819f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff77fe640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcff8107d30 0x7fcff819f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff77fe640 1 -- 172.21.15.50:0/39303029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff8102ad0 msgr2=0x7fcff81982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff77fe640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff8102ad0 0x7fcff81982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff77fe640 1 -- 172.21.15.50:0/39303029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff819fee0 con 0x7fcff8105400 2024-04-23T18:18:20.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff77fe640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff8105400 0x7fcff8198800 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800e9f0 tx=0x7fcfe800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:20.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff57fa640 1 -- 172.21.15.50:0/39303029 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe800ccc0 con 0x7fcff8105400 2024-04-23T18:18:20.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff57fa640 1 -- 172.21.15.50:0/39303029 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfe80054d0 con 0x7fcff8105400 2024-04-23T18:18:20.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcff57fa640 1 -- 172.21.15.50:0/39303029 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8010430 con 0x7fcff8105400 2024-04-23T18:18:20.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.516+0000 7fcffe530640 1 -- 172.21.15.50:0/39303029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff81a01d0 con 0x7fcff8105400 2024-04-23T18:18:20.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.517+0000 7fcffe530640 1 -- 172.21.15.50:0/39303029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff81a06d0 con 0x7fcff8105400 2024-04-23T18:18:20.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.518+0000 7fcffe530640 1 -- 172.21.15.50:0/39303029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcff80fdf30 con 0x7fcff8105400 2024-04-23T18:18:20.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.519+0000 7fcff57fa640 1 -- 172.21.15.50:0/39303029 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fcfe80046c0 con 0x7fcff8105400 2024-04-23T18:18:20.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.519+0000 7fcff57fa640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcfe00a3bb0 0x7fcfe00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:20.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.520+0000 7fcff7fff640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcfe00a3bb0 0x7fcfe00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:20.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.520+0000 7fcff57fa640 1 -- 172.21.15.50:0/39303029 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fcfe80d0780 con 0x7fcff8105400 2024-04-23T18:18:20.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.520+0000 7fcff7fff640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcfe00a3bb0 0x7fcfe00a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcfec000c00 tx=0x7fcfec0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:20.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.524+0000 7fcff57fa640 1 -- 172.21.15.50:0/39303029 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcfe809d120 con 0x7fcff8105400 2024-04-23T18:18:20.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:20 smithi177 ceph-mon[32308]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-23T18:18:20.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:20 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/692506020' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:20.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:20 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1673259065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:20.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:20 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2727035929' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:20.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:20 smithi188 ceph-mon[32694]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-23T18:18:20.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:20 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/692506020' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:20.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:20 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1673259065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:20.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:20 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2727035929' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:20.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.714+0000 7fcffe530640 1 -- 172.21.15.50:0/39303029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcff8062b50 con 0x7fcff8105400 2024-04-23T18:18:20.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.714+0000 7fcff57fa640 1 -- 172.21.15.50:0/39303029 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fcfe809ca20 con 0x7fcff8105400 2024-04-23T18:18:20.708 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2024-04-23T18:18:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.717+0000 7fcffe530640 1 -- 172.21.15.50:0/39303029 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcfe00a3bb0 msgr2=0x7fcfe00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.717+0000 7fcffe530640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcfe00a3bb0 0x7fcfe00a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcfec000c00 tx=0x7fcfec0314a0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.717+0000 7fcffe530640 1 -- 172.21.15.50:0/39303029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff8105400 msgr2=0x7fcff8198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.717+0000 7fcffe530640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff8105400 0x7fcff8198800 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800e9f0 tx=0x7fcfe800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.717+0000 7fcffe530640 1 -- 172.21.15.50:0/39303029 shutdown_connections 2024-04-23T18:18:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.717+0000 7fcffe530640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fcfe00a3bb0 0x7fcfe00a6070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.717+0000 7fcffe530640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcff8107d30 0x7fcff819f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.717+0000 7fcffe530640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcff8105400 0x7fcff8198800 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.717+0000 7fcffe530640 1 --2- 172.21.15.50:0/39303029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff8102ad0 0x7fcff81982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.717+0000 7fcffe530640 1 -- 172.21.15.50:0/39303029 >> 172.21.15.50:0/39303029 conn(0x7fcff80fc440 msgr2=0x7fcff8104990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:20.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.717+0000 7fcffe530640 1 -- 172.21.15.50:0/39303029 shutdown_connections 2024-04-23T18:18:20.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.717+0000 7fcffe530640 1 -- 172.21.15.50:0/39303029 wait complete. 2024-04-23T18:18:20.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:18:20.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.831+0000 7f828ffe8640 1 -- 172.21.15.50:0/568890244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82881029a0 msgr2=0x7f82881092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.831+0000 7f828ffe8640 1 --2- 172.21.15.50:0/568890244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82881029a0 0x7f82881092c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8284009ca0 tx=0x7f828402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.831+0000 7f828ffe8640 1 -- 172.21.15.50:0/568890244 shutdown_connections 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.831+0000 7f828ffe8640 1 --2- 172.21.15.50:0/568890244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82881029a0 0x7f82881092c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.831+0000 7f828ffe8640 1 --2- 172.21.15.50:0/568890244 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8288101fe0 0x7f8288102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.831+0000 7f828ffe8640 1 --2- 172.21.15.50:0/568890244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8288100e90 0x7f82881012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.831+0000 7f828ffe8640 1 -- 172.21.15.50:0/568890244 >> 172.21.15.50:0/568890244 conn(0x7f82880fc400 msgr2=0x7f82880fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.831+0000 7f828ffe8640 1 -- 172.21.15.50:0/568890244 shutdown_connections 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.831+0000 7f828ffe8640 1 -- 172.21.15.50:0/568890244 wait complete. 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828ffe8640 1 Processor -- start 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828ffe8640 1 -- start start 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828ffe8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8288100e90 0x7f828810b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828ffe8640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8288101fe0 0x7f8288109e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828ffe8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82881029a0 0x7f828810a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828ffe8640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f828810db30 con 0x7f82881029a0 2024-04-23T18:18:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828ffe8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f828810d9b0 con 0x7f8288100e90 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828ffe8640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f828810dcb0 con 0x7f8288101fe0 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828e585640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82881029a0 0x7f828810a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828e585640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82881029a0 0x7f828810a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40426/0 (socket says 172.21.15.50:40426) 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828e585640 1 -- 172.21.15.50:0/667401778 learned_addr learned my addr 172.21.15.50:0/667401778 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828d583640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8288101fe0 0x7f8288109e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.832+0000 7f828dd84640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8288100e90 0x7f828810b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.833+0000 7f828e585640 1 -- 172.21.15.50:0/667401778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8288101fe0 msgr2=0x7f8288109e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.833+0000 7f828e585640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8288101fe0 0x7f8288109e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.833+0000 7f828e585640 1 -- 172.21.15.50:0/667401778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8288100e90 msgr2=0x7f828810b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.833+0000 7f828e585640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8288100e90 0x7f828810b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.833+0000 7f828e585640 1 -- 172.21.15.50:0/667401778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828810ac40 con 0x7f82881029a0 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.833+0000 7f828e585640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82881029a0 0x7f828810a3b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f82840097b0 tx=0x7f8284003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.833+0000 7f827effd640 1 -- 172.21.15.50:0/667401778 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284004240 con 0x7f82881029a0 2024-04-23T18:18:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.833+0000 7f828ffe8640 1 -- 172.21.15.50:0/667401778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82881a22b0 con 0x7f82881029a0 2024-04-23T18:18:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.833+0000 7f827effd640 1 -- 172.21.15.50:0/667401778 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82840317c0 con 0x7f82881029a0 2024-04-23T18:18:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.833+0000 7f827effd640 1 -- 172.21.15.50:0/667401778 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828402fd80 con 0x7f82881029a0 2024-04-23T18:18:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.833+0000 7f828ffe8640 1 -- 172.21.15.50:0/667401778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82881a2850 con 0x7f82881029a0 2024-04-23T18:18:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.834+0000 7f828ffe8640 1 -- 172.21.15.50:0/667401778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82881081a0 con 0x7f82881029a0 2024-04-23T18:18:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.835+0000 7f827effd640 1 -- 172.21.15.50:0/667401778 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f82840043e0 con 0x7f82881029a0 2024-04-23T18:18:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.836+0000 7f827effd640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f82740a3bb0 0x7f82740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.836+0000 7f828dd84640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f82740a3bb0 0x7f82740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.836+0000 7f827effd640 1 -- 172.21.15.50:0/667401778 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f82840f1d10 con 0x7f82881029a0 2024-04-23T18:18:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.836+0000 7f828dd84640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f82740a3bb0 0x7f82740a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8280005fd0 tx=0x7f8280005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:20.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:20.840+0000 7f827effd640 1 -- 172.21.15.50:0/667401778 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f82840be6f0 con 0x7f82881029a0 2024-04-23T18:18:21.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.087+0000 7f828ffe8640 1 -- 172.21.15.50:0/667401778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8288062b50 con 0x7f82881029a0 2024-04-23T18:18:21.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.088+0000 7f827effd640 1 -- 172.21.15.50:0/667401778 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11875 (secure 0 0 0) 0x7f82840de070 con 0x7f82881029a0 2024-04-23T18:18:21.081 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:21.082 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:16:37.965071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:18:21.082 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 20 2024-04-23T18:18:21.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.090+0000 7f828ffe8640 1 -- 172.21.15.50:0/667401778 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f82740a3bb0 msgr2=0x7f82740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.090+0000 7f828ffe8640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f82740a3bb0 0x7f82740a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8280005fd0 tx=0x7f8280005850 comp rx=0 tx=0).stop 2024-04-23T18:18:21.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.090+0000 7f828ffe8640 1 -- 172.21.15.50:0/667401778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82881029a0 msgr2=0x7f828810a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.090+0000 7f828ffe8640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82881029a0 0x7f828810a3b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f82840097b0 tx=0x7f8284003c20 comp rx=0 tx=0).stop 2024-04-23T18:18:21.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.090+0000 7f828ffe8640 1 -- 172.21.15.50:0/667401778 shutdown_connections 2024-04-23T18:18:21.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.090+0000 7f828ffe8640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f82740a3bb0 0x7f82740a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.090+0000 7f828ffe8640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82881029a0 0x7f828810a3b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.090+0000 7f828ffe8640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8288101fe0 0x7f8288109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.090+0000 7f828ffe8640 1 --2- 172.21.15.50:0/667401778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8288100e90 0x7f828810b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.090+0000 7f828ffe8640 1 -- 172.21.15.50:0/667401778 >> 172.21.15.50:0/667401778 conn(0x7f82880fc400 msgr2=0x7f8288105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.090+0000 7f828ffe8640 1 -- 172.21.15.50:0/667401778 shutdown_connections 2024-04-23T18:18:21.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.090+0000 7f828ffe8640 1 -- 172.21.15.50:0/667401778 wait complete. 2024-04-23T18:18:21.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:18:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.208+0000 7f81954e1640 1 -- 172.21.15.50:0/669640571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81901031b0 msgr2=0x7f8190103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.208+0000 7f81954e1640 1 --2- 172.21.15.50:0/669640571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81901031b0 0x7f8190103630 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8180009900 tx=0x7f818002a410 comp rx=0 tx=0).stop 2024-04-23T18:18:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.208+0000 7f81954e1640 1 -- 172.21.15.50:0/669640571 shutdown_connections 2024-04-23T18:18:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.208+0000 7f81954e1640 1 --2- 172.21.15.50:0/669640571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8190103b70 0x7f819010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.208+0000 7f81954e1640 1 --2- 172.21.15.50:0/669640571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81901031b0 0x7f8190103630 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.208+0000 7f81954e1640 1 --2- 172.21.15.50:0/669640571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8190101fc0 0x7f81901023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.208+0000 7f81954e1640 1 -- 172.21.15.50:0/669640571 >> 172.21.15.50:0/669640571 conn(0x7f81900fd550 msgr2=0x7f81900ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.208+0000 7f81954e1640 1 -- 172.21.15.50:0/669640571 shutdown_connections 2024-04-23T18:18:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.208+0000 7f81954e1640 1 -- 172.21.15.50:0/669640571 wait complete. 2024-04-23T18:18:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.208+0000 7f81954e1640 1 Processor -- start 2024-04-23T18:18:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f81954e1640 1 -- start start 2024-04-23T18:18:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f81954e1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8190101fc0 0x7f8190075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f81954e1640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81901031b0 0x7f8190074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f81954e1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8190103b70 0x7f8190074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f81954e1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f819010db50 con 0x7f8190101fc0 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f81954e1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f819010d9d0 con 0x7f8190103b70 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f81954e1640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f819010dcd0 con 0x7f81901031b0 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f818effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8190101fc0 0x7f8190075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f818effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8190101fc0 0x7f8190075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40450/0 (socket says 172.21.15.50:40450) 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f818effd640 1 -- 172.21.15.50:0/3461821456 learned_addr learned my addr 172.21.15.50:0/3461821456 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f818e7fc640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81901031b0 0x7f8190074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f818effd640 1 -- 172.21.15.50:0/3461821456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81901031b0 msgr2=0x7f8190074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f818effd640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81901031b0 0x7f8190074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f818effd640 1 -- 172.21.15.50:0/3461821456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8190103b70 msgr2=0x7f8190074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f818f7fe640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8190103b70 0x7f8190074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f818effd640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8190103b70 0x7f8190074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f818effd640 1 -- 172.21.15.50:0/3461821456 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8190075110 con 0x7f8190101fc0 2024-04-23T18:18:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f818effd640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8190101fc0 0x7f8190075d00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f817800d950 tx=0x7f817800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f8187fff640 1 -- 172.21.15.50:0/3461821456 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8178017070 con 0x7f8190101fc0 2024-04-23T18:18:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f8187fff640 1 -- 172.21.15.50:0/3461821456 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8178009c30 con 0x7f8190101fc0 2024-04-23T18:18:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f81954e1640 1 -- 172.21.15.50:0/3461821456 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81901a2320 con 0x7f8190101fc0 2024-04-23T18:18:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.209+0000 7f8187fff640 1 -- 172.21.15.50:0/3461821456 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817800f620 con 0x7f8190101fc0 2024-04-23T18:18:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.210+0000 7f81954e1640 1 -- 172.21.15.50:0/3461821456 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81901a2820 con 0x7f8190101fc0 2024-04-23T18:18:21.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.211+0000 7f81954e1640 1 -- 172.21.15.50:0/3461821456 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8190109380 con 0x7f8190101fc0 2024-04-23T18:18:21.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.212+0000 7f8187fff640 1 -- 172.21.15.50:0/3461821456 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f817801a070 con 0x7f8190101fc0 2024-04-23T18:18:21.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.212+0000 7f8187fff640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f817c0a3bb0 0x7f817c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.213+0000 7f818e7fc640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f817c0a3bb0 0x7f817c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.213+0000 7f8187fff640 1 -- 172.21.15.50:0/3461821456 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f81780cfef0 con 0x7f8190101fc0 2024-04-23T18:18:21.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.213+0000 7f818e7fc640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f817c0a3bb0 0x7f817c0a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8180031500 tx=0x7f8180005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.217+0000 7f8187fff640 1 -- 172.21.15.50:0/3461821456 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8178009de0 con 0x7f8190101fc0 2024-04-23T18:18:21.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:21 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1790591529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:21.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:21 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/39303029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:21.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:21 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/667401778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:21.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.410+0000 7f81954e1640 1 -- 172.21.15.50:0/3461821456 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8190062b50 con 0x7f8190101fc0 2024-04-23T18:18:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.411+0000 7f8187fff640 1 -- 172.21.15.50:0/3461821456 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22661 (secure 0 0 0) 0x7f817809c8d0 con 0x7f8190101fc0 2024-04-23T18:18:21.405 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:21.406 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":89,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:17:36.102736+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:18:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.414+0000 7f81954e1640 1 -- 172.21.15.50:0/3461821456 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f817c0a3bb0 msgr2=0x7f817c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.414+0000 7f81954e1640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f817c0a3bb0 0x7f817c0a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8180031500 tx=0x7f8180005fd0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.414+0000 7f81954e1640 1 -- 172.21.15.50:0/3461821456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8190101fc0 msgr2=0x7f8190075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.414+0000 7f81954e1640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8190101fc0 0x7f8190075d00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f817800d950 tx=0x7f817800de10 comp rx=0 tx=0).stop 2024-04-23T18:18:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.414+0000 7f81954e1640 1 -- 172.21.15.50:0/3461821456 shutdown_connections 2024-04-23T18:18:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.414+0000 7f81954e1640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f817c0a3bb0 0x7f817c0a6070 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.414+0000 7f81954e1640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8190103b70 0x7f8190074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.414+0000 7f81954e1640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81901031b0 0x7f8190074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.414+0000 7f81954e1640 1 --2- 172.21.15.50:0/3461821456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8190101fc0 0x7f8190075d00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.414+0000 7f81954e1640 1 -- 172.21.15.50:0/3461821456 >> 172.21.15.50:0/3461821456 conn(0x7f81900fd550 msgr2=0x7f8190106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.414+0000 7f81954e1640 1 -- 172.21.15.50:0/3461821456 shutdown_connections 2024-04-23T18:18:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.414+0000 7f81954e1640 1 -- 172.21.15.50:0/3461821456 wait complete. 2024-04-23T18:18:21.419 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T18:16:33.316468+0000', 'modified': '2024-04-23T18:16:37.965071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24506, 'mds_1': 24499, 'mds_2': 24475}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24440': {'gid': 24440, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6835/475975073', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6834', 'nonce': 475975073}, {'type': 'v1', 'addr': '172.21.15.177:6835', 'nonce': 475975073}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6835/1248812622', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6834', 'nonce': 1248812622}, {'type': 'v1', 'addr': '172.21.15.188:6835', 'nonce': 1248812622}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6837/1048893636', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6836', 'nonce': 1048893636}, {'type': 'v1', 'addr': '172.21.15.177:6837', 'nonce': 1048893636}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.188:6837/2190889779', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6836', 'nonce': 2190889779}, {'type': 'v1', 'addr': '172.21.15.188:6837', 'nonce': 2190889779}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.188:6839/2202634301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6838', 'nonce': 2202634301}, {'type': 'v1', 'addr': '172.21.15.188:6839', 'nonce': 2202634301}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.177:6839/2936205355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 2936205355}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 2936205355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24506, 'qdb_cluster': [24475, 24506, 24499]} 2024-04-23T18:18:21.419 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-23T18:18:21.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-23T18:18:21.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.528+0000 7f02fbf44640 1 -- 172.21.15.50:0/1943794318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f4107c70 msgr2=0x7f02f410a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.528+0000 7f02fbf44640 1 --2- 172.21.15.50:0/1943794318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f4107c70 0x7f02f410a0c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f02e4009900 tx=0x7f02e402a410 comp rx=0 tx=0).stop 2024-04-23T18:18:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.528+0000 7f02fbf44640 1 -- 172.21.15.50:0/1943794318 shutdown_connections 2024-04-23T18:18:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.528+0000 7f02fbf44640 1 --2- 172.21.15.50:0/1943794318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f4107c70 0x7f02f410a0c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.528+0000 7f02fbf44640 1 --2- 172.21.15.50:0/1943794318 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02f4105340 0x7f02f4107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.528+0000 7f02fbf44640 1 --2- 172.21.15.50:0/1943794318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f4102a10 0x7f02f4104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.528+0000 7f02fbf44640 1 -- 172.21.15.50:0/1943794318 >> 172.21.15.50:0/1943794318 conn(0x7f02f40fc400 msgr2=0x7f02f40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02fbf44640 1 -- 172.21.15.50:0/1943794318 shutdown_connections 2024-04-23T18:18:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02fbf44640 1 -- 172.21.15.50:0/1943794318 wait complete. 2024-04-23T18:18:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02fbf44640 1 Processor -- start 2024-04-23T18:18:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02fbf44640 1 -- start start 2024-04-23T18:18:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02fbf44640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02f4102a10 0x7f02f4198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02fbf44640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f4105340 0x7f02f41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02fbf44640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f4107c70 0x7f02f419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02fbf44640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f02f410dad0 con 0x7f02f4105340 2024-04-23T18:18:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02fbf44640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02f410d950 con 0x7f02f4107c70 2024-04-23T18:18:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02fbf44640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f02f410dc50 con 0x7f02f4102a10 2024-04-23T18:18:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02f94df640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f4105340 0x7f02f41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02f94df640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f4105340 0x7f02f41986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40470/0 (socket says 172.21.15.50:40470) 2024-04-23T18:18:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02f94df640 1 -- 172.21.15.50:0/858205209 learned_addr learned my addr 172.21.15.50:0/858205209 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02f9ce0640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02f4102a10 0x7f02f4198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.529+0000 7f02fa4e1640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f4107c70 0x7f02f419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.530+0000 7f02f94df640 1 -- 172.21.15.50:0/858205209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02f4102a10 msgr2=0x7f02f4198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.530+0000 7f02f94df640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02f4102a10 0x7f02f4198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.530+0000 7f02f94df640 1 -- 172.21.15.50:0/858205209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f4107c70 msgr2=0x7f02f419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.530+0000 7f02f94df640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f4107c70 0x7f02f419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.530+0000 7f02f94df640 1 -- 172.21.15.50:0/858205209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f419fda0 con 0x7f02f4105340 2024-04-23T18:18:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.530+0000 7f02fa4e1640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f4107c70 0x7f02f419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.530+0000 7f02f94df640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f4105340 0x7f02f41986c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f02e800c970 tx=0x7f02e800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.530+0000 7f02f2ffd640 1 -- 172.21.15.50:0/858205209 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8007bb0 con 0x7f02f4105340 2024-04-23T18:18:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.530+0000 7f02f2ffd640 1 -- 172.21.15.50:0/858205209 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02e8007d50 con 0x7f02f4105340 2024-04-23T18:18:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.530+0000 7f02fbf44640 1 -- 172.21.15.50:0/858205209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f41a0090 con 0x7f02f4105340 2024-04-23T18:18:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.530+0000 7f02f2ffd640 1 -- 172.21.15.50:0/858205209 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8018730 con 0x7f02f4105340 2024-04-23T18:18:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.530+0000 7f02fbf44640 1 -- 172.21.15.50:0/858205209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f41a0590 con 0x7f02f4105340 2024-04-23T18:18:21.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.532+0000 7f02f2ffd640 1 -- 172.21.15.50:0/858205209 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f02e800f990 con 0x7f02f4105340 2024-04-23T18:18:21.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.532+0000 7f02fec68740 1 Processor -- start 2024-04-23T18:18:21.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.532+0000 7f02fec68740 1 -- start start 2024-04-23T18:18:21.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fec68740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 0x557da55affa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fec68740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 0x557da55b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fec68740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55b1710 0x557da55b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fec68740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557da5514c50 con 0x557da55afb80 2024-04-23T18:18:21.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fec68740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557da54b1830 con 0x557da55b1710 2024-04-23T18:18:21.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fec68740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557da5510c10 con 0x557da55b0d50 2024-04-23T18:18:21.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fa4e1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 0x557da55affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02f2ffd640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f02dc0a3bb0 0x7f02dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fa4e1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 0x557da55affa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40480/0 (socket says 172.21.15.50:40480) 2024-04-23T18:18:21.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fa4e1640 1 -- 172.21.15.50:0/4076856666 learned_addr learned my addr 172.21.15.50:0/4076856666 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02f2ffd640 1 -- 172.21.15.50:0/858205209 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f02e80d0450 con 0x7f02f4105340 2024-04-23T18:18:21.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fa4e1640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f02dc0a3bb0 0x7f02dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02f94df640 1 --2- 172.21.15.50:0/4076856666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55b1710 0x557da55b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02f9ce0640 1 --2- 172.21.15.50:0/4076856666 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 0x557da55b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fa4e1640 1 -- 172.21.15.50:0/4076856666 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 msgr2=0x557da55b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fa4e1640 1 --2- 172.21.15.50:0/4076856666 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 0x557da55b11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fa4e1640 1 -- 172.21.15.50:0/4076856666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55b1710 msgr2=0x557da55b7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fa4e1640 1 --2- 172.21.15.50:0/4076856666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55b1710 0x557da55b7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fa4e1640 1 -- 172.21.15.50:0/4076856666 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557da52272d0 con 0x557da55afb80 2024-04-23T18:18:21.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.533+0000 7f02fa4e1640 1 --2- 172.21.15.50:0/4076856666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 0x557da55affa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f02e4005940 tx=0x7f02e40318e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483744fa28472ef0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02dbfff640 1 -- 172.21.15.50:0/4076856666 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e4035070 con 0x557da55afb80 2024-04-23T18:18:21.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02dbfff640 1 -- 172.21.15.50:0/4076856666 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02e404c070 con 0x557da55afb80 2024-04-23T18:18:21.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02dbfff640 1 -- 172.21.15.50:0/4076856666 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e4004a80 con 0x557da55afb80 2024-04-23T18:18:21.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02fec68740 1 -- 172.21.15.50:0/4076856666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 msgr2=0x557da55affa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02fec68740 1 --2- 172.21.15.50:0/4076856666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 0x557da55affa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f02e4005940 tx=0x7f02e40318e0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02fa4e1640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f02dc0a3bb0 0x7f02dc0a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f02e404b000 tx=0x7f02e4049000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02fec68740 1 -- 172.21.15.50:0/4076856666 shutdown_connections 2024-04-23T18:18:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02fec68740 1 --2- 172.21.15.50:0/4076856666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55b1710 0x557da55b7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02fec68740 1 --2- 172.21.15.50:0/4076856666 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 0x557da55b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02fec68740 1 --2- 172.21.15.50:0/4076856666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 0x557da55affa0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02fec68740 1 -- 172.21.15.50:0/4076856666 >> 172.21.15.50:0/4076856666 conn(0x557da55ab070 msgr2=0x557da55ab470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02fec68740 1 -- 172.21.15.50:0/4076856666 shutdown_connections 2024-04-23T18:18:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.534+0000 7f02fec68740 1 -- 172.21.15.50:0/4076856666 wait complete. 2024-04-23T18:18:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02fec68740 1 -- start start 2024-04-23T18:18:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02fec68740 1 Processor -- start 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02fec68740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55afb80 0x557da55ba0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02fec68740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55b0d50 0x557da55ba600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02fec68740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55b1710 0x557da55c1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02fec68740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557da55b8720 con 0x557da55b0d50 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02fec68740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557da54af470 con 0x557da55b1710 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02fec68740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557da55b88a0 con 0x557da55afb80 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02fec68740 10 client.0 _ll_register_callbacks cb 0x557da54d4f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02f94df640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55b0d50 0x557da55ba600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02f94df640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55b0d50 0x557da55ba600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40494/0 (socket says 172.21.15.50:40494) 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02f94df640 1 -- 172.21.15.50:0/2745983306 learned_addr learned my addr 172.21.15.50:0/2745983306 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02fa4e1640 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55b1710 0x557da55c1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.535+0000 7f02f9ce0640 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55afb80 0x557da55ba0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02f94df640 1 -- 172.21.15.50:0/2745983306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55afb80 msgr2=0x557da55ba0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02f94df640 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55afb80 0x557da55ba0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02f94df640 1 -- 172.21.15.50:0/2745983306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55b1710 msgr2=0x557da55c1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02f94df640 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55b1710 0x557da55c1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02f94df640 1 -- 172.21.15.50:0/2745983306 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557da52f16b0 con 0x557da55b0d50 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02f94df640 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55b0d50 0x557da55ba600 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f02e80cdf50 tx=0x7f02e809c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d241b034644b42f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02d97fa640 1 -- 172.21.15.50:0/2745983306 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e80a0ae0 con 0x557da55b0d50 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02d97fa640 1 -- 172.21.15.50:0/2745983306 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f02e80ce060 con 0x557da55b0d50 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02fec68740 1 -- 172.21.15.50:0/2745983306 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02e404c070 con 0x557da55b0d50 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02d97fa640 1 -- 172.21.15.50:0/2745983306 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f02e80a53a0 con 0x557da55b0d50 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02fec68740 10 client.15195 fetch_fsmap learned FSMap version 20 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02fec68740 1 -- 172.21.15.50:0/2745983306 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x557da55c4990 con 0x557da55b0d50 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02d97fa640 1 -- 172.21.15.50:0/2745983306 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02e404c070 con 0x557da55b0d50 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02d97fa640 1 -- 172.21.15.50:0/2745983306 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f02e8020030 con 0x557da55b0d50 2024-04-23T18:18:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02fec68740 10 client.15195 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T18:18:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02fec68740 10 client.15195 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-23T18:18:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02fec68740 20 client.15195 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02fec68740 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x557da55c6fc0 0x557da55c8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02fec68740 4 client.15195 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.536+0000 7f02fec68740 1 -- 172.21.15.50:0/2745983306 --> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557da5522670 con 0x557da55c6fc0 2024-04-23T18:18:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.537+0000 7f02f9ce0640 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x557da55c6fc0 0x557da55c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.537+0000 7f02f9ce0640 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x557da55c6fc0 0x557da55c8430 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e16c13c6775fc298 server_cookie=8fa610f500b0a4b6 in_seq=0 out_seq=0 2024-04-23T18:18:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.537+0000 7f02d97fa640 10 client.15195 ms_handle_connect on v2:172.21.15.188:6836/2190889779 2024-04-23T18:18:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02d97fa640 1 -- 172.21.15.50:0/2745983306 <== mds.2 v2:172.21.15.188:6836/2190889779 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557da5522670 con 0x557da55c6fc0 2024-04-23T18:18:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02d97fa640 10 client.15195 handle_command_reply: tid=0 2024-04-23T18:18:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02fec68740 1 client.15195 shutdown 2024-04-23T18:18:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02fec68740 1 -- 172.21.15.50:0/2745983306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55b0d50 msgr2=0x557da55ba600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02fec68740 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55b0d50 0x557da55ba600 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f02e80cdf50 tx=0x7f02e809c3a0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02fec68740 1 -- 172.21.15.50:0/2745983306 shutdown_connections 2024-04-23T18:18:21.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02fec68740 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x557da55c6fc0 0x557da55c8430 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02fec68740 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55b1710 0x557da55c1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02fec68740 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55b0d50 0x557da55ba600 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02d97fa640 0 client.15195 ms_handle_reset on v2:172.21.15.188:6836/2190889779 2024-04-23T18:18:21.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02fec68740 1 --2- 172.21.15.50:0/2745983306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55afb80 0x557da55ba0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02fec68740 1 -- 172.21.15.50:0/2745983306 >> 172.21.15.50:0/2745983306 conn(0x557da55ab070 msgr2=0x557da55aa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02fec68740 1 -- 172.21.15.50:0/2745983306 shutdown_connections 2024-04-23T18:18:21.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02fec68740 1 -- 172.21.15.50:0/2745983306 wait complete. 2024-04-23T18:18:21.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.538+0000 7f02fec68740 20 client.15195 trim_cache size 0 max 16384 2024-04-23T18:18:21.544 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:21 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1790591529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:21.544 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:21 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/39303029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:21.544 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:21 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/667401778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:21.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02fec68740 1 Processor -- start 2024-04-23T18:18:21.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02fec68740 1 -- start start 2024-04-23T18:18:21.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02fec68740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55ab070 0x557da55a9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02fec68740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55afb80 0x557da55c1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02fec68740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 0x557da55a9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02fec68740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557da55c6770 con 0x557da55ab070 2024-04-23T18:18:21.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02fec68740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557da55c6e40 con 0x557da55afb80 2024-04-23T18:18:21.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02fec68740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557da55c7510 con 0x557da55b0d50 2024-04-23T18:18:21.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02f9ce0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55ab070 0x557da55a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02f9ce0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55ab070 0x557da55a9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40510/0 (socket says 172.21.15.50:40510) 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02f9ce0640 1 -- 172.21.15.50:0/201515592 learned_addr learned my addr 172.21.15.50:0/201515592 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02f9ce0640 1 -- 172.21.15.50:0/201515592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 msgr2=0x557da55a9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02f9ce0640 1 --2- 172.21.15.50:0/201515592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 0x557da55a9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02f9ce0640 1 -- 172.21.15.50:0/201515592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55afb80 msgr2=0x557da55c1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02f9ce0640 1 --2- 172.21.15.50:0/201515592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55afb80 0x557da55c1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02f9ce0640 1 -- 172.21.15.50:0/201515592 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557da55c8760 con 0x557da55ab070 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02f9ce0640 1 --2- 172.21.15.50:0/201515592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55ab070 0x557da55a9940 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x557da55c8dc0 tx=0x7f02ec009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b78ad73d1fc1f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.557+0000 7f02db7fe640 1 -- 172.21.15.50:0/201515592 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec013e20 con 0x557da55ab070 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02db7fe640 1 -- 172.21.15.50:0/201515592 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02ec01b3b0 con 0x557da55ab070 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02db7fe640 1 -- 172.21.15.50:0/201515592 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec005660 con 0x557da55ab070 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02fec68740 1 -- 172.21.15.50:0/201515592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55ab070 msgr2=0x557da55a9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02fec68740 1 --2- 172.21.15.50:0/201515592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55ab070 0x557da55a9940 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x557da55c8dc0 tx=0x7f02ec009680 comp rx=0 tx=0).stop 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02fec68740 1 -- 172.21.15.50:0/201515592 shutdown_connections 2024-04-23T18:18:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02fec68740 1 --2- 172.21.15.50:0/201515592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 0x557da55a9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02fec68740 1 --2- 172.21.15.50:0/201515592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55afb80 0x557da55c1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02fec68740 1 --2- 172.21.15.50:0/201515592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55ab070 0x557da55a9940 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02fec68740 1 -- 172.21.15.50:0/201515592 >> 172.21.15.50:0/201515592 conn(0x557da55b4100 msgr2=0x557da55a60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02fec68740 1 -- 172.21.15.50:0/201515592 shutdown_connections 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02fec68740 1 -- 172.21.15.50:0/201515592 wait complete. 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02fec68740 1 -- start start 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.558+0000 7f02fec68740 1 Processor -- start 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02fec68740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55ab070 0x557da55bd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02fec68740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 0x557da55bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02fec68740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 0x557da55c0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02fec68740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557da55c82b0 con 0x557da55afb80 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02fec68740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557da55c7be0 con 0x557da55ab070 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02fec68740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557da55c8980 con 0x557da55b0d50 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02fec68740 10 client.0 _ll_register_callbacks cb 0x7f02e8007f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:21.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02f94df640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 0x557da55bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02f94df640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 0x557da55bfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40524/0 (socket says 172.21.15.50:40524) 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02f94df640 1 -- 172.21.15.50:0/616007677 learned_addr learned my addr 172.21.15.50:0/616007677 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02f9ce0640 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55ab070 0x557da55bd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02fa4e1640 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 0x557da55c0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02f94df640 1 -- 172.21.15.50:0/616007677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 msgr2=0x557da55c0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02f94df640 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 0x557da55c0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02f94df640 1 -- 172.21.15.50:0/616007677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55ab070 msgr2=0x557da55bd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02f94df640 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55ab070 0x557da55bd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02f94df640 1 -- 172.21.15.50:0/616007677 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557da55c0870 con 0x557da55afb80 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02fa4e1640 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 0x557da55c0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02f94df640 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 0x557da55bfb30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f02e809c6c0 tx=0x7f02e800fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa471ae53b5509b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.559+0000 7f02d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02d97fa640 1 -- 172.21.15.50:0/616007677 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e809c500 con 0x557da55afb80 2024-04-23T18:18:21.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02d97fa640 1 -- 172.21.15.50:0/616007677 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f02e809bb80 con 0x557da55afb80 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02fec68740 1 -- 172.21.15.50:0/616007677 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02ec01b3b0 con 0x557da55afb80 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02d97fa640 1 -- 172.21.15.50:0/616007677 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f02e80a0ae0 con 0x557da55afb80 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02fec68740 10 client.15201 fetch_fsmap learned FSMap version 20 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02fec68740 1 -- 172.21.15.50:0/616007677 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x557da5635850 con 0x557da55afb80 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02d97fa640 1 -- 172.21.15.50:0/616007677 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02ec01b3b0 con 0x557da55afb80 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02d97fa640 1 -- 172.21.15.50:0/616007677 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f02e8020050 con 0x557da55afb80 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02fec68740 10 client.15201 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02fec68740 10 client.15201 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02fec68740 20 client.15201 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02fec68740 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x557da55b1710 0x557da56390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02fec68740 4 client.15201 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02fec68740 1 -- 172.21.15.50:0/616007677 --> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] -- command(tid 0: {"prefix": "status"}) -- 0x557da551c4e0 con 0x557da55b1710 2024-04-23T18:18:21.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.560+0000 7f02f9ce0640 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x557da55b1710 0x557da56390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02d97fa640 10 client.15201 ms_handle_connect on v2:172.21.15.188:6836/2190889779 2024-04-23T18:18:21.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02f9ce0640 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x557da55b1710 0x557da56390b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=302ec9b95ec0f0d8 server_cookie=955986c1fbd89ed1 in_seq=0 out_seq=0 2024-04-23T18:18:21.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02d97fa640 1 -- 172.21.15.50:0/616007677 <== mds.2 v2:172.21.15.188:6836/2190889779 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x557da551c4e0 con 0x557da55b1710 2024-04-23T18:18:21.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02d97fa640 10 client.15201 handle_command_reply: tid=0 2024-04-23T18:18:21.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02fec68740 1 client.15201 shutdown 2024-04-23T18:18:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02fec68740 1 -- 172.21.15.50:0/616007677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 msgr2=0x557da55bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02fec68740 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 0x557da55bfb30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f02e809c6c0 tx=0x7f02e800fc80 comp rx=0 tx=0).stop 2024-04-23T18:18:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02fec68740 1 -- 172.21.15.50:0/616007677 shutdown_connections 2024-04-23T18:18:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02fec68740 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x557da55b1710 0x557da56390b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02fec68740 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557da55b0d50 0x557da55c0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02fec68740 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557da55afb80 0x557da55bfb30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02d97fa640 0 client.15201 ms_handle_reset on v2:172.21.15.188:6836/2190889779 2024-04-23T18:18:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02fec68740 1 --2- 172.21.15.50:0/616007677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557da55ab070 0x557da55bd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02fec68740 1 -- 172.21.15.50:0/616007677 >> 172.21.15.50:0/616007677 conn(0x557da55b4100 msgr2=0x557da55add50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02fec68740 1 -- 172.21.15.50:0/616007677 shutdown_connections 2024-04-23T18:18:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.561+0000 7f02fec68740 1 -- 172.21.15.50:0/616007677 wait complete. 2024-04-23T18:18:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fec68740 20 client.15201 trim_cache size 0 max 16384 2024-04-23T18:18:21.563 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:18:21.563 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:18:21.563 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 2, 2024-04-23T18:18:21.563 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 24475, 2024-04-23T18:18:21.563 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2024-04-23T18:18:21.563 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2024-04-23T18:18:21.563 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2024-04-23T18:18:21.563 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 104.6020582, 2024-04-23T18:18:21.563 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 20, 2024-04-23T18:18:21.563 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 89, 2024-04-23T18:18:21.564 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 87, 2024-04-23T18:18:21.564 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 126.63746713 2024-04-23T18:18:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:18:21.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fbf44640 1 -- 172.21.15.50:0/858205209 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f02dc0a3bb0 msgr2=0x7f02dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fbf44640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f02dc0a3bb0 0x7f02dc0a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f02e404b000 tx=0x7f02e4049000 comp rx=0 tx=0).stop 2024-04-23T18:18:21.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fbf44640 1 -- 172.21.15.50:0/858205209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f4105340 msgr2=0x7f02f41986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fbf44640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f4105340 0x7f02f41986c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f02e800c970 tx=0x7f02e800ce30 comp rx=0 tx=0).stop 2024-04-23T18:18:21.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fbf44640 1 -- 172.21.15.50:0/858205209 shutdown_connections 2024-04-23T18:18:21.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fbf44640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f02dc0a3bb0 0x7f02dc0a6070 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fbf44640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f4107c70 0x7f02f419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fbf44640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02f4105340 0x7f02f41986c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fbf44640 1 --2- 172.21.15.50:0/858205209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02f4102a10 0x7f02f4198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fbf44640 1 -- 172.21.15.50:0/858205209 >> 172.21.15.50:0/858205209 conn(0x7f02f40fc400 msgr2=0x7f02f4103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fbf44640 1 -- 172.21.15.50:0/858205209 shutdown_connections 2024-04-23T18:18:21.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.562+0000 7f02fbf44640 1 -- 172.21.15.50:0/858205209 wait complete. 2024-04-23T18:18:21.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-23T18:18:21.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:21 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1790591529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:21.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:21 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/39303029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:21.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:21 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/667401778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:21.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.676+0000 7fd1d5607640 1 -- 172.21.15.50:0/3085999371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1d0105360 msgr2=0x7fd1d0107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.676+0000 7fd1d5607640 1 --2- 172.21.15.50:0/3085999371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1d0105360 0x7fd1d0107750 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009900 tx=0x7fd1c402a410 comp rx=0 tx=0).stop 2024-04-23T18:18:21.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.676+0000 7fd1d5607640 1 -- 172.21.15.50:0/3085999371 shutdown_connections 2024-04-23T18:18:21.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.676+0000 7fd1d5607640 1 --2- 172.21.15.50:0/3085999371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0107c90 0x7fd1d010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.676+0000 7fd1d5607640 1 --2- 172.21.15.50:0/3085999371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1d0105360 0x7fd1d0107750 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.676+0000 7fd1d5607640 1 --2- 172.21.15.50:0/3085999371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1d0102a30 0x7fd1d0104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.676+0000 7fd1d5607640 1 -- 172.21.15.50:0/3085999371 >> 172.21.15.50:0/3085999371 conn(0x7fd1d00fc440 msgr2=0x7fd1d00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.676+0000 7fd1d5607640 1 -- 172.21.15.50:0/3085999371 shutdown_connections 2024-04-23T18:18:21.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.676+0000 7fd1d5607640 1 -- 172.21.15.50:0/3085999371 wait complete. 2024-04-23T18:18:21.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1d5607640 1 Processor -- start 2024-04-23T18:18:21.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1d5607640 1 -- start start 2024-04-23T18:18:21.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1d5607640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0102a30 0x7fd1d01023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1d5607640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1d0105360 0x7fd1d01009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1d5607640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1d0107c90 0x7fd1d0100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1d5607640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d010db00 con 0x7fd1d0105360 2024-04-23T18:18:21.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1d5607640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d010d980 con 0x7fd1d0107c90 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1d5607640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d010dc80 con 0x7fd1d0102a30 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1ce7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1d0105360 0x7fd1d01009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1ce7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1d0105360 0x7fd1d01009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57460/0 (socket says 172.21.15.50:57460) 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1ce7fc640 1 -- 172.21.15.50:0/1040440687 learned_addr learned my addr 172.21.15.50:0/1040440687 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1ceffd640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0102a30 0x7fd1d01023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1cf7fe640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1d0107c90 0x7fd1d0100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1ce7fc640 1 -- 172.21.15.50:0/1040440687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0102a30 msgr2=0x7fd1d01023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0102a30 0x7fd1d01023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1ce7fc640 1 -- 172.21.15.50:0/1040440687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1d0107c90 msgr2=0x7fd1d0100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1d0107c90 0x7fd1d0100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1ce7fc640 1 -- 172.21.15.50:0/1040440687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d01017b0 con 0x7fd1d0105360 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1ceffd640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0102a30 0x7fd1d01023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.677+0000 7fd1cf7fe640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1d0107c90 0x7fd1d0100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:21.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.678+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1d0105360 0x7fd1d01009e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4031500 tx=0x7fd1c4003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.678+0000 7fd1b7fff640 1 -- 172.21.15.50:0/1040440687 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c403f070 con 0x7fd1d0105360 2024-04-23T18:18:21.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.678+0000 7fd1d5607640 1 -- 172.21.15.50:0/1040440687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d01a22c0 con 0x7fd1d0105360 2024-04-23T18:18:21.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.678+0000 7fd1b7fff640 1 -- 172.21.15.50:0/1040440687 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c4003e70 con 0x7fd1d0105360 2024-04-23T18:18:21.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.678+0000 7fd1b7fff640 1 -- 172.21.15.50:0/1040440687 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4035030 con 0x7fd1d0105360 2024-04-23T18:18:21.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.678+0000 7fd1d5607640 1 -- 172.21.15.50:0/1040440687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d01a2860 con 0x7fd1d0105360 2024-04-23T18:18:21.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.680+0000 7fd1b7fff640 1 -- 172.21.15.50:0/1040440687 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd1c4042070 con 0x7fd1d0105360 2024-04-23T18:18:21.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.680+0000 7fd1d83f1740 1 Processor -- start 2024-04-23T18:18:21.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.680+0000 7fd1d83f1740 1 -- start start 2024-04-23T18:18:21.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.680+0000 7fd1d83f1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 0x5608ea421fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.680+0000 7fd1d83f1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea422d50 0x5608ea4231d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.680+0000 7fd1d83f1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea423710 0x5608ea429fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.680+0000 7fd1d83f1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5608ea386c50 con 0x5608ea421b80 2024-04-23T18:18:21.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.680+0000 7fd1d83f1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5608ea323830 con 0x5608ea422d50 2024-04-23T18:18:21.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.680+0000 7fd1d83f1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5608ea382c10 con 0x5608ea423710 2024-04-23T18:18:21.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1cf7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 0x5608ea421fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1b7fff640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd1b80a3bb0 0x7fd1b80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1cf7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 0x5608ea421fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57468/0 (socket says 172.21.15.50:57468) 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1cf7fe640 1 -- 172.21.15.50:0/1498690201 learned_addr learned my addr 172.21.15.50:0/1498690201 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1cf7fe640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd1b80a3bb0 0x7fd1b80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1b7fff640 1 -- 172.21.15.50:0/1040440687 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fd1c40f1100 con 0x7fd1d0105360 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/1498690201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea423710 0x5608ea429fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1ceffd640 1 --2- 172.21.15.50:0/1498690201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea422d50 0x5608ea4231d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1cf7fe640 1 -- 172.21.15.50:0/1498690201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea423710 msgr2=0x5608ea429fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1cf7fe640 1 --2- 172.21.15.50:0/1498690201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea423710 0x5608ea429fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1cf7fe640 1 -- 172.21.15.50:0/1498690201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea422d50 msgr2=0x5608ea4231d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1cf7fe640 1 --2- 172.21.15.50:0/1498690201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea422d50 0x5608ea4231d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1cf7fe640 1 -- 172.21.15.50:0/1498690201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608ea0992d0 con 0x5608ea421b80 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1cf7fe640 1 --2- 172.21.15.50:0/1498690201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 0x5608ea421fa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8014a10 tx=0x7fd1c8014ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a07eb06d3956b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1b57fa640 1 -- 172.21.15.50:0/1498690201 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8014250 con 0x5608ea421b80 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1b57fa640 1 -- 172.21.15.50:0/1498690201 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c8023070 con 0x5608ea421b80 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1cf7fe640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd1b80a3bb0 0x7fd1b80a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8022000 tx=0x7fd1c8020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.681+0000 7fd1b57fa640 1 -- 172.21.15.50:0/1498690201 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8013960 con 0x5608ea421b80 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.682+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1498690201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 msgr2=0x5608ea421fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.682+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/1498690201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 0x5608ea421fa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8014a10 tx=0x7fd1c8014ed0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.682+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1498690201 shutdown_connections 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.682+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/1498690201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea423710 0x5608ea429fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.682+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/1498690201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea422d50 0x5608ea4231d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.682+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/1498690201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 0x5608ea421fa0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.682+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1498690201 >> 172.21.15.50:0/1498690201 conn(0x5608ea41d070 msgr2=0x5608ea41d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.682+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1498690201 shutdown_connections 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.682+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1498690201 wait complete. 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.682+0000 7fd1d83f1740 1 -- start start 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.682+0000 7fd1d83f1740 1 Processor -- start 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1d83f1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 0x5608ea42bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1d83f1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea422d50 0x5608ea42c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1d83f1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 0x5608ea433550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1d83f1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5608ea42a720 con 0x5608ea421b80 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1d83f1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5608ea321470 con 0x5608ea423710 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1d83f1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5608ea42a8a0 con 0x5608ea422d50 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1d83f1740 10 client.0 _ll_register_callbacks cb 0x5608ea346f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1ceffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 0x5608ea42bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1ceffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 0x5608ea42bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57472/0 (socket says 172.21.15.50:57472) 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1ceffd640 1 -- 172.21.15.50:0/1901565780 learned_addr learned my addr 172.21.15.50:0/1901565780 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea422d50 0x5608ea42c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1ceffd640 1 -- 172.21.15.50:0/1901565780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea422d50 msgr2=0x5608ea42c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1ceffd640 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea422d50 0x5608ea42c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1cf7fe640 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 0x5608ea433550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1ceffd640 1 -- 172.21.15.50:0/1901565780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 msgr2=0x5608ea433550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1ceffd640 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 0x5608ea433550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1ceffd640 1 -- 172.21.15.50:0/1901565780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608ea1636b0 con 0x5608ea421b80 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea422d50 0x5608ea42c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1ceffd640 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 0x5608ea42bfd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0010700 tx=0x7fd1c0010bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=566a3d5dd81e3669 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1b2ff5640 1 -- 172.21.15.50:0/1901565780 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0017070 con 0x5608ea421b80 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1b2ff5640 1 -- 172.21.15.50:0/1901565780 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fd1c000ba60 con 0x5608ea421b80 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.683+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1901565780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c8023070 con 0x5608ea421b80 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.684+0000 7fd1b2ff5640 1 -- 172.21.15.50:0/1901565780 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd1c000bc60 con 0x5608ea421b80 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.684+0000 7fd1d83f1740 10 client.15213 fetch_fsmap learned FSMap version 20 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.684+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1901565780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5608ea436900 con 0x5608ea421b80 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.684+0000 7fd1b2ff5640 1 -- 172.21.15.50:0/1901565780 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c8023070 con 0x5608ea421b80 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.684+0000 7fd1b2ff5640 1 -- 172.21.15.50:0/1901565780 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fd1c0004a70 con 0x5608ea421b80 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.684+0000 7fd1d83f1740 10 client.15213 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.684+0000 7fd1d83f1740 10 client.15213 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:18:21.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.684+0000 7fd1d83f1740 20 client.15213 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.684+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x5608ea438f90 0x5608ea43a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.684+0000 7fd1d83f1740 4 client.15213 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.684+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1901565780 --> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608ea394670 con 0x5608ea438f90 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.684+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x5608ea438f90 0x5608ea43a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.685+0000 7fd1b2ff5640 10 client.15213 ms_handle_connect on v2:172.21.15.188:6838/2202634301 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.685+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x5608ea438f90 0x5608ea43a400 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e38f9f2b45fabefc server_cookie=5daf8989682084f1 in_seq=0 out_seq=0 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.685+0000 7fd1b2ff5640 1 -- 172.21.15.50:0/1901565780 <== mds.1 v2:172.21.15.188:6838/2202634301 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5608ea394670 con 0x5608ea438f90 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.685+0000 7fd1b2ff5640 10 client.15213 handle_command_reply: tid=0 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.685+0000 7fd1d83f1740 1 client.15213 shutdown 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.686+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1901565780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 msgr2=0x5608ea42bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.686+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 0x5608ea42bfd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0010700 tx=0x7fd1c0010bc0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.686+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1901565780 shutdown_connections 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.686+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x5608ea438f90 0x5608ea43a400 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.686+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 0x5608ea433550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.686+0000 7fd1b2ff5640 0 client.15213 ms_handle_reset on v2:172.21.15.188:6838/2202634301 2024-04-23T18:18:21.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.686+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea422d50 0x5608ea42c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.686+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/1901565780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea421b80 0x5608ea42bfd0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.686+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1901565780 >> 172.21.15.50:0/1901565780 conn(0x5608ea41d070 msgr2=0x5608ea41c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.686+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1901565780 shutdown_connections 2024-04-23T18:18:21.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.686+0000 7fd1d83f1740 1 -- 172.21.15.50:0/1901565780 wait complete. 2024-04-23T18:18:21.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.686+0000 7fd1d83f1740 20 client.15213 trim_cache size 0 max 16384 2024-04-23T18:18:21.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.704+0000 7fd1d83f1740 1 Processor -- start 2024-04-23T18:18:21.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.704+0000 7fd1d83f1740 1 -- start start 2024-04-23T18:18:21.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1d83f1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea421b80 0x5608ea4180b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1d83f1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 0x5608ea433470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1d83f1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 0x5608ea41ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1d83f1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5608ea438740 con 0x5608ea422d50 2024-04-23T18:18:21.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1d83f1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5608ea438e10 con 0x5608ea423710 2024-04-23T18:18:21.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1d83f1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5608ea4394e0 con 0x5608ea421b80 2024-04-23T18:18:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1ce7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 0x5608ea433470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1ce7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 0x5608ea433470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57482/0 (socket says 172.21.15.50:57482) 2024-04-23T18:18:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1ce7fc640 1 -- 172.21.15.50:0/3350699687 learned_addr learned my addr 172.21.15.50:0/3350699687 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1ce7fc640 1 -- 172.21.15.50:0/3350699687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea421b80 msgr2=0x5608ea4180b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/3350699687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea421b80 0x5608ea4180b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1ce7fc640 1 -- 172.21.15.50:0/3350699687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 msgr2=0x5608ea41ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/3350699687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 0x5608ea41ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1ce7fc640 1 -- 172.21.15.50:0/3350699687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608ea41dd40 con 0x5608ea422d50 2024-04-23T18:18:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/3350699687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 0x5608ea433470 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40eec00 tx=0x7fd1c4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf4c13a33a3ea76 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1b4ff9640 1 -- 172.21.15.50:0/3350699687 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4049040 con 0x5608ea422d50 2024-04-23T18:18:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1b4ff9640 1 -- 172.21.15.50:0/3350699687 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c40c6050 con 0x5608ea422d50 2024-04-23T18:18:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.705+0000 7fd1b4ff9640 1 -- 172.21.15.50:0/3350699687 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c403be60 con 0x5608ea422d50 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.706+0000 7fd1d83f1740 1 -- 172.21.15.50:0/3350699687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 msgr2=0x5608ea433470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.706+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/3350699687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 0x5608ea433470 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40eec00 tx=0x7fd1c4009260 comp rx=0 tx=0).stop 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.706+0000 7fd1d83f1740 1 -- 172.21.15.50:0/3350699687 shutdown_connections 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.706+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/3350699687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 0x5608ea41ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.706+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/3350699687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 0x5608ea433470 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.706+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/3350699687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea421b80 0x5608ea4180b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.706+0000 7fd1d83f1740 1 -- 172.21.15.50:0/3350699687 >> 172.21.15.50:0/3350699687 conn(0x5608ea41d070 msgr2=0x5608ea41d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.706+0000 7fd1d83f1740 1 -- 172.21.15.50:0/3350699687 shutdown_connections 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.706+0000 7fd1d83f1740 1 -- 172.21.15.50:0/3350699687 wait complete. 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.706+0000 7fd1d83f1740 1 -- start start 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.706+0000 7fd1d83f1740 1 Processor -- start 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.706+0000 7fd1d83f1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea421b80 0x5608ea431460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1d83f1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 0x5608ea49af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1d83f1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 0x5608ea49d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1d83f1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5608ea43a280 con 0x5608ea422d50 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1d83f1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5608ea439bb0 con 0x5608ea423710 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1d83f1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5608ea43a950 con 0x5608ea421b80 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1d83f1740 10 client.0 _ll_register_callbacks cb 0x7fd1c000e430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1ce7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 0x5608ea49af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1ce7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 0x5608ea49af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57492/0 (socket says 172.21.15.50:57492) 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1ce7fc640 1 -- 172.21.15.50:0/2960676002 learned_addr learned my addr 172.21.15.50:0/2960676002 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1cf7fe640 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 0x5608ea49d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1ceffd640 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea421b80 0x5608ea431460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1ce7fc640 1 -- 172.21.15.50:0/2960676002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea421b80 msgr2=0x5608ea431460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea421b80 0x5608ea431460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1ce7fc640 1 -- 172.21.15.50:0/2960676002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 msgr2=0x5608ea49d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 0x5608ea49d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1ce7fc640 1 -- 172.21.15.50:0/2960676002 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608ea4295f0 con 0x5608ea422d50 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1ceffd640 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea421b80 0x5608ea431460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1ce7fc640 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 0x5608ea49af20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4031fd0 tx=0x7fd1c40c1910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe9158bd76a5088 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1b2ff5640 1 -- 172.21.15.50:0/2960676002 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c40db060 con 0x5608ea422d50 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1b2ff5640 1 -- 172.21.15.50:0/2960676002 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fd1c40bd640 con 0x5608ea422d50 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1d83f1740 1 -- 172.21.15.50:0/2960676002 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c40c6050 con 0x5608ea422d50 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1b2ff5640 1 -- 172.21.15.50:0/2960676002 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd1c403be60 con 0x5608ea422d50 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1d83f1740 10 client.15219 fetch_fsmap learned FSMap version 20 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.707+0000 7fd1d83f1740 1 -- 172.21.15.50:0/2960676002 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5608ea49ea80 con 0x5608ea422d50 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.708+0000 7fd1b2ff5640 1 -- 172.21.15.50:0/2960676002 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c40c6050 con 0x5608ea422d50 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.708+0000 7fd1b2ff5640 1 -- 172.21.15.50:0/2960676002 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fd1c40d6b60 con 0x5608ea422d50 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.708+0000 7fd1d83f1740 10 client.15219 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.708+0000 7fd1d83f1740 10 client.15219 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:18:21.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.708+0000 7fd1d83f1740 20 client.15219 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.708+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x5608ea4a3460 0x5608ea4a2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.708+0000 7fd1d83f1740 4 client.15219 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.708+0000 7fd1d83f1740 1 -- 172.21.15.50:0/2960676002 --> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] -- command(tid 0: {"prefix": "status"}) -- 0x5608ea38e4e0 con 0x5608ea4a3460 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.708+0000 7fd1ceffd640 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x5608ea4a3460 0x5608ea4a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1b2ff5640 10 client.15219 ms_handle_connect on v2:172.21.15.188:6838/2202634301 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1ceffd640 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x5608ea4a3460 0x5608ea4a2170 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=23634fd60c90c0d server_cookie=a213b5e76fca7d36 in_seq=0 out_seq=0 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1b2ff5640 1 -- 172.21.15.50:0/2960676002 <== mds.1 v2:172.21.15.188:6838/2202634301 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5608ea38e4e0 con 0x5608ea4a3460 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1b2ff5640 10 client.15219 handle_command_reply: tid=0 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1d83f1740 1 client.15219 shutdown 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1d83f1740 1 -- 172.21.15.50:0/2960676002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 msgr2=0x5608ea49af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 0x5608ea49af20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4031fd0 tx=0x7fd1c40c1910 comp rx=0 tx=0).stop 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1d83f1740 1 -- 172.21.15.50:0/2960676002 shutdown_connections 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x5608ea4a3460 0x5608ea4a2170 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608ea423710 0x5608ea49d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608ea422d50 0x5608ea49af20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1d83f1740 1 --2- 172.21.15.50:0/2960676002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608ea421b80 0x5608ea431460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1b2ff5640 0 client.15219 ms_handle_reset on v2:172.21.15.188:6838/2202634301 2024-04-23T18:18:21.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1d83f1740 1 -- 172.21.15.50:0/2960676002 >> 172.21.15.50:0/2960676002 conn(0x5608ea41d070 msgr2=0x5608ea41d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1d83f1740 1 -- 172.21.15.50:0/2960676002 shutdown_connections 2024-04-23T18:18:21.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1d83f1740 1 -- 172.21.15.50:0/2960676002 wait complete. 2024-04-23T18:18:21.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.709+0000 7fd1d83f1740 20 client.15219 trim_cache size 0 max 16384 2024-04-23T18:18:21.710 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:18:21.710 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:18:21.710 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 1, 2024-04-23T18:18:21.711 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 24499, 2024-04-23T18:18:21.711 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2024-04-23T18:18:21.711 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2024-04-23T18:18:21.711 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2024-04-23T18:18:21.711 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 105.761030225, 2024-04-23T18:18:21.711 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 20, 2024-04-23T18:18:21.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 89, 2024-04-23T18:18:21.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 87, 2024-04-23T18:18:21.711 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 121.249378515 2024-04-23T18:18:21.711 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:18:21.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.710+0000 7fd1d5607640 1 -- 172.21.15.50:0/1040440687 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd1b80a3bb0 msgr2=0x7fd1b80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.710+0000 7fd1d5607640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd1b80a3bb0 0x7fd1b80a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8022000 tx=0x7fd1c8020000 comp rx=0 tx=0).stop 2024-04-23T18:18:21.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.710+0000 7fd1d5607640 1 -- 172.21.15.50:0/1040440687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1d0105360 msgr2=0x7fd1d01009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.710+0000 7fd1d5607640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1d0105360 0x7fd1d01009e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4031500 tx=0x7fd1c4003c80 comp rx=0 tx=0).stop 2024-04-23T18:18:21.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.710+0000 7fd1d5607640 1 -- 172.21.15.50:0/1040440687 shutdown_connections 2024-04-23T18:18:21.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.710+0000 7fd1d5607640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fd1b80a3bb0 0x7fd1b80a6070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.710+0000 7fd1d5607640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1d0107c90 0x7fd1d0100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.710+0000 7fd1d5607640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1d0105360 0x7fd1d01009e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.710+0000 7fd1d5607640 1 --2- 172.21.15.50:0/1040440687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1d0102a30 0x7fd1d01023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.710+0000 7fd1d5607640 1 -- 172.21.15.50:0/1040440687 >> 172.21.15.50:0/1040440687 conn(0x7fd1d00fc440 msgr2=0x7fd1d01034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.710+0000 7fd1d5607640 1 -- 172.21.15.50:0/1040440687 shutdown_connections 2024-04-23T18:18:21.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.710+0000 7fd1d5607640 1 -- 172.21.15.50:0/1040440687 wait complete. 2024-04-23T18:18:21.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T18:18:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.822+0000 7f86048a9640 1 -- 172.21.15.50:0/1794548004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8600101fe0 msgr2=0x7f8600102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.822+0000 7f86048a9640 1 --2- 172.21.15.50:0/1794548004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8600101fe0 0x7f8600102460 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009900 tx=0x7f85f402a410 comp rx=0 tx=0).stop 2024-04-23T18:18:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 -- 172.21.15.50:0/1794548004 shutdown_connections 2024-04-23T18:18:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 --2- 172.21.15.50:0/1794548004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86001029a0 0x7f86001092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 --2- 172.21.15.50:0/1794548004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8600101fe0 0x7f8600102460 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 --2- 172.21.15.50:0/1794548004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8600100e90 0x7f86001012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 -- 172.21.15.50:0/1794548004 >> 172.21.15.50:0/1794548004 conn(0x7f86000fc400 msgr2=0x7f86000fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 -- 172.21.15.50:0/1794548004 shutdown_connections 2024-04-23T18:18:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 -- 172.21.15.50:0/1794548004 wait complete. 2024-04-23T18:18:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 Processor -- start 2024-04-23T18:18:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 -- start start 2024-04-23T18:18:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8600100e90 0x7f86001981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8600101fe0 0x7f8600198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86001029a0 0x7f860019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f860010db90 con 0x7f8600100e90 2024-04-23T18:18:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f860010da10 con 0x7f8600101fe0 2024-04-23T18:18:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.823+0000 7f86048a9640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f860010dd10 con 0x7f86001029a0 2024-04-23T18:18:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85fe59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8600100e90 0x7f86001981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85fe59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8600100e90 0x7f86001981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57514/0 (socket says 172.21.15.50:57514) 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85fe59c640 1 -- 172.21.15.50:0/3055426735 learned_addr learned my addr 172.21.15.50:0/3055426735 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85fdd9b640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8600101fe0 0x7f8600198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85fed9d640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86001029a0 0x7f860019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85fe59c640 1 -- 172.21.15.50:0/3055426735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86001029a0 msgr2=0x7f860019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85fe59c640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86001029a0 0x7f860019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85fe59c640 1 -- 172.21.15.50:0/3055426735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8600101fe0 msgr2=0x7f8600198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85fe59c640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8600101fe0 0x7f8600198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85fe59c640 1 -- 172.21.15.50:0/3055426735 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860019fdf0 con 0x7f8600100e90 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85fe59c640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8600100e90 0x7f86001981d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f85f000cce0 tx=0x7f85f000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85ef7fe640 1 -- 172.21.15.50:0/3055426735 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f0004b20 con 0x7f8600100e90 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f86048a9640 1 -- 172.21.15.50:0/3055426735 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86001a00e0 con 0x7f8600100e90 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85ef7fe640 1 -- 172.21.15.50:0/3055426735 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f0004cc0 con 0x7f8600100e90 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f85ef7fe640 1 -- 172.21.15.50:0/3055426735 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f000f6b0 con 0x7f8600100e90 2024-04-23T18:18:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.824+0000 7f86048a9640 1 -- 172.21.15.50:0/3055426735 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86001a0600 con 0x7f8600100e90 2024-04-23T18:18:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.826+0000 7f85ef7fe640 1 -- 172.21.15.50:0/3055426735 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f85f000f850 con 0x7f8600100e90 2024-04-23T18:18:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f86075bd740 1 Processor -- start 2024-04-23T18:18:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f86075bd740 1 -- start start 2024-04-23T18:18:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f86075bd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44293b80 0x560b44293fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f86075bd740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 0x560b442951d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f86075bd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44295710 0x560b4429bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f86075bd740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560b441f8c50 con 0x560b44294d50 2024-04-23T18:18:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f86075bd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560b44195830 con 0x560b44293b80 2024-04-23T18:18:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f86075bd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560b441f4c10 con 0x560b44295710 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f85fe59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 0x560b442951d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f85fe59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 0x560b442951d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57516/0 (socket says 172.21.15.50:57516) 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f85fe59c640 1 -- 172.21.15.50:0/271893220 learned_addr learned my addr 172.21.15.50:0/271893220 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f85ef7fe640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f85e80a3bb0 0x7f85e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f85fed9d640 1 --2- 172.21.15.50:0/271893220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44293b80 0x560b44293fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f85fdd9b640 1 --2- 172.21.15.50:0/271893220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44295710 0x560b4429bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f85fed9d640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f85e80a3bb0 0x7f85e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f85fe59c640 1 -- 172.21.15.50:0/271893220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44295710 msgr2=0x560b4429bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.827+0000 7f85fe59c640 1 --2- 172.21.15.50:0/271893220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44295710 0x560b4429bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.828+0000 7f85fe59c640 1 -- 172.21.15.50:0/271893220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44293b80 msgr2=0x560b44293fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.828+0000 7f85fe59c640 1 --2- 172.21.15.50:0/271893220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44293b80 0x560b44293fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.828+0000 7f85ef7fe640 1 -- 172.21.15.50:0/3055426735 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f85f00d04e0 con 0x7f8600100e90 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.828+0000 7f85fe59c640 1 -- 172.21.15.50:0/271893220 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b43f0b2d0 con 0x560b44294d50 2024-04-23T18:18:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.828+0000 7f85fe59c640 1 --2- 172.21.15.50:0/271893220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 0x560b442951d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f85f0099150 tx=0x7f85f00519c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e7ba9d317e0a72 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.828+0000 7f85fed9d640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f85e80a3bb0 0x7f85e80a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f85f80058c0 tx=0x7f85f8004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.828+0000 7f85ecff9640 1 -- 172.21.15.50:0/271893220 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f001d070 con 0x560b44294d50 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.828+0000 7f85ecff9640 1 -- 172.21.15.50:0/271893220 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f009ce80 con 0x560b44294d50 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.828+0000 7f85ecff9640 1 -- 172.21.15.50:0/271893220 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f009aa80 con 0x560b44294d50 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 -- 172.21.15.50:0/271893220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 msgr2=0x560b442951d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 --2- 172.21.15.50:0/271893220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 0x560b442951d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f85f0099150 tx=0x7f85f00519c0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 -- 172.21.15.50:0/271893220 shutdown_connections 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 --2- 172.21.15.50:0/271893220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44295710 0x560b4429bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 --2- 172.21.15.50:0/271893220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 0x560b442951d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 --2- 172.21.15.50:0/271893220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44293b80 0x560b44293fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 -- 172.21.15.50:0/271893220 >> 172.21.15.50:0/271893220 conn(0x560b4428f070 msgr2=0x560b4428f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 -- 172.21.15.50:0/271893220 shutdown_connections 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 -- 172.21.15.50:0/271893220 wait complete. 2024-04-23T18:18:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 -- start start 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 Processor -- start 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44293b80 0x560b4429df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 0x560b4429e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44295710 0x560b442a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560b4429c6c0 con 0x560b44294d50 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560b44193470 con 0x560b44293b80 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560b4429c840 con 0x560b44295710 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f86075bd740 10 client.0 _ll_register_callbacks cb 0x560b441b8f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.829+0000 7f85fdd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 0x560b4429e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85fdd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 0x560b4429e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57526/0 (socket says 172.21.15.50:57526) 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85fdd9b640 1 -- 172.21.15.50:0/693032589 learned_addr learned my addr 172.21.15.50:0/693032589 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85fed9d640 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44295710 0x560b442a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85fe59c640 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44293b80 0x560b4429df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85fdd9b640 1 -- 172.21.15.50:0/693032589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44295710 msgr2=0x560b442a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85fdd9b640 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44295710 0x560b442a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85fdd9b640 1 -- 172.21.15.50:0/693032589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44293b80 msgr2=0x560b4429df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85fdd9b640 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44293b80 0x560b4429df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85fdd9b640 1 -- 172.21.15.50:0/693032589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b43fd56b0 con 0x560b44294d50 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85fed9d640 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44295710 0x560b442a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85fe59c640 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44293b80 0x560b4429df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85fdd9b640 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 0x560b4429e480 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f85f4006010 tx=0x7f85f40044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de00fc32ad5a73f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85e5ffb640 1 -- 172.21.15.50:0/693032589 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4035070 con 0x560b44294d50 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85e5ffb640 1 -- 172.21.15.50:0/693032589 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f85f4003a80 con 0x560b44294d50 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f86075bd740 1 -- 172.21.15.50:0/693032589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85f009ce80 con 0x560b44294d50 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85e5ffb640 1 -- 172.21.15.50:0/693032589 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f85f4003c80 con 0x560b44294d50 2024-04-23T18:18:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f86075bd740 10 client.15231 fetch_fsmap learned FSMap version 20 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f86075bd740 1 -- 172.21.15.50:0/693032589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560b442a88a0 con 0x560b44294d50 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.830+0000 7f85e5ffb640 1 -- 172.21.15.50:0/693032589 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f009ce80 con 0x560b44294d50 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.831+0000 7f85e5ffb640 1 -- 172.21.15.50:0/693032589 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f85f403b5f0 con 0x560b44294d50 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.831+0000 7f86075bd740 10 client.15231 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.831+0000 7f86075bd740 10 client.15231 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.831+0000 7f86075bd740 20 client.15231 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.831+0000 7f86075bd740 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560b442aaf80 0x560b442ac3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.831+0000 7f86075bd740 4 client.15231 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.831+0000 7f86075bd740 1 -- 172.21.15.50:0/693032589 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b44206670 con 0x560b442aaf80 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.831+0000 7f85fe59c640 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560b442aaf80 0x560b442ac3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.831+0000 7f85fe59c640 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560b442aaf80 0x560b442ac3f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdcf65321ebf3a0d server_cookie=2d8315dbbc1455ee in_seq=0 out_seq=0 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.831+0000 7f85e5ffb640 10 client.15231 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f85e5ffb640 1 -- 172.21.15.50:0/693032589 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560b44206670 con 0x560b442aaf80 2024-04-23T18:18:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f85e5ffb640 10 client.15231 handle_command_reply: tid=0 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f86075bd740 1 client.15231 shutdown 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f86075bd740 1 -- 172.21.15.50:0/693032589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 msgr2=0x560b4429e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f86075bd740 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 0x560b4429e480 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f85f4006010 tx=0x7f85f40044f0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f86075bd740 1 -- 172.21.15.50:0/693032589 shutdown_connections 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f86075bd740 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560b442aaf80 0x560b442ac3f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f86075bd740 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44295710 0x560b442a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f86075bd740 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44294d50 0x560b4429e480 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f85e5ffb640 0 client.15231 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f86075bd740 1 --2- 172.21.15.50:0/693032589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44293b80 0x560b4429df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f86075bd740 1 -- 172.21.15.50:0/693032589 >> 172.21.15.50:0/693032589 conn(0x560b4428f070 msgr2=0x560b4428e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f86075bd740 1 -- 172.21.15.50:0/693032589 shutdown_connections 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f86075bd740 1 -- 172.21.15.50:0/693032589 wait complete. 2024-04-23T18:18:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.832+0000 7f86075bd740 20 client.15231 trim_cache size 0 max 16384 2024-04-23T18:18:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f86075bd740 1 Processor -- start 2024-04-23T18:18:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f86075bd740 1 -- start start 2024-04-23T18:18:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f86075bd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44293b80 0x560b44295130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f86075bd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44294d50 0x560b442a5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f86075bd740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 0x560b4429db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f86075bd740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560b442aa730 con 0x560b44295710 2024-04-23T18:18:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f86075bd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560b442aae00 con 0x560b44294d50 2024-04-23T18:18:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f86075bd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560b442ab4d0 con 0x560b44293b80 2024-04-23T18:18:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85fed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 0x560b4429db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85fed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 0x560b4429db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57530/0 (socket says 172.21.15.50:57530) 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85fed9d640 1 -- 172.21.15.50:0/3048121062 learned_addr learned my addr 172.21.15.50:0/3048121062 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85fdd9b640 1 --2- 172.21.15.50:0/3048121062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44294d50 0x560b442a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85fe59c640 1 --2- 172.21.15.50:0/3048121062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44293b80 0x560b44295130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85fed9d640 1 -- 172.21.15.50:0/3048121062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44293b80 msgr2=0x560b44295130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85fed9d640 1 --2- 172.21.15.50:0/3048121062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44293b80 0x560b44295130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85fed9d640 1 -- 172.21.15.50:0/3048121062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44294d50 msgr2=0x560b442a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85fed9d640 1 --2- 172.21.15.50:0/3048121062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44294d50 0x560b442a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85fed9d640 1 -- 172.21.15.50:0/3048121062 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b442947d0 con 0x560b44295710 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85fe59c640 1 --2- 172.21.15.50:0/3048121062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44293b80 0x560b44295130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85fed9d640 1 --2- 172.21.15.50:0/3048121062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 0x560b4429db50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f85f8011810 tx=0x7f85f800b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b75be5130a78dc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85e7fff640 1 -- 172.21.15.50:0/3048121062 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f800fb40 con 0x560b44295710 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85e7fff640 1 -- 172.21.15.50:0/3048121062 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f8018040 con 0x560b44295710 2024-04-23T18:18:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.851+0000 7f85e7fff640 1 -- 172.21.15.50:0/3048121062 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f80203f0 con 0x560b44295710 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.852+0000 7f86075bd740 1 -- 172.21.15.50:0/3048121062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 msgr2=0x560b4429db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.852+0000 7f86075bd740 1 --2- 172.21.15.50:0/3048121062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 0x560b4429db50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f85f8011810 tx=0x7f85f800b440 comp rx=0 tx=0).stop 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.852+0000 7f86075bd740 1 -- 172.21.15.50:0/3048121062 shutdown_connections 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.852+0000 7f86075bd740 1 --2- 172.21.15.50:0/3048121062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 0x560b4429db50 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.852+0000 7f86075bd740 1 --2- 172.21.15.50:0/3048121062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44294d50 0x560b442a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.852+0000 7f86075bd740 1 --2- 172.21.15.50:0/3048121062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44293b80 0x560b44295130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.852+0000 7f86075bd740 1 -- 172.21.15.50:0/3048121062 >> 172.21.15.50:0/3048121062 conn(0x560b4428f070 msgr2=0x560b44298800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.852+0000 7f86075bd740 1 -- 172.21.15.50:0/3048121062 shutdown_connections 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.852+0000 7f86075bd740 1 -- 172.21.15.50:0/3048121062 wait complete. 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.852+0000 7f86075bd740 1 -- start start 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.852+0000 7f86075bd740 1 Processor -- start 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.852+0000 7f86075bd740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44293b80 0x560b442a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f86075bd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44294d50 0x560b442a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f86075bd740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 0x560b442a4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f86075bd740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560b442ac270 con 0x560b44295710 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f86075bd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560b442abba0 con 0x560b44294d50 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f86075bd740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560b442ac940 con 0x560b44293b80 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f86075bd740 10 client.0 _ll_register_callbacks cb 0x7f85f4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85fed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 0x560b442a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85fed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 0x560b442a4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57532/0 (socket says 172.21.15.50:57532) 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85fed9d640 1 -- 172.21.15.50:0/1734945639 learned_addr learned my addr 172.21.15.50:0/1734945639 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85fdd9b640 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44294d50 0x560b442a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85fe59c640 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44293b80 0x560b442a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85fed9d640 1 -- 172.21.15.50:0/1734945639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44293b80 msgr2=0x560b442a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85fed9d640 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44293b80 0x560b442a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85fed9d640 1 -- 172.21.15.50:0/1734945639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44294d50 msgr2=0x560b442a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85fed9d640 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44294d50 0x560b442a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85fed9d640 1 -- 172.21.15.50:0/1734945639 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b442a4660 con 0x560b44295710 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85fdd9b640 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44294d50 0x560b442a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85fed9d640 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 0x560b442a4070 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f85f800bfd0 tx=0x7f85f8028610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a70b2b3900641a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85e5ffb640 1 -- 172.21.15.50:0/1734945639 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8011e50 con 0x560b44295710 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85e5ffb640 1 -- 172.21.15.50:0/1734945639 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f85f800f940 con 0x560b44295710 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f86075bd740 1 -- 172.21.15.50:0/1734945639 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85f8018040 con 0x560b44295710 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f85e5ffb640 1 -- 172.21.15.50:0/1734945639 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f85f800fb40 con 0x560b44295710 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f86075bd740 10 client.15237 fetch_fsmap learned FSMap version 20 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.853+0000 7f86075bd740 1 -- 172.21.15.50:0/1734945639 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560b44319740 con 0x560b44295710 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.854+0000 7f85e5ffb640 1 -- 172.21.15.50:0/1734945639 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f8018040 con 0x560b44295710 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.854+0000 7f85e5ffb640 1 -- 172.21.15.50:0/1734945639 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f85f80049a0 con 0x560b44295710 2024-04-23T18:18:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.854+0000 7f86075bd740 10 client.15237 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.854+0000 7f86075bd740 10 client.15237 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.854+0000 7f86075bd740 20 client.15237 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.854+0000 7f86075bd740 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560b4431df70 0x560b4431cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.854+0000 7f86075bd740 4 client.15237 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.854+0000 7f86075bd740 1 -- 172.21.15.50:0/1734945639 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "status"}) -- 0x560b442004e0 con 0x560b4431df70 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.854+0000 7f85fe59c640 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560b4431df70 0x560b4431cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f85e5ffb640 10 client.15237 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f85fe59c640 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560b4431df70 0x560b4431cc80 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2a19ff2b794079b server_cookie=99d0cd27cf592ee2 in_seq=0 out_seq=0 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f85e5ffb640 1 -- 172.21.15.50:0/1734945639 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x560b442004e0 con 0x560b4431df70 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f85e5ffb640 10 client.15237 handle_command_reply: tid=0 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86075bd740 1 client.15237 shutdown 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86075bd740 1 -- 172.21.15.50:0/1734945639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 msgr2=0x560b442a4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86075bd740 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 0x560b442a4070 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f85f800bfd0 tx=0x7f85f8028610 comp rx=0 tx=0).stop 2024-04-23T18:18:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86075bd740 1 -- 172.21.15.50:0/1734945639 shutdown_connections 2024-04-23T18:18:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86075bd740 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560b4431df70 0x560b4431cc80 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86075bd740 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560b44295710 0x560b442a4070 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86075bd740 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b44294d50 0x560b442a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86075bd740 1 --2- 172.21.15.50:0/1734945639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560b44293b80 0x560b442a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f85e5ffb640 0 client.15237 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86075bd740 1 -- 172.21.15.50:0/1734945639 >> 172.21.15.50:0/1734945639 conn(0x560b4428f070 msgr2=0x560b44298800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86075bd740 1 -- 172.21.15.50:0/1734945639 shutdown_connections 2024-04-23T18:18:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86075bd740 1 -- 172.21.15.50:0/1734945639 wait complete. 2024-04-23T18:18:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86075bd740 20 client.15237 trim_cache size 0 max 16384 2024-04-23T18:18:21.856 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:18:21.856 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:18:21.857 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2024-04-23T18:18:21.857 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 24506, 2024-04-23T18:18:21.857 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2024-04-23T18:18:21.857 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2024-04-23T18:18:21.857 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2024-04-23T18:18:21.857 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 107.888137848, 2024-04-23T18:18:21.857 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 20, 2024-04-23T18:18:21.857 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 89, 2024-04-23T18:18:21.857 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 87, 2024-04-23T18:18:21.857 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 113.057253649 2024-04-23T18:18:21.857 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:18:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86048a9640 1 -- 172.21.15.50:0/3055426735 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f85e80a3bb0 msgr2=0x7f85e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.855+0000 7f86048a9640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f85e80a3bb0 0x7f85e80a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f85f80058c0 tx=0x7f85f8004730 comp rx=0 tx=0).stop 2024-04-23T18:18:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.856+0000 7f86048a9640 1 -- 172.21.15.50:0/3055426735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8600100e90 msgr2=0x7f86001981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.856+0000 7f86048a9640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8600100e90 0x7f86001981d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f85f000cce0 tx=0x7f85f000a620 comp rx=0 tx=0).stop 2024-04-23T18:18:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.856+0000 7f86048a9640 1 -- 172.21.15.50:0/3055426735 shutdown_connections 2024-04-23T18:18:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.856+0000 7f86048a9640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f85e80a3bb0 0x7f85e80a6070 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.856+0000 7f86048a9640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86001029a0 0x7f860019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.856+0000 7f86048a9640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8600101fe0 0x7f8600198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.856+0000 7f86048a9640 1 --2- 172.21.15.50:0/3055426735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8600100e90 0x7f86001981d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.856+0000 7f86048a9640 1 -- 172.21.15.50:0/3055426735 >> 172.21.15.50:0/3055426735 conn(0x7f86000fc400 msgr2=0x7f8600105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.856+0000 7f86048a9640 1 -- 172.21.15.50:0/3055426735 shutdown_connections 2024-04-23T18:18:21.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.856+0000 7f86048a9640 1 -- 172.21.15.50:0/3055426735 wait complete. 2024-04-23T18:18:21.860 INFO:teuthology.run_tasks:Running task sequential... 2024-04-23T18:18:21.872 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-23T18:18:21.872 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-23T18:18:21.873 INFO:teuthology.task.print:Enabling mgr modules 2024-04-23T18:18:21.873 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-23T18:18:21.873 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:18:21.874 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi050.front.sepia.ceph.com 2024-04-23T18:18:21.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-23T18:18:21.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.981+0000 7f31c598e640 1 -- 172.21.15.50:0/1394655371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c01053c0 msgr2=0x7f31c01077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.981+0000 7f31c598e640 1 --2- 172.21.15.50:0/1394655371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c01053c0 0x7f31c01077b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009900 tx=0x7f31a802a410 comp rx=0 tx=0).stop 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.981+0000 7f31c598e640 1 -- 172.21.15.50:0/1394655371 shutdown_connections 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.981+0000 7f31c598e640 1 --2- 172.21.15.50:0/1394655371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31c0107cf0 0x7f31c010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.981+0000 7f31c598e640 1 --2- 172.21.15.50:0/1394655371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c01053c0 0x7f31c01077b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.981+0000 7f31c598e640 1 --2- 172.21.15.50:0/1394655371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31c0102a90 0x7f31c0104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.981+0000 7f31c598e640 1 -- 172.21.15.50:0/1394655371 >> 172.21.15.50:0/1394655371 conn(0x7f31c00fc420 msgr2=0x7f31c00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.981+0000 7f31c598e640 1 -- 172.21.15.50:0/1394655371 shutdown_connections 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.981+0000 7f31c598e640 1 -- 172.21.15.50:0/1394655371 wait complete. 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.982+0000 7f31c598e640 1 Processor -- start 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.982+0000 7f31c598e640 1 -- start start 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.982+0000 7f31c598e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31c0102a90 0x7f31c01981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.982+0000 7f31c598e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c01053c0 0x7f31c01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.982+0000 7f31c598e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31c0107cf0 0x7f31c019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.982+0000 7f31c598e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f31c010daf0 con 0x7f31c01053c0 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.982+0000 7f31c598e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f31c010d970 con 0x7f31c0102a90 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.982+0000 7f31c598e640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f31c010dc70 con 0x7f31c0107cf0 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.982+0000 7f31be7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c01053c0 0x7f31c01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.982+0000 7f31be7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c01053c0 0x7f31c01986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57552/0 (socket says 172.21.15.50:57552) 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.982+0000 7f31be7fc640 1 -- 172.21.15.50:0/1430763869 learned_addr learned my addr 172.21.15.50:0/1430763869 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.982+0000 7f31beffd640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31c0102a90 0x7f31c01981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31bf7fe640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31c0107cf0 0x7f31c019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31be7fc640 1 -- 172.21.15.50:0/1430763869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31c0107cf0 msgr2=0x7f31c019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31be7fc640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31c0107cf0 0x7f31c019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31be7fc640 1 -- 172.21.15.50:0/1430763869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31c0102a90 msgr2=0x7f31c01981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31be7fc640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31c0102a90 0x7f31c01981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31be7fc640 1 -- 172.21.15.50:0/1430763869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c019fdc0 con 0x7f31c01053c0 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31bf7fe640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31c0107cf0 0x7f31c019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:21.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31be7fc640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c01053c0 0x7f31c01986e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f31a8006010 tx=0x7f31a80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31c498c640 1 -- 172.21.15.50:0/1430763869 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a803f030 con 0x7f31c01053c0 2024-04-23T18:18:21.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31c598e640 1 -- 172.21.15.50:0/1430763869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c01a0050 con 0x7f31c01053c0 2024-04-23T18:18:21.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31c498c640 1 -- 172.21.15.50:0/1430763869 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31a8031910 con 0x7f31c01053c0 2024-04-23T18:18:21.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31c498c640 1 -- 172.21.15.50:0/1430763869 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a8035070 con 0x7f31c01053c0 2024-04-23T18:18:21.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.983+0000 7f31c598e640 1 -- 172.21.15.50:0/1430763869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c01a0590 con 0x7f31c01053c0 2024-04-23T18:18:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.984+0000 7f31c598e640 1 -- 172.21.15.50:0/1430763869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31c00fdf10 con 0x7f31c01053c0 2024-04-23T18:18:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.985+0000 7f31c498c640 1 -- 172.21.15.50:0/1430763869 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f31a8042030 con 0x7f31c01053c0 2024-04-23T18:18:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.986+0000 7f31c498c640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f31ac0a3c50 0x7f31ac0a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.986+0000 7f31beffd640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f31ac0a3c50 0x7f31ac0a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.986+0000 7f31c498c640 1 -- 172.21.15.50:0/1430763869 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f31a80f09a0 con 0x7f31c01053c0 2024-04-23T18:18:21.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.986+0000 7f31beffd640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f31ac0a3c50 0x7f31ac0a6110 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009bd0 tx=0x7f31b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:21.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:21.990+0000 7f31c498c640 1 -- 172.21.15.50:0/1430763869 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31a80bd380 con 0x7f31c01053c0 2024-04-23T18:18:22.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:22.231+0000 7f31c598e640 1 -- 172.21.15.50:0/1430763869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f31c0062b50 con 0x7f31c01053c0 2024-04-23T18:18:22.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:22 smithi050 ceph-mon[29187]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:18:22.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:22 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3461821456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:22.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:22 smithi177 ceph-mon[32308]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:18:22.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:22 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3461821456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:22.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:22 smithi188 ceph-mon[32694]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T18:18:22.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:22 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3461821456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:23.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.153+0000 7f31c498c640 1 -- 172.21.15.50:0/1430763869 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f31a8003e60 con 0x7f31c01053c0 2024-04-23T18:18:23.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.156+0000 7f31c598e640 1 -- 172.21.15.50:0/1430763869 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f31ac0a3c50 msgr2=0x7f31ac0a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:23.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.156+0000 7f31c598e640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f31ac0a3c50 0x7f31ac0a6110 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009bd0 tx=0x7f31b8005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:23.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.156+0000 7f31c598e640 1 -- 172.21.15.50:0/1430763869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c01053c0 msgr2=0x7f31c01986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:23.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.156+0000 7f31c598e640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c01053c0 0x7f31c01986e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f31a8006010 tx=0x7f31a80042f0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.157+0000 7f31c598e640 1 -- 172.21.15.50:0/1430763869 shutdown_connections 2024-04-23T18:18:23.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.157+0000 7f31c598e640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f31ac0a3c50 0x7f31ac0a6110 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.157+0000 7f31c598e640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f31c0107cf0 0x7f31c019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.157+0000 7f31c598e640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c01053c0 0x7f31c01986e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.157+0000 7f31c598e640 1 --2- 172.21.15.50:0/1430763869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31c0102a90 0x7f31c01981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.157+0000 7f31c598e640 1 -- 172.21.15.50:0/1430763869 >> 172.21.15.50:0/1430763869 conn(0x7f31c00fc420 msgr2=0x7f31c0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:23.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.158+0000 7f31c598e640 1 -- 172.21.15.50:0/1430763869 shutdown_connections 2024-04-23T18:18:23.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.158+0000 7f31c598e640 1 -- 172.21.15.50:0/1430763869 wait complete. 2024-04-23T18:18:23.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-23T18:18:23.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.285+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2823962599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7cc105360 msgr2=0x7fc7cc107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:23.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.285+0000 7fc7d2ca0640 1 --2- 172.21.15.50:0/2823962599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7cc105360 0x7fc7cc107750 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8009900 tx=0x7fc7b802a410 comp rx=0 tx=0).stop 2024-04-23T18:18:23.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.285+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2823962599 shutdown_connections 2024-04-23T18:18:23.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.285+0000 7fc7d2ca0640 1 --2- 172.21.15.50:0/2823962599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7cc107c90 0x7fc7cc10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.285+0000 7fc7d2ca0640 1 --2- 172.21.15.50:0/2823962599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7cc105360 0x7fc7cc107750 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.285+0000 7fc7d2ca0640 1 --2- 172.21.15.50:0/2823962599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7cc102a30 0x7fc7cc104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.285+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2823962599 >> 172.21.15.50:0/2823962599 conn(0x7fc7cc0fc420 msgr2=0x7fc7cc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.285+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2823962599 shutdown_connections 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.285+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2823962599 wait complete. 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d2ca0640 1 Processor -- start 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d2ca0640 1 -- start start 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d2ca0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7cc102a30 0x7fc7cc198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d2ca0640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7cc105360 0x7fc7cc1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d2ca0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7cc107c90 0x7fc7cc19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d2ca0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc10daf0 con 0x7fc7cc107c90 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d2ca0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc10d970 con 0x7fc7cc102a30 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d2ca0640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc10dc70 con 0x7fc7cc105360 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d123d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7cc107c90 0x7fc7cc19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d123d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7cc107c90 0x7fc7cc19f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57584/0 (socket says 172.21.15.50:57584) 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d123d640 1 -- 172.21.15.50:0/2469689065 learned_addr learned my addr 172.21.15.50:0/2469689065 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d0a3c640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7cc102a30 0x7fc7cc198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7cbfff640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7cc105360 0x7fc7cc1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d123d640 1 -- 172.21.15.50:0/2469689065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7cc105360 msgr2=0x7fc7cc1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d123d640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7cc105360 0x7fc7cc1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d123d640 1 -- 172.21.15.50:0/2469689065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7cc102a30 msgr2=0x7fc7cc198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d123d640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7cc102a30 0x7fc7cc198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.286+0000 7fc7d123d640 1 -- 172.21.15.50:0/2469689065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7cc19fda0 con 0x7fc7cc107c90 2024-04-23T18:18:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.287+0000 7fc7d123d640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7cc107c90 0x7fc7cc19f730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009fd0 tx=0x7fc7c400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.287+0000 7fc7c9ffb640 1 -- 172.21.15.50:0/2469689065 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4017070 con 0x7fc7cc107c90 2024-04-23T18:18:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.287+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2469689065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7cc1a0090 con 0x7fc7cc107c90 2024-04-23T18:18:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.287+0000 7fc7c9ffb640 1 -- 172.21.15.50:0/2469689065 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7c4003e20 con 0x7fc7cc107c90 2024-04-23T18:18:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.287+0000 7fc7c9ffb640 1 -- 172.21.15.50:0/2469689065 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4010460 con 0x7fc7cc107c90 2024-04-23T18:18:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.287+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2469689065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7cc1a0560 con 0x7fc7cc107c90 2024-04-23T18:18:23.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.288+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2469689065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7cc0fdf10 con 0x7fc7cc107c90 2024-04-23T18:18:23.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.289+0000 7fc7c9ffb640 1 -- 172.21.15.50:0/2469689065 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fc7c4003970 con 0x7fc7cc107c90 2024-04-23T18:18:23.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.290+0000 7fc7c9ffb640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc7bc0a3ca0 0x7fc7bc0a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:23.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.290+0000 7fc7d0a3c640 1 -- 172.21.15.50:0/2469689065 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc7bc0a3ca0 msgr2=0x7fc7bc0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1004018784 2024-04-23T18:18:23.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.290+0000 7fc7c9ffb640 1 -- 172.21.15.50:0/2469689065 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fc7c40d01b0 con 0x7fc7cc107c90 2024-04-23T18:18:23.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.290+0000 7fc7d0a3c640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc7bc0a3ca0 0x7fc7bc0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:18:23.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.294+0000 7fc7c9ffb640 1 -- 172.21.15.50:0/2469689065 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc7c409c500 con 0x7fc7cc107c90 2024-04-23T18:18:23.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:23 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1430763869' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T18:18:23.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:23 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: ignoring --setuser ceph since I am not root 2024-04-23T18:18:23.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:23 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: ignoring --setgroup ceph since I am not root 2024-04-23T18:18:23.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:23 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:23.347+0000 7f0f9aaff200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:18:23.478 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.484+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2469689065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fc7cc062b50 con 0x7fc7cc107c90 2024-04-23T18:18:23.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.488+0000 7fc7c9ffb640 1 -- 172.21.15.50:0/2469689065 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7fc7c40a0800 con 0x7fc7cc107c90 2024-04-23T18:18:23.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.490+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2469689065 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc7bc0a3ca0 msgr2=0x7fc7bc0a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:18:23.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.490+0000 7fc7d2ca0640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc7bc0a3ca0 0x7fc7bc0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.490+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2469689065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7cc107c90 msgr2=0x7fc7cc19f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:23.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.490+0000 7fc7d2ca0640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7cc107c90 0x7fc7cc19f730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009fd0 tx=0x7fc7c400b260 comp rx=0 tx=0).stop 2024-04-23T18:18:23.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.490+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2469689065 shutdown_connections 2024-04-23T18:18:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.490+0000 7fc7d2ca0640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fc7bc0a3ca0 0x7fc7bc0a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.490+0000 7fc7d2ca0640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7cc107c90 0x7fc7cc19f730 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.490+0000 7fc7d2ca0640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7cc105360 0x7fc7cc1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.490+0000 7fc7d2ca0640 1 --2- 172.21.15.50:0/2469689065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7cc102a30 0x7fc7cc198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.490+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2469689065 >> 172.21.15.50:0/2469689065 conn(0x7fc7cc0fc420 msgr2=0x7fc7cc103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.490+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2469689065 shutdown_connections 2024-04-23T18:18:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.490+0000 7fc7d2ca0640 1 -- 172.21.15.50:0/2469689065 wait complete. 2024-04-23T18:18:23.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-23T18:18:23.513 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:23 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1430763869' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T18:18:23.513 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: ignoring --setuser ceph since I am not root 2024-04-23T18:18:23.513 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: ignoring --setgroup ceph since I am not root 2024-04-23T18:18:23.513 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:23 smithi188 ceph-mgr[34510]: -- 172.21.15.188:0/2104269038 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5581d50d85a0 con 0x5581d50ac800 2024-04-23T18:18:23.513 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:23.334+0000 7fb69908c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:18:23.513 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:23.518+0000 7fb69908c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:18:23.517 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:23 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1430763869' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T18:18:23.517 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: ignoring --setuser ceph since I am not root 2024-04-23T18:18:23.518 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: ignoring --setgroup ceph since I am not root 2024-04-23T18:18:23.518 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:23 smithi177 ceph-mgr[33733]: -- 172.21.15.177:0/506479657 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556907dba5a0 con 0x556907d8e800 2024-04-23T18:18:23.518 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:23.339+0000 7fe169b27200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T18:18:23.518 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:23.524+0000 7fe169b27200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:18:23.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.606+0000 7f307095d640 1 -- 172.21.15.50:0/1231358064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c102040 msgr2=0x7f306c1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:23.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.606+0000 7f307095d640 1 --2- 172.21.15.50:0/1231358064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c102040 0x7f306c1024c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f305c009900 tx=0x7f305c02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:23.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.606+0000 7f307095d640 1 -- 172.21.15.50:0/1231358064 shutdown_connections 2024-04-23T18:18:23.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.606+0000 7f307095d640 1 --2- 172.21.15.50:0/1231358064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f306c102a00 0x7f306c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.606+0000 7f307095d640 1 --2- 172.21.15.50:0/1231358064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c102040 0x7f306c1024c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.606+0000 7f307095d640 1 --2- 172.21.15.50:0/1231358064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f306c100e60 0x7f306c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.606+0000 7f307095d640 1 -- 172.21.15.50:0/1231358064 >> 172.21.15.50:0/1231358064 conn(0x7f306c0fc420 msgr2=0x7f306c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:23.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.606+0000 7f307095d640 1 -- 172.21.15.50:0/1231358064 shutdown_connections 2024-04-23T18:18:23.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.606+0000 7f307095d640 1 -- 172.21.15.50:0/1231358064 wait complete. 2024-04-23T18:18:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.606+0000 7f307095d640 1 Processor -- start 2024-04-23T18:18:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f307095d640 1 -- start start 2024-04-23T18:18:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f307095d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f306c100e60 0x7f306c06a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f307095d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f306c102040 0x7f306c19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f307095d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c102a00 0x7f306c19f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f307095d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f306c10ca50 con 0x7f306c102a00 2024-04-23T18:18:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f307095d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f306c10c8d0 con 0x7f306c102040 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f307095d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f306c10cbd0 con 0x7f306c100e60 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f306ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c102a00 0x7f306c19f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f306ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c102a00 0x7f306c19f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57590/0 (socket says 172.21.15.50:57590) 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f306ad9d640 1 -- 172.21.15.50:0/4202502219 learned_addr learned my addr 172.21.15.50:0/4202502219 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f306a59c640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f306c100e60 0x7f306c06a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f3069d9b640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f306c102040 0x7f306c19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f306ad9d640 1 -- 172.21.15.50:0/4202502219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f306c100e60 msgr2=0x7f306c06a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f306ad9d640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f306c100e60 0x7f306c06a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f306ad9d640 1 -- 172.21.15.50:0/4202502219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f306c102040 msgr2=0x7f306c19ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f306ad9d640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f306c102040 0x7f306c19ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f306ad9d640 1 -- 172.21.15.50:0/4202502219 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f306c19fd70 con 0x7f306c102a00 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f306ad9d640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c102a00 0x7f306c19f4b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f306400eb10 tx=0x7f306400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.607+0000 7f305b7fe640 1 -- 172.21.15.50:0/4202502219 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064004b20 con 0x7f306c102a00 2024-04-23T18:18:23.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.608+0000 7f305b7fe640 1 -- 172.21.15.50:0/4202502219 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3064004cc0 con 0x7f306c102a00 2024-04-23T18:18:23.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.608+0000 7f307095d640 1 -- 172.21.15.50:0/4202502219 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f306c1a0060 con 0x7f306c102a00 2024-04-23T18:18:23.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.608+0000 7f305b7fe640 1 -- 172.21.15.50:0/4202502219 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064010670 con 0x7f306c102a00 2024-04-23T18:18:23.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.608+0000 7f307095d640 1 -- 172.21.15.50:0/4202502219 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f306c1a05d0 con 0x7f306c102a00 2024-04-23T18:18:23.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.609+0000 7f307095d640 1 -- 172.21.15.50:0/4202502219 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f306c108200 con 0x7f306c102a00 2024-04-23T18:18:23.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.611+0000 7f305b7fe640 1 -- 172.21.15.50:0/4202502219 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f3064005020 con 0x7f306c102a00 2024-04-23T18:18:23.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.611+0000 7f305b7fe640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f30540a3d20 0x7f30540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:23.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.611+0000 7f306a59c640 1 -- 172.21.15.50:0/4202502219 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f30540a3d20 msgr2=0x7f30540a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1004018784 2024-04-23T18:18:23.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.611+0000 7f305b7fe640 1 -- 172.21.15.50:0/4202502219 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f30640d0900 con 0x7f306c102a00 2024-04-23T18:18:23.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.611+0000 7f306a59c640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f30540a3d20 0x7f30540a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:18:23.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.616+0000 7f305b7fe640 1 -- 172.21.15.50:0/4202502219 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f306409d2e0 con 0x7f306c102a00 2024-04-23T18:18:23.787 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:23.626+0000 7fe169b27200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:18:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.804+0000 7f307095d640 1 -- 172.21.15.50:0/4202502219 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f306c062bc0 con 0x7f306c102a00 2024-04-23T18:18:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.807+0000 7f305b7fe640 1 -- 172.21.15.50:0/4202502219 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7f306409cbe0 con 0x7f306c102a00 2024-04-23T18:18:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.809+0000 7f307095d640 1 -- 172.21.15.50:0/4202502219 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f30540a3d20 msgr2=0x7f30540a61e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:18:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.809+0000 7f307095d640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f30540a3d20 0x7f30540a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.809+0000 7f307095d640 1 -- 172.21.15.50:0/4202502219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c102a00 msgr2=0x7f306c19f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.809+0000 7f307095d640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c102a00 0x7f306c19f4b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f306400eb10 tx=0x7f306400c2d0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.809+0000 7f307095d640 1 -- 172.21.15.50:0/4202502219 shutdown_connections 2024-04-23T18:18:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.809+0000 7f307095d640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f30540a3d20 0x7f30540a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.809+0000 7f307095d640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c102a00 0x7f306c19f4b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.809+0000 7f307095d640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f306c102040 0x7f306c19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.809+0000 7f307095d640 1 --2- 172.21.15.50:0/4202502219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f306c100e60 0x7f306c06a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.809+0000 7f307095d640 1 -- 172.21.15.50:0/4202502219 >> 172.21.15.50:0/4202502219 conn(0x7f306c0fc420 msgr2=0x7f306c1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.810+0000 7f307095d640 1 -- 172.21.15.50:0/4202502219 shutdown_connections 2024-04-23T18:18:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:23.810+0000 7f307095d640 1 -- 172.21.15.50:0/4202502219 wait complete. 2024-04-23T18:18:23.814 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:23 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:23.543+0000 7f0f9aaff200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:18:23.814 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:23 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:23.654+0000 7f0f9aaff200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:18:23.814 INFO:teuthology.run_tasks:Running task kclient... 2024-04-23T18:18:23.840 INFO:tasks.kclient:Mounting kernel clients... 2024-04-23T18:18:23.840 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-23T18:18:23.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:18:23.841 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2024-04-23T18:18:23.841 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:23.621+0000 7fb69908c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T18:18:23.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:18:23.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link delete ceph-brx 2024-04-23T18:18:23.923 INFO:teuthology.orchestra.run.smithi050.stderr:Cannot find device "ceph-brx" 2024-04-23T18:18:23.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:18:23.925 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-23T18:18:23.925 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-23T18:18:23.925 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-23T18:18:23.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-23T18:18:24.066 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:23 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:23.981+0000 7f0f9aaff200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:18:24.066 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:24 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:24.072+0000 7f0f9aaff200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:18:24.218 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:18:24.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:24 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1430763869' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T18:18:24.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:24 smithi177 ceph-mon[32308]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-23T18:18:24.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:24 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2469689065' entity='client.admin' 2024-04-23T18:18:24.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:24 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4202502219' entity='client.admin' 2024-04-23T18:18:24.288 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:23.933+0000 7fe169b27200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:18:24.288 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:24 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:24.017+0000 7fe169b27200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:18:24.329 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:23.926+0000 7fb69908c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T18:18:24.330 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:24 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:24.010+0000 7fb69908c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T18:18:24.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:24 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1430763869' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T18:18:24.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:24 smithi188 ceph-mon[32694]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-23T18:18:24.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:24 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2469689065' entity='client.admin' 2024-04-23T18:18:24.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:24 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4202502219' entity='client.admin' 2024-04-23T18:18:24.341 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:24 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1430763869' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T18:18:24.341 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:24 smithi050 ceph-mon[29187]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-23T18:18:24.341 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:24 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2469689065' entity='client.admin' 2024-04-23T18:18:24.341 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:24 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4202502219' entity='client.admin' 2024-04-23T18:18:25.522 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:25 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:25.186+0000 7fb69908c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:18:25.522 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:25 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:25.356+0000 7fb69908c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:18:25.533 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:25 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:25.200+0000 7fe169b27200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:18:25.533 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:25 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:25.370+0000 7fe169b27200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:18:25.608 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:25 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:25.343+0000 7f0f9aaff200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T18:18:25.608 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:25 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:25.525+0000 7f0f9aaff200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T18:18:25.787 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:25 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:25.539+0000 7fe169b27200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:18:25.811 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:25 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:25.527+0000 7fb69908c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:18:25.886 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:25 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:25.704+0000 7f0f9aaff200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T18:18:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.914+0000 7fb340493700 1 -- 172.21.15.50:0/793765442 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb338108ab0 msgr2=0x7fb338108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.914+0000 7fb340493700 1 --2- 172.21.15.50:0/793765442 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb338108ab0 0x7fb338108e90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb334009990 tx=0x7fb334009ca0 comp rx=0 tx=0).stop 2024-04-23T18:18:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.914+0000 7fb340493700 1 -- 172.21.15.50:0/793765442 shutdown_connections 2024-04-23T18:18:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.914+0000 7fb340493700 1 --2- 172.21.15.50:0/793765442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb338103480 0x7fb33810f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.914+0000 7fb340493700 1 --2- 172.21.15.50:0/793765442 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb338108ab0 0x7fb338108e90 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.914+0000 7fb340493700 1 --2- 172.21.15.50:0/793765442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb338102ac0 0x7fb338102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.914+0000 7fb340493700 1 -- 172.21.15.50:0/793765442 >> 172.21.15.50:0/793765442 conn(0x7fb338079bf0 msgr2=0x7fb33807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.914+0000 7fb340493700 1 -- 172.21.15.50:0/793765442 shutdown_connections 2024-04-23T18:18:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.915+0000 7fb340493700 1 -- 172.21.15.50:0/793765442 wait complete. 2024-04-23T18:18:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.915+0000 7fb340493700 1 Processor -- start 2024-04-23T18:18:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.915+0000 7fb340493700 1 -- start start 2024-04-23T18:18:25.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.915+0000 7fb340493700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb338102ac0 0x7fb338199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:25.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.915+0000 7fb340493700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb338103480 0x7fb3381996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.915+0000 7fb340493700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb338108ab0 0x7fb33819da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.915+0000 7fb340493700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb338112cb0 con 0x7fb338108ab0 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.915+0000 7fb340493700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb338112b30 con 0x7fb338102ac0 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.915+0000 7fb340493700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb338112e30 con 0x7fb338103480 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33ea30700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb338108ab0 0x7fb33819da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33ea30700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb338108ab0 0x7fb33819da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57616/0 (socket says 172.21.15.50:57616) 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33ea30700 1 -- 172.21.15.50:0/2085675748 learned_addr learned my addr 172.21.15.50:0/2085675748 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33e22f700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb338102ac0 0x7fb338199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33da2e700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb338103480 0x7fb3381996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33e22f700 1 -- 172.21.15.50:0/2085675748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb338103480 msgr2=0x7fb3381996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33e22f700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb338103480 0x7fb3381996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33e22f700 1 -- 172.21.15.50:0/2085675748 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb338108ab0 msgr2=0x7fb33819da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33e22f700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb338108ab0 0x7fb33819da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:25.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33e22f700 1 -- 172.21.15.50:0/2085675748 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb33819e170 con 0x7fb338102ac0 2024-04-23T18:18:25.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33ea30700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb338108ab0 0x7fb33819da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:25.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33da2e700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb338103480 0x7fb3381996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:25.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb33e22f700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb338102ac0 0x7fb338199160 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb3340059d0 tx=0x7fb3340047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:25.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb32f7fe700 1 -- 172.21.15.50:0/2085675748 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334027070 con 0x7fb338102ac0 2024-04-23T18:18:25.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb32f7fe700 1 -- 172.21.15.50:0/2085675748 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb334022470 con 0x7fb338102ac0 2024-04-23T18:18:25.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.916+0000 7fb340493700 1 -- 172.21.15.50:0/2085675748 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb33819e400 con 0x7fb338102ac0 2024-04-23T18:18:25.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.917+0000 7fb32f7fe700 1 -- 172.21.15.50:0/2085675748 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334017650 con 0x7fb338102ac0 2024-04-23T18:18:25.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.917+0000 7fb340493700 1 -- 172.21.15.50:0/2085675748 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3381a5ce0 con 0x7fb338102ac0 2024-04-23T18:18:25.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.917+0000 7fb340493700 1 -- 172.21.15.50:0/2085675748 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb33810b040 con 0x7fb338102ac0 2024-04-23T18:18:25.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.919+0000 7fb32f7fe700 1 -- 172.21.15.50:0/2085675748 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fb33402a070 con 0x7fb338102ac0 2024-04-23T18:18:25.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.919+0000 7fb32f7fe700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3240a3c90 0x7fb3240a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:25.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.919+0000 7fb33da2e700 1 -- 172.21.15.50:0/2085675748 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3240a3c90 msgr2=0x7fb3240a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1004018784 2024-04-23T18:18:25.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.919+0000 7fb32f7fe700 1 -- 172.21.15.50:0/2085675748 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fb3340d1200 con 0x7fb338102ac0 2024-04-23T18:18:25.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.919+0000 7fb33da2e700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3240a3c90 0x7fb3240a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:18:25.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:25.923+0000 7fb32f7fe700 1 -- 172.21.15.50:0/2085675748 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb334013070 con 0x7fb338102ac0 2024-04-23T18:18:26.079 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:25 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:25.817+0000 7fb69908c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:18:26.079 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:25 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:25.909+0000 7fb69908c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:18:26.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.120+0000 7fb33da2e700 1 -- 172.21.15.50:0/2085675748 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3240a3c90 msgr2=0x7fb3240a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1004018784 2024-04-23T18:18:26.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.120+0000 7fb33da2e700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3240a3c90 0x7fb3240a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:18:26.122 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:25 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:25.827+0000 7fe169b27200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:18:26.122 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:25 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:25.920+0000 7fe169b27200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:18:26.122 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:26 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:26.128+0000 7fe169b27200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:18:26.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.181+0000 7fb340493700 1 -- 172.21.15.50:0/2085675748 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb33819a4c0 con 0x7fb338102ac0 2024-04-23T18:18:26.175 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:26 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:26.019+0000 7f0f9aaff200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T18:18:26.175 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:26 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:26.117+0000 7f0f9aaff200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T18:18:26.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.182+0000 7fb32f7fe700 1 -- 172.21.15.50:0/2085675748 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) ==== 53+0+73 (secure 0 0 0) 0x7fb33409dde0 con 0x7fb338102ac0 2024-04-23T18:18:26.176 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-23T18:18:26.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.184+0000 7fb340493700 1 -- 172.21.15.50:0/2085675748 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3240a3c90 msgr2=0x7fb3240a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:18:26.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.184+0000 7fb340493700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3240a3c90 0x7fb3240a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:26.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.184+0000 7fb340493700 1 -- 172.21.15.50:0/2085675748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb338102ac0 msgr2=0x7fb338199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:26.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.184+0000 7fb340493700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb338102ac0 0x7fb338199160 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb3340059d0 tx=0x7fb3340047b0 comp rx=0 tx=0).stop 2024-04-23T18:18:26.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.184+0000 7fb340493700 1 -- 172.21.15.50:0/2085675748 shutdown_connections 2024-04-23T18:18:26.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.184+0000 7fb340493700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb338108ab0 0x7fb33819da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:26.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.184+0000 7fb340493700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7fb3240a3c90 0x7fb3240a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:26.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.184+0000 7fb340493700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb338102ac0 0x7fb338199160 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:26.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.184+0000 7fb340493700 1 --2- 172.21.15.50:0/2085675748 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb338103480 0x7fb3381996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:26.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.184+0000 7fb340493700 1 -- 172.21.15.50:0/2085675748 >> 172.21.15.50:0/2085675748 conn(0x7fb338079bf0 msgr2=0x7fb338100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:26.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.184+0000 7fb340493700 1 -- 172.21.15.50:0/2085675748 shutdown_connections 2024-04-23T18:18:26.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26.184+0000 7fb340493700 1 -- 172.21.15.50:0/2085675748 wait complete. 2024-04-23T18:18:26.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:26 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2085675748' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T18:18:26.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:26 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:26.333+0000 7f0f9aaff200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:18:26.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:26 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2085675748' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T18:18:26.538 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:26 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:26.323+0000 7fe169b27200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:18:26.579 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:26 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:26.116+0000 7fb69908c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T18:18:26.579 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:26 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:26.311+0000 7fb69908c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:18:26.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:26 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2085675748' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T18:18:26.740 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:26 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:26.537+0000 7f0f9aaff200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T18:18:26.778 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-23T18:18:26.778 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-23T18:18:26.778 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2024-04-23T18:18:26.778 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-23T18:18:26.778 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-23T18:18:26.778 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-23T18:18:26.778 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-23T18:18:26.778 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-23T18:18:26.778 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-23T18:18:26.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:18:26.779 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2024-04-23T18:18:26.795 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-23T18:18:26.795 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-23T18:18:26.795 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2024-04-23T18:18:26.795 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2024-04-23T18:18:26.795 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2024-04-23T18:18:26.795 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2024-04-23T18:18:26.795 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T18:18:26.795 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 42073sec preferred_lft 42073sec 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link noprefixroute 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2024-04-23T18:18:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2024-04-23T18:18:26.797 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-23T18:18:26.797 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:18:26.797 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2024-04-23T18:18:26.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link add name ceph-brx type bridge 2024-04-23T18:18:26.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip addr flush dev ceph-brx 2024-04-23T18:18:26.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set ceph-brx up 2024-04-23T18:18:26.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-23T18:18:26.797 DEBUG:teuthology.orchestra.run.smithi050:> ') 2024-04-23T18:18:26.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:18:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:18:27.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:18:27.004 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-23T18:18:27.030 INFO:teuthology.orchestra.run.smithi050.stdout:1 2024-04-23T18:18:27.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:18:27.032 DEBUG:teuthology.orchestra.run.smithi050:> route 2024-04-23T18:18:27.101 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2024-04-23T18:18:27.101 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-23T18:18:27.101 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-23T18:18:27.101 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-23T18:18:27.101 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-23T18:18:27.102 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:18:27.102 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2024-04-23T18:18:27.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-23T18:18:27.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-23T18:18:27.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-23T18:18:27.102 DEBUG:teuthology.orchestra.run.smithi050:> ') 2024-04-23T18:18:27.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:18:27.287 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:26 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:26.920+0000 7fe169b27200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:18:27.296 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:27 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:27.194+0000 7f0f9aaff200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:18:27.320 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:26 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:26.906+0000 7fb69908c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T18:18:27.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:18:27.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:18:27.377 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2024-04-23T18:18:27.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:18:27.394 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list-id 2024-04-23T18:18:27.450 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:18:27.450 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2024-04-23T18:18:27.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-23T18:18:27.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-23T18:18:27.451 DEBUG:teuthology.orchestra.run.smithi050:> ') 2024-04-23T18:18:27.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:18:27.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:18:27.574 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-23T18:18:27.574 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:18:27.575 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2024-04-23T18:18:27.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-23T18:18:27.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-23T18:18:27.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-23T18:18:27.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-23T18:18:27.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-23T18:18:27.575 DEBUG:teuthology.orchestra.run.smithi050:> ') 2024-04-23T18:18:27.579 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:27 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:27.327+0000 7fb69908c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:18:27.579 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:27 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:27.423+0000 7fb69908c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:18:27.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:18:27.630 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:27 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:27.342+0000 7fe169b27200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:18:27.630 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:27 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:27.438+0000 7fe169b27200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:18:27.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:18:27.706 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T18:18:27.706 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2024-04-23T18:18:27.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2024-04-23T18:18:27.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set dev brx.0 master ceph-brx 2024-04-23T18:18:27.706 DEBUG:teuthology.orchestra.run.smithi050:> ') 2024-04-23T18:18:27.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:18:27.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:18:27.784 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T18:18:27.784 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-23T18:18:27.800 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-23T18:18:27.800 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T18:18:27.800 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-23T18:18:27.854 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-23T18:18:27.854 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-23T18:18:27.855 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/0c4ee723-281e-4755-96b0-c936df51685c 2024-04-23T18:18:27.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:18:27.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/0c4ee723-281e-4755-96b0-c936df51685c /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-23T18:18:27.915 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:27 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:27.667+0000 7f0f9aaff200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T18:18:27.915 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:27 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:27.774+0000 7f0f9aaff200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T18:18:27.947 INFO:teuthology.orchestra.run.smithi050.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-23T18:18:27.948 INFO:teuthology.orchestra.run.smithi050.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-23T18:18:28.063 INFO:teuthology.orchestra.run.smithi050.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-23T18:18:28.064 INFO:teuthology.orchestra.run.smithi050.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-23T18:18:28.064 INFO:teuthology.orchestra.run.smithi050.stdout:mount.ceph: resolved to: "172.21.15.50:3300,172.21.15.177:3300,172.21.15.188:3300" 2024-04-23T18:18:28.064 INFO:teuthology.orchestra.run.smithi050.stdout:mount.ceph: trying mount with new device syntax: 0@5d677974-019c-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/0c4ee723-281e-4755-96b0-c936df51685c 2024-04-23T18:18:28.064 INFO:teuthology.orchestra.run.smithi050.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.50:3300/172.21.15.177:3300/172.21.15.188:3300" will pass to kernel 2024-04-23T18:18:28.064 INFO:teuthology.orchestra.run.smithi050.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-23T18:18:28.065 INFO:teuthology.orchestra.run.smithi050.stdout: You just mounted a file system that supports labels which does not 2024-04-23T18:18:28.065 INFO:teuthology.orchestra.run.smithi050.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-23T18:18:28.065 INFO:teuthology.orchestra.run.smithi050.stdout: applications will generate AVC messages and not be allowed access to 2024-04-23T18:18:28.065 INFO:teuthology.orchestra.run.smithi050.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-23T18:18:28.066 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-23T18:18:28.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:18:28.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-23T18:18:28.079 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:27 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:27.623+0000 7fb69908c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:18:28.134 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T18:18:28.194 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:27 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:27.993+0000 7f0f9aaff200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:18:28.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T18:18:28.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:18:28.235 INFO:teuthology.orchestra.run.smithi050.stdout:client24956 2024-04-23T18:18:28.235 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:18:28.235 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T18:18:28.293 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T18:18:28.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T18:18:28.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:18:28.389 INFO:teuthology.orchestra.run.smithi050.stdout:client24956 2024-04-23T18:18:28.390 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T18:18:28.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T18:18:28.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:18:28.448 INFO:teuthology.orchestra.run.smithi050.stdout:client24956 2024-04-23T18:18:28.448 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/5d677974-019c-11ef-bc93-c7b262605968.client24956/status') 2024-04-23T18:18:28.474 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:28.148+0000 7fb69908c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:18:28.474 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:28.239+0000 7fb69908c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:18:28.492 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:27 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:27.637+0000 7fe169b27200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T18:18:28.492 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:28 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:28.164+0000 7fe169b27200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:18:28.492 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:28 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:28.257+0000 7fe169b27200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:18:28.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T18:18:28.508 INFO:teuthology.orchestra.run.smithi050.stdout:instance: client.24956 (3)172.21.15.50:0/2078406337 2024-04-23T18:18:28.508 INFO:teuthology.orchestra.run.smithi050.stdout:blocklisted: false 2024-04-23T18:18:28.508 INFO:teuthology.orchestra.run.smithi050.stderr:0+1 records in 2024-04-23T18:18:28.508 INFO:teuthology.orchestra.run.smithi050.stderr:0+1 records out 2024-04-23T18:18:28.508 INFO:teuthology.orchestra.run.smithi050.stderr:71 bytes copied, 2.7532e-05 s, 2.6 MB/s 2024-04-23T18:18:28.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T18:18:28.516 INFO:teuthology.run_tasks:Running task exec... 2024-04-23T18:18:28.525 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:18:28.526 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi050.front.sepia.ceph.com 2024-04-23T18:18:28.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-23T18:18:28.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.636+0000 7f728415d640 1 -- 172.21.15.50:0/3976004368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f727c102a00 msgr2=0x7f727c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:28.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.636+0000 7f728415d640 1 --2- 172.21.15.50:0/3976004368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f727c102a00 0x7f727c109320 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7274009ca0 tx=0x7f727402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.636+0000 7f728415d640 1 -- 172.21.15.50:0/3976004368 shutdown_connections 2024-04-23T18:18:28.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.636+0000 7f728415d640 1 --2- 172.21.15.50:0/3976004368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f727c102a00 0x7f727c109320 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.636+0000 7f728415d640 1 --2- 172.21.15.50:0/3976004368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f727c102040 0x7f727c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.636+0000 7f728415d640 1 --2- 172.21.15.50:0/3976004368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f727c100e60 0x7f727c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.636+0000 7f728415d640 1 -- 172.21.15.50:0/3976004368 >> 172.21.15.50:0/3976004368 conn(0x7f727c0fc420 msgr2=0x7f727c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:28.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.636+0000 7f728415d640 1 -- 172.21.15.50:0/3976004368 shutdown_connections 2024-04-23T18:18:28.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.636+0000 7f728415d640 1 -- 172.21.15.50:0/3976004368 wait complete. 2024-04-23T18:18:28.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.637+0000 7f728415d640 1 Processor -- start 2024-04-23T18:18:28.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.637+0000 7f728415d640 1 -- start start 2024-04-23T18:18:28.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.637+0000 7f728415d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f727c100e60 0x7f727c104e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:28.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.637+0000 7f728415d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f727c102040 0x7f727c1034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:28.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.637+0000 7f728415d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f727c102a00 0x7f727c1039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:28.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.637+0000 7f728415d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f727c10ca10 con 0x7f727c100e60 2024-04-23T18:18:28.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.637+0000 7f728415d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f727c10c890 con 0x7f727c102a00 2024-04-23T18:18:28.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.637+0000 7f728415d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f727c10cb90 con 0x7f727c102040 2024-04-23T18:18:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.637+0000 7f7281ef9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f727c100e60 0x7f727c104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.637+0000 7f7281ef9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f727c100e60 0x7f727c104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57626/0 (socket says 172.21.15.50:57626) 2024-04-23T18:18:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.637+0000 7f7281ef9640 1 -- 172.21.15.50:0/2333389762 learned_addr learned my addr 172.21.15.50:0/2333389762 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f72816f8640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f727c102040 0x7f727c1034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f72826fa640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f727c102a00 0x7f727c1039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f7281ef9640 1 -- 172.21.15.50:0/2333389762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f727c102040 msgr2=0x7f727c1034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f7281ef9640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f727c102040 0x7f727c1034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f7281ef9640 1 -- 172.21.15.50:0/2333389762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f727c102a00 msgr2=0x7f727c1039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f7281ef9640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f727c102a00 0x7f727c1039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f7281ef9640 1 -- 172.21.15.50:0/2333389762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727c1042b0 con 0x7f727c100e60 2024-04-23T18:18:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f7281ef9640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f727c100e60 0x7f727c104e20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f726c00e9c0 tx=0x7f726c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f7272ffd640 1 -- 172.21.15.50:0/2333389762 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c00cd80 con 0x7f727c100e60 2024-04-23T18:18:28.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f728415d640 1 -- 172.21.15.50:0/2333389762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f727c1a2260 con 0x7f727c100e60 2024-04-23T18:18:28.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f7272ffd640 1 -- 172.21.15.50:0/2333389762 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f726c0035f0 con 0x7f727c100e60 2024-04-23T18:18:28.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f7272ffd640 1 -- 172.21.15.50:0/2333389762 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c010690 con 0x7f727c100e60 2024-04-23T18:18:28.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.638+0000 7f728415d640 1 -- 172.21.15.50:0/2333389762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f727c1a2730 con 0x7f727c100e60 2024-04-23T18:18:28.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.640+0000 7f728415d640 1 -- 172.21.15.50:0/2333389762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f727c1a29f0 con 0x7f727c100e60 2024-04-23T18:18:28.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.640+0000 7f7272ffd640 1 -- 172.21.15.50:0/2333389762 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f726c004000 con 0x7f727c100e60 2024-04-23T18:18:28.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.641+0000 7f7272ffd640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f72680a3ca0 0x7f72680a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:28.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.641+0000 7f72816f8640 1 -- 172.21.15.50:0/2333389762 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f72680a3ca0 msgr2=0x7f72680a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1004018784 2024-04-23T18:18:28.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.641+0000 7f7272ffd640 1 -- 172.21.15.50:0/2333389762 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f726c0d0980 con 0x7f727c100e60 2024-04-23T18:18:28.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.641+0000 7f72816f8640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f72680a3ca0 0x7f72680a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:18:28.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.645+0000 7f7272ffd640 1 -- 172.21.15.50:0/2333389762 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f726c09d320 con 0x7f727c100e60 2024-04-23T18:18:28.787 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:28 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:28.499+0000 7fe169b27200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:18:28.787 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:28 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:28.631+0000 7fe169b27200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:18:28.788 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:28 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:28.727+0000 7fe169b27200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:18:28.818 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:28.481+0000 7fb69908c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:18:28.818 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:28.615+0000 7fb69908c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:18:28.818 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:28.710+0000 7fb69908c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:18:28.818 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:28.825+0000 7fb69908c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:18:28.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.842+0000 7f72816f8640 1 -- 172.21.15.50:0/2333389762 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f72680a3ca0 msgr2=0x7f72680a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1004018784 2024-04-23T18:18:28.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.842+0000 7f72816f8640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f72680a3ca0 0x7f72680a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:18:28.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.902+0000 7f728415d640 1 -- 172.21.15.50:0/2333389762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f727c1a2bc0 con 0x7f727c100e60 2024-04-23T18:18:28.933 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:28 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:28.556+0000 7f0f9aaff200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T18:18:28.934 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:28 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:28.667+0000 7f0f9aaff200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T18:18:28.934 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:28 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:28.939+0000 7f0f9aaff200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T18:18:28.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.954+0000 7f7272ffd640 1 -- 172.21.15.50:0/2333389762 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v21) ==== 117+0+0 (secure 0 0 0) 0x7f726c09cc20 con 0x7f727c100e60 2024-04-23T18:18:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.956+0000 7f728415d640 1 -- 172.21.15.50:0/2333389762 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f72680a3ca0 msgr2=0x7f72680a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:18:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.956+0000 7f728415d640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f72680a3ca0 0x7f72680a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.956+0000 7f728415d640 1 -- 172.21.15.50:0/2333389762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f727c100e60 msgr2=0x7f727c104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.956+0000 7f728415d640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f727c100e60 0x7f727c104e20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f726c00e9c0 tx=0x7f726c00ee80 comp rx=0 tx=0).stop 2024-04-23T18:18:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.956+0000 7f728415d640 1 -- 172.21.15.50:0/2333389762 shutdown_connections 2024-04-23T18:18:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.956+0000 7f728415d640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f72680a3ca0 0x7f72680a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.956+0000 7f728415d640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f727c102a00 0x7f727c1039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.956+0000 7f728415d640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f727c102040 0x7f727c1034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.956+0000 7f728415d640 1 --2- 172.21.15.50:0/2333389762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f727c100e60 0x7f727c104e20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.956+0000 7f728415d640 1 -- 172.21.15.50:0/2333389762 >> 172.21.15.50:0/2333389762 conn(0x7f727c0fc420 msgr2=0x7f727c0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.956+0000 7f728415d640 1 -- 172.21.15.50:0/2333389762 shutdown_connections 2024-04-23T18:18:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:28.956+0000 7f728415d640 1 -- 172.21.15.50:0/2333389762 wait complete. 2024-04-23T18:18:28.962 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-23T18:18:28.973 INFO:teuthology.run_tasks:Running task exec... 2024-04-23T18:18:28.985 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:18:28.985 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi050.front.sepia.ceph.com 2024-04-23T18:18:28.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-23T18:18:29.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:28 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2333389762' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-23T18:18:29.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.097+0000 7f712aa59640 1 -- 172.21.15.50:0/3653636326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124102a00 msgr2=0x7f7124109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:29.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.097+0000 7f712aa59640 1 --2- 172.21.15.50:0/3653636326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124102a00 0x7f7124109320 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f711c009ca0 tx=0x7f711c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 -- 172.21.15.50:0/3653636326 shutdown_connections 2024-04-23T18:18:29.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 --2- 172.21.15.50:0/3653636326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124102a00 0x7f7124109320 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 --2- 172.21.15.50:0/3653636326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7124102040 0x7f71241024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 --2- 172.21.15.50:0/3653636326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7124100e60 0x7f7124101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 -- 172.21.15.50:0/3653636326 >> 172.21.15.50:0/3653636326 conn(0x7f71240fc420 msgr2=0x7f71240fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 -- 172.21.15.50:0/3653636326 shutdown_connections 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 -- 172.21.15.50:0/3653636326 wait complete. 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 Processor -- start 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 -- start start 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7124100e60 0x7f7124198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7124102040 0x7f71241986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124102a00 0x7f712419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f712410ca00 con 0x7f7124102a00 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f712410c880 con 0x7f7124102040 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.098+0000 7f712aa59640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f712410cb80 con 0x7f7124100e60 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f7128ff6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124102a00 0x7f712419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f7128ff6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124102a00 0x7f712419f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57652/0 (socket says 172.21.15.50:57652) 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f7128ff6640 1 -- 172.21.15.50:0/486371419 learned_addr learned my addr 172.21.15.50:0/486371419 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f71237fe640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7124102040 0x7f71241986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f7123fff640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7124100e60 0x7f7124198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f7123fff640 1 -- 172.21.15.50:0/486371419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7124102040 msgr2=0x7f71241986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f7123fff640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7124102040 0x7f71241986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f7123fff640 1 -- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124102a00 msgr2=0x7f712419f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f7123fff640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124102a00 0x7f712419f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f7123fff640 1 -- 172.21.15.50:0/486371419 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f712419fd30 con 0x7f7124100e60 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f7128ff6640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124102a00 0x7f712419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f7123fff640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7124100e60 0x7f7124198160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f711400e9c0 tx=0x7f711400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f71217fa640 1 -- 172.21.15.50:0/486371419 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711400ccc0 con 0x7f7124100e60 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f71217fa640 1 -- 172.21.15.50:0/486371419 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71140035f0 con 0x7f7124100e60 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f712aa59640 1 -- 172.21.15.50:0/486371419 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f712419ffc0 con 0x7f7124100e60 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.099+0000 7f71217fa640 1 -- 172.21.15.50:0/486371419 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711400ce60 con 0x7f7124100e60 2024-04-23T18:18:29.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.100+0000 7f712aa59640 1 -- 172.21.15.50:0/486371419 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71241a0480 con 0x7f7124100e60 2024-04-23T18:18:29.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.100+0000 7f712aa59640 1 -- 172.21.15.50:0/486371419 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71241a0740 con 0x7f7124100e60 2024-04-23T18:18:29.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.102+0000 7f71217fa640 1 -- 172.21.15.50:0/486371419 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f7114004000 con 0x7f7124100e60 2024-04-23T18:18:29.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.102+0000 7f71217fa640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f710c0a3ca0 0x7f710c0a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:29.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.102+0000 7f71237fe640 1 -- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f710c0a3ca0 msgr2=0x7f710c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1004018784 2024-04-23T18:18:29.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.102+0000 7f71217fa640 1 -- 172.21.15.50:0/486371419 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7f71140d0a10 con 0x7f7124100e60 2024-04-23T18:18:29.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.102+0000 7f71237fe640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f710c0a3ca0 0x7f710c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T18:18:29.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.106+0000 7f71217fa640 1 -- 172.21.15.50:0/486371419 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7114043e20 con 0x7f7124100e60 2024-04-23T18:18:29.140 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:28 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2333389762' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-23T18:18:29.140 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:28 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:28.841+0000 7fe169b27200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:18:29.184 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:28 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2333389762' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-23T18:18:29.185 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:29 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:29.087+0000 7f0f9aaff200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T18:18:29.185 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:29 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:29.191+0000 7f0f9aaff200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T18:18:29.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.303+0000 7f71237fe640 1 -- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f710c0a3ca0 msgr2=0x7f710c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1004018784 2024-04-23T18:18:29.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.303+0000 7f71237fe640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f710c0a3ca0 0x7f710c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T18:18:29.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.319+0000 7f712aa59640 1 -- 172.21.15.50:0/486371419 --> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f71241a0910 con 0x7f710c0a3ca0 2024-04-23T18:18:29.444 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:29 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:29.315+0000 7f0f9aaff200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T18:18:29.538 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:29 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:29.147+0000 7fe169b27200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:18:29.579 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:29.131+0000 7fb69908c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:18:29.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.704+0000 7f71237fe640 1 -- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f710c0a3ca0 msgr2=0x7f710c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1004018784 2024-04-23T18:18:29.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.704+0000 7f71237fe640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f710c0a3ca0 0x7f710c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T18:18:29.944 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:29 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:29.628+0000 7f0f9aaff200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T18:18:29.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:29.970+0000 7f71217fa640 1 -- 172.21.15.50:0/486371419 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7f7114099360 con 0x7f7124100e60 2024-04-23T18:18:30.038 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:29 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:29.746+0000 7fe169b27200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:18:30.038 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:18:29 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y[33729]: 2024-04-23T18:18:29.960+0000 7fe169b27200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:18:30.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:29 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2333389762' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-23T18:18:30.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:29 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:30.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:29 smithi177 ceph-mon[32308]: Standby manager daemon z restarted 2024-04-23T18:18:30.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:29 smithi177 ceph-mon[32308]: Standby manager daemon z started 2024-04-23T18:18:30.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:29 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.188:0/233952861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:18:30.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:29 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.188:0/233952861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:18:30.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:29 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.188:0/233952861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:18:30.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:29 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.188:0/233952861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:18:30.079 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:29.727+0000 7fb69908c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:18:30.079 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:18:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z[34472]: 2024-04-23T18:18:29.942+0000 7fb69908c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:18:30.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:29 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2333389762' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-23T18:18:30.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:29 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:30.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:29 smithi188 ceph-mon[32694]: Standby manager daemon z restarted 2024-04-23T18:18:30.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:29 smithi188 ceph-mon[32694]: Standby manager daemon z started 2024-04-23T18:18:30.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:29 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.188:0/233952861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:18:30.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:29 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.188:0/233952861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:18:30.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:29 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.188:0/233952861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:18:30.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:29 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.188:0/233952861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:18:30.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:29 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2333389762' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-23T18:18:30.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:29 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:18:30.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:29 smithi050 ceph-mon[29187]: Standby manager daemon z restarted 2024-04-23T18:18:30.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:29 smithi050 ceph-mon[29187]: Standby manager daemon z started 2024-04-23T18:18:30.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:29 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.188:0/233952861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T18:18:30.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:29 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.188:0/233952861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:18:30.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:29 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.188:0/233952861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T18:18:30.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:29 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.188:0/233952861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:18:30.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:30.505+0000 7f71237fe640 1 -- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f710c0a3ca0 msgr2=0x7f710c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/1004018784 2024-04-23T18:18:30.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:30.505+0000 7f71237fe640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f710c0a3ca0 0x7f710c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T18:18:30.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:30.517+0000 7f71217fa640 1 -- 172.21.15.50:0/486371419 <== mon.1 v2:172.21.15.188:3300/0 8 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7f71140ce590 con 0x7f7124100e60 2024-04-23T18:18:30.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:30.518+0000 7f71217fa640 1 -- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f710c0a3ca0 msgr2=0x7f710c0a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T18:18:30.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:30.518+0000 7f71217fa640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/1004018784,v1:172.21.15.50:6801/1004018784] conn(0x7f710c0a3ca0 0x7f710c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:30.659 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:30.270+0000 7f0f9aaff200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T18:18:30.659 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:30.500+0000 7f0f9aaff200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T18:18:30.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:30.751+0000 7f0efcc38700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:30.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:30.751+0000 7f0efcc38700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:30.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:30.752+0000 7f0efcc38700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:30.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:30.752+0000 7f0efcc38700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:30.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:30.752+0000 7f0efcc38700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:30.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:30.755+0000 7f0ef9431700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:30.945 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:30.755+0000 7f0ef9431700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:30.946 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:30.755+0000 7f0ef9431700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:30.946 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:30.755+0000 7f0ef9431700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:30.946 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:18:30.755+0000 7f0ef9431700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:18:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: mgrmap e20: x(active, since 8m), standbys: y, z 2024-04-23T18:18:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.177:0/79381647' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:18:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: Standby manager daemon y restarted 2024-04-23T18:18:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: Standby manager daemon y started 2024-04-23T18:18:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.177:0/79381647' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:18:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.177:0/79381647' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:18:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.? 172.21.15.177:0/79381647' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:18:31.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: Active manager daemon x restarted 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: Activating manager daemon x 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: mgrmap e21: x(active, starting, since 0.0105277s), standbys: y, z 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:18:31.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:18:31.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:18:31.291 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:18:31.291 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:18:31.291 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:18:31.291 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:18:31.291 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:18:31.291 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:18:31.291 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: Manager daemon x is now available 2024-04-23T18:18:31.291 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:18:31.291 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:18:31.291 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:30 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:18:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: mgrmap e20: x(active, since 8m), standbys: y, z 2024-04-23T18:18:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.177:0/79381647' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:18:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: Standby manager daemon y restarted 2024-04-23T18:18:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: Standby manager daemon y started 2024-04-23T18:18:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.177:0/79381647' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:18:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.177:0/79381647' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:18:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.? 172.21.15.177:0/79381647' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:18:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: Active manager daemon x restarted 2024-04-23T18:18:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: Activating manager daemon x 2024-04-23T18:18:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T18:18:31.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: mgrmap e21: x(active, starting, since 0.0105277s), standbys: y, z 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:18:31.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:18:31.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:18:31.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:18:31.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:18:31.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: Manager daemon x is now available 2024-04-23T18:18:31.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:18:31.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:18:31.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:30 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:18:31.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: mgrmap e20: x(active, since 8m), standbys: y, z 2024-04-23T18:18:31.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.177:0/79381647' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T18:18:31.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: Standby manager daemon y restarted 2024-04-23T18:18:31.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: Standby manager daemon y started 2024-04-23T18:18:31.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.177:0/79381647' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T18:18:31.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.177:0/79381647' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T18:18:31.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.? 172.21.15.177:0/79381647' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T18:18:31.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: Active manager daemon x restarted 2024-04-23T18:18:31.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: Activating manager daemon x 2024-04-23T18:18:31.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: mgrmap e21: x(active, starting, since 0.0105277s), standbys: y, z 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:18:31.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T18:18:31.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T18:18:31.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T18:18:31.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T18:18:31.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T18:18:31.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T18:18:31.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T18:18:31.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: Manager daemon x is now available 2024-04-23T18:18:31.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:18:31.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T18:18:31.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:30 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T18:18:31.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.524+0000 7f71217fa640 1 -- 172.21.15.50:0/486371419 <== mon.1 v2:172.21.15.188:3300/0 9 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f711400ccc0 con 0x7f7124100e60 2024-04-23T18:18:31.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.525+0000 7f71217fa640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f710c035660 0x7f710c035a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.525+0000 7f71217fa640 1 -- 172.21.15.50:0/486371419 --> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f7114014ec0 con 0x7f710c035660 2024-04-23T18:18:31.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.527+0000 7f71237fe640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f710c035660 0x7f710c035a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.529+0000 7f71237fe640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f710c035660 0x7f710c035a40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7124199620 tx=0x7f7118005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.594+0000 7f71217fa640 1 -- 172.21.15.50:0/486371419 <== mgr.25018 v2:172.21.15.50:6800/4210816660 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7114014ec0 con 0x7f710c035660 2024-04-23T18:18:31.589 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path / 2024-04-23T18:18:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.597+0000 7f712aa59640 1 -- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f710c035660 msgr2=0x7f710c035a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.597+0000 7f712aa59640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f710c035660 0x7f710c035a40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7124199620 tx=0x7f7118005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.597+0000 7f712aa59640 1 -- 172.21.15.50:0/486371419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7124100e60 msgr2=0x7f7124198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.597+0000 7f712aa59640 1 --2- 172.21.15.50:0/486371419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7124100e60 0x7f7124198160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f711400e9c0 tx=0x7f711400ee80 comp rx=0 tx=0).stop 2024-04-23T18:18:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.597+0000 7f7128ff6640 1 -- 172.21.15.50:0/486371419 reap_dead start 2024-04-23T18:18:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.597+0000 7f712aa59640 1 -- 172.21.15.50:0/486371419 shutdown_connections 2024-04-23T18:18:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.597+0000 7f712aa59640 1 -- 172.21.15.50:0/486371419 >> 172.21.15.50:0/486371419 conn(0x7f71240fc420 msgr2=0x7f71240fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.597+0000 7f712aa59640 1 -- 172.21.15.50:0/486371419 shutdown_connections 2024-04-23T18:18:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.597+0000 7f712aa59640 1 -- 172.21.15.50:0/486371419 wait complete. 2024-04-23T18:18:31.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.716+0000 7fc42bc59640 1 -- 172.21.15.50:0/4274440282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc424102a00 msgr2=0x7fc424109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.716+0000 7fc42bc59640 1 --2- 172.21.15.50:0/4274440282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc424102a00 0x7fc424109320 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc41c009ca0 tx=0x7fc41c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.716+0000 7fc42bc59640 1 -- 172.21.15.50:0/4274440282 shutdown_connections 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.716+0000 7fc42bc59640 1 --2- 172.21.15.50:0/4274440282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc424102a00 0x7fc424109320 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.716+0000 7fc42bc59640 1 --2- 172.21.15.50:0/4274440282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc424102040 0x7fc4241024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.716+0000 7fc42bc59640 1 --2- 172.21.15.50:0/4274440282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc424100e60 0x7fc424101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.716+0000 7fc42bc59640 1 -- 172.21.15.50:0/4274440282 >> 172.21.15.50:0/4274440282 conn(0x7fc4240fc420 msgr2=0x7fc4240fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.716+0000 7fc42bc59640 1 -- 172.21.15.50:0/4274440282 shutdown_connections 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.716+0000 7fc42bc59640 1 -- 172.21.15.50:0/4274440282 wait complete. 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.716+0000 7fc42bc59640 1 Processor -- start 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc42bc59640 1 -- start start 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc42bc59640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc424100e60 0x7fc4241980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc42bc59640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc424102040 0x7fc424198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc42bc59640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc424102a00 0x7fc42419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc42bc59640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc42410ca10 con 0x7fc424100e60 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc42bc59640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc42410c890 con 0x7fc424102040 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc42bc59640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc42410cb90 con 0x7fc424102a00 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc4299f5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc424100e60 0x7fc4241980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc4299f5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc424100e60 0x7fc4241980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48990/0 (socket says 172.21.15.50:48990) 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc4299f5640 1 -- 172.21.15.50:0/3090650119 learned_addr learned my addr 172.21.15.50:0/3090650119 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc4291f4640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc424102040 0x7fc424198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc42a1f6640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc424102a00 0x7fc42419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc4299f5640 1 -- 172.21.15.50:0/3090650119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc424102a00 msgr2=0x7fc42419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc4299f5640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc424102a00 0x7fc42419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc4299f5640 1 -- 172.21.15.50:0/3090650119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc424102040 msgr2=0x7fc424198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc4299f5640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc424102040 0x7fc424198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc4299f5640 1 -- 172.21.15.50:0/3090650119 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc42419fd10 con 0x7fc424100e60 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc4299f5640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc424100e60 0x7fc4241980f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc41400e9c0 tx=0x7fc41400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc41affd640 1 -- 172.21.15.50:0/3090650119 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41400cd80 con 0x7fc424100e60 2024-04-23T18:18:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.717+0000 7fc41affd640 1 -- 172.21.15.50:0/3090650119 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4140035f0 con 0x7fc424100e60 2024-04-23T18:18:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.718+0000 7fc42bc59640 1 -- 172.21.15.50:0/3090650119 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc42419ffa0 con 0x7fc424100e60 2024-04-23T18:18:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.718+0000 7fc41affd640 1 -- 172.21.15.50:0/3090650119 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc414010690 con 0x7fc424100e60 2024-04-23T18:18:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.718+0000 7fc42bc59640 1 -- 172.21.15.50:0/3090650119 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4241a0410 con 0x7fc424100e60 2024-04-23T18:18:31.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.719+0000 7fc42bc59640 1 -- 172.21.15.50:0/3090650119 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4241a06d0 con 0x7fc424100e60 2024-04-23T18:18:31.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.720+0000 7fc41affd640 1 -- 172.21.15.50:0/3090650119 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fc414004000 con 0x7fc424100e60 2024-04-23T18:18:31.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.720+0000 7fc41affd640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4100a3a60 0x7fc4100a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:31.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.721+0000 7fc4291f4640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4100a3a60 0x7fc4100a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:31.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.721+0000 7fc41affd640 1 -- 172.21.15.50:0/3090650119 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fc4140d05f0 con 0x7fc424100e60 2024-04-23T18:18:31.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.721+0000 7fc4291f4640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4100a3a60 0x7fc4100a5f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc424199600 tx=0x7fc42000b390 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:31.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.725+0000 7fc41affd640 1 -- 172.21.15.50:0/3090650119 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc41409d1b0 con 0x7fc424100e60 2024-04-23T18:18:31.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.938+0000 7fc42bc59640 1 -- 172.21.15.50:0/3090650119 --> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fc4240fe560 con 0x7fc4100a3a60 2024-04-23T18:18:31.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.996+0000 7fc41affd640 1 -- 172.21.15.50:0/3090650119 <== mgr.25018 v2:172.21.15.50:6800/4210816660 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fc4240fe560 con 0x7fc4100a3a60 2024-04-23T18:18:31.990 INFO:teuthology.orchestra.run.smithi050.stdout:Retention added to path / 2024-04-23T18:18:31.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.998+0000 7fc42bc59640 1 -- 172.21.15.50:0/3090650119 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4100a3a60 msgr2=0x7fc4100a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.998+0000 7fc42bc59640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4100a3a60 0x7fc4100a5f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc424199600 tx=0x7fc42000b390 comp rx=0 tx=0).stop 2024-04-23T18:18:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.998+0000 7fc42bc59640 1 -- 172.21.15.50:0/3090650119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc424100e60 msgr2=0x7fc4241980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.998+0000 7fc42bc59640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc424100e60 0x7fc4241980f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc41400e9c0 tx=0x7fc41400ee80 comp rx=0 tx=0).stop 2024-04-23T18:18:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.998+0000 7fc42bc59640 1 -- 172.21.15.50:0/3090650119 shutdown_connections 2024-04-23T18:18:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.998+0000 7fc42bc59640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4100a3a60 0x7fc4100a5f20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.998+0000 7fc42bc59640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc424102a00 0x7fc42419f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.998+0000 7fc42bc59640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc424102040 0x7fc424198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.998+0000 7fc42bc59640 1 --2- 172.21.15.50:0/3090650119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc424100e60 0x7fc4241980f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.998+0000 7fc42bc59640 1 -- 172.21.15.50:0/3090650119 >> 172.21.15.50:0/3090650119 conn(0x7fc4240fc420 msgr2=0x7fc4240fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.998+0000 7fc42bc59640 1 -- 172.21.15.50:0/3090650119 shutdown_connections 2024-04-23T18:18:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:31.999+0000 7fc42bc59640 1 -- 172.21.15.50:0/3090650119 wait complete. 2024-04-23T18:18:32.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-23T18:18:32.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:31 smithi177 ceph-mon[32308]: [23/Apr/2024:18:18:31] ENGINE Bus STARTING 2024-04-23T18:18:32.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:31 smithi177 ceph-mon[32308]: mgrmap e22: x(active, since 1.01957s), standbys: y, z 2024-04-23T18:18:32.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.113+0000 7f5f1f919640 1 -- 172.21.15.50:0/206795741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f18105410 msgr2=0x7f5f18107800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.113+0000 7f5f1f919640 1 --2- 172.21.15.50:0/206795741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f18105410 0x7f5f18107800 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009900 tx=0x7f5f0c02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:32.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.113+0000 7f5f1f919640 1 -- 172.21.15.50:0/206795741 shutdown_connections 2024-04-23T18:18:32.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.113+0000 7f5f1f919640 1 --2- 172.21.15.50:0/206795741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f18107d40 0x7f5f1810a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.113+0000 7f5f1f919640 1 --2- 172.21.15.50:0/206795741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f18105410 0x7f5f18107800 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.113+0000 7f5f1f919640 1 --2- 172.21.15.50:0/206795741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f18102ae0 0x7f5f18104ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.113+0000 7f5f1f919640 1 -- 172.21.15.50:0/206795741 >> 172.21.15.50:0/206795741 conn(0x7f5f180fc4d0 msgr2=0x7f5f180fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:32.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.113+0000 7f5f1f919640 1 -- 172.21.15.50:0/206795741 shutdown_connections 2024-04-23T18:18:32.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.113+0000 7f5f1f919640 1 -- 172.21.15.50:0/206795741 wait complete. 2024-04-23T18:18:32.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1f919640 1 Processor -- start 2024-04-23T18:18:32.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1f919640 1 -- start start 2024-04-23T18:18:32.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1f919640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f18102ae0 0x7f5f18198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:32.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1f919640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f18105410 0x7f5f181986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:32.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1f919640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f18107d40 0x7f5f1819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:32.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1f919640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5f1810daf0 con 0x7f5f18105410 2024-04-23T18:18:32.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1f919640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5f1810d970 con 0x7f5f18107d40 2024-04-23T18:18:32.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1f919640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5f1810dc70 con 0x7f5f18102ae0 2024-04-23T18:18:32.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1d6b5640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f18102ae0 0x7f5f18198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1ceb4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f18105410 0x7f5f181986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1d6b5640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f18102ae0 0x7f5f18198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:42778/0 (socket says 172.21.15.50:42778) 2024-04-23T18:18:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1d6b5640 1 -- 172.21.15.50:0/3760802915 learned_addr learned my addr 172.21.15.50:0/3760802915 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1deb6640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f18107d40 0x7f5f1819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1d6b5640 1 -- 172.21.15.50:0/3760802915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f18107d40 msgr2=0x7f5f1819f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1d6b5640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f18107d40 0x7f5f1819f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1d6b5640 1 -- 172.21.15.50:0/3760802915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f18105410 msgr2=0x7f5f181986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1d6b5640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f18105410 0x7f5f181986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1d6b5640 1 -- 172.21.15.50:0/3760802915 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f1819fd80 con 0x7f5f18102ae0 2024-04-23T18:18:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.114+0000 7f5f1ceb4640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f18105410 0x7f5f181986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.115+0000 7f5f1d6b5640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f18102ae0 0x7f5f18198160 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000e9f0 tx=0x7f5f1000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:32.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.115+0000 7f5f167fc640 1 -- 172.21.15.50:0/3760802915 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1000cd80 con 0x7f5f18102ae0 2024-04-23T18:18:32.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.115+0000 7f5f167fc640 1 -- 172.21.15.50:0/3760802915 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f100108e0 con 0x7f5f18102ae0 2024-04-23T18:18:32.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.115+0000 7f5f167fc640 1 -- 172.21.15.50:0/3760802915 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f10018980 con 0x7f5f18102ae0 2024-04-23T18:18:32.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.115+0000 7f5f1f919640 1 -- 172.21.15.50:0/3760802915 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f181a0070 con 0x7f5f18102ae0 2024-04-23T18:18:32.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.115+0000 7f5f1f919640 1 -- 172.21.15.50:0/3760802915 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f181a0590 con 0x7f5f18102ae0 2024-04-23T18:18:32.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.116+0000 7f5f1f919640 1 -- 172.21.15.50:0/3760802915 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f180fdfc0 con 0x7f5f18102ae0 2024-04-23T18:18:32.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.117+0000 7f5f167fc640 1 -- 172.21.15.50:0/3760802915 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f5f10010430 con 0x7f5f18102ae0 2024-04-23T18:18:32.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.117+0000 7f5f167fc640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5f000a3a60 0x7f5f000a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:32.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.118+0000 7f5f1ceb4640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5f000a3a60 0x7f5f000a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:32.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.118+0000 7f5f167fc640 1 -- 172.21.15.50:0/3760802915 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f5f100d07b0 con 0x7f5f18102ae0 2024-04-23T18:18:32.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.118+0000 7f5f1ceb4640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5f000a3a60 0x7f5f000a5f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009230 tx=0x7f5f0c0314a0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:32.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.121+0000 7f5f167fc640 1 -- 172.21.15.50:0/3760802915 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5f1009cd20 con 0x7f5f18102ae0 2024-04-23T18:18:32.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:31 smithi050 ceph-mon[29187]: [23/Apr/2024:18:18:31] ENGINE Bus STARTING 2024-04-23T18:18:32.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:31 smithi050 ceph-mon[29187]: mgrmap e22: x(active, since 1.01957s), standbys: y, z 2024-04-23T18:18:32.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.336+0000 7f5f1f919640 1 -- 172.21.15.50:0/3760802915 --> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f5f18062390 con 0x7f5f000a3a60 2024-04-23T18:18:32.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:31 smithi188 ceph-mon[32694]: [23/Apr/2024:18:18:31] ENGINE Bus STARTING 2024-04-23T18:18:32.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:31 smithi188 ceph-mon[32694]: mgrmap e22: x(active, since 1.01957s), standbys: y, z 2024-04-23T18:18:32.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.344+0000 7f5f167fc640 1 -- 172.21.15.50:0/3760802915 <== mgr.25018 v2:172.21.15.50:6800/4210816660 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f5f18062390 con 0x7f5f000a3a60 2024-04-23T18:18:32.338 INFO:teuthology.orchestra.run.smithi050.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-23T00:00:00", "created": "2024-04-23T18:18:31", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-23T18:18:32.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.346+0000 7f5f1f919640 1 -- 172.21.15.50:0/3760802915 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5f000a3a60 msgr2=0x7f5f000a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.346+0000 7f5f1f919640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5f000a3a60 0x7f5f000a5f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009230 tx=0x7f5f0c0314a0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.346+0000 7f5f1f919640 1 -- 172.21.15.50:0/3760802915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f18102ae0 msgr2=0x7f5f18198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.346+0000 7f5f1f919640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f18102ae0 0x7f5f18198160 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000e9f0 tx=0x7f5f1000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.346+0000 7f5f1f919640 1 -- 172.21.15.50:0/3760802915 shutdown_connections 2024-04-23T18:18:32.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.346+0000 7f5f1f919640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5f000a3a60 0x7f5f000a5f20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.346+0000 7f5f1f919640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f18107d40 0x7f5f1819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.346+0000 7f5f1f919640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f18105410 0x7f5f181986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.346+0000 7f5f1f919640 1 --2- 172.21.15.50:0/3760802915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f18102ae0 0x7f5f18198160 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.346+0000 7f5f1f919640 1 -- 172.21.15.50:0/3760802915 >> 172.21.15.50:0/3760802915 conn(0x7f5f180fc4d0 msgr2=0x7f5f181035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:32.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.346+0000 7f5f1f919640 1 -- 172.21.15.50:0/3760802915 shutdown_connections 2024-04-23T18:18:32.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.346+0000 7f5f1f919640 1 -- 172.21.15.50:0/3760802915 wait complete. 2024-04-23T18:18:32.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-23T18:18:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.468+0000 7f0f7995d640 1 -- 172.21.15.50:0/377063399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f74105360 msgr2=0x7f0f74107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.468+0000 7f0f7995d640 1 --2- 172.21.15.50:0/377063399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f74105360 0x7f0f74107750 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0f68009900 tx=0x7f0f6802a410 comp rx=0 tx=0).stop 2024-04-23T18:18:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.468+0000 7f0f7995d640 1 -- 172.21.15.50:0/377063399 shutdown_connections 2024-04-23T18:18:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.468+0000 7f0f7995d640 1 --2- 172.21.15.50:0/377063399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f74107c90 0x7f0f7410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.468+0000 7f0f7995d640 1 --2- 172.21.15.50:0/377063399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f74105360 0x7f0f74107750 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.468+0000 7f0f7995d640 1 --2- 172.21.15.50:0/377063399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f74102a30 0x7f0f74104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.468+0000 7f0f7995d640 1 -- 172.21.15.50:0/377063399 >> 172.21.15.50:0/377063399 conn(0x7f0f740fc420 msgr2=0x7f0f740fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.468+0000 7f0f7995d640 1 -- 172.21.15.50:0/377063399 shutdown_connections 2024-04-23T18:18:32.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.468+0000 7f0f7995d640 1 -- 172.21.15.50:0/377063399 wait complete. 2024-04-23T18:18:32.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.468+0000 7f0f7995d640 1 Processor -- start 2024-04-23T18:18:32.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f7995d640 1 -- start start 2024-04-23T18:18:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f7995d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f74102a30 0x7f0f741981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f7995d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f74105360 0x7f0f741986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f7995d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f74107c90 0x7f0f7419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f7995d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7410daf0 con 0x7f0f74107c90 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f7995d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7410d970 con 0x7f0f74105360 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f7995d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7410dc70 con 0x7f0f74102a30 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f737fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f74107c90 0x7f0f7419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f727fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f74105360 0x7f0f741986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f72ffd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f74102a30 0x7f0f741981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f727fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f74105360 0x7f0f741986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:53722/0 (socket says 172.21.15.50:53722) 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f737fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f74107c90 0x7f0f7419f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49024/0 (socket says 172.21.15.50:49024) 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f727fc640 1 -- 172.21.15.50:0/2767894844 learned_addr learned my addr 172.21.15.50:0/2767894844 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f727fc640 1 -- 172.21.15.50:0/2767894844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f74102a30 msgr2=0x7f0f741981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f727fc640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f74102a30 0x7f0f741981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f727fc640 1 -- 172.21.15.50:0/2767894844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f74107c90 msgr2=0x7f0f7419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f727fc640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f74107c90 0x7f0f7419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f727fc640 1 -- 172.21.15.50:0/2767894844 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7419fdc0 con 0x7f0f74105360 2024-04-23T18:18:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.469+0000 7f0f737fe640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f74107c90 0x7f0f7419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.470+0000 7f0f727fc640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f74105360 0x7f0f741986e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f68006010 tx=0x7f0f680042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.470+0000 7f0f7895b640 1 -- 172.21.15.50:0/2767894844 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6803f030 con 0x7f0f74105360 2024-04-23T18:18:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.470+0000 7f0f7895b640 1 -- 172.21.15.50:0/2767894844 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f68031910 con 0x7f0f74105360 2024-04-23T18:18:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.470+0000 7f0f7895b640 1 -- 172.21.15.50:0/2767894844 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f68035070 con 0x7f0f74105360 2024-04-23T18:18:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.470+0000 7f0f7995d640 1 -- 172.21.15.50:0/2767894844 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f741a0050 con 0x7f0f74105360 2024-04-23T18:18:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.470+0000 7f0f7995d640 1 -- 172.21.15.50:0/2767894844 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f741a0540 con 0x7f0f74105360 2024-04-23T18:18:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.471+0000 7f0f7995d640 1 -- 172.21.15.50:0/2767894844 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f740fdf10 con 0x7f0f74105360 2024-04-23T18:18:32.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.472+0000 7f0f7895b640 1 -- 172.21.15.50:0/2767894844 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f0f68042030 con 0x7f0f74105360 2024-04-23T18:18:32.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.473+0000 7f0f7895b640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0f5c0a3a60 0x7f0f5c0a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:32.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.473+0000 7f0f7895b640 1 -- 172.21.15.50:0/2767894844 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f0f680f0a10 con 0x7f0f74105360 2024-04-23T18:18:32.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.473+0000 7f0f72ffd640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0f5c0a3a60 0x7f0f5c0a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:32.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.473+0000 7f0f72ffd640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0f5c0a3a60 0x7f0f5c0a5f20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009bd0 tx=0x7f0f64005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:32.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.477+0000 7f0f7895b640 1 -- 172.21.15.50:0/2767894844 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f68003e60 con 0x7f0f74105360 2024-04-23T18:18:32.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.728+0000 7f0f7995d640 1 -- 172.21.15.50:0/2767894844 --> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f0f74062390 con 0x7f0f5c0a3a60 2024-04-23T18:18:32.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.735+0000 7f0f7895b640 1 -- 172.21.15.50:0/2767894844 <== mgr.25018 v2:172.21.15.50:6800/4210816660 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f0f74062390 con 0x7f0f5c0a3a60 2024-04-23T18:18:32.729 INFO:teuthology.orchestra.run.smithi050.stdout:/ 1m 6m3h 2024-04-23T18:18:32.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.737+0000 7f0f7995d640 1 -- 172.21.15.50:0/2767894844 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0f5c0a3a60 msgr2=0x7f0f5c0a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.737+0000 7f0f7995d640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0f5c0a3a60 0x7f0f5c0a5f20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009bd0 tx=0x7f0f64005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:32.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.737+0000 7f0f7995d640 1 -- 172.21.15.50:0/2767894844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f74105360 msgr2=0x7f0f741986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.737+0000 7f0f7995d640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f74105360 0x7f0f741986e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f68006010 tx=0x7f0f680042f0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.737+0000 7f0f7995d640 1 -- 172.21.15.50:0/2767894844 shutdown_connections 2024-04-23T18:18:32.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.737+0000 7f0f7995d640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0f5c0a3a60 0x7f0f5c0a5f20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.737+0000 7f0f7995d640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f74107c90 0x7f0f7419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.737+0000 7f0f7995d640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f74105360 0x7f0f741986e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.737+0000 7f0f7995d640 1 --2- 172.21.15.50:0/2767894844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f74102a30 0x7f0f741981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.737+0000 7f0f7995d640 1 -- 172.21.15.50:0/2767894844 >> 172.21.15.50:0/2767894844 conn(0x7f0f740fc420 msgr2=0x7f0f74103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:32.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.738+0000 7f0f7995d640 1 -- 172.21.15.50:0/2767894844 shutdown_connections 2024-04-23T18:18:32.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.738+0000 7f0f7995d640 1 -- 172.21.15.50:0/2767894844 wait complete. 2024-04-23T18:18:32.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-23T18:18:32.770 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-23T18:18:32.780 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-23T18:18:32.790 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi050.front.sepia.ceph.com 2024-04-23T18:18:32.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-23T18:18:32.813 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-23T18:18:32.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:18:32.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.936+0000 7f99ea507640 1 -- 172.21.15.50:0/3325416565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99e4107c70 msgr2=0x7f99e410a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.936+0000 7f99ea507640 1 --2- 172.21.15.50:0/3325416565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99e4107c70 0x7f99e410a0c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f99dc009ca0 tx=0x7f99dc02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.936+0000 7f99ea507640 1 -- 172.21.15.50:0/3325416565 shutdown_connections 2024-04-23T18:18:32.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.936+0000 7f99ea507640 1 --2- 172.21.15.50:0/3325416565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99e4107c70 0x7f99e410a0c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.936+0000 7f99ea507640 1 --2- 172.21.15.50:0/3325416565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e4105340 0x7f99e4107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.936+0000 7f99ea507640 1 --2- 172.21.15.50:0/3325416565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e4102a10 0x7f99e4104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.936+0000 7f99ea507640 1 -- 172.21.15.50:0/3325416565 >> 172.21.15.50:0/3325416565 conn(0x7f99e40fc400 msgr2=0x7f99e40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:32.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.936+0000 7f99ea507640 1 -- 172.21.15.50:0/3325416565 shutdown_connections 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.936+0000 7f99ea507640 1 -- 172.21.15.50:0/3325416565 wait complete. 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99ea507640 1 Processor -- start 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99ea507640 1 -- start start 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99ea507640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e4102a10 0x7f99e4102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99ea507640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e4105340 0x7f99e41009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99ea507640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99e4107c70 0x7f99e4100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99ea507640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f99e410dae0 con 0x7f99e4107c70 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99ea507640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f99e410d960 con 0x7f99e4102a10 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99ea507640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f99e410dc60 con 0x7f99e4105340 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99e8aa4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99e4107c70 0x7f99e4100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99e8aa4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99e4107c70 0x7f99e4100f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49068/0 (socket says 172.21.15.50:49068) 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99e8aa4640 1 -- 172.21.15.50:0/3116108616 learned_addr learned my addr 172.21.15.50:0/3116108616 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99e37fe640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e4105340 0x7f99e41009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:32.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.937+0000 7f99e3fff640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e4102a10 0x7f99e4102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99e37fe640 1 -- 172.21.15.50:0/3116108616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e4102a10 msgr2=0x7f99e4102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99e37fe640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e4102a10 0x7f99e4102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99e37fe640 1 -- 172.21.15.50:0/3116108616 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99e4107c70 msgr2=0x7f99e4100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99e37fe640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99e4107c70 0x7f99e4100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99e37fe640 1 -- 172.21.15.50:0/3116108616 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e4101790 con 0x7f99e4105340 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99e3fff640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e4102a10 0x7f99e4102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99e8aa4640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99e4107c70 0x7f99e4100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99e37fe640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e4105340 0x7f99e41009c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f99d800e9c0 tx=0x7f99d800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99e17fa640 1 -- 172.21.15.50:0/3116108616 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d800cd80 con 0x7f99e4105340 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99e17fa640 1 -- 172.21.15.50:0/3116108616 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99d80035f0 con 0x7f99e4105340 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99ea507640 1 -- 172.21.15.50:0/3116108616 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99e41a22b0 con 0x7f99e4105340 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99e17fa640 1 -- 172.21.15.50:0/3116108616 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d8010690 con 0x7f99e4105340 2024-04-23T18:18:32.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.938+0000 7f99ea507640 1 -- 172.21.15.50:0/3116108616 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99e41a2800 con 0x7f99e4105340 2024-04-23T18:18:32.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.939+0000 7f99ea507640 1 -- 172.21.15.50:0/3116108616 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99e40fdef0 con 0x7f99e4105340 2024-04-23T18:18:32.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.940+0000 7f99e17fa640 1 -- 172.21.15.50:0/3116108616 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f99d80037a0 con 0x7f99e4105340 2024-04-23T18:18:32.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.941+0000 7f99e17fa640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99cc0a39b0 0x7f99cc0a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:32.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.941+0000 7f99e3fff640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99cc0a39b0 0x7f99cc0a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:32.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.941+0000 7f99e17fa640 1 -- 172.21.15.50:0/3116108616 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f99d80d06e0 con 0x7f99e4105340 2024-04-23T18:18:32.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.941+0000 7f99e3fff640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99cc0a39b0 0x7f99cc0a5e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009fc0 tx=0x7f99d4005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:32.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.944+0000 7f99e17fa640 1 -- 172.21.15.50:0/3116108616 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99d809d2a0 con 0x7f99e4105340 2024-04-23T18:18:32.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:32.988+0000 7f99e17fa640 1 -- 172.21.15.50:0/3116108616 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f99d8009310 con 0x7f99e4105340 2024-04-23T18:18:33.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.143+0000 7f99ea507640 1 -- 172.21.15.50:0/3116108616 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f99e402d810 con 0x7f99e4105340 2024-04-23T18:18:33.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.145+0000 7f99e17fa640 1 -- 172.21.15.50:0/3116108616 <== mon.1 v2:172.21.15.188:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23038 (secure 0 0 0) 0x7f99d8045780 con 0x7f99e4105340 2024-04-23T18:18:33.139 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:33.139 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":90,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:18:30.504176+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/1479623929":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/3248139242":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6800/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/2484340243":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/444338023":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:18:33.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.147+0000 7f99ea507640 1 -- 172.21.15.50:0/3116108616 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99cc0a39b0 msgr2=0x7f99cc0a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.147+0000 7f99ea507640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99cc0a39b0 0x7f99cc0a5e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009fc0 tx=0x7f99d4005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:33.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.147+0000 7f99ea507640 1 -- 172.21.15.50:0/3116108616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e4105340 msgr2=0x7f99e41009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.147+0000 7f99ea507640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e4105340 0x7f99e41009c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f99d800e9c0 tx=0x7f99d800ee80 comp rx=0 tx=0).stop 2024-04-23T18:18:33.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.147+0000 7f99ea507640 1 -- 172.21.15.50:0/3116108616 shutdown_connections 2024-04-23T18:18:33.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.147+0000 7f99ea507640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99cc0a39b0 0x7f99cc0a5e70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.147+0000 7f99ea507640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99e4107c70 0x7f99e4100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.147+0000 7f99ea507640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e4105340 0x7f99e41009c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.147+0000 7f99ea507640 1 --2- 172.21.15.50:0/3116108616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e4102a10 0x7f99e4102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.147+0000 7f99ea507640 1 -- 172.21.15.50:0/3116108616 >> 172.21.15.50:0/3116108616 conn(0x7f99e40fc400 msgr2=0x7f99e41032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:33.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.147+0000 7f99ea507640 1 -- 172.21.15.50:0/3116108616 shutdown_connections 2024-04-23T18:18:33.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.147+0000 7f99ea507640 1 -- 172.21.15.50:0/3116108616 wait complete. 2024-04-23T18:18:33.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:32 smithi050 ceph-mon[29187]: [23/Apr/2024:18:18:31] ENGINE Serving on http://172.21.15.50:8765 2024-04-23T18:18:33.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:32 smithi050 ceph-mon[29187]: [23/Apr/2024:18:18:31] ENGINE Serving on https://172.21.15.50:7150 2024-04-23T18:18:33.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:32 smithi050 ceph-mon[29187]: [23/Apr/2024:18:18:31] ENGINE Bus STARTED 2024-04-23T18:18:33.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:32 smithi050 ceph-mon[29187]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:33.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:32 smithi050 ceph-mon[29187]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:18:33.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:32 smithi050 ceph-mon[29187]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:33.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:32 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:32 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:32 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:32 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:32 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:32 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.153 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:12:02.493647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:16:31.679148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:16:32.269478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:18:33.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:18:33.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.274+0000 7f7d506ef640 1 -- 172.21.15.50:0/471518117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d481029a0 msgr2=0x7f7d481092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.274+0000 7f7d506ef640 1 --2- 172.21.15.50:0/471518117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d481029a0 0x7f7d481092c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009be0 tx=0x7f7d4002a6c0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.274+0000 7f7d506ef640 1 -- 172.21.15.50:0/471518117 shutdown_connections 2024-04-23T18:18:33.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.274+0000 7f7d506ef640 1 --2- 172.21.15.50:0/471518117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d481029a0 0x7f7d481092c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.274+0000 7f7d506ef640 1 --2- 172.21.15.50:0/471518117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d48101fe0 0x7f7d48102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.274+0000 7f7d506ef640 1 --2- 172.21.15.50:0/471518117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d48100e90 0x7f7d481012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.274+0000 7f7d506ef640 1 -- 172.21.15.50:0/471518117 >> 172.21.15.50:0/471518117 conn(0x7f7d480fc400 msgr2=0x7f7d480fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.274+0000 7f7d506ef640 1 -- 172.21.15.50:0/471518117 shutdown_connections 2024-04-23T18:18:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.274+0000 7f7d506ef640 1 -- 172.21.15.50:0/471518117 wait complete. 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.275+0000 7f7d506ef640 1 Processor -- start 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.275+0000 7f7d506ef640 1 -- start start 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.275+0000 7f7d506ef640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d48100e90 0x7f7d481970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.275+0000 7f7d506ef640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d48101fe0 0x7f7d48197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d506ef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d481029a0 0x7f7d4819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d506ef640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4810c9e0 con 0x7f7d48100e90 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d506ef640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4810c860 con 0x7f7d481029a0 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d506ef640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4810cb60 con 0x7f7d48101fe0 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d4e48b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d48100e90 0x7f7d481970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d4e48b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d48100e90 0x7f7d481970e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49086/0 (socket says 172.21.15.50:49086) 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d4e48b640 1 -- 172.21.15.50:0/2416212593 learned_addr learned my addr 172.21.15.50:0/2416212593 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d4ec8c640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d481029a0 0x7f7d4819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d4dc8a640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d48101fe0 0x7f7d48197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d4e48b640 1 -- 172.21.15.50:0/2416212593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d48101fe0 msgr2=0x7f7d48197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d4e48b640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d48101fe0 0x7f7d48197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d4e48b640 1 -- 172.21.15.50:0/2416212593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d481029a0 msgr2=0x7f7d4819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d4e48b640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d481029a0 0x7f7d4819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d4e48b640 1 -- 172.21.15.50:0/2416212593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d4819ed00 con 0x7f7d48100e90 2024-04-23T18:18:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d4ec8c640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d481029a0 0x7f7d4819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d4e48b640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d48100e90 0x7f7d481970e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7d4400e9c0 tx=0x7f7d4400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d3f7fe640 1 -- 172.21.15.50:0/2416212593 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4400cde0 con 0x7f7d48100e90 2024-04-23T18:18:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d506ef640 1 -- 172.21.15.50:0/2416212593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d4819eff0 con 0x7f7d48100e90 2024-04-23T18:18:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d3f7fe640 1 -- 172.21.15.50:0/2416212593 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d440035f0 con 0x7f7d48100e90 2024-04-23T18:18:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d506ef640 1 -- 172.21.15.50:0/2416212593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d4819f4b0 con 0x7f7d48100e90 2024-04-23T18:18:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.276+0000 7f7d3f7fe640 1 -- 172.21.15.50:0/2416212593 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d44010690 con 0x7f7d48100e90 2024-04-23T18:18:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.277+0000 7f7d506ef640 1 -- 172.21.15.50:0/2416212593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d48108220 con 0x7f7d48100e90 2024-04-23T18:18:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.279+0000 7f7d3f7fe640 1 -- 172.21.15.50:0/2416212593 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f7d44004000 con 0x7f7d48100e90 2024-04-23T18:18:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.279+0000 7f7d3f7fe640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7d340a3d20 0x7f7d340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.279+0000 7f7d4dc8a640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7d340a3d20 0x7f7d340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.279+0000 7f7d3f7fe640 1 -- 172.21.15.50:0/2416212593 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f7d440d0b40 con 0x7f7d48100e90 2024-04-23T18:18:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.280+0000 7f7d4dc8a640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7d340a3d20 0x7f7d340a61e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7d481985f0 tx=0x7f7d38005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.283+0000 7f7d3f7fe640 1 -- 172.21.15.50:0/2416212593 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d4409d280 con 0x7f7d48100e90 2024-04-23T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:32 smithi177 ceph-mon[32308]: [23/Apr/2024:18:18:31] ENGINE Serving on http://172.21.15.50:8765 2024-04-23T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:32 smithi177 ceph-mon[32308]: [23/Apr/2024:18:18:31] ENGINE Serving on https://172.21.15.50:7150 2024-04-23T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:32 smithi177 ceph-mon[32308]: [23/Apr/2024:18:18:31] ENGINE Bus STARTED 2024-04-23T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:32 smithi177 ceph-mon[32308]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:32 smithi177 ceph-mon[32308]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:32 smithi177 ceph-mon[32308]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:32 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:32 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:32 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:32 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:32 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:32 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:32 smithi188 ceph-mon[32694]: [23/Apr/2024:18:18:31] ENGINE Serving on http://172.21.15.50:8765 2024-04-23T18:18:33.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:32 smithi188 ceph-mon[32694]: [23/Apr/2024:18:18:31] ENGINE Serving on https://172.21.15.50:7150 2024-04-23T18:18:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:32 smithi188 ceph-mon[32694]: [23/Apr/2024:18:18:31] ENGINE Bus STARTED 2024-04-23T18:18:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:32 smithi188 ceph-mon[32694]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:32 smithi188 ceph-mon[32694]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:18:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:32 smithi188 ceph-mon[32694]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:32 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:32 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:32 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:32 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:32 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:32 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:33.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.472+0000 7f7d506ef640 1 -- 172.21.15.50:0/2416212593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7d4802d830 con 0x7f7d48100e90 2024-04-23T18:18:33.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.473+0000 7f7d3f7fe640 1 -- 172.21.15.50:0/2416212593 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f7d4409cb80 con 0x7f7d48100e90 2024-04-23T18:18:33.467 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-23T18:18:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.475+0000 7f7d506ef640 1 -- 172.21.15.50:0/2416212593 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7d340a3d20 msgr2=0x7f7d340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.475+0000 7f7d506ef640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7d340a3d20 0x7f7d340a61e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7d481985f0 tx=0x7f7d38005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.475+0000 7f7d506ef640 1 -- 172.21.15.50:0/2416212593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d48100e90 msgr2=0x7f7d481970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.475+0000 7f7d506ef640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d48100e90 0x7f7d481970e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7d4400e9c0 tx=0x7f7d4400ee80 comp rx=0 tx=0).stop 2024-04-23T18:18:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.475+0000 7f7d506ef640 1 -- 172.21.15.50:0/2416212593 shutdown_connections 2024-04-23T18:18:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.475+0000 7f7d506ef640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7d340a3d20 0x7f7d340a61e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.475+0000 7f7d506ef640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d481029a0 0x7f7d4819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.475+0000 7f7d506ef640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d48101fe0 0x7f7d48197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.475+0000 7f7d506ef640 1 --2- 172.21.15.50:0/2416212593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d48100e90 0x7f7d481970e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.475+0000 7f7d506ef640 1 -- 172.21.15.50:0/2416212593 >> 172.21.15.50:0/2416212593 conn(0x7f7d480fc400 msgr2=0x7f7d481053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:33.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.475+0000 7f7d506ef640 1 -- 172.21.15.50:0/2416212593 shutdown_connections 2024-04-23T18:18:33.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.475+0000 7f7d506ef640 1 -- 172.21.15.50:0/2416212593 wait complete. 2024-04-23T18:18:33.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.597+0000 7fa281b8e640 1 -- 172.21.15.50:0/1071961932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa27c101fc0 msgr2=0x7fa27c102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.597+0000 7fa281b8e640 1 --2- 172.21.15.50:0/1071961932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa27c101fc0 0x7fa27c102440 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa270009900 tx=0x7fa27002a410 comp rx=0 tx=0).stop 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.597+0000 7fa281b8e640 1 -- 172.21.15.50:0/1071961932 shutdown_connections 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.597+0000 7fa281b8e640 1 --2- 172.21.15.50:0/1071961932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa27c102980 0x7fa27c1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.597+0000 7fa281b8e640 1 --2- 172.21.15.50:0/1071961932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa27c101fc0 0x7fa27c102440 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.597+0000 7fa281b8e640 1 --2- 172.21.15.50:0/1071961932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa27c100e70 0x7fa27c101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.597+0000 7fa281b8e640 1 -- 172.21.15.50:0/1071961932 >> 172.21.15.50:0/1071961932 conn(0x7fa27c0fc3e0 msgr2=0x7fa27c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.597+0000 7fa281b8e640 1 -- 172.21.15.50:0/1071961932 shutdown_connections 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.597+0000 7fa281b8e640 1 -- 172.21.15.50:0/1071961932 wait complete. 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.597+0000 7fa281b8e640 1 Processor -- start 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa281b8e640 1 -- start start 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa281b8e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa27c100e70 0x7fa27c069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa281b8e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa27c101fc0 0x7fa27c19cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:33.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa281b8e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa27c102980 0x7fa27c19f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa281b8e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa27c10ca10 con 0x7fa27c102980 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa281b8e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa27c10c890 con 0x7fa27c101fc0 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa281b8e640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa27c10cb90 con 0x7fa27c100e70 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa27bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa27c102980 0x7fa27c19f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa27b7fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa27c100e70 0x7fa27c069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa27bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa27c102980 0x7fa27c19f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49104/0 (socket says 172.21.15.50:49104) 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa27affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa27c101fc0 0x7fa27c19cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa27bfff640 1 -- 172.21.15.50:0/1707204894 learned_addr learned my addr 172.21.15.50:0/1707204894 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa27b7fe640 1 -- 172.21.15.50:0/1707204894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa27c101fc0 msgr2=0x7fa27c19cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa27b7fe640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa27c101fc0 0x7fa27c19cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa27b7fe640 1 -- 172.21.15.50:0/1707204894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa27c102980 msgr2=0x7fa27c19f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa27b7fe640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa27c102980 0x7fa27c19f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa27b7fe640 1 -- 172.21.15.50:0/1707204894 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27c19fe60 con 0x7fa27c100e70 2024-04-23T18:18:33.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa27bfff640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa27c102980 0x7fa27c19f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.598+0000 7fa27b7fe640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa27c100e70 0x7fa27c069c50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00e9f0 tx=0x7fa26c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.599+0000 7fa278ff9640 1 -- 172.21.15.50:0/1707204894 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c00ccc0 con 0x7fa27c100e70 2024-04-23T18:18:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.599+0000 7fa281b8e640 1 -- 172.21.15.50:0/1707204894 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa27c1a00f0 con 0x7fa27c100e70 2024-04-23T18:18:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.599+0000 7fa278ff9640 1 -- 172.21.15.50:0/1707204894 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa26c005020 con 0x7fa27c100e70 2024-04-23T18:18:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.599+0000 7fa278ff9640 1 -- 172.21.15.50:0/1707204894 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c004a00 con 0x7fa27c100e70 2024-04-23T18:18:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.599+0000 7fa281b8e640 1 -- 172.21.15.50:0/1707204894 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa27c1a0610 con 0x7fa27c100e70 2024-04-23T18:18:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.600+0000 7fa281b8e640 1 -- 172.21.15.50:0/1707204894 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa27c068d70 con 0x7fa27c100e70 2024-04-23T18:18:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.602+0000 7fa278ff9640 1 -- 172.21.15.50:0/1707204894 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7fa26c004ba0 con 0x7fa27c100e70 2024-04-23T18:18:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.602+0000 7fa278ff9640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa2640a3d20 0x7fa2640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.602+0000 7fa278ff9640 1 -- 172.21.15.50:0/1707204894 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fa26c0d0870 con 0x7fa27c100e70 2024-04-23T18:18:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.602+0000 7fa27affd640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa2640a3d20 0x7fa2640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.603+0000 7fa27affd640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa2640a3d20 0x7fa2640a61e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa270005ab0 tx=0x7fa270005a20 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.605+0000 7fa278ff9640 1 -- 172.21.15.50:0/1707204894 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa26c09cf20 con 0x7fa27c100e70 2024-04-23T18:18:33.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.802+0000 7fa281b8e640 1 -- 172.21.15.50:0/1707204894 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa27c02d5c0 con 0x7fa27c100e70 2024-04-23T18:18:33.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.802+0000 7fa278ff9640 1 -- 172.21.15.50:0/1707204894 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fa26c09c820 con 0x7fa27c100e70 2024-04-23T18:18:33.796 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2024-04-23T18:18:33.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.804+0000 7fa281b8e640 1 -- 172.21.15.50:0/1707204894 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa2640a3d20 msgr2=0x7fa2640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.804+0000 7fa281b8e640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa2640a3d20 0x7fa2640a61e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa270005ab0 tx=0x7fa270005a20 comp rx=0 tx=0).stop 2024-04-23T18:18:33.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.804+0000 7fa281b8e640 1 -- 172.21.15.50:0/1707204894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa27c100e70 msgr2=0x7fa27c069c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.804+0000 7fa281b8e640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa27c100e70 0x7fa27c069c50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00e9f0 tx=0x7fa26c00eeb0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.804+0000 7fa281b8e640 1 -- 172.21.15.50:0/1707204894 shutdown_connections 2024-04-23T18:18:33.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.804+0000 7fa281b8e640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa2640a3d20 0x7fa2640a61e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.804+0000 7fa281b8e640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa27c102980 0x7fa27c19f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.804+0000 7fa281b8e640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa27c101fc0 0x7fa27c19cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.804+0000 7fa281b8e640 1 --2- 172.21.15.50:0/1707204894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa27c100e70 0x7fa27c069c50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.804+0000 7fa281b8e640 1 -- 172.21.15.50:0/1707204894 >> 172.21.15.50:0/1707204894 conn(0x7fa27c0fc3e0 msgr2=0x7fa27c105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:33.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.805+0000 7fa281b8e640 1 -- 172.21.15.50:0/1707204894 shutdown_connections 2024-04-23T18:18:33.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.805+0000 7fa281b8e640 1 -- 172.21.15.50:0/1707204894 wait complete. 2024-04-23T18:18:33.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:18:33.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.923+0000 7f6d1237d640 1 -- 172.21.15.50:0/1875134921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d0c101b20 msgr2=0x7f6d0c101fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.923+0000 7f6d1237d640 1 --2- 172.21.15.50:0/1875134921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d0c101b20 0x7f6d0c101fa0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009ca0 tx=0x7f6d0402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.923+0000 7f6d1237d640 1 -- 172.21.15.50:0/1875134921 shutdown_connections 2024-04-23T18:18:33.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.923+0000 7f6d1237d640 1 --2- 172.21.15.50:0/1875134921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d0c101b20 0x7f6d0c101fa0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.923+0000 7f6d1237d640 1 --2- 172.21.15.50:0/1875134921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d0c101180 0x7f6d0c1015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.923+0000 7f6d1237d640 1 --2- 172.21.15.50:0/1875134921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d0c100820 0x7f6d0c100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.923+0000 7f6d1237d640 1 -- 172.21.15.50:0/1875134921 >> 172.21.15.50:0/1875134921 conn(0x7f6d0c0fc440 msgr2=0x7f6d0c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:33.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.923+0000 7f6d1237d640 1 -- 172.21.15.50:0/1875134921 shutdown_connections 2024-04-23T18:18:33.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.923+0000 7f6d1237d640 1 -- 172.21.15.50:0/1875134921 wait complete. 2024-04-23T18:18:33.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.924+0000 7f6d1237d640 1 Processor -- start 2024-04-23T18:18:33.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.924+0000 7f6d1237d640 1 -- start start 2024-04-23T18:18:33.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.924+0000 7f6d1237d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d0c100820 0x7f6d0c19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:33.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.924+0000 7f6d1237d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d0c101180 0x7f6d0c19a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:33.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.924+0000 7f6d1237d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d0c101b20 0x7f6d0c1a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.924+0000 7f6d1237d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c10fd00 con 0x7f6d0c100820 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.924+0000 7f6d1237d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c10fb80 con 0x7f6d0c101180 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.924+0000 7f6d1237d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c10fe80 con 0x7f6d0c101b20 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.925+0000 7f6d0bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d0c100820 0x7f6d0c19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.925+0000 7f6d0bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d0c100820 0x7f6d0c19a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49130/0 (socket says 172.21.15.50:49130) 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.925+0000 7f6d0bfff640 1 -- 172.21.15.50:0/4095005470 learned_addr learned my addr 172.21.15.50:0/4095005470 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.925+0000 7f6d1091a640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d0c101b20 0x7f6d0c1a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.925+0000 7f6d0b7fe640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d0c101180 0x7f6d0c19a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.925+0000 7f6d1091a640 1 -- 172.21.15.50:0/4095005470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d0c101180 msgr2=0x7f6d0c19a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.925+0000 7f6d1091a640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d0c101180 0x7f6d0c19a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.925+0000 7f6d1091a640 1 -- 172.21.15.50:0/4095005470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d0c100820 msgr2=0x7f6d0c19a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.925+0000 7f6d1091a640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d0c100820 0x7f6d0c19a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.925+0000 7f6d1091a640 1 -- 172.21.15.50:0/4095005470 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c1a1f40 con 0x7f6d0c101b20 2024-04-23T18:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.925+0000 7f6d0bfff640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d0c100820 0x7f6d0c19a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:33.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.925+0000 7f6d1091a640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d0c101b20 0x7f6d0c1a18d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6d04031a20 tx=0x7f6d040317a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:33.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.926+0000 7f6d097fa640 1 -- 172.21.15.50:0/4095005470 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04003c30 con 0x7f6d0c101b20 2024-04-23T18:18:33.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.926+0000 7f6d097fa640 1 -- 172.21.15.50:0/4095005470 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d04003dd0 con 0x7f6d0c101b20 2024-04-23T18:18:33.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.926+0000 7f6d1237d640 1 -- 172.21.15.50:0/4095005470 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0c1a21d0 con 0x7f6d0c101b20 2024-04-23T18:18:33.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.926+0000 7f6d097fa640 1 -- 172.21.15.50:0/4095005470 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0402fb60 con 0x7f6d0c101b20 2024-04-23T18:18:33.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.926+0000 7f6d1237d640 1 -- 172.21.15.50:0/4095005470 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0c1a2720 con 0x7f6d0c101b20 2024-04-23T18:18:33.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.927+0000 7f6d1237d640 1 -- 172.21.15.50:0/4095005470 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d0c10b570 con 0x7f6d0c101b20 2024-04-23T18:18:33.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.928+0000 7f6d097fa640 1 -- 172.21.15.50:0/4095005470 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f6d0402fd20 con 0x7f6d0c101b20 2024-04-23T18:18:33.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.929+0000 7f6d097fa640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6cf40a3d20 0x7f6cf40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:33.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.929+0000 7f6d097fa640 1 -- 172.21.15.50:0/4095005470 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f6d040f1d80 con 0x7f6d0c101b20 2024-04-23T18:18:33.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.929+0000 7f6d0bfff640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6cf40a3d20 0x7f6cf40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:33.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.929+0000 7f6d0bfff640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6cf40a3d20 0x7f6cf40a61e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc011550 tx=0x7f6cfc005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:33.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:33.932+0000 7f6d097fa640 1 -- 172.21.15.50:0/4095005470 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d040be5d0 con 0x7f6d0c101b20 2024-04-23T18:18:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.131+0000 7f6d1237d640 1 -- 172.21.15.50:0/4095005470 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6d0c02d5c0 con 0x7f6d0c101b20 2024-04-23T18:18:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.131+0000 7f6d097fa640 1 -- 172.21.15.50:0/4095005470 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f6d040bded0 con 0x7f6d0c101b20 2024-04-23T18:18:34.124 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2024-04-23T18:18:34.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.133+0000 7f6d1237d640 1 -- 172.21.15.50:0/4095005470 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6cf40a3d20 msgr2=0x7f6cf40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.133+0000 7f6d1237d640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6cf40a3d20 0x7f6cf40a61e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc011550 tx=0x7f6cfc005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.133+0000 7f6d1237d640 1 -- 172.21.15.50:0/4095005470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d0c101b20 msgr2=0x7f6d0c1a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.133+0000 7f6d1237d640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d0c101b20 0x7f6d0c1a18d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6d04031a20 tx=0x7f6d040317a0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.134+0000 7f6d1237d640 1 -- 172.21.15.50:0/4095005470 shutdown_connections 2024-04-23T18:18:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.134+0000 7f6d1237d640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6cf40a3d20 0x7f6cf40a61e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.134+0000 7f6d1237d640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d0c101b20 0x7f6d0c1a18d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.134+0000 7f6d1237d640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d0c101180 0x7f6d0c19a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.134+0000 7f6d1237d640 1 --2- 172.21.15.50:0/4095005470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d0c100820 0x7f6d0c19a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.134+0000 7f6d1237d640 1 -- 172.21.15.50:0/4095005470 >> 172.21.15.50:0/4095005470 conn(0x7f6d0c0fc440 msgr2=0x7f6d0c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.134+0000 7f6d1237d640 1 -- 172.21.15.50:0/4095005470 shutdown_connections 2024-04-23T18:18:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.134+0000 7f6d1237d640 1 -- 172.21.15.50:0/4095005470 wait complete. 2024-04-23T18:18:34.137 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:34.137 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3116108616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2416212593' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:33 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1707204894' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:34.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:18:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.252+0000 7fad8b605640 1 -- 172.21.15.50:0/3441398742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad84102a70 msgr2=0x7fad84104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.252+0000 7fad8b605640 1 --2- 172.21.15.50:0/3441398742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad84102a70 0x7fad84104e60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fad80009900 tx=0x7fad8002a410 comp rx=0 tx=0).stop 2024-04-23T18:18:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.252+0000 7fad8b605640 1 -- 172.21.15.50:0/3441398742 shutdown_connections 2024-04-23T18:18:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.252+0000 7fad8b605640 1 --2- 172.21.15.50:0/3441398742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad84107cd0 0x7fad8410a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.252+0000 7fad8b605640 1 --2- 172.21.15.50:0/3441398742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad841053a0 0x7fad84107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.252+0000 7fad8b605640 1 --2- 172.21.15.50:0/3441398742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad84102a70 0x7fad84104e60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.252+0000 7fad8b605640 1 -- 172.21.15.50:0/3441398742 >> 172.21.15.50:0/3441398742 conn(0x7fad840fc400 msgr2=0x7fad840fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.252+0000 7fad8b605640 1 -- 172.21.15.50:0/3441398742 shutdown_connections 2024-04-23T18:18:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.252+0000 7fad8b605640 1 -- 172.21.15.50:0/3441398742 wait complete. 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad8b605640 1 Processor -- start 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad8b605640 1 -- start start 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad8b605640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad84102a70 0x7fad84198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad8b605640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad841053a0 0x7fad84198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad8b605640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad84107cd0 0x7fad8419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad8b605640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fad8410dad0 con 0x7fad841053a0 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad8b605640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fad8410d950 con 0x7fad84102a70 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad8b605640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fad8410dc50 con 0x7fad84107cd0 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad88ba0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad841053a0 0x7fad84198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad89ba2640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad84107cd0 0x7fad8419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad88ba0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad841053a0 0x7fad84198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49146/0 (socket says 172.21.15.50:49146) 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad88ba0640 1 -- 172.21.15.50:0/2492813782 learned_addr learned my addr 172.21.15.50:0/2492813782 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.253+0000 7fad893a1640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad84102a70 0x7fad84198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:34.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.254+0000 7fad88ba0640 1 -- 172.21.15.50:0/2492813782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad84107cd0 msgr2=0x7fad8419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.254+0000 7fad88ba0640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad84107cd0 0x7fad8419f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.254+0000 7fad88ba0640 1 -- 172.21.15.50:0/2492813782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad84102a70 msgr2=0x7fad84198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.254+0000 7fad88ba0640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad84102a70 0x7fad84198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.254+0000 7fad88ba0640 1 -- 172.21.15.50:0/2492813782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8419fe70 con 0x7fad841053a0 2024-04-23T18:18:34.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.254+0000 7fad89ba2640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad84107cd0 0x7fad8419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:34.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.254+0000 7fad88ba0640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad841053a0 0x7fad84198790 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fad7400e9f0 tx=0x7fad7400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:34.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.254+0000 7fad7a7fc640 1 -- 172.21.15.50:0/2492813782 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7400ccc0 con 0x7fad841053a0 2024-04-23T18:18:34.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.254+0000 7fad8b605640 1 -- 172.21.15.50:0/2492813782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad841a0160 con 0x7fad841053a0 2024-04-23T18:18:34.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.254+0000 7fad7a7fc640 1 -- 172.21.15.50:0/2492813782 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad740054d0 con 0x7fad841053a0 2024-04-23T18:18:34.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.254+0000 7fad7a7fc640 1 -- 172.21.15.50:0/2492813782 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad74010430 con 0x7fad841053a0 2024-04-23T18:18:34.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.254+0000 7fad8b605640 1 -- 172.21.15.50:0/2492813782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad841a06b0 con 0x7fad841053a0 2024-04-23T18:18:34.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.255+0000 7fad8b605640 1 -- 172.21.15.50:0/2492813782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad840fdef0 con 0x7fad841053a0 2024-04-23T18:18:34.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.257+0000 7fad7a7fc640 1 -- 172.21.15.50:0/2492813782 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7fad740046c0 con 0x7fad841053a0 2024-04-23T18:18:34.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.257+0000 7fad7a7fc640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fad700a3da0 0x7fad700a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:34.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.257+0000 7fad7a7fc640 1 -- 172.21.15.50:0/2492813782 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fad740d0b50 con 0x7fad841053a0 2024-04-23T18:18:34.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.257+0000 7fad893a1640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fad700a3da0 0x7fad700a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:34.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.258+0000 7fad893a1640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fad700a3da0 0x7fad700a6260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad80000c00 tx=0x7fad800314a0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:34.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.264+0000 7fad7a7fc640 1 -- 172.21.15.50:0/2492813782 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad7409d1a0 con 0x7fad841053a0 2024-04-23T18:18:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:18:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-23T18:18:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3116108616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2416212593' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:34.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:34.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:34.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:34.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:34.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:33 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1707204894' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:34.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:18:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:18:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-23T18:18:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3116108616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2416212593' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:34.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:34.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:33 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1707204894' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:34.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.459+0000 7fad8b605640 1 -- 172.21.15.50:0/2492813782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad8402d810 con 0x7fad841053a0 2024-04-23T18:18:34.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.461+0000 7fad7a7fc640 1 -- 172.21.15.50:0/2492813782 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23038 (secure 0 0 0) 0x7fad7409caa0 con 0x7fad841053a0 2024-04-23T18:18:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:34.454 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":90,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:18:30.504176+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/1479623929":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/3248139242":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6800/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/2484340243":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/444338023":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:18:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.463+0000 7fad8b605640 1 -- 172.21.15.50:0/2492813782 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fad700a3da0 msgr2=0x7fad700a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.463+0000 7fad8b605640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fad700a3da0 0x7fad700a6260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad80000c00 tx=0x7fad800314a0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.463+0000 7fad8b605640 1 -- 172.21.15.50:0/2492813782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad841053a0 msgr2=0x7fad84198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.463+0000 7fad8b605640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad841053a0 0x7fad84198790 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fad7400e9f0 tx=0x7fad7400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.463+0000 7fad8b605640 1 -- 172.21.15.50:0/2492813782 shutdown_connections 2024-04-23T18:18:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.463+0000 7fad8b605640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fad700a3da0 0x7fad700a6260 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.463+0000 7fad8b605640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad84107cd0 0x7fad8419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.463+0000 7fad8b605640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad841053a0 0x7fad84198790 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.463+0000 7fad8b605640 1 --2- 172.21.15.50:0/2492813782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad84102a70 0x7fad84198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.463+0000 7fad8b605640 1 -- 172.21.15.50:0/2492813782 >> 172.21.15.50:0/2492813782 conn(0x7fad840fc400 msgr2=0x7fad84104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.463+0000 7fad8b605640 1 -- 172.21.15.50:0/2492813782 shutdown_connections 2024-04-23T18:18:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.463+0000 7fad8b605640 1 -- 172.21.15.50:0/2492813782 wait complete. 2024-04-23T18:18:34.468 INFO:tasks.quiescer.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:12:02.493647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:16:31.679148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:16:32.269478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:18:34.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:18:34.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.585+0000 7fe14a47c640 1 -- 172.21.15.50:0/1170271135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1440749b0 msgr2=0x7fe144074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.585+0000 7fe14a47c640 1 --2- 172.21.15.50:0/1170271135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1440749b0 0x7fe144074e30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe138009930 tx=0x7fe13802a420 comp rx=0 tx=0).stop 2024-04-23T18:18:34.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.585+0000 7fe14a47c640 1 -- 172.21.15.50:0/1170271135 shutdown_connections 2024-04-23T18:18:34.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.585+0000 7fe14a47c640 1 --2- 172.21.15.50:0/1170271135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe14410e580 0x7fe144110970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.585+0000 7fe14a47c640 1 --2- 172.21.15.50:0/1170271135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1440749b0 0x7fe144074e30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.585+0000 7fe14a47c640 1 --2- 172.21.15.50:0/1170271135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe144075f90 0x7fe1440743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.585+0000 7fe14a47c640 1 -- 172.21.15.50:0/1170271135 >> 172.21.15.50:0/1170271135 conn(0x7fe1440fd520 msgr2=0x7fe1440ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.585+0000 7fe14a47c640 1 -- 172.21.15.50:0/1170271135 shutdown_connections 2024-04-23T18:18:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.585+0000 7fe14a47c640 1 -- 172.21.15.50:0/1170271135 wait complete. 2024-04-23T18:18:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe14a47c640 1 Processor -- start 2024-04-23T18:18:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe14a47c640 1 -- start start 2024-04-23T18:18:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe14a47c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe1440749b0 0x7fe14419e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe14a47c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe144075f90 0x7fe14419ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe14a47c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe14410e580 0x7fe1441a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe14a47c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe1441140f0 con 0x7fe144075f90 2024-04-23T18:18:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe14a47c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe144113f70 con 0x7fe14410e580 2024-04-23T18:18:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe14a47c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe144114270 con 0x7fe1440749b0 2024-04-23T18:18:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe1437fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe144075f90 0x7fe14419ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe1437fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe144075f90 0x7fe14419ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49160/0 (socket says 172.21.15.50:49160) 2024-04-23T18:18:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe1437fe640 1 -- 172.21.15.50:0/3758968833 learned_addr learned my addr 172.21.15.50:0/3758968833 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe143fff640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe1440749b0 0x7fe14419e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe148a19640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe14410e580 0x7fe1441a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe143fff640 1 -- 172.21.15.50:0/3758968833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe14410e580 msgr2=0x7fe1441a5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe143fff640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe14410e580 0x7fe1441a5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe143fff640 1 -- 172.21.15.50:0/3758968833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe144075f90 msgr2=0x7fe14419ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe143fff640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe144075f90 0x7fe14419ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe143fff640 1 -- 172.21.15.50:0/3758968833 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1441a6420 con 0x7fe1440749b0 2024-04-23T18:18:34.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.586+0000 7fe1437fe640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe144075f90 0x7fe14419ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:34.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.587+0000 7fe143fff640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe1440749b0 0x7fe14419e850 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe13400e9f0 tx=0x7fe13400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:34.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.587+0000 7fe1417fa640 1 -- 172.21.15.50:0/3758968833 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13400ccc0 con 0x7fe1440749b0 2024-04-23T18:18:34.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.587+0000 7fe14a47c640 1 -- 172.21.15.50:0/3758968833 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1441a6710 con 0x7fe1440749b0 2024-04-23T18:18:34.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.587+0000 7fe1417fa640 1 -- 172.21.15.50:0/3758968833 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe134005020 con 0x7fe1440749b0 2024-04-23T18:18:34.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.587+0000 7fe1417fa640 1 -- 172.21.15.50:0/3758968833 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134004a00 con 0x7fe1440749b0 2024-04-23T18:18:34.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.587+0000 7fe14a47c640 1 -- 172.21.15.50:0/3758968833 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1441a6c10 con 0x7fe1440749b0 2024-04-23T18:18:34.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.588+0000 7fe14a47c640 1 -- 172.21.15.50:0/3758968833 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe144104640 con 0x7fe1440749b0 2024-04-23T18:18:34.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.589+0000 7fe1417fa640 1 -- 172.21.15.50:0/3758968833 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7fe134004ba0 con 0x7fe1440749b0 2024-04-23T18:18:34.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.590+0000 7fe1417fa640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe12c0a3d20 0x7fe12c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:34.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.590+0000 7fe1437fe640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe12c0a3d20 0x7fe12c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:34.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.590+0000 7fe1417fa640 1 -- 172.21.15.50:0/3758968833 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fe1340d08d0 con 0x7fe1440749b0 2024-04-23T18:18:34.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.590+0000 7fe1437fe640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe12c0a3d20 0x7fe12c0a61e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe14419fd10 tx=0x7fe1380314c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:34.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.593+0000 7fe1417fa640 1 -- 172.21.15.50:0/3758968833 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe13409cf20 con 0x7fe1440749b0 2024-04-23T18:18:34.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.800+0000 7fe14a47c640 1 -- 172.21.15.50:0/3758968833 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe144062b50 con 0x7fe1440749b0 2024-04-23T18:18:34.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.800+0000 7fe1417fa640 1 -- 172.21.15.50:0/3758968833 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fe13409c820 con 0x7fe1440749b0 2024-04-23T18:18:34.793 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-23T18:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.803+0000 7fe14a47c640 1 -- 172.21.15.50:0/3758968833 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe12c0a3d20 msgr2=0x7fe12c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.803+0000 7fe14a47c640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe12c0a3d20 0x7fe12c0a61e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe14419fd10 tx=0x7fe1380314c0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.803+0000 7fe14a47c640 1 -- 172.21.15.50:0/3758968833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe1440749b0 msgr2=0x7fe14419e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.803+0000 7fe14a47c640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe1440749b0 0x7fe14419e850 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe13400e9f0 tx=0x7fe13400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.803+0000 7fe14a47c640 1 -- 172.21.15.50:0/3758968833 shutdown_connections 2024-04-23T18:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.803+0000 7fe14a47c640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe12c0a3d20 0x7fe12c0a61e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.803+0000 7fe14a47c640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe14410e580 0x7fe1441a5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.803+0000 7fe14a47c640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe144075f90 0x7fe14419ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.803+0000 7fe14a47c640 1 --2- 172.21.15.50:0/3758968833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe1440749b0 0x7fe14419e850 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.803+0000 7fe14a47c640 1 -- 172.21.15.50:0/3758968833 >> 172.21.15.50:0/3758968833 conn(0x7fe1440fd520 msgr2=0x7fe144101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.803+0000 7fe14a47c640 1 -- 172.21.15.50:0/3758968833 shutdown_connections 2024-04-23T18:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.803+0000 7fe14a47c640 1 -- 172.21.15.50:0/3758968833 wait complete. 2024-04-23T18:18:34.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:18:34.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.926+0000 7f1b038ae640 1 -- 172.21.15.50:0/825230391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1afc102b50 msgr2=0x7f1afc106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.926+0000 7f1b038ae640 1 --2- 172.21.15.50:0/825230391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1afc102b50 0x7f1afc106b90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009900 tx=0x7f1aec02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.926+0000 7f1b038ae640 1 -- 172.21.15.50:0/825230391 shutdown_connections 2024-04-23T18:18:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.926+0000 7f1b038ae640 1 --2- 172.21.15.50:0/825230391 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1afc107270 0x7f1afc10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.926+0000 7f1b038ae640 1 --2- 172.21.15.50:0/825230391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1afc102b50 0x7f1afc106b90 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.926+0000 7f1b038ae640 1 --2- 172.21.15.50:0/825230391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1afc1021a0 0x7f1afc102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.926+0000 7f1b038ae640 1 -- 172.21.15.50:0/825230391 >> 172.21.15.50:0/825230391 conn(0x7f1afc0793e0 msgr2=0x7f1afc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.926+0000 7f1b038ae640 1 -- 172.21.15.50:0/825230391 shutdown_connections 2024-04-23T18:18:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.926+0000 7f1b038ae640 1 -- 172.21.15.50:0/825230391 wait complete. 2024-04-23T18:18:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b038ae640 1 Processor -- start 2024-04-23T18:18:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b038ae640 1 -- start start 2024-04-23T18:18:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b038ae640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1afc1021a0 0x7f1afc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b038ae640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1afc107270 0x7f1afc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b038ae640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1afc19cfd0 0x7f1afc19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b038ae640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc112190 con 0x7f1afc107270 2024-04-23T18:18:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b038ae640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc112010 con 0x7f1afc19cfd0 2024-04-23T18:18:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b038ae640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc112310 con 0x7f1afc1021a0 2024-04-23T18:18:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b00e49640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1afc107270 0x7f1afc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b0164a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1afc1021a0 0x7f1afc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b0164a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1afc1021a0 0x7f1afc198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:42972/0 (socket says 172.21.15.50:42972) 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b0164a640 1 -- 172.21.15.50:0/3825086338 learned_addr learned my addr 172.21.15.50:0/3825086338 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b01e4b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1afc19cfd0 0x7f1afc19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b0164a640 1 -- 172.21.15.50:0/3825086338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1afc19cfd0 msgr2=0x7f1afc19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b0164a640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1afc19cfd0 0x7f1afc19d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b0164a640 1 -- 172.21.15.50:0/3825086338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1afc107270 msgr2=0x7f1afc198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b0164a640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1afc107270 0x7f1afc198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b0164a640 1 -- 172.21.15.50:0/3825086338 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1afc19dac0 con 0x7f1afc1021a0 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.927+0000 7f1b00e49640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1afc107270 0x7f1afc198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.928+0000 7f1b0164a640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1afc1021a0 0x7f1afc198710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1af800e9f0 tx=0x7f1af800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.928+0000 7f1af27fc640 1 -- 172.21.15.50:0/3825086338 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af800cd80 con 0x7f1afc1021a0 2024-04-23T18:18:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.928+0000 7f1af27fc640 1 -- 172.21.15.50:0/3825086338 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1af80108e0 con 0x7f1afc1021a0 2024-04-23T18:18:34.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.928+0000 7f1b038ae640 1 -- 172.21.15.50:0/3825086338 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1afc19ddb0 con 0x7f1afc1021a0 2024-04-23T18:18:34.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.928+0000 7f1af27fc640 1 -- 172.21.15.50:0/3825086338 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af8018980 con 0x7f1afc1021a0 2024-04-23T18:18:34.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.928+0000 7f1b038ae640 1 -- 172.21.15.50:0/3825086338 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1afc1a5570 con 0x7f1afc1021a0 2024-04-23T18:18:34.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.929+0000 7f1b038ae640 1 -- 172.21.15.50:0/3825086338 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1afc0684e0 con 0x7f1afc1021a0 2024-04-23T18:18:34.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.930+0000 7f1af27fc640 1 -- 172.21.15.50:0/3825086338 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f1af8010430 con 0x7f1afc1021a0 2024-04-23T18:18:34.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.931+0000 7f1af27fc640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1ae80a3d20 0x7f1ae80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:34.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.931+0000 7f1b00e49640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1ae80a3d20 0x7f1ae80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:34.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.931+0000 7f1af27fc640 1 -- 172.21.15.50:0/3825086338 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f1af80d0de0 con 0x7f1afc1021a0 2024-04-23T18:18:34.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.932+0000 7f1b00e49640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1ae80a3d20 0x7f1ae80a61e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1aec000c00 tx=0x7f1aec031400 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:34.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:34.934+0000 7f1af27fc640 1 -- 172.21.15.50:0/3825086338 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1af809cde0 con 0x7f1afc1021a0 2024-04-23T18:18:35.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.018+0000 7f1af27fc640 1 -- 172.21.15.50:0/3825086338 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f1af8099450 con 0x7f1afc1021a0 2024-04-23T18:18:35.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.136+0000 7f1b038ae640 1 -- 172.21.15.50:0/3825086338 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1afc10a500 con 0x7f1afc1021a0 2024-04-23T18:18:35.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.137+0000 7f1af27fc640 1 -- 172.21.15.50:0/3825086338 <== mon.1 v2:172.21.15.188:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f1af80ce1d0 con 0x7f1afc1021a0 2024-04-23T18:18:35.131 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2024-04-23T18:18:35.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.139+0000 7f1b038ae640 1 -- 172.21.15.50:0/3825086338 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1ae80a3d20 msgr2=0x7f1ae80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.139+0000 7f1b038ae640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1ae80a3d20 0x7f1ae80a61e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1aec000c00 tx=0x7f1aec031400 comp rx=0 tx=0).stop 2024-04-23T18:18:35.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.139+0000 7f1b038ae640 1 -- 172.21.15.50:0/3825086338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1afc1021a0 msgr2=0x7f1afc198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.139+0000 7f1b038ae640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1afc1021a0 0x7f1afc198710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1af800e9f0 tx=0x7f1af800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.139+0000 7f1b038ae640 1 -- 172.21.15.50:0/3825086338 shutdown_connections 2024-04-23T18:18:35.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.139+0000 7f1b038ae640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1ae80a3d20 0x7f1ae80a61e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.139+0000 7f1b038ae640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1afc19cfd0 0x7f1afc19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.139+0000 7f1b038ae640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1afc107270 0x7f1afc198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.139+0000 7f1b038ae640 1 --2- 172.21.15.50:0/3825086338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1afc1021a0 0x7f1afc198710 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.139+0000 7f1b038ae640 1 -- 172.21.15.50:0/3825086338 >> 172.21.15.50:0/3825086338 conn(0x7f1afc0793e0 msgr2=0x7f1afc100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:35.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.139+0000 7f1b038ae640 1 -- 172.21.15.50:0/3825086338 shutdown_connections 2024-04-23T18:18:35.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.139+0000 7f1b038ae640 1 -- 172.21.15.50:0/3825086338 wait complete. 2024-04-23T18:18:35.144 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-23T18:18:35.144 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: Unable to set osd_memory_target on smithi188 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T18:18:35.144 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-23T18:18:35.144 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-23T18:18:35.144 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:35.144 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:35.144 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:35.144 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:35.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:35.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:35.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-04-23T18:18:35.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:18:35.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:18:35.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:18:35.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: Updating smithi050:/etc/ceph/ceph.conf 2024-04-23T18:18:35.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-23T18:18:35.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-23T18:18:35.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4095005470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:35.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2492813782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:35.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:34 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3758968833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:35.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:18:35.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.264+0000 7f181e8a7640 1 -- 172.21.15.50:0/1130309206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1818101a50 msgr2=0x7f1818105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.264+0000 7f181e8a7640 1 --2- 172.21.15.50:0/1130309206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1818101a50 0x7f1818105a90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f180c009900 tx=0x7f180c02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.264+0000 7f181e8a7640 1 -- 172.21.15.50:0/1130309206 shutdown_connections 2024-04-23T18:18:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.264+0000 7f181e8a7640 1 --2- 172.21.15.50:0/1130309206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1818106170 0x7f181810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.264+0000 7f181e8a7640 1 --2- 172.21.15.50:0/1130309206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1818101a50 0x7f1818105a90 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.264+0000 7f181e8a7640 1 --2- 172.21.15.50:0/1130309206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18181010a0 0x7f1818101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.264+0000 7f181e8a7640 1 -- 172.21.15.50:0/1130309206 >> 172.21.15.50:0/1130309206 conn(0x7f18180771d0 msgr2=0x7f18180775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.264+0000 7f181e8a7640 1 -- 172.21.15.50:0/1130309206 shutdown_connections 2024-04-23T18:18:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.264+0000 7f181e8a7640 1 -- 172.21.15.50:0/1130309206 wait complete. 2024-04-23T18:18:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.265+0000 7f181e8a7640 1 Processor -- start 2024-04-23T18:18:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.265+0000 7f181e8a7640 1 -- start start 2024-04-23T18:18:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.265+0000 7f181e8a7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18181010a0 0x7f181819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.265+0000 7f181e8a7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1818106170 0x7f181819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.265+0000 7f181e8a7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18181985a0 0x7f1818198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.265+0000 7f181e8a7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f18181110b0 con 0x7f18181010a0 2024-04-23T18:18:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.265+0000 7f181e8a7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1818110f30 con 0x7f1818106170 2024-04-23T18:18:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.265+0000 7f181e8a7640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1818111230 con 0x7f18181985a0 2024-04-23T18:18:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.265+0000 7f1817fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18181010a0 0x7f181819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f1817fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18181010a0 0x7f181819e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49204/0 (socket says 172.21.15.50:49204) 2024-04-23T18:18:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f1817fff640 1 -- 172.21.15.50:0/2102803724 learned_addr learned my addr 172.21.15.50:0/2102803724 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f181ce44640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18181985a0 0x7f1818198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f18177fe640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1818106170 0x7f181819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f1817fff640 1 -- 172.21.15.50:0/2102803724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18181985a0 msgr2=0x7f1818198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f1817fff640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18181985a0 0x7f1818198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f1817fff640 1 -- 172.21.15.50:0/2102803724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1818106170 msgr2=0x7f181819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f1817fff640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1818106170 0x7f181819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f1817fff640 1 -- 172.21.15.50:0/2102803724 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18181990c0 con 0x7f18181010a0 2024-04-23T18:18:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f1817fff640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18181010a0 0x7f181819e3f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f180800ea60 tx=0x7f180800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f18157fa640 1 -- 172.21.15.50:0/2102803724 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180800cde0 con 0x7f18181010a0 2024-04-23T18:18:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f181e8a7640 1 -- 172.21.15.50:0/2102803724 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18181993b0 con 0x7f18181010a0 2024-04-23T18:18:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f18157fa640 1 -- 172.21.15.50:0/2102803724 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18080035f0 con 0x7f18181010a0 2024-04-23T18:18:35.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f18157fa640 1 -- 172.21.15.50:0/2102803724 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1808010690 con 0x7f18181010a0 2024-04-23T18:18:35.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.266+0000 7f181e8a7640 1 -- 172.21.15.50:0/2102803724 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18181a5650 con 0x7f18181010a0 2024-04-23T18:18:35.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.267+0000 7f181e8a7640 1 -- 172.21.15.50:0/2102803724 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1818102570 con 0x7f18181010a0 2024-04-23T18:18:35.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.268+0000 7f18157fa640 1 -- 172.21.15.50:0/2102803724 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f18080037a0 con 0x7f18181010a0 2024-04-23T18:18:35.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.269+0000 7f18157fa640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f18000a3db0 0x7f18000a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:35.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.269+0000 7f18177fe640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f18000a3db0 0x7f18000a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:35.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.269+0000 7f18157fa640 1 -- 172.21.15.50:0/2102803724 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f18080d0ce0 con 0x7f18181010a0 2024-04-23T18:18:35.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.270+0000 7f18177fe640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f18000a3db0 0x7f18000a6270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f180c000c00 tx=0x7f180c032bc0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:35.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.274+0000 7f18157fa640 1 -- 172.21.15.50:0/2102803724 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f180809d220 con 0x7f18181010a0 2024-04-23T18:18:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-23T18:18:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: Unable to set osd_memory_target on smithi188 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T18:18:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-23T18:18:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-23T18:18:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:35.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:35.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-04-23T18:18:35.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:18:35.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:18:35.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:18:35.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: Updating smithi050:/etc/ceph/ceph.conf 2024-04-23T18:18:35.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-23T18:18:35.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-23T18:18:35.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4095005470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:35.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2492813782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:35.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:34 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3758968833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: Unable to set osd_memory_target on smithi188 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:18:35.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:18:35.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: Updating smithi050:/etc/ceph/ceph.conf 2024-04-23T18:18:35.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-23T18:18:35.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-23T18:18:35.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4095005470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:35.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2492813782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:35.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:34 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3758968833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:35.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.464+0000 7f181e8a7640 1 -- 172.21.15.50:0/2102803724 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f181810a410 con 0x7f18181010a0 2024-04-23T18:18:35.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.464+0000 7f18157fa640 1 -- 172.21.15.50:0/2102803724 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f180809cb20 con 0x7f18181010a0 2024-04-23T18:18:35.457 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2024-04-23T18:18:35.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.466+0000 7f181e8a7640 1 -- 172.21.15.50:0/2102803724 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f18000a3db0 msgr2=0x7f18000a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.466+0000 7f181e8a7640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f18000a3db0 0x7f18000a6270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f180c000c00 tx=0x7f180c032bc0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.466+0000 7f181e8a7640 1 -- 172.21.15.50:0/2102803724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18181010a0 msgr2=0x7f181819e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.466+0000 7f181e8a7640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18181010a0 0x7f181819e3f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f180800ea60 tx=0x7f180800ef20 comp rx=0 tx=0).stop 2024-04-23T18:18:35.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.466+0000 7f181e8a7640 1 -- 172.21.15.50:0/2102803724 shutdown_connections 2024-04-23T18:18:35.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.466+0000 7f181e8a7640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f18000a3db0 0x7f18000a6270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.466+0000 7f181e8a7640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18181985a0 0x7f1818198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.466+0000 7f181e8a7640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1818106170 0x7f181819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.466+0000 7f181e8a7640 1 --2- 172.21.15.50:0/2102803724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18181010a0 0x7f181819e3f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.466+0000 7f181e8a7640 1 -- 172.21.15.50:0/2102803724 >> 172.21.15.50:0/2102803724 conn(0x7f18180771d0 msgr2=0x7f18180fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:35.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.466+0000 7f181e8a7640 1 -- 172.21.15.50:0/2102803724 shutdown_connections 2024-04-23T18:18:35.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.466+0000 7f181e8a7640 1 -- 172.21.15.50:0/2102803724 wait complete. 2024-04-23T18:18:35.470 INFO:tasks.quiescer.ceph_manager:waiting for clean 2024-04-23T18:18:35.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:18:35.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.580+0000 7f2103594640 1 -- 172.21.15.50:0/1242323585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20fc106170 msgr2=0x7f20fc10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.580+0000 7f2103594640 1 --2- 172.21.15.50:0/1242323585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20fc106170 0x7f20fc10dd40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009ca0 tx=0x7f20f402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.580+0000 7f2103594640 1 -- 172.21.15.50:0/1242323585 shutdown_connections 2024-04-23T18:18:35.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.580+0000 7f2103594640 1 --2- 172.21.15.50:0/1242323585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20fc106170 0x7f20fc10dd40 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.580+0000 7f2103594640 1 --2- 172.21.15.50:0/1242323585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20fc101a50 0x7f20fc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.580+0000 7f2103594640 1 --2- 172.21.15.50:0/1242323585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc1010a0 0x7f20fc101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.580+0000 7f2103594640 1 -- 172.21.15.50:0/1242323585 >> 172.21.15.50:0/1242323585 conn(0x7f20fc077f50 msgr2=0x7f20fc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.580+0000 7f2103594640 1 -- 172.21.15.50:0/1242323585 shutdown_connections 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.580+0000 7f2103594640 1 -- 172.21.15.50:0/1242323585 wait complete. 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2103594640 1 Processor -- start 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2103594640 1 -- start start 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2103594640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20fc1010a0 0x7f20fc19e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2103594640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20fc106170 0x7f20fc19e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2103594640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc1985c0 0x7f20fc198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2103594640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc111130 con 0x7f20fc106170 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2103594640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc110fb0 con 0x7f20fc1010a0 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2103594640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc1112b0 con 0x7f20fc1985c0 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2100b2f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20fc106170 0x7f20fc19e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2101330640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20fc1010a0 0x7f20fc19e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2100b2f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20fc106170 0x7f20fc19e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49230/0 (socket says 172.21.15.50:49230) 2024-04-23T18:18:35.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2100b2f640 1 -- 172.21.15.50:0/4110210363 learned_addr learned my addr 172.21.15.50:0/4110210363 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:35.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2101b31640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc1985c0 0x7f20fc198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:35.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2100b2f640 1 -- 172.21.15.50:0/4110210363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc1985c0 msgr2=0x7f20fc198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2100b2f640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc1985c0 0x7f20fc198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2100b2f640 1 -- 172.21.15.50:0/4110210363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20fc1010a0 msgr2=0x7f20fc19e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2100b2f640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20fc1010a0 0x7f20fc19e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2100b2f640 1 -- 172.21.15.50:0/4110210363 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20fc1990e0 con 0x7f20fc106170 2024-04-23T18:18:35.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.581+0000 7f2101330640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20fc1010a0 0x7f20fc19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:35.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.582+0000 7f2100b2f640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20fc106170 0x7f20fc19e950 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9c0 tx=0x7f20ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:35.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.582+0000 7f20f27fc640 1 -- 172.21.15.50:0/4110210363 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec00cd80 con 0x7f20fc106170 2024-04-23T18:18:35.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.582+0000 7f20f27fc640 1 -- 172.21.15.50:0/4110210363 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20ec0054d0 con 0x7f20fc106170 2024-04-23T18:18:35.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.582+0000 7f20f27fc640 1 -- 172.21.15.50:0/4110210363 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec010430 con 0x7f20fc106170 2024-04-23T18:18:35.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.582+0000 7f2103594640 1 -- 172.21.15.50:0/4110210363 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20fc1993d0 con 0x7f20fc106170 2024-04-23T18:18:35.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.582+0000 7f2103594640 1 -- 172.21.15.50:0/4110210363 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20fc1a5620 con 0x7f20fc106170 2024-04-23T18:18:35.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.583+0000 7f2103594640 1 -- 172.21.15.50:0/4110210363 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20fc068d70 con 0x7f20fc106170 2024-04-23T18:18:35.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.584+0000 7f20f27fc640 1 -- 172.21.15.50:0/4110210363 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f20ec004fd0 con 0x7f20fc106170 2024-04-23T18:18:35.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.585+0000 7f20f27fc640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f20e80a3db0 0x7f20e80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:35.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.585+0000 7f2101330640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f20e80a3db0 0x7f20e80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:35.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.585+0000 7f20f27fc640 1 -- 172.21.15.50:0/4110210363 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f20ec0d0d60 con 0x7f20fc106170 2024-04-23T18:18:35.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.585+0000 7f2101330640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f20e80a3db0 0x7f20e80a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f20f8005fd0 tx=0x7f20f800b390 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:35.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.589+0000 7f20f27fc640 1 -- 172.21.15.50:0/4110210363 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f20ec09d240 con 0x7f20fc106170 2024-04-23T18:18:35.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.772+0000 7f2103594640 1 -- 172.21.15.50:0/4110210363 --> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f20fc101ac0 con 0x7f20e80a3db0 2024-04-23T18:18:35.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.780+0000 7f20f27fc640 1 -- 172.21.15.50:0/4110210363 <== mgr.25018 v2:172.21.15.50:6800/4210816660 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257741 (secure 0 0 0) 0x7f20fc101ac0 con 0x7f20e80a3db0 2024-04-23T18:18:35.776 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:35.777 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-23T18:18:35.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.783+0000 7f2103594640 1 -- 172.21.15.50:0/4110210363 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f20e80a3db0 msgr2=0x7f20e80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.783+0000 7f2103594640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f20e80a3db0 0x7f20e80a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f20f8005fd0 tx=0x7f20f800b390 comp rx=0 tx=0).stop 2024-04-23T18:18:35.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.783+0000 7f2103594640 1 -- 172.21.15.50:0/4110210363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20fc106170 msgr2=0x7f20fc19e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.783+0000 7f2103594640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20fc106170 0x7f20fc19e950 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9c0 tx=0x7f20ec00ee80 comp rx=0 tx=0).stop 2024-04-23T18:18:35.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.783+0000 7f2103594640 1 -- 172.21.15.50:0/4110210363 shutdown_connections 2024-04-23T18:18:35.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.783+0000 7f2103594640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f20e80a3db0 0x7f20e80a6270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.783+0000 7f2103594640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20fc1985c0 0x7f20fc198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.783+0000 7f2103594640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20fc106170 0x7f20fc19e950 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.783+0000 7f2103594640 1 --2- 172.21.15.50:0/4110210363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20fc1010a0 0x7f20fc19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.783+0000 7f2103594640 1 -- 172.21.15.50:0/4110210363 >> 172.21.15.50:0/4110210363 conn(0x7f20fc077f50 msgr2=0x7f20fc0ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:35.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.783+0000 7f2103594640 1 -- 172.21.15.50:0/4110210363 shutdown_connections 2024-04-23T18:18:35.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.783+0000 7f2103594640 1 -- 172.21.15.50:0/4110210363 wait complete. 2024-04-23T18:18:35.787 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-23T18:18:34.524804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188308,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":940,"num_read_kb":634,"num_write":475,"num_write_kb":4382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181883,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":397756,"kb_used_data":15296,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124544068,"statfs":{"total":1151940427776,"available":1151533125632,"internally_reserved":0,"allocated":15663104,"data_stored":9276020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991529"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672105+0000","last_change":"2024-04-23T18:16:33.987102+0000","last_active":"2024-04-23T18:18:30.672105+0000","last_peered":"2024-04-23T18:18:30.672105+0000","last_clean":"2024-04-23T18:18:30.672105+0000","last_became_active":"2024-04-23T18:16:33.986716+0000","last_became_peered":"2024-04-23T18:16:33.986716+0000","last_unstale":"2024-04-23T18:18:30.672105+0000","last_undegraded":"2024-04-23T18:18:30.672105+0000","last_fullsized":"2024-04-23T18:18:30.672105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:41.895219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880751+0000","last_change":"2024-04-23T18:16:33.995703+0000","last_active":"2024-04-23T18:18:30.880751+0000","last_peered":"2024-04-23T18:18:30.880751+0000","last_clean":"2024-04-23T18:18:30.880751+0000","last_became_active":"2024-04-23T18:16:33.995242+0000","last_became_peered":"2024-04-23T18:16:33.995242+0000","last_unstale":"2024-04-23T18:18:30.880751+0000","last_undegraded":"2024-04-23T18:18:30.880751+0000","last_fullsized":"2024-04-23T18:18:30.880751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:57.813822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880606+0000","last_change":"2024-04-23T18:16:34.000864+0000","last_active":"2024-04-23T18:18:30.880606+0000","last_peered":"2024-04-23T18:18:30.880606+0000","last_clean":"2024-04-23T18:18:30.880606+0000","last_became_active":"2024-04-23T18:16:33.998934+0000","last_became_peered":"2024-04-23T18:16:33.998934+0000","last_unstale":"2024-04-23T18:18:30.880606+0000","last_undegraded":"2024-04-23T18:18:30.880606+0000","last_fullsized":"2024-04-23T18:18:30.880606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:47:29.481697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672137+0000","last_change":"2024-04-23T18:16:33.989351+0000","last_active":"2024-04-23T18:18:30.672137+0000","last_peered":"2024-04-23T18:18:30.672137+0000","last_clean":"2024-04-23T18:18:30.672137+0000","last_became_active":"2024-04-23T18:16:33.989030+0000","last_became_peered":"2024-04-23T18:16:33.989030+0000","last_unstale":"2024-04-23T18:18:30.672137+0000","last_undegraded":"2024-04-23T18:18:30.672137+0000","last_fullsized":"2024-04-23T18:18:30.672137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:29:03.023675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608347+0000","last_change":"2024-04-23T18:16:33.996062+0000","last_active":"2024-04-23T18:18:30.608347+0000","last_peered":"2024-04-23T18:18:30.608347+0000","last_clean":"2024-04-23T18:18:30.608347+0000","last_became_active":"2024-04-23T18:16:33.995732+0000","last_became_peered":"2024-04-23T18:16:33.995732+0000","last_unstale":"2024-04-23T18:18:30.608347+0000","last_undegraded":"2024-04-23T18:18:30.608347+0000","last_fullsized":"2024-04-23T18:18:30.608347+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:04.175580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.617038+0000","last_change":"2024-04-23T18:16:33.989895+0000","last_active":"2024-04-23T18:18:30.617038+0000","last_peered":"2024-04-23T18:18:30.617038+0000","last_clean":"2024-04-23T18:18:30.617038+0000","last_became_active":"2024-04-23T18:16:33.989547+0000","last_became_peered":"2024-04-23T18:16:33.989547+0000","last_unstale":"2024-04-23T18:18:30.617038+0000","last_undegraded":"2024-04-23T18:18:30.617038+0000","last_fullsized":"2024-04-23T18:18:30.617038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:12:37.317974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672251+0000","last_change":"2024-04-23T18:16:33.999636+0000","last_active":"2024-04-23T18:18:30.672251+0000","last_peered":"2024-04-23T18:18:30.672251+0000","last_clean":"2024-04-23T18:18:30.672251+0000","last_became_active":"2024-04-23T18:16:33.999282+0000","last_became_peered":"2024-04-23T18:16:33.999282+0000","last_unstale":"2024-04-23T18:18:30.672251+0000","last_undegraded":"2024-04-23T18:18:30.672251+0000","last_fullsized":"2024-04-23T18:18:30.672251+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:28.841392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608063+0000","last_change":"2024-04-23T18:16:33.996602+0000","last_active":"2024-04-23T18:18:30.608063+0000","last_peered":"2024-04-23T18:18:30.608063+0000","last_clean":"2024-04-23T18:18:30.608063+0000","last_became_active":"2024-04-23T18:16:33.994401+0000","last_became_peered":"2024-04-23T18:16:33.994401+0000","last_unstale":"2024-04-23T18:18:30.608063+0000","last_undegraded":"2024-04-23T18:18:30.608063+0000","last_fullsized":"2024-04-23T18:18:30.608063+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:31:54.553537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510925+0000","last_change":"2024-04-23T18:16:33.994763+0000","last_active":"2024-04-23T18:18:30.510925+0000","last_peered":"2024-04-23T18:18:30.510925+0000","last_clean":"2024-04-23T18:18:30.510925+0000","last_became_active":"2024-04-23T18:16:33.994408+0000","last_became_peered":"2024-04-23T18:16:33.994408+0000","last_unstale":"2024-04-23T18:18:30.510925+0000","last_undegraded":"2024-04-23T18:18:30.510925+0000","last_fullsized":"2024-04-23T18:18:30.510925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:02.706160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.617268+0000","last_change":"2024-04-23T18:16:33.993435+0000","last_active":"2024-04-23T18:18:30.617268+0000","last_peered":"2024-04-23T18:18:30.617268+0000","last_clean":"2024-04-23T18:18:30.617268+0000","last_became_active":"2024-04-23T18:16:33.993108+0000","last_became_peered":"2024-04-23T18:16:33.993108+0000","last_unstale":"2024-04-23T18:18:30.617268+0000","last_undegraded":"2024-04-23T18:18:30.617268+0000","last_fullsized":"2024-04-23T18:18:30.617268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:28:32.584896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672170+0000","last_change":"2024-04-23T18:16:33.997526+0000","last_active":"2024-04-23T18:18:30.672170+0000","last_peered":"2024-04-23T18:18:30.672170+0000","last_clean":"2024-04-23T18:18:30.672170+0000","last_became_active":"2024-04-23T18:16:33.997189+0000","last_became_peered":"2024-04-23T18:16:33.997189+0000","last_unstale":"2024-04-23T18:18:30.672170+0000","last_undegraded":"2024-04-23T18:18:30.672170+0000","last_fullsized":"2024-04-23T18:18:30.672170+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:11:41.208261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671985+0000","last_change":"2024-04-23T18:16:33.989775+0000","last_active":"2024-04-23T18:18:30.671985+0000","last_peered":"2024-04-23T18:18:30.671985+0000","last_clean":"2024-04-23T18:18:30.671985+0000","last_became_active":"2024-04-23T18:16:33.989382+0000","last_became_peered":"2024-04-23T18:16:33.989382+0000","last_unstale":"2024-04-23T18:18:30.671985+0000","last_undegraded":"2024-04-23T18:18:30.671985+0000","last_fullsized":"2024-04-23T18:18:30.671985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:20:17.756258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672114+0000","last_change":"2024-04-23T18:16:34.005871+0000","last_active":"2024-04-23T18:18:30.672114+0000","last_peered":"2024-04-23T18:18:30.672114+0000","last_clean":"2024-04-23T18:18:30.672114+0000","last_became_active":"2024-04-23T18:16:34.005218+0000","last_became_peered":"2024-04-23T18:16:34.005218+0000","last_unstale":"2024-04-23T18:18:30.672114+0000","last_undegraded":"2024-04-23T18:18:30.672114+0000","last_fullsized":"2024-04-23T18:18:30.672114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:30:03.864306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880355+0000","last_change":"2024-04-23T18:16:33.995054+0000","last_active":"2024-04-23T18:18:30.880355+0000","last_peered":"2024-04-23T18:18:30.880355+0000","last_clean":"2024-04-23T18:18:30.880355+0000","last_became_active":"2024-04-23T18:16:33.994389+0000","last_became_peered":"2024-04-23T18:16:33.994389+0000","last_unstale":"2024-04-23T18:18:30.880355+0000","last_undegraded":"2024-04-23T18:18:30.880355+0000","last_fullsized":"2024-04-23T18:18:30.880355+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:27.344616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880258+0000","last_change":"2024-04-23T18:16:33.984490+0000","last_active":"2024-04-23T18:18:30.880258+0000","last_peered":"2024-04-23T18:18:30.880258+0000","last_clean":"2024-04-23T18:18:30.880258+0000","last_became_active":"2024-04-23T18:16:33.984071+0000","last_became_peered":"2024-04-23T18:16:33.984071+0000","last_unstale":"2024-04-23T18:18:30.880258+0000","last_undegraded":"2024-04-23T18:18:30.880258+0000","last_fullsized":"2024-04-23T18:18:30.880258+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:23.589290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510517+0000","last_change":"2024-04-23T18:16:34.002163+0000","last_active":"2024-04-23T18:18:30.510517+0000","last_peered":"2024-04-23T18:18:30.510517+0000","last_clean":"2024-04-23T18:18:30.510517+0000","last_became_active":"2024-04-23T18:16:34.001764+0000","last_became_peered":"2024-04-23T18:16:34.001764+0000","last_unstale":"2024-04-23T18:18:30.510517+0000","last_undegraded":"2024-04-23T18:18:30.510517+0000","last_fullsized":"2024-04-23T18:18:30.510517+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:16:03.616358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672114+0000","last_change":"2024-04-23T18:16:33.987586+0000","last_active":"2024-04-23T18:18:30.672114+0000","last_peered":"2024-04-23T18:18:30.672114+0000","last_clean":"2024-04-23T18:18:30.672114+0000","last_became_active":"2024-04-23T18:16:33.987266+0000","last_became_peered":"2024-04-23T18:16:33.987266+0000","last_unstale":"2024-04-23T18:18:30.672114+0000","last_undegraded":"2024-04-23T18:18:30.672114+0000","last_fullsized":"2024-04-23T18:18:30.672114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:41.610529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508920+0000","last_change":"2024-04-23T18:16:33.992938+0000","last_active":"2024-04-23T18:18:30.508920+0000","last_peered":"2024-04-23T18:18:30.508920+0000","last_clean":"2024-04-23T18:18:30.508920+0000","last_became_active":"2024-04-23T18:16:33.990441+0000","last_became_peered":"2024-04-23T18:16:33.990441+0000","last_unstale":"2024-04-23T18:18:30.508920+0000","last_undegraded":"2024-04-23T18:18:30.508920+0000","last_fullsized":"2024-04-23T18:18:30.508920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:10.689333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512446+0000","last_change":"2024-04-23T18:16:33.994844+0000","last_active":"2024-04-23T18:18:30.512446+0000","last_peered":"2024-04-23T18:18:30.512446+0000","last_clean":"2024-04-23T18:18:30.512446+0000","last_became_active":"2024-04-23T18:16:33.994505+0000","last_became_peered":"2024-04-23T18:16:33.994505+0000","last_unstale":"2024-04-23T18:18:30.512446+0000","last_undegraded":"2024-04-23T18:18:30.512446+0000","last_fullsized":"2024-04-23T18:18:30.512446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:32:33.445266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508971+0000","last_change":"2024-04-23T18:16:32.958841+0000","last_active":"2024-04-23T18:18:30.508971+0000","last_peered":"2024-04-23T18:18:30.508971+0000","last_clean":"2024-04-23T18:18:30.508971+0000","last_became_active":"2024-04-23T18:16:32.958529+0000","last_became_peered":"2024-04-23T18:16:32.958529+0000","last_unstale":"2024-04-23T18:18:30.508971+0000","last_undegraded":"2024-04-23T18:18:30.508971+0000","last_fullsized":"2024-04-23T18:18:30.508971+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:47:09.933786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608592+0000","last_change":"2024-04-23T18:16:33.992108+0000","last_active":"2024-04-23T18:18:30.608592+0000","last_peered":"2024-04-23T18:18:30.608592+0000","last_clean":"2024-04-23T18:18:30.608592+0000","last_became_active":"2024-04-23T18:16:33.991751+0000","last_became_peered":"2024-04-23T18:16:33.991751+0000","last_unstale":"2024-04-23T18:18:30.608592+0000","last_undegraded":"2024-04-23T18:18:30.608592+0000","last_fullsized":"2024-04-23T18:18:30.608592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:07:57.416263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608867+0000","last_change":"2024-04-23T18:16:32.960461+0000","last_active":"2024-04-23T18:18:30.608867+0000","last_peered":"2024-04-23T18:18:30.608867+0000","last_clean":"2024-04-23T18:18:30.608867+0000","last_became_active":"2024-04-23T18:16:32.960115+0000","last_became_peered":"2024-04-23T18:16:32.960115+0000","last_unstale":"2024-04-23T18:18:30.608867+0000","last_undegraded":"2024-04-23T18:18:30.608867+0000","last_fullsized":"2024-04-23T18:18:30.608867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:30.533043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880852+0000","last_change":"2024-04-23T18:16:34.001831+0000","last_active":"2024-04-23T18:18:30.880852+0000","last_peered":"2024-04-23T18:18:30.880852+0000","last_clean":"2024-04-23T18:18:30.880852+0000","last_became_active":"2024-04-23T18:16:34.001486+0000","last_became_peered":"2024-04-23T18:16:34.001486+0000","last_unstale":"2024-04-23T18:18:30.880852+0000","last_undegraded":"2024-04-23T18:18:30.880852+0000","last_fullsized":"2024-04-23T18:18:30.880852+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:06:40.452366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608622+0000","last_change":"2024-04-23T18:16:32.960387+0000","last_active":"2024-04-23T18:18:30.608622+0000","last_peered":"2024-04-23T18:18:30.608622+0000","last_clean":"2024-04-23T18:18:30.608622+0000","last_became_active":"2024-04-23T18:16:32.960046+0000","last_became_peered":"2024-04-23T18:16:32.960046+0000","last_unstale":"2024-04-23T18:18:30.608622+0000","last_undegraded":"2024-04-23T18:18:30.608622+0000","last_fullsized":"2024-04-23T18:18:30.608622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:03.296653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510543+0000","last_change":"2024-04-23T18:16:33.996520+0000","last_active":"2024-04-23T18:18:30.510543+0000","last_peered":"2024-04-23T18:18:30.510543+0000","last_clean":"2024-04-23T18:18:30.510543+0000","last_became_active":"2024-04-23T18:16:33.995636+0000","last_became_peered":"2024-04-23T18:16:33.995636+0000","last_unstale":"2024-04-23T18:18:30.510543+0000","last_undegraded":"2024-04-23T18:18:30.510543+0000","last_fullsized":"2024-04-23T18:18:30.510543+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:29.665407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608556+0000","last_change":"2024-04-23T18:16:32.956813+0000","last_active":"2024-04-23T18:18:30.608556+0000","last_peered":"2024-04-23T18:18:30.608556+0000","last_clean":"2024-04-23T18:18:30.608556+0000","last_became_active":"2024-04-23T18:16:32.956505+0000","last_became_peered":"2024-04-23T18:16:32.956505+0000","last_unstale":"2024-04-23T18:18:30.608556+0000","last_undegraded":"2024-04-23T18:18:30.608556+0000","last_fullsized":"2024-04-23T18:18:30.608556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:36:27.330353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.614156+0000","last_change":"2024-04-23T18:16:33.999754+0000","last_active":"2024-04-23T18:18:30.614156+0000","last_peered":"2024-04-23T18:18:30.614156+0000","last_clean":"2024-04-23T18:18:30.614156+0000","last_became_active":"2024-04-23T18:16:33.999034+0000","last_became_peered":"2024-04-23T18:16:33.999034+0000","last_unstale":"2024-04-23T18:18:30.614156+0000","last_undegraded":"2024-04-23T18:18:30.614156+0000","last_fullsized":"2024-04-23T18:18:30.614156+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:26:08.595745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":160,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:31.190208+0000","last_change":"2024-04-23T18:17:36.120371+0000","last_active":"2024-04-23T18:18:31.190208+0000","last_peered":"2024-04-23T18:18:31.190208+0000","last_clean":"2024-04-23T18:18:31.190208+0000","last_became_active":"2024-04-23T18:17:36.116149+0000","last_became_peered":"2024-04-23T18:17:36.116149+0000","last_unstale":"2024-04-23T18:18:31.190208+0000","last_undegraded":"2024-04-23T18:18:31.190208+0000","last_fullsized":"2024-04-23T18:18:31.190208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:54.994254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608344+0000","last_change":"2024-04-23T18:16:33.995166+0000","last_active":"2024-04-23T18:18:30.608344+0000","last_peered":"2024-04-23T18:18:30.608344+0000","last_clean":"2024-04-23T18:18:30.608344+0000","last_became_active":"2024-04-23T18:16:33.994347+0000","last_became_peered":"2024-04-23T18:16:33.994347+0000","last_unstale":"2024-04-23T18:18:30.608344+0000","last_undegraded":"2024-04-23T18:18:30.608344+0000","last_fullsized":"2024-04-23T18:18:30.608344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.305837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.616932+0000","last_change":"2024-04-23T18:16:32.959358+0000","last_active":"2024-04-23T18:18:30.616932+0000","last_peered":"2024-04-23T18:18:30.616932+0000","last_clean":"2024-04-23T18:18:30.616932+0000","last_became_active":"2024-04-23T18:16:32.959036+0000","last_became_peered":"2024-04-23T18:16:32.959036+0000","last_unstale":"2024-04-23T18:18:30.616932+0000","last_undegraded":"2024-04-23T18:18:30.616932+0000","last_fullsized":"2024-04-23T18:18:30.616932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:03:15.734754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.614091+0000","last_change":"2024-04-23T18:16:33.988578+0000","last_active":"2024-04-23T18:18:30.614091+0000","last_peered":"2024-04-23T18:18:30.614091+0000","last_clean":"2024-04-23T18:18:30.614091+0000","last_became_active":"2024-04-23T18:16:33.988248+0000","last_became_peered":"2024-04-23T18:16:33.988248+0000","last_unstale":"2024-04-23T18:18:30.614091+0000","last_undegraded":"2024-04-23T18:18:30.614091+0000","last_fullsized":"2024-04-23T18:18:30.614091+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:45:15.538971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671882+0000","last_change":"2024-04-23T18:16:32.952945+0000","last_active":"2024-04-23T18:18:30.671882+0000","last_peered":"2024-04-23T18:18:30.671882+0000","last_clean":"2024-04-23T18:18:30.671882+0000","last_became_active":"2024-04-23T18:16:32.952602+0000","last_became_peered":"2024-04-23T18:16:32.952602+0000","last_unstale":"2024-04-23T18:18:30.671882+0000","last_undegraded":"2024-04-23T18:18:30.671882+0000","last_fullsized":"2024-04-23T18:18:30.671882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:20.037431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608129+0000","last_change":"2024-04-23T18:16:33.996897+0000","last_active":"2024-04-23T18:18:30.608129+0000","last_peered":"2024-04-23T18:18:30.608129+0000","last_clean":"2024-04-23T18:18:30.608129+0000","last_became_active":"2024-04-23T18:16:33.996214+0000","last_became_peered":"2024-04-23T18:16:33.996214+0000","last_unstale":"2024-04-23T18:18:30.608129+0000","last_undegraded":"2024-04-23T18:18:30.608129+0000","last_fullsized":"2024-04-23T18:18:30.608129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:34.931697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.613841+0000","last_change":"2024-04-23T18:16:32.959637+0000","last_active":"2024-04-23T18:18:30.613841+0000","last_peered":"2024-04-23T18:18:30.613841+0000","last_clean":"2024-04-23T18:18:30.613841+0000","last_became_active":"2024-04-23T18:16:32.959332+0000","last_became_peered":"2024-04-23T18:16:32.959332+0000","last_unstale":"2024-04-23T18:18:30.613841+0000","last_undegraded":"2024-04-23T18:18:30.613841+0000","last_fullsized":"2024-04-23T18:18:30.613841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:30:49.089101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508653+0000","last_change":"2024-04-23T18:16:33.986288+0000","last_active":"2024-04-23T18:18:30.508653+0000","last_peered":"2024-04-23T18:18:30.508653+0000","last_clean":"2024-04-23T18:18:30.508653+0000","last_became_active":"2024-04-23T18:16:33.985866+0000","last_became_peered":"2024-04-23T18:16:33.985866+0000","last_unstale":"2024-04-23T18:18:30.508653+0000","last_undegraded":"2024-04-23T18:18:30.508653+0000","last_fullsized":"2024-04-23T18:18:30.508653+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:30:17.675952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":140,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.761189+0000","last_change":"2024-04-23T18:16:32.958466+0000","last_active":"2024-04-23T18:18:30.761189+0000","last_peered":"2024-04-23T18:18:30.761189+0000","last_clean":"2024-04-23T18:18:30.761189+0000","last_became_active":"2024-04-23T18:16:32.958155+0000","last_became_peered":"2024-04-23T18:16:32.958155+0000","last_unstale":"2024-04-23T18:18:30.761189+0000","last_undegraded":"2024-04-23T18:18:30.761189+0000","last_fullsized":"2024-04-23T18:18:30.761189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:15:49.073934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510826+0000","last_change":"2024-04-23T18:16:33.996156+0000","last_active":"2024-04-23T18:18:30.510826+0000","last_peered":"2024-04-23T18:18:30.510826+0000","last_clean":"2024-04-23T18:18:30.510826+0000","last_became_active":"2024-04-23T18:16:33.995711+0000","last_became_peered":"2024-04-23T18:16:33.995711+0000","last_unstale":"2024-04-23T18:18:30.510826+0000","last_undegraded":"2024-04-23T18:18:30.510826+0000","last_fullsized":"2024-04-23T18:18:30.510826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:45.272598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672184+0000","last_change":"2024-04-23T18:17:36.217460+0000","last_active":"2024-04-23T18:18:30.672184+0000","last_peered":"2024-04-23T18:18:30.672184+0000","last_clean":"2024-04-23T18:18:30.672184+0000","last_became_active":"2024-04-23T18:17:36.214196+0000","last_became_peered":"2024-04-23T18:17:36.214196+0000","last_unstale":"2024-04-23T18:18:30.672184+0000","last_undegraded":"2024-04-23T18:18:30.672184+0000","last_fullsized":"2024-04-23T18:18:30.672184+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:34:55.557943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880842+0000","last_change":"2024-04-23T18:17:36.114238+0000","last_active":"2024-04-23T18:18:30.880842+0000","last_peered":"2024-04-23T18:18:30.880842+0000","last_clean":"2024-04-23T18:18:30.880842+0000","last_became_active":"2024-04-23T18:17:36.113810+0000","last_became_peered":"2024-04-23T18:17:36.113810+0000","last_unstale":"2024-04-23T18:18:30.880842+0000","last_undegraded":"2024-04-23T18:18:30.880842+0000","last_fullsized":"2024-04-23T18:18:30.880842+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:53.854596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.614213+0000","last_change":"2024-04-23T18:16:32.966725+0000","last_active":"2024-04-23T18:18:30.614213+0000","last_peered":"2024-04-23T18:18:30.614213+0000","last_clean":"2024-04-23T18:18:30.614213+0000","last_became_active":"2024-04-23T18:16:32.966304+0000","last_became_peered":"2024-04-23T18:16:32.966304+0000","last_unstale":"2024-04-23T18:18:30.614213+0000","last_undegraded":"2024-04-23T18:18:30.614213+0000","last_fullsized":"2024-04-23T18:18:30.614213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:45.191051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510708+0000","last_change":"2024-04-23T18:16:33.994667+0000","last_active":"2024-04-23T18:18:30.510708+0000","last_peered":"2024-04-23T18:18:30.510708+0000","last_clean":"2024-04-23T18:18:30.510708+0000","last_became_active":"2024-04-23T18:16:33.994305+0000","last_became_peered":"2024-04-23T18:16:33.994305+0000","last_unstale":"2024-04-23T18:18:30.510708+0000","last_undegraded":"2024-04-23T18:18:30.510708+0000","last_fullsized":"2024-04-23T18:18:30.510708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:23.807075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'37","reported_seq":298,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:31.190262+0000","last_change":"2024-04-23T18:16:32.957946+0000","last_active":"2024-04-23T18:18:31.190262+0000","last_peered":"2024-04-23T18:18:31.190262+0000","last_clean":"2024-04-23T18:18:31.190262+0000","last_became_active":"2024-04-23T18:16:32.957478+0000","last_became_peered":"2024-04-23T18:16:32.957478+0000","last_unstale":"2024-04-23T18:18:31.190262+0000","last_undegraded":"2024-04-23T18:18:31.190262+0000","last_fullsized":"2024-04-23T18:18:31.190262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:16.850604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66735,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":240,"num_read_kb":185,"num_write":38,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608533+0000","last_change":"2024-04-23T18:16:33.989537+0000","last_active":"2024-04-23T18:18:30.608533+0000","last_peered":"2024-04-23T18:18:30.608533+0000","last_clean":"2024-04-23T18:18:30.608533+0000","last_became_active":"2024-04-23T18:16:33.989180+0000","last_became_peered":"2024-04-23T18:16:33.989180+0000","last_unstale":"2024-04-23T18:18:30.608533+0000","last_undegraded":"2024-04-23T18:18:30.608533+0000","last_fullsized":"2024-04-23T18:18:30.608533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:53:18.700347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.511992+0000","last_change":"2024-04-23T18:16:32.967314+0000","last_active":"2024-04-23T18:18:30.511992+0000","last_peered":"2024-04-23T18:18:30.511992+0000","last_clean":"2024-04-23T18:18:30.511992+0000","last_became_active":"2024-04-23T18:16:32.966861+0000","last_became_peered":"2024-04-23T18:16:32.966861+0000","last_unstale":"2024-04-23T18:18:30.511992+0000","last_undegraded":"2024-04-23T18:18:30.511992+0000","last_fullsized":"2024-04-23T18:18:30.511992+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:00:25.606402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671655+0000","last_change":"2024-04-23T18:16:33.992593+0000","last_active":"2024-04-23T18:18:30.671655+0000","last_peered":"2024-04-23T18:18:30.671655+0000","last_clean":"2024-04-23T18:18:30.671655+0000","last_became_active":"2024-04-23T18:16:33.992228+0000","last_became_peered":"2024-04-23T18:16:33.992228+0000","last_unstale":"2024-04-23T18:18:30.671655+0000","last_undegraded":"2024-04-23T18:18:30.671655+0000","last_fullsized":"2024-04-23T18:18:30.671655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:12:23.836829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.613908+0000","last_change":"2024-04-23T18:16:32.961740+0000","last_active":"2024-04-23T18:18:30.613908+0000","last_peered":"2024-04-23T18:18:30.613908+0000","last_clean":"2024-04-23T18:18:30.613908+0000","last_became_active":"2024-04-23T18:16:32.961331+0000","last_became_peered":"2024-04-23T18:16:32.961331+0000","last_unstale":"2024-04-23T18:18:30.613908+0000","last_undegraded":"2024-04-23T18:18:30.613908+0000","last_fullsized":"2024-04-23T18:18:30.613908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:59:17.665288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671999+0000","last_change":"2024-04-23T18:17:36.111151+0000","last_active":"2024-04-23T18:18:30.671999+0000","last_peered":"2024-04-23T18:18:30.671999+0000","last_clean":"2024-04-23T18:18:30.671999+0000","last_became_active":"2024-04-23T18:17:36.110770+0000","last_became_peered":"2024-04-23T18:17:36.110770+0000","last_unstale":"2024-04-23T18:18:30.671999+0000","last_undegraded":"2024-04-23T18:18:30.671999+0000","last_fullsized":"2024-04-23T18:18:30.671999+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:01:03.456738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:31.133054+0000","last_change":"2024-04-23T18:16:32.954935+0000","last_active":"2024-04-23T18:18:31.133054+0000","last_peered":"2024-04-23T18:18:31.133054+0000","last_clean":"2024-04-23T18:18:31.133054+0000","last_became_active":"2024-04-23T18:16:32.954595+0000","last_became_peered":"2024-04-23T18:16:32.954595+0000","last_unstale":"2024-04-23T18:18:31.133054+0000","last_undegraded":"2024-04-23T18:18:31.133054+0000","last_fullsized":"2024-04-23T18:18:31.133054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:22.954101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671780+0000","last_change":"2024-04-23T18:16:34.000215+0000","last_active":"2024-04-23T18:18:30.671780+0000","last_peered":"2024-04-23T18:18:30.671780+0000","last_clean":"2024-04-23T18:18:30.671780+0000","last_became_active":"2024-04-23T18:16:33.999855+0000","last_became_peered":"2024-04-23T18:16:33.999855+0000","last_unstale":"2024-04-23T18:18:30.671780+0000","last_undegraded":"2024-04-23T18:18:30.671780+0000","last_fullsized":"2024-04-23T18:18:30.671780+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:14.686916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671868+0000","last_change":"2024-04-23T18:17:36.127721+0000","last_active":"2024-04-23T18:18:30.671868+0000","last_peered":"2024-04-23T18:18:30.671868+0000","last_clean":"2024-04-23T18:18:30.671868+0000","last_became_active":"2024-04-23T18:17:36.113735+0000","last_became_peered":"2024-04-23T18:17:36.113735+0000","last_unstale":"2024-04-23T18:18:30.671868+0000","last_undegraded":"2024-04-23T18:18:30.671868+0000","last_fullsized":"2024-04-23T18:18:30.671868+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:56:41.395925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508868+0000","last_change":"2024-04-23T18:17:36.214244+0000","last_active":"2024-04-23T18:18:30.508868+0000","last_peered":"2024-04-23T18:18:30.508868+0000","last_clean":"2024-04-23T18:18:30.508868+0000","last_became_active":"2024-04-23T18:17:36.213897+0000","last_became_peered":"2024-04-23T18:17:36.213897+0000","last_unstale":"2024-04-23T18:18:30.508868+0000","last_undegraded":"2024-04-23T18:18:30.508868+0000","last_fullsized":"2024-04-23T18:18:30.508868+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:44:24.935747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608788+0000","last_change":"2024-04-23T18:16:32.959253+0000","last_active":"2024-04-23T18:18:30.608788+0000","last_peered":"2024-04-23T18:18:30.608788+0000","last_clean":"2024-04-23T18:18:30.608788+0000","last_became_active":"2024-04-23T18:16:32.958929+0000","last_became_peered":"2024-04-23T18:16:32.958929+0000","last_unstale":"2024-04-23T18:18:30.608788+0000","last_undegraded":"2024-04-23T18:18:30.608788+0000","last_fullsized":"2024-04-23T18:18:30.608788+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:36.317516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672022+0000","last_change":"2024-04-23T18:16:33.979464+0000","last_active":"2024-04-23T18:18:30.672022+0000","last_peered":"2024-04-23T18:18:30.672022+0000","last_clean":"2024-04-23T18:18:30.672022+0000","last_became_active":"2024-04-23T18:16:33.979106+0000","last_became_peered":"2024-04-23T18:16:33.979106+0000","last_unstale":"2024-04-23T18:18:30.672022+0000","last_undegraded":"2024-04-23T18:18:30.672022+0000","last_fullsized":"2024-04-23T18:18:30.672022+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:00.125514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508806+0000","last_change":"2024-04-23T18:16:32.960812+0000","last_active":"2024-04-23T18:18:30.508806+0000","last_peered":"2024-04-23T18:18:30.508806+0000","last_clean":"2024-04-23T18:18:30.508806+0000","last_became_active":"2024-04-23T18:16:32.959807+0000","last_became_peered":"2024-04-23T18:16:32.959807+0000","last_unstale":"2024-04-23T18:18:30.508806+0000","last_undegraded":"2024-04-23T18:18:30.508806+0000","last_fullsized":"2024-04-23T18:18:30.508806+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:45.097443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.617178+0000","last_change":"2024-04-23T18:16:33.992891+0000","last_active":"2024-04-23T18:18:30.617178+0000","last_peered":"2024-04-23T18:18:30.617178+0000","last_clean":"2024-04-23T18:18:30.617178+0000","last_became_active":"2024-04-23T18:16:33.992559+0000","last_became_peered":"2024-04-23T18:16:33.992559+0000","last_unstale":"2024-04-23T18:18:30.617178+0000","last_undegraded":"2024-04-23T18:18:30.617178+0000","last_fullsized":"2024-04-23T18:18:30.617178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:48.615158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671986+0000","last_change":"2024-04-23T18:16:32.957015+0000","last_active":"2024-04-23T18:18:30.671986+0000","last_peered":"2024-04-23T18:18:30.671986+0000","last_clean":"2024-04-23T18:18:30.671986+0000","last_became_active":"2024-04-23T18:16:32.956690+0000","last_became_peered":"2024-04-23T18:16:32.956690+0000","last_unstale":"2024-04-23T18:18:30.671986+0000","last_undegraded":"2024-04-23T18:18:30.671986+0000","last_fullsized":"2024-04-23T18:18:30.671986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:20:42.087521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.617121+0000","last_change":"2024-04-23T18:16:33.998906+0000","last_active":"2024-04-23T18:18:30.617121+0000","last_peered":"2024-04-23T18:18:30.617121+0000","last_clean":"2024-04-23T18:18:30.617121+0000","last_became_active":"2024-04-23T18:16:33.997297+0000","last_became_peered":"2024-04-23T18:16:33.997297+0000","last_unstale":"2024-04-23T18:18:30.617121+0000","last_undegraded":"2024-04-23T18:18:30.617121+0000","last_fullsized":"2024-04-23T18:18:30.617121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:06.668830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.614008+0000","last_change":"2024-04-23T18:17:36.113962+0000","last_active":"2024-04-23T18:18:30.614008+0000","last_peered":"2024-04-23T18:18:30.614008+0000","last_clean":"2024-04-23T18:18:30.614008+0000","last_became_active":"2024-04-23T18:17:36.113573+0000","last_became_peered":"2024-04-23T18:17:36.113573+0000","last_unstale":"2024-04-23T18:18:30.614008+0000","last_undegraded":"2024-04-23T18:18:30.614008+0000","last_fullsized":"2024-04-23T18:18:30.614008+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:47:25.408144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'6","reported_seq":141,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671805+0000","last_change":"2024-04-23T18:16:32.961423+0000","last_active":"2024-04-23T18:18:30.671805+0000","last_peered":"2024-04-23T18:18:30.671805+0000","last_clean":"2024-04-23T18:18:30.671805+0000","last_became_active":"2024-04-23T18:16:32.961116+0000","last_became_peered":"2024-04-23T18:16:32.961116+0000","last_unstale":"2024-04-23T18:18:30.671805+0000","last_undegraded":"2024-04-23T18:18:30.671805+0000","last_fullsized":"2024-04-23T18:18:30.671805+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:28.525642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608360+0000","last_change":"2024-04-23T18:16:33.990747+0000","last_active":"2024-04-23T18:18:30.608360+0000","last_peered":"2024-04-23T18:18:30.608360+0000","last_clean":"2024-04-23T18:18:30.608360+0000","last_became_active":"2024-04-23T18:16:33.989972+0000","last_became_peered":"2024-04-23T18:16:33.989972+0000","last_unstale":"2024-04-23T18:18:30.608360+0000","last_undegraded":"2024-04-23T18:18:30.608360+0000","last_fullsized":"2024-04-23T18:18:30.608360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:04.667446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671772+0000","last_change":"2024-04-23T18:16:32.952821+0000","last_active":"2024-04-23T18:18:30.671772+0000","last_peered":"2024-04-23T18:18:30.671772+0000","last_clean":"2024-04-23T18:18:30.671772+0000","last_became_active":"2024-04-23T18:16:32.952470+0000","last_became_peered":"2024-04-23T18:16:32.952470+0000","last_unstale":"2024-04-23T18:18:30.671772+0000","last_undegraded":"2024-04-23T18:18:30.671772+0000","last_fullsized":"2024-04-23T18:18:30.671772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:19.392670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671938+0000","last_change":"2024-04-23T18:15:25.867370+0000","last_active":"2024-04-23T18:18:30.671938+0000","last_peered":"2024-04-23T18:18:30.671938+0000","last_clean":"2024-04-23T18:18:30.671938+0000","last_became_active":"2024-04-23T18:15:25.866815+0000","last_became_peered":"2024-04-23T18:15:25.866815+0000","last_unstale":"2024-04-23T18:18:30.671938+0000","last_undegraded":"2024-04-23T18:18:30.671938+0000","last_fullsized":"2024-04-23T18:18:30.671938+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_clean_scrub_stamp":"2024-04-23T18:12:02.913350+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:42.482422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671577+0000","last_change":"2024-04-23T18:16:33.992988+0000","last_active":"2024-04-23T18:18:30.671577+0000","last_peered":"2024-04-23T18:18:30.671577+0000","last_clean":"2024-04-23T18:18:30.671577+0000","last_became_active":"2024-04-23T18:16:33.992650+0000","last_became_peered":"2024-04-23T18:16:33.992650+0000","last_unstale":"2024-04-23T18:18:30.671577+0000","last_undegraded":"2024-04-23T18:18:30.671577+0000","last_fullsized":"2024-04-23T18:18:30.671577+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:54.054708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672214+0000","last_change":"2024-04-23T18:16:32.957318+0000","last_active":"2024-04-23T18:18:30.672214+0000","last_peered":"2024-04-23T18:18:30.672214+0000","last_clean":"2024-04-23T18:18:30.672214+0000","last_became_active":"2024-04-23T18:16:32.956973+0000","last_became_peered":"2024-04-23T18:16:32.956973+0000","last_unstale":"2024-04-23T18:18:30.672214+0000","last_undegraded":"2024-04-23T18:18:30.672214+0000","last_fullsized":"2024-04-23T18:18:30.672214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:35.684825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672061+0000","last_change":"2024-04-23T18:17:36.117253+0000","last_active":"2024-04-23T18:18:30.672061+0000","last_peered":"2024-04-23T18:18:30.672061+0000","last_clean":"2024-04-23T18:18:30.672061+0000","last_became_active":"2024-04-23T18:17:36.111210+0000","last_became_peered":"2024-04-23T18:17:36.111210+0000","last_unstale":"2024-04-23T18:18:30.672061+0000","last_undegraded":"2024-04-23T18:18:30.672061+0000","last_fullsized":"2024-04-23T18:18:30.672061+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:31.599001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672229+0000","last_change":"2024-04-23T18:16:33.996345+0000","last_active":"2024-04-23T18:18:30.672229+0000","last_peered":"2024-04-23T18:18:30.672229+0000","last_clean":"2024-04-23T18:18:30.672229+0000","last_became_active":"2024-04-23T18:16:33.993802+0000","last_became_peered":"2024-04-23T18:16:33.993802+0000","last_unstale":"2024-04-23T18:18:30.672229+0000","last_undegraded":"2024-04-23T18:18:30.672229+0000","last_fullsized":"2024-04-23T18:18:30.672229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:32.410330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'44","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.722525+0000","last_change":"2024-04-23T18:16:32.957953+0000","last_active":"2024-04-23T18:18:30.722525+0000","last_peered":"2024-04-23T18:18:30.722525+0000","last_clean":"2024-04-23T18:18:30.722525+0000","last_became_active":"2024-04-23T18:16:32.957616+0000","last_became_peered":"2024-04-23T18:16:32.957616+0000","last_unstale":"2024-04-23T18:18:30.722525+0000","last_undegraded":"2024-04-23T18:18:30.722525+0000","last_fullsized":"2024-04-23T18:18:30.722525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:15.485373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.609061+0000","last_change":"2024-04-23T18:16:33.985685+0000","last_active":"2024-04-23T18:18:30.609061+0000","last_peered":"2024-04-23T18:18:30.609061+0000","last_clean":"2024-04-23T18:18:30.609061+0000","last_became_active":"2024-04-23T18:16:33.984631+0000","last_became_peered":"2024-04-23T18:16:33.984631+0000","last_unstale":"2024-04-23T18:18:30.609061+0000","last_undegraded":"2024-04-23T18:18:30.609061+0000","last_fullsized":"2024-04-23T18:18:30.609061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:45.893285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672398+0000","last_change":"2024-04-23T18:16:32.954281+0000","last_active":"2024-04-23T18:18:30.672398+0000","last_peered":"2024-04-23T18:18:30.672398+0000","last_clean":"2024-04-23T18:18:30.672398+0000","last_became_active":"2024-04-23T18:16:32.953988+0000","last_became_peered":"2024-04-23T18:16:32.953988+0000","last_unstale":"2024-04-23T18:18:30.672398+0000","last_undegraded":"2024-04-23T18:18:30.672398+0000","last_fullsized":"2024-04-23T18:18:30.672398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:09.049861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880698+0000","last_change":"2024-04-23T18:16:33.994860+0000","last_active":"2024-04-23T18:18:30.880698+0000","last_peered":"2024-04-23T18:18:30.880698+0000","last_clean":"2024-04-23T18:18:30.880698+0000","last_became_active":"2024-04-23T18:16:33.993438+0000","last_became_peered":"2024-04-23T18:16:33.993438+0000","last_unstale":"2024-04-23T18:18:30.880698+0000","last_undegraded":"2024-04-23T18:18:30.880698+0000","last_fullsized":"2024-04-23T18:18:30.880698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:55.565873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510367+0000","last_change":"2024-04-23T18:16:32.959251+0000","last_active":"2024-04-23T18:18:30.510367+0000","last_peered":"2024-04-23T18:18:30.510367+0000","last_clean":"2024-04-23T18:18:30.510367+0000","last_became_active":"2024-04-23T18:16:32.958937+0000","last_became_peered":"2024-04-23T18:16:32.958937+0000","last_unstale":"2024-04-23T18:18:30.510367+0000","last_undegraded":"2024-04-23T18:18:30.510367+0000","last_fullsized":"2024-04-23T18:18:30.510367+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.126634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608249+0000","last_change":"2024-04-23T18:16:33.995480+0000","last_active":"2024-04-23T18:18:30.608249+0000","last_peered":"2024-04-23T18:18:30.608249+0000","last_clean":"2024-04-23T18:18:30.608249+0000","last_became_active":"2024-04-23T18:16:33.994361+0000","last_became_peered":"2024-04-23T18:16:33.994361+0000","last_unstale":"2024-04-23T18:18:30.608249+0000","last_undegraded":"2024-04-23T18:18:30.608249+0000","last_fullsized":"2024-04-23T18:18:30.608249+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:59:35.240562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608282+0000","last_change":"2024-04-23T18:16:32.967787+0000","last_active":"2024-04-23T18:18:30.608282+0000","last_peered":"2024-04-23T18:18:30.608282+0000","last_clean":"2024-04-23T18:18:30.608282+0000","last_became_active":"2024-04-23T18:16:32.967463+0000","last_became_peered":"2024-04-23T18:16:32.967463+0000","last_unstale":"2024-04-23T18:18:30.608282+0000","last_undegraded":"2024-04-23T18:18:30.608282+0000","last_fullsized":"2024-04-23T18:18:30.608282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:58.743682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671931+0000","last_change":"2024-04-23T18:16:33.980558+0000","last_active":"2024-04-23T18:18:30.671931+0000","last_peered":"2024-04-23T18:18:30.671931+0000","last_clean":"2024-04-23T18:18:30.671931+0000","last_became_active":"2024-04-23T18:16:33.980146+0000","last_became_peered":"2024-04-23T18:16:33.980146+0000","last_unstale":"2024-04-23T18:18:30.671931+0000","last_undegraded":"2024-04-23T18:18:30.671931+0000","last_fullsized":"2024-04-23T18:18:30.671931+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:58.001833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512195+0000","last_change":"2024-04-23T18:16:32.962139+0000","last_active":"2024-04-23T18:18:30.512195+0000","last_peered":"2024-04-23T18:18:30.512195+0000","last_clean":"2024-04-23T18:18:30.512195+0000","last_became_active":"2024-04-23T18:16:32.961785+0000","last_became_peered":"2024-04-23T18:16:32.961785+0000","last_unstale":"2024-04-23T18:18:30.512195+0000","last_undegraded":"2024-04-23T18:18:30.512195+0000","last_fullsized":"2024-04-23T18:18:30.512195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:09.619414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508926+0000","last_change":"2024-04-23T18:17:36.214255+0000","last_active":"2024-04-23T18:18:30.508926+0000","last_peered":"2024-04-23T18:18:30.508926+0000","last_clean":"2024-04-23T18:18:30.508926+0000","last_became_active":"2024-04-23T18:17:36.213832+0000","last_became_peered":"2024-04-23T18:17:36.213832+0000","last_unstale":"2024-04-23T18:18:30.508926+0000","last_undegraded":"2024-04-23T18:18:30.508926+0000","last_fullsized":"2024-04-23T18:18:30.508926+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:59:35.497710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608350+0000","last_change":"2024-04-23T18:16:32.956965+0000","last_active":"2024-04-23T18:18:30.608350+0000","last_peered":"2024-04-23T18:18:30.608350+0000","last_clean":"2024-04-23T18:18:30.608350+0000","last_became_active":"2024-04-23T18:16:32.956672+0000","last_became_peered":"2024-04-23T18:16:32.956672+0000","last_unstale":"2024-04-23T18:18:30.608350+0000","last_undegraded":"2024-04-23T18:18:30.608350+0000","last_fullsized":"2024-04-23T18:18:30.608350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:00:44.470809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672198+0000","last_change":"2024-04-23T18:16:34.002711+0000","last_active":"2024-04-23T18:18:30.672198+0000","last_peered":"2024-04-23T18:18:30.672198+0000","last_clean":"2024-04-23T18:18:30.672198+0000","last_became_active":"2024-04-23T18:16:34.002365+0000","last_became_peered":"2024-04-23T18:16:34.002365+0000","last_unstale":"2024-04-23T18:18:30.672198+0000","last_undegraded":"2024-04-23T18:18:30.672198+0000","last_fullsized":"2024-04-23T18:18:30.672198+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:52:11.982735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512283+0000","last_change":"2024-04-23T18:16:32.965121+0000","last_active":"2024-04-23T18:18:30.512283+0000","last_peered":"2024-04-23T18:18:30.512283+0000","last_clean":"2024-04-23T18:18:30.512283+0000","last_became_active":"2024-04-23T18:16:32.964732+0000","last_became_peered":"2024-04-23T18:16:32.964732+0000","last_unstale":"2024-04-23T18:18:30.512283+0000","last_undegraded":"2024-04-23T18:18:30.512283+0000","last_fullsized":"2024-04-23T18:18:30.512283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:19:34.690245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672288+0000","last_change":"2024-04-23T18:16:33.996357+0000","last_active":"2024-04-23T18:18:30.672288+0000","last_peered":"2024-04-23T18:18:30.672288+0000","last_clean":"2024-04-23T18:18:30.672288+0000","last_became_active":"2024-04-23T18:16:33.996017+0000","last_became_peered":"2024-04-23T18:16:33.996017+0000","last_unstale":"2024-04-23T18:18:30.672288+0000","last_undegraded":"2024-04-23T18:18:30.672288+0000","last_fullsized":"2024-04-23T18:18:30.672288+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:20:57.321228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.720073+0000","last_change":"2024-04-23T18:16:32.961705+0000","last_active":"2024-04-23T18:18:30.720073+0000","last_peered":"2024-04-23T18:18:30.720073+0000","last_clean":"2024-04-23T18:18:30.720073+0000","last_became_active":"2024-04-23T18:16:32.961342+0000","last_became_peered":"2024-04-23T18:16:32.961342+0000","last_unstale":"2024-04-23T18:18:30.720073+0000","last_undegraded":"2024-04-23T18:18:30.720073+0000","last_fullsized":"2024-04-23T18:18:30.720073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:51.340910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672001+0000","last_change":"2024-04-23T18:16:33.992944+0000","last_active":"2024-04-23T18:18:30.672001+0000","last_peered":"2024-04-23T18:18:30.672001+0000","last_clean":"2024-04-23T18:18:30.672001+0000","last_became_active":"2024-04-23T18:16:33.992579+0000","last_became_peered":"2024-04-23T18:16:33.992579+0000","last_unstale":"2024-04-23T18:18:30.672001+0000","last_undegraded":"2024-04-23T18:18:30.672001+0000","last_fullsized":"2024-04-23T18:18:30.672001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:13:37.829588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672489+0000","last_change":"2024-04-23T18:16:32.955192+0000","last_active":"2024-04-23T18:18:30.672489+0000","last_peered":"2024-04-23T18:18:30.672489+0000","last_clean":"2024-04-23T18:18:30.672489+0000","last_became_active":"2024-04-23T18:16:32.954860+0000","last_became_peered":"2024-04-23T18:16:32.954860+0000","last_unstale":"2024-04-23T18:18:30.672489+0000","last_undegraded":"2024-04-23T18:18:30.672489+0000","last_fullsized":"2024-04-23T18:18:30.672489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:20.796060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672367+0000","last_change":"2024-04-23T18:16:33.986211+0000","last_active":"2024-04-23T18:18:30.672367+0000","last_peered":"2024-04-23T18:18:30.672367+0000","last_clean":"2024-04-23T18:18:30.672367+0000","last_became_active":"2024-04-23T18:16:33.985679+0000","last_became_peered":"2024-04-23T18:16:33.985679+0000","last_unstale":"2024-04-23T18:18:30.672367+0000","last_undegraded":"2024-04-23T18:18:30.672367+0000","last_fullsized":"2024-04-23T18:18:30.672367+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:03:38.797864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672089+0000","last_change":"2024-04-23T18:16:33.979462+0000","last_active":"2024-04-23T18:18:30.672089+0000","last_peered":"2024-04-23T18:18:30.672089+0000","last_clean":"2024-04-23T18:18:30.672089+0000","last_became_active":"2024-04-23T18:16:33.979114+0000","last_became_peered":"2024-04-23T18:16:33.979114+0000","last_unstale":"2024-04-23T18:18:30.672089+0000","last_undegraded":"2024-04-23T18:18:30.672089+0000","last_fullsized":"2024-04-23T18:18:30.672089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:16:49.708240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512027+0000","last_change":"2024-04-23T18:16:33.979255+0000","last_active":"2024-04-23T18:18:30.512027+0000","last_peered":"2024-04-23T18:18:30.512027+0000","last_clean":"2024-04-23T18:18:30.512027+0000","last_became_active":"2024-04-23T18:16:33.978934+0000","last_became_peered":"2024-04-23T18:16:33.978934+0000","last_unstale":"2024-04-23T18:18:30.512027+0000","last_undegraded":"2024-04-23T18:18:30.512027+0000","last_fullsized":"2024-04-23T18:18:30.512027+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:10.104564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.511957+0000","last_change":"2024-04-23T18:16:33.981877+0000","last_active":"2024-04-23T18:18:30.511957+0000","last_peered":"2024-04-23T18:18:30.511957+0000","last_clean":"2024-04-23T18:18:30.511957+0000","last_became_active":"2024-04-23T18:16:33.981536+0000","last_became_peered":"2024-04-23T18:16:33.981536+0000","last_unstale":"2024-04-23T18:18:30.511957+0000","last_undegraded":"2024-04-23T18:18:30.511957+0000","last_fullsized":"2024-04-23T18:18:30.511957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:00:15.056779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880509+0000","last_change":"2024-04-23T18:16:33.999090+0000","last_active":"2024-04-23T18:18:30.880509+0000","last_peered":"2024-04-23T18:18:30.880509+0000","last_clean":"2024-04-23T18:18:30.880509+0000","last_became_active":"2024-04-23T18:16:33.998513+0000","last_became_peered":"2024-04-23T18:16:33.998513+0000","last_unstale":"2024-04-23T18:18:30.880509+0000","last_undegraded":"2024-04-23T18:18:30.880509+0000","last_fullsized":"2024-04-23T18:18:30.880509+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:14.115406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672082+0000","last_change":"2024-04-23T18:16:33.993375+0000","last_active":"2024-04-23T18:18:30.672082+0000","last_peered":"2024-04-23T18:18:30.672082+0000","last_clean":"2024-04-23T18:18:30.672082+0000","last_became_active":"2024-04-23T18:16:33.991841+0000","last_became_peered":"2024-04-23T18:16:33.991841+0000","last_unstale":"2024-04-23T18:18:30.672082+0000","last_undegraded":"2024-04-23T18:18:30.672082+0000","last_fullsized":"2024-04-23T18:18:30.672082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:12.777495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672189+0000","last_change":"2024-04-23T18:16:33.983621+0000","last_active":"2024-04-23T18:18:30.672189+0000","last_peered":"2024-04-23T18:18:30.672189+0000","last_clean":"2024-04-23T18:18:30.672189+0000","last_became_active":"2024-04-23T18:16:33.983356+0000","last_became_peered":"2024-04-23T18:16:33.983356+0000","last_unstale":"2024-04-23T18:18:30.672189+0000","last_undegraded":"2024-04-23T18:18:30.672189+0000","last_fullsized":"2024-04-23T18:18:30.672189+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:27:19.391281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512250+0000","last_change":"2024-04-23T18:16:33.987952+0000","last_active":"2024-04-23T18:18:30.512250+0000","last_peered":"2024-04-23T18:18:30.512250+0000","last_clean":"2024-04-23T18:18:30.512250+0000","last_became_active":"2024-04-23T18:16:33.987334+0000","last_became_peered":"2024-04-23T18:16:33.987334+0000","last_unstale":"2024-04-23T18:18:30.512250+0000","last_undegraded":"2024-04-23T18:18:30.512250+0000","last_fullsized":"2024-04-23T18:18:30.512250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:41.902779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.609205+0000","last_change":"2024-04-23T18:16:33.983914+0000","last_active":"2024-04-23T18:18:30.609205+0000","last_peered":"2024-04-23T18:18:30.609205+0000","last_clean":"2024-04-23T18:18:30.609205+0000","last_became_active":"2024-04-23T18:16:33.983444+0000","last_became_peered":"2024-04-23T18:16:33.983444+0000","last_unstale":"2024-04-23T18:18:30.609205+0000","last_undegraded":"2024-04-23T18:18:30.609205+0000","last_fullsized":"2024-04-23T18:18:30.609205+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:10:17.630049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.613877+0000","last_change":"2024-04-23T18:16:33.992496+0000","last_active":"2024-04-23T18:18:30.613877+0000","last_peered":"2024-04-23T18:18:30.613877+0000","last_clean":"2024-04-23T18:18:30.613877+0000","last_became_active":"2024-04-23T18:16:33.992155+0000","last_became_peered":"2024-04-23T18:16:33.992155+0000","last_unstale":"2024-04-23T18:18:30.613877+0000","last_undegraded":"2024-04-23T18:18:30.613877+0000","last_fullsized":"2024-04-23T18:18:30.613877+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:48.835992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608101+0000","last_change":"2024-04-23T18:16:33.984409+0000","last_active":"2024-04-23T18:18:30.608101+0000","last_peered":"2024-04-23T18:18:30.608101+0000","last_clean":"2024-04-23T18:18:30.608101+0000","last_became_active":"2024-04-23T18:16:33.984073+0000","last_became_peered":"2024-04-23T18:16:33.984073+0000","last_unstale":"2024-04-23T18:18:30.608101+0000","last_undegraded":"2024-04-23T18:18:30.608101+0000","last_fullsized":"2024-04-23T18:18:30.608101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:45:37.796783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510525+0000","last_change":"2024-04-23T18:16:33.986376+0000","last_active":"2024-04-23T18:18:30.510525+0000","last_peered":"2024-04-23T18:18:30.510525+0000","last_clean":"2024-04-23T18:18:30.510525+0000","last_became_active":"2024-04-23T18:16:33.986012+0000","last_became_peered":"2024-04-23T18:16:33.986012+0000","last_unstale":"2024-04-23T18:18:30.510525+0000","last_undegraded":"2024-04-23T18:18:30.510525+0000","last_fullsized":"2024-04-23T18:18:30.510525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:26.940215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672288+0000","last_change":"2024-04-23T18:16:33.999795+0000","last_active":"2024-04-23T18:18:30.672288+0000","last_peered":"2024-04-23T18:18:30.672288+0000","last_clean":"2024-04-23T18:18:30.672288+0000","last_became_active":"2024-04-23T18:16:33.997034+0000","last_became_peered":"2024-04-23T18:16:33.997034+0000","last_unstale":"2024-04-23T18:18:30.672288+0000","last_undegraded":"2024-04-23T18:18:30.672288+0000","last_fullsized":"2024-04-23T18:18:30.672288+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:23.983846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672329+0000","last_change":"2024-04-23T18:16:33.985495+0000","last_active":"2024-04-23T18:18:30.672329+0000","last_peered":"2024-04-23T18:18:30.672329+0000","last_clean":"2024-04-23T18:18:30.672329+0000","last_became_active":"2024-04-23T18:16:33.985178+0000","last_became_peered":"2024-04-23T18:16:33.985178+0000","last_unstale":"2024-04-23T18:18:30.672329+0000","last_undegraded":"2024-04-23T18:18:30.672329+0000","last_fullsized":"2024-04-23T18:18:30.672329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:22:25.325756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":597706,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":804,"num_read_kb":515,"num_write":292,"num_write_kb":1442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1147,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1793118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":231,"ondisk_log_size":231,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318307,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":477387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579945,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32964,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712188,"statfs":{"total":95995035648,"available":95961280512,"internally_reserved":0,"allocated":1830912,"data_stored":1268691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776174,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36708,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708444,"statfs":{"total":95995035648,"available":95957446656,"internally_reserved":0,"allocated":1400832,"data_stored":870705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037810,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1388544,"data_stored":870491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712408,"statfs":{"total":95995035648,"available":95961505792,"internally_reserved":0,"allocated":1601536,"data_stored":1071493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":476878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":544951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40472,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704680,"statfs":{"total":95995035648,"available":95953592320,"internally_reserved":0,"allocated":999424,"data_stored":469193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1081344,"data_stored":544134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41000,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704152,"statfs":{"total":95995035648,"available":95953051648,"internally_reserved":0,"allocated":1540096,"data_stored":1002005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":999424,"data_stored":478937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28724,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716428,"statfs":{"total":95995035648,"available":95965622272,"internally_reserved":0,"allocated":1748992,"data_stored":1201155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":67463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":201535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:18:35.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:18:35.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.937+0000 7f09fdcd0640 1 -- 172.21.15.50:0/570875038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f810c570 msgr2=0x7f09f810e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.937+0000 7f09fdcd0640 1 --2- 172.21.15.50:0/570875038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f810c570 0x7f09f810e9e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009ca0 tx=0x7f09f002a950 comp rx=0 tx=0).stop 2024-04-23T18:18:35.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.937+0000 7f09fdcd0640 1 -- 172.21.15.50:0/570875038 shutdown_connections 2024-04-23T18:18:35.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.937+0000 7f09fdcd0640 1 --2- 172.21.15.50:0/570875038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f810c570 0x7f09f810e9e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.937+0000 7f09fdcd0640 1 --2- 172.21.15.50:0/570875038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09f8103b50 0x7f09f810c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.937+0000 7f09fdcd0640 1 --2- 172.21.15.50:0/570875038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09f8103230 0x7f09f8103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.937+0000 7f09fdcd0640 1 -- 172.21.15.50:0/570875038 >> 172.21.15.50:0/570875038 conn(0x7f09f8077fb0 msgr2=0x7f09f80783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:35.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.937+0000 7f09fdcd0640 1 -- 172.21.15.50:0/570875038 shutdown_connections 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.937+0000 7f09fdcd0640 1 -- 172.21.15.50:0/570875038 wait complete. 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.938+0000 7f09fdcd0640 1 Processor -- start 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.938+0000 7f09fdcd0640 1 -- start start 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.938+0000 7f09fdcd0640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09f8103230 0x7f09f8198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.938+0000 7f09fdcd0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09f8103b50 0x7f09f8198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.938+0000 7f09fdcd0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f810c570 0x7f09f819cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.938+0000 7f09fdcd0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f09f81121b0 con 0x7f09f810c570 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.938+0000 7f09fdcd0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f09f8112030 con 0x7f09f8103b50 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.938+0000 7f09fdcd0640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f09f8112330 con 0x7f09f8103230 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.938+0000 7f09f7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f810c570 0x7f09f819cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.938+0000 7f09f7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f810c570 0x7f09f819cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49250/0 (socket says 172.21.15.50:49250) 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.938+0000 7f09f7fff640 1 -- 172.21.15.50:0/3146146479 learned_addr learned my addr 172.21.15.50:0/3146146479 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.938+0000 7f09f77fe640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09f8103230 0x7f09f8198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09f6ffd640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09f8103b50 0x7f09f8198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09f7fff640 1 -- 172.21.15.50:0/3146146479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09f8103230 msgr2=0x7f09f8198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09f7fff640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09f8103230 0x7f09f8198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09f7fff640 1 -- 172.21.15.50:0/3146146479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09f8103b50 msgr2=0x7f09f8198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09f7fff640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09f8103b50 0x7f09f8198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09f7fff640 1 -- 172.21.15.50:0/3146146479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f819d630 con 0x7f09f810c570 2024-04-23T18:18:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09f6ffd640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09f8103b50 0x7f09f8198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09f7fff640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f810c570 0x7f09f819cf00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f09f0004af0 tx=0x7f09f0031f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09f4ff9640 1 -- 172.21.15.50:0/3146146479 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f0034040 con 0x7f09f810c570 2024-04-23T18:18:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09f4ff9640 1 -- 172.21.15.50:0/3146146479 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09f00308f0 con 0x7f09f810c570 2024-04-23T18:18:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09f4ff9640 1 -- 172.21.15.50:0/3146146479 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f0004570 con 0x7f09f810c570 2024-04-23T18:18:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09fdcd0640 1 -- 172.21.15.50:0/3146146479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f819d8c0 con 0x7f09f810c570 2024-04-23T18:18:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.939+0000 7f09fdcd0640 1 -- 172.21.15.50:0/3146146479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f81a51b0 con 0x7f09f810c570 2024-04-23T18:18:35.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.940+0000 7f09fdcd0640 1 -- 172.21.15.50:0/3146146479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f80691c0 con 0x7f09f810c570 2024-04-23T18:18:35.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.941+0000 7f09f4ff9640 1 -- 172.21.15.50:0/3146146479 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f09f0037020 con 0x7f09f810c570 2024-04-23T18:18:35.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.942+0000 7f09f4ff9640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f09e00a3db0 0x7f09e00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:35.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.942+0000 7f09f77fe640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f09e00a3db0 0x7f09e00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:35.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.942+0000 7f09f4ff9640 1 -- 172.21.15.50:0/3146146479 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f09f00f16a0 con 0x7f09f810c570 2024-04-23T18:18:35.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.943+0000 7f09f77fe640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f09e00a3db0 0x7f09e00a6270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009bd0 tx=0x7f09ec005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:35.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:35.947+0000 7f09f4ff9640 1 -- 172.21.15.50:0/3146146479 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f09f00bdc90 con 0x7f09f810c570 2024-04-23T18:18:36.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.131+0000 7f09fdcd0640 1 -- 172.21.15.50:0/3146146479 --> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f09f8062390 con 0x7f09e00a3db0 2024-04-23T18:18:36.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.138+0000 7f09f4ff9640 1 -- 172.21.15.50:0/3146146479 <== mgr.25018 v2:172.21.15.50:6800/4210816660 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257741 (secure 0 0 0) 0x7f09f8062390 con 0x7f09e00a3db0 2024-04-23T18:18:36.133 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:36.135 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-23T18:18:36.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.141+0000 7f09fdcd0640 1 -- 172.21.15.50:0/3146146479 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f09e00a3db0 msgr2=0x7f09e00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.141+0000 7f09fdcd0640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f09e00a3db0 0x7f09e00a6270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009bd0 tx=0x7f09ec005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:36.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.141+0000 7f09fdcd0640 1 -- 172.21.15.50:0/3146146479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f810c570 msgr2=0x7f09f819cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.141+0000 7f09fdcd0640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f810c570 0x7f09f819cf00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f09f0004af0 tx=0x7f09f0031f50 comp rx=0 tx=0).stop 2024-04-23T18:18:36.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.141+0000 7f09fdcd0640 1 -- 172.21.15.50:0/3146146479 shutdown_connections 2024-04-23T18:18:36.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.141+0000 7f09fdcd0640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f09e00a3db0 0x7f09e00a6270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.141+0000 7f09fdcd0640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f810c570 0x7f09f819cf00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.141+0000 7f09fdcd0640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09f8103b50 0x7f09f8198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.141+0000 7f09fdcd0640 1 --2- 172.21.15.50:0/3146146479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09f8103230 0x7f09f8198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.141+0000 7f09fdcd0640 1 -- 172.21.15.50:0/3146146479 >> 172.21.15.50:0/3146146479 conn(0x7f09f8077fb0 msgr2=0x7f09f80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:36.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.141+0000 7f09fdcd0640 1 -- 172.21.15.50:0/3146146479 shutdown_connections 2024-04-23T18:18:36.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.141+0000 7f09fdcd0640 1 -- 172.21.15.50:0/3146146479 wait complete. 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: Updating smithi050:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: Updating smithi050:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.146 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.146 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.146 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.146 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.146 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3825086338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:36.146 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:18:36.146 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:18:36.146 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:18:36.146 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.146 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:36 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2102803724' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:36.147 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-23T18:18:34.524804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188308,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":940,"num_read_kb":634,"num_write":475,"num_write_kb":4382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181883,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":397756,"kb_used_data":15296,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124544068,"statfs":{"total":1151940427776,"available":1151533125632,"internally_reserved":0,"allocated":15663104,"data_stored":9276020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991529"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672105+0000","last_change":"2024-04-23T18:16:33.987102+0000","last_active":"2024-04-23T18:18:30.672105+0000","last_peered":"2024-04-23T18:18:30.672105+0000","last_clean":"2024-04-23T18:18:30.672105+0000","last_became_active":"2024-04-23T18:16:33.986716+0000","last_became_peered":"2024-04-23T18:16:33.986716+0000","last_unstale":"2024-04-23T18:18:30.672105+0000","last_undegraded":"2024-04-23T18:18:30.672105+0000","last_fullsized":"2024-04-23T18:18:30.672105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:41.895219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880751+0000","last_change":"2024-04-23T18:16:33.995703+0000","last_active":"2024-04-23T18:18:30.880751+0000","last_peered":"2024-04-23T18:18:30.880751+0000","last_clean":"2024-04-23T18:18:30.880751+0000","last_became_active":"2024-04-23T18:16:33.995242+0000","last_became_peered":"2024-04-23T18:16:33.995242+0000","last_unstale":"2024-04-23T18:18:30.880751+0000","last_undegraded":"2024-04-23T18:18:30.880751+0000","last_fullsized":"2024-04-23T18:18:30.880751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:57.813822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880606+0000","last_change":"2024-04-23T18:16:34.000864+0000","last_active":"2024-04-23T18:18:30.880606+0000","last_peered":"2024-04-23T18:18:30.880606+0000","last_clean":"2024-04-23T18:18:30.880606+0000","last_became_active":"2024-04-23T18:16:33.998934+0000","last_became_peered":"2024-04-23T18:16:33.998934+0000","last_unstale":"2024-04-23T18:18:30.880606+0000","last_undegraded":"2024-04-23T18:18:30.880606+0000","last_fullsized":"2024-04-23T18:18:30.880606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:47:29.481697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672137+0000","last_change":"2024-04-23T18:16:33.989351+0000","last_active":"2024-04-23T18:18:30.672137+0000","last_peered":"2024-04-23T18:18:30.672137+0000","last_clean":"2024-04-23T18:18:30.672137+0000","last_became_active":"2024-04-23T18:16:33.989030+0000","last_became_peered":"2024-04-23T18:16:33.989030+0000","last_unstale":"2024-04-23T18:18:30.672137+0000","last_undegraded":"2024-04-23T18:18:30.672137+0000","last_fullsized":"2024-04-23T18:18:30.672137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:29:03.023675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608347+0000","last_change":"2024-04-23T18:16:33.996062+0000","last_active":"2024-04-23T18:18:30.608347+0000","last_peered":"2024-04-23T18:18:30.608347+0000","last_clean":"2024-04-23T18:18:30.608347+0000","last_became_active":"2024-04-23T18:16:33.995732+0000","last_became_peered":"2024-04-23T18:16:33.995732+0000","last_unstale":"2024-04-23T18:18:30.608347+0000","last_undegraded":"2024-04-23T18:18:30.608347+0000","last_fullsized":"2024-04-23T18:18:30.608347+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:04.175580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.617038+0000","last_change":"2024-04-23T18:16:33.989895+0000","last_active":"2024-04-23T18:18:30.617038+0000","last_peered":"2024-04-23T18:18:30.617038+0000","last_clean":"2024-04-23T18:18:30.617038+0000","last_became_active":"2024-04-23T18:16:33.989547+0000","last_became_peered":"2024-04-23T18:16:33.989547+0000","last_unstale":"2024-04-23T18:18:30.617038+0000","last_undegraded":"2024-04-23T18:18:30.617038+0000","last_fullsized":"2024-04-23T18:18:30.617038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:12:37.317974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672251+0000","last_change":"2024-04-23T18:16:33.999636+0000","last_active":"2024-04-23T18:18:30.672251+0000","last_peered":"2024-04-23T18:18:30.672251+0000","last_clean":"2024-04-23T18:18:30.672251+0000","last_became_active":"2024-04-23T18:16:33.999282+0000","last_became_peered":"2024-04-23T18:16:33.999282+0000","last_unstale":"2024-04-23T18:18:30.672251+0000","last_undegraded":"2024-04-23T18:18:30.672251+0000","last_fullsized":"2024-04-23T18:18:30.672251+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:28.841392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608063+0000","last_change":"2024-04-23T18:16:33.996602+0000","last_active":"2024-04-23T18:18:30.608063+0000","last_peered":"2024-04-23T18:18:30.608063+0000","last_clean":"2024-04-23T18:18:30.608063+0000","last_became_active":"2024-04-23T18:16:33.994401+0000","last_became_peered":"2024-04-23T18:16:33.994401+0000","last_unstale":"2024-04-23T18:18:30.608063+0000","last_undegraded":"2024-04-23T18:18:30.608063+0000","last_fullsized":"2024-04-23T18:18:30.608063+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:31:54.553537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510925+0000","last_change":"2024-04-23T18:16:33.994763+0000","last_active":"2024-04-23T18:18:30.510925+0000","last_peered":"2024-04-23T18:18:30.510925+0000","last_clean":"2024-04-23T18:18:30.510925+0000","last_became_active":"2024-04-23T18:16:33.994408+0000","last_became_peered":"2024-04-23T18:16:33.994408+0000","last_unstale":"2024-04-23T18:18:30.510925+0000","last_undegraded":"2024-04-23T18:18:30.510925+0000","last_fullsized":"2024-04-23T18:18:30.510925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:02.706160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.617268+0000","last_change":"2024-04-23T18:16:33.993435+0000","last_active":"2024-04-23T18:18:30.617268+0000","last_peered":"2024-04-23T18:18:30.617268+0000","last_clean":"2024-04-23T18:18:30.617268+0000","last_became_active":"2024-04-23T18:16:33.993108+0000","last_became_peered":"2024-04-23T18:16:33.993108+0000","last_unstale":"2024-04-23T18:18:30.617268+0000","last_undegraded":"2024-04-23T18:18:30.617268+0000","last_fullsized":"2024-04-23T18:18:30.617268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:28:32.584896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672170+0000","last_change":"2024-04-23T18:16:33.997526+0000","last_active":"2024-04-23T18:18:30.672170+0000","last_peered":"2024-04-23T18:18:30.672170+0000","last_clean":"2024-04-23T18:18:30.672170+0000","last_became_active":"2024-04-23T18:16:33.997189+0000","last_became_peered":"2024-04-23T18:16:33.997189+0000","last_unstale":"2024-04-23T18:18:30.672170+0000","last_undegraded":"2024-04-23T18:18:30.672170+0000","last_fullsized":"2024-04-23T18:18:30.672170+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:11:41.208261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671985+0000","last_change":"2024-04-23T18:16:33.989775+0000","last_active":"2024-04-23T18:18:30.671985+0000","last_peered":"2024-04-23T18:18:30.671985+0000","last_clean":"2024-04-23T18:18:30.671985+0000","last_became_active":"2024-04-23T18:16:33.989382+0000","last_became_peered":"2024-04-23T18:16:33.989382+0000","last_unstale":"2024-04-23T18:18:30.671985+0000","last_undegraded":"2024-04-23T18:18:30.671985+0000","last_fullsized":"2024-04-23T18:18:30.671985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:20:17.756258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672114+0000","last_change":"2024-04-23T18:16:34.005871+0000","last_active":"2024-04-23T18:18:30.672114+0000","last_peered":"2024-04-23T18:18:30.672114+0000","last_clean":"2024-04-23T18:18:30.672114+0000","last_became_active":"2024-04-23T18:16:34.005218+0000","last_became_peered":"2024-04-23T18:16:34.005218+0000","last_unstale":"2024-04-23T18:18:30.672114+0000","last_undegraded":"2024-04-23T18:18:30.672114+0000","last_fullsized":"2024-04-23T18:18:30.672114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:30:03.864306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880355+0000","last_change":"2024-04-23T18:16:33.995054+0000","last_active":"2024-04-23T18:18:30.880355+0000","last_peered":"2024-04-23T18:18:30.880355+0000","last_clean":"2024-04-23T18:18:30.880355+0000","last_became_active":"2024-04-23T18:16:33.994389+0000","last_became_peered":"2024-04-23T18:16:33.994389+0000","last_unstale":"2024-04-23T18:18:30.880355+0000","last_undegraded":"2024-04-23T18:18:30.880355+0000","last_fullsized":"2024-04-23T18:18:30.880355+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:27.344616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880258+0000","last_change":"2024-04-23T18:16:33.984490+0000","last_active":"2024-04-23T18:18:30.880258+0000","last_peered":"2024-04-23T18:18:30.880258+0000","last_clean":"2024-04-23T18:18:30.880258+0000","last_became_active":"2024-04-23T18:16:33.984071+0000","last_became_peered":"2024-04-23T18:16:33.984071+0000","last_unstale":"2024-04-23T18:18:30.880258+0000","last_undegraded":"2024-04-23T18:18:30.880258+0000","last_fullsized":"2024-04-23T18:18:30.880258+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:23.589290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510517+0000","last_change":"2024-04-23T18:16:34.002163+0000","last_active":"2024-04-23T18:18:30.510517+0000","last_peered":"2024-04-23T18:18:30.510517+0000","last_clean":"2024-04-23T18:18:30.510517+0000","last_became_active":"2024-04-23T18:16:34.001764+0000","last_became_peered":"2024-04-23T18:16:34.001764+0000","last_unstale":"2024-04-23T18:18:30.510517+0000","last_undegraded":"2024-04-23T18:18:30.510517+0000","last_fullsized":"2024-04-23T18:18:30.510517+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:16:03.616358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672114+0000","last_change":"2024-04-23T18:16:33.987586+0000","last_active":"2024-04-23T18:18:30.672114+0000","last_peered":"2024-04-23T18:18:30.672114+0000","last_clean":"2024-04-23T18:18:30.672114+0000","last_became_active":"2024-04-23T18:16:33.987266+0000","last_became_peered":"2024-04-23T18:16:33.987266+0000","last_unstale":"2024-04-23T18:18:30.672114+0000","last_undegraded":"2024-04-23T18:18:30.672114+0000","last_fullsized":"2024-04-23T18:18:30.672114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:41.610529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508920+0000","last_change":"2024-04-23T18:16:33.992938+0000","last_active":"2024-04-23T18:18:30.508920+0000","last_peered":"2024-04-23T18:18:30.508920+0000","last_clean":"2024-04-23T18:18:30.508920+0000","last_became_active":"2024-04-23T18:16:33.990441+0000","last_became_peered":"2024-04-23T18:16:33.990441+0000","last_unstale":"2024-04-23T18:18:30.508920+0000","last_undegraded":"2024-04-23T18:18:30.508920+0000","last_fullsized":"2024-04-23T18:18:30.508920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:10.689333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512446+0000","last_change":"2024-04-23T18:16:33.994844+0000","last_active":"2024-04-23T18:18:30.512446+0000","last_peered":"2024-04-23T18:18:30.512446+0000","last_clean":"2024-04-23T18:18:30.512446+0000","last_became_active":"2024-04-23T18:16:33.994505+0000","last_became_peered":"2024-04-23T18:16:33.994505+0000","last_unstale":"2024-04-23T18:18:30.512446+0000","last_undegraded":"2024-04-23T18:18:30.512446+0000","last_fullsized":"2024-04-23T18:18:30.512446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:32:33.445266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508971+0000","last_change":"2024-04-23T18:16:32.958841+0000","last_active":"2024-04-23T18:18:30.508971+0000","last_peered":"2024-04-23T18:18:30.508971+0000","last_clean":"2024-04-23T18:18:30.508971+0000","last_became_active":"2024-04-23T18:16:32.958529+0000","last_became_peered":"2024-04-23T18:16:32.958529+0000","last_unstale":"2024-04-23T18:18:30.508971+0000","last_undegraded":"2024-04-23T18:18:30.508971+0000","last_fullsized":"2024-04-23T18:18:30.508971+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:47:09.933786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608592+0000","last_change":"2024-04-23T18:16:33.992108+0000","last_active":"2024-04-23T18:18:30.608592+0000","last_peered":"2024-04-23T18:18:30.608592+0000","last_clean":"2024-04-23T18:18:30.608592+0000","last_became_active":"2024-04-23T18:16:33.991751+0000","last_became_peered":"2024-04-23T18:16:33.991751+0000","last_unstale":"2024-04-23T18:18:30.608592+0000","last_undegraded":"2024-04-23T18:18:30.608592+0000","last_fullsized":"2024-04-23T18:18:30.608592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:07:57.416263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608867+0000","last_change":"2024-04-23T18:16:32.960461+0000","last_active":"2024-04-23T18:18:30.608867+0000","last_peered":"2024-04-23T18:18:30.608867+0000","last_clean":"2024-04-23T18:18:30.608867+0000","last_became_active":"2024-04-23T18:16:32.960115+0000","last_became_peered":"2024-04-23T18:16:32.960115+0000","last_unstale":"2024-04-23T18:18:30.608867+0000","last_undegraded":"2024-04-23T18:18:30.608867+0000","last_fullsized":"2024-04-23T18:18:30.608867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:30.533043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880852+0000","last_change":"2024-04-23T18:16:34.001831+0000","last_active":"2024-04-23T18:18:30.880852+0000","last_peered":"2024-04-23T18:18:30.880852+0000","last_clean":"2024-04-23T18:18:30.880852+0000","last_became_active":"2024-04-23T18:16:34.001486+0000","last_became_peered":"2024-04-23T18:16:34.001486+0000","last_unstale":"2024-04-23T18:18:30.880852+0000","last_undegraded":"2024-04-23T18:18:30.880852+0000","last_fullsized":"2024-04-23T18:18:30.880852+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:06:40.452366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608622+0000","last_change":"2024-04-23T18:16:32.960387+0000","last_active":"2024-04-23T18:18:30.608622+0000","last_peered":"2024-04-23T18:18:30.608622+0000","last_clean":"2024-04-23T18:18:30.608622+0000","last_became_active":"2024-04-23T18:16:32.960046+0000","last_became_peered":"2024-04-23T18:16:32.960046+0000","last_unstale":"2024-04-23T18:18:30.608622+0000","last_undegraded":"2024-04-23T18:18:30.608622+0000","last_fullsized":"2024-04-23T18:18:30.608622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:03.296653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510543+0000","last_change":"2024-04-23T18:16:33.996520+0000","last_active":"2024-04-23T18:18:30.510543+0000","last_peered":"2024-04-23T18:18:30.510543+0000","last_clean":"2024-04-23T18:18:30.510543+0000","last_became_active":"2024-04-23T18:16:33.995636+0000","last_became_peered":"2024-04-23T18:16:33.995636+0000","last_unstale":"2024-04-23T18:18:30.510543+0000","last_undegraded":"2024-04-23T18:18:30.510543+0000","last_fullsized":"2024-04-23T18:18:30.510543+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:29.665407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608556+0000","last_change":"2024-04-23T18:16:32.956813+0000","last_active":"2024-04-23T18:18:30.608556+0000","last_peered":"2024-04-23T18:18:30.608556+0000","last_clean":"2024-04-23T18:18:30.608556+0000","last_became_active":"2024-04-23T18:16:32.956505+0000","last_became_peered":"2024-04-23T18:16:32.956505+0000","last_unstale":"2024-04-23T18:18:30.608556+0000","last_undegraded":"2024-04-23T18:18:30.608556+0000","last_fullsized":"2024-04-23T18:18:30.608556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:36:27.330353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.614156+0000","last_change":"2024-04-23T18:16:33.999754+0000","last_active":"2024-04-23T18:18:30.614156+0000","last_peered":"2024-04-23T18:18:30.614156+0000","last_clean":"2024-04-23T18:18:30.614156+0000","last_became_active":"2024-04-23T18:16:33.999034+0000","last_became_peered":"2024-04-23T18:16:33.999034+0000","last_unstale":"2024-04-23T18:18:30.614156+0000","last_undegraded":"2024-04-23T18:18:30.614156+0000","last_fullsized":"2024-04-23T18:18:30.614156+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:26:08.595745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":160,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:31.190208+0000","last_change":"2024-04-23T18:17:36.120371+0000","last_active":"2024-04-23T18:18:31.190208+0000","last_peered":"2024-04-23T18:18:31.190208+0000","last_clean":"2024-04-23T18:18:31.190208+0000","last_became_active":"2024-04-23T18:17:36.116149+0000","last_became_peered":"2024-04-23T18:17:36.116149+0000","last_unstale":"2024-04-23T18:18:31.190208+0000","last_undegraded":"2024-04-23T18:18:31.190208+0000","last_fullsized":"2024-04-23T18:18:31.190208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:54.994254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608344+0000","last_change":"2024-04-23T18:16:33.995166+0000","last_active":"2024-04-23T18:18:30.608344+0000","last_peered":"2024-04-23T18:18:30.608344+0000","last_clean":"2024-04-23T18:18:30.608344+0000","last_became_active":"2024-04-23T18:16:33.994347+0000","last_became_peered":"2024-04-23T18:16:33.994347+0000","last_unstale":"2024-04-23T18:18:30.608344+0000","last_undegraded":"2024-04-23T18:18:30.608344+0000","last_fullsized":"2024-04-23T18:18:30.608344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.305837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.616932+0000","last_change":"2024-04-23T18:16:32.959358+0000","last_active":"2024-04-23T18:18:30.616932+0000","last_peered":"2024-04-23T18:18:30.616932+0000","last_clean":"2024-04-23T18:18:30.616932+0000","last_became_active":"2024-04-23T18:16:32.959036+0000","last_became_peered":"2024-04-23T18:16:32.959036+0000","last_unstale":"2024-04-23T18:18:30.616932+0000","last_undegraded":"2024-04-23T18:18:30.616932+0000","last_fullsized":"2024-04-23T18:18:30.616932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:03:15.734754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.614091+0000","last_change":"2024-04-23T18:16:33.988578+0000","last_active":"2024-04-23T18:18:30.614091+0000","last_peered":"2024-04-23T18:18:30.614091+0000","last_clean":"2024-04-23T18:18:30.614091+0000","last_became_active":"2024-04-23T18:16:33.988248+0000","last_became_peered":"2024-04-23T18:16:33.988248+0000","last_unstale":"2024-04-23T18:18:30.614091+0000","last_undegraded":"2024-04-23T18:18:30.614091+0000","last_fullsized":"2024-04-23T18:18:30.614091+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:45:15.538971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671882+0000","last_change":"2024-04-23T18:16:32.952945+0000","last_active":"2024-04-23T18:18:30.671882+0000","last_peered":"2024-04-23T18:18:30.671882+0000","last_clean":"2024-04-23T18:18:30.671882+0000","last_became_active":"2024-04-23T18:16:32.952602+0000","last_became_peered":"2024-04-23T18:16:32.952602+0000","last_unstale":"2024-04-23T18:18:30.671882+0000","last_undegraded":"2024-04-23T18:18:30.671882+0000","last_fullsized":"2024-04-23T18:18:30.671882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:20.037431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608129+0000","last_change":"2024-04-23T18:16:33.996897+0000","last_active":"2024-04-23T18:18:30.608129+0000","last_peered":"2024-04-23T18:18:30.608129+0000","last_clean":"2024-04-23T18:18:30.608129+0000","last_became_active":"2024-04-23T18:16:33.996214+0000","last_became_peered":"2024-04-23T18:16:33.996214+0000","last_unstale":"2024-04-23T18:18:30.608129+0000","last_undegraded":"2024-04-23T18:18:30.608129+0000","last_fullsized":"2024-04-23T18:18:30.608129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:34.931697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.613841+0000","last_change":"2024-04-23T18:16:32.959637+0000","last_active":"2024-04-23T18:18:30.613841+0000","last_peered":"2024-04-23T18:18:30.613841+0000","last_clean":"2024-04-23T18:18:30.613841+0000","last_became_active":"2024-04-23T18:16:32.959332+0000","last_became_peered":"2024-04-23T18:16:32.959332+0000","last_unstale":"2024-04-23T18:18:30.613841+0000","last_undegraded":"2024-04-23T18:18:30.613841+0000","last_fullsized":"2024-04-23T18:18:30.613841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:30:49.089101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508653+0000","last_change":"2024-04-23T18:16:33.986288+0000","last_active":"2024-04-23T18:18:30.508653+0000","last_peered":"2024-04-23T18:18:30.508653+0000","last_clean":"2024-04-23T18:18:30.508653+0000","last_became_active":"2024-04-23T18:16:33.985866+0000","last_became_peered":"2024-04-23T18:16:33.985866+0000","last_unstale":"2024-04-23T18:18:30.508653+0000","last_undegraded":"2024-04-23T18:18:30.508653+0000","last_fullsized":"2024-04-23T18:18:30.508653+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:30:17.675952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":140,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.761189+0000","last_change":"2024-04-23T18:16:32.958466+0000","last_active":"2024-04-23T18:18:30.761189+0000","last_peered":"2024-04-23T18:18:30.761189+0000","last_clean":"2024-04-23T18:18:30.761189+0000","last_became_active":"2024-04-23T18:16:32.958155+0000","last_became_peered":"2024-04-23T18:16:32.958155+0000","last_unstale":"2024-04-23T18:18:30.761189+0000","last_undegraded":"2024-04-23T18:18:30.761189+0000","last_fullsized":"2024-04-23T18:18:30.761189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:15:49.073934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510826+0000","last_change":"2024-04-23T18:16:33.996156+0000","last_active":"2024-04-23T18:18:30.510826+0000","last_peered":"2024-04-23T18:18:30.510826+0000","last_clean":"2024-04-23T18:18:30.510826+0000","last_became_active":"2024-04-23T18:16:33.995711+0000","last_became_peered":"2024-04-23T18:16:33.995711+0000","last_unstale":"2024-04-23T18:18:30.510826+0000","last_undegraded":"2024-04-23T18:18:30.510826+0000","last_fullsized":"2024-04-23T18:18:30.510826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:45.272598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672184+0000","last_change":"2024-04-23T18:17:36.217460+0000","last_active":"2024-04-23T18:18:30.672184+0000","last_peered":"2024-04-23T18:18:30.672184+0000","last_clean":"2024-04-23T18:18:30.672184+0000","last_became_active":"2024-04-23T18:17:36.214196+0000","last_became_peered":"2024-04-23T18:17:36.214196+0000","last_unstale":"2024-04-23T18:18:30.672184+0000","last_undegraded":"2024-04-23T18:18:30.672184+0000","last_fullsized":"2024-04-23T18:18:30.672184+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:34:55.557943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880842+0000","last_change":"2024-04-23T18:17:36.114238+0000","last_active":"2024-04-23T18:18:30.880842+0000","last_peered":"2024-04-23T18:18:30.880842+0000","last_clean":"2024-04-23T18:18:30.880842+0000","last_became_active":"2024-04-23T18:17:36.113810+0000","last_became_peered":"2024-04-23T18:17:36.113810+0000","last_unstale":"2024-04-23T18:18:30.880842+0000","last_undegraded":"2024-04-23T18:18:30.880842+0000","last_fullsized":"2024-04-23T18:18:30.880842+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:53.854596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.614213+0000","last_change":"2024-04-23T18:16:32.966725+0000","last_active":"2024-04-23T18:18:30.614213+0000","last_peered":"2024-04-23T18:18:30.614213+0000","last_clean":"2024-04-23T18:18:30.614213+0000","last_became_active":"2024-04-23T18:16:32.966304+0000","last_became_peered":"2024-04-23T18:16:32.966304+0000","last_unstale":"2024-04-23T18:18:30.614213+0000","last_undegraded":"2024-04-23T18:18:30.614213+0000","last_fullsized":"2024-04-23T18:18:30.614213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:45.191051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510708+0000","last_change":"2024-04-23T18:16:33.994667+0000","last_active":"2024-04-23T18:18:30.510708+0000","last_peered":"2024-04-23T18:18:30.510708+0000","last_clean":"2024-04-23T18:18:30.510708+0000","last_became_active":"2024-04-23T18:16:33.994305+0000","last_became_peered":"2024-04-23T18:16:33.994305+0000","last_unstale":"2024-04-23T18:18:30.510708+0000","last_undegraded":"2024-04-23T18:18:30.510708+0000","last_fullsized":"2024-04-23T18:18:30.510708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:23.807075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'37","reported_seq":298,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:31.190262+0000","last_change":"2024-04-23T18:16:32.957946+0000","last_active":"2024-04-23T18:18:31.190262+0000","last_peered":"2024-04-23T18:18:31.190262+0000","last_clean":"2024-04-23T18:18:31.190262+0000","last_became_active":"2024-04-23T18:16:32.957478+0000","last_became_peered":"2024-04-23T18:16:32.957478+0000","last_unstale":"2024-04-23T18:18:31.190262+0000","last_undegraded":"2024-04-23T18:18:31.190262+0000","last_fullsized":"2024-04-23T18:18:31.190262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:16.850604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66735,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":240,"num_read_kb":185,"num_write":38,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608533+0000","last_change":"2024-04-23T18:16:33.989537+0000","last_active":"2024-04-23T18:18:30.608533+0000","last_peered":"2024-04-23T18:18:30.608533+0000","last_clean":"2024-04-23T18:18:30.608533+0000","last_became_active":"2024-04-23T18:16:33.989180+0000","last_became_peered":"2024-04-23T18:16:33.989180+0000","last_unstale":"2024-04-23T18:18:30.608533+0000","last_undegraded":"2024-04-23T18:18:30.608533+0000","last_fullsized":"2024-04-23T18:18:30.608533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:53:18.700347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.511992+0000","last_change":"2024-04-23T18:16:32.967314+0000","last_active":"2024-04-23T18:18:30.511992+0000","last_peered":"2024-04-23T18:18:30.511992+0000","last_clean":"2024-04-23T18:18:30.511992+0000","last_became_active":"2024-04-23T18:16:32.966861+0000","last_became_peered":"2024-04-23T18:16:32.966861+0000","last_unstale":"2024-04-23T18:18:30.511992+0000","last_undegraded":"2024-04-23T18:18:30.511992+0000","last_fullsized":"2024-04-23T18:18:30.511992+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:00:25.606402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671655+0000","last_change":"2024-04-23T18:16:33.992593+0000","last_active":"2024-04-23T18:18:30.671655+0000","last_peered":"2024-04-23T18:18:30.671655+0000","last_clean":"2024-04-23T18:18:30.671655+0000","last_became_active":"2024-04-23T18:16:33.992228+0000","last_became_peered":"2024-04-23T18:16:33.992228+0000","last_unstale":"2024-04-23T18:18:30.671655+0000","last_undegraded":"2024-04-23T18:18:30.671655+0000","last_fullsized":"2024-04-23T18:18:30.671655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:12:23.836829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.613908+0000","last_change":"2024-04-23T18:16:32.961740+0000","last_active":"2024-04-23T18:18:30.613908+0000","last_peered":"2024-04-23T18:18:30.613908+0000","last_clean":"2024-04-23T18:18:30.613908+0000","last_became_active":"2024-04-23T18:16:32.961331+0000","last_became_peered":"2024-04-23T18:16:32.961331+0000","last_unstale":"2024-04-23T18:18:30.613908+0000","last_undegraded":"2024-04-23T18:18:30.613908+0000","last_fullsized":"2024-04-23T18:18:30.613908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:59:17.665288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671999+0000","last_change":"2024-04-23T18:17:36.111151+0000","last_active":"2024-04-23T18:18:30.671999+0000","last_peered":"2024-04-23T18:18:30.671999+0000","last_clean":"2024-04-23T18:18:30.671999+0000","last_became_active":"2024-04-23T18:17:36.110770+0000","last_became_peered":"2024-04-23T18:17:36.110770+0000","last_unstale":"2024-04-23T18:18:30.671999+0000","last_undegraded":"2024-04-23T18:18:30.671999+0000","last_fullsized":"2024-04-23T18:18:30.671999+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:01:03.456738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:31.133054+0000","last_change":"2024-04-23T18:16:32.954935+0000","last_active":"2024-04-23T18:18:31.133054+0000","last_peered":"2024-04-23T18:18:31.133054+0000","last_clean":"2024-04-23T18:18:31.133054+0000","last_became_active":"2024-04-23T18:16:32.954595+0000","last_became_peered":"2024-04-23T18:16:32.954595+0000","last_unstale":"2024-04-23T18:18:31.133054+0000","last_undegraded":"2024-04-23T18:18:31.133054+0000","last_fullsized":"2024-04-23T18:18:31.133054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:22.954101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671780+0000","last_change":"2024-04-23T18:16:34.000215+0000","last_active":"2024-04-23T18:18:30.671780+0000","last_peered":"2024-04-23T18:18:30.671780+0000","last_clean":"2024-04-23T18:18:30.671780+0000","last_became_active":"2024-04-23T18:16:33.999855+0000","last_became_peered":"2024-04-23T18:16:33.999855+0000","last_unstale":"2024-04-23T18:18:30.671780+0000","last_undegraded":"2024-04-23T18:18:30.671780+0000","last_fullsized":"2024-04-23T18:18:30.671780+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:14.686916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671868+0000","last_change":"2024-04-23T18:17:36.127721+0000","last_active":"2024-04-23T18:18:30.671868+0000","last_peered":"2024-04-23T18:18:30.671868+0000","last_clean":"2024-04-23T18:18:30.671868+0000","last_became_active":"2024-04-23T18:17:36.113735+0000","last_became_peered":"2024-04-23T18:17:36.113735+0000","last_unstale":"2024-04-23T18:18:30.671868+0000","last_undegraded":"2024-04-23T18:18:30.671868+0000","last_fullsized":"2024-04-23T18:18:30.671868+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:56:41.395925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508868+0000","last_change":"2024-04-23T18:17:36.214244+0000","last_active":"2024-04-23T18:18:30.508868+0000","last_peered":"2024-04-23T18:18:30.508868+0000","last_clean":"2024-04-23T18:18:30.508868+0000","last_became_active":"2024-04-23T18:17:36.213897+0000","last_became_peered":"2024-04-23T18:17:36.213897+0000","last_unstale":"2024-04-23T18:18:30.508868+0000","last_undegraded":"2024-04-23T18:18:30.508868+0000","last_fullsized":"2024-04-23T18:18:30.508868+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:44:24.935747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608788+0000","last_change":"2024-04-23T18:16:32.959253+0000","last_active":"2024-04-23T18:18:30.608788+0000","last_peered":"2024-04-23T18:18:30.608788+0000","last_clean":"2024-04-23T18:18:30.608788+0000","last_became_active":"2024-04-23T18:16:32.958929+0000","last_became_peered":"2024-04-23T18:16:32.958929+0000","last_unstale":"2024-04-23T18:18:30.608788+0000","last_undegraded":"2024-04-23T18:18:30.608788+0000","last_fullsized":"2024-04-23T18:18:30.608788+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:36.317516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672022+0000","last_change":"2024-04-23T18:16:33.979464+0000","last_active":"2024-04-23T18:18:30.672022+0000","last_peered":"2024-04-23T18:18:30.672022+0000","last_clean":"2024-04-23T18:18:30.672022+0000","last_became_active":"2024-04-23T18:16:33.979106+0000","last_became_peered":"2024-04-23T18:16:33.979106+0000","last_unstale":"2024-04-23T18:18:30.672022+0000","last_undegraded":"2024-04-23T18:18:30.672022+0000","last_fullsized":"2024-04-23T18:18:30.672022+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:00.125514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508806+0000","last_change":"2024-04-23T18:16:32.960812+0000","last_active":"2024-04-23T18:18:30.508806+0000","last_peered":"2024-04-23T18:18:30.508806+0000","last_clean":"2024-04-23T18:18:30.508806+0000","last_became_active":"2024-04-23T18:16:32.959807+0000","last_became_peered":"2024-04-23T18:16:32.959807+0000","last_unstale":"2024-04-23T18:18:30.508806+0000","last_undegraded":"2024-04-23T18:18:30.508806+0000","last_fullsized":"2024-04-23T18:18:30.508806+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:45.097443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.617178+0000","last_change":"2024-04-23T18:16:33.992891+0000","last_active":"2024-04-23T18:18:30.617178+0000","last_peered":"2024-04-23T18:18:30.617178+0000","last_clean":"2024-04-23T18:18:30.617178+0000","last_became_active":"2024-04-23T18:16:33.992559+0000","last_became_peered":"2024-04-23T18:16:33.992559+0000","last_unstale":"2024-04-23T18:18:30.617178+0000","last_undegraded":"2024-04-23T18:18:30.617178+0000","last_fullsized":"2024-04-23T18:18:30.617178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:48.615158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671986+0000","last_change":"2024-04-23T18:16:32.957015+0000","last_active":"2024-04-23T18:18:30.671986+0000","last_peered":"2024-04-23T18:18:30.671986+0000","last_clean":"2024-04-23T18:18:30.671986+0000","last_became_active":"2024-04-23T18:16:32.956690+0000","last_became_peered":"2024-04-23T18:16:32.956690+0000","last_unstale":"2024-04-23T18:18:30.671986+0000","last_undegraded":"2024-04-23T18:18:30.671986+0000","last_fullsized":"2024-04-23T18:18:30.671986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:20:42.087521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.617121+0000","last_change":"2024-04-23T18:16:33.998906+0000","last_active":"2024-04-23T18:18:30.617121+0000","last_peered":"2024-04-23T18:18:30.617121+0000","last_clean":"2024-04-23T18:18:30.617121+0000","last_became_active":"2024-04-23T18:16:33.997297+0000","last_became_peered":"2024-04-23T18:16:33.997297+0000","last_unstale":"2024-04-23T18:18:30.617121+0000","last_undegraded":"2024-04-23T18:18:30.617121+0000","last_fullsized":"2024-04-23T18:18:30.617121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:06.668830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.614008+0000","last_change":"2024-04-23T18:17:36.113962+0000","last_active":"2024-04-23T18:18:30.614008+0000","last_peered":"2024-04-23T18:18:30.614008+0000","last_clean":"2024-04-23T18:18:30.614008+0000","last_became_active":"2024-04-23T18:17:36.113573+0000","last_became_peered":"2024-04-23T18:17:36.113573+0000","last_unstale":"2024-04-23T18:18:30.614008+0000","last_undegraded":"2024-04-23T18:18:30.614008+0000","last_fullsized":"2024-04-23T18:18:30.614008+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:47:25.408144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'6","reported_seq":141,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671805+0000","last_change":"2024-04-23T18:16:32.961423+0000","last_active":"2024-04-23T18:18:30.671805+0000","last_peered":"2024-04-23T18:18:30.671805+0000","last_clean":"2024-04-23T18:18:30.671805+0000","last_became_active":"2024-04-23T18:16:32.961116+0000","last_became_peered":"2024-04-23T18:16:32.961116+0000","last_unstale":"2024-04-23T18:18:30.671805+0000","last_undegraded":"2024-04-23T18:18:30.671805+0000","last_fullsized":"2024-04-23T18:18:30.671805+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:28.525642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608360+0000","last_change":"2024-04-23T18:16:33.990747+0000","last_active":"2024-04-23T18:18:30.608360+0000","last_peered":"2024-04-23T18:18:30.608360+0000","last_clean":"2024-04-23T18:18:30.608360+0000","last_became_active":"2024-04-23T18:16:33.989972+0000","last_became_peered":"2024-04-23T18:16:33.989972+0000","last_unstale":"2024-04-23T18:18:30.608360+0000","last_undegraded":"2024-04-23T18:18:30.608360+0000","last_fullsized":"2024-04-23T18:18:30.608360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:04.667446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671772+0000","last_change":"2024-04-23T18:16:32.952821+0000","last_active":"2024-04-23T18:18:30.671772+0000","last_peered":"2024-04-23T18:18:30.671772+0000","last_clean":"2024-04-23T18:18:30.671772+0000","last_became_active":"2024-04-23T18:16:32.952470+0000","last_became_peered":"2024-04-23T18:16:32.952470+0000","last_unstale":"2024-04-23T18:18:30.671772+0000","last_undegraded":"2024-04-23T18:18:30.671772+0000","last_fullsized":"2024-04-23T18:18:30.671772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:19.392670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671938+0000","last_change":"2024-04-23T18:15:25.867370+0000","last_active":"2024-04-23T18:18:30.671938+0000","last_peered":"2024-04-23T18:18:30.671938+0000","last_clean":"2024-04-23T18:18:30.671938+0000","last_became_active":"2024-04-23T18:15:25.866815+0000","last_became_peered":"2024-04-23T18:15:25.866815+0000","last_unstale":"2024-04-23T18:18:30.671938+0000","last_undegraded":"2024-04-23T18:18:30.671938+0000","last_fullsized":"2024-04-23T18:18:30.671938+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_clean_scrub_stamp":"2024-04-23T18:12:02.913350+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:42.482422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671577+0000","last_change":"2024-04-23T18:16:33.992988+0000","last_active":"2024-04-23T18:18:30.671577+0000","last_peered":"2024-04-23T18:18:30.671577+0000","last_clean":"2024-04-23T18:18:30.671577+0000","last_became_active":"2024-04-23T18:16:33.992650+0000","last_became_peered":"2024-04-23T18:16:33.992650+0000","last_unstale":"2024-04-23T18:18:30.671577+0000","last_undegraded":"2024-04-23T18:18:30.671577+0000","last_fullsized":"2024-04-23T18:18:30.671577+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:54.054708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672214+0000","last_change":"2024-04-23T18:16:32.957318+0000","last_active":"2024-04-23T18:18:30.672214+0000","last_peered":"2024-04-23T18:18:30.672214+0000","last_clean":"2024-04-23T18:18:30.672214+0000","last_became_active":"2024-04-23T18:16:32.956973+0000","last_became_peered":"2024-04-23T18:16:32.956973+0000","last_unstale":"2024-04-23T18:18:30.672214+0000","last_undegraded":"2024-04-23T18:18:30.672214+0000","last_fullsized":"2024-04-23T18:18:30.672214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:35.684825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672061+0000","last_change":"2024-04-23T18:17:36.117253+0000","last_active":"2024-04-23T18:18:30.672061+0000","last_peered":"2024-04-23T18:18:30.672061+0000","last_clean":"2024-04-23T18:18:30.672061+0000","last_became_active":"2024-04-23T18:17:36.111210+0000","last_became_peered":"2024-04-23T18:17:36.111210+0000","last_unstale":"2024-04-23T18:18:30.672061+0000","last_undegraded":"2024-04-23T18:18:30.672061+0000","last_fullsized":"2024-04-23T18:18:30.672061+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:31.599001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672229+0000","last_change":"2024-04-23T18:16:33.996345+0000","last_active":"2024-04-23T18:18:30.672229+0000","last_peered":"2024-04-23T18:18:30.672229+0000","last_clean":"2024-04-23T18:18:30.672229+0000","last_became_active":"2024-04-23T18:16:33.993802+0000","last_became_peered":"2024-04-23T18:16:33.993802+0000","last_unstale":"2024-04-23T18:18:30.672229+0000","last_undegraded":"2024-04-23T18:18:30.672229+0000","last_fullsized":"2024-04-23T18:18:30.672229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:32.410330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'44","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.722525+0000","last_change":"2024-04-23T18:16:32.957953+0000","last_active":"2024-04-23T18:18:30.722525+0000","last_peered":"2024-04-23T18:18:30.722525+0000","last_clean":"2024-04-23T18:18:30.722525+0000","last_became_active":"2024-04-23T18:16:32.957616+0000","last_became_peered":"2024-04-23T18:16:32.957616+0000","last_unstale":"2024-04-23T18:18:30.722525+0000","last_undegraded":"2024-04-23T18:18:30.722525+0000","last_fullsized":"2024-04-23T18:18:30.722525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:15.485373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.609061+0000","last_change":"2024-04-23T18:16:33.985685+0000","last_active":"2024-04-23T18:18:30.609061+0000","last_peered":"2024-04-23T18:18:30.609061+0000","last_clean":"2024-04-23T18:18:30.609061+0000","last_became_active":"2024-04-23T18:16:33.984631+0000","last_became_peered":"2024-04-23T18:16:33.984631+0000","last_unstale":"2024-04-23T18:18:30.609061+0000","last_undegraded":"2024-04-23T18:18:30.609061+0000","last_fullsized":"2024-04-23T18:18:30.609061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:45.893285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672398+0000","last_change":"2024-04-23T18:16:32.954281+0000","last_active":"2024-04-23T18:18:30.672398+0000","last_peered":"2024-04-23T18:18:30.672398+0000","last_clean":"2024-04-23T18:18:30.672398+0000","last_became_active":"2024-04-23T18:16:32.953988+0000","last_became_peered":"2024-04-23T18:16:32.953988+0000","last_unstale":"2024-04-23T18:18:30.672398+0000","last_undegraded":"2024-04-23T18:18:30.672398+0000","last_fullsized":"2024-04-23T18:18:30.672398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:09.049861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880698+0000","last_change":"2024-04-23T18:16:33.994860+0000","last_active":"2024-04-23T18:18:30.880698+0000","last_peered":"2024-04-23T18:18:30.880698+0000","last_clean":"2024-04-23T18:18:30.880698+0000","last_became_active":"2024-04-23T18:16:33.993438+0000","last_became_peered":"2024-04-23T18:16:33.993438+0000","last_unstale":"2024-04-23T18:18:30.880698+0000","last_undegraded":"2024-04-23T18:18:30.880698+0000","last_fullsized":"2024-04-23T18:18:30.880698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:55.565873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510367+0000","last_change":"2024-04-23T18:16:32.959251+0000","last_active":"2024-04-23T18:18:30.510367+0000","last_peered":"2024-04-23T18:18:30.510367+0000","last_clean":"2024-04-23T18:18:30.510367+0000","last_became_active":"2024-04-23T18:16:32.958937+0000","last_became_peered":"2024-04-23T18:16:32.958937+0000","last_unstale":"2024-04-23T18:18:30.510367+0000","last_undegraded":"2024-04-23T18:18:30.510367+0000","last_fullsized":"2024-04-23T18:18:30.510367+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.126634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608249+0000","last_change":"2024-04-23T18:16:33.995480+0000","last_active":"2024-04-23T18:18:30.608249+0000","last_peered":"2024-04-23T18:18:30.608249+0000","last_clean":"2024-04-23T18:18:30.608249+0000","last_became_active":"2024-04-23T18:16:33.994361+0000","last_became_peered":"2024-04-23T18:16:33.994361+0000","last_unstale":"2024-04-23T18:18:30.608249+0000","last_undegraded":"2024-04-23T18:18:30.608249+0000","last_fullsized":"2024-04-23T18:18:30.608249+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:59:35.240562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608282+0000","last_change":"2024-04-23T18:16:32.967787+0000","last_active":"2024-04-23T18:18:30.608282+0000","last_peered":"2024-04-23T18:18:30.608282+0000","last_clean":"2024-04-23T18:18:30.608282+0000","last_became_active":"2024-04-23T18:16:32.967463+0000","last_became_peered":"2024-04-23T18:16:32.967463+0000","last_unstale":"2024-04-23T18:18:30.608282+0000","last_undegraded":"2024-04-23T18:18:30.608282+0000","last_fullsized":"2024-04-23T18:18:30.608282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:58.743682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671931+0000","last_change":"2024-04-23T18:16:33.980558+0000","last_active":"2024-04-23T18:18:30.671931+0000","last_peered":"2024-04-23T18:18:30.671931+0000","last_clean":"2024-04-23T18:18:30.671931+0000","last_became_active":"2024-04-23T18:16:33.980146+0000","last_became_peered":"2024-04-23T18:16:33.980146+0000","last_unstale":"2024-04-23T18:18:30.671931+0000","last_undegraded":"2024-04-23T18:18:30.671931+0000","last_fullsized":"2024-04-23T18:18:30.671931+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:58.001833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512195+0000","last_change":"2024-04-23T18:16:32.962139+0000","last_active":"2024-04-23T18:18:30.512195+0000","last_peered":"2024-04-23T18:18:30.512195+0000","last_clean":"2024-04-23T18:18:30.512195+0000","last_became_active":"2024-04-23T18:16:32.961785+0000","last_became_peered":"2024-04-23T18:16:32.961785+0000","last_unstale":"2024-04-23T18:18:30.512195+0000","last_undegraded":"2024-04-23T18:18:30.512195+0000","last_fullsized":"2024-04-23T18:18:30.512195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:09.619414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508926+0000","last_change":"2024-04-23T18:17:36.214255+0000","last_active":"2024-04-23T18:18:30.508926+0000","last_peered":"2024-04-23T18:18:30.508926+0000","last_clean":"2024-04-23T18:18:30.508926+0000","last_became_active":"2024-04-23T18:17:36.213832+0000","last_became_peered":"2024-04-23T18:17:36.213832+0000","last_unstale":"2024-04-23T18:18:30.508926+0000","last_undegraded":"2024-04-23T18:18:30.508926+0000","last_fullsized":"2024-04-23T18:18:30.508926+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:59:35.497710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608350+0000","last_change":"2024-04-23T18:16:32.956965+0000","last_active":"2024-04-23T18:18:30.608350+0000","last_peered":"2024-04-23T18:18:30.608350+0000","last_clean":"2024-04-23T18:18:30.608350+0000","last_became_active":"2024-04-23T18:16:32.956672+0000","last_became_peered":"2024-04-23T18:16:32.956672+0000","last_unstale":"2024-04-23T18:18:30.608350+0000","last_undegraded":"2024-04-23T18:18:30.608350+0000","last_fullsized":"2024-04-23T18:18:30.608350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:00:44.470809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672198+0000","last_change":"2024-04-23T18:16:34.002711+0000","last_active":"2024-04-23T18:18:30.672198+0000","last_peered":"2024-04-23T18:18:30.672198+0000","last_clean":"2024-04-23T18:18:30.672198+0000","last_became_active":"2024-04-23T18:16:34.002365+0000","last_became_peered":"2024-04-23T18:16:34.002365+0000","last_unstale":"2024-04-23T18:18:30.672198+0000","last_undegraded":"2024-04-23T18:18:30.672198+0000","last_fullsized":"2024-04-23T18:18:30.672198+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:52:11.982735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512283+0000","last_change":"2024-04-23T18:16:32.965121+0000","last_active":"2024-04-23T18:18:30.512283+0000","last_peered":"2024-04-23T18:18:30.512283+0000","last_clean":"2024-04-23T18:18:30.512283+0000","last_became_active":"2024-04-23T18:16:32.964732+0000","last_became_peered":"2024-04-23T18:16:32.964732+0000","last_unstale":"2024-04-23T18:18:30.512283+0000","last_undegraded":"2024-04-23T18:18:30.512283+0000","last_fullsized":"2024-04-23T18:18:30.512283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:19:34.690245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672288+0000","last_change":"2024-04-23T18:16:33.996357+0000","last_active":"2024-04-23T18:18:30.672288+0000","last_peered":"2024-04-23T18:18:30.672288+0000","last_clean":"2024-04-23T18:18:30.672288+0000","last_became_active":"2024-04-23T18:16:33.996017+0000","last_became_peered":"2024-04-23T18:16:33.996017+0000","last_unstale":"2024-04-23T18:18:30.672288+0000","last_undegraded":"2024-04-23T18:18:30.672288+0000","last_fullsized":"2024-04-23T18:18:30.672288+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:20:57.321228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.720073+0000","last_change":"2024-04-23T18:16:32.961705+0000","last_active":"2024-04-23T18:18:30.720073+0000","last_peered":"2024-04-23T18:18:30.720073+0000","last_clean":"2024-04-23T18:18:30.720073+0000","last_became_active":"2024-04-23T18:16:32.961342+0000","last_became_peered":"2024-04-23T18:16:32.961342+0000","last_unstale":"2024-04-23T18:18:30.720073+0000","last_undegraded":"2024-04-23T18:18:30.720073+0000","last_fullsized":"2024-04-23T18:18:30.720073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:51.340910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672001+0000","last_change":"2024-04-23T18:16:33.992944+0000","last_active":"2024-04-23T18:18:30.672001+0000","last_peered":"2024-04-23T18:18:30.672001+0000","last_clean":"2024-04-23T18:18:30.672001+0000","last_became_active":"2024-04-23T18:16:33.992579+0000","last_became_peered":"2024-04-23T18:16:33.992579+0000","last_unstale":"2024-04-23T18:18:30.672001+0000","last_undegraded":"2024-04-23T18:18:30.672001+0000","last_fullsized":"2024-04-23T18:18:30.672001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:13:37.829588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672489+0000","last_change":"2024-04-23T18:16:32.955192+0000","last_active":"2024-04-23T18:18:30.672489+0000","last_peered":"2024-04-23T18:18:30.672489+0000","last_clean":"2024-04-23T18:18:30.672489+0000","last_became_active":"2024-04-23T18:16:32.954860+0000","last_became_peered":"2024-04-23T18:16:32.954860+0000","last_unstale":"2024-04-23T18:18:30.672489+0000","last_undegraded":"2024-04-23T18:18:30.672489+0000","last_fullsized":"2024-04-23T18:18:30.672489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:20.796060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672367+0000","last_change":"2024-04-23T18:16:33.986211+0000","last_active":"2024-04-23T18:18:30.672367+0000","last_peered":"2024-04-23T18:18:30.672367+0000","last_clean":"2024-04-23T18:18:30.672367+0000","last_became_active":"2024-04-23T18:16:33.985679+0000","last_became_peered":"2024-04-23T18:16:33.985679+0000","last_unstale":"2024-04-23T18:18:30.672367+0000","last_undegraded":"2024-04-23T18:18:30.672367+0000","last_fullsized":"2024-04-23T18:18:30.672367+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:03:38.797864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672089+0000","last_change":"2024-04-23T18:16:33.979462+0000","last_active":"2024-04-23T18:18:30.672089+0000","last_peered":"2024-04-23T18:18:30.672089+0000","last_clean":"2024-04-23T18:18:30.672089+0000","last_became_active":"2024-04-23T18:16:33.979114+0000","last_became_peered":"2024-04-23T18:16:33.979114+0000","last_unstale":"2024-04-23T18:18:30.672089+0000","last_undegraded":"2024-04-23T18:18:30.672089+0000","last_fullsized":"2024-04-23T18:18:30.672089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:16:49.708240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512027+0000","last_change":"2024-04-23T18:16:33.979255+0000","last_active":"2024-04-23T18:18:30.512027+0000","last_peered":"2024-04-23T18:18:30.512027+0000","last_clean":"2024-04-23T18:18:30.512027+0000","last_became_active":"2024-04-23T18:16:33.978934+0000","last_became_peered":"2024-04-23T18:16:33.978934+0000","last_unstale":"2024-04-23T18:18:30.512027+0000","last_undegraded":"2024-04-23T18:18:30.512027+0000","last_fullsized":"2024-04-23T18:18:30.512027+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:10.104564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.511957+0000","last_change":"2024-04-23T18:16:33.981877+0000","last_active":"2024-04-23T18:18:30.511957+0000","last_peered":"2024-04-23T18:18:30.511957+0000","last_clean":"2024-04-23T18:18:30.511957+0000","last_became_active":"2024-04-23T18:16:33.981536+0000","last_became_peered":"2024-04-23T18:16:33.981536+0000","last_unstale":"2024-04-23T18:18:30.511957+0000","last_undegraded":"2024-04-23T18:18:30.511957+0000","last_fullsized":"2024-04-23T18:18:30.511957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:00:15.056779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880509+0000","last_change":"2024-04-23T18:16:33.999090+0000","last_active":"2024-04-23T18:18:30.880509+0000","last_peered":"2024-04-23T18:18:30.880509+0000","last_clean":"2024-04-23T18:18:30.880509+0000","last_became_active":"2024-04-23T18:16:33.998513+0000","last_became_peered":"2024-04-23T18:16:33.998513+0000","last_unstale":"2024-04-23T18:18:30.880509+0000","last_undegraded":"2024-04-23T18:18:30.880509+0000","last_fullsized":"2024-04-23T18:18:30.880509+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:14.115406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672082+0000","last_change":"2024-04-23T18:16:33.993375+0000","last_active":"2024-04-23T18:18:30.672082+0000","last_peered":"2024-04-23T18:18:30.672082+0000","last_clean":"2024-04-23T18:18:30.672082+0000","last_became_active":"2024-04-23T18:16:33.991841+0000","last_became_peered":"2024-04-23T18:16:33.991841+0000","last_unstale":"2024-04-23T18:18:30.672082+0000","last_undegraded":"2024-04-23T18:18:30.672082+0000","last_fullsized":"2024-04-23T18:18:30.672082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:12.777495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672189+0000","last_change":"2024-04-23T18:16:33.983621+0000","last_active":"2024-04-23T18:18:30.672189+0000","last_peered":"2024-04-23T18:18:30.672189+0000","last_clean":"2024-04-23T18:18:30.672189+0000","last_became_active":"2024-04-23T18:16:33.983356+0000","last_became_peered":"2024-04-23T18:16:33.983356+0000","last_unstale":"2024-04-23T18:18:30.672189+0000","last_undegraded":"2024-04-23T18:18:30.672189+0000","last_fullsized":"2024-04-23T18:18:30.672189+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:27:19.391281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512250+0000","last_change":"2024-04-23T18:16:33.987952+0000","last_active":"2024-04-23T18:18:30.512250+0000","last_peered":"2024-04-23T18:18:30.512250+0000","last_clean":"2024-04-23T18:18:30.512250+0000","last_became_active":"2024-04-23T18:16:33.987334+0000","last_became_peered":"2024-04-23T18:16:33.987334+0000","last_unstale":"2024-04-23T18:18:30.512250+0000","last_undegraded":"2024-04-23T18:18:30.512250+0000","last_fullsized":"2024-04-23T18:18:30.512250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:41.902779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.609205+0000","last_change":"2024-04-23T18:16:33.983914+0000","last_active":"2024-04-23T18:18:30.609205+0000","last_peered":"2024-04-23T18:18:30.609205+0000","last_clean":"2024-04-23T18:18:30.609205+0000","last_became_active":"2024-04-23T18:16:33.983444+0000","last_became_peered":"2024-04-23T18:16:33.983444+0000","last_unstale":"2024-04-23T18:18:30.609205+0000","last_undegraded":"2024-04-23T18:18:30.609205+0000","last_fullsized":"2024-04-23T18:18:30.609205+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:10:17.630049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.613877+0000","last_change":"2024-04-23T18:16:33.992496+0000","last_active":"2024-04-23T18:18:30.613877+0000","last_peered":"2024-04-23T18:18:30.613877+0000","last_clean":"2024-04-23T18:18:30.613877+0000","last_became_active":"2024-04-23T18:16:33.992155+0000","last_became_peered":"2024-04-23T18:16:33.992155+0000","last_unstale":"2024-04-23T18:18:30.613877+0000","last_undegraded":"2024-04-23T18:18:30.613877+0000","last_fullsized":"2024-04-23T18:18:30.613877+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:48.835992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608101+0000","last_change":"2024-04-23T18:16:33.984409+0000","last_active":"2024-04-23T18:18:30.608101+0000","last_peered":"2024-04-23T18:18:30.608101+0000","last_clean":"2024-04-23T18:18:30.608101+0000","last_became_active":"2024-04-23T18:16:33.984073+0000","last_became_peered":"2024-04-23T18:16:33.984073+0000","last_unstale":"2024-04-23T18:18:30.608101+0000","last_undegraded":"2024-04-23T18:18:30.608101+0000","last_fullsized":"2024-04-23T18:18:30.608101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:45:37.796783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510525+0000","last_change":"2024-04-23T18:16:33.986376+0000","last_active":"2024-04-23T18:18:30.510525+0000","last_peered":"2024-04-23T18:18:30.510525+0000","last_clean":"2024-04-23T18:18:30.510525+0000","last_became_active":"2024-04-23T18:16:33.986012+0000","last_became_peered":"2024-04-23T18:16:33.986012+0000","last_unstale":"2024-04-23T18:18:30.510525+0000","last_undegraded":"2024-04-23T18:18:30.510525+0000","last_fullsized":"2024-04-23T18:18:30.510525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:26.940215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672288+0000","last_change":"2024-04-23T18:16:33.999795+0000","last_active":"2024-04-23T18:18:30.672288+0000","last_peered":"2024-04-23T18:18:30.672288+0000","last_clean":"2024-04-23T18:18:30.672288+0000","last_became_active":"2024-04-23T18:16:33.997034+0000","last_became_peered":"2024-04-23T18:16:33.997034+0000","last_unstale":"2024-04-23T18:18:30.672288+0000","last_undegraded":"2024-04-23T18:18:30.672288+0000","last_fullsized":"2024-04-23T18:18:30.672288+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:23.983846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672329+0000","last_change":"2024-04-23T18:16:33.985495+0000","last_active":"2024-04-23T18:18:30.672329+0000","last_peered":"2024-04-23T18:18:30.672329+0000","last_clean":"2024-04-23T18:18:30.672329+0000","last_became_active":"2024-04-23T18:16:33.985178+0000","last_became_peered":"2024-04-23T18:16:33.985178+0000","last_unstale":"2024-04-23T18:18:30.672329+0000","last_undegraded":"2024-04-23T18:18:30.672329+0000","last_fullsized":"2024-04-23T18:18:30.672329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:22:25.325756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":597706,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":804,"num_read_kb":515,"num_write":292,"num_write_kb":1442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1147,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1793118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":231,"ondisk_log_size":231,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318307,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":477387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579945,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32964,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712188,"statfs":{"total":95995035648,"available":95961280512,"internally_reserved":0,"allocated":1830912,"data_stored":1268691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776174,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36708,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708444,"statfs":{"total":95995035648,"available":95957446656,"internally_reserved":0,"allocated":1400832,"data_stored":870705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037810,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1388544,"data_stored":870491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712408,"statfs":{"total":95995035648,"available":95961505792,"internally_reserved":0,"allocated":1601536,"data_stored":1071493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":476878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":544951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40472,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704680,"statfs":{"total":95995035648,"available":95953592320,"internally_reserved":0,"allocated":999424,"data_stored":469193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1081344,"data_stored":544134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41000,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704152,"statfs":{"total":95995035648,"available":95953051648,"internally_reserved":0,"allocated":1540096,"data_stored":1002005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":999424,"data_stored":478937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28724,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716428,"statfs":{"total":95995035648,"available":95965622272,"internally_reserved":0,"allocated":1748992,"data_stored":1201155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":67463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":201535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:18:36.150 INFO:tasks.quiescer.ceph_manager:clean! 2024-04-23T18:18:36.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T18:18:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:18:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:18:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: Updating smithi050:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:18:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:18:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:18:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:18:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:18:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:18:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:18:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: Updating smithi050:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:18:36.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3825086338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.289 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:36 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2102803724' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:36.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.296+0000 7f13435bb640 1 -- 172.21.15.50:0/1017947364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f133c101020 msgr2=0x7f133c101400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.296+0000 7f13435bb640 1 --2- 172.21.15.50:0/1017947364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f133c101020 0x7f133c101400 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1338009900 tx=0x7f133802a480 comp rx=0 tx=0).stop 2024-04-23T18:18:36.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.296+0000 7f13435bb640 1 -- 172.21.15.50:0/1017947364 shutdown_connections 2024-04-23T18:18:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.296+0000 7f13435bb640 1 --2- 172.21.15.50:0/1017947364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f133c106340 0x7f133c1067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.296+0000 7f13435bb640 1 --2- 172.21.15.50:0/1017947364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f133c101940 0x7f133c105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.296+0000 7f13435bb640 1 --2- 172.21.15.50:0/1017947364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f133c101020 0x7f133c101400 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.296+0000 7f13435bb640 1 -- 172.21.15.50:0/1017947364 >> 172.21.15.50:0/1017947364 conn(0x7f133c077fb0 msgr2=0x7f133c0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.297+0000 7f13435bb640 1 -- 172.21.15.50:0/1017947364 shutdown_connections 2024-04-23T18:18:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.297+0000 7f13435bb640 1 -- 172.21.15.50:0/1017947364 wait complete. 2024-04-23T18:18:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.297+0000 7f13435bb640 1 Processor -- start 2024-04-23T18:18:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.297+0000 7f13435bb640 1 -- start start 2024-04-23T18:18:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.297+0000 7f13435bb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f133c101020 0x7f133c19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.297+0000 7f13435bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f133c101940 0x7f133c19f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.297+0000 7f13435bb640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f133c106340 0x7f133c1a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.297+0000 7f13435bb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f133c1187f0 con 0x7f133c101940 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.297+0000 7f13435bb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f133c118670 con 0x7f133c101020 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.297+0000 7f13435bb640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f133c118970 con 0x7f133c106340 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.297+0000 7f1340b56640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f133c101940 0x7f133c19f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f1340b56640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f133c101940 0x7f133c19f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49272/0 (socket says 172.21.15.50:49272) 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f1340b56640 1 -- 172.21.15.50:0/375333234 learned_addr learned my addr 172.21.15.50:0/375333234 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f1341357640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f133c101020 0x7f133c19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f1341b58640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f133c106340 0x7f133c1a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f1340b56640 1 -- 172.21.15.50:0/375333234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f133c106340 msgr2=0x7f133c1a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f1340b56640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f133c106340 0x7f133c1a3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f1340b56640 1 -- 172.21.15.50:0/375333234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f133c101020 msgr2=0x7f133c19ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f1340b56640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f133c101020 0x7f133c19ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f1340b56640 1 -- 172.21.15.50:0/375333234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f133c1a3d80 con 0x7f133c101940 2024-04-23T18:18:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f1341b58640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f133c106340 0x7f133c1a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f1340b56640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f133c101940 0x7f133c19f2d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f132c00d770 tx=0x7f132c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f13327fc640 1 -- 172.21.15.50:0/375333234 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c017070 con 0x7f133c101940 2024-04-23T18:18:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f13327fc640 1 -- 172.21.15.50:0/375333234 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f132c0054d0 con 0x7f133c101940 2024-04-23T18:18:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f13435bb640 1 -- 172.21.15.50:0/375333234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f133c1a4070 con 0x7f133c101940 2024-04-23T18:18:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f13327fc640 1 -- 172.21.15.50:0/375333234 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c004c30 con 0x7f133c101940 2024-04-23T18:18:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.298+0000 7f13435bb640 1 -- 172.21.15.50:0/375333234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f133c1ab900 con 0x7f133c101940 2024-04-23T18:18:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.299+0000 7f13435bb640 1 -- 172.21.15.50:0/375333234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f133c0691c0 con 0x7f133c101940 2024-04-23T18:18:36.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.300+0000 7f13327fc640 1 -- 172.21.15.50:0/375333234 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f132c004e10 con 0x7f133c101940 2024-04-23T18:18:36.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.301+0000 7f13327fc640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f13280a3e30 0x7f13280a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:36.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.301+0000 7f13327fc640 1 -- 172.21.15.50:0/375333234 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f132c0d0490 con 0x7f133c101940 2024-04-23T18:18:36.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.301+0000 7f1341357640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f13280a3e30 0x7f13280a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:36.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.301+0000 7f1341357640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f13280a3e30 0x7f13280a62f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1338031440 tx=0x7f13380095f0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:36.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.305+0000 7f13327fc640 1 -- 172.21.15.50:0/375333234 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f132c09c9d0 con 0x7f133c101940 2024-04-23T18:18:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:18:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:18:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: Updating smithi050:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T18:18:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:18:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:18:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:18:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-23T18:18:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: Updating smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:18:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: Updating smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:18:36.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: Updating smithi050:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3825086338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:18:36.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:18:36.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:36 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2102803724' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:36.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.489+0000 7f13435bb640 1 -- 172.21.15.50:0/375333234 --> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f133c1054f0 con 0x7f13280a3e30 2024-04-23T18:18:36.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.496+0000 7f13327fc640 1 -- 172.21.15.50:0/375333234 <== mgr.25018 v2:172.21.15.50:6800/4210816660 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257741 (secure 0 0 0) 0x7f133c1054f0 con 0x7f13280a3e30 2024-04-23T18:18:36.491 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:36.493 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-23T18:18:36.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.499+0000 7f13435bb640 1 -- 172.21.15.50:0/375333234 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f13280a3e30 msgr2=0x7f13280a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.499+0000 7f13435bb640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f13280a3e30 0x7f13280a62f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1338031440 tx=0x7f13380095f0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.499+0000 7f13435bb640 1 -- 172.21.15.50:0/375333234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f133c101940 msgr2=0x7f133c19f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.499+0000 7f13435bb640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f133c101940 0x7f133c19f2d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f132c00d770 tx=0x7f132c00dc30 comp rx=0 tx=0).stop 2024-04-23T18:18:36.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.499+0000 7f13435bb640 1 -- 172.21.15.50:0/375333234 shutdown_connections 2024-04-23T18:18:36.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.499+0000 7f13435bb640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f13280a3e30 0x7f13280a62f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.499+0000 7f13435bb640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f133c106340 0x7f133c1a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.499+0000 7f13435bb640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f133c101940 0x7f133c19f2d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.499+0000 7f13435bb640 1 --2- 172.21.15.50:0/375333234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f133c101020 0x7f133c19ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.499+0000 7f13435bb640 1 -- 172.21.15.50:0/375333234 >> 172.21.15.50:0/375333234 conn(0x7f133c077fb0 msgr2=0x7f133c103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:36.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.499+0000 7f13435bb640 1 -- 172.21.15.50:0/375333234 shutdown_connections 2024-04-23T18:18:36.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.499+0000 7f13435bb640 1 -- 172.21.15.50:0/375333234 wait complete. 2024-04-23T18:18:36.503 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-23T18:18:34.524804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188308,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":940,"num_read_kb":634,"num_write":475,"num_write_kb":4382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181883,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":397756,"kb_used_data":15296,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124544068,"statfs":{"total":1151940427776,"available":1151533125632,"internally_reserved":0,"allocated":15663104,"data_stored":9276020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991529"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672105+0000","last_change":"2024-04-23T18:16:33.987102+0000","last_active":"2024-04-23T18:18:30.672105+0000","last_peered":"2024-04-23T18:18:30.672105+0000","last_clean":"2024-04-23T18:18:30.672105+0000","last_became_active":"2024-04-23T18:16:33.986716+0000","last_became_peered":"2024-04-23T18:16:33.986716+0000","last_unstale":"2024-04-23T18:18:30.672105+0000","last_undegraded":"2024-04-23T18:18:30.672105+0000","last_fullsized":"2024-04-23T18:18:30.672105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:41.895219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880751+0000","last_change":"2024-04-23T18:16:33.995703+0000","last_active":"2024-04-23T18:18:30.880751+0000","last_peered":"2024-04-23T18:18:30.880751+0000","last_clean":"2024-04-23T18:18:30.880751+0000","last_became_active":"2024-04-23T18:16:33.995242+0000","last_became_peered":"2024-04-23T18:16:33.995242+0000","last_unstale":"2024-04-23T18:18:30.880751+0000","last_undegraded":"2024-04-23T18:18:30.880751+0000","last_fullsized":"2024-04-23T18:18:30.880751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:57.813822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880606+0000","last_change":"2024-04-23T18:16:34.000864+0000","last_active":"2024-04-23T18:18:30.880606+0000","last_peered":"2024-04-23T18:18:30.880606+0000","last_clean":"2024-04-23T18:18:30.880606+0000","last_became_active":"2024-04-23T18:16:33.998934+0000","last_became_peered":"2024-04-23T18:16:33.998934+0000","last_unstale":"2024-04-23T18:18:30.880606+0000","last_undegraded":"2024-04-23T18:18:30.880606+0000","last_fullsized":"2024-04-23T18:18:30.880606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:47:29.481697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672137+0000","last_change":"2024-04-23T18:16:33.989351+0000","last_active":"2024-04-23T18:18:30.672137+0000","last_peered":"2024-04-23T18:18:30.672137+0000","last_clean":"2024-04-23T18:18:30.672137+0000","last_became_active":"2024-04-23T18:16:33.989030+0000","last_became_peered":"2024-04-23T18:16:33.989030+0000","last_unstale":"2024-04-23T18:18:30.672137+0000","last_undegraded":"2024-04-23T18:18:30.672137+0000","last_fullsized":"2024-04-23T18:18:30.672137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:29:03.023675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608347+0000","last_change":"2024-04-23T18:16:33.996062+0000","last_active":"2024-04-23T18:18:30.608347+0000","last_peered":"2024-04-23T18:18:30.608347+0000","last_clean":"2024-04-23T18:18:30.608347+0000","last_became_active":"2024-04-23T18:16:33.995732+0000","last_became_peered":"2024-04-23T18:16:33.995732+0000","last_unstale":"2024-04-23T18:18:30.608347+0000","last_undegraded":"2024-04-23T18:18:30.608347+0000","last_fullsized":"2024-04-23T18:18:30.608347+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:04.175580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.617038+0000","last_change":"2024-04-23T18:16:33.989895+0000","last_active":"2024-04-23T18:18:30.617038+0000","last_peered":"2024-04-23T18:18:30.617038+0000","last_clean":"2024-04-23T18:18:30.617038+0000","last_became_active":"2024-04-23T18:16:33.989547+0000","last_became_peered":"2024-04-23T18:16:33.989547+0000","last_unstale":"2024-04-23T18:18:30.617038+0000","last_undegraded":"2024-04-23T18:18:30.617038+0000","last_fullsized":"2024-04-23T18:18:30.617038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:12:37.317974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672251+0000","last_change":"2024-04-23T18:16:33.999636+0000","last_active":"2024-04-23T18:18:30.672251+0000","last_peered":"2024-04-23T18:18:30.672251+0000","last_clean":"2024-04-23T18:18:30.672251+0000","last_became_active":"2024-04-23T18:16:33.999282+0000","last_became_peered":"2024-04-23T18:16:33.999282+0000","last_unstale":"2024-04-23T18:18:30.672251+0000","last_undegraded":"2024-04-23T18:18:30.672251+0000","last_fullsized":"2024-04-23T18:18:30.672251+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:28.841392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608063+0000","last_change":"2024-04-23T18:16:33.996602+0000","last_active":"2024-04-23T18:18:30.608063+0000","last_peered":"2024-04-23T18:18:30.608063+0000","last_clean":"2024-04-23T18:18:30.608063+0000","last_became_active":"2024-04-23T18:16:33.994401+0000","last_became_peered":"2024-04-23T18:16:33.994401+0000","last_unstale":"2024-04-23T18:18:30.608063+0000","last_undegraded":"2024-04-23T18:18:30.608063+0000","last_fullsized":"2024-04-23T18:18:30.608063+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:31:54.553537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510925+0000","last_change":"2024-04-23T18:16:33.994763+0000","last_active":"2024-04-23T18:18:30.510925+0000","last_peered":"2024-04-23T18:18:30.510925+0000","last_clean":"2024-04-23T18:18:30.510925+0000","last_became_active":"2024-04-23T18:16:33.994408+0000","last_became_peered":"2024-04-23T18:16:33.994408+0000","last_unstale":"2024-04-23T18:18:30.510925+0000","last_undegraded":"2024-04-23T18:18:30.510925+0000","last_fullsized":"2024-04-23T18:18:30.510925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:02.706160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.617268+0000","last_change":"2024-04-23T18:16:33.993435+0000","last_active":"2024-04-23T18:18:30.617268+0000","last_peered":"2024-04-23T18:18:30.617268+0000","last_clean":"2024-04-23T18:18:30.617268+0000","last_became_active":"2024-04-23T18:16:33.993108+0000","last_became_peered":"2024-04-23T18:16:33.993108+0000","last_unstale":"2024-04-23T18:18:30.617268+0000","last_undegraded":"2024-04-23T18:18:30.617268+0000","last_fullsized":"2024-04-23T18:18:30.617268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:28:32.584896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672170+0000","last_change":"2024-04-23T18:16:33.997526+0000","last_active":"2024-04-23T18:18:30.672170+0000","last_peered":"2024-04-23T18:18:30.672170+0000","last_clean":"2024-04-23T18:18:30.672170+0000","last_became_active":"2024-04-23T18:16:33.997189+0000","last_became_peered":"2024-04-23T18:16:33.997189+0000","last_unstale":"2024-04-23T18:18:30.672170+0000","last_undegraded":"2024-04-23T18:18:30.672170+0000","last_fullsized":"2024-04-23T18:18:30.672170+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:11:41.208261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671985+0000","last_change":"2024-04-23T18:16:33.989775+0000","last_active":"2024-04-23T18:18:30.671985+0000","last_peered":"2024-04-23T18:18:30.671985+0000","last_clean":"2024-04-23T18:18:30.671985+0000","last_became_active":"2024-04-23T18:16:33.989382+0000","last_became_peered":"2024-04-23T18:16:33.989382+0000","last_unstale":"2024-04-23T18:18:30.671985+0000","last_undegraded":"2024-04-23T18:18:30.671985+0000","last_fullsized":"2024-04-23T18:18:30.671985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:20:17.756258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672114+0000","last_change":"2024-04-23T18:16:34.005871+0000","last_active":"2024-04-23T18:18:30.672114+0000","last_peered":"2024-04-23T18:18:30.672114+0000","last_clean":"2024-04-23T18:18:30.672114+0000","last_became_active":"2024-04-23T18:16:34.005218+0000","last_became_peered":"2024-04-23T18:16:34.005218+0000","last_unstale":"2024-04-23T18:18:30.672114+0000","last_undegraded":"2024-04-23T18:18:30.672114+0000","last_fullsized":"2024-04-23T18:18:30.672114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:30:03.864306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880355+0000","last_change":"2024-04-23T18:16:33.995054+0000","last_active":"2024-04-23T18:18:30.880355+0000","last_peered":"2024-04-23T18:18:30.880355+0000","last_clean":"2024-04-23T18:18:30.880355+0000","last_became_active":"2024-04-23T18:16:33.994389+0000","last_became_peered":"2024-04-23T18:16:33.994389+0000","last_unstale":"2024-04-23T18:18:30.880355+0000","last_undegraded":"2024-04-23T18:18:30.880355+0000","last_fullsized":"2024-04-23T18:18:30.880355+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:27.344616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880258+0000","last_change":"2024-04-23T18:16:33.984490+0000","last_active":"2024-04-23T18:18:30.880258+0000","last_peered":"2024-04-23T18:18:30.880258+0000","last_clean":"2024-04-23T18:18:30.880258+0000","last_became_active":"2024-04-23T18:16:33.984071+0000","last_became_peered":"2024-04-23T18:16:33.984071+0000","last_unstale":"2024-04-23T18:18:30.880258+0000","last_undegraded":"2024-04-23T18:18:30.880258+0000","last_fullsized":"2024-04-23T18:18:30.880258+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:43:23.589290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510517+0000","last_change":"2024-04-23T18:16:34.002163+0000","last_active":"2024-04-23T18:18:30.510517+0000","last_peered":"2024-04-23T18:18:30.510517+0000","last_clean":"2024-04-23T18:18:30.510517+0000","last_became_active":"2024-04-23T18:16:34.001764+0000","last_became_peered":"2024-04-23T18:16:34.001764+0000","last_unstale":"2024-04-23T18:18:30.510517+0000","last_undegraded":"2024-04-23T18:18:30.510517+0000","last_fullsized":"2024-04-23T18:18:30.510517+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:16:03.616358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672114+0000","last_change":"2024-04-23T18:16:33.987586+0000","last_active":"2024-04-23T18:18:30.672114+0000","last_peered":"2024-04-23T18:18:30.672114+0000","last_clean":"2024-04-23T18:18:30.672114+0000","last_became_active":"2024-04-23T18:16:33.987266+0000","last_became_peered":"2024-04-23T18:16:33.987266+0000","last_unstale":"2024-04-23T18:18:30.672114+0000","last_undegraded":"2024-04-23T18:18:30.672114+0000","last_fullsized":"2024-04-23T18:18:30.672114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:46:41.610529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508920+0000","last_change":"2024-04-23T18:16:33.992938+0000","last_active":"2024-04-23T18:18:30.508920+0000","last_peered":"2024-04-23T18:18:30.508920+0000","last_clean":"2024-04-23T18:18:30.508920+0000","last_became_active":"2024-04-23T18:16:33.990441+0000","last_became_peered":"2024-04-23T18:16:33.990441+0000","last_unstale":"2024-04-23T18:18:30.508920+0000","last_undegraded":"2024-04-23T18:18:30.508920+0000","last_fullsized":"2024-04-23T18:18:30.508920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:10.689333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512446+0000","last_change":"2024-04-23T18:16:33.994844+0000","last_active":"2024-04-23T18:18:30.512446+0000","last_peered":"2024-04-23T18:18:30.512446+0000","last_clean":"2024-04-23T18:18:30.512446+0000","last_became_active":"2024-04-23T18:16:33.994505+0000","last_became_peered":"2024-04-23T18:16:33.994505+0000","last_unstale":"2024-04-23T18:18:30.512446+0000","last_undegraded":"2024-04-23T18:18:30.512446+0000","last_fullsized":"2024-04-23T18:18:30.512446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:32:33.445266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508971+0000","last_change":"2024-04-23T18:16:32.958841+0000","last_active":"2024-04-23T18:18:30.508971+0000","last_peered":"2024-04-23T18:18:30.508971+0000","last_clean":"2024-04-23T18:18:30.508971+0000","last_became_active":"2024-04-23T18:16:32.958529+0000","last_became_peered":"2024-04-23T18:16:32.958529+0000","last_unstale":"2024-04-23T18:18:30.508971+0000","last_undegraded":"2024-04-23T18:18:30.508971+0000","last_fullsized":"2024-04-23T18:18:30.508971+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:47:09.933786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608592+0000","last_change":"2024-04-23T18:16:33.992108+0000","last_active":"2024-04-23T18:18:30.608592+0000","last_peered":"2024-04-23T18:18:30.608592+0000","last_clean":"2024-04-23T18:18:30.608592+0000","last_became_active":"2024-04-23T18:16:33.991751+0000","last_became_peered":"2024-04-23T18:16:33.991751+0000","last_unstale":"2024-04-23T18:18:30.608592+0000","last_undegraded":"2024-04-23T18:18:30.608592+0000","last_fullsized":"2024-04-23T18:18:30.608592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:07:57.416263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608867+0000","last_change":"2024-04-23T18:16:32.960461+0000","last_active":"2024-04-23T18:18:30.608867+0000","last_peered":"2024-04-23T18:18:30.608867+0000","last_clean":"2024-04-23T18:18:30.608867+0000","last_became_active":"2024-04-23T18:16:32.960115+0000","last_became_peered":"2024-04-23T18:16:32.960115+0000","last_unstale":"2024-04-23T18:18:30.608867+0000","last_undegraded":"2024-04-23T18:18:30.608867+0000","last_fullsized":"2024-04-23T18:18:30.608867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:30.533043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880852+0000","last_change":"2024-04-23T18:16:34.001831+0000","last_active":"2024-04-23T18:18:30.880852+0000","last_peered":"2024-04-23T18:18:30.880852+0000","last_clean":"2024-04-23T18:18:30.880852+0000","last_became_active":"2024-04-23T18:16:34.001486+0000","last_became_peered":"2024-04-23T18:16:34.001486+0000","last_unstale":"2024-04-23T18:18:30.880852+0000","last_undegraded":"2024-04-23T18:18:30.880852+0000","last_fullsized":"2024-04-23T18:18:30.880852+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:06:40.452366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608622+0000","last_change":"2024-04-23T18:16:32.960387+0000","last_active":"2024-04-23T18:18:30.608622+0000","last_peered":"2024-04-23T18:18:30.608622+0000","last_clean":"2024-04-23T18:18:30.608622+0000","last_became_active":"2024-04-23T18:16:32.960046+0000","last_became_peered":"2024-04-23T18:16:32.960046+0000","last_unstale":"2024-04-23T18:18:30.608622+0000","last_undegraded":"2024-04-23T18:18:30.608622+0000","last_fullsized":"2024-04-23T18:18:30.608622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:03.296653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510543+0000","last_change":"2024-04-23T18:16:33.996520+0000","last_active":"2024-04-23T18:18:30.510543+0000","last_peered":"2024-04-23T18:18:30.510543+0000","last_clean":"2024-04-23T18:18:30.510543+0000","last_became_active":"2024-04-23T18:16:33.995636+0000","last_became_peered":"2024-04-23T18:16:33.995636+0000","last_unstale":"2024-04-23T18:18:30.510543+0000","last_undegraded":"2024-04-23T18:18:30.510543+0000","last_fullsized":"2024-04-23T18:18:30.510543+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:29.665407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608556+0000","last_change":"2024-04-23T18:16:32.956813+0000","last_active":"2024-04-23T18:18:30.608556+0000","last_peered":"2024-04-23T18:18:30.608556+0000","last_clean":"2024-04-23T18:18:30.608556+0000","last_became_active":"2024-04-23T18:16:32.956505+0000","last_became_peered":"2024-04-23T18:16:32.956505+0000","last_unstale":"2024-04-23T18:18:30.608556+0000","last_undegraded":"2024-04-23T18:18:30.608556+0000","last_fullsized":"2024-04-23T18:18:30.608556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:36:27.330353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.614156+0000","last_change":"2024-04-23T18:16:33.999754+0000","last_active":"2024-04-23T18:18:30.614156+0000","last_peered":"2024-04-23T18:18:30.614156+0000","last_clean":"2024-04-23T18:18:30.614156+0000","last_became_active":"2024-04-23T18:16:33.999034+0000","last_became_peered":"2024-04-23T18:16:33.999034+0000","last_unstale":"2024-04-23T18:18:30.614156+0000","last_undegraded":"2024-04-23T18:18:30.614156+0000","last_fullsized":"2024-04-23T18:18:30.614156+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:26:08.595745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":160,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:31.190208+0000","last_change":"2024-04-23T18:17:36.120371+0000","last_active":"2024-04-23T18:18:31.190208+0000","last_peered":"2024-04-23T18:18:31.190208+0000","last_clean":"2024-04-23T18:18:31.190208+0000","last_became_active":"2024-04-23T18:17:36.116149+0000","last_became_peered":"2024-04-23T18:17:36.116149+0000","last_unstale":"2024-04-23T18:18:31.190208+0000","last_undegraded":"2024-04-23T18:18:31.190208+0000","last_fullsized":"2024-04-23T18:18:31.190208+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:54.994254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608344+0000","last_change":"2024-04-23T18:16:33.995166+0000","last_active":"2024-04-23T18:18:30.608344+0000","last_peered":"2024-04-23T18:18:30.608344+0000","last_clean":"2024-04-23T18:18:30.608344+0000","last_became_active":"2024-04-23T18:16:33.994347+0000","last_became_peered":"2024-04-23T18:16:33.994347+0000","last_unstale":"2024-04-23T18:18:30.608344+0000","last_undegraded":"2024-04-23T18:18:30.608344+0000","last_fullsized":"2024-04-23T18:18:30.608344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.305837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.616932+0000","last_change":"2024-04-23T18:16:32.959358+0000","last_active":"2024-04-23T18:18:30.616932+0000","last_peered":"2024-04-23T18:18:30.616932+0000","last_clean":"2024-04-23T18:18:30.616932+0000","last_became_active":"2024-04-23T18:16:32.959036+0000","last_became_peered":"2024-04-23T18:16:32.959036+0000","last_unstale":"2024-04-23T18:18:30.616932+0000","last_undegraded":"2024-04-23T18:18:30.616932+0000","last_fullsized":"2024-04-23T18:18:30.616932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:03:15.734754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.614091+0000","last_change":"2024-04-23T18:16:33.988578+0000","last_active":"2024-04-23T18:18:30.614091+0000","last_peered":"2024-04-23T18:18:30.614091+0000","last_clean":"2024-04-23T18:18:30.614091+0000","last_became_active":"2024-04-23T18:16:33.988248+0000","last_became_peered":"2024-04-23T18:16:33.988248+0000","last_unstale":"2024-04-23T18:18:30.614091+0000","last_undegraded":"2024-04-23T18:18:30.614091+0000","last_fullsized":"2024-04-23T18:18:30.614091+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:45:15.538971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671882+0000","last_change":"2024-04-23T18:16:32.952945+0000","last_active":"2024-04-23T18:18:30.671882+0000","last_peered":"2024-04-23T18:18:30.671882+0000","last_clean":"2024-04-23T18:18:30.671882+0000","last_became_active":"2024-04-23T18:16:32.952602+0000","last_became_peered":"2024-04-23T18:16:32.952602+0000","last_unstale":"2024-04-23T18:18:30.671882+0000","last_undegraded":"2024-04-23T18:18:30.671882+0000","last_fullsized":"2024-04-23T18:18:30.671882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:20.037431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608129+0000","last_change":"2024-04-23T18:16:33.996897+0000","last_active":"2024-04-23T18:18:30.608129+0000","last_peered":"2024-04-23T18:18:30.608129+0000","last_clean":"2024-04-23T18:18:30.608129+0000","last_became_active":"2024-04-23T18:16:33.996214+0000","last_became_peered":"2024-04-23T18:16:33.996214+0000","last_unstale":"2024-04-23T18:18:30.608129+0000","last_undegraded":"2024-04-23T18:18:30.608129+0000","last_fullsized":"2024-04-23T18:18:30.608129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:34.931697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.613841+0000","last_change":"2024-04-23T18:16:32.959637+0000","last_active":"2024-04-23T18:18:30.613841+0000","last_peered":"2024-04-23T18:18:30.613841+0000","last_clean":"2024-04-23T18:18:30.613841+0000","last_became_active":"2024-04-23T18:16:32.959332+0000","last_became_peered":"2024-04-23T18:16:32.959332+0000","last_unstale":"2024-04-23T18:18:30.613841+0000","last_undegraded":"2024-04-23T18:18:30.613841+0000","last_fullsized":"2024-04-23T18:18:30.613841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:30:49.089101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508653+0000","last_change":"2024-04-23T18:16:33.986288+0000","last_active":"2024-04-23T18:18:30.508653+0000","last_peered":"2024-04-23T18:18:30.508653+0000","last_clean":"2024-04-23T18:18:30.508653+0000","last_became_active":"2024-04-23T18:16:33.985866+0000","last_became_peered":"2024-04-23T18:16:33.985866+0000","last_unstale":"2024-04-23T18:18:30.508653+0000","last_undegraded":"2024-04-23T18:18:30.508653+0000","last_fullsized":"2024-04-23T18:18:30.508653+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:30:17.675952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":140,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.761189+0000","last_change":"2024-04-23T18:16:32.958466+0000","last_active":"2024-04-23T18:18:30.761189+0000","last_peered":"2024-04-23T18:18:30.761189+0000","last_clean":"2024-04-23T18:18:30.761189+0000","last_became_active":"2024-04-23T18:16:32.958155+0000","last_became_peered":"2024-04-23T18:16:32.958155+0000","last_unstale":"2024-04-23T18:18:30.761189+0000","last_undegraded":"2024-04-23T18:18:30.761189+0000","last_fullsized":"2024-04-23T18:18:30.761189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:15:49.073934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510826+0000","last_change":"2024-04-23T18:16:33.996156+0000","last_active":"2024-04-23T18:18:30.510826+0000","last_peered":"2024-04-23T18:18:30.510826+0000","last_clean":"2024-04-23T18:18:30.510826+0000","last_became_active":"2024-04-23T18:16:33.995711+0000","last_became_peered":"2024-04-23T18:16:33.995711+0000","last_unstale":"2024-04-23T18:18:30.510826+0000","last_undegraded":"2024-04-23T18:18:30.510826+0000","last_fullsized":"2024-04-23T18:18:30.510826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:45.272598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672184+0000","last_change":"2024-04-23T18:17:36.217460+0000","last_active":"2024-04-23T18:18:30.672184+0000","last_peered":"2024-04-23T18:18:30.672184+0000","last_clean":"2024-04-23T18:18:30.672184+0000","last_became_active":"2024-04-23T18:17:36.214196+0000","last_became_peered":"2024-04-23T18:17:36.214196+0000","last_unstale":"2024-04-23T18:18:30.672184+0000","last_undegraded":"2024-04-23T18:18:30.672184+0000","last_fullsized":"2024-04-23T18:18:30.672184+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:34:55.557943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880842+0000","last_change":"2024-04-23T18:17:36.114238+0000","last_active":"2024-04-23T18:18:30.880842+0000","last_peered":"2024-04-23T18:18:30.880842+0000","last_clean":"2024-04-23T18:18:30.880842+0000","last_became_active":"2024-04-23T18:17:36.113810+0000","last_became_peered":"2024-04-23T18:17:36.113810+0000","last_unstale":"2024-04-23T18:18:30.880842+0000","last_undegraded":"2024-04-23T18:18:30.880842+0000","last_fullsized":"2024-04-23T18:18:30.880842+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:53.854596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.614213+0000","last_change":"2024-04-23T18:16:32.966725+0000","last_active":"2024-04-23T18:18:30.614213+0000","last_peered":"2024-04-23T18:18:30.614213+0000","last_clean":"2024-04-23T18:18:30.614213+0000","last_became_active":"2024-04-23T18:16:32.966304+0000","last_became_peered":"2024-04-23T18:16:32.966304+0000","last_unstale":"2024-04-23T18:18:30.614213+0000","last_undegraded":"2024-04-23T18:18:30.614213+0000","last_fullsized":"2024-04-23T18:18:30.614213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:45.191051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510708+0000","last_change":"2024-04-23T18:16:33.994667+0000","last_active":"2024-04-23T18:18:30.510708+0000","last_peered":"2024-04-23T18:18:30.510708+0000","last_clean":"2024-04-23T18:18:30.510708+0000","last_became_active":"2024-04-23T18:16:33.994305+0000","last_became_peered":"2024-04-23T18:16:33.994305+0000","last_unstale":"2024-04-23T18:18:30.510708+0000","last_undegraded":"2024-04-23T18:18:30.510708+0000","last_fullsized":"2024-04-23T18:18:30.510708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:23.807075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'37","reported_seq":298,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:31.190262+0000","last_change":"2024-04-23T18:16:32.957946+0000","last_active":"2024-04-23T18:18:31.190262+0000","last_peered":"2024-04-23T18:18:31.190262+0000","last_clean":"2024-04-23T18:18:31.190262+0000","last_became_active":"2024-04-23T18:16:32.957478+0000","last_became_peered":"2024-04-23T18:16:32.957478+0000","last_unstale":"2024-04-23T18:18:31.190262+0000","last_undegraded":"2024-04-23T18:18:31.190262+0000","last_fullsized":"2024-04-23T18:18:31.190262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:16.850604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66735,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":240,"num_read_kb":185,"num_write":38,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608533+0000","last_change":"2024-04-23T18:16:33.989537+0000","last_active":"2024-04-23T18:18:30.608533+0000","last_peered":"2024-04-23T18:18:30.608533+0000","last_clean":"2024-04-23T18:18:30.608533+0000","last_became_active":"2024-04-23T18:16:33.989180+0000","last_became_peered":"2024-04-23T18:16:33.989180+0000","last_unstale":"2024-04-23T18:18:30.608533+0000","last_undegraded":"2024-04-23T18:18:30.608533+0000","last_fullsized":"2024-04-23T18:18:30.608533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:53:18.700347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.511992+0000","last_change":"2024-04-23T18:16:32.967314+0000","last_active":"2024-04-23T18:18:30.511992+0000","last_peered":"2024-04-23T18:18:30.511992+0000","last_clean":"2024-04-23T18:18:30.511992+0000","last_became_active":"2024-04-23T18:16:32.966861+0000","last_became_peered":"2024-04-23T18:16:32.966861+0000","last_unstale":"2024-04-23T18:18:30.511992+0000","last_undegraded":"2024-04-23T18:18:30.511992+0000","last_fullsized":"2024-04-23T18:18:30.511992+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:00:25.606402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671655+0000","last_change":"2024-04-23T18:16:33.992593+0000","last_active":"2024-04-23T18:18:30.671655+0000","last_peered":"2024-04-23T18:18:30.671655+0000","last_clean":"2024-04-23T18:18:30.671655+0000","last_became_active":"2024-04-23T18:16:33.992228+0000","last_became_peered":"2024-04-23T18:16:33.992228+0000","last_unstale":"2024-04-23T18:18:30.671655+0000","last_undegraded":"2024-04-23T18:18:30.671655+0000","last_fullsized":"2024-04-23T18:18:30.671655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:12:23.836829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.613908+0000","last_change":"2024-04-23T18:16:32.961740+0000","last_active":"2024-04-23T18:18:30.613908+0000","last_peered":"2024-04-23T18:18:30.613908+0000","last_clean":"2024-04-23T18:18:30.613908+0000","last_became_active":"2024-04-23T18:16:32.961331+0000","last_became_peered":"2024-04-23T18:16:32.961331+0000","last_unstale":"2024-04-23T18:18:30.613908+0000","last_undegraded":"2024-04-23T18:18:30.613908+0000","last_fullsized":"2024-04-23T18:18:30.613908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:59:17.665288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671999+0000","last_change":"2024-04-23T18:17:36.111151+0000","last_active":"2024-04-23T18:18:30.671999+0000","last_peered":"2024-04-23T18:18:30.671999+0000","last_clean":"2024-04-23T18:18:30.671999+0000","last_became_active":"2024-04-23T18:17:36.110770+0000","last_became_peered":"2024-04-23T18:17:36.110770+0000","last_unstale":"2024-04-23T18:18:30.671999+0000","last_undegraded":"2024-04-23T18:18:30.671999+0000","last_fullsized":"2024-04-23T18:18:30.671999+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:01:03.456738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:31.133054+0000","last_change":"2024-04-23T18:16:32.954935+0000","last_active":"2024-04-23T18:18:31.133054+0000","last_peered":"2024-04-23T18:18:31.133054+0000","last_clean":"2024-04-23T18:18:31.133054+0000","last_became_active":"2024-04-23T18:16:32.954595+0000","last_became_peered":"2024-04-23T18:16:32.954595+0000","last_unstale":"2024-04-23T18:18:31.133054+0000","last_undegraded":"2024-04-23T18:18:31.133054+0000","last_fullsized":"2024-04-23T18:18:31.133054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:22.954101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671780+0000","last_change":"2024-04-23T18:16:34.000215+0000","last_active":"2024-04-23T18:18:30.671780+0000","last_peered":"2024-04-23T18:18:30.671780+0000","last_clean":"2024-04-23T18:18:30.671780+0000","last_became_active":"2024-04-23T18:16:33.999855+0000","last_became_peered":"2024-04-23T18:16:33.999855+0000","last_unstale":"2024-04-23T18:18:30.671780+0000","last_undegraded":"2024-04-23T18:18:30.671780+0000","last_fullsized":"2024-04-23T18:18:30.671780+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:46:14.686916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671868+0000","last_change":"2024-04-23T18:17:36.127721+0000","last_active":"2024-04-23T18:18:30.671868+0000","last_peered":"2024-04-23T18:18:30.671868+0000","last_clean":"2024-04-23T18:18:30.671868+0000","last_became_active":"2024-04-23T18:17:36.113735+0000","last_became_peered":"2024-04-23T18:17:36.113735+0000","last_unstale":"2024-04-23T18:18:30.671868+0000","last_undegraded":"2024-04-23T18:18:30.671868+0000","last_fullsized":"2024-04-23T18:18:30.671868+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:56:41.395925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508868+0000","last_change":"2024-04-23T18:17:36.214244+0000","last_active":"2024-04-23T18:18:30.508868+0000","last_peered":"2024-04-23T18:18:30.508868+0000","last_clean":"2024-04-23T18:18:30.508868+0000","last_became_active":"2024-04-23T18:17:36.213897+0000","last_became_peered":"2024-04-23T18:17:36.213897+0000","last_unstale":"2024-04-23T18:18:30.508868+0000","last_undegraded":"2024-04-23T18:18:30.508868+0000","last_fullsized":"2024-04-23T18:18:30.508868+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:44:24.935747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608788+0000","last_change":"2024-04-23T18:16:32.959253+0000","last_active":"2024-04-23T18:18:30.608788+0000","last_peered":"2024-04-23T18:18:30.608788+0000","last_clean":"2024-04-23T18:18:30.608788+0000","last_became_active":"2024-04-23T18:16:32.958929+0000","last_became_peered":"2024-04-23T18:16:32.958929+0000","last_unstale":"2024-04-23T18:18:30.608788+0000","last_undegraded":"2024-04-23T18:18:30.608788+0000","last_fullsized":"2024-04-23T18:18:30.608788+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:10:36.317516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672022+0000","last_change":"2024-04-23T18:16:33.979464+0000","last_active":"2024-04-23T18:18:30.672022+0000","last_peered":"2024-04-23T18:18:30.672022+0000","last_clean":"2024-04-23T18:18:30.672022+0000","last_became_active":"2024-04-23T18:16:33.979106+0000","last_became_peered":"2024-04-23T18:16:33.979106+0000","last_unstale":"2024-04-23T18:18:30.672022+0000","last_undegraded":"2024-04-23T18:18:30.672022+0000","last_fullsized":"2024-04-23T18:18:30.672022+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:00.125514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508806+0000","last_change":"2024-04-23T18:16:32.960812+0000","last_active":"2024-04-23T18:18:30.508806+0000","last_peered":"2024-04-23T18:18:30.508806+0000","last_clean":"2024-04-23T18:18:30.508806+0000","last_became_active":"2024-04-23T18:16:32.959807+0000","last_became_peered":"2024-04-23T18:16:32.959807+0000","last_unstale":"2024-04-23T18:18:30.508806+0000","last_undegraded":"2024-04-23T18:18:30.508806+0000","last_fullsized":"2024-04-23T18:18:30.508806+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:45.097443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.617178+0000","last_change":"2024-04-23T18:16:33.992891+0000","last_active":"2024-04-23T18:18:30.617178+0000","last_peered":"2024-04-23T18:18:30.617178+0000","last_clean":"2024-04-23T18:18:30.617178+0000","last_became_active":"2024-04-23T18:16:33.992559+0000","last_became_peered":"2024-04-23T18:16:33.992559+0000","last_unstale":"2024-04-23T18:18:30.617178+0000","last_undegraded":"2024-04-23T18:18:30.617178+0000","last_fullsized":"2024-04-23T18:18:30.617178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:48.615158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671986+0000","last_change":"2024-04-23T18:16:32.957015+0000","last_active":"2024-04-23T18:18:30.671986+0000","last_peered":"2024-04-23T18:18:30.671986+0000","last_clean":"2024-04-23T18:18:30.671986+0000","last_became_active":"2024-04-23T18:16:32.956690+0000","last_became_peered":"2024-04-23T18:16:32.956690+0000","last_unstale":"2024-04-23T18:18:30.671986+0000","last_undegraded":"2024-04-23T18:18:30.671986+0000","last_fullsized":"2024-04-23T18:18:30.671986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:20:42.087521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.617121+0000","last_change":"2024-04-23T18:16:33.998906+0000","last_active":"2024-04-23T18:18:30.617121+0000","last_peered":"2024-04-23T18:18:30.617121+0000","last_clean":"2024-04-23T18:18:30.617121+0000","last_became_active":"2024-04-23T18:16:33.997297+0000","last_became_peered":"2024-04-23T18:16:33.997297+0000","last_unstale":"2024-04-23T18:18:30.617121+0000","last_undegraded":"2024-04-23T18:18:30.617121+0000","last_fullsized":"2024-04-23T18:18:30.617121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:21:06.668830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.614008+0000","last_change":"2024-04-23T18:17:36.113962+0000","last_active":"2024-04-23T18:18:30.614008+0000","last_peered":"2024-04-23T18:18:30.614008+0000","last_clean":"2024-04-23T18:18:30.614008+0000","last_became_active":"2024-04-23T18:17:36.113573+0000","last_became_peered":"2024-04-23T18:17:36.113573+0000","last_unstale":"2024-04-23T18:18:30.614008+0000","last_undegraded":"2024-04-23T18:18:30.614008+0000","last_fullsized":"2024-04-23T18:18:30.614008+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:47:25.408144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'6","reported_seq":141,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671805+0000","last_change":"2024-04-23T18:16:32.961423+0000","last_active":"2024-04-23T18:18:30.671805+0000","last_peered":"2024-04-23T18:18:30.671805+0000","last_clean":"2024-04-23T18:18:30.671805+0000","last_became_active":"2024-04-23T18:16:32.961116+0000","last_became_peered":"2024-04-23T18:16:32.961116+0000","last_unstale":"2024-04-23T18:18:30.671805+0000","last_undegraded":"2024-04-23T18:18:30.671805+0000","last_fullsized":"2024-04-23T18:18:30.671805+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:28.525642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608360+0000","last_change":"2024-04-23T18:16:33.990747+0000","last_active":"2024-04-23T18:18:30.608360+0000","last_peered":"2024-04-23T18:18:30.608360+0000","last_clean":"2024-04-23T18:18:30.608360+0000","last_became_active":"2024-04-23T18:16:33.989972+0000","last_became_peered":"2024-04-23T18:16:33.989972+0000","last_unstale":"2024-04-23T18:18:30.608360+0000","last_undegraded":"2024-04-23T18:18:30.608360+0000","last_fullsized":"2024-04-23T18:18:30.608360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:41:04.667446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671772+0000","last_change":"2024-04-23T18:16:32.952821+0000","last_active":"2024-04-23T18:18:30.671772+0000","last_peered":"2024-04-23T18:18:30.671772+0000","last_clean":"2024-04-23T18:18:30.671772+0000","last_became_active":"2024-04-23T18:16:32.952470+0000","last_became_peered":"2024-04-23T18:16:32.952470+0000","last_unstale":"2024-04-23T18:18:30.671772+0000","last_undegraded":"2024-04-23T18:18:30.671772+0000","last_fullsized":"2024-04-23T18:18:30.671772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:30:19.392670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671938+0000","last_change":"2024-04-23T18:15:25.867370+0000","last_active":"2024-04-23T18:18:30.671938+0000","last_peered":"2024-04-23T18:18:30.671938+0000","last_clean":"2024-04-23T18:18:30.671938+0000","last_became_active":"2024-04-23T18:15:25.866815+0000","last_became_peered":"2024-04-23T18:15:25.866815+0000","last_unstale":"2024-04-23T18:18:30.671938+0000","last_undegraded":"2024-04-23T18:18:30.671938+0000","last_fullsized":"2024-04-23T18:18:30.671938+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:12:02.913350+0000","last_clean_scrub_stamp":"2024-04-23T18:12:02.913350+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:42.482422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671577+0000","last_change":"2024-04-23T18:16:33.992988+0000","last_active":"2024-04-23T18:18:30.671577+0000","last_peered":"2024-04-23T18:18:30.671577+0000","last_clean":"2024-04-23T18:18:30.671577+0000","last_became_active":"2024-04-23T18:16:33.992650+0000","last_became_peered":"2024-04-23T18:16:33.992650+0000","last_unstale":"2024-04-23T18:18:30.671577+0000","last_undegraded":"2024-04-23T18:18:30.671577+0000","last_fullsized":"2024-04-23T18:18:30.671577+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:54.054708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672214+0000","last_change":"2024-04-23T18:16:32.957318+0000","last_active":"2024-04-23T18:18:30.672214+0000","last_peered":"2024-04-23T18:18:30.672214+0000","last_clean":"2024-04-23T18:18:30.672214+0000","last_became_active":"2024-04-23T18:16:32.956973+0000","last_became_peered":"2024-04-23T18:16:32.956973+0000","last_unstale":"2024-04-23T18:18:30.672214+0000","last_undegraded":"2024-04-23T18:18:30.672214+0000","last_fullsized":"2024-04-23T18:18:30.672214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:10:35.684825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672061+0000","last_change":"2024-04-23T18:17:36.117253+0000","last_active":"2024-04-23T18:18:30.672061+0000","last_peered":"2024-04-23T18:18:30.672061+0000","last_clean":"2024-04-23T18:18:30.672061+0000","last_became_active":"2024-04-23T18:17:36.111210+0000","last_became_peered":"2024-04-23T18:17:36.111210+0000","last_unstale":"2024-04-23T18:18:30.672061+0000","last_undegraded":"2024-04-23T18:18:30.672061+0000","last_fullsized":"2024-04-23T18:18:30.672061+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:31.599001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672229+0000","last_change":"2024-04-23T18:16:33.996345+0000","last_active":"2024-04-23T18:18:30.672229+0000","last_peered":"2024-04-23T18:18:30.672229+0000","last_clean":"2024-04-23T18:18:30.672229+0000","last_became_active":"2024-04-23T18:16:33.993802+0000","last_became_peered":"2024-04-23T18:16:33.993802+0000","last_unstale":"2024-04-23T18:18:30.672229+0000","last_undegraded":"2024-04-23T18:18:30.672229+0000","last_fullsized":"2024-04-23T18:18:30.672229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:32.410330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'44","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.722525+0000","last_change":"2024-04-23T18:16:32.957953+0000","last_active":"2024-04-23T18:18:30.722525+0000","last_peered":"2024-04-23T18:18:30.722525+0000","last_clean":"2024-04-23T18:18:30.722525+0000","last_became_active":"2024-04-23T18:16:32.957616+0000","last_became_peered":"2024-04-23T18:16:32.957616+0000","last_unstale":"2024-04-23T18:18:30.722525+0000","last_undegraded":"2024-04-23T18:18:30.722525+0000","last_fullsized":"2024-04-23T18:18:30.722525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:15.485373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.609061+0000","last_change":"2024-04-23T18:16:33.985685+0000","last_active":"2024-04-23T18:18:30.609061+0000","last_peered":"2024-04-23T18:18:30.609061+0000","last_clean":"2024-04-23T18:18:30.609061+0000","last_became_active":"2024-04-23T18:16:33.984631+0000","last_became_peered":"2024-04-23T18:16:33.984631+0000","last_unstale":"2024-04-23T18:18:30.609061+0000","last_undegraded":"2024-04-23T18:18:30.609061+0000","last_fullsized":"2024-04-23T18:18:30.609061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:45.893285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672398+0000","last_change":"2024-04-23T18:16:32.954281+0000","last_active":"2024-04-23T18:18:30.672398+0000","last_peered":"2024-04-23T18:18:30.672398+0000","last_clean":"2024-04-23T18:18:30.672398+0000","last_became_active":"2024-04-23T18:16:32.953988+0000","last_became_peered":"2024-04-23T18:16:32.953988+0000","last_unstale":"2024-04-23T18:18:30.672398+0000","last_undegraded":"2024-04-23T18:18:30.672398+0000","last_fullsized":"2024-04-23T18:18:30.672398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:09.049861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880698+0000","last_change":"2024-04-23T18:16:33.994860+0000","last_active":"2024-04-23T18:18:30.880698+0000","last_peered":"2024-04-23T18:18:30.880698+0000","last_clean":"2024-04-23T18:18:30.880698+0000","last_became_active":"2024-04-23T18:16:33.993438+0000","last_became_peered":"2024-04-23T18:16:33.993438+0000","last_unstale":"2024-04-23T18:18:30.880698+0000","last_undegraded":"2024-04-23T18:18:30.880698+0000","last_fullsized":"2024-04-23T18:18:30.880698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:55.565873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510367+0000","last_change":"2024-04-23T18:16:32.959251+0000","last_active":"2024-04-23T18:18:30.510367+0000","last_peered":"2024-04-23T18:18:30.510367+0000","last_clean":"2024-04-23T18:18:30.510367+0000","last_became_active":"2024-04-23T18:16:32.958937+0000","last_became_peered":"2024-04-23T18:16:32.958937+0000","last_unstale":"2024-04-23T18:18:30.510367+0000","last_undegraded":"2024-04-23T18:18:30.510367+0000","last_fullsized":"2024-04-23T18:18:30.510367+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.126634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608249+0000","last_change":"2024-04-23T18:16:33.995480+0000","last_active":"2024-04-23T18:18:30.608249+0000","last_peered":"2024-04-23T18:18:30.608249+0000","last_clean":"2024-04-23T18:18:30.608249+0000","last_became_active":"2024-04-23T18:16:33.994361+0000","last_became_peered":"2024-04-23T18:16:33.994361+0000","last_unstale":"2024-04-23T18:18:30.608249+0000","last_undegraded":"2024-04-23T18:18:30.608249+0000","last_fullsized":"2024-04-23T18:18:30.608249+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:59:35.240562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608282+0000","last_change":"2024-04-23T18:16:32.967787+0000","last_active":"2024-04-23T18:18:30.608282+0000","last_peered":"2024-04-23T18:18:30.608282+0000","last_clean":"2024-04-23T18:18:30.608282+0000","last_became_active":"2024-04-23T18:16:32.967463+0000","last_became_peered":"2024-04-23T18:16:32.967463+0000","last_unstale":"2024-04-23T18:18:30.608282+0000","last_undegraded":"2024-04-23T18:18:30.608282+0000","last_fullsized":"2024-04-23T18:18:30.608282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:58.743682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.671931+0000","last_change":"2024-04-23T18:16:33.980558+0000","last_active":"2024-04-23T18:18:30.671931+0000","last_peered":"2024-04-23T18:18:30.671931+0000","last_clean":"2024-04-23T18:18:30.671931+0000","last_became_active":"2024-04-23T18:16:33.980146+0000","last_became_peered":"2024-04-23T18:16:33.980146+0000","last_unstale":"2024-04-23T18:18:30.671931+0000","last_undegraded":"2024-04-23T18:18:30.671931+0000","last_fullsized":"2024-04-23T18:18:30.671931+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:29:58.001833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512195+0000","last_change":"2024-04-23T18:16:32.962139+0000","last_active":"2024-04-23T18:18:30.512195+0000","last_peered":"2024-04-23T18:18:30.512195+0000","last_clean":"2024-04-23T18:18:30.512195+0000","last_became_active":"2024-04-23T18:16:32.961785+0000","last_became_peered":"2024-04-23T18:16:32.961785+0000","last_unstale":"2024-04-23T18:18:30.512195+0000","last_undegraded":"2024-04-23T18:18:30.512195+0000","last_fullsized":"2024-04-23T18:18:30.512195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:03:09.619414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.508926+0000","last_change":"2024-04-23T18:17:36.214255+0000","last_active":"2024-04-23T18:18:30.508926+0000","last_peered":"2024-04-23T18:18:30.508926+0000","last_clean":"2024-04-23T18:18:30.508926+0000","last_became_active":"2024-04-23T18:17:36.213832+0000","last_became_peered":"2024-04-23T18:17:36.213832+0000","last_unstale":"2024-04-23T18:18:30.508926+0000","last_undegraded":"2024-04-23T18:18:30.508926+0000","last_fullsized":"2024-04-23T18:18:30.508926+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:59:35.497710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608350+0000","last_change":"2024-04-23T18:16:32.956965+0000","last_active":"2024-04-23T18:18:30.608350+0000","last_peered":"2024-04-23T18:18:30.608350+0000","last_clean":"2024-04-23T18:18:30.608350+0000","last_became_active":"2024-04-23T18:16:32.956672+0000","last_became_peered":"2024-04-23T18:16:32.956672+0000","last_unstale":"2024-04-23T18:18:30.608350+0000","last_undegraded":"2024-04-23T18:18:30.608350+0000","last_fullsized":"2024-04-23T18:18:30.608350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:00:44.470809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672198+0000","last_change":"2024-04-23T18:16:34.002711+0000","last_active":"2024-04-23T18:18:30.672198+0000","last_peered":"2024-04-23T18:18:30.672198+0000","last_clean":"2024-04-23T18:18:30.672198+0000","last_became_active":"2024-04-23T18:16:34.002365+0000","last_became_peered":"2024-04-23T18:16:34.002365+0000","last_unstale":"2024-04-23T18:18:30.672198+0000","last_undegraded":"2024-04-23T18:18:30.672198+0000","last_fullsized":"2024-04-23T18:18:30.672198+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:52:11.982735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512283+0000","last_change":"2024-04-23T18:16:32.965121+0000","last_active":"2024-04-23T18:18:30.512283+0000","last_peered":"2024-04-23T18:18:30.512283+0000","last_clean":"2024-04-23T18:18:30.512283+0000","last_became_active":"2024-04-23T18:16:32.964732+0000","last_became_peered":"2024-04-23T18:16:32.964732+0000","last_unstale":"2024-04-23T18:18:30.512283+0000","last_undegraded":"2024-04-23T18:18:30.512283+0000","last_fullsized":"2024-04-23T18:18:30.512283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:19:34.690245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672288+0000","last_change":"2024-04-23T18:16:33.996357+0000","last_active":"2024-04-23T18:18:30.672288+0000","last_peered":"2024-04-23T18:18:30.672288+0000","last_clean":"2024-04-23T18:18:30.672288+0000","last_became_active":"2024-04-23T18:16:33.996017+0000","last_became_peered":"2024-04-23T18:16:33.996017+0000","last_unstale":"2024-04-23T18:18:30.672288+0000","last_undegraded":"2024-04-23T18:18:30.672288+0000","last_fullsized":"2024-04-23T18:18:30.672288+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:20:57.321228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.720073+0000","last_change":"2024-04-23T18:16:32.961705+0000","last_active":"2024-04-23T18:18:30.720073+0000","last_peered":"2024-04-23T18:18:30.720073+0000","last_clean":"2024-04-23T18:18:30.720073+0000","last_became_active":"2024-04-23T18:16:32.961342+0000","last_became_peered":"2024-04-23T18:16:32.961342+0000","last_unstale":"2024-04-23T18:18:30.720073+0000","last_undegraded":"2024-04-23T18:18:30.720073+0000","last_fullsized":"2024-04-23T18:18:30.720073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:51.340910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672001+0000","last_change":"2024-04-23T18:16:33.992944+0000","last_active":"2024-04-23T18:18:30.672001+0000","last_peered":"2024-04-23T18:18:30.672001+0000","last_clean":"2024-04-23T18:18:30.672001+0000","last_became_active":"2024-04-23T18:16:33.992579+0000","last_became_peered":"2024-04-23T18:16:33.992579+0000","last_unstale":"2024-04-23T18:18:30.672001+0000","last_undegraded":"2024-04-23T18:18:30.672001+0000","last_fullsized":"2024-04-23T18:18:30.672001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:13:37.829588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672489+0000","last_change":"2024-04-23T18:16:32.955192+0000","last_active":"2024-04-23T18:18:30.672489+0000","last_peered":"2024-04-23T18:18:30.672489+0000","last_clean":"2024-04-23T18:18:30.672489+0000","last_became_active":"2024-04-23T18:16:32.954860+0000","last_became_peered":"2024-04-23T18:16:32.954860+0000","last_unstale":"2024-04-23T18:18:30.672489+0000","last_undegraded":"2024-04-23T18:18:30.672489+0000","last_fullsized":"2024-04-23T18:18:30.672489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:31.935783+0000","last_clean_scrub_stamp":"2024-04-23T18:16:31.935783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:20:20.796060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672367+0000","last_change":"2024-04-23T18:16:33.986211+0000","last_active":"2024-04-23T18:18:30.672367+0000","last_peered":"2024-04-23T18:18:30.672367+0000","last_clean":"2024-04-23T18:18:30.672367+0000","last_became_active":"2024-04-23T18:16:33.985679+0000","last_became_peered":"2024-04-23T18:16:33.985679+0000","last_unstale":"2024-04-23T18:18:30.672367+0000","last_undegraded":"2024-04-23T18:18:30.672367+0000","last_fullsized":"2024-04-23T18:18:30.672367+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:03:38.797864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672089+0000","last_change":"2024-04-23T18:16:33.979462+0000","last_active":"2024-04-23T18:18:30.672089+0000","last_peered":"2024-04-23T18:18:30.672089+0000","last_clean":"2024-04-23T18:18:30.672089+0000","last_became_active":"2024-04-23T18:16:33.979114+0000","last_became_peered":"2024-04-23T18:16:33.979114+0000","last_unstale":"2024-04-23T18:18:30.672089+0000","last_undegraded":"2024-04-23T18:18:30.672089+0000","last_fullsized":"2024-04-23T18:18:30.672089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:16:49.708240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512027+0000","last_change":"2024-04-23T18:16:33.979255+0000","last_active":"2024-04-23T18:18:30.512027+0000","last_peered":"2024-04-23T18:18:30.512027+0000","last_clean":"2024-04-23T18:18:30.512027+0000","last_became_active":"2024-04-23T18:16:33.978934+0000","last_became_peered":"2024-04-23T18:16:33.978934+0000","last_unstale":"2024-04-23T18:18:30.512027+0000","last_undegraded":"2024-04-23T18:18:30.512027+0000","last_fullsized":"2024-04-23T18:18:30.512027+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:10.104564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.511957+0000","last_change":"2024-04-23T18:16:33.981877+0000","last_active":"2024-04-23T18:18:30.511957+0000","last_peered":"2024-04-23T18:18:30.511957+0000","last_clean":"2024-04-23T18:18:30.511957+0000","last_became_active":"2024-04-23T18:16:33.981536+0000","last_became_peered":"2024-04-23T18:16:33.981536+0000","last_unstale":"2024-04-23T18:18:30.511957+0000","last_undegraded":"2024-04-23T18:18:30.511957+0000","last_fullsized":"2024-04-23T18:18:30.511957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:00:15.056779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.880509+0000","last_change":"2024-04-23T18:16:33.999090+0000","last_active":"2024-04-23T18:18:30.880509+0000","last_peered":"2024-04-23T18:18:30.880509+0000","last_clean":"2024-04-23T18:18:30.880509+0000","last_became_active":"2024-04-23T18:16:33.998513+0000","last_became_peered":"2024-04-23T18:16:33.998513+0000","last_unstale":"2024-04-23T18:18:30.880509+0000","last_undegraded":"2024-04-23T18:18:30.880509+0000","last_fullsized":"2024-04-23T18:18:30.880509+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:01:14.115406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672082+0000","last_change":"2024-04-23T18:16:33.993375+0000","last_active":"2024-04-23T18:18:30.672082+0000","last_peered":"2024-04-23T18:18:30.672082+0000","last_clean":"2024-04-23T18:18:30.672082+0000","last_became_active":"2024-04-23T18:16:33.991841+0000","last_became_peered":"2024-04-23T18:16:33.991841+0000","last_unstale":"2024-04-23T18:18:30.672082+0000","last_undegraded":"2024-04-23T18:18:30.672082+0000","last_fullsized":"2024-04-23T18:18:30.672082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:00:12.777495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672189+0000","last_change":"2024-04-23T18:16:33.983621+0000","last_active":"2024-04-23T18:18:30.672189+0000","last_peered":"2024-04-23T18:18:30.672189+0000","last_clean":"2024-04-23T18:18:30.672189+0000","last_became_active":"2024-04-23T18:16:33.983356+0000","last_became_peered":"2024-04-23T18:16:33.983356+0000","last_unstale":"2024-04-23T18:18:30.672189+0000","last_undegraded":"2024-04-23T18:18:30.672189+0000","last_fullsized":"2024-04-23T18:18:30.672189+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:27:19.391281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.512250+0000","last_change":"2024-04-23T18:16:33.987952+0000","last_active":"2024-04-23T18:18:30.512250+0000","last_peered":"2024-04-23T18:18:30.512250+0000","last_clean":"2024-04-23T18:18:30.512250+0000","last_became_active":"2024-04-23T18:16:33.987334+0000","last_became_peered":"2024-04-23T18:16:33.987334+0000","last_unstale":"2024-04-23T18:18:30.512250+0000","last_undegraded":"2024-04-23T18:18:30.512250+0000","last_fullsized":"2024-04-23T18:18:30.512250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:57:41.902779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.609205+0000","last_change":"2024-04-23T18:16:33.983914+0000","last_active":"2024-04-23T18:18:30.609205+0000","last_peered":"2024-04-23T18:18:30.609205+0000","last_clean":"2024-04-23T18:18:30.609205+0000","last_became_active":"2024-04-23T18:16:33.983444+0000","last_became_peered":"2024-04-23T18:16:33.983444+0000","last_unstale":"2024-04-23T18:18:30.609205+0000","last_undegraded":"2024-04-23T18:18:30.609205+0000","last_fullsized":"2024-04-23T18:18:30.609205+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:10:17.630049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.613877+0000","last_change":"2024-04-23T18:16:33.992496+0000","last_active":"2024-04-23T18:18:30.613877+0000","last_peered":"2024-04-23T18:18:30.613877+0000","last_clean":"2024-04-23T18:18:30.613877+0000","last_became_active":"2024-04-23T18:16:33.992155+0000","last_became_peered":"2024-04-23T18:16:33.992155+0000","last_unstale":"2024-04-23T18:18:30.613877+0000","last_undegraded":"2024-04-23T18:18:30.613877+0000","last_fullsized":"2024-04-23T18:18:30.613877+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:48.835992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.608101+0000","last_change":"2024-04-23T18:16:33.984409+0000","last_active":"2024-04-23T18:18:30.608101+0000","last_peered":"2024-04-23T18:18:30.608101+0000","last_clean":"2024-04-23T18:18:30.608101+0000","last_became_active":"2024-04-23T18:16:33.984073+0000","last_became_peered":"2024-04-23T18:16:33.984073+0000","last_unstale":"2024-04-23T18:18:30.608101+0000","last_undegraded":"2024-04-23T18:18:30.608101+0000","last_fullsized":"2024-04-23T18:18:30.608101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:45:37.796783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.510525+0000","last_change":"2024-04-23T18:16:33.986376+0000","last_active":"2024-04-23T18:18:30.510525+0000","last_peered":"2024-04-23T18:18:30.510525+0000","last_clean":"2024-04-23T18:18:30.510525+0000","last_became_active":"2024-04-23T18:16:33.986012+0000","last_became_peered":"2024-04-23T18:16:33.986012+0000","last_unstale":"2024-04-23T18:18:30.510525+0000","last_undegraded":"2024-04-23T18:18:30.510525+0000","last_fullsized":"2024-04-23T18:18:30.510525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:07:26.940215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672288+0000","last_change":"2024-04-23T18:16:33.999795+0000","last_active":"2024-04-23T18:18:30.672288+0000","last_peered":"2024-04-23T18:18:30.672288+0000","last_clean":"2024-04-23T18:18:30.672288+0000","last_became_active":"2024-04-23T18:16:33.997034+0000","last_became_peered":"2024-04-23T18:16:33.997034+0000","last_unstale":"2024-04-23T18:18:30.672288+0000","last_undegraded":"2024-04-23T18:18:30.672288+0000","last_fullsized":"2024-04-23T18:18:30.672288+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:56:23.983846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T18:18:30.672329+0000","last_change":"2024-04-23T18:16:33.985495+0000","last_active":"2024-04-23T18:18:30.672329+0000","last_peered":"2024-04-23T18:18:30.672329+0000","last_clean":"2024-04-23T18:18:30.672329+0000","last_became_active":"2024-04-23T18:16:33.985178+0000","last_became_peered":"2024-04-23T18:16:33.985178+0000","last_unstale":"2024-04-23T18:18:30.672329+0000","last_undegraded":"2024-04-23T18:18:30.672329+0000","last_fullsized":"2024-04-23T18:18:30.672329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T18:16:32.941292+0000","last_clean_scrub_stamp":"2024-04-23T18:16:32.941292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:22:25.325756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":597706,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":804,"num_read_kb":515,"num_write":292,"num_write_kb":1442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1147,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1793118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":231,"ondisk_log_size":231,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318307,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":477387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579945,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32964,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712188,"statfs":{"total":95995035648,"available":95961280512,"internally_reserved":0,"allocated":1830912,"data_stored":1268691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776174,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36708,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708444,"statfs":{"total":95995035648,"available":95957446656,"internally_reserved":0,"allocated":1400832,"data_stored":870705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037810,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1388544,"data_stored":870491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32744,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712408,"statfs":{"total":95995035648,"available":95961505792,"internally_reserved":0,"allocated":1601536,"data_stored":1071493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":476878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":544951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40472,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704680,"statfs":{"total":95995035648,"available":95953592320,"internally_reserved":0,"allocated":999424,"data_stored":469193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1081344,"data_stored":544134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41000,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704152,"statfs":{"total":95995035648,"available":95953051648,"internally_reserved":0,"allocated":1540096,"data_stored":1002005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":999424,"data_stored":478937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28724,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716428,"statfs":{"total":95995035648,"available":95965622272,"internally_reserved":0,"allocated":1748992,"data_stored":1201155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":67463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":201535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T18:18:36.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:18:36.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.653+0000 7f7de9243640 1 -- 172.21.15.50:0/2241564204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4102bf0 msgr2=0x7f7de4106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.653+0000 7f7de9243640 1 --2- 172.21.15.50:0/2241564204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4102bf0 0x7f7de4106ba0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009900 tx=0x7f7dcc02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:36.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.653+0000 7f7de9243640 1 -- 172.21.15.50:0/2241564204 shutdown_connections 2024-04-23T18:18:36.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.653+0000 7f7de9243640 1 --2- 172.21.15.50:0/2241564204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7de4107280 0x7f7de410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.653+0000 7f7de9243640 1 --2- 172.21.15.50:0/2241564204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4102bf0 0x7f7de4106ba0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.653+0000 7f7de9243640 1 --2- 172.21.15.50:0/2241564204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7de4102240 0x7f7de4102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.653+0000 7f7de9243640 1 -- 172.21.15.50:0/2241564204 >> 172.21.15.50:0/2241564204 conn(0x7f7de40790e0 msgr2=0x7f7de40794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:36.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.653+0000 7f7de9243640 1 -- 172.21.15.50:0/2241564204 shutdown_connections 2024-04-23T18:18:36.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.653+0000 7f7de9243640 1 -- 172.21.15.50:0/2241564204 wait complete. 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de9243640 1 Processor -- start 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de9243640 1 -- start start 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de9243640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4102240 0x7f7de41a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de9243640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7de4107280 0x7f7de41a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de9243640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7de419a7b0 0x7f7de419ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de9243640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7de41121f0 con 0x7f7de4102240 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de9243640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7de4112070 con 0x7f7de419a7b0 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de9243640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7de4112370 con 0x7f7de4107280 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4102240 0x7f7de41a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4102240 0x7f7de41a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49294/0 (socket says 172.21.15.50:49294) 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de2d9d640 1 -- 172.21.15.50:0/156900633 learned_addr learned my addr 172.21.15.50:0/156900633 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de359e640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7de419a7b0 0x7f7de419ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7ddbfff640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7de4107280 0x7f7de41a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de2d9d640 1 -- 172.21.15.50:0/156900633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7de4107280 msgr2=0x7f7de41a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de2d9d640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7de4107280 0x7f7de41a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de2d9d640 1 -- 172.21.15.50:0/156900633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7de419a7b0 msgr2=0x7f7de419ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de2d9d640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7de419a7b0 0x7f7de419ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de2d9d640 1 -- 172.21.15.50:0/156900633 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de419b340 con 0x7f7de4102240 2024-04-23T18:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.654+0000 7f7de2d9d640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4102240 0x7f7de41a0640 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400c890 tx=0x7f7dd400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.655+0000 7f7de0d99640 1 -- 172.21.15.50:0/156900633 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4004220 con 0x7f7de4102240 2024-04-23T18:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.655+0000 7f7de9243640 1 -- 172.21.15.50:0/156900633 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de419b630 con 0x7f7de4102240 2024-04-23T18:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.655+0000 7f7de0d99640 1 -- 172.21.15.50:0/156900633 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dd40043c0 con 0x7f7de4102240 2024-04-23T18:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.655+0000 7f7de0d99640 1 -- 172.21.15.50:0/156900633 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4004a00 con 0x7f7de4102240 2024-04-23T18:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.655+0000 7f7de9243640 1 -- 172.21.15.50:0/156900633 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de41a77e0 con 0x7f7de4102240 2024-04-23T18:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.656+0000 7f7de9243640 1 -- 172.21.15.50:0/156900633 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7de4103680 con 0x7f7de4102240 2024-04-23T18:18:36.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.657+0000 7f7de0d99640 1 -- 172.21.15.50:0/156900633 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f7dd40079b0 con 0x7f7de4102240 2024-04-23T18:18:36.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.658+0000 7f7de0d99640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7dd00a3db0 0x7f7dd00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:36.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.658+0000 7f7de0d99640 1 -- 172.21.15.50:0/156900633 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f7dd40d0a40 con 0x7f7de4102240 2024-04-23T18:18:36.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.658+0000 7f7ddbfff640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7dd00a3db0 0x7f7dd00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:36.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.658+0000 7f7ddbfff640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7dd00a3db0 0x7f7dd00a6270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc02aa00 tx=0x7f7dcc005a20 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:36.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.661+0000 7f7de0d99640 1 -- 172.21.15.50:0/156900633 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7de419b630 con 0x7f7de4102240 2024-04-23T18:18:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.886+0000 7f7de9243640 1 -- 172.21.15.50:0/156900633 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7de40691c0 con 0x7f7de4102240 2024-04-23T18:18:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.887+0000 7f7de0d99640 1 -- 172.21.15.50:0/156900633 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11875 (secure 0 0 0) 0x7f7dd4014030 con 0x7f7de4102240 2024-04-23T18:18:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:36.880 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:18:28.904281+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:18:36.880 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 21 2024-04-23T18:18:36.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.888+0000 7f7de9243640 1 -- 172.21.15.50:0/156900633 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7dd00a3db0 msgr2=0x7f7dd00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.889+0000 7f7de9243640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7dd00a3db0 0x7f7dd00a6270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc02aa00 tx=0x7f7dcc005a20 comp rx=0 tx=0).stop 2024-04-23T18:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.889+0000 7f7de9243640 1 -- 172.21.15.50:0/156900633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4102240 msgr2=0x7f7de41a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.889+0000 7f7de9243640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4102240 0x7f7de41a0640 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400c890 tx=0x7f7dd400cd50 comp rx=0 tx=0).stop 2024-04-23T18:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.889+0000 7f7de9243640 1 -- 172.21.15.50:0/156900633 shutdown_connections 2024-04-23T18:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.889+0000 7f7de9243640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7dd00a3db0 0x7f7dd00a6270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.889+0000 7f7de9243640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7de419a7b0 0x7f7de419ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.889+0000 7f7de9243640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7de4107280 0x7f7de41a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.889+0000 7f7de9243640 1 --2- 172.21.15.50:0/156900633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4102240 0x7f7de41a0640 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.889+0000 7f7de9243640 1 -- 172.21.15.50:0/156900633 >> 172.21.15.50:0/156900633 conn(0x7f7de40790e0 msgr2=0x7f7de41003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.889+0000 7f7de9243640 1 -- 172.21.15.50:0/156900633 shutdown_connections 2024-04-23T18:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:36.889+0000 7f7de9243640 1 -- 172.21.15.50:0/156900633 wait complete. 2024-04-23T18:18:36.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:18:36.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.002+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2366873885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c50107270 msgr2=0x7f2c5010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.002+0000 7f2c57d4b640 1 --2- 172.21.15.50:0/2366873885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c50107270 0x7f2c5010ee40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2c48009ca0 tx=0x7f2c4802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.002+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2366873885 shutdown_connections 2024-04-23T18:18:36.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.002+0000 7f2c57d4b640 1 --2- 172.21.15.50:0/2366873885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c50107270 0x7f2c5010ee40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.002+0000 7f2c57d4b640 1 --2- 172.21.15.50:0/2366873885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c5006b290 0x7f2c50106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.002+0000 7f2c57d4b640 1 --2- 172.21.15.50:0/2366873885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c5006a8e0 0x7f2c5006acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.002+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2366873885 >> 172.21.15.50:0/2366873885 conn(0x7f2c50077f20 msgr2=0x7f2c50078330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:36.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.002+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2366873885 shutdown_connections 2024-04-23T18:18:36.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.002+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2366873885 wait complete. 2024-04-23T18:18:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c57d4b640 1 Processor -- start 2024-04-23T18:18:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c57d4b640 1 -- start start 2024-04-23T18:18:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c57d4b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c5006a8e0 0x7f2c50100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c57d4b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c50107270 0x7f2c501013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c57d4b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c50101900 0x7f2c501a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c57d4b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2c501121e0 con 0x7f2c50107270 2024-04-23T18:18:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c57d4b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2c50112060 con 0x7f2c50101900 2024-04-23T18:18:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c57d4b640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2c50112360 con 0x7f2c5006a8e0 2024-04-23T18:18:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c552e6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c50107270 0x7f2c501013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c552e6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c50107270 0x7f2c501013c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49312/0 (socket says 172.21.15.50:49312) 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c552e6640 1 -- 172.21.15.50:0/2059918676 learned_addr learned my addr 172.21.15.50:0/2059918676 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c562e8640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c50101900 0x7f2c501a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c55ae7640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c5006a8e0 0x7f2c50100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c552e6640 1 -- 172.21.15.50:0/2059918676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c5006a8e0 msgr2=0x7f2c50100e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c552e6640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c5006a8e0 0x7f2c50100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c552e6640 1 -- 172.21.15.50:0/2059918676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c50101900 msgr2=0x7f2c501a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c552e6640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c50101900 0x7f2c501a68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c552e6640 1 -- 172.21.15.50:0/2059918676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c501a6fd0 con 0x7f2c50107270 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.003+0000 7f2c562e8640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c50101900 0x7f2c501a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.004+0000 7f2c552e6640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c50107270 0x7f2c501013c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000d770 tx=0x7f2c4000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.004+0000 7f2c46ffd640 1 -- 172.21.15.50:0/2059918676 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40017070 con 0x7f2c50107270 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.004+0000 7f2c46ffd640 1 -- 172.21.15.50:0/2059918676 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c40009c30 con 0x7f2c50107270 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.004+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2059918676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c501a72c0 con 0x7f2c50107270 2024-04-23T18:18:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.004+0000 7f2c46ffd640 1 -- 172.21.15.50:0/2059918676 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4000f620 con 0x7f2c50107270 2024-04-23T18:18:37.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.004+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2059918676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c501a7860 con 0x7f2c50107270 2024-04-23T18:18:37.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.005+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2059918676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c500691c0 con 0x7f2c50107270 2024-04-23T18:18:37.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.006+0000 7f2c46ffd640 1 -- 172.21.15.50:0/2059918676 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f2c4001a070 con 0x7f2c50107270 2024-04-23T18:18:37.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.006+0000 7f2c46ffd640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2c3c0a3e60 0x7f2c3c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.007+0000 7f2c55ae7640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2c3c0a3e60 0x7f2c3c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.007+0000 7f2c46ffd640 1 -- 172.21.15.50:0/2059918676 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f2c400d03f0 con 0x7f2c50107270 2024-04-23T18:18:37.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.007+0000 7f2c55ae7640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2c3c0a3e60 0x7f2c3c0a6320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c005fd0 tx=0x7f2c4c005850 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:37.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.010+0000 7f2c46ffd640 1 -- 172.21.15.50:0/2059918676 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c4009c9b0 con 0x7f2c50107270 2024-04-23T18:18:37.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.199+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2059918676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c5010a500 con 0x7f2c50107270 2024-04-23T18:18:37.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.201+0000 7f2c46ffd640 1 -- 172.21.15.50:0/2059918676 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23038 (secure 0 0 0) 0x7f2c40096e20 con 0x7f2c50107270 2024-04-23T18:18:37.194 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:37.194 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":90,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:18:30.504176+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/1479623929":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/3248139242":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6800/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/2484340243":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/444338023":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:18:37.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:37 smithi050 ceph-mon[29187]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:18:37.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:37 smithi050 ceph-mon[29187]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:18:37.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:37 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/156900633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.203+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2059918676 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2c3c0a3e60 msgr2=0x7f2c3c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.203+0000 7f2c57d4b640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2c3c0a3e60 0x7f2c3c0a6320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c005fd0 tx=0x7f2c4c005850 comp rx=0 tx=0).stop 2024-04-23T18:18:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.203+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2059918676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c50107270 msgr2=0x7f2c501013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.203+0000 7f2c57d4b640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c50107270 0x7f2c501013c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000d770 tx=0x7f2c4000dc30 comp rx=0 tx=0).stop 2024-04-23T18:18:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.203+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2059918676 shutdown_connections 2024-04-23T18:18:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.203+0000 7f2c57d4b640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2c3c0a3e60 0x7f2c3c0a6320 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.203+0000 7f2c57d4b640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c50101900 0x7f2c501a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.203+0000 7f2c57d4b640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c50107270 0x7f2c501013c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.203+0000 7f2c57d4b640 1 --2- 172.21.15.50:0/2059918676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c5006a8e0 0x7f2c50100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.203+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2059918676 >> 172.21.15.50:0/2059918676 conn(0x7f2c50077f20 msgr2=0x7f2c500feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.203+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2059918676 shutdown_connections 2024-04-23T18:18:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.203+0000 7f2c57d4b640 1 -- 172.21.15.50:0/2059918676 wait complete. 2024-04-23T18:18:37.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:12:02.493647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:16:31.679148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:16:32.269478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:18:37.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:18:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:37 smithi177 ceph-mon[32308]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:18:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:37 smithi177 ceph-mon[32308]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:18:37.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:37 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/156900633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.320+0000 7feedeb19640 1 -- 172.21.15.50:0/415483023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feed8107240 msgr2=0x7feed810ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.320+0000 7feedeb19640 1 --2- 172.21.15.50:0/415483023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feed8107240 0x7feed810ee10 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7feed0009ca0 tx=0x7feed002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.320+0000 7feedeb19640 1 -- 172.21.15.50:0/415483023 shutdown_connections 2024-04-23T18:18:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.320+0000 7feedeb19640 1 --2- 172.21.15.50:0/415483023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feed8107240 0x7feed810ee10 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.320+0000 7feedeb19640 1 --2- 172.21.15.50:0/415483023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feed8102b20 0x7feed8106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.320+0000 7feedeb19640 1 --2- 172.21.15.50:0/415483023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feed8102170 0x7feed8102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.320+0000 7feedeb19640 1 -- 172.21.15.50:0/415483023 >> 172.21.15.50:0/415483023 conn(0x7feed80fd9d0 msgr2=0x7feed80ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.320+0000 7feedeb19640 1 -- 172.21.15.50:0/415483023 shutdown_connections 2024-04-23T18:18:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.320+0000 7feedeb19640 1 -- 172.21.15.50:0/415483023 wait complete. 2024-04-23T18:18:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedeb19640 1 Processor -- start 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedeb19640 1 -- start start 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedeb19640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feed8102170 0x7feed81986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedeb19640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feed8107240 0x7feed8198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedeb19640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feed819cf30 0x7feed819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedeb19640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7feed81121e0 con 0x7feed819cf30 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedeb19640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feed8112060 con 0x7feed8107240 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedeb19640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7feed8112360 con 0x7feed8102170 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedd0b6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feed819cf30 0x7feed819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedd0b6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feed819cf30 0x7feed819d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49324/0 (socket says 172.21.15.50:49324) 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedd0b6640 1 -- 172.21.15.50:0/2038543441 learned_addr learned my addr 172.21.15.50:0/2038543441 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feed7fff640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feed8107240 0x7feed8198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedc8b5640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feed8102170 0x7feed81986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedd0b6640 1 -- 172.21.15.50:0/2038543441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feed8102170 msgr2=0x7feed81986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedd0b6640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feed8102170 0x7feed81986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedd0b6640 1 -- 172.21.15.50:0/2038543441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feed8107240 msgr2=0x7feed8198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedd0b6640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feed8107240 0x7feed8198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedd0b6640 1 -- 172.21.15.50:0/2038543441 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed819da20 con 0x7feed819cf30 2024-04-23T18:18:37.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feed7fff640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feed8107240 0x7feed8198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:37.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.321+0000 7feedd0b6640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feed819cf30 0x7feed819d3e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7feed0009430 tx=0x7feed00319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:37.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.322+0000 7feed5ffb640 1 -- 172.21.15.50:0/2038543441 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed0003c30 con 0x7feed819cf30 2024-04-23T18:18:37.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.322+0000 7feed5ffb640 1 -- 172.21.15.50:0/2038543441 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feed0003dd0 con 0x7feed819cf30 2024-04-23T18:18:37.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.322+0000 7feedeb19640 1 -- 172.21.15.50:0/2038543441 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feed819dcb0 con 0x7feed819cf30 2024-04-23T18:18:37.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.322+0000 7feed5ffb640 1 -- 172.21.15.50:0/2038543441 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed002faa0 con 0x7feed819cf30 2024-04-23T18:18:37.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.322+0000 7feedeb19640 1 -- 172.21.15.50:0/2038543441 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feed81a54d0 con 0x7feed819cf30 2024-04-23T18:18:37.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.323+0000 7feedeb19640 1 -- 172.21.15.50:0/2038543441 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feed80684e0 con 0x7feed819cf30 2024-04-23T18:18:37.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.324+0000 7feed5ffb640 1 -- 172.21.15.50:0/2038543441 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7feed002fc60 con 0x7feed819cf30 2024-04-23T18:18:37.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.325+0000 7feed5ffb640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feec80a3db0 0x7feec80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.325+0000 7feedc8b5640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feec80a3db0 0x7feec80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.325+0000 7feed5ffb640 1 -- 172.21.15.50:0/2038543441 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7feed00f1f80 con 0x7feed819cf30 2024-04-23T18:18:37.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.325+0000 7feedc8b5640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feec80a3db0 0x7feec80a6270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feecc005fd0 tx=0x7feecc00b590 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:37.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.329+0000 7feed5ffb640 1 -- 172.21.15.50:0/2038543441 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feed00be570 con 0x7feed819cf30 2024-04-23T18:18:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:37 smithi188 ceph-mon[32694]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:18:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:37 smithi188 ceph-mon[32694]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:18:37.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:37 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/156900633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:37.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.514+0000 7feedeb19640 1 -- 172.21.15.50:0/2038543441 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7feed8102b20 con 0x7feed819cf30 2024-04-23T18:18:37.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.514+0000 7feed5ffb640 1 -- 172.21.15.50:0/2038543441 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7feed00bde70 con 0x7feed819cf30 2024-04-23T18:18:37.507 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-23T18:18:37.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.516+0000 7feedeb19640 1 -- 172.21.15.50:0/2038543441 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feec80a3db0 msgr2=0x7feec80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.516+0000 7feedeb19640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feec80a3db0 0x7feec80a6270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feecc005fd0 tx=0x7feecc00b590 comp rx=0 tx=0).stop 2024-04-23T18:18:37.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.516+0000 7feedeb19640 1 -- 172.21.15.50:0/2038543441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feed819cf30 msgr2=0x7feed819d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.516+0000 7feedeb19640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feed819cf30 0x7feed819d3e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7feed0009430 tx=0x7feed00319f0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.516+0000 7feedeb19640 1 -- 172.21.15.50:0/2038543441 shutdown_connections 2024-04-23T18:18:37.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.516+0000 7feedeb19640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feec80a3db0 0x7feec80a6270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.516+0000 7feedeb19640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feed819cf30 0x7feed819d3e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.516+0000 7feedeb19640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feed8107240 0x7feed8198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.516+0000 7feedeb19640 1 --2- 172.21.15.50:0/2038543441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feed8102170 0x7feed81986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.516+0000 7feedeb19640 1 -- 172.21.15.50:0/2038543441 >> 172.21.15.50:0/2038543441 conn(0x7feed80fd9d0 msgr2=0x7feed80ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:37.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.517+0000 7feedeb19640 1 -- 172.21.15.50:0/2038543441 shutdown_connections 2024-04-23T18:18:37.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.517+0000 7feedeb19640 1 -- 172.21.15.50:0/2038543441 wait complete. 2024-04-23T18:18:37.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:18:37.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.633+0000 7f08d7e91640 1 -- 172.21.15.50:0/678235118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08d0108020 msgr2=0x7f08d0108400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.633+0000 7f08d7e91640 1 --2- 172.21.15.50:0/678235118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08d0108020 0x7f08d0108400 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009900 tx=0x7f08cc02a480 comp rx=0 tx=0).stop 2024-04-23T18:18:37.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.633+0000 7f08d7e91640 1 -- 172.21.15.50:0/678235118 shutdown_connections 2024-04-23T18:18:37.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.633+0000 7f08d7e91640 1 --2- 172.21.15.50:0/678235118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08d006af70 0x7f08d010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.633+0000 7f08d7e91640 1 --2- 172.21.15.50:0/678235118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08d006a5d0 0x7f08d006aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.633+0000 7f08d7e91640 1 --2- 172.21.15.50:0/678235118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08d0108020 0x7f08d0108400 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.633+0000 7f08d7e91640 1 -- 172.21.15.50:0/678235118 >> 172.21.15.50:0/678235118 conn(0x7f08d0076c50 msgr2=0x7f08d0077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:37.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.633+0000 7f08d7e91640 1 -- 172.21.15.50:0/678235118 shutdown_connections 2024-04-23T18:18:37.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.633+0000 7f08d7e91640 1 -- 172.21.15.50:0/678235118 wait complete. 2024-04-23T18:18:37.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d7e91640 1 Processor -- start 2024-04-23T18:18:37.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d7e91640 1 -- start start 2024-04-23T18:18:37.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d7e91640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08d006a5d0 0x7f08d01a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d7e91640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08d006af70 0x7f08d01a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d7e91640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08d0108020 0x7f08d019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d7e91640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f08d01121d0 con 0x7f08d0108020 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d7e91640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f08d0112050 con 0x7f08d006af70 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d7e91640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f08d0112350 con 0x7f08d006a5d0 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d642e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08d0108020 0x7f08d019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d642e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08d0108020 0x7f08d019a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49332/0 (socket says 172.21.15.50:49332) 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d642e640 1 -- 172.21.15.50:0/2291885031 learned_addr learned my addr 172.21.15.50:0/2291885031 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d542c640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08d006af70 0x7f08d01a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d5c2d640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08d006a5d0 0x7f08d01a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d642e640 1 -- 172.21.15.50:0/2291885031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08d006a5d0 msgr2=0x7f08d01a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d642e640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08d006a5d0 0x7f08d01a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d642e640 1 -- 172.21.15.50:0/2291885031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08d006af70 msgr2=0x7f08d01a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d642e640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08d006af70 0x7f08d01a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.634+0000 7f08d642e640 1 -- 172.21.15.50:0/2291885031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d019aff0 con 0x7f08d0108020 2024-04-23T18:18:37.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.635+0000 7f08d642e640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08d0108020 0x7f08d019a730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f08c800da40 tx=0x7f08c800df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:37.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.635+0000 7f08c6ffd640 1 -- 172.21.15.50:0/2291885031 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c8017070 con 0x7f08d0108020 2024-04-23T18:18:37.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.635+0000 7f08c6ffd640 1 -- 172.21.15.50:0/2291885031 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08c8003e20 con 0x7f08d0108020 2024-04-23T18:18:37.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.635+0000 7f08d7e91640 1 -- 172.21.15.50:0/2291885031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d019b2e0 con 0x7f08d0108020 2024-04-23T18:18:37.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.635+0000 7f08c6ffd640 1 -- 172.21.15.50:0/2291885031 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c8010460 con 0x7f08d0108020 2024-04-23T18:18:37.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.635+0000 7f08d7e91640 1 -- 172.21.15.50:0/2291885031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d01a75f0 con 0x7f08d0108020 2024-04-23T18:18:37.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.636+0000 7f08d7e91640 1 -- 172.21.15.50:0/2291885031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08d0103660 con 0x7f08d0108020 2024-04-23T18:18:37.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.637+0000 7f08c6ffd640 1 -- 172.21.15.50:0/2291885031 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f08c8004020 con 0x7f08d0108020 2024-04-23T18:18:37.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.638+0000 7f08c6ffd640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f08bc0a3db0 0x7f08bc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.638+0000 7f08d5c2d640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f08bc0a3db0 0x7f08bc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.638+0000 7f08c6ffd640 1 -- 172.21.15.50:0/2291885031 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f08c80d05f0 con 0x7f08d0108020 2024-04-23T18:18:37.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.638+0000 7f08d5c2d640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f08bc0a3db0 0x7f08bc0a6270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f08cc000c00 tx=0x7f08cc031400 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:37.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.642+0000 7f08c6ffd640 1 -- 172.21.15.50:0/2291885031 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f08c809cb30 con 0x7f08d0108020 2024-04-23T18:18:37.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.826+0000 7f08d7e91640 1 -- 172.21.15.50:0/2291885031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f08d01a77c0 con 0x7f08d0108020 2024-04-23T18:18:37.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.827+0000 7f08c6ffd640 1 -- 172.21.15.50:0/2291885031 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f08c809c430 con 0x7f08d0108020 2024-04-23T18:18:37.819 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2024-04-23T18:18:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.829+0000 7f08d7e91640 1 -- 172.21.15.50:0/2291885031 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f08bc0a3db0 msgr2=0x7f08bc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.829+0000 7f08d7e91640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f08bc0a3db0 0x7f08bc0a6270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f08cc000c00 tx=0x7f08cc031400 comp rx=0 tx=0).stop 2024-04-23T18:18:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.829+0000 7f08d7e91640 1 -- 172.21.15.50:0/2291885031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08d0108020 msgr2=0x7f08d019a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.829+0000 7f08d7e91640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08d0108020 0x7f08d019a730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f08c800da40 tx=0x7f08c800df00 comp rx=0 tx=0).stop 2024-04-23T18:18:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.829+0000 7f08d7e91640 1 -- 172.21.15.50:0/2291885031 shutdown_connections 2024-04-23T18:18:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.829+0000 7f08d7e91640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f08bc0a3db0 0x7f08bc0a6270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.829+0000 7f08d7e91640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08d0108020 0x7f08d019a730 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.829+0000 7f08d7e91640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08d006af70 0x7f08d01a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.829+0000 7f08d7e91640 1 --2- 172.21.15.50:0/2291885031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08d006a5d0 0x7f08d01a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.829+0000 7f08d7e91640 1 -- 172.21.15.50:0/2291885031 >> 172.21.15.50:0/2291885031 conn(0x7f08d0076c50 msgr2=0x7f08d010d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.829+0000 7f08d7e91640 1 -- 172.21.15.50:0/2291885031 shutdown_connections 2024-04-23T18:18:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.829+0000 7f08d7e91640 1 -- 172.21.15.50:0/2291885031 wait complete. 2024-04-23T18:18:37.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:18:37.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.945+0000 7f2d23843640 1 -- 172.21.15.50:0/1980798347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d1c101a40 msgr2=0x7f2d1c105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.945+0000 7f2d23843640 1 --2- 172.21.15.50:0/1980798347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d1c101a40 0x7f2d1c105a80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c009900 tx=0x7f2d0c02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:37.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.945+0000 7f2d23843640 1 -- 172.21.15.50:0/1980798347 shutdown_connections 2024-04-23T18:18:37.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.945+0000 7f2d23843640 1 --2- 172.21.15.50:0/1980798347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d1c106160 0x7f2d1c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.945+0000 7f2d23843640 1 --2- 172.21.15.50:0/1980798347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d1c101a40 0x7f2d1c105a80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.945+0000 7f2d23843640 1 --2- 172.21.15.50:0/1980798347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d1c101090 0x7f2d1c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.945+0000 7f2d23843640 1 -- 172.21.15.50:0/1980798347 >> 172.21.15.50:0/1980798347 conn(0x7f2d1c077f50 msgr2=0x7f2d1c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:37.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.945+0000 7f2d23843640 1 -- 172.21.15.50:0/1980798347 shutdown_connections 2024-04-23T18:18:37.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.945+0000 7f2d23843640 1 -- 172.21.15.50:0/1980798347 wait complete. 2024-04-23T18:18:37.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.945+0000 7f2d23843640 1 Processor -- start 2024-04-23T18:18:37.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.945+0000 7f2d23843640 1 -- start start 2024-04-23T18:18:37.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d23843640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d1c101090 0x7f2d1c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d23843640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d1c106160 0x7f2d1c199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d23843640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d1c19a0b0 0x7f2d1c1a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d23843640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c111080 con 0x7f2d1c19a0b0 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d23843640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c110f00 con 0x7f2d1c101090 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d23843640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c111200 con 0x7f2d1c106160 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d21de0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d1c19a0b0 0x7f2d1c1a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d21de0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d1c19a0b0 0x7f2d1c1a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49348/0 (socket says 172.21.15.50:49348) 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d21de0640 1 -- 172.21.15.50:0/3443446302 learned_addr learned my addr 172.21.15.50:0/3443446302 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d20dde640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d1c106160 0x7f2d1c199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d215df640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d1c101090 0x7f2d1c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d21de0640 1 -- 172.21.15.50:0/3443446302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d1c106160 msgr2=0x7f2d1c199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d21de0640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d1c106160 0x7f2d1c199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d21de0640 1 -- 172.21.15.50:0/3443446302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d1c101090 msgr2=0x7f2d1c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d21de0640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d1c101090 0x7f2d1c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d21de0640 1 -- 172.21.15.50:0/3443446302 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c1a4e30 con 0x7f2d1c19a0b0 2024-04-23T18:18:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d215df640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d1c101090 0x7f2d1c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:37.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.946+0000 7f2d21de0640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d1c19a0b0 0x7f2d1c1a4660 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009fd0 tx=0x7f2d1400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:37.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.947+0000 7f2d127fc640 1 -- 172.21.15.50:0/3443446302 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d14017070 con 0x7f2d1c19a0b0 2024-04-23T18:18:37.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.947+0000 7f2d127fc640 1 -- 172.21.15.50:0/3443446302 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d14003e20 con 0x7f2d1c19a0b0 2024-04-23T18:18:37.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.947+0000 7f2d23843640 1 -- 172.21.15.50:0/3443446302 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1c1a5120 con 0x7f2d1c19a0b0 2024-04-23T18:18:37.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.947+0000 7f2d127fc640 1 -- 172.21.15.50:0/3443446302 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d14010460 con 0x7f2d1c19a0b0 2024-04-23T18:18:37.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.947+0000 7f2d23843640 1 -- 172.21.15.50:0/3443446302 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d1c1a5640 con 0x7f2d1c19a0b0 2024-04-23T18:18:37.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.948+0000 7f2d23843640 1 -- 172.21.15.50:0/3443446302 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d1c102560 con 0x7f2d1c19a0b0 2024-04-23T18:18:37.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.949+0000 7f2d127fc640 1 -- 172.21.15.50:0/3443446302 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f2d14003970 con 0x7f2d1c19a0b0 2024-04-23T18:18:37.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.949+0000 7f2d127fc640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2d080a3e30 0x7f2d080a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:37.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.950+0000 7f2d215df640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2d080a3e30 0x7f2d080a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:37.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.950+0000 7f2d127fc640 1 -- 172.21.15.50:0/3443446302 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f2d140d06b0 con 0x7f2d1c19a0b0 2024-04-23T18:18:37.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.950+0000 7f2d215df640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2d080a3e30 0x7f2d080a62f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d18005d00 tx=0x7f2d18009550 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:37.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:37.954+0000 7f2d127fc640 1 -- 172.21.15.50:0/3443446302 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d1409cb90 con 0x7f2d1c19a0b0 2024-04-23T18:18:38.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.143+0000 7f2d23843640 1 -- 172.21.15.50:0/3443446302 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2d1c109390 con 0x7f2d1c19a0b0 2024-04-23T18:18:38.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.143+0000 7f2d127fc640 1 -- 172.21.15.50:0/3443446302 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f2d1409c490 con 0x7f2d1c19a0b0 2024-04-23T18:18:38.136 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2024-04-23T18:18:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.146+0000 7f2d23843640 1 -- 172.21.15.50:0/3443446302 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2d080a3e30 msgr2=0x7f2d080a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.146+0000 7f2d23843640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2d080a3e30 0x7f2d080a62f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d18005d00 tx=0x7f2d18009550 comp rx=0 tx=0).stop 2024-04-23T18:18:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.146+0000 7f2d23843640 1 -- 172.21.15.50:0/3443446302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d1c19a0b0 msgr2=0x7f2d1c1a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.146+0000 7f2d23843640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d1c19a0b0 0x7f2d1c1a4660 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009fd0 tx=0x7f2d1400b260 comp rx=0 tx=0).stop 2024-04-23T18:18:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.146+0000 7f2d23843640 1 -- 172.21.15.50:0/3443446302 shutdown_connections 2024-04-23T18:18:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.146+0000 7f2d23843640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2d080a3e30 0x7f2d080a62f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.146+0000 7f2d23843640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d1c19a0b0 0x7f2d1c1a4660 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.146+0000 7f2d23843640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d1c106160 0x7f2d1c199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.146+0000 7f2d23843640 1 --2- 172.21.15.50:0/3443446302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d1c101090 0x7f2d1c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.146+0000 7f2d23843640 1 -- 172.21.15.50:0/3443446302 >> 172.21.15.50:0/3443446302 conn(0x7f2d1c077f50 msgr2=0x7f2d1c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.146+0000 7f2d23843640 1 -- 172.21.15.50:0/3443446302 shutdown_connections 2024-04-23T18:18:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.146+0000 7f2d23843640 1 -- 172.21.15.50:0/3443446302 wait complete. 2024-04-23T18:18:38.149 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:38 smithi050 ceph-mon[29187]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:18:38.149 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:38 smithi050 ceph-mon[29187]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 65 op/s 2024-04-23T18:18:38.149 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:38 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2059918676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:38.149 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:38 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2038543441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:38.149 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:38 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2291885031' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:38.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:18:38.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 -- 172.21.15.50:0/427803382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74341010a0 msgr2=0x7f7434101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 --2- 172.21.15.50:0/427803382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74341010a0 0x7f7434101480 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7430009900 tx=0x7f743002a410 comp rx=0 tx=0).stop 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 -- 172.21.15.50:0/427803382 shutdown_connections 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 --2- 172.21.15.50:0/427803382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7434106170 0x7f743410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 --2- 172.21.15.50:0/427803382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7434101a50 0x7f7434105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 --2- 172.21.15.50:0/427803382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74341010a0 0x7f7434101480 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 -- 172.21.15.50:0/427803382 >> 172.21.15.50:0/427803382 conn(0x7f7434077f50 msgr2=0x7f7434078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 -- 172.21.15.50:0/427803382 shutdown_connections 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 -- 172.21.15.50:0/427803382 wait complete. 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 Processor -- start 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 -- start start 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74341010a0 0x7f7434198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7434106170 0x7f7434198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f743419cfe0 0x7f743419d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:38.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f74341110b0 con 0x7f74341010a0 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7434110f30 con 0x7f7434106170 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.260+0000 7f743b544640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7434111230 con 0x7f743419cfe0 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f74392e0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74341010a0 0x7f7434198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f74392e0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74341010a0 0x7f7434198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49364/0 (socket says 172.21.15.50:49364) 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f74392e0640 1 -- 172.21.15.50:0/686951543 learned_addr learned my addr 172.21.15.50:0/686951543 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f7439ae1640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f743419cfe0 0x7f743419d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f7438adf640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7434106170 0x7f7434198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f74392e0640 1 -- 172.21.15.50:0/686951543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f743419cfe0 msgr2=0x7f743419d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f74392e0640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f743419cfe0 0x7f743419d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f74392e0640 1 -- 172.21.15.50:0/686951543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7434106170 msgr2=0x7f7434198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f74392e0640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7434106170 0x7f7434198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f74392e0640 1 -- 172.21.15.50:0/686951543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743419dad0 con 0x7f74341010a0 2024-04-23T18:18:38.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f7438adf640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7434106170 0x7f7434198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:38.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f74392e0640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74341010a0 0x7f7434198720 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7430005940 tx=0x7f74300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:38.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f742a7fc640 1 -- 172.21.15.50:0/686951543 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743003f030 con 0x7f74341010a0 2024-04-23T18:18:38.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f742a7fc640 1 -- 172.21.15.50:0/686951543 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7430003cb0 con 0x7f74341010a0 2024-04-23T18:18:38.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f742a7fc640 1 -- 172.21.15.50:0/686951543 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7430035070 con 0x7f74341010a0 2024-04-23T18:18:38.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f743b544640 1 -- 172.21.15.50:0/686951543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f743419dd60 con 0x7f74341010a0 2024-04-23T18:18:38.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.261+0000 7f743b544640 1 -- 172.21.15.50:0/686951543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74341a5520 con 0x7f74341010a0 2024-04-23T18:18:38.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.262+0000 7f743b544640 1 -- 172.21.15.50:0/686951543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7434068d70 con 0x7f74341010a0 2024-04-23T18:18:38.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.264+0000 7f742a7fc640 1 -- 172.21.15.50:0/686951543 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f7430042030 con 0x7f74341010a0 2024-04-23T18:18:38.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.264+0000 7f742a7fc640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f74200a3db0 0x7f74200a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:38.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.264+0000 7f7438adf640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f74200a3db0 0x7f74200a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:38.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.264+0000 7f742a7fc640 1 -- 172.21.15.50:0/686951543 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f74300f15b0 con 0x7f74341010a0 2024-04-23T18:18:38.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.265+0000 7f7438adf640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f74200a3db0 0x7f74200a6270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f743406a1a0 tx=0x7f7424005d90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:38.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.268+0000 7f742a7fc640 1 -- 172.21.15.50:0/686951543 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f74300bda90 con 0x7f74341010a0 2024-04-23T18:18:38.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:38 smithi177 ceph-mon[32308]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:18:38.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:38 smithi177 ceph-mon[32308]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 65 op/s 2024-04-23T18:18:38.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:38 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2059918676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:38.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:38 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2038543441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:38.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:38 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2291885031' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:38.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:38 smithi188 ceph-mon[32694]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T18:18:38.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:38 smithi188 ceph-mon[32694]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 65 op/s 2024-04-23T18:18:38.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:38 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2059918676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:38.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:38 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2038543441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:18:38.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:38 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2291885031' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:18:38.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.493+0000 7f743b544640 1 -- 172.21.15.50:0/686951543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7434101a80 con 0x7f74341010a0 2024-04-23T18:18:38.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.494+0000 7f742a7fc640 1 -- 172.21.15.50:0/686951543 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11875 (secure 0 0 0) 0x7f74300bd390 con 0x7f74341010a0 2024-04-23T18:18:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:38.487 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:18:28.904281+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:18:38.487 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 21 2024-04-23T18:18:38.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.496+0000 7f743b544640 1 -- 172.21.15.50:0/686951543 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f74200a3db0 msgr2=0x7f74200a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.496+0000 7f743b544640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f74200a3db0 0x7f74200a6270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f743406a1a0 tx=0x7f7424005d90 comp rx=0 tx=0).stop 2024-04-23T18:18:38.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.496+0000 7f743b544640 1 -- 172.21.15.50:0/686951543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74341010a0 msgr2=0x7f7434198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.496+0000 7f743b544640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74341010a0 0x7f7434198720 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7430005940 tx=0x7f74300042f0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.496+0000 7f743b544640 1 -- 172.21.15.50:0/686951543 shutdown_connections 2024-04-23T18:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.496+0000 7f743b544640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f74200a3db0 0x7f74200a6270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.496+0000 7f743b544640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f743419cfe0 0x7f743419d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.496+0000 7f743b544640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7434106170 0x7f7434198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.496+0000 7f743b544640 1 --2- 172.21.15.50:0/686951543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74341010a0 0x7f7434198720 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.496+0000 7f743b544640 1 -- 172.21.15.50:0/686951543 >> 172.21.15.50:0/686951543 conn(0x7f7434077f50 msgr2=0x7f7434104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.497+0000 7f743b544640 1 -- 172.21.15.50:0/686951543 shutdown_connections 2024-04-23T18:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.497+0000 7f743b544640 1 -- 172.21.15.50:0/686951543 wait complete. 2024-04-23T18:18:38.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:18:38.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.610+0000 7f2abda33640 1 -- 172.21.15.50:0/22617779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab8106170 msgr2=0x7f2ab810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.610+0000 7f2abda33640 1 --2- 172.21.15.50:0/22617779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab8106170 0x7f2ab810dd40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009900 tx=0x7f2ab002a410 comp rx=0 tx=0).stop 2024-04-23T18:18:38.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.610+0000 7f2abda33640 1 -- 172.21.15.50:0/22617779 shutdown_connections 2024-04-23T18:18:38.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.610+0000 7f2abda33640 1 --2- 172.21.15.50:0/22617779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab8106170 0x7f2ab810dd40 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.610+0000 7f2abda33640 1 --2- 172.21.15.50:0/22617779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ab8101a50 0x7f2ab8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.610+0000 7f2abda33640 1 --2- 172.21.15.50:0/22617779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ab81010a0 0x7f2ab8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.610+0000 7f2abda33640 1 -- 172.21.15.50:0/22617779 >> 172.21.15.50:0/22617779 conn(0x7f2ab8077f50 msgr2=0x7f2ab8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:38.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.610+0000 7f2abda33640 1 -- 172.21.15.50:0/22617779 shutdown_connections 2024-04-23T18:18:38.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.610+0000 7f2abda33640 1 -- 172.21.15.50:0/22617779 wait complete. 2024-04-23T18:18:38.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.610+0000 7f2abda33640 1 Processor -- start 2024-04-23T18:18:38.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.610+0000 7f2abda33640 1 -- start start 2024-04-23T18:18:38.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2abda33640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ab81010a0 0x7f2ab81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:38.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2abda33640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ab8106170 0x7f2ab8198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:38.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2abda33640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab819cfb0 0x7f2ab819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:38.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2abda33640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81121f0 con 0x7f2ab819cfb0 2024-04-23T18:18:38.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2abda33640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab8112070 con 0x7f2ab8106170 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2abda33640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab8112370 con 0x7f2ab81010a0 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2ab77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab819cfb0 0x7f2ab819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2ab77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab819cfb0 0x7f2ab819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49374/0 (socket says 172.21.15.50:49374) 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2ab77fe640 1 -- 172.21.15.50:0/1454749267 learned_addr learned my addr 172.21.15.50:0/1454749267 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2ab6ffd640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ab81010a0 0x7f2ab81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2ab67fc640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ab8106170 0x7f2ab8198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2ab77fe640 1 -- 172.21.15.50:0/1454749267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ab81010a0 msgr2=0x7f2ab81986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2ab77fe640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ab81010a0 0x7f2ab81986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2ab77fe640 1 -- 172.21.15.50:0/1454749267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ab8106170 msgr2=0x7f2ab8198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2ab77fe640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ab8106170 0x7f2ab8198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2ab77fe640 1 -- 172.21.15.50:0/1454749267 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab819daa0 con 0x7f2ab819cfb0 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2ab6ffd640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ab81010a0 0x7f2ab81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2ab77fe640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab819cfb0 0x7f2ab819d460 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0004770 tx=0x7f2ab0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:38.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2abca31640 1 -- 172.21.15.50:0/1454749267 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab003f030 con 0x7f2ab819cfb0 2024-04-23T18:18:38.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2abca31640 1 -- 172.21.15.50:0/1454749267 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ab0004440 con 0x7f2ab819cfb0 2024-04-23T18:18:38.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2abda33640 1 -- 172.21.15.50:0/1454749267 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab819dd30 con 0x7f2ab819cfb0 2024-04-23T18:18:38.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2abca31640 1 -- 172.21.15.50:0/1454749267 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0035070 con 0x7f2ab819cfb0 2024-04-23T18:18:38.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.611+0000 7f2abda33640 1 -- 172.21.15.50:0/1454749267 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab81a5550 con 0x7f2ab819cfb0 2024-04-23T18:18:38.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.612+0000 7f2abda33640 1 -- 172.21.15.50:0/1454749267 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab8068d70 con 0x7f2ab819cfb0 2024-04-23T18:18:38.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.614+0000 7f2abca31640 1 -- 172.21.15.50:0/1454749267 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f2ab0042030 con 0x7f2ab819cfb0 2024-04-23T18:18:38.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.614+0000 7f2abca31640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2aa00a3e30 0x7f2aa00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:38.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.614+0000 7f2ab6ffd640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2aa00a3e30 0x7f2aa00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:38.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.614+0000 7f2abca31640 1 -- 172.21.15.50:0/1454749267 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f2ab00f0f90 con 0x7f2ab819cfb0 2024-04-23T18:18:38.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.615+0000 7f2ab6ffd640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2aa00a3e30 0x7f2aa00a62f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8009b50 tx=0x7f2aa8005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:38.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.618+0000 7f2abca31640 1 -- 172.21.15.50:0/1454749267 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ab00bd4d0 con 0x7f2ab819cfb0 2024-04-23T18:18:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.807+0000 7f2abda33640 1 -- 172.21.15.50:0/1454749267 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ab8199a40 con 0x7f2ab819cfb0 2024-04-23T18:18:38.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.809+0000 7f2abca31640 1 -- 172.21.15.50:0/1454749267 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23038 (secure 0 0 0) 0x7f2ab8199a40 con 0x7f2ab819cfb0 2024-04-23T18:18:38.801 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:38.802 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":90,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:18:30.504176+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/1479623929":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/3248139242":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6800/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/2484340243":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/444338023":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:18:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.811+0000 7f2abda33640 1 -- 172.21.15.50:0/1454749267 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2aa00a3e30 msgr2=0x7f2aa00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.811+0000 7f2abda33640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2aa00a3e30 0x7f2aa00a62f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8009b50 tx=0x7f2aa8005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.811+0000 7f2abda33640 1 -- 172.21.15.50:0/1454749267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab819cfb0 msgr2=0x7f2ab819d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.811+0000 7f2abda33640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab819cfb0 0x7f2ab819d460 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0004770 tx=0x7f2ab0004040 comp rx=0 tx=0).stop 2024-04-23T18:18:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.811+0000 7f2abda33640 1 -- 172.21.15.50:0/1454749267 shutdown_connections 2024-04-23T18:18:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.811+0000 7f2abda33640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2aa00a3e30 0x7f2aa00a62f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.811+0000 7f2abda33640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab819cfb0 0x7f2ab819d460 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.811+0000 7f2abda33640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ab8106170 0x7f2ab8198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.811+0000 7f2abda33640 1 --2- 172.21.15.50:0/1454749267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ab81010a0 0x7f2ab81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.811+0000 7f2abda33640 1 -- 172.21.15.50:0/1454749267 >> 172.21.15.50:0/1454749267 conn(0x7f2ab8077f50 msgr2=0x7f2ab80fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.811+0000 7f2abda33640 1 -- 172.21.15.50:0/1454749267 shutdown_connections 2024-04-23T18:18:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.811+0000 7f2abda33640 1 -- 172.21.15.50:0/1454749267 wait complete. 2024-04-23T18:18:38.815 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 444943 2024-04-23T18:18:38.815 DEBUG:tasks.quiescer:Yielding 2024-04-23T18:18:38.816 INFO:teuthology.run_tasks:Running task workunit... 2024-04-23T18:18:38.828 INFO:tasks.workunit:Pulling workunits from ref 57e0caf76fdb7ab8b1358588f292d08519163844 2024-04-23T18:18:38.829 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-23T18:18:38.829 INFO:tasks.workunit:timeout=3h 2024-04-23T18:18:38.829 INFO:tasks.workunit:cleanup=True 2024-04-23T18:18:38.829 DEBUG:teuthology.orchestra.run.smithi050:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-23T18:18:38.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:18:38.845 INFO:teuthology.orchestra.run.smithi050.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-23T18:18:38.845 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-23T18:18:38.845 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-23T18:18:38.846 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T18:18:38.846 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-23T18:18:38.846 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-23 18:16:39.419367900 +0000 2024-04-23T18:18:38.846 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-23 18:16:39.419367900 +0000 2024-04-23T18:18:38.846 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-23 18:18:28.137298770 +0000 2024-04-23T18:18:38.846 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-04-23 18:16:39.419367900 +0000 2024-04-23T18:18:38.846 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-23T18:18:38.846 DEBUG:teuthology.orchestra.run.smithi050:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-23T18:18:38.916 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 57e0caf76fdb7ab8b1358588f292d08519163844 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.941+0000 7f7acb15c640 1 -- 172.21.15.50:0/3707134034 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac4103230 msgr2=0x7f7ac4103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.941+0000 7f7acb15c640 1 --2- 172.21.15.50:0/3707134034 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac4103230 0x7f7ac4103610 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4009900 tx=0x7f7ab402a480 comp rx=0 tx=0).stop 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.941+0000 7f7acb15c640 1 -- 172.21.15.50:0/3707134034 shutdown_connections 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.941+0000 7f7acb15c640 1 --2- 172.21.15.50:0/3707134034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ac410c570 0x7f7ac410e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.941+0000 7f7acb15c640 1 --2- 172.21.15.50:0/3707134034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac4103b50 0x7f7ac410c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.941+0000 7f7acb15c640 1 --2- 172.21.15.50:0/3707134034 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac4103230 0x7f7ac4103610 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.941+0000 7f7acb15c640 1 -- 172.21.15.50:0/3707134034 >> 172.21.15.50:0/3707134034 conn(0x7f7ac4077fb0 msgr2=0x7f7ac40783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.941+0000 7f7acb15c640 1 -- 172.21.15.50:0/3707134034 shutdown_connections 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.941+0000 7f7acb15c640 1 -- 172.21.15.50:0/3707134034 wait complete. 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7acb15c640 1 Processor -- start 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7acb15c640 1 -- start start 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7acb15c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ac4103230 0x7f7ac4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7acb15c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac4103b50 0x7f7ac4198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7acb15c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac410c570 0x7f7ac419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7acb15c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac41121b0 con 0x7f7ac410c570 2024-04-23T18:18:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7acb15c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac4112030 con 0x7f7ac4103230 2024-04-23T18:18:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7acb15c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac4112330 con 0x7f7ac4103b50 2024-04-23T18:18:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7ac96f9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac410c570 0x7f7ac419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7ac96f9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac410c570 0x7f7ac419cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49378/0 (socket says 172.21.15.50:49378) 2024-04-23T18:18:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7ac96f9640 1 -- 172.21.15.50:0/2343029450 learned_addr learned my addr 172.21.15.50:0/2343029450 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7ac8ef8640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ac4103230 0x7f7ac4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7ac3fff640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac4103b50 0x7f7ac4198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7ac96f9640 1 -- 172.21.15.50:0/2343029450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac4103b50 msgr2=0x7f7ac4198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7ac96f9640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac4103b50 0x7f7ac4198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7ac96f9640 1 -- 172.21.15.50:0/2343029450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ac4103230 msgr2=0x7f7ac4198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7ac96f9640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ac4103230 0x7f7ac4198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7ac96f9640 1 -- 172.21.15.50:0/2343029450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac419d720 con 0x7f7ac410c570 2024-04-23T18:18:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7ac8ef8640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ac4103230 0x7f7ac4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.942+0000 7f7ac96f9640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac410c570 0x7f7ac419cff0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7abc00da40 tx=0x7f7abc00df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.943+0000 7f7ac1ffb640 1 -- 172.21.15.50:0/2343029450 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc017070 con 0x7f7ac410c570 2024-04-23T18:18:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.943+0000 7f7acb15c640 1 -- 172.21.15.50:0/2343029450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ac419da10 con 0x7f7ac410c570 2024-04-23T18:18:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.943+0000 7f7ac1ffb640 1 -- 172.21.15.50:0/2343029450 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7abc003e20 con 0x7f7ac410c570 2024-04-23T18:18:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.943+0000 7f7ac1ffb640 1 -- 172.21.15.50:0/2343029450 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc010460 con 0x7f7ac410c570 2024-04-23T18:18:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.943+0000 7f7acb15c640 1 -- 172.21.15.50:0/2343029450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ac41a52a0 con 0x7f7ac410c570 2024-04-23T18:18:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.944+0000 7f7acb15c640 1 -- 172.21.15.50:0/2343029450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ac40691c0 con 0x7f7ac410c570 2024-04-23T18:18:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.945+0000 7f7ac1ffb640 1 -- 172.21.15.50:0/2343029450 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f7abc004020 con 0x7f7ac410c570 2024-04-23T18:18:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.945+0000 7f7ac1ffb640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7aac0a3db0 0x7f7aac0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.946+0000 7f7ac1ffb640 1 -- 172.21.15.50:0/2343029450 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f7abc0d06b0 con 0x7f7ac410c570 2024-04-23T18:18:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.946+0000 7f7ac8ef8640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7aac0a3db0 0x7f7aac0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.946+0000 7f7ac8ef8640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7aac0a3db0 0x7f7aac0a6270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4031440 tx=0x7f7ab4005a20 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:38.951+0000 7f7ac1ffb640 1 -- 172.21.15.50:0/2343029450 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7abc09cbf0 con 0x7f7ac410c570 2024-04-23T18:18:38.950 INFO:tasks.workunit.client.0.smithi050.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-23T18:18:39.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.180+0000 7f7acb15c640 1 -- 172.21.15.50:0/2343029450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ac41076f0 con 0x7f7ac410c570 2024-04-23T18:18:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.181+0000 7f7ac1ffb640 1 -- 172.21.15.50:0/2343029450 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11875 (secure 0 0 0) 0x7f7abc09c4f0 con 0x7f7ac410c570 2024-04-23T18:18:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:39.174 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:18:28.904281+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:18:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 21 2024-04-23T18:18:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.183+0000 7f7acb15c640 1 -- 172.21.15.50:0/2343029450 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7aac0a3db0 msgr2=0x7f7aac0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.183+0000 7f7acb15c640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7aac0a3db0 0x7f7aac0a6270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4031440 tx=0x7f7ab4005a20 comp rx=0 tx=0).stop 2024-04-23T18:18:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.183+0000 7f7acb15c640 1 -- 172.21.15.50:0/2343029450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac410c570 msgr2=0x7f7ac419cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.183+0000 7f7acb15c640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac410c570 0x7f7ac419cff0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7abc00da40 tx=0x7f7abc00df00 comp rx=0 tx=0).stop 2024-04-23T18:18:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.183+0000 7f7acb15c640 1 -- 172.21.15.50:0/2343029450 shutdown_connections 2024-04-23T18:18:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.183+0000 7f7acb15c640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7aac0a3db0 0x7f7aac0a6270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.183+0000 7f7acb15c640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac410c570 0x7f7ac419cff0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.183+0000 7f7acb15c640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac4103b50 0x7f7ac4198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.183+0000 7f7acb15c640 1 --2- 172.21.15.50:0/2343029450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ac4103230 0x7f7ac4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.183+0000 7f7acb15c640 1 -- 172.21.15.50:0/2343029450 >> 172.21.15.50:0/2343029450 conn(0x7f7ac4077fb0 msgr2=0x7f7ac40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.183+0000 7f7acb15c640 1 -- 172.21.15.50:0/2343029450 shutdown_connections 2024-04-23T18:18:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.183+0000 7f7acb15c640 1 -- 172.21.15.50:0/2343029450 wait complete. 2024-04-23T18:18:39.186 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:39 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3443446302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:39.186 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:39 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/686951543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:39.186 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:39 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1454749267' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:39.187 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T18:16:33.316468+0000', 'modified': '2024-04-23T18:18:28.904281+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24506, 'mds_1': 24499, 'mds_2': 24475}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24440': {'gid': 24440, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6835/475975073', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6834', 'nonce': 475975073}, {'type': 'v1', 'addr': '172.21.15.177:6835', 'nonce': 475975073}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6835/1248812622', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6834', 'nonce': 1248812622}, {'type': 'v1', 'addr': '172.21.15.188:6835', 'nonce': 1248812622}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6837/1048893636', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6836', 'nonce': 1048893636}, {'type': 'v1', 'addr': '172.21.15.177:6837', 'nonce': 1048893636}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.188:6837/2190889779', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6836', 'nonce': 2190889779}, {'type': 'v1', 'addr': '172.21.15.188:6837', 'nonce': 2190889779}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.188:6839/2202634301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6838', 'nonce': 2202634301}, {'type': 'v1', 'addr': '172.21.15.188:6839', 'nonce': 2202634301}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.177:6839/2936205355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 2936205355}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 2936205355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24506, 'qdb_cluster': [24475, 24499, 24506]} 2024-04-23T18:18:39.188 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-23T18:18:39.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-23T18:18:39.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:39 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3443446302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:39.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:39 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/686951543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:39.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:39 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1454749267' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:39.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:39 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3443446302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:18:39.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:39 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/686951543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:39.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:39 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1454749267' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:18:39.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.338+0000 7f78b7210640 1 -- 172.21.15.50:0/996205245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78b0102b50 msgr2=0x7f78b0106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.338+0000 7f78b7210640 1 --2- 172.21.15.50:0/996205245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78b0102b50 0x7f78b0106b90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f78a0009900 tx=0x7f78a002a410 comp rx=0 tx=0).stop 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.338+0000 7f78b7210640 1 -- 172.21.15.50:0/996205245 shutdown_connections 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.338+0000 7f78b7210640 1 --2- 172.21.15.50:0/996205245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b0107270 0x7f78b010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.338+0000 7f78b7210640 1 --2- 172.21.15.50:0/996205245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78b0102b50 0x7f78b0106b90 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.338+0000 7f78b7210640 1 --2- 172.21.15.50:0/996205245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78b01021a0 0x7f78b0102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.338+0000 7f78b7210640 1 -- 172.21.15.50:0/996205245 >> 172.21.15.50:0/996205245 conn(0x7f78b00fda20 msgr2=0x7f78b00ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.338+0000 7f78b7210640 1 -- 172.21.15.50:0/996205245 shutdown_connections 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.338+0000 7f78b7210640 1 -- 172.21.15.50:0/996205245 wait complete. 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b7210640 1 Processor -- start 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b7210640 1 -- start start 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b7210640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78b01021a0 0x7f78b0198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b7210640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b0107270 0x7f78b0198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b7210640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78b019cf70 0x7f78b019d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b7210640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01121e0 con 0x7f78b019cf70 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b7210640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f78b0112060 con 0x7f78b01021a0 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b7210640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f78b0112360 con 0x7f78b0107270 2024-04-23T18:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b57ad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78b019cf70 0x7f78b019d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b57ad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78b019cf70 0x7f78b019d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49396/0 (socket says 172.21.15.50:49396) 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b57ad640 1 -- 172.21.15.50:0/1260519402 learned_addr learned my addr 172.21.15.50:0/1260519402 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78a7fff640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b0107270 0x7f78b0198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b4fac640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78b01021a0 0x7f78b0198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b57ad640 1 -- 172.21.15.50:0/1260519402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b0107270 msgr2=0x7f78b0198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b57ad640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b0107270 0x7f78b0198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b57ad640 1 -- 172.21.15.50:0/1260519402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78b01021a0 msgr2=0x7f78b0198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b57ad640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78b01021a0 0x7f78b0198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b57ad640 1 -- 172.21.15.50:0/1260519402 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b019da60 con 0x7f78b019cf70 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.339+0000 7f78b57ad640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78b019cf70 0x7f78b019d420 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f78a800eaa0 tx=0x7f78a800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.340+0000 7f78a5ffb640 1 -- 172.21.15.50:0/1260519402 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a800cdb0 con 0x7f78b019cf70 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.340+0000 7f78b7210640 1 -- 172.21.15.50:0/1260519402 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b019dd50 con 0x7f78b019cf70 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.340+0000 7f78a5ffb640 1 -- 172.21.15.50:0/1260519402 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a8005020 con 0x7f78b019cf70 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.340+0000 7f78a5ffb640 1 -- 172.21.15.50:0/1260519402 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a80116d0 con 0x7f78b019cf70 2024-04-23T18:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.340+0000 7f78b7210640 1 -- 172.21.15.50:0/1260519402 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b01a5510 con 0x7f78b019cf70 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.342+0000 7f78a5ffb640 1 -- 172.21.15.50:0/1260519402 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f78a80051d0 con 0x7f78b019cf70 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.342+0000 7f78ba003740 1 Processor -- start 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.342+0000 7f78ba003740 1 -- start start 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.342+0000 7f78a5ffb640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f78980a3db0 0x7f78980a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78ba003740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 0x55675f9f1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78ba003740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f1760 0x55675f9fb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78ba003740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9fe220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78ba003740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55675f956c50 con 0x55675f9f0db0 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78ba003740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55675f8f3830 con 0x55675f9f1760 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78ba003740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55675f952c10 con 0x55675f9fbdd0 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78a7fff640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f78980a3db0 0x7f78980a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78a5ffb640 1 -- 172.21.15.50:0/1260519402 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f78a80d0b30 con 0x7f78b019cf70 2024-04-23T18:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78b4fac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 0x55675f9f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78b4fac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 0x55675f9f1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49412/0 (socket says 172.21.15.50:49412) 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78b4fac640 1 -- 172.21.15.50:0/1309937275 learned_addr learned my addr 172.21.15.50:0/1309937275 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78b57ad640 1 --2- 172.21.15.50:0/1309937275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9fe220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78b4fac640 1 -- 172.21.15.50:0/1309937275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 msgr2=0x55675f9fe220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78b4fac640 1 --2- 172.21.15.50:0/1309937275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9fe220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78b4fac640 1 -- 172.21.15.50:0/1309937275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f1760 msgr2=0x55675f9fb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78b4fac640 1 --2- 172.21.15.50:0/1309937275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f1760 0x55675f9fb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78b4fac640 1 -- 172.21.15.50:0/1309937275 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55675f6692d0 con 0x55675f9f0db0 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78b57ad640 1 --2- 172.21.15.50:0/1309937275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9fe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78a7fff640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f78980a3db0 0x7f78980a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f78a0031440 tx=0x7f78a0005b70 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f78b4fac640 1 --2- 172.21.15.50:0/1309937275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 0x55675f9f1190 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00cc60 tx=0x7f78ac00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182a6ed1b2f9d925 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f7896ffd640 1 -- 172.21.15.50:0/1309937275 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac014070 con 0x55675f9f0db0 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f7896ffd640 1 -- 172.21.15.50:0/1309937275 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78ac0047a0 con 0x55675f9f0db0 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.343+0000 7f7896ffd640 1 -- 172.21.15.50:0/1309937275 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac010670 con 0x55675f9f0db0 2024-04-23T18:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.344+0000 7f78ba003740 1 -- 172.21.15.50:0/1309937275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 msgr2=0x55675f9f1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.344+0000 7f78ba003740 1 --2- 172.21.15.50:0/1309937275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 0x55675f9f1190 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00cc60 tx=0x7f78ac00e720 comp rx=0 tx=0).stop 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.344+0000 7f78ba003740 1 -- 172.21.15.50:0/1309937275 shutdown_connections 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.344+0000 7f78ba003740 1 --2- 172.21.15.50:0/1309937275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9fe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.344+0000 7f78ba003740 1 --2- 172.21.15.50:0/1309937275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f1760 0x55675f9fb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.344+0000 7f78ba003740 1 --2- 172.21.15.50:0/1309937275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 0x55675f9f1190 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.344+0000 7f78ba003740 1 -- 172.21.15.50:0/1309937275 >> 172.21.15.50:0/1309937275 conn(0x55675f9ed070 msgr2=0x55675f9ed450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.344+0000 7f78ba003740 1 -- 172.21.15.50:0/1309937275 shutdown_connections 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.344+0000 7f78ba003740 1 -- 172.21.15.50:0/1309937275 wait complete. 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.344+0000 7f78ba003740 1 -- start start 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.344+0000 7f78ba003740 1 Processor -- start 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78ba003740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 0x55675f9f8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78ba003740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f1760 0x55675f9f9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78ba003740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9f9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78ba003740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55675f9fed40 con 0x55675f9f0db0 2024-04-23T18:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78ba003740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55675f8f1470 con 0x55675f9f1760 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78ba003740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55675f9feec0 con 0x55675f9fbdd0 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78ba003740 10 client.0 _ll_register_callbacks cb 0x55675f916f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78b4fac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 0x55675f9f8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78b4fac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 0x55675f9f8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49414/0 (socket says 172.21.15.50:49414) 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78b4fac640 1 -- 172.21.15.50:0/377110216 learned_addr learned my addr 172.21.15.50:0/377110216 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78a7fff640 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f1760 0x55675f9f9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78b57ad640 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9f9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78b4fac640 1 -- 172.21.15.50:0/377110216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 msgr2=0x55675f9f9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78b4fac640 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9f9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78b4fac640 1 -- 172.21.15.50:0/377110216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f1760 msgr2=0x55675f9f9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78b4fac640 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f1760 0x55675f9f9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78b4fac640 1 -- 172.21.15.50:0/377110216 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55675f7336b0 con 0x55675f9f0db0 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.345+0000 7f78b4fac640 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 0x55675f9f8cf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f78ac010c60 tx=0x7f78ac010c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27bf68fbf00df8ea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f788ffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f788ffff640 1 -- 172.21.15.50:0/377110216 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac014070 con 0x55675f9f0db0 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f788ffff640 1 -- 172.21.15.50:0/377110216 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f78ac004020 con 0x55675f9f0db0 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f78ba003740 1 -- 172.21.15.50:0/377110216 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78ac0047a0 con 0x55675f9f0db0 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f788ffff640 1 -- 172.21.15.50:0/377110216 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f78ac010e30 con 0x55675f9f0db0 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f78ba003740 10 client.15477 fetch_fsmap learned FSMap version 21 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f78ba003740 1 -- 172.21.15.50:0/377110216 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55675fa0af70 con 0x55675f9f0db0 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f788ffff640 1 -- 172.21.15.50:0/377110216 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78ac0047a0 con 0x55675f9f0db0 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f788ffff640 1 -- 172.21.15.50:0/377110216 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f78ac0035f0 con 0x55675f9f0db0 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f78ba003740 10 client.15477 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f78ba003740 10 client.15477 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f78ba003740 20 client.15477 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f78ba003740 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55675fa0d130 0x55675fa0e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f78ba003740 4 client.15477 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.346+0000 7f78ba003740 1 -- 172.21.15.50:0/377110216 --> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55675f964670 con 0x55675fa0d130 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.347+0000 7f78a7fff640 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55675fa0d130 0x55675fa0e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.347+0000 7f78a7fff640 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55675fa0d130 0x55675fa0e5a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=10066665e8401d93 server_cookie=8aa475a23bbe4095 in_seq=0 out_seq=0 2024-04-23T18:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.347+0000 7f788ffff640 10 client.15477 ms_handle_connect on v2:172.21.15.188:6836/2190889779 2024-04-23T18:18:39.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f788ffff640 1 -- 172.21.15.50:0/377110216 <== mds.2 v2:172.21.15.188:6836/2190889779 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55675f964670 con 0x55675fa0d130 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f788ffff640 10 client.15477 handle_command_reply: tid=0 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f78ba003740 1 client.15477 shutdown 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f78ba003740 1 -- 172.21.15.50:0/377110216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 msgr2=0x55675f9f8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f78ba003740 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 0x55675f9f8cf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f78ac010c60 tx=0x7f78ac010c90 comp rx=0 tx=0).stop 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f78ba003740 1 -- 172.21.15.50:0/377110216 shutdown_connections 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f78ba003740 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55675fa0d130 0x55675fa0e5a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f78ba003740 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9f9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f78ba003740 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f1760 0x55675f9f9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f788ffff640 0 client.15477 ms_handle_reset on v2:172.21.15.188:6836/2190889779 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f78ba003740 1 --2- 172.21.15.50:0/377110216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f0db0 0x55675f9f8cf0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f78ba003740 1 -- 172.21.15.50:0/377110216 >> 172.21.15.50:0/377110216 conn(0x55675f9ed070 msgr2=0x55675f9f2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f78ba003740 1 -- 172.21.15.50:0/377110216 shutdown_connections 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f78ba003740 1 -- 172.21.15.50:0/377110216 wait complete. 2024-04-23T18:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.348+0000 7f78ba003740 20 client.15477 trim_cache size 0 max 16384 2024-04-23T18:18:39.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.366+0000 7f78ba003740 1 Processor -- start 2024-04-23T18:18:39.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78ba003740 1 -- start start 2024-04-23T18:18:39.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78ba003740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f0db0 0x55675fa0ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78ba003740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 0x55675f9fb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78ba003740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78ba003740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55675fa0c8e0 con 0x55675f9f1760 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78ba003740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55675fa0cfb0 con 0x55675f9f0db0 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78ba003740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55675fa0d680 con 0x55675f9fbdd0 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78a7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 0x55675f9fb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78a7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 0x55675f9fb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49428/0 (socket says 172.21.15.50:49428) 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78a7fff640 1 -- 172.21.15.50:0/1471550427 learned_addr learned my addr 172.21.15.50:0/1471550427 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78b4fac640 1 --2- 172.21.15.50:0/1471550427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f0db0 0x55675fa0ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78b57ad640 1 --2- 172.21.15.50:0/1471550427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78a7fff640 1 -- 172.21.15.50:0/1471550427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 msgr2=0x55675f9f33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78a7fff640 1 --2- 172.21.15.50:0/1471550427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9f33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78a7fff640 1 -- 172.21.15.50:0/1471550427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f0db0 msgr2=0x55675fa0ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78a7fff640 1 --2- 172.21.15.50:0/1471550427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f0db0 0x55675fa0ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78a7fff640 1 -- 172.21.15.50:0/1471550427 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55675f9f1500 con 0x55675f9f1760 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78b4fac640 1 --2- 172.21.15.50:0/1471550427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f0db0 0x55675fa0ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78a7fff640 1 --2- 172.21.15.50:0/1471550427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 0x55675f9fb420 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f78b006b820 tx=0x7f78a003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581c8b289377542f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.367+0000 7f78967fc640 1 -- 172.21.15.50:0/1471550427 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0035040 con 0x55675f9f1760 2024-04-23T18:18:39.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.368+0000 7f78967fc640 1 -- 172.21.15.50:0/1471550427 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a0041070 con 0x55675f9f1760 2024-04-23T18:18:39.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.368+0000 7f78967fc640 1 -- 172.21.15.50:0/1471550427 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a00039e0 con 0x55675f9f1760 2024-04-23T18:18:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.368+0000 7f78ba003740 1 -- 172.21.15.50:0/1471550427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 msgr2=0x55675f9fb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.368+0000 7f78ba003740 1 --2- 172.21.15.50:0/1471550427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 0x55675f9fb420 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f78b006b820 tx=0x7f78a003f000 comp rx=0 tx=0).stop 2024-04-23T18:18:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.368+0000 7f78ba003740 1 -- 172.21.15.50:0/1471550427 shutdown_connections 2024-04-23T18:18:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.368+0000 7f78ba003740 1 --2- 172.21.15.50:0/1471550427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9fbdd0 0x55675f9f33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.368+0000 7f78ba003740 1 --2- 172.21.15.50:0/1471550427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 0x55675f9fb420 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.368+0000 7f78ba003740 1 --2- 172.21.15.50:0/1471550427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9f0db0 0x55675fa0ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.368+0000 7f78ba003740 1 -- 172.21.15.50:0/1471550427 >> 172.21.15.50:0/1471550427 conn(0x55675f9ed070 msgr2=0x55675f9ed450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.368+0000 7f78ba003740 1 -- 172.21.15.50:0/1471550427 shutdown_connections 2024-04-23T18:18:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.368+0000 7f78ba003740 1 -- 172.21.15.50:0/1471550427 wait complete. 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.368+0000 7f78ba003740 1 -- start start 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78ba003740 1 Processor -- start 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78ba003740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9f0db0 0x55675f9f59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78ba003740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 0x55675f9f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78ba003740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9fbdd0 0x55675f9f6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78ba003740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55675fa0e420 con 0x55675f9f1760 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78ba003740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55675fa0dd50 con 0x55675f9fbdd0 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78ba003740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55675fa0eaf0 con 0x55675f9f0db0 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78ba003740 10 client.0 _ll_register_callbacks cb 0x7f78ac00cab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78a7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 0x55675f9f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78a7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 0x55675f9f5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49440/0 (socket says 172.21.15.50:49440) 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78a7fff640 1 -- 172.21.15.50:0/2117293885 learned_addr learned my addr 172.21.15.50:0/2117293885 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78b57ad640 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9fbdd0 0x55675f9f6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78b4fac640 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9f0db0 0x55675f9f59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78a7fff640 1 -- 172.21.15.50:0/2117293885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9f0db0 msgr2=0x55675f9f59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78a7fff640 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9f0db0 0x55675f9f59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78a7fff640 1 -- 172.21.15.50:0/2117293885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9fbdd0 msgr2=0x55675f9f6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78a7fff640 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9fbdd0 0x55675f9f6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78a7fff640 1 -- 172.21.15.50:0/2117293885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55675fa147d0 con 0x55675f9f1760 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.369+0000 7f78b57ad640 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9fbdd0 0x55675f9f6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f78a7fff640 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 0x55675f9f5f30 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f78a00096c0 tx=0x7f78a0005660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd7f6abc3aad62d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f788ffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f788ffff640 1 -- 172.21.15.50:0/2117293885 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0056070 con 0x55675f9f1760 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f788ffff640 1 -- 172.21.15.50:0/2117293885 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f78a0004890 con 0x55675f9f1760 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f78ba003740 1 -- 172.21.15.50:0/2117293885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78a0041070 con 0x55675f9f1760 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f788ffff640 1 -- 172.21.15.50:0/2117293885 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f78a0004a90 con 0x55675f9f1760 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f78ba003740 10 client.15483 fetch_fsmap learned FSMap version 21 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f78ba003740 1 -- 172.21.15.50:0/2117293885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55675fa73230 con 0x55675f9f1760 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f788ffff640 1 -- 172.21.15.50:0/2117293885 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a0041070 con 0x55675f9f1760 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f788ffff640 1 -- 172.21.15.50:0/2117293885 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f78a00324b0 con 0x55675f9f1760 2024-04-23T18:18:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f78ba003740 10 client.15483 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f78ba003740 10 client.15483 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f78ba003740 20 client.15483 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f78ba003740 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55675fa779c0 0x55675fa766d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f78ba003740 4 client.15483 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.370+0000 7f78ba003740 1 -- 172.21.15.50:0/2117293885 --> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] -- command(tid 0: {"prefix": "status"}) -- 0x55675f95e4e0 con 0x55675fa779c0 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78b4fac640 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55675fa779c0 0x55675fa766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f788ffff640 10 client.15483 ms_handle_connect on v2:172.21.15.188:6836/2190889779 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78b4fac640 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55675fa779c0 0x55675fa766d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=96ad9282e027c8f8 server_cookie=1683b5dde82384eb in_seq=0 out_seq=0 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f788ffff640 1 -- 172.21.15.50:0/2117293885 <== mds.2 v2:172.21.15.188:6836/2190889779 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55675f95e4e0 con 0x55675fa779c0 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f788ffff640 10 client.15483 handle_command_reply: tid=0 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78ba003740 1 client.15483 shutdown 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78ba003740 1 -- 172.21.15.50:0/2117293885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 msgr2=0x55675f9f5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78ba003740 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 0x55675f9f5f30 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f78a00096c0 tx=0x7f78a0005660 comp rx=0 tx=0).stop 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78ba003740 1 -- 172.21.15.50:0/2117293885 shutdown_connections 2024-04-23T18:18:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78ba003740 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.188:6836/2190889779,v1:172.21.15.188:6837/2190889779] conn(0x55675fa779c0 0x55675fa766d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78ba003740 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675f9fbdd0 0x55675f9f6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f788ffff640 0 client.15483 ms_handle_reset on v2:172.21.15.188:6836/2190889779 2024-04-23T18:18:39.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78ba003740 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55675f9f1760 0x55675f9f5f30 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78ba003740 1 --2- 172.21.15.50:0/2117293885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55675f9f0db0 0x55675f9f59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78ba003740 1 -- 172.21.15.50:0/2117293885 >> 172.21.15.50:0/2117293885 conn(0x55675f9ed070 msgr2=0x55675fa07110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78ba003740 1 -- 172.21.15.50:0/2117293885 shutdown_connections 2024-04-23T18:18:39.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.371+0000 7f78ba003740 1 -- 172.21.15.50:0/2117293885 wait complete. 2024-04-23T18:18:39.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78ba003740 20 client.15483 trim_cache size 0 max 16384 2024-04-23T18:18:39.370 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:18:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:18:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 2, 2024-04-23T18:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 24475, 2024-04-23T18:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2024-04-23T18:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2024-04-23T18:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2024-04-23T18:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 122.41107855600001, 2024-04-23T18:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 21, 2024-04-23T18:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 90, 2024-04-23T18:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 89, 2024-04-23T18:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 144.446487515 2024-04-23T18:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:18:39.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78b7210640 1 -- 172.21.15.50:0/1260519402 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f78980a3db0 msgr2=0x7f78980a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78b7210640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f78980a3db0 0x7f78980a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f78a0031440 tx=0x7f78a0005b70 comp rx=0 tx=0).stop 2024-04-23T18:18:39.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78b7210640 1 -- 172.21.15.50:0/1260519402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78b019cf70 msgr2=0x7f78b019d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78b7210640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78b019cf70 0x7f78b019d420 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f78a800eaa0 tx=0x7f78a800ef60 comp rx=0 tx=0).stop 2024-04-23T18:18:39.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78b7210640 1 -- 172.21.15.50:0/1260519402 shutdown_connections 2024-04-23T18:18:39.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78b7210640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f78980a3db0 0x7f78980a6270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78b7210640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78b019cf70 0x7f78b019d420 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78b7210640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b0107270 0x7f78b0198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78b7210640 1 --2- 172.21.15.50:0/1260519402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78b01021a0 0x7f78b0198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78b7210640 1 -- 172.21.15.50:0/1260519402 >> 172.21.15.50:0/1260519402 conn(0x7f78b00fda20 msgr2=0x7f78b00ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78b7210640 1 -- 172.21.15.50:0/1260519402 shutdown_connections 2024-04-23T18:18:39.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.372+0000 7f78b7210640 1 -- 172.21.15.50:0/1260519402 wait complete. 2024-04-23T18:18:39.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-23T18:18:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.486+0000 7f2bc9a60640 1 -- 172.21.15.50:0/3932484563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bc4107250 msgr2=0x7f2bc410ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.486+0000 7f2bc9a60640 1 --2- 172.21.15.50:0/3932484563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bc4107250 0x7f2bc410ee20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009ca0 tx=0x7f2bbc02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.486+0000 7f2bc9a60640 1 -- 172.21.15.50:0/3932484563 shutdown_connections 2024-04-23T18:18:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.486+0000 7f2bc9a60640 1 --2- 172.21.15.50:0/3932484563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bc4107250 0x7f2bc410ee20 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.486+0000 7f2bc9a60640 1 --2- 172.21.15.50:0/3932484563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bc4102b30 0x7f2bc4106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.486+0000 7f2bc9a60640 1 --2- 172.21.15.50:0/3932484563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bc4102180 0x7f2bc4102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.486+0000 7f2bc9a60640 1 -- 172.21.15.50:0/3932484563 >> 172.21.15.50:0/3932484563 conn(0x7f2bc40792d0 msgr2=0x7f2bc40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.486+0000 7f2bc9a60640 1 -- 172.21.15.50:0/3932484563 shutdown_connections 2024-04-23T18:18:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.487+0000 7f2bc9a60640 1 -- 172.21.15.50:0/3932484563 wait complete. 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.487+0000 7f2bc9a60640 1 Processor -- start 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.487+0000 7f2bc9a60640 1 -- start start 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.487+0000 7f2bc9a60640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bc4102180 0x7f2bc4198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.487+0000 7f2bc9a60640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bc4107250 0x7f2bc4198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.487+0000 7f2bc9a60640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bc419cf90 0x7f2bc419d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.487+0000 7f2bc9a60640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc41121a0 con 0x7f2bc4107250 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.487+0000 7f2bc9a60640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc4112020 con 0x7f2bc419cf90 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.487+0000 7f2bc9a60640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc4112320 con 0x7f2bc4102180 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.487+0000 7f2bbbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bc4107250 0x7f2bc4198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.487+0000 7f2bbbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bc4107250 0x7f2bc4198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49460/0 (socket says 172.21.15.50:49460) 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.487+0000 7f2bbbfff640 1 -- 172.21.15.50:0/2545510252 learned_addr learned my addr 172.21.15.50:0/2545510252 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bc37fe640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bc419cf90 0x7f2bc419d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bc2ffd640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bc4102180 0x7f2bc4198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bbbfff640 1 -- 172.21.15.50:0/2545510252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bc4102180 msgr2=0x7f2bc4198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bbbfff640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bc4102180 0x7f2bc4198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bbbfff640 1 -- 172.21.15.50:0/2545510252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bc419cf90 msgr2=0x7f2bc419d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bbbfff640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bc419cf90 0x7f2bc419d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bbbfff640 1 -- 172.21.15.50:0/2545510252 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc419da80 con 0x7f2bc4107250 2024-04-23T18:18:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bc2ffd640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bc4102180 0x7f2bc4198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bbbfff640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bc4107250 0x7f2bc4198c60 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00d770 tx=0x7f2bac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bc8a5e640 1 -- 172.21.15.50:0/2545510252 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac017070 con 0x7f2bc4107250 2024-04-23T18:18:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bc9a60640 1 -- 172.21.15.50:0/2545510252 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc419dd70 con 0x7f2bc4107250 2024-04-23T18:18:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bc8a5e640 1 -- 172.21.15.50:0/2545510252 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bac009c30 con 0x7f2bc4107250 2024-04-23T18:18:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bc8a5e640 1 -- 172.21.15.50:0/2545510252 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac00f620 con 0x7f2bc4107250 2024-04-23T18:18:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.488+0000 7f2bc9a60640 1 -- 172.21.15.50:0/2545510252 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc41a5560 con 0x7f2bc4107250 2024-04-23T18:18:39.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.490+0000 7f2bc8a5e640 1 -- 172.21.15.50:0/2545510252 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f2bac01a070 con 0x7f2bc4107250 2024-04-23T18:18:39.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bcc775740 1 Processor -- start 2024-04-23T18:18:39.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bcc775740 1 -- start start 2024-04-23T18:18:39.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bcc775740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeecc50 0x55700aeed030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bcc775740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aef7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bcc775740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aef8210 0x55700aef8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bcc775740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55700ae52c50 con 0x55700aeed600 2024-04-23T18:18:39.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bcc775740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55700adef830 con 0x55700aeecc50 2024-04-23T18:18:39.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bcc775740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55700ae4ec10 con 0x55700aef8210 2024-04-23T18:18:39.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bc2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aef7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bc2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aef7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49464/0 (socket says 172.21.15.50:49464) 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bc2ffd640 1 -- 172.21.15.50:0/3084514213 learned_addr learned my addr 172.21.15.50:0/3084514213 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bc8a5e640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2bb00a3db0 0x7f2bb00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bbbfff640 1 --2- 172.21.15.50:0/3084514213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aef8210 0x55700aef8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.491+0000 7f2bc37fe640 1 --2- 172.21.15.50:0/3084514213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeecc50 0x55700aeed030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.492+0000 7f2bc37fe640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2bb00a3db0 0x7f2bb00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.492+0000 7f2bc2ffd640 1 -- 172.21.15.50:0/3084514213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aef8210 msgr2=0x55700aef8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.492+0000 7f2bc2ffd640 1 --2- 172.21.15.50:0/3084514213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aef8210 0x55700aef8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.492+0000 7f2bc2ffd640 1 -- 172.21.15.50:0/3084514213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeecc50 msgr2=0x55700aeed030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.492+0000 7f2bc2ffd640 1 --2- 172.21.15.50:0/3084514213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeecc50 0x55700aeed030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.492+0000 7f2bc2ffd640 1 -- 172.21.15.50:0/3084514213 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55700ab652d0 con 0x55700aeed600 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.492+0000 7f2bc8a5e640 1 -- 172.21.15.50:0/2545510252 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f2bac0d0470 con 0x7f2bc4107250 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.492+0000 7f2bc2ffd640 1 --2- 172.21.15.50:0/3084514213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aef7bf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4009d50 tx=0x7f2bb40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1318f02948567c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.492+0000 7f2bb9ffb640 1 -- 172.21.15.50:0/3084514213 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb4012650 con 0x55700aeed600 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.492+0000 7f2bb9ffb640 1 -- 172.21.15.50:0/3084514213 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bb4009e40 con 0x55700aeed600 2024-04-23T18:18:39.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.492+0000 7f2bc37fe640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2bb00a3db0 0x7f2bb00a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc005fe0 tx=0x7f2bbc0095c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.492+0000 7f2bb9ffb640 1 -- 172.21.15.50:0/3084514213 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb400fc00 con 0x55700aeed600 2024-04-23T18:18:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.493+0000 7f2bcc775740 1 -- 172.21.15.50:0/3084514213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 msgr2=0x55700aef7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.493+0000 7f2bcc775740 1 --2- 172.21.15.50:0/3084514213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aef7bf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4009d50 tx=0x7f2bb40117b0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.493+0000 7f2bcc775740 1 -- 172.21.15.50:0/3084514213 shutdown_connections 2024-04-23T18:18:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.493+0000 7f2bcc775740 1 --2- 172.21.15.50:0/3084514213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aef8210 0x55700aef8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.493+0000 7f2bcc775740 1 --2- 172.21.15.50:0/3084514213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aef7bf0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.493+0000 7f2bcc775740 1 --2- 172.21.15.50:0/3084514213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeecc50 0x55700aeed030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.493+0000 7f2bcc775740 1 -- 172.21.15.50:0/3084514213 >> 172.21.15.50:0/3084514213 conn(0x55700aee9070 msgr2=0x55700aee9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.493+0000 7f2bcc775740 1 -- 172.21.15.50:0/3084514213 shutdown_connections 2024-04-23T18:18:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.493+0000 7f2bcc775740 1 -- 172.21.15.50:0/3084514213 wait complete. 2024-04-23T18:18:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.493+0000 7f2bcc775740 1 -- start start 2024-04-23T18:18:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.493+0000 7f2bcc775740 1 Processor -- start 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bcc775740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aeecc50 0x55700aefbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bcc775740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aefc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bcc775740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aef8210 0x55700aefc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bcc775740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55700aeff190 con 0x55700aeed600 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bcc775740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55700aded470 con 0x55700aef8210 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bcc775740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55700aeff310 con 0x55700aeecc50 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bcc775740 10 client.0 _ll_register_callbacks cb 0x55700ae12f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bbbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aefc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bbbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aefc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49480/0 (socket says 172.21.15.50:49480) 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bbbfff640 1 -- 172.21.15.50:0/2615235014 learned_addr learned my addr 172.21.15.50:0/2615235014 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bc2ffd640 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aeecc50 0x55700aefbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bc37fe640 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aef8210 0x55700aefc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bbbfff640 1 -- 172.21.15.50:0/2615235014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aeecc50 msgr2=0x55700aefbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bbbfff640 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aeecc50 0x55700aefbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bbbfff640 1 -- 172.21.15.50:0/2615235014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aef8210 msgr2=0x55700aefc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bbbfff640 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aef8210 0x55700aefc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bbbfff640 1 -- 172.21.15.50:0/2615235014 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55700ac2f6b0 con 0x55700aeed600 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bbbfff640 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aefc0b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2bac098c80 tx=0x7f2bac09beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4a69a47f941593 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2ba2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2ba2ffd640 1 -- 172.21.15.50:0/2615235014 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac021070 con 0x55700aeed600 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2ba2ffd640 1 -- 172.21.15.50:0/2615235014 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f2bac0cddc0 con 0x55700aeed600 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.494+0000 7f2bcc775740 1 -- 172.21.15.50:0/2615235014 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55700ae60670 con 0x55700aeed600 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.495+0000 7f2ba2ffd640 1 -- 172.21.15.50:0/2615235014 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2bac0b5050 con 0x55700aeed600 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.495+0000 7f2bcc775740 10 client.15495 fetch_fsmap learned FSMap version 21 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.495+0000 7f2bcc775740 1 -- 172.21.15.50:0/2615235014 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55700af0b450 con 0x55700aeed600 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.495+0000 7f2ba2ffd640 1 -- 172.21.15.50:0/2615235014 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55700ae60670 con 0x55700aeed600 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.495+0000 7f2ba2ffd640 1 -- 172.21.15.50:0/2615235014 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2bac0a0110 con 0x55700aeed600 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.495+0000 7f2bcc775740 10 client.15495 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:18:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.495+0000 7f2bcc775740 10 client.15495 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.495+0000 7f2bcc775740 20 client.15495 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.495+0000 7f2bcc775740 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55700af0da30 0x55700af0eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.495+0000 7f2bcc775740 4 client.15495 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.495+0000 7f2bcc775740 1 -- 172.21.15.50:0/2615235014 --> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55700ae5a4e0 con 0x55700af0da30 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.495+0000 7f2bc2ffd640 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55700af0da30 0x55700af0eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.496+0000 7f2ba2ffd640 10 client.15495 ms_handle_connect on v2:172.21.15.188:6838/2202634301 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.496+0000 7f2bc2ffd640 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55700af0da30 0x55700af0eea0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=29715cce7d432d00 server_cookie=efce9cfc0c11581 in_seq=0 out_seq=0 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2ba2ffd640 1 -- 172.21.15.50:0/2615235014 <== mds.1 v2:172.21.15.188:6838/2202634301 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55700ae5a4e0 con 0x55700af0da30 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2ba2ffd640 10 client.15495 handle_command_reply: tid=0 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2bcc775740 1 client.15495 shutdown 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2bcc775740 1 -- 172.21.15.50:0/2615235014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 msgr2=0x55700aefc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2bcc775740 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aefc0b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2bac098c80 tx=0x7f2bac09beb0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2bcc775740 1 -- 172.21.15.50:0/2615235014 shutdown_connections 2024-04-23T18:18:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2bcc775740 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55700af0da30 0x55700af0eea0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2bcc775740 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aef8210 0x55700aefc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2bcc775740 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aefc0b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2ba2ffd640 0 client.15495 ms_handle_reset on v2:172.21.15.188:6838/2202634301 2024-04-23T18:18:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2bcc775740 1 --2- 172.21.15.50:0/2615235014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aeecc50 0x55700aefbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2bcc775740 1 -- 172.21.15.50:0/2615235014 >> 172.21.15.50:0/2615235014 conn(0x55700aee9070 msgr2=0x55700aeeeec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2bcc775740 1 -- 172.21.15.50:0/2615235014 shutdown_connections 2024-04-23T18:18:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2bcc775740 1 -- 172.21.15.50:0/2615235014 wait complete. 2024-04-23T18:18:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.497+0000 7f2bcc775740 20 client.15495 trim_cache size 0 max 16384 2024-04-23T18:18:39.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bcc775740 1 Processor -- start 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bcc775740 1 -- start start 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bcc775740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aeecc50 0x55700aeef150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bcc775740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeed600 0x55700aee9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bcc775740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aef8210 0x55700aef9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bcc775740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55700af0d1e0 con 0x55700aef8210 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bcc775740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55700af0d8b0 con 0x55700aeed600 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bcc775740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55700af0df80 con 0x55700aeecc50 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bc37fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aef8210 0x55700aef9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bc37fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aef8210 0x55700aef9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49492/0 (socket says 172.21.15.50:49492) 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bc37fe640 1 -- 172.21.15.50:0/3278823610 learned_addr learned my addr 172.21.15.50:0/3278823610 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bbbfff640 1 --2- 172.21.15.50:0/3278823610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeed600 0x55700aee9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bc37fe640 1 -- 172.21.15.50:0/3278823610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aeecc50 msgr2=0x55700aeef150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bc37fe640 1 --2- 172.21.15.50:0/3278823610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aeecc50 0x55700aeef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bc37fe640 1 -- 172.21.15.50:0/3278823610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeed600 msgr2=0x55700aee9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bc37fe640 1 --2- 172.21.15.50:0/3278823610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeed600 0x55700aee9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bc37fe640 1 -- 172.21.15.50:0/3278823610 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55700aee4700 con 0x55700aef8210 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.516+0000 7f2bc37fe640 1 --2- 172.21.15.50:0/3278823610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aef8210 0x55700aef9600 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc032ea0 tx=0x7f2bbc042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e9a459b8a7608 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bb97fa640 1 -- 172.21.15.50:0/3278823610 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc054070 con 0x55700aef8210 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bb97fa640 1 -- 172.21.15.50:0/3278823610 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bbc036070 con 0x55700aef8210 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bb97fa640 1 -- 172.21.15.50:0/3278823610 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc0045a0 con 0x55700aef8210 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bcc775740 1 -- 172.21.15.50:0/3278823610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aef8210 msgr2=0x55700aef9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bcc775740 1 --2- 172.21.15.50:0/3278823610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aef8210 0x55700aef9600 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc032ea0 tx=0x7f2bbc042000 comp rx=0 tx=0).stop 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bcc775740 1 -- 172.21.15.50:0/3278823610 shutdown_connections 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bcc775740 1 --2- 172.21.15.50:0/3278823610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aef8210 0x55700aef9600 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bcc775740 1 --2- 172.21.15.50:0/3278823610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeed600 0x55700aee9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bcc775740 1 --2- 172.21.15.50:0/3278823610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aeecc50 0x55700aeef150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bcc775740 1 -- 172.21.15.50:0/3278823610 >> 172.21.15.50:0/3278823610 conn(0x55700aee9070 msgr2=0x55700aef5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bcc775740 1 -- 172.21.15.50:0/3278823610 shutdown_connections 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bcc775740 1 -- 172.21.15.50:0/3278823610 wait complete. 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.517+0000 7f2bcc775740 1 -- start start 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bcc775740 1 Processor -- start 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bcc775740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeecc50 0x55700af17260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bcc775740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aef8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bcc775740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aef8210 0x55700aef8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bcc775740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55700af0ed20 con 0x55700aeed600 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bcc775740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55700af0e650 con 0x55700aeecc50 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bcc775740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55700af0f3f0 con 0x55700aef8210 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bcc775740 10 client.0 _ll_register_callbacks cb 0x7f2bac0cd9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bbbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aef8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bbbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aef8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49494/0 (socket says 172.21.15.50:49494) 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bbbfff640 1 -- 172.21.15.50:0/2073384118 learned_addr learned my addr 172.21.15.50:0/2073384118 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bc2ffd640 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeecc50 0x55700af17260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bc37fe640 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aef8210 0x55700aef8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bbbfff640 1 -- 172.21.15.50:0/2073384118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aef8210 msgr2=0x55700aef8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bbbfff640 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aef8210 0x55700aef8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bbbfff640 1 -- 172.21.15.50:0/2073384118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeecc50 msgr2=0x55700af17260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bbbfff640 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeecc50 0x55700af17260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.518+0000 7f2bbbfff640 1 -- 172.21.15.50:0/2073384118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55700aef90e0 con 0x55700aeed600 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2bbbfff640 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aef8630 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00d740 tx=0x7f2bac0a0340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896b8ec04325dc00 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2ba2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2ba2ffd640 1 -- 172.21.15.50:0/2073384118 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac0b9030 con 0x55700aeed600 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2ba2ffd640 1 -- 172.21.15.50:0/2073384118 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f2bac098170 con 0x55700aeed600 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2bcc775740 1 -- 172.21.15.50:0/2073384118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bbc036070 con 0x55700aeed600 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2ba2ffd640 1 -- 172.21.15.50:0/2073384118 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2bac021040 con 0x55700aeed600 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2bcc775740 10 client.15501 fetch_fsmap learned FSMap version 21 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2bcc775740 1 -- 172.21.15.50:0/2073384118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55700af15ba0 con 0x55700aeed600 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2ba2ffd640 1 -- 172.21.15.50:0/2073384118 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bbc036070 con 0x55700aeed600 2024-04-23T18:18:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2ba2ffd640 1 -- 172.21.15.50:0/2073384118 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2bac0cddc0 con 0x55700aeed600 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2bcc775740 10 client.15501 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2bcc775740 10 client.15501 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2bcc775740 20 client.15501 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2bcc775740 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55700af78300 0x55700af77010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2bcc775740 4 client.15501 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.519+0000 7f2bcc775740 1 -- 172.21.15.50:0/2073384118 --> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] -- command(tid 0: {"prefix": "status"}) -- 0x55700add69a0 con 0x55700af78300 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bc2ffd640 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55700af78300 0x55700af77010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bc2ffd640 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55700af78300 0x55700af77010 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1189398eca1f23fa server_cookie=907d227accd3484f in_seq=0 out_seq=0 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2ba2ffd640 10 client.15501 ms_handle_connect on v2:172.21.15.188:6838/2202634301 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2ba2ffd640 1 -- 172.21.15.50:0/2073384118 <== mds.1 v2:172.21.15.188:6838/2202634301 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55700add69a0 con 0x55700af78300 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2ba2ffd640 10 client.15501 handle_command_reply: tid=0 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bcc775740 1 client.15501 shutdown 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bcc775740 1 -- 172.21.15.50:0/2073384118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 msgr2=0x55700aef8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bcc775740 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aef8630 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00d740 tx=0x7f2bac0a0340 comp rx=0 tx=0).stop 2024-04-23T18:18:39.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bcc775740 1 -- 172.21.15.50:0/2073384118 shutdown_connections 2024-04-23T18:18:39.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bcc775740 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.188:6838/2202634301,v1:172.21.15.188:6839/2202634301] conn(0x55700af78300 0x55700af77010 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bcc775740 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55700aef8210 0x55700aef8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bcc775740 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55700aeed600 0x55700aef8630 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bcc775740 1 --2- 172.21.15.50:0/2073384118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55700aeecc50 0x55700af17260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2ba2ffd640 0 client.15501 ms_handle_reset on v2:172.21.15.188:6838/2202634301 2024-04-23T18:18:39.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bcc775740 1 -- 172.21.15.50:0/2073384118 >> 172.21.15.50:0/2073384118 conn(0x55700aee9070 msgr2=0x55700aef6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bcc775740 1 -- 172.21.15.50:0/2073384118 shutdown_connections 2024-04-23T18:18:39.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.520+0000 7f2bcc775740 1 -- 172.21.15.50:0/2073384118 wait complete. 2024-04-23T18:18:39.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bcc775740 20 client.15501 trim_cache size 0 max 16384 2024-04-23T18:18:39.519 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:18:39.519 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:18:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 1, 2024-04-23T18:18:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 24499, 2024-04-23T18:18:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2024-04-23T18:18:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2024-04-23T18:18:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2024-04-23T18:18:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 123.571419365, 2024-04-23T18:18:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 21, 2024-04-23T18:18:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 90, 2024-04-23T18:18:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 89, 2024-04-23T18:18:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 139.059767529 2024-04-23T18:18:39.520 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:18:39.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bc9a60640 1 -- 172.21.15.50:0/2545510252 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2bb00a3db0 msgr2=0x7f2bb00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bc9a60640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2bb00a3db0 0x7f2bb00a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc005fe0 tx=0x7f2bbc0095c0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bc9a60640 1 -- 172.21.15.50:0/2545510252 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bc4107250 msgr2=0x7f2bc4198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bc9a60640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bc4107250 0x7f2bc4198c60 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00d770 tx=0x7f2bac00dc30 comp rx=0 tx=0).stop 2024-04-23T18:18:39.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bc9a60640 1 -- 172.21.15.50:0/2545510252 shutdown_connections 2024-04-23T18:18:39.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bc9a60640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2bb00a3db0 0x7f2bb00a6270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bc9a60640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bc419cf90 0x7f2bc419d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bc9a60640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bc4107250 0x7f2bc4198c60 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bc9a60640 1 --2- 172.21.15.50:0/2545510252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bc4102180 0x7f2bc4198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bc9a60640 1 -- 172.21.15.50:0/2545510252 >> 172.21.15.50:0/2545510252 conn(0x7f2bc40792d0 msgr2=0x7f2bc4100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bc9a60640 1 -- 172.21.15.50:0/2545510252 shutdown_connections 2024-04-23T18:18:39.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.521+0000 7f2bc9a60640 1 -- 172.21.15.50:0/2545510252 wait complete. 2024-04-23T18:18:39.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T18:18:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.636+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/2351710966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc4410c600 msgr2=0x7fcc4410ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.636+0000 7fcc4b8ae640 1 --2- 172.21.15.50:0/2351710966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc4410c600 0x7fcc4410ea50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c009ca0 tx=0x7fcc3c02a950 comp rx=0 tx=0).stop 2024-04-23T18:18:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.636+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/2351710966 shutdown_connections 2024-04-23T18:18:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.636+0000 7fcc4b8ae640 1 --2- 172.21.15.50:0/2351710966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc4410c600 0x7fcc4410ea50 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.636+0000 7fcc4b8ae640 1 --2- 172.21.15.50:0/2351710966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc44103b50 0x7fcc4410c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.636+0000 7fcc4b8ae640 1 --2- 172.21.15.50:0/2351710966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc44103230 0x7fcc44103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.636+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/2351710966 >> 172.21.15.50:0/2351710966 conn(0x7fcc44077f80 msgr2=0x7fcc44078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.636+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/2351710966 shutdown_connections 2024-04-23T18:18:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.636+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/2351710966 wait complete. 2024-04-23T18:18:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.636+0000 7fcc4b8ae640 1 Processor -- start 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4b8ae640 1 -- start start 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4b8ae640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc44103230 0x7fcc44198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4b8ae640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc44103b50 0x7fcc44198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4b8ae640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc4410c600 0x7fcc4419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4b8ae640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcc44112210 con 0x7fcc44103230 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4b8ae640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcc44112090 con 0x7fcc4410c600 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4b8ae640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcc44112390 con 0x7fcc44103b50 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4964a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc44103230 0x7fcc44198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4964a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc44103230 0x7fcc44198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49514/0 (socket says 172.21.15.50:49514) 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4964a640 1 -- 172.21.15.50:0/1375228987 learned_addr learned my addr 172.21.15.50:0/1375228987 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc48e49640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc44103b50 0x7fcc44198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc49e4b640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc4410c600 0x7fcc4419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4964a640 1 -- 172.21.15.50:0/1375228987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc44103b50 msgr2=0x7fcc44198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4964a640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc44103b50 0x7fcc44198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4964a640 1 -- 172.21.15.50:0/1375228987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc4410c600 msgr2=0x7fcc4419d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4964a640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc4410c600 0x7fcc4419d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4964a640 1 -- 172.21.15.50:0/1375228987 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4419d740 con 0x7fcc44103230 2024-04-23T18:18:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc49e4b640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc4410c600 0x7fcc4419d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.637+0000 7fcc4964a640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc44103230 0x7fcc44198750 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000d950 tx=0x7fcc4000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.638+0000 7fcc3a7fc640 1 -- 172.21.15.50:0/1375228987 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40017070 con 0x7fcc44103230 2024-04-23T18:18:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.638+0000 7fcc3a7fc640 1 -- 172.21.15.50:0/1375228987 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc40009c30 con 0x7fcc44103230 2024-04-23T18:18:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.638+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/1375228987 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4419da30 con 0x7fcc44103230 2024-04-23T18:18:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.638+0000 7fcc3a7fc640 1 -- 172.21.15.50:0/1375228987 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4000f620 con 0x7fcc44103230 2024-04-23T18:18:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.638+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/1375228987 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc441a52c0 con 0x7fcc44103230 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.640+0000 7fcc3a7fc640 1 -- 172.21.15.50:0/1375228987 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fcc40010460 con 0x7fcc44103230 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.640+0000 7fcc4e5c6740 1 Processor -- start 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.640+0000 7fcc4e5c6740 1 -- start start 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc4e5c6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180cc4db0 0x557180cc5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc4e5c6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 0x557180ccf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc4e5c6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180ccfdd0 0x557180cd2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc4e5c6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557180c2ac50 con 0x557180ccfdd0 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc4e5c6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557180bc7830 con 0x557180cc5760 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc4e5c6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557180c26c10 con 0x557180cc4db0 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc48e49640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180ccfdd0 0x557180cd2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc3a7fc640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcc300a3e30 0x7fcc300a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc48e49640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180ccfdd0 0x557180cd2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49526/0 (socket says 172.21.15.50:49526) 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc48e49640 1 -- 172.21.15.50:0/1198276833 learned_addr learned my addr 172.21.15.50:0/1198276833 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc49e4b640 1 --2- 172.21.15.50:0/1198276833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180cc4db0 0x557180cc5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc49e4b640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcc300a3e30 0x7fcc300a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc4964a640 1 --2- 172.21.15.50:0/1198276833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 0x557180ccf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc3a7fc640 1 -- 172.21.15.50:0/1375228987 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fcc400d0590 con 0x7fcc44103230 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc48e49640 1 -- 172.21.15.50:0/1198276833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180cc4db0 msgr2=0x557180cc5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc48e49640 1 --2- 172.21.15.50:0/1198276833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180cc4db0 0x557180cc5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc48e49640 1 -- 172.21.15.50:0/1198276833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 msgr2=0x557180ccf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc48e49640 1 --2- 172.21.15.50:0/1198276833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 0x557180ccf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc48e49640 1 -- 172.21.15.50:0/1198276833 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55718093d2d0 con 0x557180ccfdd0 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc4964a640 1 --2- 172.21.15.50:0/1198276833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 0x557180ccf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc48e49640 1 --2- 172.21.15.50:0/1198276833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180ccfdd0 0x557180cd2220 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009c10 tx=0x7fcc34011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d38d1bb05c868c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc49e4b640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcc300a3e30 0x7fcc300a62f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c030040 tx=0x7fcc3c0095c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.641+0000 7fcc2f7fe640 1 -- 172.21.15.50:0/1198276833 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc34004230 con 0x557180ccfdd0 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.642+0000 7fcc2f7fe640 1 -- 172.21.15.50:0/1198276833 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc340043d0 con 0x557180ccfdd0 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.642+0000 7fcc2f7fe640 1 -- 172.21.15.50:0/1198276833 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc34004d60 con 0x557180ccfdd0 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.642+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/1198276833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180ccfdd0 msgr2=0x557180cd2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.642+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/1198276833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180ccfdd0 0x557180cd2220 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009c10 tx=0x7fcc34011810 comp rx=0 tx=0).stop 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.642+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/1198276833 shutdown_connections 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.642+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/1198276833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180ccfdd0 0x557180cd2220 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.642+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/1198276833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 0x557180ccf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.642+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/1198276833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180cc4db0 0x557180cc5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.642+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/1198276833 >> 172.21.15.50:0/1198276833 conn(0x557180cc1070 msgr2=0x557180cc1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.642+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/1198276833 shutdown_connections 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.642+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/1198276833 wait complete. 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4e5c6740 1 -- start start 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4e5c6740 1 Processor -- start 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4e5c6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180cccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4e5c6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 0x557180ccd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4e5c6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180ccfdd0 0x557180ccd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4e5c6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557180cd2d40 con 0x557180cc4db0 2024-04-23T18:18:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4e5c6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557180bc5470 con 0x557180cc5760 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4e5c6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557180cd2ec0 con 0x557180ccfdd0 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4e5c6740 10 client.0 _ll_register_callbacks cb 0x557180beaf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4964a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180cccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4964a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180cccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49540/0 (socket says 172.21.15.50:49540) 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4964a640 1 -- 172.21.15.50:0/2552438443 learned_addr learned my addr 172.21.15.50:0/2552438443 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc48e49640 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 0x557180ccd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc49e4b640 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180ccfdd0 0x557180ccd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4964a640 1 -- 172.21.15.50:0/2552438443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180ccfdd0 msgr2=0x557180ccd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4964a640 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180ccfdd0 0x557180ccd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4964a640 1 -- 172.21.15.50:0/2552438443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 msgr2=0x557180ccd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.643+0000 7fcc4964a640 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 0x557180ccd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc4964a640 1 -- 172.21.15.50:0/2552438443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557180a076b0 con 0x557180cc4db0 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc4964a640 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180cccc90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcc40098810 tx=0x7fcc400209d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c815943ce12eebb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc2cff9640 1 -- 172.21.15.50:0/2552438443 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40020040 con 0x557180cc4db0 2024-04-23T18:18:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc2cff9640 1 -- 172.21.15.50:0/2552438443 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fcc40016170 con 0x557180cc4db0 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/2552438443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc340043d0 con 0x557180cc4db0 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc2cff9640 1 -- 172.21.15.50:0/2552438443 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcc40009de0 con 0x557180cc4db0 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc4e5c6740 10 client.15513 fetch_fsmap learned FSMap version 21 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/2552438443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557180cdefb0 con 0x557180cc4db0 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc2cff9640 1 -- 172.21.15.50:0/2552438443 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc400b0e50 con 0x557180cc4db0 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc2cff9640 1 -- 172.21.15.50:0/2552438443 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcc400cdec0 con 0x557180cc4db0 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc4e5c6740 10 client.15513 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc4e5c6740 10 client.15513 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc4e5c6740 20 client.15513 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557180ce15c0 0x557180ce2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc4e5c6740 4 client.15513 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.644+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/2552438443 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557180c38670 con 0x557180ce15c0 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.645+0000 7fcc48e49640 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557180ce15c0 0x557180ce2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.645+0000 7fcc48e49640 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557180ce15c0 0x557180ce2a30 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f163483594be0a5 server_cookie=2ebabc309047eac0 in_seq=0 out_seq=0 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.645+0000 7fcc2cff9640 10 client.15513 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc2cff9640 1 -- 172.21.15.50:0/2552438443 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557180c38670 con 0x557180ce15c0 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc2cff9640 10 client.15513 handle_command_reply: tid=0 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc4e5c6740 1 client.15513 shutdown 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/2552438443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 msgr2=0x557180cccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180cccc90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcc40098810 tx=0x7fcc400209d0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/2552438443 shutdown_connections 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557180ce15c0 0x557180ce2a30 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180ccfdd0 0x557180ccd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 0x557180ccd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc2cff9640 0 client.15513 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/2552438443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180cccc90 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/2552438443 >> 172.21.15.50:0/2552438443 conn(0x557180cc1070 msgr2=0x557180cc6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/2552438443 shutdown_connections 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/2552438443 wait complete. 2024-04-23T18:18:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.646+0000 7fcc4e5c6740 20 client.15513 trim_cache size 0 max 16384 2024-04-23T18:18:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.664+0000 7fcc4e5c6740 1 Processor -- start 2024-04-23T18:18:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.664+0000 7fcc4e5c6740 1 -- start start 2024-04-23T18:18:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4e5c6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180cc7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4e5c6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 0x557180cc9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4e5c6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180ccfdd0 0x557180cca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4e5c6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557180ce0d70 con 0x557180cc4db0 2024-04-23T18:18:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4e5c6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557180ce1440 con 0x557180cc5760 2024-04-23T18:18:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4e5c6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557180ce1b10 con 0x557180ccfdd0 2024-04-23T18:18:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4964a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180cc7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4964a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180cc7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49554/0 (socket says 172.21.15.50:49554) 2024-04-23T18:18:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4964a640 1 -- 172.21.15.50:0/456035112 learned_addr learned my addr 172.21.15.50:0/456035112 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4964a640 1 -- 172.21.15.50:0/456035112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180ccfdd0 msgr2=0x557180cca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4964a640 1 --2- 172.21.15.50:0/456035112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180ccfdd0 0x557180cca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4964a640 1 -- 172.21.15.50:0/456035112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 msgr2=0x557180cc9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4964a640 1 --2- 172.21.15.50:0/456035112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 0x557180cc9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4964a640 1 -- 172.21.15.50:0/456035112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557180ccb1e0 con 0x557180cc4db0 2024-04-23T18:18:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc4964a640 1 --2- 172.21.15.50:0/456035112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180cc7360 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcc4009c340 tx=0x7fcc400a50a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb77bf03f760173c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc2effd640 1 -- 172.21.15.50:0/456035112 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40020070 con 0x557180cc4db0 2024-04-23T18:18:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc2effd640 1 -- 172.21.15.50:0/456035112 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc400b0e50 con 0x557180cc4db0 2024-04-23T18:18:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.665+0000 7fcc2effd640 1 -- 172.21.15.50:0/456035112 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40016170 con 0x557180cc4db0 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.666+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/456035112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 msgr2=0x557180cc7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.666+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/456035112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180cc7360 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcc4009c340 tx=0x7fcc400a50a0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.666+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/456035112 shutdown_connections 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.666+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/456035112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180ccfdd0 0x557180cca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.666+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/456035112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180cc5760 0x557180cc9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.666+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/456035112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180cc7360 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.666+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/456035112 >> 172.21.15.50:0/456035112 conn(0x557180cc1070 msgr2=0x557180cc1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.666+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/456035112 shutdown_connections 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.666+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/456035112 wait complete. 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.666+0000 7fcc4e5c6740 1 -- start start 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.666+0000 7fcc4e5c6740 1 Processor -- start 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4e5c6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180ceb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4e5c6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180cc5760 0x557180ce4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4e5c6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180ccfdd0 0x557180ce49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4e5c6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557180ce28b0 con 0x557180cc4db0 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4e5c6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557180ce21e0 con 0x557180ccfdd0 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4e5c6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557180ce2f80 con 0x557180cc5760 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4e5c6740 10 client.0 _ll_register_callbacks cb 0x7fcc400cdaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4964a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180ceb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4964a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180ceb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49568/0 (socket says 172.21.15.50:49568) 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4964a640 1 -- 172.21.15.50:0/384302597 learned_addr learned my addr 172.21.15.50:0/384302597 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc48e49640 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180cc5760 0x557180ce4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4964a640 1 -- 172.21.15.50:0/384302597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180cc5760 msgr2=0x557180ce4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4964a640 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180cc5760 0x557180ce4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc49e4b640 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180ccfdd0 0x557180ce49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4964a640 1 -- 172.21.15.50:0/384302597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180ccfdd0 msgr2=0x557180ce49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4964a640 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180ccfdd0 0x557180ce49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4964a640 1 -- 172.21.15.50:0/384302597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557180ce51d0 con 0x557180cc4db0 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4964a640 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180ceb0d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcc400a4af0 tx=0x7fcc400ce160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e983ff9d0bf8fc0c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc2cff9640 1 -- 172.21.15.50:0/384302597 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40020070 con 0x557180cc4db0 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc2cff9640 1 -- 172.21.15.50:0/384302597 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fcc40098300 con 0x557180cc4db0 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/384302597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc400b0e50 con 0x557180cc4db0 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.667+0000 7fcc2cff9640 1 -- 172.21.15.50:0/384302597 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcc4001e120 con 0x557180cc4db0 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.668+0000 7fcc4e5c6740 10 client.15519 fetch_fsmap learned FSMap version 21 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.668+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/384302597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557180d47630 con 0x557180cc4db0 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.668+0000 7fcc2cff9640 1 -- 172.21.15.50:0/384302597 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc400b0e50 con 0x557180cc4db0 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.668+0000 7fcc2cff9640 1 -- 172.21.15.50:0/384302597 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcc4009b400 con 0x557180cc4db0 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.668+0000 7fcc4e5c6740 10 client.15519 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.668+0000 7fcc4e5c6740 10 client.15519 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T18:18:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.668+0000 7fcc4e5c6740 20 client.15519 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.668+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557180d4bb60 0x557180d4a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.668+0000 7fcc4e5c6740 4 client.15519 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.668+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/384302597 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "status"}) -- 0x557180c324e0 con 0x557180d4bb60 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.668+0000 7fcc48e49640 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557180d4bb60 0x557180d4a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc48e49640 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557180d4bb60 0x557180d4a870 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6863d0b140a6997c server_cookie=3bb9ce16582bd691 in_seq=0 out_seq=0 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc2cff9640 10 client.15519 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc2cff9640 1 -- 172.21.15.50:0/384302597 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x557180c324e0 con 0x557180d4bb60 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc2cff9640 10 client.15519 handle_command_reply: tid=0 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc4e5c6740 1 client.15519 shutdown 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/384302597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 msgr2=0x557180ceb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180ceb0d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcc400a4af0 tx=0x7fcc400ce160 comp rx=0 tx=0).stop 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/384302597 shutdown_connections 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557180d4bb60 0x557180d4a870 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557180ccfdd0 0x557180ce49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc2cff9640 0 client.15519 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557180cc5760 0x557180ce4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc4e5c6740 1 --2- 172.21.15.50:0/384302597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557180cc4db0 0x557180ceb0d0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/384302597 >> 172.21.15.50:0/384302597 conn(0x557180cc1070 msgr2=0x557180ccb560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/384302597 shutdown_connections 2024-04-23T18:18:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc4e5c6740 1 -- 172.21.15.50:0/384302597 wait complete. 2024-04-23T18:18:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.669+0000 7fcc4e5c6740 20 client.15519 trim_cache size 0 max 16384 2024-04-23T18:18:39.668 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:18:39.668 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "5d677974-019c-11ef-bc93-c7b262605968", 2024-04-23T18:18:39.668 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2024-04-23T18:18:39.668 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 24506, 2024-04-23T18:18:39.668 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2024-04-23T18:18:39.669 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2024-04-23T18:18:39.669 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2024-04-23T18:18:39.669 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 125.701193657, 2024-04-23T18:18:39.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 21, 2024-04-23T18:18:39.669 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 90, 2024-04-23T18:18:39.669 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 89, 2024-04-23T18:18:39.669 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 130.870309642 2024-04-23T18:18:39.669 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:18:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.670+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/1375228987 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcc300a3e30 msgr2=0x7fcc300a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.670+0000 7fcc4b8ae640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcc300a3e30 0x7fcc300a62f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c030040 tx=0x7fcc3c0095c0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.670+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/1375228987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc44103230 msgr2=0x7fcc44198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.670+0000 7fcc4b8ae640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc44103230 0x7fcc44198750 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000d950 tx=0x7fcc4000de10 comp rx=0 tx=0).stop 2024-04-23T18:18:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.670+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/1375228987 shutdown_connections 2024-04-23T18:18:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.670+0000 7fcc4b8ae640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcc300a3e30 0x7fcc300a62f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.670+0000 7fcc4b8ae640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc4410c600 0x7fcc4419d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.670+0000 7fcc4b8ae640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc44103b50 0x7fcc44198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.670+0000 7fcc4b8ae640 1 --2- 172.21.15.50:0/1375228987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc44103230 0x7fcc44198750 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.670+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/1375228987 >> 172.21.15.50:0/1375228987 conn(0x7fcc44077f80 msgr2=0x7fcc440fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.670+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/1375228987 shutdown_connections 2024-04-23T18:18:39.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:39.670+0000 7fcc4b8ae640 1 -- 172.21.15.50:0/1375228987 wait complete. 2024-04-23T18:18:39.673 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-23T18:18:40.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:40 smithi177 ceph-mon[32308]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 131 KiB/s wr, 75 op/s 2024-04-23T18:18:40.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:40 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2343029450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:40.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:40 smithi188 ceph-mon[32694]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 131 KiB/s wr, 75 op/s 2024-04-23T18:18:40.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:40 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2343029450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:40.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:40 smithi050 ceph-mon[29187]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 131 KiB/s wr, 75 op/s 2024-04-23T18:18:40.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:40 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2343029450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:42.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:42 smithi177 ceph-mon[32308]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 58 op/s 2024-04-23T18:18:42.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:42 smithi188 ceph-mon[32694]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 58 op/s 2024-04-23T18:18:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:42 smithi050 ceph-mon[29187]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 58 op/s 2024-04-23T18:18:44.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:44 smithi177 ceph-mon[32308]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-04-23T18:18:44.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:44 smithi188 ceph-mon[32694]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-04-23T18:18:44.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:44 smithi050 ceph-mon[29187]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-04-23T18:18:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.956+0000 7f020428a640 1 -- 172.21.15.50:0/2398521261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01fc102b50 msgr2=0x7f01fc106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.956+0000 7f020428a640 1 --2- 172.21.15.50:0/2398521261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01fc102b50 0x7f01fc106b90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009900 tx=0x7f01ec02a410 comp rx=0 tx=0).stop 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.956+0000 7f020428a640 1 -- 172.21.15.50:0/2398521261 shutdown_connections 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.956+0000 7f020428a640 1 --2- 172.21.15.50:0/2398521261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01fc107270 0x7f01fc10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.956+0000 7f020428a640 1 --2- 172.21.15.50:0/2398521261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01fc102b50 0x7f01fc106b90 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.956+0000 7f020428a640 1 --2- 172.21.15.50:0/2398521261 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01fc1021a0 0x7f01fc102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.956+0000 7f020428a640 1 -- 172.21.15.50:0/2398521261 >> 172.21.15.50:0/2398521261 conn(0x7f01fc0fda00 msgr2=0x7f01fc0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.956+0000 7f020428a640 1 -- 172.21.15.50:0/2398521261 shutdown_connections 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.956+0000 7f020428a640 1 -- 172.21.15.50:0/2398521261 wait complete. 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.957+0000 7f020428a640 1 Processor -- start 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.957+0000 7f020428a640 1 -- start start 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.957+0000 7f020428a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01fc1021a0 0x7f01fc1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.957+0000 7f020428a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01fc107270 0x7f01fc198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.957+0000 7f020428a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01fc19cfb0 0x7f01fc19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.957+0000 7f020428a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1121b0 con 0x7f01fc19cfb0 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.957+0000 7f020428a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc112030 con 0x7f01fc107270 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.957+0000 7f020428a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc112330 con 0x7f01fc1021a0 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.957+0000 7f0202827640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01fc19cfb0 0x7f01fc19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.957+0000 7f0202827640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01fc19cfb0 0x7f01fc19d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46856/0 (socket says 172.21.15.50:46856) 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.957+0000 7f0202827640 1 -- 172.21.15.50:0/4123823872 learned_addr learned my addr 172.21.15.50:0/4123823872 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.957+0000 7f0202026640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01fc1021a0 0x7f01fc1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f0201825640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01fc107270 0x7f01fc198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f0202827640 1 -- 172.21.15.50:0/4123823872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01fc1021a0 msgr2=0x7f01fc1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f0202827640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01fc1021a0 0x7f01fc1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f0202827640 1 -- 172.21.15.50:0/4123823872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01fc107270 msgr2=0x7f01fc198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f0202827640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01fc107270 0x7f01fc198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f0202827640 1 -- 172.21.15.50:0/4123823872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01fc19daa0 con 0x7f01fc19cfb0 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f0201825640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01fc107270 0x7f01fc198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f0202827640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01fc19cfb0 0x7f01fc19d460 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f01f400eaa0 tx=0x7f01f400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f01f37fe640 1 -- 172.21.15.50:0/4123823872 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f400cdb0 con 0x7f01fc19cfb0 2024-04-23T18:18:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f020428a640 1 -- 172.21.15.50:0/4123823872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01fc19dd90 con 0x7f01fc19cfb0 2024-04-23T18:18:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f01f37fe640 1 -- 172.21.15.50:0/4123823872 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01f4005020 con 0x7f01fc19cfb0 2024-04-23T18:18:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f01f37fe640 1 -- 172.21.15.50:0/4123823872 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f40116d0 con 0x7f01fc19cfb0 2024-04-23T18:18:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.958+0000 7f020428a640 1 -- 172.21.15.50:0/4123823872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01fc1a55a0 con 0x7f01fc19cfb0 2024-04-23T18:18:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.961+0000 7f01f37fe640 1 -- 172.21.15.50:0/4123823872 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f01f40054a0 con 0x7f01fc19cfb0 2024-04-23T18:18:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.961+0000 7f0206fa2740 1 Processor -- start 2024-04-23T18:18:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.961+0000 7f0206fa2740 1 -- start start 2024-04-23T18:18:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.961+0000 7f0206fa2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 0x55bf457cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.961+0000 7f0206fa2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457ccf40 0x55bf457d72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.961+0000 7f0206fa2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457d7800 0x55bf457d9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.961+0000 7f0206fa2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bf45731970 con 0x55bf457cc620 2024-04-23T18:18:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.961+0000 7f0206fa2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bf456ce4a0 con 0x55bf457ccf40 2024-04-23T18:18:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.961+0000 7f0206fa2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bf4572d930 con 0x55bf457d7800 2024-04-23T18:18:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.961+0000 7f0201825640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 0x55bf457cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f0201825640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 0x55bf457cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46870/0 (socket says 172.21.15.50:46870) 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f0201825640 1 -- 172.21.15.50:0/686496864 learned_addr learned my addr 172.21.15.50:0/686496864 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f0202026640 1 --2- 172.21.15.50:0/686496864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457ccf40 0x55bf457d72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f0201825640 1 --2- 172.21.15.50:0/686496864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457d7800 0x55bf457d9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f0201825640 1 -- 172.21.15.50:0/686496864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457d7800 msgr2=0x55bf457d9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f0201825640 1 --2- 172.21.15.50:0/686496864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457d7800 0x55bf457d9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f0201825640 1 -- 172.21.15.50:0/686496864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457ccf40 msgr2=0x55bf457d72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f0201825640 1 --2- 172.21.15.50:0/686496864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457ccf40 0x55bf457d72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f0201825640 1 -- 172.21.15.50:0/686496864 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf45444210 con 0x55bf457cc620 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f01f37fe640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f01e80a3e60 0x7f01e80a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f0201825640 1 --2- 172.21.15.50:0/686496864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 0x55bf457cca00 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f01ec005c40 tx=0x7f01ec005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc3b075a9dec8d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f01f0bfe640 1 -- 172.21.15.50:0/686496864 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec035050 con 0x55bf457cc620 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f0202026640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f01e80a3e60 0x7f01e80a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f01f37fe640 1 -- 172.21.15.50:0/4123823872 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f01f40d0aa0 con 0x7f01fc19cfb0 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f01f0bfe640 1 -- 172.21.15.50:0/686496864 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01ec031620 con 0x55bf457cc620 2024-04-23T18:18:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.962+0000 7f01f0bfe640 1 -- 172.21.15.50:0/686496864 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec005610 con 0x55bf457cc620 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.963+0000 7f0206fa2740 1 -- 172.21.15.50:0/686496864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 msgr2=0x55bf457cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.963+0000 7f0202026640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f01e80a3e60 0x7f01e80a6320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55bf457cdbc0 tx=0x7f01f8005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.963+0000 7f0206fa2740 1 --2- 172.21.15.50:0/686496864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 0x55bf457cca00 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f01ec005c40 tx=0x7f01ec005bd0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.963+0000 7f0206fa2740 1 -- 172.21.15.50:0/686496864 shutdown_connections 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.963+0000 7f0206fa2740 1 --2- 172.21.15.50:0/686496864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457d7800 0x55bf457d9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.963+0000 7f0206fa2740 1 --2- 172.21.15.50:0/686496864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457ccf40 0x55bf457d72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.963+0000 7f0206fa2740 1 --2- 172.21.15.50:0/686496864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 0x55bf457cca00 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.963+0000 7f0206fa2740 1 -- 172.21.15.50:0/686496864 >> 172.21.15.50:0/686496864 conn(0x55bf457c8b30 msgr2=0x55bf457c8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.963+0000 7f0206fa2740 1 -- 172.21.15.50:0/686496864 shutdown_connections 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.963+0000 7f0206fa2740 1 -- 172.21.15.50:0/686496864 wait complete. 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0206fa2740 1 -- start start 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0206fa2740 1 Processor -- start 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0206fa2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 0x55bf457d7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0206fa2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457ccf40 0x55bf457d4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0206fa2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457d49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0206fa2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bf457da3d0 con 0x55bf457cc620 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0206fa2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bf456cc0b0 con 0x55bf457d7800 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0206fa2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bf457da550 con 0x55bf457ccf40 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0202026640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 0x55bf457d7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0206fa2740 10 client.0 _ll_register_callbacks cb 0x55bf456b9d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0202026640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 0x55bf457d7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46872/0 (socket says 172.21.15.50:46872) 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0202026640 1 -- 172.21.15.50:0/2209507527 learned_addr learned my addr 172.21.15.50:0/2209507527 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0201825640 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457ccf40 0x55bf457d4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0202827640 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457d49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0202026640 1 -- 172.21.15.50:0/2209507527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457ccf40 msgr2=0x55bf457d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.964+0000 7f0202026640 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457ccf40 0x55bf457d4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f0202026640 1 -- 172.21.15.50:0/2209507527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 msgr2=0x55bf457d49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f0202026640 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457d49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f0202026640 1 -- 172.21.15.50:0/2209507527 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf45510620 con 0x55bf457cc620 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f0201825640 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457ccf40 0x55bf457d4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f0202827640 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457d49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f0202026640 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 0x55bf457d7160 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f01f80045e0 tx=0x7f01f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e87665f1c9ce4e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f01e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f01e5ffb640 1 -- 172.21.15.50:0/2209507527 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f801e680 con 0x55bf457cc620 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f01e5ffb640 1 -- 172.21.15.50:0/2209507527 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f01f801e820 con 0x55bf457cc620 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f0206fa2740 1 -- 172.21.15.50:0/2209507527 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01ec031620 con 0x55bf457cc620 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f01e5ffb640 1 -- 172.21.15.50:0/2209507527 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f01f801ea20 con 0x55bf457cc620 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f0206fa2740 10 client.15531 fetch_fsmap learned FSMap version 21 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f0206fa2740 1 -- 172.21.15.50:0/2209507527 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bf457e63b0 con 0x55bf457cc620 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.965+0000 7f01e5ffb640 1 -- 172.21.15.50:0/2209507527 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01ec031620 con 0x55bf457cc620 2024-04-23T18:18:45.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.966+0000 7f01e5ffb640 1 -- 172.21.15.50:0/2209507527 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f01f8027070 con 0x55bf457cc620 2024-04-23T18:18:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.966+0000 7f0206fa2740 10 client.15531 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:18:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.966+0000 7f0206fa2740 10 client.15531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.966+0000 7f0206fa2740 20 client.15531 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.966+0000 7f0206fa2740 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf457e7870 0x55bf457e9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.966+0000 7f0206fa2740 4 client.15531 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.966+0000 7f0206fa2740 1 -- 172.21.15.50:0/2209507527 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf4573f390 con 0x55bf457e7870 2024-04-23T18:18:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.966+0000 7f0201825640 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf457e7870 0x55bf457e9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.966+0000 7f01e5ffb640 10 client.15531 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.966+0000 7f0201825640 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf457e7870 0x55bf457e9cc0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e8fa7c1f85f7723 server_cookie=86e50ac4b2ba0036 in_seq=0 out_seq=0 2024-04-23T18:18:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f01e5ffb640 1 -- 172.21.15.50:0/2209507527 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf4573f390 con 0x55bf457e7870 2024-04-23T18:18:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f01e5ffb640 10 client.15531 handle_command_reply: tid=0 2024-04-23T18:18:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f0206fa2740 1 client.15531 shutdown 2024-04-23T18:18:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f0206fa2740 1 -- 172.21.15.50:0/2209507527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 msgr2=0x55bf457d7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f0206fa2740 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 0x55bf457d7160 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f01f80045e0 tx=0x7f01f8004730 comp rx=0 tx=0).stop 2024-04-23T18:18:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f0206fa2740 1 -- 172.21.15.50:0/2209507527 shutdown_connections 2024-04-23T18:18:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f0206fa2740 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf457e7870 0x55bf457e9cc0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f0206fa2740 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457d49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f0206fa2740 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457ccf40 0x55bf457d4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f01e5ffb640 0 client.15531 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f0206fa2740 1 --2- 172.21.15.50:0/2209507527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457cc620 0x55bf457d7160 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f0206fa2740 1 -- 172.21.15.50:0/2209507527 >> 172.21.15.50:0/2209507527 conn(0x55bf457c8b30 msgr2=0x55bf457ce8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f0206fa2740 1 -- 172.21.15.50:0/2209507527 shutdown_connections 2024-04-23T18:18:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.967+0000 7f0206fa2740 1 -- 172.21.15.50:0/2209507527 wait complete. 2024-04-23T18:18:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.968+0000 7f0206fa2740 20 client.15531 trim_cache size 0 max 16384 2024-04-23T18:18:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.983+0000 7f0206fa2740 1 Processor -- start 2024-04-23T18:18:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.983+0000 7f0206fa2740 1 -- start start 2024-04-23T18:18:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.983+0000 7f0206fa2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457cc620 0x55bf457e3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.983+0000 7f0206fa2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 0x55bf457cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.983+0000 7f0206fa2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457d6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.983+0000 7f0206fa2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1b0380 con 0x55bf457ccf40 2024-04-23T18:18:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.983+0000 7f0206fa2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1adc50 con 0x55bf457d7800 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.983+0000 7f0206fa2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1abac0 con 0x55bf457cc620 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f0201825640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 0x55bf457cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f0201825640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 0x55bf457cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46874/0 (socket says 172.21.15.50:46874) 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f0201825640 1 -- 172.21.15.50:0/1115676895 learned_addr learned my addr 172.21.15.50:0/1115676895 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f0202827640 1 --2- 172.21.15.50:0/1115676895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457d6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f0202026640 1 --2- 172.21.15.50:0/1115676895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457cc620 0x55bf457e3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f0201825640 1 -- 172.21.15.50:0/1115676895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457cc620 msgr2=0x55bf457e3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f0201825640 1 --2- 172.21.15.50:0/1115676895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457cc620 0x55bf457e3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f0201825640 1 -- 172.21.15.50:0/1115676895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 msgr2=0x55bf457d6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f0201825640 1 --2- 172.21.15.50:0/1115676895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457d6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f0201825640 1 -- 172.21.15.50:0/1115676895 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf457d3400 con 0x55bf457ccf40 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f0201825640 1 --2- 172.21.15.50:0/1115676895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 0x55bf457cf970 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f01ec005ec0 tx=0x7f01ec005310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41db09599f1aca85 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f01e7fff640 1 -- 172.21.15.50:0/1115676895 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec035040 con 0x55bf457ccf40 2024-04-23T18:18:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f01e7fff640 1 -- 172.21.15.50:0/1115676895 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01ec005570 con 0x55bf457ccf40 2024-04-23T18:18:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.984+0000 7f01e7fff640 1 -- 172.21.15.50:0/1115676895 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec0037e0 con 0x55bf457ccf40 2024-04-23T18:18:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.985+0000 7f0206fa2740 1 -- 172.21.15.50:0/1115676895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 msgr2=0x55bf457cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.985+0000 7f0206fa2740 1 --2- 172.21.15.50:0/1115676895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 0x55bf457cf970 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f01ec005ec0 tx=0x7f01ec005310 comp rx=0 tx=0).stop 2024-04-23T18:18:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.985+0000 7f0206fa2740 1 -- 172.21.15.50:0/1115676895 shutdown_connections 2024-04-23T18:18:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.985+0000 7f0206fa2740 1 --2- 172.21.15.50:0/1115676895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457d6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.985+0000 7f0206fa2740 1 --2- 172.21.15.50:0/1115676895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 0x55bf457cf970 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.985+0000 7f0206fa2740 1 --2- 172.21.15.50:0/1115676895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457cc620 0x55bf457e3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.985+0000 7f0206fa2740 1 -- 172.21.15.50:0/1115676895 >> 172.21.15.50:0/1115676895 conn(0x55bf457c8b30 msgr2=0x55bf457cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.985+0000 7f0206fa2740 1 -- 172.21.15.50:0/1115676895 shutdown_connections 2024-04-23T18:18:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.985+0000 7f0206fa2740 1 -- 172.21.15.50:0/1115676895 wait complete. 2024-04-23T18:18:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0206fa2740 1 -- start start 2024-04-23T18:18:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0206fa2740 1 Processor -- start 2024-04-23T18:18:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0206fa2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457cc620 0x55bf457db330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0206fa2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 0x55bf457db870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0206fa2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457dbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0206fa2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1ac110 con 0x55bf457ccf40 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0206fa2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1ad580 con 0x55bf457d7800 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0206fa2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1ac7e0 con 0x55bf457cc620 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0206fa2740 10 client.0 _ll_register_callbacks cb 0x7f01f80044a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0201825640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 0x55bf457db870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0201825640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 0x55bf457db870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46886/0 (socket says 172.21.15.50:46886) 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0201825640 1 -- 172.21.15.50:0/1909456706 learned_addr learned my addr 172.21.15.50:0/1909456706 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0202827640 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457dbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0202026640 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457cc620 0x55bf457db330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0201825640 1 -- 172.21.15.50:0/1909456706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457cc620 msgr2=0x55bf457db330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.986+0000 7f0201825640 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457cc620 0x55bf457db330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f0201825640 1 -- 172.21.15.50:0/1909456706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 msgr2=0x55bf457dbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f0201825640 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457dbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f0201825640 1 -- 172.21.15.50:0/1909456706 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf457ccd70 con 0x55bf457ccf40 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f0201825640 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 0x55bf457db870 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f01ec031440 tx=0x7f01ec032ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7ab5c4727dff1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f01e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f01e5ffb640 1 -- 172.21.15.50:0/1909456706 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec060060 con 0x55bf457ccf40 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f01e5ffb640 1 -- 172.21.15.50:0/1909456706 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f01ec04f050 con 0x55bf457ccf40 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f0206fa2740 1 -- 172.21.15.50:0/1909456706 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01ec005570 con 0x55bf457ccf40 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f01e5ffb640 1 -- 172.21.15.50:0/1909456706 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f01ec035040 con 0x55bf457ccf40 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f0206fa2740 10 client.15537 fetch_fsmap learned FSMap version 21 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f0206fa2740 1 -- 172.21.15.50:0/1909456706 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bf4584e140 con 0x55bf457ccf40 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f01e5ffb640 1 -- 172.21.15.50:0/1909456706 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01ec005570 con 0x55bf457ccf40 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f01e5ffb640 1 -- 172.21.15.50:0/1909456706 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f01ec003a70 con 0x55bf457ccf40 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f0206fa2740 10 client.15537 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.987+0000 7f0206fa2740 10 client.15537 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.988+0000 7f0206fa2740 20 client.15537 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.988+0000 7f0206fa2740 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf4584e370 0x55bf45850760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.988+0000 7f0206fa2740 4 client.15537 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:18:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.988+0000 7f0206fa2740 1 -- 172.21.15.50:0/1909456706 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bf45739200 con 0x55bf4584e370 2024-04-23T18:18:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.988+0000 7f0202026640 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf4584e370 0x55bf45850760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.988+0000 7f0202026640 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf4584e370 0x55bf45850760 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1f1f2d49f36a8d3 server_cookie=332a845c0c060e60 in_seq=0 out_seq=0 2024-04-23T18:18:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:45.988+0000 7f01e5ffb640 10 client.15537 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f01e5ffb640 1 -- 172.21.15.50:0/1909456706 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bf45739200 con 0x55bf4584e370 2024-04-23T18:18:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f01e5ffb640 10 client.15537 handle_command_reply: tid=0 2024-04-23T18:18:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f0206fa2740 1 client.15537 shutdown 2024-04-23T18:18:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f0206fa2740 1 -- 172.21.15.50:0/1909456706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 msgr2=0x55bf457db870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f0206fa2740 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 0x55bf457db870 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f01ec031440 tx=0x7f01ec032ba0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f0206fa2740 1 -- 172.21.15.50:0/1909456706 shutdown_connections 2024-04-23T18:18:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f0206fa2740 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf4584e370 0x55bf45850760 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f0206fa2740 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf457d7800 0x55bf457dbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f01e5ffb640 0 client.15537 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f0206fa2740 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf457ccf40 0x55bf457db870 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f0206fa2740 1 --2- 172.21.15.50:0/1909456706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf457cc620 0x55bf457db330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f0206fa2740 1 -- 172.21.15.50:0/1909456706 >> 172.21.15.50:0/1909456706 conn(0x55bf457c8b30 msgr2=0x55bf457c7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f0206fa2740 1 -- 172.21.15.50:0/1909456706 shutdown_connections 2024-04-23T18:18:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f0206fa2740 1 -- 172.21.15.50:0/1909456706 wait complete. 2024-04-23T18:18:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.000+0000 7f0206fa2740 20 client.15537 trim_cache size 0 max 16384 2024-04-23T18:18:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:18:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:18:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:18:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:18:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.001+0000 7f020428a640 1 -- 172.21.15.50:0/4123823872 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f01e80a3e60 msgr2=0x7f01e80a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.001+0000 7f020428a640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f01e80a3e60 0x7f01e80a6320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55bf457cdbc0 tx=0x7f01f8005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.001+0000 7f020428a640 1 -- 172.21.15.50:0/4123823872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01fc19cfb0 msgr2=0x7f01fc19d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.001+0000 7f020428a640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01fc19cfb0 0x7f01fc19d460 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f01f400eaa0 tx=0x7f01f400ef60 comp rx=0 tx=0).stop 2024-04-23T18:18:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.001+0000 7f020428a640 1 -- 172.21.15.50:0/4123823872 shutdown_connections 2024-04-23T18:18:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.001+0000 7f020428a640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f01e80a3e60 0x7f01e80a6320 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.001+0000 7f020428a640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01fc19cfb0 0x7f01fc19d460 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.001+0000 7f020428a640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01fc107270 0x7f01fc198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.001+0000 7f020428a640 1 --2- 172.21.15.50:0/4123823872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01fc1021a0 0x7f01fc1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.001+0000 7f020428a640 1 -- 172.21.15.50:0/4123823872 >> 172.21.15.50:0/4123823872 conn(0x7f01fc0fda00 msgr2=0x7f01fc0ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.001+0000 7f020428a640 1 -- 172.21.15.50:0/4123823872 shutdown_connections 2024-04-23T18:18:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:46.001+0000 7f020428a640 1 -- 172.21.15.50:0/4123823872 wait complete. 2024-04-23T18:18:46.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:46 smithi177 ceph-mon[32308]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-23T18:18:46.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:46 smithi188 ceph-mon[32694]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-23T18:18:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:46 smithi050 ceph-mon[29187]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-23T18:18:48.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:48 smithi177 ceph-mon[32308]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-23T18:18:48.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:48 smithi188 ceph-mon[32694]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-23T18:18:48.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:48 smithi050 ceph-mon[29187]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-23T18:18:50.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:50 smithi177 ceph-mon[32308]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 41 KiB/s wr, 26 op/s 2024-04-23T18:18:50.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:50 smithi188 ceph-mon[32694]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 41 KiB/s wr, 26 op/s 2024-04-23T18:18:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:50 smithi050 ceph-mon[29187]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 41 KiB/s wr, 26 op/s 2024-04-23T18:18:52.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:52 smithi177 ceph-mon[32308]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-23T18:18:52.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:52 smithi188 ceph-mon[32694]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-23T18:18:52.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:52 smithi050 ceph-mon[29187]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-23T18:18:54.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:54 smithi177 ceph-mon[32308]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-04-23T18:18:54.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:54 smithi188 ceph-mon[32694]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-04-23T18:18:54.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:54 smithi050 ceph-mon[29187]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-04-23T18:18:54.674 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:18:54.674 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-23T18:18:54.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:18:54.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.789+0000 7f785f6ef640 1 -- 172.21.15.50:0/2799711826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78581031e0 msgr2=0x7f78581035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:54.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.789+0000 7f785f6ef640 1 --2- 172.21.15.50:0/2799711826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78581031e0 0x7f78581035c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7854009900 tx=0x7f785402a510 comp rx=0 tx=0).stop 2024-04-23T18:18:54.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.789+0000 7f785f6ef640 1 -- 172.21.15.50:0/2799711826 shutdown_connections 2024-04-23T18:18:54.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.789+0000 7f785f6ef640 1 --2- 172.21.15.50:0/2799711826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f785810c520 0x7f785810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:54.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.789+0000 7f785f6ef640 1 --2- 172.21.15.50:0/2799711826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7858103b00 0x7f785810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:54.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.789+0000 7f785f6ef640 1 --2- 172.21.15.50:0/2799711826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78581031e0 0x7f78581035c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:54.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.789+0000 7f785f6ef640 1 -- 172.21.15.50:0/2799711826 >> 172.21.15.50:0/2799711826 conn(0x7f7858077f50 msgr2=0x7f7858078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:54.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.789+0000 7f785f6ef640 1 -- 172.21.15.50:0/2799711826 shutdown_connections 2024-04-23T18:18:54.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.789+0000 7f785f6ef640 1 -- 172.21.15.50:0/2799711826 wait complete. 2024-04-23T18:18:54.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.790+0000 7f785f6ef640 1 Processor -- start 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.790+0000 7f785f6ef640 1 -- start start 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.790+0000 7f785f6ef640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78581031e0 0x7f785819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.790+0000 7f785f6ef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7858103b00 0x7f785819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.790+0000 7f785f6ef640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f785810c520 0x7f7858198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.790+0000 7f785f6ef640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f78581121b0 con 0x7f78581031e0 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.790+0000 7f785f6ef640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7858112030 con 0x7f7858103b00 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.790+0000 7f785f6ef640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7858112330 con 0x7f785810c520 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.790+0000 7f785d48b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78581031e0 0x7f785819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.790+0000 7f785d48b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78581031e0 0x7f785819e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59852/0 (socket says 172.21.15.50:59852) 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f785d48b640 1 -- 172.21.15.50:0/3921438081 learned_addr learned my addr 172.21.15.50:0/3921438081 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f785cc8a640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7858103b00 0x7f785819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f785dc8c640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f785810c520 0x7f7858198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f785d48b640 1 -- 172.21.15.50:0/3921438081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f785810c520 msgr2=0x7f7858198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:54.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f785d48b640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f785810c520 0x7f7858198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:54.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f785d48b640 1 -- 172.21.15.50:0/3921438081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7858103b00 msgr2=0x7f785819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:54.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f785d48b640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7858103b00 0x7f785819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:54.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f785d48b640 1 -- 172.21.15.50:0/3921438081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7858198d10 con 0x7f78581031e0 2024-04-23T18:18:54.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f785d48b640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78581031e0 0x7f785819e3e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7854005940 tx=0x7f7854004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:54.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f784e7fc640 1 -- 172.21.15.50:0/3921438081 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785403f030 con 0x7f78581031e0 2024-04-23T18:18:54.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f785f6ef640 1 -- 172.21.15.50:0/3921438081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7858198fa0 con 0x7f78581031e0 2024-04-23T18:18:54.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f784e7fc640 1 -- 172.21.15.50:0/3921438081 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7854003aa0 con 0x7f78581031e0 2024-04-23T18:18:54.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f784e7fc640 1 -- 172.21.15.50:0/3921438081 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7854035070 con 0x7f78581031e0 2024-04-23T18:18:54.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.791+0000 7f785f6ef640 1 -- 172.21.15.50:0/3921438081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78581a52a0 con 0x7f78581031e0 2024-04-23T18:18:54.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.792+0000 7f785f6ef640 1 -- 172.21.15.50:0/3921438081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7858068d70 con 0x7f78581031e0 2024-04-23T18:18:54.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.794+0000 7f784e7fc640 1 -- 172.21.15.50:0/3921438081 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f7854042030 con 0x7f78581031e0 2024-04-23T18:18:54.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.795+0000 7f784e7fc640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f78440a3db0 0x7f78440a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:54.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.795+0000 7f785cc8a640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f78440a3db0 0x7f78440a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:54.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.795+0000 7f784e7fc640 1 -- 172.21.15.50:0/3921438081 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f78540f0a60 con 0x7f78581031e0 2024-04-23T18:18:54.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.795+0000 7f785cc8a640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f78440a3db0 0x7f78440a6270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f78581008f0 tx=0x7f7848005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:54.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:54.801+0000 7f784e7fc640 1 -- 172.21.15.50:0/3921438081 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78540bc950 con 0x7f78581031e0 2024-04-23T18:18:55.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.056+0000 7f785f6ef640 1 -- 172.21.15.50:0/3921438081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78581076a0 con 0x7f78581031e0 2024-04-23T18:18:55.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.056+0000 7f784e7fc640 1 -- 172.21.15.50:0/3921438081 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11875 (secure 0 0 0) 0x7f78540c0c50 con 0x7f78581031e0 2024-04-23T18:18:55.050 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:18:55.051 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:18:28.904281+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:18:55.051 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 21 2024-04-23T18:18:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.059+0000 7f785f6ef640 1 -- 172.21.15.50:0/3921438081 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f78440a3db0 msgr2=0x7f78440a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.059+0000 7f785f6ef640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f78440a3db0 0x7f78440a6270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f78581008f0 tx=0x7f7848005f90 comp rx=0 tx=0).stop 2024-04-23T18:18:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.059+0000 7f785f6ef640 1 -- 172.21.15.50:0/3921438081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78581031e0 msgr2=0x7f785819e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.059+0000 7f785f6ef640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78581031e0 0x7f785819e3e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7854005940 tx=0x7f7854004270 comp rx=0 tx=0).stop 2024-04-23T18:18:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.059+0000 7f785f6ef640 1 -- 172.21.15.50:0/3921438081 shutdown_connections 2024-04-23T18:18:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.059+0000 7f785f6ef640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f78440a3db0 0x7f78440a6270 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.059+0000 7f785f6ef640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f785810c520 0x7f7858198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.059+0000 7f785f6ef640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7858103b00 0x7f785819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.059+0000 7f785f6ef640 1 --2- 172.21.15.50:0/3921438081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78581031e0 0x7f785819e3e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.059+0000 7f785f6ef640 1 -- 172.21.15.50:0/3921438081 >> 172.21.15.50:0/3921438081 conn(0x7f7858077f50 msgr2=0x7f78580fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.059+0000 7f785f6ef640 1 -- 172.21.15.50:0/3921438081 shutdown_connections 2024-04-23T18:18:55.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.059+0000 7f785f6ef640 1 -- 172.21.15.50:0/3921438081 wait complete. 2024-04-23T18:18:55.068 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-23T18:18:55.068 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-23T18:18:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.199+0000 7f5be364e640 1 -- 172.21.15.50:0/95240383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc1010a0 msgr2=0x7f5bdc101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.199+0000 7f5be364e640 1 --2- 172.21.15.50:0/95240383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc1010a0 0x7f5bdc101480 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009900 tx=0x7f5bd802a410 comp rx=0 tx=0).stop 2024-04-23T18:18:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.199+0000 7f5be364e640 1 -- 172.21.15.50:0/95240383 shutdown_connections 2024-04-23T18:18:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.199+0000 7f5be364e640 1 --2- 172.21.15.50:0/95240383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bdc106170 0x7f5bdc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.199+0000 7f5be364e640 1 --2- 172.21.15.50:0/95240383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bdc101a50 0x7f5bdc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.199+0000 7f5be364e640 1 --2- 172.21.15.50:0/95240383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc1010a0 0x7f5bdc101480 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.199+0000 7f5be364e640 1 -- 172.21.15.50:0/95240383 >> 172.21.15.50:0/95240383 conn(0x7f5bdc077f50 msgr2=0x7f5bdc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.199+0000 7f5be364e640 1 -- 172.21.15.50:0/95240383 shutdown_connections 2024-04-23T18:18:55.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.199+0000 7f5be364e640 1 -- 172.21.15.50:0/95240383 wait complete. 2024-04-23T18:18:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be364e640 1 Processor -- start 2024-04-23T18:18:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be364e640 1 -- start start 2024-04-23T18:18:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be364e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc1010a0 0x7f5bdc102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be364e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bdc106170 0x7f5bdc103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be364e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bdc1035b0 0x7f5bdc1a69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be364e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc111150 con 0x7f5bdc1010a0 2024-04-23T18:18:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be364e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc110fd0 con 0x7f5bdc106170 2024-04-23T18:18:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be364e640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1112d0 con 0x7f5bdc1035b0 2024-04-23T18:18:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be13ea640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc1010a0 0x7f5bdc102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be13ea640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc1010a0 0x7f5bdc102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59870/0 (socket says 172.21.15.50:59870) 2024-04-23T18:18:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be13ea640 1 -- 172.21.15.50:0/1608597182 learned_addr learned my addr 172.21.15.50:0/1608597182 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be1beb640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bdc1035b0 0x7f5bdc1a69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be0be9640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bdc106170 0x7f5bdc103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be13ea640 1 -- 172.21.15.50:0/1608597182 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bdc1035b0 msgr2=0x7f5bdc1a69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be13ea640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bdc1035b0 0x7f5bdc1a69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be13ea640 1 -- 172.21.15.50:0/1608597182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bdc106170 msgr2=0x7f5bdc103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be13ea640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bdc106170 0x7f5bdc103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.200+0000 7f5be13ea640 1 -- 172.21.15.50:0/1608597182 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bdc1a70b0 con 0x7f5bdc1010a0 2024-04-23T18:18:55.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.201+0000 7f5be13ea640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc1010a0 0x7f5bdc102b30 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8005940 tx=0x7f5bd8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:55.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.201+0000 7f5bd27fc640 1 -- 172.21.15.50:0/1608597182 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd803f030 con 0x7f5bdc1010a0 2024-04-23T18:18:55.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.201+0000 7f5be364e640 1 -- 172.21.15.50:0/1608597182 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bdc1a7340 con 0x7f5bdc1010a0 2024-04-23T18:18:55.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.201+0000 7f5bd27fc640 1 -- 172.21.15.50:0/1608597182 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd8003aa0 con 0x7f5bdc1010a0 2024-04-23T18:18:55.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.201+0000 7f5bd27fc640 1 -- 172.21.15.50:0/1608597182 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8035070 con 0x7f5bdc1010a0 2024-04-23T18:18:55.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.201+0000 7f5be364e640 1 -- 172.21.15.50:0/1608597182 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bdc1a7880 con 0x7f5bdc1010a0 2024-04-23T18:18:55.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.203+0000 7f5bd27fc640 1 -- 172.21.15.50:0/1608597182 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f5bd8042030 con 0x7f5bdc1010a0 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.204+0000 7f5bd27fc640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5bc80a3de0 0x7f5bc80a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.204+0000 7f5bd27fc640 1 -- 172.21.15.50:0/1608597182 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f5bd80f0ae0 con 0x7f5bdc1010a0 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.204+0000 7f5be0be9640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5bc80a3de0 0x7f5bc80a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.204+0000 7f5be6366740 1 Processor -- start 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be6366740 1 -- start start 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be0be9640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5bc80a3de0 0x7f5bc80a62a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc1045c0 tx=0x7f5bcc008040 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be6366740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 0x56385f5ca480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be6366740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca9c0 0x56385f5d4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be6366740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5d7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be6366740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56385f52f370 con 0x56385f5d5090 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be6366740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56385f4cc2e0 con 0x56385f5ca0a0 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be6366740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56385f52b330 con 0x56385f5ca9c0 2024-04-23T18:18:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be1beb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5d7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be1beb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5d7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59882/0 (socket says 172.21.15.50:59882) 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be1beb640 1 -- 172.21.15.50:0/1037116984 learned_addr learned my addr 172.21.15.50:0/1037116984 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be13ea640 1 --2- 172.21.15.50:0/1037116984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 0x56385f5ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be0be9640 1 --2- 172.21.15.50:0/1037116984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca9c0 0x56385f5d4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be1beb640 1 -- 172.21.15.50:0/1037116984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca9c0 msgr2=0x56385f5d4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be1beb640 1 --2- 172.21.15.50:0/1037116984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca9c0 0x56385f5d4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be1beb640 1 -- 172.21.15.50:0/1037116984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 msgr2=0x56385f5ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be1beb640 1 --2- 172.21.15.50:0/1037116984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 0x56385f5ca480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be1beb640 1 -- 172.21.15.50:0/1037116984 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56385f30dd80 con 0x56385f5d5090 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be0be9640 1 --2- 172.21.15.50:0/1037116984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca9c0 0x56385f5d4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be13ea640 1 --2- 172.21.15.50:0/1037116984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 0x56385f5ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.205+0000 7f5be1beb640 1 --2- 172.21.15.50:0/1037116984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5d7480 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009fd0 tx=0x7f5bd40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb2d1a3ccf79ebe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.206+0000 7f5bc77fe640 1 -- 172.21.15.50:0/1037116984 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4004d20 con 0x56385f5d5090 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.206+0000 7f5bc77fe640 1 -- 172.21.15.50:0/1037116984 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd4004050 con 0x56385f5d5090 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.206+0000 7f5bc77fe640 1 -- 172.21.15.50:0/1037116984 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4005280 con 0x56385f5d5090 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.207+0000 7f5be6366740 1 -- 172.21.15.50:0/1037116984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 msgr2=0x56385f5d7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.207+0000 7f5be6366740 1 --2- 172.21.15.50:0/1037116984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5d7480 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009fd0 tx=0x7f5bd40118d0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.207+0000 7f5be6366740 1 -- 172.21.15.50:0/1037116984 shutdown_connections 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.207+0000 7f5be6366740 1 --2- 172.21.15.50:0/1037116984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5d7480 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.207+0000 7f5be6366740 1 --2- 172.21.15.50:0/1037116984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca9c0 0x56385f5d4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.207+0000 7f5be6366740 1 --2- 172.21.15.50:0/1037116984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 0x56385f5ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.207+0000 7f5be6366740 1 -- 172.21.15.50:0/1037116984 >> 172.21.15.50:0/1037116984 conn(0x56385f5c63c0 msgr2=0x56385f5c67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.207+0000 7f5be6366740 1 -- 172.21.15.50:0/1037116984 shutdown_connections 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.207+0000 7f5be6366740 1 -- 172.21.15.50:0/1037116984 wait complete. 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.207+0000 7f5be6366740 1 -- start start 2024-04-23T18:18:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.207+0000 7f5be6366740 1 Processor -- start 2024-04-23T18:18:55.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be6366740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca0a0 0x56385f5d1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be6366740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5ca9c0 0x56385f5d2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be6366740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5d5090 0x56385f5d2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be6366740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56385f5d7e80 con 0x56385f5ca9c0 2024-04-23T18:18:55.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be6366740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56385f4c9fb0 con 0x56385f5d5090 2024-04-23T18:18:55.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be6366740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56385f5d8000 con 0x56385f5ca0a0 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be6366740 10 client.0 _ll_register_callbacks cb 0x56385f4c99a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be0be9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5ca9c0 0x56385f5d2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be0be9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5ca9c0 0x56385f5d2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59886/0 (socket says 172.21.15.50:59886) 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be0be9640 1 -- 172.21.15.50:0/2728993478 learned_addr learned my addr 172.21.15.50:0/2728993478 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be13ea640 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca0a0 0x56385f5d1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be1beb640 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5d5090 0x56385f5d2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be0be9640 1 -- 172.21.15.50:0/2728993478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca0a0 msgr2=0x56385f5d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be0be9640 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca0a0 0x56385f5d1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be0be9640 1 -- 172.21.15.50:0/2728993478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5d5090 msgr2=0x56385f5d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be0be9640 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5d5090 0x56385f5d2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be0be9640 1 -- 172.21.15.50:0/2728993478 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385f36e510 con 0x56385f5ca9c0 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be1beb640 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5d5090 0x56385f5d2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5be0be9640 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5ca9c0 0x56385f5d2530 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00de90 tx=0x7f5bcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be17069b3b0c4ba0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5bc4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5bc4ff9640 1 -- 172.21.15.50:0/2728993478 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc014e40 con 0x56385f5ca9c0 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.208+0000 7f5bc4ff9640 1 -- 172.21.15.50:0/2728993478 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f5bcc0125f0 con 0x56385f5ca9c0 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.209+0000 7f5be6366740 1 -- 172.21.15.50:0/2728993478 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bd4004050 con 0x56385f5ca9c0 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.209+0000 7f5bc4ff9640 1 -- 172.21.15.50:0/2728993478 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5bcc0059c0 con 0x56385f5ca9c0 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.209+0000 7f5be6366740 10 client.15555 fetch_fsmap learned FSMap version 21 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.209+0000 7f5be6366740 1 -- 172.21.15.50:0/2728993478 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56385f201670 con 0x56385f5ca9c0 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.209+0000 7f5bc4ff9640 1 -- 172.21.15.50:0/2728993478 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd4004050 con 0x56385f5ca9c0 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.209+0000 7f5bc4ff9640 1 -- 172.21.15.50:0/2728993478 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f5bcc013070 con 0x56385f5ca9c0 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.209+0000 7f5be6366740 10 client.15555 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.209+0000 7f5be6366740 10 client.15555 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.209+0000 7f5be6366740 20 client.15555 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.209+0000 7f5be6366740 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56385f5e5440 0x56385f5e7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.209+0000 7f5be6366740 4 client.15555 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.209+0000 7f5be6366740 1 -- 172.21.15.50:0/2728993478 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56385f53cd90 con 0x56385f5e5440 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.210+0000 7f5be13ea640 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56385f5e5440 0x56385f5e7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.210+0000 7f5be13ea640 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56385f5e5440 0x56385f5e7860 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4d0a17d06f57263 server_cookie=151cfdfb70597482 in_seq=0 out_seq=0 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.210+0000 7f5bc4ff9640 10 client.15555 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5bc4ff9640 1 -- 172.21.15.50:0/2728993478 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56385f53cd90 con 0x56385f5e5440 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5bc4ff9640 10 client.15555 handle_command_reply: tid=0 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5be6366740 1 client.15555 shutdown 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5be6366740 1 -- 172.21.15.50:0/2728993478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5ca9c0 msgr2=0x56385f5d2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5be6366740 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5ca9c0 0x56385f5d2530 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00de90 tx=0x7f5bcc0046c0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5be6366740 1 -- 172.21.15.50:0/2728993478 shutdown_connections 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5be6366740 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56385f5e5440 0x56385f5e7860 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5be6366740 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5d5090 0x56385f5d2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5be6366740 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5ca9c0 0x56385f5d2530 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5be6366740 1 --2- 172.21.15.50:0/2728993478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca0a0 0x56385f5d1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5be6366740 1 -- 172.21.15.50:0/2728993478 >> 172.21.15.50:0/2728993478 conn(0x56385f5c63c0 msgr2=0x56385f5cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5bc4ff9640 0 client.15555 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:55.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5be6366740 1 -- 172.21.15.50:0/2728993478 shutdown_connections 2024-04-23T18:18:55.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5be6366740 1 -- 172.21.15.50:0/2728993478 wait complete. 2024-04-23T18:18:55.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.211+0000 7f5be6366740 20 client.15555 trim_cache size 0 max 16384 2024-04-23T18:18:55.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.230+0000 7f5be6366740 1 Processor -- start 2024-04-23T18:18:55.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.230+0000 7f5be6366740 1 -- start start 2024-04-23T18:18:55.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.230+0000 7f5be6366740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca0a0 0x56385f5e0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.230+0000 7f5be6366740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca9c0 0x56385f5ed130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.230+0000 7f5be6366740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5ed670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be6366740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1b2680 con 0x56385f5d5090 2024-04-23T18:18:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be6366740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1aff50 con 0x56385f5ca9c0 2024-04-23T18:18:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be6366740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1addc0 con 0x56385f5ca0a0 2024-04-23T18:18:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be0be9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca9c0 0x56385f5ed130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be1beb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5ed670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be0be9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca9c0 0x56385f5ed130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:35260/0 (socket says 172.21.15.50:35260) 2024-04-23T18:18:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be0be9640 1 -- 172.21.15.50:0/1886037161 learned_addr learned my addr 172.21.15.50:0/1886037161 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be1beb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5ed670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59894/0 (socket says 172.21.15.50:59894) 2024-04-23T18:18:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be13ea640 1 --2- 172.21.15.50:0/1886037161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca0a0 0x56385f5e0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be1beb640 1 -- 172.21.15.50:0/1886037161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca0a0 msgr2=0x56385f5e0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be1beb640 1 --2- 172.21.15.50:0/1886037161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca0a0 0x56385f5e0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be1beb640 1 -- 172.21.15.50:0/1886037161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca9c0 msgr2=0x56385f5ed130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be1beb640 1 --2- 172.21.15.50:0/1886037161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca9c0 0x56385f5ed130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be1beb640 1 -- 172.21.15.50:0/1886037161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56385f5ca7f0 con 0x56385f5d5090 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.231+0000 7f5be1beb640 1 --2- 172.21.15.50:0/1886037161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5ed670 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5bd401b040 tx=0x7f5bd4003a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179e094d3271b2c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.232+0000 7f5bc6ffd640 1 -- 172.21.15.50:0/1886037161 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4003d20 con 0x56385f5d5090 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.232+0000 7f5bc6ffd640 1 -- 172.21.15.50:0/1886037161 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd4005570 con 0x56385f5d5090 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.232+0000 7f5bc6ffd640 1 -- 172.21.15.50:0/1886037161 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd400fa40 con 0x56385f5d5090 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.232+0000 7f5be6366740 1 -- 172.21.15.50:0/1886037161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 msgr2=0x56385f5ed670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.232+0000 7f5be6366740 1 --2- 172.21.15.50:0/1886037161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5ed670 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5bd401b040 tx=0x7f5bd4003a30 comp rx=0 tx=0).stop 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.233+0000 7f5be6366740 1 -- 172.21.15.50:0/1886037161 shutdown_connections 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.233+0000 7f5be6366740 1 --2- 172.21.15.50:0/1886037161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5ed670 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.233+0000 7f5be6366740 1 --2- 172.21.15.50:0/1886037161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca9c0 0x56385f5ed130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.233+0000 7f5be6366740 1 --2- 172.21.15.50:0/1886037161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca0a0 0x56385f5e0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.233+0000 7f5be6366740 1 -- 172.21.15.50:0/1886037161 >> 172.21.15.50:0/1886037161 conn(0x56385f5c63c0 msgr2=0x56385f5d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.233+0000 7f5be6366740 1 -- 172.21.15.50:0/1886037161 shutdown_connections 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.233+0000 7f5be6366740 1 -- 172.21.15.50:0/1886037161 wait complete. 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.233+0000 7f5be6366740 1 -- start start 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.233+0000 7f5be6366740 1 Processor -- start 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.233+0000 7f5be6366740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 0x56385f5dd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.233+0000 7f5be6366740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca9c0 0x56385f5dd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be6366740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5da220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be6366740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1ae410 con 0x56385f5d5090 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be6366740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1af880 con 0x56385f5ca0a0 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be6366740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc1aeae0 con 0x56385f5ca9c0 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be1beb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5da220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be13ea640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 0x56385f5dd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be6366740 10 client.0 _ll_register_callbacks cb 0x7f5bcc00db40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be1beb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5da220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59900/0 (socket says 172.21.15.50:59900) 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be13ea640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 0x56385f5dd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:35262/0 (socket says 172.21.15.50:35262) 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be1beb640 1 -- 172.21.15.50:0/2476386949 learned_addr learned my addr 172.21.15.50:0/2476386949 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be0be9640 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca9c0 0x56385f5dd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be1beb640 1 -- 172.21.15.50:0/2476386949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca9c0 msgr2=0x56385f5dd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be1beb640 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca9c0 0x56385f5dd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be1beb640 1 -- 172.21.15.50:0/2476386949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 msgr2=0x56385f5dd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be1beb640 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 0x56385f5dd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be1beb640 1 -- 172.21.15.50:0/2476386949 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385f5daa20 con 0x56385f5d5090 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be13ea640 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 0x56385f5dd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be1beb640 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5da220 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4006010 tx=0x7f5bd40052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ef6a34468ff3f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5bc4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5bc4ff9640 1 -- 172.21.15.50:0/2476386949 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4032070 con 0x56385f5d5090 2024-04-23T18:18:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5bc4ff9640 1 -- 172.21.15.50:0/2476386949 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f5bd400f4e0 con 0x56385f5d5090 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.234+0000 7f5be6366740 1 -- 172.21.15.50:0/2476386949 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bd4005570 con 0x56385f5d5090 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.235+0000 7f5bc4ff9640 1 -- 172.21.15.50:0/2476386949 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5bd403e050 con 0x56385f5d5090 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.235+0000 7f5be6366740 10 client.15561 fetch_fsmap learned FSMap version 21 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.235+0000 7f5be6366740 1 -- 172.21.15.50:0/2476386949 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56385f64c450 con 0x56385f5d5090 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.235+0000 7f5bc4ff9640 1 -- 172.21.15.50:0/2476386949 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd4005570 con 0x56385f5d5090 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.235+0000 7f5bc4ff9640 1 -- 172.21.15.50:0/2476386949 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f5bd4025850 con 0x56385f5d5090 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.235+0000 7f5be6366740 10 client.15561 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.235+0000 7f5be6366740 10 client.15561 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.235+0000 7f5be6366740 20 client.15561 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.235+0000 7f5be6366740 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56385f64c990 0x56385f64ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.235+0000 7f5be6366740 4 client.15561 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.235+0000 7f5be6366740 1 -- 172.21.15.50:0/2476386949 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x56385f536c00 con 0x56385f64c990 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.235+0000 7f5be13ea640 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56385f64c990 0x56385f64ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.236+0000 7f5bc4ff9640 10 client.15561 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.236+0000 7f5be13ea640 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56385f64c990 0x56385f64ed80 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f55f76ff4780f22f server_cookie=3f50a50d11c662a6 in_seq=0 out_seq=0 2024-04-23T18:18:55.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5bc4ff9640 1 -- 172.21.15.50:0/2476386949 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x56385f536c00 con 0x56385f64c990 2024-04-23T18:18:55.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5bc4ff9640 10 client.15561 handle_command_reply: tid=0 2024-04-23T18:18:55.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5be6366740 1 client.15561 shutdown 2024-04-23T18:18:55.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5be6366740 1 -- 172.21.15.50:0/2476386949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 msgr2=0x56385f5da220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5be6366740 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5da220 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4006010 tx=0x7f5bd40052f0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5be6366740 1 -- 172.21.15.50:0/2476386949 shutdown_connections 2024-04-23T18:18:55.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5be6366740 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56385f64c990 0x56385f64ed80 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5be6366740 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56385f5d5090 0x56385f5da220 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5be6366740 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56385f5ca9c0 0x56385f5dd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5be6366740 1 --2- 172.21.15.50:0/2476386949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385f5ca0a0 0x56385f5dd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5be6366740 1 -- 172.21.15.50:0/2476386949 >> 172.21.15.50:0/2476386949 conn(0x56385f5c63c0 msgr2=0x56385f5d2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5bc4ff9640 0 client.15561 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:55.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5be6366740 1 -- 172.21.15.50:0/2476386949 shutdown_connections 2024-04-23T18:18:55.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5be6366740 1 -- 172.21.15.50:0/2476386949 wait complete. 2024-04-23T18:18:55.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.247+0000 7f5be6366740 20 client.15561 trim_cache size 0 max 16384 2024-04-23T18:18:55.242 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:18:55.242 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 21, 2024-04-23T18:18:55.242 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 2, 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "f5ca02e5": { 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 2, 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 90.0, 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:18:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:18:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:18:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:18:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:18:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:18:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:18:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:18:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:18:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:18:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:18:55.244 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:18:55.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.248+0000 7f5be364e640 1 -- 172.21.15.50:0/1608597182 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5bc80a3de0 msgr2=0x7f5bc80a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.248+0000 7f5be364e640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5bc80a3de0 0x7f5bc80a62a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc1045c0 tx=0x7f5bcc008040 comp rx=0 tx=0).stop 2024-04-23T18:18:55.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.248+0000 7f5be364e640 1 -- 172.21.15.50:0/1608597182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc1010a0 msgr2=0x7f5bdc102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:55.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.248+0000 7f5be364e640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc1010a0 0x7f5bdc102b30 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8005940 tx=0x7f5bd8004270 comp rx=0 tx=0).stop 2024-04-23T18:18:55.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.248+0000 7f5be364e640 1 -- 172.21.15.50:0/1608597182 shutdown_connections 2024-04-23T18:18:55.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.248+0000 7f5be364e640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5bc80a3de0 0x7f5bc80a62a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.248+0000 7f5be364e640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bdc1035b0 0x7f5bdc1a69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.248+0000 7f5be364e640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bdc106170 0x7f5bdc103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.248+0000 7f5be364e640 1 --2- 172.21.15.50:0/1608597182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc1010a0 0x7f5bdc102b30 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:55.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.248+0000 7f5be364e640 1 -- 172.21.15.50:0/1608597182 >> 172.21.15.50:0/1608597182 conn(0x7f5bdc077f50 msgr2=0x7f5bdc100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:55.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.248+0000 7f5be364e640 1 -- 172.21.15.50:0/1608597182 shutdown_connections 2024-04-23T18:18:55.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:18:55.248+0000 7f5be364e640 1 -- 172.21.15.50:0/1608597182 wait complete. 2024-04-23T18:18:55.254 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'f5ca02e5', quiesce took 0.6 seconds. Will release after: 30.0 2024-04-23T18:18:56.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:56 smithi177 ceph-mon[32308]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-04-23T18:18:56.288 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:56 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3921438081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:56.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:56 smithi188 ceph-mon[32694]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-04-23T18:18:56.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:56 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3921438081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:56.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:56 smithi050 ceph-mon[29187]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-04-23T18:18:56.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:56 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3921438081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:18:58.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:18:58 smithi188 ceph-mon[32694]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-04-23T18:18:58.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:18:58 smithi050 ceph-mon[29187]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-04-23T18:18:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:18:58 smithi177 ceph-mon[32308]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-04-23T18:18:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.137+0000 7f5431519640 1 -- 172.21.15.50:0/233650417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f542c101aa0 msgr2=0x7f542c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.137+0000 7f5431519640 1 --2- 172.21.15.50:0/233650417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f542c101aa0 0x7f542c105a90 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5420009900 tx=0x7f542002a410 comp rx=0 tx=0).stop 2024-04-23T18:18:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.137+0000 7f5431519640 1 -- 172.21.15.50:0/233650417 shutdown_connections 2024-04-23T18:18:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.137+0000 7f5431519640 1 --2- 172.21.15.50:0/233650417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f542c106170 0x7f542c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.137+0000 7f5431519640 1 --2- 172.21.15.50:0/233650417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f542c101aa0 0x7f542c105a90 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.137+0000 7f5431519640 1 --2- 172.21.15.50:0/233650417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f542c1010f0 0x7f542c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.137+0000 7f5431519640 1 -- 172.21.15.50:0/233650417 >> 172.21.15.50:0/233650417 conn(0x7f542c077fd0 msgr2=0x7f542c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.137+0000 7f5431519640 1 -- 172.21.15.50:0/233650417 shutdown_connections 2024-04-23T18:18:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.137+0000 7f5431519640 1 -- 172.21.15.50:0/233650417 wait complete. 2024-04-23T18:18:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.138+0000 7f5431519640 1 Processor -- start 2024-04-23T18:18:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.138+0000 7f5431519640 1 -- start start 2024-04-23T18:18:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.138+0000 7f5431519640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f542c1010f0 0x7f542c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.138+0000 7f5431519640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f542c106170 0x7f542c198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.138+0000 7f5431519640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f542c19cf30 0x7f542c19d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.138+0000 7f5431519640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f542c111090 con 0x7f542c1010f0 2024-04-23T18:18:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.138+0000 7f5431519640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f542c110f10 con 0x7f542c19cf30 2024-04-23T18:18:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.138+0000 7f5431519640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f542c111210 con 0x7f542c106170 2024-04-23T18:18:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f542affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f542c1010f0 0x7f542c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f542affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f542c1010f0 0x7f542c198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59914/0 (socket says 172.21.15.50:59914) 2024-04-23T18:18:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f542affd640 1 -- 172.21.15.50:0/1804449473 learned_addr learned my addr 172.21.15.50:0/1804449473 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f542a7fc640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f542c106170 0x7f542c198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f542b7fe640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f542c19cf30 0x7f542c19d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f542affd640 1 -- 172.21.15.50:0/1804449473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f542c106170 msgr2=0x7f542c198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f542affd640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f542c106170 0x7f542c198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f542affd640 1 -- 172.21.15.50:0/1804449473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f542c19cf30 msgr2=0x7f542c19d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f542affd640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f542c19cf30 0x7f542c19d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f542affd640 1 -- 172.21.15.50:0/1804449473 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c19da20 con 0x7f542c1010f0 2024-04-23T18:18:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f542a7fc640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f542c106170 0x7f542c198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f542affd640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f542c1010f0 0x7f542c198670 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f541c00d950 tx=0x7f541c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f5413fff640 1 -- 172.21.15.50:0/1804449473 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c017030 con 0x7f542c1010f0 2024-04-23T18:18:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f5413fff640 1 -- 172.21.15.50:0/1804449473 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f541c004020 con 0x7f542c1010f0 2024-04-23T18:18:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f5431519640 1 -- 172.21.15.50:0/1804449473 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542c19dd10 con 0x7f542c1010f0 2024-04-23T18:18:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f5413fff640 1 -- 172.21.15.50:0/1804449473 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c0047a0 con 0x7f542c1010f0 2024-04-23T18:18:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.139+0000 7f5431519640 1 -- 172.21.15.50:0/1804449473 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f542c1a5520 con 0x7f542c1010f0 2024-04-23T18:18:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.143+0000 7f5413fff640 1 -- 172.21.15.50:0/1804449473 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f541c009a30 con 0x7f542c1010f0 2024-04-23T18:18:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.143+0000 7f543422e740 1 Processor -- start 2024-04-23T18:18:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.143+0000 7f543422e740 1 -- start start 2024-04-23T18:18:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f543422e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e8aa30 0x559d38e8ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f543422e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38e956d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f543422e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e95c10 0x559d38e98000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f543422e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559d38deb970 con 0x559d38e8b350 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f543422e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559d38d884a0 con 0x559d38e95c10 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f5413fff640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f54140a3ee0 0x7f54140a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f543422e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559d38de7930 con 0x559d38e8aa30 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f542affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38e956d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f542b7fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e8aa30 0x559d38e8ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f542b7fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e8aa30 0x559d38e8ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:52186/0 (socket says 172.21.15.50:52186) 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f542b7fe640 1 -- 172.21.15.50:0/1933010146 learned_addr learned my addr 172.21.15.50:0/1933010146 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f542a7fc640 1 --2- 172.21.15.50:0/1933010146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e95c10 0x559d38e98000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f542b7fe640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f54140a3ee0 0x7f54140a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f5413fff640 1 -- 172.21.15.50:0/1804449473 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f541c0d0470 con 0x7f542c1010f0 2024-04-23T18:18:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f542b7fe640 1 -- 172.21.15.50:0/1933010146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e95c10 msgr2=0x559d38e98000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f542b7fe640 1 --2- 172.21.15.50:0/1933010146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e95c10 0x559d38e98000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f542b7fe640 1 -- 172.21.15.50:0/1933010146 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 msgr2=0x559d38e956d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f542b7fe640 1 --2- 172.21.15.50:0/1933010146 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38e956d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.144+0000 7f542b7fe640 1 -- 172.21.15.50:0/1933010146 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d38afe210 con 0x559d38e8aa30 2024-04-23T18:18:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.145+0000 7f542b7fe640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f54140a3ee0 0x7f54140a63a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5424009fd0 tx=0x7f542400e000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.145+0000 7f542b7fe640 1 --2- 172.21.15.50:0/1933010146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e8aa30 0x559d38e8ae10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5424014e00 tx=0x7f54240117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70e20c0e4d84a3ba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.145+0000 7f54117fa640 1 -- 172.21.15.50:0/1933010146 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424009c20 con 0x559d38e8aa30 2024-04-23T18:18:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.145+0000 7f54117fa640 1 -- 172.21.15.50:0/1933010146 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5424025070 con 0x559d38e8aa30 2024-04-23T18:18:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.145+0000 7f54117fa640 1 -- 172.21.15.50:0/1933010146 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424004520 con 0x559d38e8aa30 2024-04-23T18:18:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.146+0000 7f543422e740 1 -- 172.21.15.50:0/1933010146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e8aa30 msgr2=0x559d38e8ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.146+0000 7f543422e740 1 --2- 172.21.15.50:0/1933010146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e8aa30 0x559d38e8ae10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5424014e00 tx=0x7f54240117b0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.146+0000 7f543422e740 1 -- 172.21.15.50:0/1933010146 shutdown_connections 2024-04-23T18:18:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.146+0000 7f543422e740 1 --2- 172.21.15.50:0/1933010146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e95c10 0x559d38e98000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.146+0000 7f543422e740 1 --2- 172.21.15.50:0/1933010146 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38e956d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.146+0000 7f543422e740 1 --2- 172.21.15.50:0/1933010146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e8aa30 0x559d38e8ae10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.146+0000 7f543422e740 1 -- 172.21.15.50:0/1933010146 >> 172.21.15.50:0/1933010146 conn(0x559d38e86f40 msgr2=0x559d38e87320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.146+0000 7f543422e740 1 -- 172.21.15.50:0/1933010146 shutdown_connections 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.146+0000 7f543422e740 1 -- 172.21.15.50:0/1933010146 wait complete. 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f543422e740 1 -- start start 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f543422e740 1 Processor -- start 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f543422e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8aa30 0x559d38e8eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f543422e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e8b350 0x559d38e8f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f543422e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38e8f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f542affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8aa30 0x559d38e8eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f542affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8aa30 0x559d38e8eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59930/0 (socket says 172.21.15.50:59930) 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f543422e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559d38e827c0 con 0x559d38e8aa30 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f542affd640 1 -- 172.21.15.50:0/1424613982 learned_addr learned my addr 172.21.15.50:0/1424613982 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f543422e740 1 -- 172.21.15.50:0/1424613982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559d38d860b0 con 0x559d38e8b350 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f543422e740 1 -- 172.21.15.50:0/1424613982 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559d38e82940 con 0x559d38e95c10 2024-04-23T18:18:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f542a7fc640 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e8b350 0x559d38e8f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f543422e740 10 client.0 _ll_register_callbacks cb 0x559d38d73d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.147+0000 7f542b7fe640 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38e8f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f542affd640 1 -- 172.21.15.50:0/1424613982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 msgr2=0x559d38e8f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f542affd640 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38e8f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f542affd640 1 -- 172.21.15.50:0/1424613982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e8b350 msgr2=0x559d38e8f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f542affd640 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e8b350 0x559d38e8f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f542affd640 1 -- 172.21.15.50:0/1424613982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d38bca620 con 0x559d38e8aa30 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f542b7fe640 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38e8f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f542affd640 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8aa30 0x559d38e8eae0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f541c098710 tx=0x7f541c017ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1031e0aeb4649034 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f540eff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f540eff5640 1 -- 172.21.15.50:0/1424613982 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c09c9b0 con 0x559d38e8aa30 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f540eff5640 1 -- 172.21.15.50:0/1424613982 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f541c0041d0 con 0x559d38e8aa30 2024-04-23T18:18:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f543422e740 1 -- 172.21.15.50:0/1424613982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5424025070 con 0x559d38e8aa30 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f540eff5640 1 -- 172.21.15.50:0/1424613982 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f541c066dc0 con 0x559d38e8aa30 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f543422e740 10 client.15573 fetch_fsmap learned FSMap version 21 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f543422e740 1 -- 172.21.15.50:0/1424613982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559d38ea8b60 con 0x559d38e8aa30 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.148+0000 7f540eff5640 1 -- 172.21.15.50:0/1424613982 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5424025070 con 0x559d38e8aa30 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.149+0000 7f540eff5640 1 -- 172.21.15.50:0/1424613982 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f541c098920 con 0x559d38e8aa30 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.149+0000 7f543422e740 10 client.15573 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.149+0000 7f543422e740 10 client.15573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.149+0000 7f543422e740 20 client.15573 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.149+0000 7f543422e740 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559d38ea9fd0 0x559d38eac3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.149+0000 7f543422e740 4 client.15573 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.149+0000 7f543422e740 1 -- 172.21.15.50:0/1424613982 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d38df9390 con 0x559d38ea9fd0 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.149+0000 7f542a7fc640 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559d38ea9fd0 0x559d38eac3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.150+0000 7f540eff5640 10 client.15573 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.150+0000 7f542a7fc640 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559d38ea9fd0 0x559d38eac3c0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cedc9354ab4525b server_cookie=37767ddfaaaabef1 in_seq=0 out_seq=0 2024-04-23T18:18:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.150+0000 7f540eff5640 1 -- 172.21.15.50:0/1424613982 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559d38df9390 con 0x559d38ea9fd0 2024-04-23T18:18:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.150+0000 7f540eff5640 10 client.15573 handle_command_reply: tid=0 2024-04-23T18:18:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.150+0000 7f543422e740 1 client.15573 shutdown 2024-04-23T18:18:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.151+0000 7f543422e740 1 -- 172.21.15.50:0/1424613982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8aa30 msgr2=0x559d38e8eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.151+0000 7f543422e740 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8aa30 0x559d38e8eae0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f541c098710 tx=0x7f541c017ba0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.151+0000 7f543422e740 1 -- 172.21.15.50:0/1424613982 shutdown_connections 2024-04-23T18:18:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.151+0000 7f543422e740 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559d38ea9fd0 0x559d38eac3c0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.151+0000 7f543422e740 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38e8f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.151+0000 7f543422e740 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e8b350 0x559d38e8f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.151+0000 7f543422e740 1 --2- 172.21.15.50:0/1424613982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8aa30 0x559d38e8eae0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.151+0000 7f543422e740 1 -- 172.21.15.50:0/1424613982 >> 172.21.15.50:0/1424613982 conn(0x559d38e86f40 msgr2=0x559d38e81cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.151+0000 7f540eff5640 0 client.15573 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.151+0000 7f543422e740 1 -- 172.21.15.50:0/1424613982 shutdown_connections 2024-04-23T18:18:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.151+0000 7f543422e740 1 -- 172.21.15.50:0/1424613982 wait complete. 2024-04-23T18:18:59.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.151+0000 7f543422e740 20 client.15573 trim_cache size 0 max 16384 2024-04-23T18:18:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.181+0000 7f543422e740 1 Processor -- start 2024-04-23T18:18:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.181+0000 7f543422e740 1 -- start start 2024-04-23T18:18:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.181+0000 7f543422e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38e92320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.181+0000 7f543422e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38e8f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.181+0000 7f543422e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e94ec0 0x559d38eb1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.181+0000 7f543422e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1ae170 con 0x559d38e8b350 2024-04-23T18:18:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.181+0000 7f543422e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1aba40 con 0x559d38e94ec0 2024-04-23T18:18:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.181+0000 7f543422e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1a98b0 con 0x559d38e95c10 2024-04-23T18:18:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.182+0000 7f542a7fc640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38e8f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.182+0000 7f542affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38e92320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.182+0000 7f542a7fc640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38e8f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:52204/0 (socket says 172.21.15.50:52204) 2024-04-23T18:18:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.182+0000 7f542a7fc640 1 -- 172.21.15.50:0/236541993 learned_addr learned my addr 172.21.15.50:0/236541993 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.182+0000 7f542b7fe640 1 --2- 172.21.15.50:0/236541993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e94ec0 0x559d38eb1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.182+0000 7f542a7fc640 1 -- 172.21.15.50:0/236541993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e94ec0 msgr2=0x559d38eb1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.182+0000 7f542a7fc640 1 --2- 172.21.15.50:0/236541993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e94ec0 0x559d38eb1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.182+0000 7f542a7fc640 1 -- 172.21.15.50:0/236541993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 msgr2=0x559d38e92320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.182+0000 7f542a7fc640 1 --2- 172.21.15.50:0/236541993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38e92320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.182+0000 7f542a7fc640 1 -- 172.21.15.50:0/236541993 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d38e98000 con 0x559d38e95c10 2024-04-23T18:18:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.182+0000 7f542affd640 1 --2- 172.21.15.50:0/236541993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38e92320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.182+0000 7f542a7fc640 1 --2- 172.21.15.50:0/236541993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38e8f3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5420005c40 tx=0x7f54200056a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49c9986f7d487104 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:59.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.183+0000 7f5410ff9640 1 -- 172.21.15.50:0/236541993 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542004d070 con 0x559d38e95c10 2024-04-23T18:18:59.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.183+0000 7f5410ff9640 1 -- 172.21.15.50:0/236541993 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5420047340 con 0x559d38e95c10 2024-04-23T18:18:59.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.183+0000 7f5410ff9640 1 -- 172.21.15.50:0/236541993 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420035020 con 0x559d38e95c10 2024-04-23T18:18:59.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.184+0000 7f543422e740 1 -- 172.21.15.50:0/236541993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 msgr2=0x559d38e8f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:59.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.184+0000 7f543422e740 1 --2- 172.21.15.50:0/236541993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38e8f3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5420005c40 tx=0x7f54200056a0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.184+0000 7f543422e740 1 -- 172.21.15.50:0/236541993 shutdown_connections 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.184+0000 7f543422e740 1 --2- 172.21.15.50:0/236541993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e94ec0 0x559d38eb1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.184+0000 7f543422e740 1 --2- 172.21.15.50:0/236541993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38e8f3b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.184+0000 7f543422e740 1 --2- 172.21.15.50:0/236541993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38e92320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.184+0000 7f543422e740 1 -- 172.21.15.50:0/236541993 >> 172.21.15.50:0/236541993 conn(0x559d38e86f40 msgr2=0x559d38e81cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.184+0000 7f543422e740 1 -- 172.21.15.50:0/236541993 shutdown_connections 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.184+0000 7f543422e740 1 -- 172.21.15.50:0/236541993 wait complete. 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.184+0000 7f543422e740 1 -- start start 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.184+0000 7f543422e740 1 Processor -- start 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f543422e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38ea2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f543422e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e94ec0 0x559d38ea32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f543422e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38f0f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f543422e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1a9f00 con 0x559d38e8b350 2024-04-23T18:18:59.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f543422e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1ab370 con 0x559d38e94ec0 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f543422e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1aa5d0 con 0x559d38e95c10 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f543422e740 10 client.0 _ll_register_callbacks cb 0x7f541c09c130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f542affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38ea2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f542affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38ea2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59950/0 (socket says 172.21.15.50:59950) 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f542affd640 1 -- 172.21.15.50:0/2218291335 learned_addr learned my addr 172.21.15.50:0/2218291335 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f542a7fc640 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e94ec0 0x559d38ea32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f542b7fe640 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38f0f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f542affd640 1 -- 172.21.15.50:0/2218291335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 msgr2=0x559d38f0f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f542affd640 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38f0f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f542affd640 1 -- 172.21.15.50:0/2218291335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e94ec0 msgr2=0x559d38ea32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f542affd640 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e94ec0 0x559d38ea32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f542affd640 1 -- 172.21.15.50:0/2218291335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d38f0faa0 con 0x559d38e8b350 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.185+0000 7f542b7fe640 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38f0f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.186+0000 7f542affd640 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38ea2d60 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f541c09c240 tx=0x7f541c004400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806b3669fbd80643 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:18:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.186+0000 7f540eff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:18:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.186+0000 7f540eff5640 1 -- 172.21.15.50:0/2218291335 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c098550 con 0x559d38e8b350 2024-04-23T18:18:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.186+0000 7f540eff5640 1 -- 172.21.15.50:0/2218291335 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f541c0a4c90 con 0x559d38e8b350 2024-04-23T18:18:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.186+0000 7f543422e740 1 -- 172.21.15.50:0/2218291335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5420047340 con 0x559d38e8b350 2024-04-23T18:18:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.186+0000 7f540eff5640 1 -- 172.21.15.50:0/2218291335 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f541c09c9b0 con 0x559d38e8b350 2024-04-23T18:18:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.186+0000 7f543422e740 10 client.15579 fetch_fsmap learned FSMap version 21 2024-04-23T18:18:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.186+0000 7f543422e740 1 -- 172.21.15.50:0/2218291335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559d38f10bb0 con 0x559d38e8b350 2024-04-23T18:18:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.187+0000 7f540eff5640 1 -- 172.21.15.50:0/2218291335 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5420047340 con 0x559d38e8b350 2024-04-23T18:18:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.188+0000 7f540eff5640 1 -- 172.21.15.50:0/2218291335 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f541c09c510 con 0x559d38e8b350 2024-04-23T18:18:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.188+0000 7f543422e740 10 client.15579 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:18:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.188+0000 7f543422e740 10 client.15579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:18:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.188+0000 7f543422e740 20 client.15579 populate_metadata read hostname 'smithi050' 2024-04-23T18:18:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.188+0000 7f543422e740 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559d38f112d0 0x559d38f136c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:18:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.188+0000 7f543422e740 4 client.15579 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:18:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.188+0000 7f543422e740 1 -- 172.21.15.50:0/2218291335 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559d38df3200 con 0x559d38f112d0 2024-04-23T18:18:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.188+0000 7f542a7fc640 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559d38f112d0 0x559d38f136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:18:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.188+0000 7f542a7fc640 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559d38f112d0 0x559d38f136c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8832989f42bbd235 server_cookie=a5c3546a5eab5bdc in_seq=0 out_seq=0 2024-04-23T18:18:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.188+0000 7f540eff5640 10 client.15579 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.196+0000 7f540eff5640 1 -- 172.21.15.50:0/2218291335 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559d38df3200 con 0x559d38f112d0 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.196+0000 7f540eff5640 10 client.15579 handle_command_reply: tid=0 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.196+0000 7f543422e740 1 client.15579 shutdown 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.196+0000 7f543422e740 1 -- 172.21.15.50:0/2218291335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 msgr2=0x559d38ea2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.196+0000 7f543422e740 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38ea2d60 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f541c09c240 tx=0x7f541c004400 comp rx=0 tx=0).stop 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.196+0000 7f543422e740 1 -- 172.21.15.50:0/2218291335 shutdown_connections 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.196+0000 7f543422e740 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559d38f112d0 0x559d38f136c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.196+0000 7f543422e740 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559d38e95c10 0x559d38f0f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.196+0000 7f543422e740 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d38e94ec0 0x559d38ea32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.196+0000 7f543422e740 1 --2- 172.21.15.50:0/2218291335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559d38e8b350 0x559d38ea2d60 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.196+0000 7f543422e740 1 -- 172.21.15.50:0/2218291335 >> 172.21.15.50:0/2218291335 conn(0x559d38e86f40 msgr2=0x559d38e8c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f540eff5640 0 client.15579 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f543422e740 1 -- 172.21.15.50:0/2218291335 shutdown_connections 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f543422e740 1 -- 172.21.15.50:0/2218291335 wait complete. 2024-04-23T18:18:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f543422e740 20 client.15579 trim_cache size 0 max 16384 2024-04-23T18:18:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:18:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:18:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:18:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:18:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f5431519640 1 -- 172.21.15.50:0/1804449473 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f54140a3ee0 msgr2=0x7f54140a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f5431519640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f54140a3ee0 0x7f54140a63a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5424009fd0 tx=0x7f542400e000 comp rx=0 tx=0).stop 2024-04-23T18:18:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f5431519640 1 -- 172.21.15.50:0/1804449473 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f542c1010f0 msgr2=0x7f542c198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:18:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f5431519640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f542c1010f0 0x7f542c198670 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f541c00d950 tx=0x7f541c00de10 comp rx=0 tx=0).stop 2024-04-23T18:18:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f5431519640 1 -- 172.21.15.50:0/1804449473 shutdown_connections 2024-04-23T18:18:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f5431519640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f54140a3ee0 0x7f54140a63a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f5431519640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f542c19cf30 0x7f542c19d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f5431519640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f542c106170 0x7f542c198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f5431519640 1 --2- 172.21.15.50:0/1804449473 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f542c1010f0 0x7f542c198670 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:18:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.197+0000 7f5431519640 1 -- 172.21.15.50:0/1804449473 >> 172.21.15.50:0/1804449473 conn(0x7f542c077fd0 msgr2=0x7f542c0699b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:18:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.198+0000 7f5431519640 1 -- 172.21.15.50:0/1804449473 shutdown_connections 2024-04-23T18:18:59.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:18:59.198+0000 7f5431519640 1 -- 172.21.15.50:0/1804449473 wait complete. 2024-04-23T18:19:00.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:00 smithi188 ceph-mon[32694]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-04-23T18:19:00.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:00 smithi050 ceph-mon[29187]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-04-23T18:19:00.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:00 smithi177 ceph-mon[32308]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-04-23T18:19:01.444 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:19:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:19:01.008+0000 7f0edafb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:19:01.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:19:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:19:01.008+0000 7f0edafb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:19:01.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:19:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:19:01.008+0000 7f0edafb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:19:01.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:19:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:19:01.008+0000 7f0edafb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:19:01.445 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:19:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:19:01.008+0000 7f0edafb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:19:02.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:02 smithi188 ceph-mon[32694]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-23T18:19:02.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:02 smithi050 ceph-mon[29187]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-23T18:19:02.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:02 smithi177 ceph-mon[32308]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-23T18:19:04.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:04 smithi188 ceph-mon[32694]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-23T18:19:04.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:04 smithi188 ceph-mon[32694]: mgrmap e25: x(active, since 32s), standbys: y, z 2024-04-23T18:19:04.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:04 smithi050 ceph-mon[29187]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-23T18:19:04.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:04 smithi050 ceph-mon[29187]: mgrmap e25: x(active, since 32s), standbys: y, z 2024-04-23T18:19:04.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:04 smithi177 ceph-mon[32308]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-23T18:19:04.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:04 smithi177 ceph-mon[32308]: mgrmap e25: x(active, since 32s), standbys: y, z 2024-04-23T18:19:06.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:06 smithi188 ceph-mon[32694]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-04-23T18:19:06.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:06 smithi050 ceph-mon[29187]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-04-23T18:19:06.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:06 smithi177 ceph-mon[32308]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-04-23T18:19:08.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:08 smithi188 ceph-mon[32694]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-23T18:19:08.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:08 smithi050 ceph-mon[29187]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-23T18:19:08.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:08 smithi177 ceph-mon[32308]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-23T18:19:10.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:10 smithi188 ceph-mon[32694]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-23T18:19:10.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:10 smithi050 ceph-mon[29187]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-23T18:19:10.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:10 smithi177 ceph-mon[32308]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-23T18:19:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.331+0000 7f96104a7640 1 -- 172.21.15.50:0/3406989834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9608101aa0 msgr2=0x7f9608105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.331+0000 7f96104a7640 1 --2- 172.21.15.50:0/3406989834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9608101aa0 0x7f9608105a90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f95f8009900 tx=0x7f95f802a3f0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.331+0000 7f96104a7640 1 -- 172.21.15.50:0/3406989834 shutdown_connections 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.331+0000 7f96104a7640 1 --2- 172.21.15.50:0/3406989834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9608106170 0x7f960810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.331+0000 7f96104a7640 1 --2- 172.21.15.50:0/3406989834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9608101aa0 0x7f9608105a90 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.331+0000 7f96104a7640 1 --2- 172.21.15.50:0/3406989834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96081010f0 0x7f96081014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.331+0000 7f96104a7640 1 -- 172.21.15.50:0/3406989834 >> 172.21.15.50:0/3406989834 conn(0x7f9608077fd0 msgr2=0x7f96080783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.331+0000 7f96104a7640 1 -- 172.21.15.50:0/3406989834 shutdown_connections 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.331+0000 7f96104a7640 1 -- 172.21.15.50:0/3406989834 wait complete. 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f96104a7640 1 Processor -- start 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f96104a7640 1 -- start start 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f96104a7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96081010f0 0x7f96081986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f96104a7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9608106170 0x7f9608198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f96104a7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f960819cf60 0x7f960819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f960ea44640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f960819cf60 0x7f960819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f960e243640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96081010f0 0x7f96081986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f960ea44640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f960819cf60 0x7f960819d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47050/0 (socket says 172.21.15.50:47050) 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f960e243640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96081010f0 0x7f96081986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:48570/0 (socket says 172.21.15.50:48570) 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f960ea44640 1 -- 172.21.15.50:0/2030522066 learned_addr learned my addr 172.21.15.50:0/2030522066 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f96104a7640 1 -- 172.21.15.50:0/2030522066 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9608111100 con 0x7f960819cf60 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f96104a7640 1 -- 172.21.15.50:0/2030522066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9608110f80 con 0x7f96081010f0 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.332+0000 7f96104a7640 1 -- 172.21.15.50:0/2030522066 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9608111280 con 0x7f9608106170 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.333+0000 7f960da42640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9608106170 0x7f9608198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.333+0000 7f960e243640 1 -- 172.21.15.50:0/2030522066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9608106170 msgr2=0x7f9608198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.333+0000 7f960e243640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9608106170 0x7f9608198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.333+0000 7f960e243640 1 -- 172.21.15.50:0/2030522066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f960819cf60 msgr2=0x7f960819d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.333+0000 7f960e243640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f960819cf60 0x7f960819d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.333+0000 7f960e243640 1 -- 172.21.15.50:0/2030522066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960819da50 con 0x7f96081010f0 2024-04-23T18:19:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.333+0000 7f960e243640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96081010f0 0x7f96081986a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f960400e9f0 tx=0x7f960400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.333+0000 7f95ff7fe640 1 -- 172.21.15.50:0/2030522066 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960400ccc0 con 0x7f96081010f0 2024-04-23T18:19:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.333+0000 7f96104a7640 1 -- 172.21.15.50:0/2030522066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960819dd40 con 0x7f96081010f0 2024-04-23T18:19:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.333+0000 7f96104a7640 1 -- 172.21.15.50:0/2030522066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96081a5550 con 0x7f96081010f0 2024-04-23T18:19:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.333+0000 7f95ff7fe640 1 -- 172.21.15.50:0/2030522066 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96040108e0 con 0x7f96081010f0 2024-04-23T18:19:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.333+0000 7f95ff7fe640 1 -- 172.21.15.50:0/2030522066 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9604018880 con 0x7f96081010f0 2024-04-23T18:19:12.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:12 smithi188 ceph-mon[32694]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-04-23T18:19:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.336+0000 7f95ff7fe640 1 -- 172.21.15.50:0/2030522066 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f9604010430 con 0x7f96081010f0 2024-04-23T18:19:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.337+0000 7f96131c4740 1 Processor -- start 2024-04-23T18:19:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.337+0000 7f96131c4740 1 -- start start 2024-04-23T18:19:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.337+0000 7f96131c4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e861a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.337+0000 7f96131c4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e86296d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.337+0000 7f96131c4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e862c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.337+0000 7f96131c4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5606e857f970 con 0x5606e861a520 2024-04-23T18:19:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.337+0000 7f96131c4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5606e851c4a0 con 0x5606e861ae40 2024-04-23T18:19:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.337+0000 7f96131c4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5606e857b930 con 0x5606e8629c10 2024-04-23T18:19:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960ea44640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e861a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960ea44640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e861a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47060/0 (socket says 172.21.15.50:47060) 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960ea44640 1 -- 172.21.15.50:0/1915079984 learned_addr learned my addr 172.21.15.50:0/1915079984 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f95ff7fe640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f95f40a3f60 0x7f95f40a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960e243640 1 --2- 172.21.15.50:0/1915079984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e86296d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960da42640 1 --2- 172.21.15.50:0/1915079984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e862c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960ea44640 1 -- 172.21.15.50:0/1915079984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 msgr2=0x5606e862c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960ea44640 1 --2- 172.21.15.50:0/1915079984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e862c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960ea44640 1 -- 172.21.15.50:0/1915079984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 msgr2=0x5606e86296d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960ea44640 1 --2- 172.21.15.50:0/1915079984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e86296d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960ea44640 1 -- 172.21.15.50:0/1915079984 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606e8292210 con 0x5606e861a520 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960da42640 1 --2- 172.21.15.50:0/1915079984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e862c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960ea44640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f95f40a3f60 0x7f95f40a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f95ff7fe640 1 -- 172.21.15.50:0/2030522066 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f96040d0d10 con 0x7f96081010f0 2024-04-23T18:19:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.338+0000 7f960ea44640 1 --2- 172.21.15.50:0/1915079984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e861a900 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9600011570 tx=0x7f9600011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92b89cece201e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.339+0000 7f95fcbfe640 1 -- 172.21.15.50:0/1915079984 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600017050 con 0x5606e861a520 2024-04-23T18:19:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.339+0000 7f95fcbfe640 1 -- 172.21.15.50:0/1915079984 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f960001bd80 con 0x5606e861a520 2024-04-23T18:19:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.339+0000 7f95fcbfe640 1 -- 172.21.15.50:0/1915079984 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600026bd0 con 0x5606e861a520 2024-04-23T18:19:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.340+0000 7f96131c4740 1 -- 172.21.15.50:0/1915079984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 msgr2=0x5606e861a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.340+0000 7f96131c4740 1 --2- 172.21.15.50:0/1915079984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e861a900 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9600011570 tx=0x7f9600011a30 comp rx=0 tx=0).stop 2024-04-23T18:19:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.340+0000 7f960ea44640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f95f40a3f60 0x7f95f40a6420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9600023000 tx=0x7f9600024000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.340+0000 7f96131c4740 1 -- 172.21.15.50:0/1915079984 shutdown_connections 2024-04-23T18:19:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.340+0000 7f96131c4740 1 --2- 172.21.15.50:0/1915079984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e862c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.340+0000 7f96131c4740 1 --2- 172.21.15.50:0/1915079984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e86296d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.340+0000 7f96131c4740 1 --2- 172.21.15.50:0/1915079984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e861a900 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.340+0000 7f96131c4740 1 -- 172.21.15.50:0/1915079984 >> 172.21.15.50:0/1915079984 conn(0x5606e8616b30 msgr2=0x5606e8616f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.340+0000 7f96131c4740 1 -- 172.21.15.50:0/1915079984 shutdown_connections 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.340+0000 7f96131c4740 1 -- 172.21.15.50:0/1915079984 wait complete. 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.340+0000 7f96131c4740 1 -- start start 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.340+0000 7f96131c4740 1 Processor -- start 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f96131c4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e8629460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f96131c4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8626750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f96131c4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e8626c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f96131c4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5606e862c840 con 0x5606e861a520 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f96131c4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5606e851a0b0 con 0x5606e861ae40 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f96131c4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5606e862c9c0 con 0x5606e8629c10 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f96131c4740 10 client.0 _ll_register_callbacks cb 0x5606e8507d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f960e243640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e8629460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f960e243640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e8629460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47072/0 (socket says 172.21.15.50:47072) 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f960e243640 1 -- 172.21.15.50:0/367968876 learned_addr learned my addr 172.21.15.50:0/367968876 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f960da42640 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8626750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f960e243640 1 -- 172.21.15.50:0/367968876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 msgr2=0x5606e8626c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f960e243640 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e8626c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f960e243640 1 -- 172.21.15.50:0/367968876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 msgr2=0x5606e8626750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f960e243640 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8626750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f960e243640 1 -- 172.21.15.50:0/367968876 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606e835e620 con 0x5606e861a520 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f960da42640 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8626750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f960e243640 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e8629460 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9604098f80 tx=0x7f9604009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875d4b855e9f39d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.341+0000 7f95f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f95f1ffb640 1 -- 172.21.15.50:0/367968876 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96040a0e20 con 0x5606e861a520 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f95f1ffb640 1 -- 172.21.15.50:0/367968876 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f96040a5250 con 0x5606e861a520 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f96131c4740 1 -- 172.21.15.50:0/367968876 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f960001bd80 con 0x5606e861a520 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f95f1ffb640 1 -- 172.21.15.50:0/367968876 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f96040a56e0 con 0x5606e861a520 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f96131c4740 10 client.15597 fetch_fsmap learned FSMap version 21 2024-04-23T18:19:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f96131c4740 1 -- 172.21.15.50:0/367968876 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5606e86386a0 con 0x5606e861a520 2024-04-23T18:19:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f95f1ffb640 1 -- 172.21.15.50:0/367968876 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f960001bd80 con 0x5606e861a520 2024-04-23T18:19:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f95f1ffb640 1 -- 172.21.15.50:0/367968876 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f96040ce5d0 con 0x5606e861a520 2024-04-23T18:19:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f96131c4740 10 client.15597 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:19:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f96131c4740 10 client.15597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f96131c4740 20 client.15597 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f96131c4740 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5606e8639b10 0x5606e863bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f96131c4740 4 client.15597 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.342+0000 7f96131c4740 1 -- 172.21.15.50:0/367968876 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606e858d390 con 0x5606e8639b10 2024-04-23T18:19:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.343+0000 7f960da42640 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5606e8639b10 0x5606e863bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.343+0000 7f95f1ffb640 10 client.15597 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:12.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.343+0000 7f960da42640 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5606e8639b10 0x5606e863bf40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4472cbf7078fa735 server_cookie=bc453b1259405928 in_seq=0 out_seq=0 2024-04-23T18:19:12.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f95f1ffb640 1 -- 172.21.15.50:0/367968876 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5606e858d390 con 0x5606e8639b10 2024-04-23T18:19:12.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f95f1ffb640 10 client.15597 handle_command_reply: tid=0 2024-04-23T18:19:12.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f96131c4740 1 client.15597 shutdown 2024-04-23T18:19:12.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f96131c4740 1 -- 172.21.15.50:0/367968876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 msgr2=0x5606e8629460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f96131c4740 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e8629460 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9604098f80 tx=0x7f9604009700 comp rx=0 tx=0).stop 2024-04-23T18:19:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f96131c4740 1 -- 172.21.15.50:0/367968876 shutdown_connections 2024-04-23T18:19:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f96131c4740 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5606e8639b10 0x5606e863bf40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f96131c4740 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e8626c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f96131c4740 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8626750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f96131c4740 1 --2- 172.21.15.50:0/367968876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e8629460 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f95f1ffb640 0 client.15597 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f96131c4740 1 -- 172.21.15.50:0/367968876 >> 172.21.15.50:0/367968876 conn(0x5606e8616b30 msgr2=0x5606e861c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f96131c4740 1 -- 172.21.15.50:0/367968876 shutdown_connections 2024-04-23T18:19:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f96131c4740 1 -- 172.21.15.50:0/367968876 wait complete. 2024-04-23T18:19:12.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.344+0000 7f96131c4740 20 client.15597 trim_cache size 0 max 16384 2024-04-23T18:19:12.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.374+0000 7f96131c4740 1 Processor -- start 2024-04-23T18:19:12.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.374+0000 7f96131c4740 1 -- start start 2024-04-23T18:19:12.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.374+0000 7f96131c4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e8635700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.374+0000 7f96131c4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8628f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.374+0000 7f96131c4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e8641850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.374+0000 7f96131c4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec011090 con 0x5606e861a520 2024-04-23T18:19:12.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f96131c4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec00e960 con 0x5606e861ae40 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f96131c4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec00c7b0 con 0x5606e8629c10 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f960e243640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e8635700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f960e243640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e8635700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47074/0 (socket says 172.21.15.50:47074) 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f960e243640 1 -- 172.21.15.50:0/708020253 learned_addr learned my addr 172.21.15.50:0/708020253 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f960ea44640 1 --2- 172.21.15.50:0/708020253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e8641850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f960da42640 1 --2- 172.21.15.50:0/708020253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8628f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f960ea44640 1 -- 172.21.15.50:0/708020253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 msgr2=0x5606e8628f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f960ea44640 1 --2- 172.21.15.50:0/708020253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8628f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f960ea44640 1 -- 172.21.15.50:0/708020253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 msgr2=0x5606e8635700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f960ea44640 1 --2- 172.21.15.50:0/708020253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e8635700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f960ea44640 1 -- 172.21.15.50:0/708020253 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606e8616960 con 0x5606e8629c10 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f960e243640 1 --2- 172.21.15.50:0/708020253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e8635700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.375+0000 7f960ea44640 1 --2- 172.21.15.50:0/708020253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e8641850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9600011f30 tx=0x7f9600013030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9cf7f2ce4129c63 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.376+0000 7f95f3fff640 1 -- 172.21.15.50:0/708020253 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600032070 con 0x5606e8629c10 2024-04-23T18:19:12.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.376+0000 7f95f3fff640 1 -- 172.21.15.50:0/708020253 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9600009500 con 0x5606e8629c10 2024-04-23T18:19:12.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.376+0000 7f95f3fff640 1 -- 172.21.15.50:0/708020253 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600026440 con 0x5606e8629c10 2024-04-23T18:19:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.377+0000 7f96131c4740 1 -- 172.21.15.50:0/708020253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 msgr2=0x5606e8641850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.377+0000 7f96131c4740 1 --2- 172.21.15.50:0/708020253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e8641850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9600011f30 tx=0x7f9600013030 comp rx=0 tx=0).stop 2024-04-23T18:19:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.377+0000 7f96131c4740 1 -- 172.21.15.50:0/708020253 shutdown_connections 2024-04-23T18:19:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.377+0000 7f96131c4740 1 --2- 172.21.15.50:0/708020253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e8641850 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.377+0000 7f96131c4740 1 --2- 172.21.15.50:0/708020253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8628f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.377+0000 7f96131c4740 1 --2- 172.21.15.50:0/708020253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e8635700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.377+0000 7f96131c4740 1 -- 172.21.15.50:0/708020253 >> 172.21.15.50:0/708020253 conn(0x5606e8616b30 msgr2=0x5606e8626710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.377+0000 7f96131c4740 1 -- 172.21.15.50:0/708020253 shutdown_connections 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.377+0000 7f96131c4740 1 -- 172.21.15.50:0/708020253 wait complete. 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.377+0000 7f96131c4740 1 -- start start 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.377+0000 7f96131c4740 1 Processor -- start 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f96131c4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e86321a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f960e243640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e86321a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f960e243640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e86321a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47088/0 (socket says 172.21.15.50:47088) 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f96131c4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8623290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f960e243640 1 -- 172.21.15.50:0/1263205805 learned_addr learned my addr 172.21.15.50:0/1263205805 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f96131c4740 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e86239d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f96131c4740 1 -- 172.21.15.50:0/1263205805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec00ce20 con 0x5606e861a520 2024-04-23T18:19:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f96131c4740 1 -- 172.21.15.50:0/1263205805 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec00e290 con 0x5606e861ae40 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f96131c4740 1 -- 172.21.15.50:0/1263205805 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec00d4f0 con 0x5606e8629c10 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f960da42640 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8623290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f960ea44640 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e86239d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f960e243640 1 -- 172.21.15.50:0/1263205805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 msgr2=0x5606e86239d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f960e243640 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e86239d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f960e243640 1 -- 172.21.15.50:0/1263205805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 msgr2=0x5606e8623290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f960e243640 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8623290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.378+0000 7f960e243640 1 -- 172.21.15.50:0/1263205805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606e8617d60 con 0x5606e861a520 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.379+0000 7f960e243640 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e86321a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9604098f80 tx=0x7f960400e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ce300ca433547d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.379+0000 7f95f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.379+0000 7f95f1ffb640 1 -- 172.21.15.50:0/1263205805 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96040cebc0 con 0x5606e861a520 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.379+0000 7f95f1ffb640 1 -- 172.21.15.50:0/1263205805 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f9604098940 con 0x5606e861a520 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.379+0000 7f96131c4740 10 client.0 _ll_register_callbacks cb 0x7f960409c8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.379+0000 7f96131c4740 1 -- 172.21.15.50:0/1263205805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9600009500 con 0x5606e861a520 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.379+0000 7f95f1ffb640 1 -- 172.21.15.50:0/1263205805 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f960409cb20 con 0x5606e861a520 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.379+0000 7f96131c4740 10 client.15603 fetch_fsmap learned FSMap version 21 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.379+0000 7f96131c4740 1 -- 172.21.15.50:0/1263205805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({fsmap=21}) -- 0x5606e869fdc0 con 0x5606e861a520 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.380+0000 7f95f1ffb640 1 -- 172.21.15.50:0/1263205805 <== mon.0 v2:172.21.15.50:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f96040b9020 con 0x5606e861a520 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.380+0000 7f96131c4740 10 client.15603 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.380+0000 7f96131c4740 10 client.15603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.380+0000 7f96131c4740 20 client.15603 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.380+0000 7f96131c4740 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5606e86a0330 0x5606e86a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.380+0000 7f96131c4740 4 client.15603 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.380+0000 7f96131c4740 1 -- 172.21.15.50:0/1263205805 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f9600009500 con 0x5606e86a0330 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.380+0000 7f960da42640 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5606e86a0330 0x5606e86a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.381+0000 7f95f1ffb640 10 client.15603 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.381+0000 7f960da42640 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5606e86a0330 0x5606e86a27f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17eafb41287f7f9f server_cookie=55b9c99859deae1d in_seq=0 out_seq=0 2024-04-23T18:19:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.386+0000 7f95f1ffb640 1 -- 172.21.15.50:0/1263205805 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f9600009500 con 0x5606e86a0330 2024-04-23T18:19:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.386+0000 7f95f1ffb640 10 client.15603 handle_command_reply: tid=0 2024-04-23T18:19:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.386+0000 7f96131c4740 1 client.15603 shutdown 2024-04-23T18:19:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.387+0000 7f96131c4740 1 -- 172.21.15.50:0/1263205805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 msgr2=0x5606e86321a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.387+0000 7f96131c4740 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e86321a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9604098f80 tx=0x7f960400e940 comp rx=0 tx=0).stop 2024-04-23T18:19:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.387+0000 7f96131c4740 1 -- 172.21.15.50:0/1263205805 shutdown_connections 2024-04-23T18:19:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.387+0000 7f96131c4740 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5606e86a0330 0x5606e86a27f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.387+0000 7f96131c4740 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5606e8629c10 0x5606e86239d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.387+0000 7f95f1ffb640 0 client.15603 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.387+0000 7f96131c4740 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606e861ae40 0x5606e8623290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.387+0000 7f96131c4740 1 --2- 172.21.15.50:0/1263205805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5606e861a520 0x5606e86321a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.387+0000 7f96131c4740 1 -- 172.21.15.50:0/1263205805 >> 172.21.15.50:0/1263205805 conn(0x5606e8616b30 msgr2=0x5606e8621670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.387+0000 7f96131c4740 1 -- 172.21.15.50:0/1263205805 shutdown_connections 2024-04-23T18:19:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.387+0000 7f96131c4740 1 -- 172.21.15.50:0/1263205805 wait complete. 2024-04-23T18:19:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.387+0000 7f96131c4740 20 client.15603 trim_cache size 0 max 16384 2024-04-23T18:19:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:19:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:19:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:19:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:19:12.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.388+0000 7f96104a7640 1 -- 172.21.15.50:0/2030522066 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f95f40a3f60 msgr2=0x7f95f40a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:12.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.388+0000 7f96104a7640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f95f40a3f60 0x7f95f40a6420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9600023000 tx=0x7f9600024000 comp rx=0 tx=0).stop 2024-04-23T18:19:12.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.388+0000 7f96104a7640 1 -- 172.21.15.50:0/2030522066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96081010f0 msgr2=0x7f96081986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:12.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.388+0000 7f96104a7640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96081010f0 0x7f96081986a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f960400e9f0 tx=0x7f960400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.388+0000 7f96104a7640 1 -- 172.21.15.50:0/2030522066 shutdown_connections 2024-04-23T18:19:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.388+0000 7f96104a7640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f95f40a3f60 0x7f95f40a6420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.388+0000 7f96104a7640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f960819cf60 0x7f960819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.388+0000 7f96104a7640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9608106170 0x7f9608198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.388+0000 7f96104a7640 1 --2- 172.21.15.50:0/2030522066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96081010f0 0x7f96081986a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.388+0000 7f96104a7640 1 -- 172.21.15.50:0/2030522066 >> 172.21.15.50:0/2030522066 conn(0x7f9608077fd0 msgr2=0x7f9608069970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.388+0000 7f96104a7640 1 -- 172.21.15.50:0/2030522066 shutdown_connections 2024-04-23T18:19:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:12.388+0000 7f96104a7640 1 -- 172.21.15.50:0/2030522066 wait complete. 2024-04-23T18:19:12.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:12 smithi050 ceph-mon[29187]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-04-23T18:19:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:12 smithi177 ceph-mon[32308]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-04-23T18:19:14.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:14 smithi188 ceph-mon[32694]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-23T18:19:14.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:14 smithi050 ceph-mon[29187]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-23T18:19:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:14 smithi177 ceph-mon[32308]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-23T18:19:16.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:16 smithi188 ceph-mon[32694]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T18:19:16.340 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:16 smithi177 ceph-mon[32308]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T18:19:16.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:16 smithi050 ceph-mon[29187]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T18:19:18.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:18 smithi188 ceph-mon[32694]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-23T18:19:18.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:18 smithi050 ceph-mon[29187]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-23T18:19:18.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:18 smithi177 ceph-mon[32308]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-23T18:19:20.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:20 smithi188 ceph-mon[32694]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-23T18:19:20.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:20 smithi050 ceph-mon[29187]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-23T18:19:20.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:20 smithi177 ceph-mon[32308]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-23T18:19:22.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:22 smithi188 ceph-mon[32694]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T18:19:22.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:22 smithi050 ceph-mon[29187]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T18:19:22.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:22 smithi177 ceph-mon[32308]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T18:19:24.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:24 smithi188 ceph-mon[32694]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-23T18:19:24.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:24 smithi050 ceph-mon[29187]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-23T18:19:24.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:24 smithi177 ceph-mon[32308]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-23T18:19:25.187 INFO:tasks.workunit.client.0.smithi050.stderr:Note: switching to '57e0caf76fdb7ab8b1358588f292d08519163844'. 2024-04-23T18:19:25.187 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-23T18:19:25.187 INFO:tasks.workunit.client.0.smithi050.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-23T18:19:25.187 INFO:tasks.workunit.client.0.smithi050.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-23T18:19:25.187 INFO:tasks.workunit.client.0.smithi050.stderr:state without impacting any branches by switching back to a branch. 2024-04-23T18:19:25.187 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-23T18:19:25.187 INFO:tasks.workunit.client.0.smithi050.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-23T18:19:25.187 INFO:tasks.workunit.client.0.smithi050.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-23T18:19:25.187 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-23T18:19:25.187 INFO:tasks.workunit.client.0.smithi050.stderr: git switch -c 2024-04-23T18:19:25.188 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-23T18:19:25.188 INFO:tasks.workunit.client.0.smithi050.stderr:Or undo this operation with: 2024-04-23T18:19:25.188 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-23T18:19:25.188 INFO:tasks.workunit.client.0.smithi050.stderr: git switch - 2024-04-23T18:19:25.188 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-23T18:19:25.188 INFO:tasks.workunit.client.0.smithi050.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-23T18:19:25.188 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-23T18:19:25.188 INFO:tasks.workunit.client.0.smithi050.stderr:HEAD is now at 57e0caf76fd qa/suites/fs: add quiescer to the fs suite 2024-04-23T18:19:25.189 DEBUG:teuthology.orchestra.run.smithi050:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-23T18:19:25.220 INFO:tasks.workunit.client.0.smithi050.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-23T18:19:25.223 INFO:tasks.workunit.client.0.smithi050.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-23T18:19:25.223 INFO:tasks.workunit.client.0.smithi050.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-23T18:19:25.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:19:25.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.364+0000 7f36042fa640 1 -- 172.21.15.50:0/2019406700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35fc101110 msgr2=0x7f35fc1014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.364+0000 7f36042fa640 1 --2- 172.21.15.50:0/2019406700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35fc101110 0x7f35fc1014f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009900 tx=0x7f35f802a410 comp rx=0 tx=0).stop 2024-04-23T18:19:25.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.364+0000 7f36042fa640 1 -- 172.21.15.50:0/2019406700 shutdown_connections 2024-04-23T18:19:25.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.364+0000 7f36042fa640 1 --2- 172.21.15.50:0/2019406700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35fc106190 0x7f35fc10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.364+0000 7f36042fa640 1 --2- 172.21.15.50:0/2019406700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35fc101ac0 0x7f35fc105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.364+0000 7f36042fa640 1 --2- 172.21.15.50:0/2019406700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35fc101110 0x7f35fc1014f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.364+0000 7f36042fa640 1 -- 172.21.15.50:0/2019406700 >> 172.21.15.50:0/2019406700 conn(0x7f35fc077fb0 msgr2=0x7f35fc0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.364+0000 7f36042fa640 1 -- 172.21.15.50:0/2019406700 shutdown_connections 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.364+0000 7f36042fa640 1 -- 172.21.15.50:0/2019406700 wait complete. 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f36042fa640 1 Processor -- start 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f36042fa640 1 -- start start 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f36042fa640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35fc101110 0x7f35fc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f36042fa640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35fc106190 0x7f35fc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f36042fa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35fc19cfd0 0x7f35fc19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f36042fa640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc112210 con 0x7f35fc101110 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f36042fa640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc112090 con 0x7f35fc19cfd0 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f36042fa640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc112390 con 0x7f35fc106190 2024-04-23T18:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f3602096640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35fc101110 0x7f35fc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f3602096640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35fc101110 0x7f35fc198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36526/0 (socket says 172.21.15.50:36526) 2024-04-23T18:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f3602096640 1 -- 172.21.15.50:0/1996070678 learned_addr learned my addr 172.21.15.50:0/1996070678 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f3601895640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35fc106190 0x7f35fc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f3602897640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35fc19cfd0 0x7f35fc19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.365+0000 7f3602096640 1 -- 172.21.15.50:0/1996070678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35fc106190 msgr2=0x7f35fc198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.366+0000 7f3602096640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35fc106190 0x7f35fc198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.366+0000 7f3602096640 1 -- 172.21.15.50:0/1996070678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35fc19cfd0 msgr2=0x7f35fc19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.366+0000 7f3602096640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35fc19cfd0 0x7f35fc19d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.366+0000 7f3602096640 1 -- 172.21.15.50:0/1996070678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35fc19dac0 con 0x7f35fc101110 2024-04-23T18:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.366+0000 7f3602096640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35fc101110 0x7f35fc198710 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f35f8005940 tx=0x7f35f80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.366+0000 7f35f37fe640 1 -- 172.21.15.50:0/1996070678 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f803f030 con 0x7f35fc101110 2024-04-23T18:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.366+0000 7f36042fa640 1 -- 172.21.15.50:0/1996070678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35fc19dd50 con 0x7f35fc101110 2024-04-23T18:19:25.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.366+0000 7f35f37fe640 1 -- 172.21.15.50:0/1996070678 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f8003cb0 con 0x7f35fc101110 2024-04-23T18:19:25.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.366+0000 7f35f37fe640 1 -- 172.21.15.50:0/1996070678 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f8035070 con 0x7f35fc101110 2024-04-23T18:19:25.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.366+0000 7f36042fa640 1 -- 172.21.15.50:0/1996070678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35fc1a5570 con 0x7f35fc101110 2024-04-23T18:19:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.367+0000 7f36042fa640 1 -- 172.21.15.50:0/1996070678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35fc0691c0 con 0x7f35fc101110 2024-04-23T18:19:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.368+0000 7f35f37fe640 1 -- 172.21.15.50:0/1996070678 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f35f8042030 con 0x7f35fc101110 2024-04-23T18:19:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.369+0000 7f35f37fe640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f35e80a3e40 0x7f35e80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.369+0000 7f3601895640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f35e80a3e40 0x7f35e80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.369+0000 7f35f37fe640 1 -- 172.21.15.50:0/1996070678 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f35f80f12d0 con 0x7f35fc101110 2024-04-23T18:19:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.369+0000 7f3601895640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f35e80a3e40 0x7f35e80a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f35fc1009d0 tx=0x7f35ec00b390 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.374+0000 7f35f37fe640 1 -- 172.21.15.50:0/1996070678 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f35f80bd7a0 con 0x7f35fc101110 2024-04-23T18:19:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.478+0000 7fcda0210640 1 -- 172.21.15.50:0/3260817584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd98103b40 msgr2=0x7fcd9810c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.478+0000 7fcda0210640 1 --2- 172.21.15.50:0/3260817584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd98103b40 0x7fcd9810c020 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcd88009900 tx=0x7fcd8802a480 comp rx=0 tx=0).stop 2024-04-23T18:19:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.478+0000 7fcda0210640 1 -- 172.21.15.50:0/3260817584 shutdown_connections 2024-04-23T18:19:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.478+0000 7fcda0210640 1 --2- 172.21.15.50:0/3260817584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd9810c560 0x7fcd9810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.478+0000 7fcda0210640 1 --2- 172.21.15.50:0/3260817584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd98103b40 0x7fcd9810c020 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.478+0000 7fcda0210640 1 --2- 172.21.15.50:0/3260817584 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcd98103220 0x7fcd98103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.478+0000 7fcda0210640 1 -- 172.21.15.50:0/3260817584 >> 172.21.15.50:0/3260817584 conn(0x7fcd98077fd0 msgr2=0x7fcd980783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.478+0000 7fcda0210640 1 -- 172.21.15.50:0/3260817584 shutdown_connections 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.478+0000 7fcda0210640 1 -- 172.21.15.50:0/3260817584 wait complete. 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.479+0000 7fcda0210640 1 Processor -- start 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.479+0000 7fcda0210640 1 -- start start 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.479+0000 7fcda0210640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcd98103220 0x7fcd981a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.479+0000 7fcda0210640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd98103b40 0x7fcd981a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.479+0000 7fcda0210640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9810c560 0x7fcd9819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.479+0000 7fcda0210640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcd98112200 con 0x7fcd9810c560 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.479+0000 7fcda0210640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd98112080 con 0x7fcd98103b40 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.479+0000 7fcda0210640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcd98112380 con 0x7fcd98103220 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.479+0000 7fcd9e7ad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9810c560 0x7fcd9819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.479+0000 7fcd9e7ad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9810c560 0x7fcd9819a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36554/0 (socket says 172.21.15.50:36554) 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.479+0000 7fcd9e7ad640 1 -- 172.21.15.50:0/1430952495 learned_addr learned my addr 172.21.15.50:0/1430952495 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.479+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcd98103220 0x7fcd981a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.480+0000 7fcd9d7ab640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd98103b40 0x7fcd981a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.480+0000 7fcd9e7ad640 1 -- 172.21.15.50:0/1430952495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcd98103220 msgr2=0x7fcd981a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.480+0000 7fcd9e7ad640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcd98103220 0x7fcd981a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.480+0000 7fcd9e7ad640 1 -- 172.21.15.50:0/1430952495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd98103b40 msgr2=0x7fcd981a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.480+0000 7fcd9e7ad640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd98103b40 0x7fcd981a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.480+0000 7fcd9e7ad640 1 -- 172.21.15.50:0/1430952495 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd9819afd0 con 0x7fcd9810c560 2024-04-23T18:19:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.480+0000 7fcd9e7ad640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9810c560 0x7fcd9819a710 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009fd0 tx=0x7fcd9000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.480+0000 7fcd8effd640 1 -- 172.21.15.50:0/1430952495 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd90017070 con 0x7fcd9810c560 2024-04-23T18:19:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.480+0000 7fcd8effd640 1 -- 172.21.15.50:0/1430952495 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd90003e20 con 0x7fcd9810c560 2024-04-23T18:19:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.480+0000 7fcd8effd640 1 -- 172.21.15.50:0/1430952495 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd90010460 con 0x7fcd9810c560 2024-04-23T18:19:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.480+0000 7fcda0210640 1 -- 172.21.15.50:0/1430952495 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd9819b2c0 con 0x7fcd9810c560 2024-04-23T18:19:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.480+0000 7fcda0210640 1 -- 172.21.15.50:0/1430952495 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd981a75d0 con 0x7fcd9810c560 2024-04-23T18:19:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.482+0000 7fcd8effd640 1 -- 172.21.15.50:0/1430952495 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fcd90003970 con 0x7fcd9810c560 2024-04-23T18:19:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.483+0000 7fcd8effd640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcd840a3ee0 0x7fcd840a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.483+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcd840a3ee0 0x7fcd840a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.483+0000 7fcd8effd640 1 -- 172.21.15.50:0/1430952495 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fcd900d0710 con 0x7fcd9810c560 2024-04-23T18:19:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcd840a3ee0 0x7fcd840a63a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005fd0 tx=0x7fcd9400b390 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcda3001740 1 Processor -- start 2024-04-23T18:19:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcda3001740 1 -- start start 2024-04-23T18:19:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcda3001740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3ea6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcda3001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3ea6f40 0x563da3eb12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcda3001740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3eb1800 0x563da3eb3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcda3001740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563da3e0b970 con 0x563da3ea6620 2024-04-23T18:19:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcda3001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563da3da84a0 con 0x563da3ea6f40 2024-04-23T18:19:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcda3001740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563da3e07930 con 0x563da3eb1800 2024-04-23T18:19:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcd9dfac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3ea6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcd9dfac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3ea6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36558/0 (socket says 172.21.15.50:36558) 2024-04-23T18:19:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcd9dfac640 1 -- 172.21.15.50:0/3978548863 learned_addr learned my addr 172.21.15.50:0/3978548863 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcd9e7ad640 1 --2- 172.21.15.50:0/3978548863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3eb1800 0x563da3eb3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.484+0000 7fcd9d7ab640 1 --2- 172.21.15.50:0/3978548863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3ea6f40 0x563da3eb12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.485+0000 7fcd9dfac640 1 -- 172.21.15.50:0/3978548863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3eb1800 msgr2=0x563da3eb3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.485+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/3978548863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3eb1800 0x563da3eb3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.485+0000 7fcd9dfac640 1 -- 172.21.15.50:0/3978548863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3ea6f40 msgr2=0x563da3eb12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.485+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/3978548863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3ea6f40 0x563da3eb12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.485+0000 7fcd9dfac640 1 -- 172.21.15.50:0/3978548863 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563da3b1e210 con 0x563da3ea6620 2024-04-23T18:19:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.485+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/3978548863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3ea6a00 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400b620 tx=0x7fcd94013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=190f004deb7e5be7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.485+0000 7fcd83fff640 1 -- 172.21.15.50:0/3978548863 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd94003e90 con 0x563da3ea6620 2024-04-23T18:19:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.485+0000 7fcd83fff640 1 -- 172.21.15.50:0/3978548863 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd940054d0 con 0x563da3ea6620 2024-04-23T18:19:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.485+0000 7fcd83fff640 1 -- 172.21.15.50:0/3978548863 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9401e660 con 0x563da3ea6620 2024-04-23T18:19:25.485 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.486+0000 7fcda3001740 1 -- 172.21.15.50:0/3978548863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 msgr2=0x563da3ea6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.486+0000 7fcda3001740 1 --2- 172.21.15.50:0/3978548863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3ea6a00 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400b620 tx=0x7fcd94013000 comp rx=0 tx=0).stop 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.486+0000 7fcda3001740 1 -- 172.21.15.50:0/3978548863 shutdown_connections 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.486+0000 7fcda3001740 1 --2- 172.21.15.50:0/3978548863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3eb1800 0x563da3eb3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.486+0000 7fcda3001740 1 --2- 172.21.15.50:0/3978548863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3ea6f40 0x563da3eb12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.486+0000 7fcda3001740 1 --2- 172.21.15.50:0/3978548863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3ea6a00 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.486+0000 7fcda3001740 1 -- 172.21.15.50:0/3978548863 >> 172.21.15.50:0/3978548863 conn(0x563da3ea2b30 msgr2=0x563da3ea2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.486+0000 7fcda3001740 1 -- 172.21.15.50:0/3978548863 shutdown_connections 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.486+0000 7fcda3001740 1 -- 172.21.15.50:0/3978548863 wait complete. 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.486+0000 7fcda3001740 1 -- start start 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.486+0000 7fcda3001740 1 Processor -- start 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcda3001740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3eb1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcda3001740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3ea6f40 0x563da3eae410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcda3001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3eb1800 0x563da3eae950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcda3001740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563da3eb4430 con 0x563da3ea6620 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcda3001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563da3da60b0 con 0x563da3eb1800 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcda3001740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563da3eb45b0 con 0x563da3ea6f40 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcda3001740 10 client.0 _ll_register_callbacks cb 0x563da3d93d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcd9dfac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3eb1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcd9dfac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3eb1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36560/0 (socket says 172.21.15.50:36560) 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcd9dfac640 1 -- 172.21.15.50:0/2698592520 learned_addr learned my addr 172.21.15.50:0/2698592520 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcd9d7ab640 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3ea6f40 0x563da3eae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcd9e7ad640 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3eb1800 0x563da3eae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcd9dfac640 1 -- 172.21.15.50:0/2698592520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3ea6f40 msgr2=0x563da3eae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3ea6f40 0x563da3eae410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcd9dfac640 1 -- 172.21.15.50:0/2698592520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3eb1800 msgr2=0x563da3eae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3eb1800 0x563da3eae950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcd9dfac640 1 -- 172.21.15.50:0/2698592520 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563da3bea620 con 0x563da3ea6620 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3eb1120 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcd94011ca0 tx=0x7fcd9400b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74eb5e643d957f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.487+0000 7fcd817fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcd817fa640 1 -- 172.21.15.50:0/2698592520 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd94036060 con 0x563da3ea6620 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcd817fa640 1 -- 172.21.15.50:0/2698592520 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fcd94004e80 con 0x563da3ea6620 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcda3001740 1 -- 172.21.15.50:0/2698592520 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd940054d0 con 0x563da3ea6620 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcd817fa640 1 -- 172.21.15.50:0/2698592520 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcd94005080 con 0x563da3ea6620 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcda3001740 10 client.15621 fetch_fsmap learned FSMap version 21 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcda3001740 1 -- 172.21.15.50:0/2698592520 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563da3ec0360 con 0x563da3ea6620 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcd817fa640 1 -- 172.21.15.50:0/2698592520 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd940054d0 con 0x563da3ea6620 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcd817fa640 1 -- 172.21.15.50:0/2698592520 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcd9401ec70 con 0x563da3ea6620 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcda3001740 10 client.15621 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcda3001740 10 client.15621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcda3001740 20 client.15621 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcda3001740 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da3ec17c0 0x563da3ec3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcda3001740 4 client.15621 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.488+0000 7fcda3001740 1 -- 172.21.15.50:0/2698592520 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563da3e19390 con 0x563da3ec17c0 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.489+0000 7fcd9d7ab640 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da3ec17c0 0x563da3ec3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.489+0000 7fcd9d7ab640 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da3ec17c0 0x563da3ec3bb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d9f97003a6954df server_cookie=c278c8bc05a50f1 in_seq=0 out_seq=0 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.489+0000 7fcd817fa640 10 client.15621 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcd817fa640 1 -- 172.21.15.50:0/2698592520 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563da3e19390 con 0x563da3ec17c0 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcd817fa640 10 client.15621 handle_command_reply: tid=0 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcda3001740 1 client.15621 shutdown 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcda3001740 1 -- 172.21.15.50:0/2698592520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 msgr2=0x563da3eb1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcda3001740 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3eb1120 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcd94011ca0 tx=0x7fcd9400b4c0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcda3001740 1 -- 172.21.15.50:0/2698592520 shutdown_connections 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcda3001740 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da3ec17c0 0x563da3ec3bb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcda3001740 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3eb1800 0x563da3eae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcda3001740 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3ea6f40 0x563da3eae410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcd817fa640 0 client.15621 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcda3001740 1 --2- 172.21.15.50:0/2698592520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3eb1120 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcda3001740 1 -- 172.21.15.50:0/2698592520 >> 172.21.15.50:0/2698592520 conn(0x563da3ea2b30 msgr2=0x563da3ea88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcda3001740 1 -- 172.21.15.50:0/2698592520 shutdown_connections 2024-04-23T18:19:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcda3001740 1 -- 172.21.15.50:0/2698592520 wait complete. 2024-04-23T18:19:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.490+0000 7fcda3001740 20 client.15621 trim_cache size 0 max 16384 2024-04-23T18:19:25.500 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcda3001740 1 Processor -- start 2024-04-23T18:19:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcda3001740 1 -- start start 2024-04-23T18:19:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcda3001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3ea6620 0x563da3ebd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcda3001740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6f40 0x563da3eb0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcda3001740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3eb1800 0x563da3ec92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcda3001740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcd981b2560 con 0x563da3ea6f40 2024-04-23T18:19:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcda3001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd981afe30 con 0x563da3ea6620 2024-04-23T18:19:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcda3001740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcd981adc80 con 0x563da3eb1800 2024-04-23T18:19:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcd9d7ab640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6f40 0x563da3eb0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcd9d7ab640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6f40 0x563da3eb0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36568/0 (socket says 172.21.15.50:36568) 2024-04-23T18:19:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcd9d7ab640 1 -- 172.21.15.50:0/603208158 learned_addr learned my addr 172.21.15.50:0/603208158 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/603208158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3ea6620 0x563da3ebd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.505+0000 7fcd9e7ad640 1 --2- 172.21.15.50:0/603208158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3eb1800 0x563da3ec92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.506+0000 7fcd9d7ab640 1 -- 172.21.15.50:0/603208158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3eb1800 msgr2=0x563da3ec92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.506+0000 7fcd9d7ab640 1 --2- 172.21.15.50:0/603208158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3eb1800 0x563da3ec92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.506+0000 7fcd9d7ab640 1 -- 172.21.15.50:0/603208158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3ea6620 msgr2=0x563da3ebd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.506+0000 7fcd9d7ab640 1 --2- 172.21.15.50:0/603208158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3ea6620 0x563da3ebd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.506+0000 7fcd9d7ab640 1 -- 172.21.15.50:0/603208158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563da3eb3fb0 con 0x563da3ea6f40 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.506+0000 7fcd9e7ad640 1 --2- 172.21.15.50:0/603208158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3eb1800 0x563da3ec92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.506+0000 7fcd9d7ab640 1 --2- 172.21.15.50:0/603208158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6f40 0x563da3eb0c50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcd88004330 tx=0x7fcd88005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e2bd070e48f904 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.506+0000 7fcd837fe640 1 -- 172.21.15.50:0/603208158 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd88035040 con 0x563da3ea6f40 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.506+0000 7fcd837fe640 1 -- 172.21.15.50:0/603208158 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd880468f0 con 0x563da3ea6f40 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.506+0000 7fcd837fe640 1 -- 172.21.15.50:0/603208158 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd88004770 con 0x563da3ea6f40 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.507+0000 7fcda3001740 1 -- 172.21.15.50:0/603208158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6f40 msgr2=0x563da3eb0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.507+0000 7fcda3001740 1 --2- 172.21.15.50:0/603208158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6f40 0x563da3eb0c50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcd88004330 tx=0x7fcd88005a20 comp rx=0 tx=0).stop 2024-04-23T18:19:25.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.507+0000 7fcda3001740 1 -- 172.21.15.50:0/603208158 shutdown_connections 2024-04-23T18:19:25.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.507+0000 7fcda3001740 1 --2- 172.21.15.50:0/603208158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3eb1800 0x563da3ec92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.507+0000 7fcda3001740 1 --2- 172.21.15.50:0/603208158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6f40 0x563da3eb0c50 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.507+0000 7fcda3001740 1 --2- 172.21.15.50:0/603208158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3ea6620 0x563da3ebd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.507+0000 7fcda3001740 1 -- 172.21.15.50:0/603208158 >> 172.21.15.50:0/603208158 conn(0x563da3ea2b30 msgr2=0x563da3ea99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.507+0000 7fcda3001740 1 -- 172.21.15.50:0/603208158 shutdown_connections 2024-04-23T18:19:25.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.507+0000 7fcda3001740 1 -- 172.21.15.50:0/603208158 wait complete. 2024-04-23T18:19:25.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.507+0000 7fcda3001740 1 -- start start 2024-04-23T18:19:25.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.507+0000 7fcda3001740 1 Processor -- start 2024-04-23T18:19:25.505 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcda3001740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3f1fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcda3001740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3ea6f40 0x563da3f203e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcda3001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3eb1800 0x563da3eb4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcda3001740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcd981ae2f0 con 0x563da3ea6620 2024-04-23T18:19:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcda3001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd981af760 con 0x563da3eb1800 2024-04-23T18:19:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcda3001740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcd981ae9c0 con 0x563da3ea6f40 2024-04-23T18:19:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcda3001740 10 client.0 _ll_register_callbacks cb 0x7fcd94003b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcd9dfac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3f1fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcd9dfac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3f1fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36584/0 (socket says 172.21.15.50:36584) 2024-04-23T18:19:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcd9dfac640 1 -- 172.21.15.50:0/1617899839 learned_addr learned my addr 172.21.15.50:0/1617899839 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcd9e7ad640 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3eb1800 0x563da3eb4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcd9d7ab640 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3ea6f40 0x563da3f203e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcd9dfac640 1 -- 172.21.15.50:0/1617899839 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3ea6f40 msgr2=0x563da3f203e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3ea6f40 0x563da3f203e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcd9dfac640 1 -- 172.21.15.50:0/1617899839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3eb1800 msgr2=0x563da3eb4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3eb1800 0x563da3eb4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcd9dfac640 1 -- 172.21.15.50:0/1617899839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563da3eb52c0 con 0x563da3ea6620 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcd9dfac640 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3f1fea0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005790 tx=0x7fcd94005010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c02adbad7ae3d9f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.508+0000 7fcd817fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcd817fa640 1 -- 172.21.15.50:0/1617899839 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd94036040 con 0x563da3ea6620 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcda3001740 1 -- 172.21.15.50:0/1617899839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd880468f0 con 0x563da3ea6620 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcd817fa640 1 -- 172.21.15.50:0/1617899839 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fcd94005220 con 0x563da3ea6620 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcd817fa640 1 -- 172.21.15.50:0/1617899839 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcd94003950 con 0x563da3ea6620 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcda3001740 10 client.15627 fetch_fsmap learned FSMap version 21 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcda3001740 1 -- 172.21.15.50:0/1617899839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563da3f27650 con 0x563da3ea6620 2024-04-23T18:19:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcd817fa640 1 -- 172.21.15.50:0/1617899839 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd880468f0 con 0x563da3ea6620 2024-04-23T18:19:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcd817fa640 1 -- 172.21.15.50:0/1617899839 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcd940436a0 con 0x563da3ea6620 2024-04-23T18:19:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcda3001740 10 client.15627 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:19:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcda3001740 10 client.15627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcda3001740 20 client.15627 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcda3001740 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da3f27d60 0x563da3f2a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcda3001740 4 client.15627 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:19:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.509+0000 7fcda3001740 1 -- 172.21.15.50:0/1617899839 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563da3e13200 con 0x563da3f27d60 2024-04-23T18:19:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.510+0000 7fcd9d7ab640 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da3f27d60 0x563da3f2a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.510+0000 7fcd9d7ab640 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da3f27d60 0x563da3f2a150 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a95cf2aa050dde0e server_cookie=a1374d04e668435b in_seq=0 out_seq=0 2024-04-23T18:19:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.510+0000 7fcd817fa640 10 client.15627 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcd817fa640 1 -- 172.21.15.50:0/1617899839 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563da3e13200 con 0x563da3f27d60 2024-04-23T18:19:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcd817fa640 10 client.15627 handle_command_reply: tid=0 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcda3001740 1 client.15627 shutdown 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcda3001740 1 -- 172.21.15.50:0/1617899839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 msgr2=0x563da3f1fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcda3001740 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3f1fea0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005790 tx=0x7fcd94005010 comp rx=0 tx=0).stop 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcda3001740 1 -- 172.21.15.50:0/1617899839 shutdown_connections 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcda3001740 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da3f27d60 0x563da3f2a150 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcda3001740 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3eb1800 0x563da3eb4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcda3001740 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3ea6f40 0x563da3f203e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcd817fa640 0 client.15627 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcda3001740 1 --2- 172.21.15.50:0/1617899839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3ea6620 0x563da3f1fea0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcda3001740 1 -- 172.21.15.50:0/1617899839 >> 172.21.15.50:0/1617899839 conn(0x563da3ea2b30 msgr2=0x563da3ea90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcda3001740 1 -- 172.21.15.50:0/1617899839 shutdown_connections 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcda3001740 1 -- 172.21.15.50:0/1617899839 wait complete. 2024-04-23T18:19:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.514+0000 7fcda3001740 20 client.15627 trim_cache size 0 max 16384 2024-04-23T18:19:25.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:19:25.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:19:25.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:19:25.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:19:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.515+0000 7fcda0210640 1 -- 172.21.15.50:0/1430952495 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcd840a3ee0 msgr2=0x7fcd840a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.515+0000 7fcda0210640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcd840a3ee0 0x7fcd840a63a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005fd0 tx=0x7fcd9400b390 comp rx=0 tx=0).stop 2024-04-23T18:19:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.515+0000 7fcda0210640 1 -- 172.21.15.50:0/1430952495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9810c560 msgr2=0x7fcd9819a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.515+0000 7fcda0210640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9810c560 0x7fcd9819a710 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009fd0 tx=0x7fcd9000b260 comp rx=0 tx=0).stop 2024-04-23T18:19:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.515+0000 7fcda0210640 1 -- 172.21.15.50:0/1430952495 shutdown_connections 2024-04-23T18:19:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.515+0000 7fcda0210640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcd840a3ee0 0x7fcd840a63a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.515+0000 7fcda0210640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9810c560 0x7fcd9819a710 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.515+0000 7fcda0210640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd98103b40 0x7fcd981a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.515+0000 7fcda0210640 1 --2- 172.21.15.50:0/1430952495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcd98103220 0x7fcd981a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.515+0000 7fcda0210640 1 -- 172.21.15.50:0/1430952495 >> 172.21.15.50:0/1430952495 conn(0x7fcd98077fd0 msgr2=0x7fcd98108470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.515+0000 7fcda0210640 1 -- 172.21.15.50:0/1430952495 shutdown_connections 2024-04-23T18:19:25.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:25.515+0000 7fcda0210640 1 -- 172.21.15.50:0/1430952495 wait complete. 2024-04-23T18:19:25.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.611+0000 7f36042fa640 1 -- 172.21.15.50:0/1996070678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35fc1093c0 con 0x7f35fc101110 2024-04-23T18:19:25.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.612+0000 7f35f37fe640 1 -- 172.21.15.50:0/1996070678 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11875 (secure 0 0 0) 0x7f35f80bd0a0 con 0x7f35fc101110 2024-04-23T18:19:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:19:25.606 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:18:28.904281+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:19:25.606 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 21 2024-04-23T18:19:25.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.614+0000 7f36042fa640 1 -- 172.21.15.50:0/1996070678 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f35e80a3e40 msgr2=0x7f35e80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.614+0000 7f36042fa640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f35e80a3e40 0x7f35e80a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f35fc1009d0 tx=0x7f35ec00b390 comp rx=0 tx=0).stop 2024-04-23T18:19:25.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.614+0000 7f36042fa640 1 -- 172.21.15.50:0/1996070678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35fc101110 msgr2=0x7f35fc198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.614+0000 7f36042fa640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35fc101110 0x7f35fc198710 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f35f8005940 tx=0x7f35f80042f0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.614+0000 7f36042fa640 1 -- 172.21.15.50:0/1996070678 shutdown_connections 2024-04-23T18:19:25.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.614+0000 7f36042fa640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f35e80a3e40 0x7f35e80a6300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.614+0000 7f36042fa640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35fc19cfd0 0x7f35fc19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.614+0000 7f36042fa640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35fc106190 0x7f35fc198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.614+0000 7f36042fa640 1 --2- 172.21.15.50:0/1996070678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35fc101110 0x7f35fc198710 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.614+0000 7f36042fa640 1 -- 172.21.15.50:0/1996070678 >> 172.21.15.50:0/1996070678 conn(0x7f35fc077fb0 msgr2=0x7f35fc0ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.614+0000 7f36042fa640 1 -- 172.21.15.50:0/1996070678 shutdown_connections 2024-04-23T18:19:25.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.614+0000 7f36042fa640 1 -- 172.21.15.50:0/1996070678 wait complete. 2024-04-23T18:19:25.619 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id f5ca02e5 --release --await' 2024-04-23T18:19:25.619 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id f5ca02e5 --release --await 2024-04-23T18:19:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.736+0000 7f3d66b20640 1 -- 172.21.15.50:0/1839192430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d60107270 msgr2=0x7f3d6010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.736+0000 7f3d66b20640 1 --2- 172.21.15.50:0/1839192430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d60107270 0x7f3d6010ee40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009ca0 tx=0x7f3d5802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.736+0000 7f3d66b20640 1 -- 172.21.15.50:0/1839192430 shutdown_connections 2024-04-23T18:19:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.736+0000 7f3d66b20640 1 --2- 172.21.15.50:0/1839192430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d60107270 0x7f3d6010ee40 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.736+0000 7f3d66b20640 1 --2- 172.21.15.50:0/1839192430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d6006b1f0 0x7f3d60106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.736+0000 7f3d66b20640 1 --2- 172.21.15.50:0/1839192430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d6006a840 0x7f3d6006ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.736+0000 7f3d66b20640 1 -- 172.21.15.50:0/1839192430 >> 172.21.15.50:0/1839192430 conn(0x7f3d60077eb0 msgr2=0x7f3d600782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.736+0000 7f3d66b20640 1 -- 172.21.15.50:0/1839192430 shutdown_connections 2024-04-23T18:19:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.736+0000 7f3d66b20640 1 -- 172.21.15.50:0/1839192430 wait complete. 2024-04-23T18:19:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d66b20640 1 Processor -- start 2024-04-23T18:19:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d66b20640 1 -- start start 2024-04-23T18:19:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d66b20640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d6006a840 0x7f3d601986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d66b20640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d60107270 0x7f3d60198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d66b20640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d6019cf90 0x7f3d6019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d66b20640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601121b0 con 0x7f3d6019cf90 2024-04-23T18:19:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d66b20640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d60112030 con 0x7f3d6006a840 2024-04-23T18:19:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d66b20640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3d60112330 con 0x7f3d60107270 2024-04-23T18:19:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d650bd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d6019cf90 0x7f3d6019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d650bd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d6019cf90 0x7f3d6019d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36602/0 (socket says 172.21.15.50:36602) 2024-04-23T18:19:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d650bd640 1 -- 172.21.15.50:0/675890990 learned_addr learned my addr 172.21.15.50:0/675890990 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d648bc640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d6006a840 0x7f3d601986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d5ffff640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d60107270 0x7f3d60198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d650bd640 1 -- 172.21.15.50:0/675890990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d60107270 msgr2=0x7f3d60198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d650bd640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d60107270 0x7f3d60198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d650bd640 1 -- 172.21.15.50:0/675890990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d6006a840 msgr2=0x7f3d601986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d650bd640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d6006a840 0x7f3d601986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.737+0000 7f3d650bd640 1 -- 172.21.15.50:0/675890990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6019da80 con 0x7f3d6019cf90 2024-04-23T18:19:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.738+0000 7f3d650bd640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d6019cf90 0x7f3d6019d440 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3d580098d0 tx=0x7f3d580317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.738+0000 7f3d5dffb640 1 -- 172.21.15.50:0/675890990 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58003b70 con 0x7f3d6019cf90 2024-04-23T18:19:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.738+0000 7f3d5dffb640 1 -- 172.21.15.50:0/675890990 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d58003d10 con 0x7f3d6019cf90 2024-04-23T18:19:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.738+0000 7f3d66b20640 1 -- 172.21.15.50:0/675890990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6019dd10 con 0x7f3d6019cf90 2024-04-23T18:19:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.738+0000 7f3d5dffb640 1 -- 172.21.15.50:0/675890990 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5802fa20 con 0x7f3d6019cf90 2024-04-23T18:19:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.738+0000 7f3d66b20640 1 -- 172.21.15.50:0/675890990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d601a5530 con 0x7f3d6019cf90 2024-04-23T18:19:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.740+0000 7f3d5dffb640 1 -- 172.21.15.50:0/675890990 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f3d5802fc40 con 0x7f3d6019cf90 2024-04-23T18:19:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.740+0000 7f3d69835740 1 Processor -- start 2024-04-23T18:19:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d69835740 1 -- start start 2024-04-23T18:19:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d69835740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cf490 0x55fcb74cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d69835740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d5dffb640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3d4c0a3e40 0x7f3d4c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d69835740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74da480 0x55fcb74dc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d69835740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55fcb74346d0 con 0x55fcb74cfdb0 2024-04-23T18:19:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d69835740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fcb73d11a0 con 0x55fcb74cf490 2024-04-23T18:19:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d69835740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fcb7430690 con 0x55fcb74da480 2024-04-23T18:19:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d5dffb640 1 -- 172.21.15.50:0/675890990 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f3d580f1da0 con 0x7f3d6019cf90 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d648bc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d5ffff640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3d4c0a3e40 0x7f3d4c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d648bc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36618/0 (socket says 172.21.15.50:36618) 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d648bc640 1 -- 172.21.15.50:0/3384408357 learned_addr learned my addr 172.21.15.50:0/3384408357 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d5ffff640 1 --2- 172.21.15.50:0/3384408357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cf490 0x55fcb74cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d650bd640 1 --2- 172.21.15.50:0/3384408357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74da480 0x55fcb74dc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d648bc640 1 -- 172.21.15.50:0/3384408357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74da480 msgr2=0x55fcb74dc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d648bc640 1 --2- 172.21.15.50:0/3384408357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74da480 0x55fcb74dc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d648bc640 1 -- 172.21.15.50:0/3384408357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cf490 msgr2=0x55fcb74cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d648bc640 1 --2- 172.21.15.50:0/3384408357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cf490 0x55fcb74cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d648bc640 1 -- 172.21.15.50:0/3384408357 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcb73db5b0 con 0x55fcb74cfdb0 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d650bd640 1 --2- 172.21.15.50:0/3384408357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74da480 0x55fcb74dc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.741+0000 7f3d5ffff640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3d4c0a3e40 0x7f3d4c0a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3d601087a0 tx=0x7f3d4800ba50 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d648bc640 1 --2- 172.21.15.50:0/3384408357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d9f40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009cf0 tx=0x7f3d50011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b1e27d2451c2ef server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d56ffd640 1 -- 172.21.15.50:0/3384408357 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5001e070 con 0x55fcb74cfdb0 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d56ffd640 1 -- 172.21.15.50:0/3384408357 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d50009de0 con 0x55fcb74cfdb0 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d56ffd640 1 -- 172.21.15.50:0/3384408357 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50019650 con 0x55fcb74cfdb0 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d69835740 1 -- 172.21.15.50:0/3384408357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 msgr2=0x55fcb74d9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d69835740 1 --2- 172.21.15.50:0/3384408357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d9f40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009cf0 tx=0x7f3d50011810 comp rx=0 tx=0).stop 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d69835740 1 -- 172.21.15.50:0/3384408357 shutdown_connections 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d69835740 1 --2- 172.21.15.50:0/3384408357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74da480 0x55fcb74dc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d69835740 1 --2- 172.21.15.50:0/3384408357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d9f40 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d69835740 1 --2- 172.21.15.50:0/3384408357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cf490 0x55fcb74cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d69835740 1 -- 172.21.15.50:0/3384408357 >> 172.21.15.50:0/3384408357 conn(0x55fcb74cb7b0 msgr2=0x55fcb74cbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d69835740 1 -- 172.21.15.50:0/3384408357 shutdown_connections 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.742+0000 7f3d69835740 1 -- 172.21.15.50:0/3384408357 wait complete. 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d69835740 1 -- start start 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d69835740 1 Processor -- start 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d69835740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74cf490 0x55fcb74d9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d69835740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d69835740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74da480 0x55fcb74d78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d69835740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55fcb74dd310 con 0x55fcb74cfdb0 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d69835740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fcb73ceb20 con 0x55fcb74da480 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d69835740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fcb74dd490 con 0x55fcb74cf490 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d69835740 10 client.0 _ll_register_callbacks cb 0x55fcb73ce530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d5ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d5ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36630/0 (socket says 172.21.15.50:36630) 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d648bc640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74cf490 0x55fcb74d9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d5ffff640 1 -- 172.21.15.50:0/4216120657 learned_addr learned my addr 172.21.15.50:0/4216120657 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.743+0000 7f3d650bd640 1 --2- 172.21.15.50:0/4216120657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74da480 0x55fcb74d78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d5ffff640 1 -- 172.21.15.50:0/4216120657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74cf490 msgr2=0x55fcb74d9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d5ffff640 1 --2- 172.21.15.50:0/4216120657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74cf490 0x55fcb74d9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d5ffff640 1 -- 172.21.15.50:0/4216120657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74da480 msgr2=0x55fcb74d78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d5ffff640 1 --2- 172.21.15.50:0/4216120657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74da480 0x55fcb74d78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d5ffff640 1 -- 172.21.15.50:0/4216120657 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcb7146920 con 0x55fcb74cfdb0 2024-04-23T18:19:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d5ffff640 1 --2- 172.21.15.50:0/4216120657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d71b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800bfa0 tx=0x7f3d4801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d59601438390de server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d3ffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d3ffff640 1 -- 172.21.15.50:0/4216120657 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4801eb80 con 0x55fcb74cfdb0 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d69835740 1 -- 172.21.15.50:0/4216120657 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d50009de0 con 0x55fcb74cfdb0 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d3ffff640 1 -- 172.21.15.50:0/4216120657 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f3d4801ed20 con 0x55fcb74cfdb0 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d3ffff640 1 -- 172.21.15.50:0/4216120657 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3d48020400 con 0x55fcb74cfdb0 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d69835740 10 client.15639 fetch_fsmap learned FSMap version 21 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d69835740 1 -- 172.21.15.50:0/4216120657 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fcb7212d30 con 0x55fcb74cfdb0 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.744+0000 7f3d3ffff640 1 -- 172.21.15.50:0/4216120657 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d50009de0 con 0x55fcb74cfdb0 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.745+0000 7f3d3ffff640 1 -- 172.21.15.50:0/4216120657 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f3d4801f070 con 0x55fcb74cfdb0 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.745+0000 7f3d69835740 10 client.15639 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.745+0000 7f3d69835740 10 client.15639 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.745+0000 7f3d69835740 20 client.15639 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.745+0000 7f3d69835740 1 --2- 172.21.15.50:0/4216120657 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fcb74eeaf0 0x55fcb74f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.745+0000 7f3d69835740 4 client.15639 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.745+0000 7f3d69835740 1 -- 172.21.15.50:0/4216120657 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcb73f2500 con 0x55fcb74eeaf0 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.745+0000 7f3d648bc640 1 --2- 172.21.15.50:0/4216120657 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fcb74eeaf0 0x55fcb74f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.745+0000 7f3d648bc640 1 --2- 172.21.15.50:0/4216120657 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fcb74eeaf0 0x55fcb74f0f10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9aa8ad28055fefbf server_cookie=4a8652bd472bf4ac in_seq=0 out_seq=0 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.745+0000 7f3d3ffff640 10 client.15639 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d3ffff640 1 -- 172.21.15.50:0/4216120657 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fcb73f2500 con 0x55fcb74eeaf0 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d3ffff640 10 client.15639 handle_command_reply: tid=0 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d69835740 1 client.15639 shutdown 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d69835740 1 -- 172.21.15.50:0/4216120657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 msgr2=0x55fcb74d71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d69835740 1 --2- 172.21.15.50:0/4216120657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d71b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800bfa0 tx=0x7f3d4801c000 comp rx=0 tx=0).stop 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d69835740 1 -- 172.21.15.50:0/4216120657 shutdown_connections 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d69835740 1 --2- 172.21.15.50:0/4216120657 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fcb74eeaf0 0x55fcb74f0f10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d69835740 1 --2- 172.21.15.50:0/4216120657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74da480 0x55fcb74d78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d3ffff640 0 client.15639 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d69835740 1 --2- 172.21.15.50:0/4216120657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d71b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d69835740 1 --2- 172.21.15.50:0/4216120657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74cf490 0x55fcb74d9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d69835740 1 -- 172.21.15.50:0/4216120657 >> 172.21.15.50:0/4216120657 conn(0x55fcb74cb7b0 msgr2=0x55fcb74ca9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d69835740 1 -- 172.21.15.50:0/4216120657 shutdown_connections 2024-04-23T18:19:25.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d69835740 1 -- 172.21.15.50:0/4216120657 wait complete. 2024-04-23T18:19:25.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.746+0000 7f3d69835740 20 client.15639 trim_cache size 0 max 16384 2024-04-23T18:19:25.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.764+0000 7f3d69835740 1 Processor -- start 2024-04-23T18:19:25.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.764+0000 7f3d69835740 1 -- start start 2024-04-23T18:19:25.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d69835740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cf490 0x55fcb74e6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d69835740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d69835740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74da480 0x55fcb74d9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d69835740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601ae1b0 con 0x55fcb74cfdb0 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d69835740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601aba80 con 0x55fcb74cf490 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d69835740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601a98d0 con 0x55fcb74da480 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d5ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d5ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36632/0 (socket says 172.21.15.50:36632) 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d5ffff640 1 -- 172.21.15.50:0/3056296343 learned_addr learned my addr 172.21.15.50:0/3056296343 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d648bc640 1 --2- 172.21.15.50:0/3056296343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cf490 0x55fcb74e6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d650bd640 1 --2- 172.21.15.50:0/3056296343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74da480 0x55fcb74d9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d5ffff640 1 -- 172.21.15.50:0/3056296343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74da480 msgr2=0x55fcb74d9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d5ffff640 1 --2- 172.21.15.50:0/3056296343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74da480 0x55fcb74d9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d5ffff640 1 -- 172.21.15.50:0/3056296343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cf490 msgr2=0x55fcb74e6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d5ffff640 1 --2- 172.21.15.50:0/3056296343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cf490 0x55fcb74e6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d5ffff640 1 -- 172.21.15.50:0/3056296343 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcb74c5de0 con 0x55fcb74cfdb0 2024-04-23T18:19:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d650bd640 1 --2- 172.21.15.50:0/3056296343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74da480 0x55fcb74d9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:19:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d5ffff640 1 --2- 172.21.15.50:0/3056296343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d99f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3d48010760 tx=0x7f3d48012630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7821355fbbcd05 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d567fc640 1 -- 172.21.15.50:0/3056296343 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4803f070 con 0x55fcb74cfdb0 2024-04-23T18:19:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d567fc640 1 -- 172.21.15.50:0/3056296343 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d480323f0 con 0x55fcb74cfdb0 2024-04-23T18:19:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.765+0000 7f3d567fc640 1 -- 172.21.15.50:0/3056296343 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48032a90 con 0x55fcb74cfdb0 2024-04-23T18:19:25.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.766+0000 7f3d69835740 1 -- 172.21.15.50:0/3056296343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 msgr2=0x55fcb74d99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.766+0000 7f3d69835740 1 --2- 172.21.15.50:0/3056296343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d99f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3d48010760 tx=0x7f3d48012630 comp rx=0 tx=0).stop 2024-04-23T18:19:25.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.766+0000 7f3d69835740 1 -- 172.21.15.50:0/3056296343 shutdown_connections 2024-04-23T18:19:25.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.766+0000 7f3d69835740 1 --2- 172.21.15.50:0/3056296343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74da480 0x55fcb74d9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.766+0000 7f3d69835740 1 --2- 172.21.15.50:0/3056296343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74cfdb0 0x55fcb74d99f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.766+0000 7f3d69835740 1 --2- 172.21.15.50:0/3056296343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cf490 0x55fcb74e6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.766+0000 7f3d69835740 1 -- 172.21.15.50:0/3056296343 >> 172.21.15.50:0/3056296343 conn(0x55fcb74cb7b0 msgr2=0x55fcb74d0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.766+0000 7f3d69835740 1 -- 172.21.15.50:0/3056296343 shutdown_connections 2024-04-23T18:19:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.766+0000 7f3d69835740 1 -- 172.21.15.50:0/3056296343 wait complete. 2024-04-23T18:19:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d69835740 1 -- start start 2024-04-23T18:19:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d69835740 1 Processor -- start 2024-04-23T18:19:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d69835740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74cf490 0x55fcb74e2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d69835740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cfdb0 0x55fcb74e2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d69835740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74da480 0x55fcb74dd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d69835740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601a9f40 con 0x55fcb74da480 2024-04-23T18:19:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d69835740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601ab3b0 con 0x55fcb74cfdb0 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d69835740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601aa610 con 0x55fcb74cf490 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d69835740 10 client.0 _ll_register_callbacks cb 0x7f3d4800beb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d650bd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74da480 0x55fcb74dd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d650bd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74da480 0x55fcb74dd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36638/0 (socket says 172.21.15.50:36638) 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d650bd640 1 -- 172.21.15.50:0/3526032761 learned_addr learned my addr 172.21.15.50:0/3526032761 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d5ffff640 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cfdb0 0x55fcb74e2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d648bc640 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74cf490 0x55fcb74e2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d650bd640 1 -- 172.21.15.50:0/3526032761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74cf490 msgr2=0x55fcb74e2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d650bd640 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74cf490 0x55fcb74e2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d650bd640 1 -- 172.21.15.50:0/3526032761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cfdb0 msgr2=0x55fcb74e2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.767+0000 7f3d650bd640 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cfdb0 0x55fcb74e2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d650bd640 1 -- 172.21.15.50:0/3526032761 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcb74ddf90 con 0x55fcb74da480 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d648bc640 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74cf490 0x55fcb74e2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d650bd640 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74da480 0x55fcb74dd7c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009430 tx=0x7f3d580bd630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74d00e463baff5b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d3ffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d3ffff640 1 -- 172.21.15.50:0/3526032761 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d580bdc10 con 0x55fcb74da480 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d3ffff640 1 -- 172.21.15.50:0/3526032761 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f3d580d7750 con 0x55fcb74da480 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d69835740 1 -- 172.21.15.50:0/3526032761 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d480323f0 con 0x55fcb74da480 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d3ffff640 1 -- 172.21.15.50:0/3526032761 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3d580c1fc0 con 0x55fcb74da480 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d69835740 10 client.15645 fetch_fsmap learned FSMap version 21 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d69835740 1 -- 172.21.15.50:0/3526032761 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fcb7555ba0 con 0x55fcb74da480 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d3ffff640 1 -- 172.21.15.50:0/3526032761 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d480323f0 con 0x55fcb74da480 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d3ffff640 1 -- 172.21.15.50:0/3526032761 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f3d580de070 con 0x55fcb74da480 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d69835740 10 client.15645 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d69835740 10 client.15645 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d69835740 20 client.15645 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d69835740 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fcb75562c0 0x55fcb75586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d69835740 4 client.15645 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f5ca02e5", "release": true, "await": true}] 2024-04-23T18:19:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.768+0000 7f3d69835740 1 -- 172.21.15.50:0/3526032761 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f5ca02e5", "release": true, "await": true}) -- 0x55fcb74420f0 con 0x55fcb75562c0 2024-04-23T18:19:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.769+0000 7f3d648bc640 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fcb75562c0 0x55fcb75586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.769+0000 7f3d648bc640 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fcb75562c0 0x55fcb75586e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc48f3bbc1d276ab server_cookie=d79fdc3021e5e530 in_seq=0 out_seq=0 2024-04-23T18:19:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.769+0000 7f3d3ffff640 10 client.15645 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:25.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.771+0000 7f3d3ffff640 1 -- 172.21.15.50:0/3526032761 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55fcb74420f0 con 0x55fcb75562c0 2024-04-23T18:19:25.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.771+0000 7f3d3ffff640 10 client.15645 handle_command_reply: tid=0 2024-04-23T18:19:25.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.771+0000 7f3d69835740 1 client.15645 shutdown 2024-04-23T18:19:25.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.771+0000 7f3d69835740 1 -- 172.21.15.50:0/3526032761 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74da480 msgr2=0x55fcb74dd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.771+0000 7f3d69835740 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74da480 0x55fcb74dd7c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009430 tx=0x7f3d580bd630 comp rx=0 tx=0).stop 2024-04-23T18:19:25.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.771+0000 7f3d69835740 1 -- 172.21.15.50:0/3526032761 shutdown_connections 2024-04-23T18:19:25.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.771+0000 7f3d69835740 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fcb75562c0 0x55fcb75586e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.771+0000 7f3d69835740 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fcb74da480 0x55fcb74dd7c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.771+0000 7f3d69835740 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcb74cfdb0 0x55fcb74e2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.771+0000 7f3d3ffff640 0 client.15645 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:25.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.771+0000 7f3d69835740 1 --2- 172.21.15.50:0/3526032761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fcb74cf490 0x55fcb74e2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.771+0000 7f3d69835740 1 -- 172.21.15.50:0/3526032761 >> 172.21.15.50:0/3526032761 conn(0x55fcb74cb7b0 msgr2=0x55fcb74d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d69835740 1 -- 172.21.15.50:0/3526032761 shutdown_connections 2024-04-23T18:19:25.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d69835740 1 -- 172.21.15.50:0/3526032761 wait complete. 2024-04-23T18:19:25.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d69835740 20 client.15645 trim_cache size 0 max 16384 2024-04-23T18:19:25.770 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:19:25.770 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 21, 2024-04-23T18:19:25.770 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:19:25.770 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 4, 2024-04-23T18:19:25.770 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:19:25.770 INFO:teuthology.orchestra.run.smithi050.stdout: "f5ca02e5": { 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 4, 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 90.0, 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:25.771 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:25.772 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:25.772 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:25.772 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:19:25.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d66b20640 1 -- 172.21.15.50:0/675890990 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3d4c0a3e40 msgr2=0x7f3d4c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d66b20640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3d4c0a3e40 0x7f3d4c0a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3d601087a0 tx=0x7f3d4800ba50 comp rx=0 tx=0).stop 2024-04-23T18:19:25.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d66b20640 1 -- 172.21.15.50:0/675890990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d6019cf90 msgr2=0x7f3d6019d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d66b20640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d6019cf90 0x7f3d6019d440 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3d580098d0 tx=0x7f3d580317b0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d66b20640 1 -- 172.21.15.50:0/675890990 shutdown_connections 2024-04-23T18:19:25.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d66b20640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3d4c0a3e40 0x7f3d4c0a6300 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d66b20640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d6019cf90 0x7f3d6019d440 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d66b20640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d60107270 0x7f3d60198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d66b20640 1 --2- 172.21.15.50:0/675890990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d6006a840 0x7f3d601986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d66b20640 1 -- 172.21.15.50:0/675890990 >> 172.21.15.50:0/675890990 conn(0x7f3d60077eb0 msgr2=0x7f3d600fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d66b20640 1 -- 172.21.15.50:0/675890990 shutdown_connections 2024-04-23T18:19:25.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.772+0000 7f3d66b20640 1 -- 172.21.15.50:0/675890990 wait complete. 2024-04-23T18:19:25.777 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'f5ca02e5', seconds elapsed: 31.1 2024-04-23T18:19:25.777 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-23T18:19:25.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:19:25.820 INFO:tasks.workunit.client.0.smithi050.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-23T18:19:25.874 INFO:tasks.workunit.client.0.smithi050.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-23T18:19:25.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 -- 172.21.15.50:0/3010720684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd57c1021a0 msgr2=0x7fd57c102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 --2- 172.21.15.50:0/3010720684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd57c1021a0 0x7fd57c102580 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd570009900 tx=0x7fd57002a410 comp rx=0 tx=0).stop 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 -- 172.21.15.50:0/3010720684 shutdown_connections 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 --2- 172.21.15.50:0/3010720684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd57c107270 0x7fd57c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 --2- 172.21.15.50:0/3010720684 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd57c102b50 0x7fd57c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 --2- 172.21.15.50:0/3010720684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd57c1021a0 0x7fd57c102580 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 -- 172.21.15.50:0/3010720684 >> 172.21.15.50:0/3010720684 conn(0x7fd57c079350 msgr2=0x7fd57c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 -- 172.21.15.50:0/3010720684 shutdown_connections 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 -- 172.21.15.50:0/3010720684 wait complete. 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 Processor -- start 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 -- start start 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd57c1021a0 0x7fd57c19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd57c107270 0x7fd57c19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd57c1985a0 0x7fd57c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd57c112200 con 0x7fd57c1985a0 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd57c112080 con 0x7fd57c107270 2024-04-23T18:19:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.888+0000 7fd582b9b640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd57c112380 con 0x7fd57c1021a0 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd581138640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd57c1985a0 0x7fd57c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd581138640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd57c1985a0 0x7fd57c198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36658/0 (socket says 172.21.15.50:36658) 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd581138640 1 -- 172.21.15.50:0/863558836 learned_addr learned my addr 172.21.15.50:0/863558836 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd57bfff640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd57c107270 0x7fd57c19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd580937640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd57c1021a0 0x7fd57c19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd581138640 1 -- 172.21.15.50:0/863558836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd57c1021a0 msgr2=0x7fd57c19e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd581138640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd57c1021a0 0x7fd57c19e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd581138640 1 -- 172.21.15.50:0/863558836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd57c107270 msgr2=0x7fd57c19e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd581138640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd57c107270 0x7fd57c19e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd581138640 1 -- 172.21.15.50:0/863558836 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd57c1990c0 con 0x7fd57c1985a0 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd57bfff640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd57c107270 0x7fd57c19e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd581138640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd57c1985a0 0x7fd57c198a50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd574009fd0 tx=0x7fd57400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd579ffb640 1 -- 172.21.15.50:0/863558836 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd574017070 con 0x7fd57c1985a0 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd579ffb640 1 -- 172.21.15.50:0/863558836 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd574003e20 con 0x7fd57c1985a0 2024-04-23T18:19:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd582b9b640 1 -- 172.21.15.50:0/863558836 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd57c1993b0 con 0x7fd57c1985a0 2024-04-23T18:19:25.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.889+0000 7fd579ffb640 1 -- 172.21.15.50:0/863558836 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd574010460 con 0x7fd57c1985a0 2024-04-23T18:19:25.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.890+0000 7fd582b9b640 1 -- 172.21.15.50:0/863558836 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd57c1a55b0 con 0x7fd57c1985a0 2024-04-23T18:19:25.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.890+0000 7fd582b9b640 1 -- 172.21.15.50:0/863558836 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd57c0684e0 con 0x7fd57c1985a0 2024-04-23T18:19:25.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.892+0000 7fd579ffb640 1 -- 172.21.15.50:0/863558836 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fd574003970 con 0x7fd57c1985a0 2024-04-23T18:19:25.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.892+0000 7fd579ffb640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd56c0a3e40 0x7fd56c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:25.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.892+0000 7fd579ffb640 1 -- 172.21.15.50:0/863558836 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fd5740d0790 con 0x7fd57c1985a0 2024-04-23T18:19:25.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.892+0000 7fd580937640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd56c0a3e40 0x7fd56c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:25.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.893+0000 7fd580937640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd56c0a3e40 0x7fd56c0a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd570000c00 tx=0x7fd570031400 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:25.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:25.896+0000 7fd579ffb640 1 -- 172.21.15.50:0/863558836 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd57409cbf0 con 0x7fd57c1985a0 2024-04-23T18:19:25.916 INFO:tasks.workunit.client.0.smithi050.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-23T18:19:25.918 INFO:tasks.workunit.client.0.smithi050.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-23T18:19:25.918 INFO:tasks.workunit.client.0.smithi050.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-23T18:19:25.958 INFO:tasks.workunit.client.0.smithi050.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-23T18:19:25.962 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:19:25.962 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-23T18:19:26.018 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-04-23T18:19:26.020 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-04-23T18:19:26.020 DEBUG:teuthology.orchestra.run.smithi050:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=57e0caf76fdb7ab8b1358588f292d08519163844 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-04-23T18:19:26.082 INFO:tasks.workunit.client.0.smithi050.stderr:+ dbench 1 2024-04-23T18:19:26.088 INFO:tasks.workunit.client.0.smithi050.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-23T18:19:26.088 INFO:tasks.workunit.client.0.smithi050.stdout: 2024-04-23T18:19:26.088 INFO:tasks.workunit.client.0.smithi050.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-23T18:19:26.089 INFO:tasks.workunit.client.0.smithi050.stdout:0 of 1 processes prepared for launch 0 sec 2024-04-23T18:19:26.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.131+0000 7fd582b9b640 1 -- 172.21.15.50:0/863558836 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd57c199ad0 con 0x7fd57c1985a0 2024-04-23T18:19:26.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.132+0000 7fd579ffb640 1 -- 172.21.15.50:0/863558836 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11875 (secure 0 0 0) 0x7fd57c199ad0 con 0x7fd57c1985a0 2024-04-23T18:19:26.126 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:19:26.126 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:18:28.904281+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:19:26.126 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 21 2024-04-23T18:19:26.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.134+0000 7fd582b9b640 1 -- 172.21.15.50:0/863558836 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd56c0a3e40 msgr2=0x7fd56c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:26.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.134+0000 7fd582b9b640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd56c0a3e40 0x7fd56c0a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd570000c00 tx=0x7fd570031400 comp rx=0 tx=0).stop 2024-04-23T18:19:26.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.134+0000 7fd582b9b640 1 -- 172.21.15.50:0/863558836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd57c1985a0 msgr2=0x7fd57c198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.134+0000 7fd582b9b640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd57c1985a0 0x7fd57c198a50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd574009fd0 tx=0x7fd57400b260 comp rx=0 tx=0).stop 2024-04-23T18:19:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.134+0000 7fd582b9b640 1 -- 172.21.15.50:0/863558836 shutdown_connections 2024-04-23T18:19:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.134+0000 7fd582b9b640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd56c0a3e40 0x7fd56c0a6300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.134+0000 7fd582b9b640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd57c1985a0 0x7fd57c198a50 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.134+0000 7fd582b9b640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd57c107270 0x7fd57c19e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.134+0000 7fd582b9b640 1 --2- 172.21.15.50:0/863558836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd57c1021a0 0x7fd57c19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.134+0000 7fd582b9b640 1 -- 172.21.15.50:0/863558836 >> 172.21.15.50:0/863558836 conn(0x7fd57c079350 msgr2=0x7fd57c1002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.134+0000 7fd582b9b640 1 -- 172.21.15.50:0/863558836 shutdown_connections 2024-04-23T18:19:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.134+0000 7fd582b9b640 1 -- 172.21.15.50:0/863558836 wait complete. 2024-04-23T18:19:26.139 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-23T18:19:26.139 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-23T18:19:26.189 INFO:tasks.workunit.client.0.smithi050.stdout:1 of 1 processes prepared for launch 0 sec 2024-04-23T18:19:26.189 INFO:tasks.workunit.client.0.smithi050.stdout:releasing clients 2024-04-23T18:19:26.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.302+0000 7f38e6522640 1 -- 172.21.15.50:0/1410004416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38e0106170 msgr2=0x7f38e010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:26.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.302+0000 7f38e6522640 1 --2- 172.21.15.50:0/1410004416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38e0106170 0x7f38e010dd40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009ca0 tx=0x7f38d802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.302+0000 7f38e6522640 1 -- 172.21.15.50:0/1410004416 shutdown_connections 2024-04-23T18:19:26.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.302+0000 7f38e6522640 1 --2- 172.21.15.50:0/1410004416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38e0106170 0x7f38e010dd40 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.302+0000 7f38e6522640 1 --2- 172.21.15.50:0/1410004416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38e0101a50 0x7f38e0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.302+0000 7f38e6522640 1 --2- 172.21.15.50:0/1410004416 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e01010a0 0x7f38e0101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.302+0000 7f38e6522640 1 -- 172.21.15.50:0/1410004416 >> 172.21.15.50:0/1410004416 conn(0x7f38e0077f50 msgr2=0x7f38e0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.302+0000 7f38e6522640 1 -- 172.21.15.50:0/1410004416 shutdown_connections 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.302+0000 7f38e6522640 1 -- 172.21.15.50:0/1410004416 wait complete. 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.302+0000 7f38e6522640 1 Processor -- start 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.302+0000 7f38e6522640 1 -- start start 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38e6522640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e01010a0 0x7f38e010f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38e6522640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38e0106170 0x7f38e010fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38e6522640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38e0110420 0x7f38e01a6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38e6522640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f38e01121d0 con 0x7f38e0106170 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38e6522640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f38e0112050 con 0x7f38e0110420 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38e6522640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f38e0112350 con 0x7f38e01010a0 2024-04-23T18:19:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38df7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38e0106170 0x7f38e010fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38df7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38e0106170 0x7f38e010fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36674/0 (socket says 172.21.15.50:36674) 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38df7fe640 1 -- 172.21.15.50:0/1104811045 learned_addr learned my addr 172.21.15.50:0/1104811045 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38e4abf640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38e0110420 0x7f38e01a6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38dffff640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e01010a0 0x7f38e010f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38df7fe640 1 -- 172.21.15.50:0/1104811045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e01010a0 msgr2=0x7f38e010f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38df7fe640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e01010a0 0x7f38e010f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38df7fe640 1 -- 172.21.15.50:0/1104811045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38e0110420 msgr2=0x7f38e01a6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38df7fe640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38e0110420 0x7f38e01a6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38df7fe640 1 -- 172.21.15.50:0/1104811045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e01a6fc0 con 0x7f38e0106170 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38dffff640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e01010a0 0x7f38e010f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38df7fe640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38e0106170 0x7f38e010fee0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f38d400e9c0 tx=0x7f38d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.303+0000 7f38dd7fa640 1 -- 172.21.15.50:0/1104811045 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d400cd80 con 0x7f38e0106170 2024-04-23T18:19:26.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.304+0000 7f38dd7fa640 1 -- 172.21.15.50:0/1104811045 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38d40054d0 con 0x7f38e0106170 2024-04-23T18:19:26.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.304+0000 7f38dd7fa640 1 -- 172.21.15.50:0/1104811045 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4010430 con 0x7f38e0106170 2024-04-23T18:19:26.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.304+0000 7f38e6522640 1 -- 172.21.15.50:0/1104811045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e01a72b0 con 0x7f38e0106170 2024-04-23T18:19:26.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.304+0000 7f38e6522640 1 -- 172.21.15.50:0/1104811045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e01a77b0 con 0x7f38e0106170 2024-04-23T18:19:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.306+0000 7f38dd7fa640 1 -- 172.21.15.50:0/1104811045 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f38d4004fd0 con 0x7f38e0106170 2024-04-23T18:19:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.306+0000 7f38e9230740 1 Processor -- start 2024-04-23T18:19:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.306+0000 7f38e9230740 1 -- start start 2024-04-23T18:19:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.306+0000 7f38e9230740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3e4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.306+0000 7f38e9230740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 0x556aad3eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.306+0000 7f38e9230740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3ef090 0x556aad3f1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.306+0000 7f38e9230740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556aad349370 con 0x556aad3e49c0 2024-04-23T18:19:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.306+0000 7f38e9230740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556aad2e62e0 con 0x556aad3e40a0 2024-04-23T18:19:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.306+0000 7f38e9230740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556aad345330 con 0x556aad3ef090 2024-04-23T18:19:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38dd7fa640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f38c80a3e40 0x7f38c80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38e4abf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38dffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 0x556aad3eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38e4abf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3e4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:59440/0 (socket says 172.21.15.50:59440) 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38e4abf640 1 -- 172.21.15.50:0/3242539431 learned_addr learned my addr 172.21.15.50:0/3242539431 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38e4abf640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f38c80a3e40 0x7f38c80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38df7fe640 1 --2- 172.21.15.50:0/3242539431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3ef090 0x556aad3f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38dd7fa640 1 -- 172.21.15.50:0/1104811045 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f38d40d0ec0 con 0x7f38e0106170 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38dffff640 1 -- 172.21.15.50:0/3242539431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3ef090 msgr2=0x556aad3f1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38dffff640 1 --2- 172.21.15.50:0/3242539431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3ef090 0x556aad3f1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38dffff640 1 -- 172.21.15.50:0/3242539431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 msgr2=0x556aad3e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38dffff640 1 --2- 172.21.15.50:0/3242539431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3e4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38dffff640 1 -- 172.21.15.50:0/3242539431 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556aad127d80 con 0x556aad3e49c0 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38e4abf640 1 --2- 172.21.15.50:0/3242539431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38dffff640 1 --2- 172.21.15.50:0/3242539431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 0x556aad3eeb50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f38d000bc80 tx=0x7f38d000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee518c6b1e711e85 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38e4abf640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f38c80a3e40 0x7f38c80a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009a60 tx=0x7f38d80095c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38c67fc640 1 -- 172.21.15.50:0/3242539431 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d0015070 con 0x556aad3e49c0 2024-04-23T18:19:26.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38c67fc640 1 -- 172.21.15.50:0/3242539431 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38d001a910 con 0x556aad3e49c0 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.307+0000 7f38c67fc640 1 -- 172.21.15.50:0/3242539431 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d00118e0 con 0x556aad3e49c0 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.308+0000 7f38e9230740 1 -- 172.21.15.50:0/3242539431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 msgr2=0x556aad3eeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.308+0000 7f38e9230740 1 --2- 172.21.15.50:0/3242539431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 0x556aad3eeb50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f38d000bc80 tx=0x7f38d000f730 comp rx=0 tx=0).stop 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.308+0000 7f38e9230740 1 -- 172.21.15.50:0/3242539431 shutdown_connections 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.308+0000 7f38e9230740 1 --2- 172.21.15.50:0/3242539431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3ef090 0x556aad3f1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.308+0000 7f38e9230740 1 --2- 172.21.15.50:0/3242539431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 0x556aad3eeb50 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.308+0000 7f38e9230740 1 --2- 172.21.15.50:0/3242539431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.308+0000 7f38e9230740 1 -- 172.21.15.50:0/3242539431 >> 172.21.15.50:0/3242539431 conn(0x556aad3e03c0 msgr2=0x556aad3e07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.308+0000 7f38e9230740 1 -- 172.21.15.50:0/3242539431 shutdown_connections 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.308+0000 7f38e9230740 1 -- 172.21.15.50:0/3242539431 wait complete. 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38e9230740 1 -- start start 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38e9230740 1 Processor -- start 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38e9230740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3ec030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38e9230740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e49c0 0x556aad3ec570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38e9230740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 0x556aad3ecab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38e9230740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556aad3f1e80 con 0x556aad3ef090 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38e9230740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556aad2e3fb0 con 0x556aad3e40a0 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38e9230740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556aad3f2000 con 0x556aad3e49c0 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38e9230740 10 client.0 _ll_register_callbacks cb 0x556aad2e39a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38dffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3ec030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38e4abf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 0x556aad3ecab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38dffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3ec030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:59442/0 (socket says 172.21.15.50:59442) 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38dffff640 1 -- 172.21.15.50:0/432970203 learned_addr learned my addr 172.21.15.50:0/432970203 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.309+0000 7f38df7fe640 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e49c0 0x556aad3ec570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38dffff640 1 -- 172.21.15.50:0/432970203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e49c0 msgr2=0x556aad3ec570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38dffff640 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e49c0 0x556aad3ec570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38dffff640 1 -- 172.21.15.50:0/432970203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 msgr2=0x556aad3ecab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:26.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38dffff640 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 0x556aad3ecab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38dffff640 1 -- 172.21.15.50:0/432970203 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556aad188510 con 0x556aad3e40a0 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38e4abf640 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 0x556aad3ecab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38dffff640 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3ec030 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f38d0006010 tx=0x7f38d001a460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e1f1ec3c1bea836 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38bf7fe640 1 -- 172.21.15.50:0/432970203 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d0015070 con 0x556aad3e40a0 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38bf7fe640 1 -- 172.21.15.50:0/432970203 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f38d001aac0 con 0x556aad3e40a0 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38e9230740 1 -- 172.21.15.50:0/432970203 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38d001a910 con 0x556aad3e40a0 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38bf7fe640 1 -- 172.21.15.50:0/432970203 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f38d001acc0 con 0x556aad3e40a0 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38e9230740 10 client.25289 fetch_fsmap learned FSMap version 21 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38e9230740 1 -- 172.21.15.50:0/432970203 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556aad01b670 con 0x556aad3e40a0 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.310+0000 7f38bf7fe640 1 -- 172.21.15.50:0/432970203 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38d001a910 con 0x556aad3e40a0 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.311+0000 7f38bf7fe640 1 -- 172.21.15.50:0/432970203 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f38d0004aa0 con 0x556aad3e40a0 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.311+0000 7f38e9230740 10 client.25289 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:19:26.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.311+0000 7f38e9230740 10 client.25289 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:19:26.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.311+0000 7f38e9230740 20 client.25289 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:26.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.311+0000 7f38e9230740 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556aad3ff3d0 0x556aad4017c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.311+0000 7f38e9230740 4 client.25289 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:26.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.311+0000 7f38e9230740 1 -- 172.21.15.50:0/432970203 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556aad356d90 con 0x556aad3ff3d0 2024-04-23T18:19:26.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.311+0000 7f38df7fe640 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556aad3ff3d0 0x556aad4017c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.311+0000 7f38df7fe640 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556aad3ff3d0 0x556aad4017c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fbf58aaf7b1816c server_cookie=54b63ef93ee93e1d in_seq=0 out_seq=0 2024-04-23T18:19:26.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.311+0000 7f38bf7fe640 10 client.25289 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:26.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38bf7fe640 1 -- 172.21.15.50:0/432970203 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556aad356d90 con 0x556aad3ff3d0 2024-04-23T18:19:26.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38bf7fe640 10 client.25289 handle_command_reply: tid=0 2024-04-23T18:19:26.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38e9230740 1 client.25289 shutdown 2024-04-23T18:19:26.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38e9230740 1 -- 172.21.15.50:0/432970203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 msgr2=0x556aad3ec030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:26.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38e9230740 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3ec030 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f38d0006010 tx=0x7f38d001a460 comp rx=0 tx=0).stop 2024-04-23T18:19:26.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38e9230740 1 -- 172.21.15.50:0/432970203 shutdown_connections 2024-04-23T18:19:26.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38e9230740 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556aad3ff3d0 0x556aad4017c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38e9230740 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 0x556aad3ecab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38e9230740 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e49c0 0x556aad3ec570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38bf7fe640 0 client.25289 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:26.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38e9230740 1 --2- 172.21.15.50:0/432970203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3ec030 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38e9230740 1 -- 172.21.15.50:0/432970203 >> 172.21.15.50:0/432970203 conn(0x556aad3e03c0 msgr2=0x556aad3e61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:26.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38e9230740 1 -- 172.21.15.50:0/432970203 shutdown_connections 2024-04-23T18:19:26.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38e9230740 1 -- 172.21.15.50:0/432970203 wait complete. 2024-04-23T18:19:26.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.313+0000 7f38e9230740 20 client.25289 trim_cache size 0 max 16384 2024-04-23T18:19:26.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.331+0000 7f38e9230740 1 Processor -- start 2024-04-23T18:19:26.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.331+0000 7f38e9230740 1 -- start start 2024-04-23T18:19:26.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.331+0000 7f38e9230740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3fad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.331+0000 7f38e9230740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e49c0 0x556aad3edf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.331+0000 7f38e9230740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 0x556aad406df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.331+0000 7f38e9230740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f38e01b26c0 con 0x556aad3ef090 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.331+0000 7f38e9230740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f38e01aff90 con 0x556aad3e40a0 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.331+0000 7f38e9230740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f38e01adde0 con 0x556aad3e49c0 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.331+0000 7f38e4abf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 0x556aad406df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.331+0000 7f38e4abf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 0x556aad406df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36696/0 (socket says 172.21.15.50:36696) 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.331+0000 7f38e4abf640 1 -- 172.21.15.50:0/1966654267 learned_addr learned my addr 172.21.15.50:0/1966654267 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.332+0000 7f38dffff640 1 --2- 172.21.15.50:0/1966654267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3fad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.332+0000 7f38df7fe640 1 --2- 172.21.15.50:0/1966654267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e49c0 0x556aad3edf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.332+0000 7f38e4abf640 1 -- 172.21.15.50:0/1966654267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e49c0 msgr2=0x556aad3edf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.332+0000 7f38e4abf640 1 --2- 172.21.15.50:0/1966654267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e49c0 0x556aad3edf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.332+0000 7f38e4abf640 1 -- 172.21.15.50:0/1966654267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 msgr2=0x556aad3fad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.332+0000 7f38e4abf640 1 --2- 172.21.15.50:0/1966654267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3fad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.332+0000 7f38e4abf640 1 -- 172.21.15.50:0/1966654267 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556aad3e5330 con 0x556aad3ef090 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.332+0000 7f38e4abf640 1 --2- 172.21.15.50:0/1966654267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 0x556aad406df0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f38d8005fe0 tx=0x7f38d8036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a825df5b7c411c9e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.332+0000 7f38c5ffb640 1 -- 172.21.15.50:0/1966654267 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d8056070 con 0x556aad3ef090 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.332+0000 7f38c5ffb640 1 -- 172.21.15.50:0/1966654267 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38d8036070 con 0x556aad3ef090 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.332+0000 7f38c5ffb640 1 -- 172.21.15.50:0/1966654267 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d802f950 con 0x556aad3ef090 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.333+0000 7f38e9230740 1 -- 172.21.15.50:0/1966654267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 msgr2=0x556aad406df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.333+0000 7f38e9230740 1 --2- 172.21.15.50:0/1966654267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 0x556aad406df0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f38d8005fe0 tx=0x7f38d8036000 comp rx=0 tx=0).stop 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.333+0000 7f38e9230740 1 -- 172.21.15.50:0/1966654267 shutdown_connections 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.333+0000 7f38e9230740 1 --2- 172.21.15.50:0/1966654267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3ef090 0x556aad406df0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.333+0000 7f38e9230740 1 --2- 172.21.15.50:0/1966654267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e49c0 0x556aad3edf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.333+0000 7f38e9230740 1 --2- 172.21.15.50:0/1966654267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3e40a0 0x556aad3fad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.333+0000 7f38e9230740 1 -- 172.21.15.50:0/1966654267 >> 172.21.15.50:0/1966654267 conn(0x556aad3e03c0 msgr2=0x556aad3ed860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.333+0000 7f38e9230740 1 -- 172.21.15.50:0/1966654267 shutdown_connections 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.333+0000 7f38e9230740 1 -- 172.21.15.50:0/1966654267 wait complete. 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.333+0000 7f38e9230740 1 -- start start 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.333+0000 7f38e9230740 1 Processor -- start 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38e9230740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e40a0 0x556aad3e8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38e9230740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 0x556aad3e93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38e9230740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3ef090 0x556aad3e98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38e9230740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f38e01ae450 con 0x556aad3e49c0 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38e9230740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f38e01af8c0 con 0x556aad3ef090 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38e9230740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f38e01aeb20 con 0x556aad3e40a0 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38e9230740 10 client.0 _ll_register_callbacks cb 0x7f38d000be90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38df7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 0x556aad3e93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38df7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 0x556aad3e93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36704/0 (socket says 172.21.15.50:36704) 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38df7fe640 1 -- 172.21.15.50:0/3235653742 learned_addr learned my addr 172.21.15.50:0/3235653742 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38e4abf640 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3ef090 0x556aad3e98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38dffff640 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e40a0 0x556aad3e8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38df7fe640 1 -- 172.21.15.50:0/3235653742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e40a0 msgr2=0x556aad3e8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38df7fe640 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e40a0 0x556aad3e8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38df7fe640 1 -- 172.21.15.50:0/3235653742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3ef090 msgr2=0x556aad3e98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38df7fe640 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3ef090 0x556aad3e98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38df7fe640 1 -- 172.21.15.50:0/3235653742 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556aad464760 con 0x556aad3e49c0 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38df7fe640 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 0x556aad3e93a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f38d40ce500 tx=0x7f38d401c590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea569102097558d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38bf7fe640 1 -- 172.21.15.50:0/3235653742 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d409cc20 con 0x556aad3e49c0 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38bf7fe640 1 -- 172.21.15.50:0/3235653742 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f38d409af20 con 0x556aad3e49c0 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.334+0000 7f38e9230740 1 -- 172.21.15.50:0/3235653742 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38d8036070 con 0x556aad3e49c0 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38bf7fe640 1 -- 172.21.15.50:0/3235653742 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f38d40a0fd0 con 0x556aad3e49c0 2024-04-23T18:19:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38e9230740 10 client.15669 fetch_fsmap learned FSMap version 21 2024-04-23T18:19:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38e9230740 1 -- 172.21.15.50:0/3235653742 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556aad465a40 con 0x556aad3e49c0 2024-04-23T18:19:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38bf7fe640 1 -- 172.21.15.50:0/3235653742 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38d8036070 con 0x556aad3e49c0 2024-04-23T18:19:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38bf7fe640 1 -- 172.21.15.50:0/3235653742 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f38d409ce80 con 0x556aad3e49c0 2024-04-23T18:19:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38e9230740 10 client.15669 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:19:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38e9230740 10 client.15669 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:19:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38e9230740 20 client.15669 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38e9230740 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556aad466190 0x556aad4685b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38e9230740 4 client.15669 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-23T18:19:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38e9230740 1 -- 172.21.15.50:0/3235653742 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x556aad350c00 con 0x556aad466190 2024-04-23T18:19:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38dffff640 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556aad466190 0x556aad4685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:26.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38dffff640 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556aad466190 0x556aad4685b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81455b15dbaaa4db server_cookie=94d6fcdf098d99ad in_seq=0 out_seq=0 2024-04-23T18:19:26.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.335+0000 7f38bf7fe640 10 client.15669 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:26.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.398+0000 7f38bf7fe640 1 -- 172.21.15.50:0/3235653742 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x556aad350c00 con 0x556aad466190 2024-04-23T18:19:26.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.398+0000 7f38bf7fe640 10 client.15669 handle_command_reply: tid=0 2024-04-23T18:19:26.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.398+0000 7f38e9230740 1 client.15669 shutdown 2024-04-23T18:19:26.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.398+0000 7f38e9230740 1 -- 172.21.15.50:0/3235653742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 msgr2=0x556aad3e93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:26.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.398+0000 7f38e9230740 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 0x556aad3e93a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f38d40ce500 tx=0x7f38d401c590 comp rx=0 tx=0).stop 2024-04-23T18:19:26.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.398+0000 7f38e9230740 1 -- 172.21.15.50:0/3235653742 shutdown_connections 2024-04-23T18:19:26.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.398+0000 7f38e9230740 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556aad466190 0x556aad4685b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.398+0000 7f38e9230740 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556aad3ef090 0x556aad3e98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.398+0000 7f38e9230740 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556aad3e49c0 0x556aad3e93a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.398+0000 7f38bf7fe640 0 client.15669 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.398+0000 7f38e9230740 1 --2- 172.21.15.50:0/3235653742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556aad3e40a0 0x556aad3e8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.398+0000 7f38e9230740 1 -- 172.21.15.50:0/3235653742 >> 172.21.15.50:0/3235653742 conn(0x556aad3e03c0 msgr2=0x556aad3ee0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e9230740 1 -- 172.21.15.50:0/3235653742 shutdown_connections 2024-04-23T18:19:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e9230740 1 -- 172.21.15.50:0/3235653742 wait complete. 2024-04-23T18:19:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e9230740 20 client.15669 trim_cache size 0 max 16384 2024-04-23T18:19:26.394 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:19:26.394 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 21, 2024-04-23T18:19:26.394 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 6, 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "bff1ca7e": { 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 6, 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 90.0, 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:19:26.395 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:19:26.396 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:19:26.396 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:19:26.396 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:19:26.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:26.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:26.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:26.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:26.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:26.396 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:19:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e6522640 1 -- 172.21.15.50:0/1104811045 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f38c80a3e40 msgr2=0x7f38c80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e6522640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f38c80a3e40 0x7f38c80a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009a60 tx=0x7f38d80095c0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e6522640 1 -- 172.21.15.50:0/1104811045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38e0106170 msgr2=0x7f38e010fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e6522640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38e0106170 0x7f38e010fee0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f38d400e9c0 tx=0x7f38d400ee80 comp rx=0 tx=0).stop 2024-04-23T18:19:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e6522640 1 -- 172.21.15.50:0/1104811045 shutdown_connections 2024-04-23T18:19:26.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e6522640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f38c80a3e40 0x7f38c80a6300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e6522640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38e0110420 0x7f38e01a6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e6522640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38e0106170 0x7f38e010fee0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e6522640 1 --2- 172.21.15.50:0/1104811045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38e01010a0 0x7f38e010f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:26.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e6522640 1 -- 172.21.15.50:0/1104811045 >> 172.21.15.50:0/1104811045 conn(0x7f38e0077f50 msgr2=0x7f38e00fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:26.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e6522640 1 -- 172.21.15.50:0/1104811045 shutdown_connections 2024-04-23T18:19:26.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:26.399+0000 7f38e6522640 1 -- 172.21.15.50:0/1104811045 wait complete. 2024-04-23T18:19:26.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:26 smithi050 ceph-mon[29187]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-23T18:19:26.403 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:26 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1996070678' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:26.404 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'bff1ca7e', quiesce took 0.6 seconds. Will release after: 30.0 2024-04-23T18:19:26.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:26 smithi177 ceph-mon[32308]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-23T18:19:26.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:26 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1996070678' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:26.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:26 smithi188 ceph-mon[32694]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-23T18:19:26.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:26 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1996070678' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:27.189 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 10.44 MB/sec warmup 1 sec latency 849.783 ms 2024-04-23T18:19:27.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:27 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/863558836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:27.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:27 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/863558836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:27.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:27 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/863558836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:28.189 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 5.22 MB/sec warmup 2 sec latency 1849.883 ms 2024-04-23T18:19:28.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:28 smithi050 ceph-mon[29187]: pgmap v31: 97 pgs: 97 active+clean; 1.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-23T18:19:28.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:28 smithi177 ceph-mon[32308]: pgmap v31: 97 pgs: 97 active+clean; 1.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-23T18:19:28.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:28 smithi188 ceph-mon[32694]: pgmap v31: 97 pgs: 97 active+clean; 1.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-23T18:19:29.189 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 3.48 MB/sec warmup 3 sec latency 2850.006 ms 2024-04-23T18:19:30.189 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 2.61 MB/sec warmup 4 sec latency 3850.162 ms 2024-04-23T18:19:30.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:30 smithi050 ceph-mon[29187]: pgmap v32: 97 pgs: 97 active+clean; 1.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-04-23T18:19:30.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:30 smithi177 ceph-mon[32308]: pgmap v32: 97 pgs: 97 active+clean; 1.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-04-23T18:19:30.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:30 smithi188 ceph-mon[32694]: pgmap v32: 97 pgs: 97 active+clean; 1.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-04-23T18:19:31.189 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 2.09 MB/sec warmup 5 sec latency 4850.271 ms 2024-04-23T18:19:32.190 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 1.74 MB/sec warmup 6 sec latency 5850.365 ms 2024-04-23T18:19:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:32 smithi050 ceph-mon[29187]: pgmap v33: 97 pgs: 97 active+clean; 1.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-04-23T18:19:32.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:32 smithi177 ceph-mon[32308]: pgmap v33: 97 pgs: 97 active+clean; 1.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-04-23T18:19:32.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:32 smithi188 ceph-mon[32694]: pgmap v33: 97 pgs: 97 active+clean; 1.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-04-23T18:19:33.190 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 1.49 MB/sec warmup 7 sec latency 6850.467 ms 2024-04-23T18:19:34.190 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 1.31 MB/sec warmup 8 sec latency 7850.582 ms 2024-04-23T18:19:34.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:34 smithi050 ceph-mon[29187]: pgmap v34: 97 pgs: 97 active+clean; 11 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 865 KiB/s wr, 14 op/s 2024-04-23T18:19:34.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:34 smithi177 ceph-mon[32308]: pgmap v34: 97 pgs: 97 active+clean; 11 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 865 KiB/s wr, 14 op/s 2024-04-23T18:19:34.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:34 smithi188 ceph-mon[32694]: pgmap v34: 97 pgs: 97 active+clean; 11 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 865 KiB/s wr, 14 op/s 2024-04-23T18:19:35.190 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 1.16 MB/sec warmup 9 sec latency 8850.697 ms 2024-04-23T18:19:36.190 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 1.04 MB/sec warmup 10 sec latency 9850.752 ms 2024-04-23T18:19:36.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:36 smithi050 ceph-mon[29187]: pgmap v35: 97 pgs: 97 active+clean; 11 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 865 KiB/s wr, 11 op/s 2024-04-23T18:19:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:19:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:19:36.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:36 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:19:36.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:36 smithi177 ceph-mon[32308]: pgmap v35: 97 pgs: 97 active+clean; 11 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 865 KiB/s wr, 11 op/s 2024-04-23T18:19:36.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:19:36.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:19:36.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:36 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:19:36.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:36 smithi188 ceph-mon[32694]: pgmap v35: 97 pgs: 97 active+clean; 11 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 865 KiB/s wr, 11 op/s 2024-04-23T18:19:36.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:19:36.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:19:36.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:36 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:19:37.190 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.95 MB/sec warmup 11 sec latency 10850.790 ms 2024-04-23T18:19:37.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:37 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:19:37.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:37 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:19:37.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:37 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:19:37.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:37 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:19:37.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:37 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:19:37.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:37 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:19:37.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:37 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:19:37.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:37 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:19:37.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:37 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:19:38.190 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.87 MB/sec warmup 12 sec latency 11850.933 ms 2024-04-23T18:19:38.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:38 smithi050 ceph-mon[29187]: pgmap v36: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 872 KiB/s wr, 15 op/s 2024-04-23T18:19:38.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:38 smithi177 ceph-mon[32308]: pgmap v36: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 872 KiB/s wr, 15 op/s 2024-04-23T18:19:38.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:38 smithi188 ceph-mon[32694]: pgmap v36: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 872 KiB/s wr, 15 op/s 2024-04-23T18:19:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.614+0000 7fa367abb640 1 -- 172.21.15.50:0/405132459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa360102b70 msgr2=0x7fa360106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.614+0000 7fa367abb640 1 --2- 172.21.15.50:0/405132459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa360102b70 0x7fa360106bb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa350009900 tx=0x7fa35002a410 comp rx=0 tx=0).stop 2024-04-23T18:19:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.614+0000 7fa367abb640 1 -- 172.21.15.50:0/405132459 shutdown_connections 2024-04-23T18:19:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.614+0000 7fa367abb640 1 --2- 172.21.15.50:0/405132459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa360107290 0x7fa36010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.614+0000 7fa367abb640 1 --2- 172.21.15.50:0/405132459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa360102b70 0x7fa360106bb0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.614+0000 7fa367abb640 1 --2- 172.21.15.50:0/405132459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3601021c0 0x7fa3601025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.614+0000 7fa367abb640 1 -- 172.21.15.50:0/405132459 >> 172.21.15.50:0/405132459 conn(0x7fa3600fda20 msgr2=0x7fa3600ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.615+0000 7fa367abb640 1 -- 172.21.15.50:0/405132459 shutdown_connections 2024-04-23T18:19:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.615+0000 7fa367abb640 1 -- 172.21.15.50:0/405132459 wait complete. 2024-04-23T18:19:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.615+0000 7fa367abb640 1 Processor -- start 2024-04-23T18:19:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.615+0000 7fa367abb640 1 -- start start 2024-04-23T18:19:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.615+0000 7fa367abb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3601021c0 0x7fa3601986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.615+0000 7fa367abb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa360107290 0x7fa360198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.615+0000 7fa367abb640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa36019cf60 0x7fa36019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.615+0000 7fa367abb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa360112200 con 0x7fa3601021c0 2024-04-23T18:19:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.615+0000 7fa367abb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa360112080 con 0x7fa360107290 2024-04-23T18:19:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.615+0000 7fa367abb640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa360112380 con 0x7fa36019cf60 2024-04-23T18:19:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa365857640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3601021c0 0x7fa3601986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa365857640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3601021c0 0x7fa3601986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56956/0 (socket says 172.21.15.50:56956) 2024-04-23T18:19:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa365857640 1 -- 172.21.15.50:0/1413705393 learned_addr learned my addr 172.21.15.50:0/1413705393 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa365056640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa360107290 0x7fa360198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa366058640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa36019cf60 0x7fa36019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa365857640 1 -- 172.21.15.50:0/1413705393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa36019cf60 msgr2=0x7fa36019d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa365857640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa36019cf60 0x7fa36019d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa365857640 1 -- 172.21.15.50:0/1413705393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa360107290 msgr2=0x7fa360198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa365857640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa360107290 0x7fa360198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa365857640 1 -- 172.21.15.50:0/1413705393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36019da50 con 0x7fa3601021c0 2024-04-23T18:19:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa365857640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3601021c0 0x7fa3601986a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa35c00d950 tx=0x7fa35c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa356ffd640 1 -- 172.21.15.50:0/1413705393 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c017030 con 0x7fa3601021c0 2024-04-23T18:19:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa356ffd640 1 -- 172.21.15.50:0/1413705393 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa35c004020 con 0x7fa3601021c0 2024-04-23T18:19:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa367abb640 1 -- 172.21.15.50:0/1413705393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa36019dd40 con 0x7fa3601021c0 2024-04-23T18:19:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa356ffd640 1 -- 172.21.15.50:0/1413705393 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c0047a0 con 0x7fa3601021c0 2024-04-23T18:19:38.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.616+0000 7fa367abb640 1 -- 172.21.15.50:0/1413705393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3601a5550 con 0x7fa3601021c0 2024-04-23T18:19:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.619+0000 7fa356ffd640 1 -- 172.21.15.50:0/1413705393 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fa35c009a30 con 0x7fa3601021c0 2024-04-23T18:19:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.619+0000 7fa36a7cc740 1 Processor -- start 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa36a7cc740 1 -- start start 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa36a7cc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 0x55f98cf5aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa36a7cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf652c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa36a7cc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf65800 0x55f98cf67bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa36a7cc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f98cebf970 con 0x55f98cf5a620 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa36a7cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f98ce5c4a0 con 0x55f98cf5af40 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa36a7cc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f98cebb930 con 0x55f98cf65800 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa356ffd640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa34c0a3f60 0x7fa34c0a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa366058640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 0x55f98cf5aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa366058640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 0x55f98cf5aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56972/0 (socket says 172.21.15.50:56972) 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa366058640 1 -- 172.21.15.50:0/1363073850 learned_addr learned my addr 172.21.15.50:0/1363073850 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa365056640 1 --2- 172.21.15.50:0/1363073850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf65800 0x55f98cf67bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa365857640 1 --2- 172.21.15.50:0/1363073850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa366058640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa34c0a3f60 0x7fa34c0a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa356ffd640 1 -- 172.21.15.50:0/1413705393 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fa35c0d0580 con 0x7fa3601021c0 2024-04-23T18:19:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa366058640 1 -- 172.21.15.50:0/1363073850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf65800 msgr2=0x55f98cf67bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa366058640 1 --2- 172.21.15.50:0/1363073850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf65800 0x55f98cf67bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa366058640 1 -- 172.21.15.50:0/1363073850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 msgr2=0x55f98cf652c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa366058640 1 --2- 172.21.15.50:0/1363073850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf652c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.620+0000 7fa366058640 1 -- 172.21.15.50:0/1363073850 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f98cbd2210 con 0x55f98cf5a620 2024-04-23T18:19:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.621+0000 7fa366058640 1 --2- 172.21.15.50:0/1363073850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 0x55f98cf5aa00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa358014940 tx=0x7fa358014e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a39035561316c6b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.621+0000 7fa34bfff640 1 -- 172.21.15.50:0/1363073850 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa358014050 con 0x55f98cf5a620 2024-04-23T18:19:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.621+0000 7fa34bfff640 1 -- 172.21.15.50:0/1363073850 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3580141f0 con 0x55f98cf5a620 2024-04-23T18:19:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.621+0000 7fa34bfff640 1 -- 172.21.15.50:0/1363073850 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3580059d0 con 0x55f98cf5a620 2024-04-23T18:19:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.622+0000 7fa36a7cc740 1 -- 172.21.15.50:0/1363073850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 msgr2=0x55f98cf5aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.622+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/1363073850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 0x55f98cf5aa00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa358014940 tx=0x7fa358014e00 comp rx=0 tx=0).stop 2024-04-23T18:19:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.622+0000 7fa366058640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa34c0a3f60 0x7fa34c0a6420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa358021000 tx=0x7fa358020000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.622+0000 7fa36a7cc740 1 -- 172.21.15.50:0/1363073850 shutdown_connections 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.622+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/1363073850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf65800 0x55f98cf67bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.622+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/1363073850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf652c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.622+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/1363073850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 0x55f98cf5aa00 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.622+0000 7fa36a7cc740 1 -- 172.21.15.50:0/1363073850 >> 172.21.15.50:0/1363073850 conn(0x55f98cf56b30 msgr2=0x55f98cf56f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.622+0000 7fa36a7cc740 1 -- 172.21.15.50:0/1363073850 shutdown_connections 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.622+0000 7fa36a7cc740 1 -- 172.21.15.50:0/1363073850 wait complete. 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.622+0000 7fa36a7cc740 1 -- start start 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.622+0000 7fa36a7cc740 1 Processor -- start 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa36a7cc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 0x55f98cf65130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa36a7cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf62420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa36a7cc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf65800 0x55f98cf62960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa36a7cc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f98cf68430 con 0x55f98cf5a620 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa36a7cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f98ce5a0b0 con 0x55f98cf5af40 2024-04-23T18:19:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa36a7cc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f98cf685b0 con 0x55f98cf65800 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa36a7cc740 10 client.0 _ll_register_callbacks cb 0x55f98ce47d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa365857640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 0x55f98cf65130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa365857640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 0x55f98cf65130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56974/0 (socket says 172.21.15.50:56974) 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa365857640 1 -- 172.21.15.50:0/152992017 learned_addr learned my addr 172.21.15.50:0/152992017 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa365056640 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf62420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa366058640 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf65800 0x55f98cf62960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa365857640 1 -- 172.21.15.50:0/152992017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf65800 msgr2=0x55f98cf62960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa365857640 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf65800 0x55f98cf62960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa365857640 1 -- 172.21.15.50:0/152992017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 msgr2=0x55f98cf62420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa365857640 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf62420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa365857640 1 -- 172.21.15.50:0/152992017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f98cc9e620 con 0x55f98cf5a620 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa365857640 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 0x55f98cf65130 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa35c09bfe0 tx=0x7fa35c0207f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92a85ac28b4626e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.623+0000 7fa3497fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa3497fa640 1 -- 172.21.15.50:0/152992017 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c06a030 con 0x55f98cf5a620 2024-04-23T18:19:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa3497fa640 1 -- 172.21.15.50:0/152992017 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fa35c098950 con 0x55f98cf5a620 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa36a7cc740 1 -- 172.21.15.50:0/152992017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3580141f0 con 0x55f98cf5a620 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa3497fa640 1 -- 172.21.15.50:0/152992017 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa35c096050 con 0x55f98cf5a620 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa36a7cc740 10 client.15681 fetch_fsmap learned FSMap version 21 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa36a7cc740 1 -- 172.21.15.50:0/152992017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f98cf74340 con 0x55f98cf5a620 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa3497fa640 1 -- 172.21.15.50:0/152992017 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3580141f0 con 0x55f98cf5a620 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa3497fa640 1 -- 172.21.15.50:0/152992017 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fa35c09c540 con 0x55f98cf5a620 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa36a7cc740 10 client.15681 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa36a7cc740 10 client.15681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa36a7cc740 20 client.15681 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f98cf75800 0x55f98cf77c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa36a7cc740 4 client.15681 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.624+0000 7fa36a7cc740 1 -- 172.21.15.50:0/152992017 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f98cecd390 con 0x55f98cf75800 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.625+0000 7fa365056640 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f98cf75800 0x55f98cf77c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.625+0000 7fa365056640 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f98cf75800 0x55f98cf77c30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5f401707547c3d9 server_cookie=a498da884c057956 in_seq=0 out_seq=0 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.625+0000 7fa3497fa640 10 client.15681 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa3497fa640 1 -- 172.21.15.50:0/152992017 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f98cecd390 con 0x55f98cf75800 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa3497fa640 10 client.15681 handle_command_reply: tid=0 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa36a7cc740 1 client.15681 shutdown 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa36a7cc740 1 -- 172.21.15.50:0/152992017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 msgr2=0x55f98cf65130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 0x55f98cf65130 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa35c09bfe0 tx=0x7fa35c0207f0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa36a7cc740 1 -- 172.21.15.50:0/152992017 shutdown_connections 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f98cf75800 0x55f98cf77c30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf65800 0x55f98cf62960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf62420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa3497fa640 0 client.15681 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/152992017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf5a620 0x55f98cf65130 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa36a7cc740 1 -- 172.21.15.50:0/152992017 >> 172.21.15.50:0/152992017 conn(0x55f98cf56b30 msgr2=0x55f98cf5c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa36a7cc740 1 -- 172.21.15.50:0/152992017 shutdown_connections 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa36a7cc740 1 -- 172.21.15.50:0/152992017 wait complete. 2024-04-23T18:19:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.626+0000 7fa36a7cc740 20 client.15681 trim_cache size 0 max 16384 2024-04-23T18:19:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.641+0000 7fa36a7cc740 1 Processor -- start 2024-04-23T18:19:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa36a7cc740 1 -- start start 2024-04-23T18:19:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa36a7cc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf5a620 0x55f98cf712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa36a7cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf5dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa36a7cc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 0x55f98cf64c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa36a7cc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa3601b03a0 con 0x55f98cf65800 2024-04-23T18:19:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa36a7cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa3601adc70 con 0x55f98cf5af40 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa36a7cc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa3601abac0 con 0x55f98cf5a620 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa366058640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 0x55f98cf64c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa366058640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 0x55f98cf64c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56986/0 (socket says 172.21.15.50:56986) 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa366058640 1 -- 172.21.15.50:0/3838288493 learned_addr learned my addr 172.21.15.50:0/3838288493 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa365857640 1 --2- 172.21.15.50:0/3838288493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf5a620 0x55f98cf712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa365056640 1 --2- 172.21.15.50:0/3838288493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf5dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa366058640 1 -- 172.21.15.50:0/3838288493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf5a620 msgr2=0x55f98cf712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa366058640 1 --2- 172.21.15.50:0/3838288493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf5a620 0x55f98cf712f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa366058640 1 -- 172.21.15.50:0/3838288493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 msgr2=0x55f98cf5dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa366058640 1 --2- 172.21.15.50:0/3838288493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf5dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa366058640 1 -- 172.21.15.50:0/3838288493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f98cf5d140 con 0x55f98cf65800 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa365056640 1 --2- 172.21.15.50:0/3838288493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf5dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:19:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.642+0000 7fa366058640 1 --2- 172.21.15.50:0/3838288493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 0x55f98cf64c60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa358014940 tx=0x7fa358012fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b5b5aff882ff8d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.643+0000 7fa34b7fe640 1 -- 172.21.15.50:0/3838288493 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa358013400 con 0x55f98cf65800 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.643+0000 7fa34b7fe640 1 -- 172.21.15.50:0/3838288493 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa358005e10 con 0x55f98cf65800 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.643+0000 7fa34b7fe640 1 -- 172.21.15.50:0/3838288493 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3580121f0 con 0x55f98cf65800 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.643+0000 7fa36a7cc740 1 -- 172.21.15.50:0/3838288493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 msgr2=0x55f98cf64c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.643+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/3838288493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 0x55f98cf64c60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa358014940 tx=0x7fa358012fd0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.643+0000 7fa36a7cc740 1 -- 172.21.15.50:0/3838288493 shutdown_connections 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.643+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/3838288493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 0x55f98cf64c60 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.643+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/3838288493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5af40 0x55f98cf5dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.643+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/3838288493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf5a620 0x55f98cf712f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.643+0000 7fa36a7cc740 1 -- 172.21.15.50:0/3838288493 >> 172.21.15.50:0/3838288493 conn(0x55f98cf56b30 msgr2=0x55f98cf55cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.643+0000 7fa36a7cc740 1 -- 172.21.15.50:0/3838288493 shutdown_connections 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.643+0000 7fa36a7cc740 1 -- 172.21.15.50:0/3838288493 wait complete. 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa36a7cc740 1 -- start start 2024-04-23T18:19:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa36a7cc740 1 Processor -- start 2024-04-23T18:19:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa36a7cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5a620 0x55f98cf693e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa36a7cc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf5af40 0x55f98cf69920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa36a7cc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 0x55f98cf69e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa36a7cc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa3601ac130 con 0x55f98cf65800 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa36a7cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa3601ad5a0 con 0x55f98cf5a620 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa36a7cc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa3601ac800 con 0x55f98cf5af40 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa36a7cc740 10 client.0 _ll_register_callbacks cb 0x7fa35c021eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa366058640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 0x55f98cf69e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa366058640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 0x55f98cf69e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56994/0 (socket says 172.21.15.50:56994) 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa366058640 1 -- 172.21.15.50:0/2403238205 learned_addr learned my addr 172.21.15.50:0/2403238205 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa365056640 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf5af40 0x55f98cf69920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.644+0000 7fa365857640 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5a620 0x55f98cf693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa366058640 1 -- 172.21.15.50:0/2403238205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf5af40 msgr2=0x55f98cf69920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa366058640 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf5af40 0x55f98cf69920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa366058640 1 -- 172.21.15.50:0/2403238205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5a620 msgr2=0x55f98cf693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa366058640 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5a620 0x55f98cf693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa366058640 1 -- 172.21.15.50:0/2403238205 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f98cfdf410 con 0x55f98cf65800 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa365857640 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5a620 0x55f98cf693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa366058640 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 0x55f98cf69e60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa358014810 tx=0x7fa358009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fd5a5bde7a9479 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa3497fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa3497fa640 1 -- 172.21.15.50:0/2403238205 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa358013400 con 0x55f98cf65800 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa3497fa640 1 -- 172.21.15.50:0/2403238205 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fa358023040 con 0x55f98cf65800 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa36a7cc740 1 -- 172.21.15.50:0/2403238205 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa358005e10 con 0x55f98cf65800 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa3497fa640 1 -- 172.21.15.50:0/2403238205 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa358014050 con 0x55f98cf65800 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa36a7cc740 10 client.15687 fetch_fsmap learned FSMap version 21 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa36a7cc740 1 -- 172.21.15.50:0/2403238205 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f98cfe0650 con 0x55f98cf65800 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.645+0000 7fa3497fa640 1 -- 172.21.15.50:0/2403238205 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa358005e10 con 0x55f98cf65800 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.646+0000 7fa3497fa640 1 -- 172.21.15.50:0/2403238205 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fa35800f040 con 0x55f98cf65800 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.646+0000 7fa36a7cc740 10 client.15687 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T18:19:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.646+0000 7fa36a7cc740 10 client.15687 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.646+0000 7fa36a7cc740 20 client.15687 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.646+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f98cfe0b10 0x55f98cfe2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.646+0000 7fa36a7cc740 4 client.15687 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:19:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.646+0000 7fa36a7cc740 1 -- 172.21.15.50:0/2403238205 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f98cec7200 con 0x55f98cfe0b10 2024-04-23T18:19:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.646+0000 7fa365857640 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f98cfe0b10 0x55f98cfe2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.646+0000 7fa3497fa640 10 client.15687 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.646+0000 7fa365857640 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f98cfe0b10 0x55f98cfe2f30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=402b19b89aa02140 server_cookie=aeda255dac306612 in_seq=0 out_seq=0 2024-04-23T18:19:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa3497fa640 1 -- 172.21.15.50:0/2403238205 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f98cec7200 con 0x55f98cfe0b10 2024-04-23T18:19:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa3497fa640 10 client.15687 handle_command_reply: tid=0 2024-04-23T18:19:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa36a7cc740 1 client.15687 shutdown 2024-04-23T18:19:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa36a7cc740 1 -- 172.21.15.50:0/2403238205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 msgr2=0x55f98cf69e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 0x55f98cf69e60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa358014810 tx=0x7fa358009e20 comp rx=0 tx=0).stop 2024-04-23T18:19:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa36a7cc740 1 -- 172.21.15.50:0/2403238205 shutdown_connections 2024-04-23T18:19:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f98cfe0b10 0x55f98cfe2f30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f98cf65800 0x55f98cf69e60 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f98cf5af40 0x55f98cf69920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa36a7cc740 1 --2- 172.21.15.50:0/2403238205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f98cf5a620 0x55f98cf693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa36a7cc740 1 -- 172.21.15.50:0/2403238205 >> 172.21.15.50:0/2403238205 conn(0x55f98cf56b30 msgr2=0x55f98cf5bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa3497fa640 0 client.15687 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa36a7cc740 1 -- 172.21.15.50:0/2403238205 shutdown_connections 2024-04-23T18:19:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa36a7cc740 1 -- 172.21.15.50:0/2403238205 wait complete. 2024-04-23T18:19:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.669+0000 7fa36a7cc740 20 client.15687 trim_cache size 0 max 16384 2024-04-23T18:19:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:19:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:19:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:19:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:19:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.670+0000 7fa367abb640 1 -- 172.21.15.50:0/1413705393 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa34c0a3f60 msgr2=0x7fa34c0a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.670+0000 7fa367abb640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa34c0a3f60 0x7fa34c0a6420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa358021000 tx=0x7fa358020000 comp rx=0 tx=0).stop 2024-04-23T18:19:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.670+0000 7fa367abb640 1 -- 172.21.15.50:0/1413705393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3601021c0 msgr2=0x7fa3601986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.670+0000 7fa367abb640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3601021c0 0x7fa3601986a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa35c00d950 tx=0x7fa35c00de10 comp rx=0 tx=0).stop 2024-04-23T18:19:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.670+0000 7fa367abb640 1 -- 172.21.15.50:0/1413705393 shutdown_connections 2024-04-23T18:19:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.670+0000 7fa367abb640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa34c0a3f60 0x7fa34c0a6420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.670+0000 7fa367abb640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa36019cf60 0x7fa36019d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.670+0000 7fa367abb640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa360107290 0x7fa360198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.670+0000 7fa367abb640 1 --2- 172.21.15.50:0/1413705393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3601021c0 0x7fa3601986a0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.670+0000 7fa367abb640 1 -- 172.21.15.50:0/1413705393 >> 172.21.15.50:0/1413705393 conn(0x7fa3600fda20 msgr2=0x7fa3600ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.670+0000 7fa367abb640 1 -- 172.21.15.50:0/1413705393 shutdown_connections 2024-04-23T18:19:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:38.670+0000 7fa367abb640 1 -- 172.21.15.50:0/1413705393 wait complete. 2024-04-23T18:19:39.190 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.80 MB/sec warmup 13 sec latency 12851.019 ms 2024-04-23T18:19:40.190 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.75 MB/sec warmup 14 sec latency 13851.139 ms 2024-04-23T18:19:40.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:40 smithi050 ceph-mon[29187]: pgmap v37: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 857 KiB/s wr, 8 op/s 2024-04-23T18:19:40.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:40 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:40.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:40 smithi177 ceph-mon[32308]: pgmap v37: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 857 KiB/s wr, 8 op/s 2024-04-23T18:19:40.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:40 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:40.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:40 smithi188 ceph-mon[32694]: pgmap v37: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 857 KiB/s wr, 8 op/s 2024-04-23T18:19:40.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:40 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:19:41.190 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.70 MB/sec warmup 15 sec latency 14851.251 ms 2024-04-23T18:19:42.191 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.65 MB/sec warmup 16 sec latency 15851.361 ms 2024-04-23T18:19:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:42 smithi050 ceph-mon[29187]: pgmap v38: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 857 KiB/s wr, 8 op/s 2024-04-23T18:19:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:42 smithi177 ceph-mon[32308]: pgmap v38: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 857 KiB/s wr, 8 op/s 2024-04-23T18:19:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:42 smithi188 ceph-mon[32694]: pgmap v38: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 857 KiB/s wr, 8 op/s 2024-04-23T18:19:43.191 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.61 MB/sec warmup 17 sec latency 16851.484 ms 2024-04-23T18:19:44.191 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.58 MB/sec warmup 18 sec latency 17851.615 ms 2024-04-23T18:19:44.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:44 smithi050 ceph-mon[29187]: pgmap v39: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 862 KiB/s wr, 20 op/s 2024-04-23T18:19:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:44 smithi177 ceph-mon[32308]: pgmap v39: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 862 KiB/s wr, 20 op/s 2024-04-23T18:19:44.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:44 smithi188 ceph-mon[32694]: pgmap v39: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 862 KiB/s wr, 20 op/s 2024-04-23T18:19:45.191 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.55 MB/sec warmup 19 sec latency 18851.720 ms 2024-04-23T18:19:46.191 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.52 MB/sec warmup 20 sec latency 19851.846 ms 2024-04-23T18:19:46.382 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:46 smithi188 ceph-mon[32694]: pgmap v40: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-04-23T18:19:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:46 smithi050 ceph-mon[29187]: pgmap v40: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-04-23T18:19:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:46 smithi177 ceph-mon[32308]: pgmap v40: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-04-23T18:19:47.191 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.50 MB/sec warmup 21 sec latency 20851.960 ms 2024-04-23T18:19:48.191 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.47 MB/sec warmup 22 sec latency 21852.076 ms 2024-04-23T18:19:48.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:48 smithi050 ceph-mon[29187]: pgmap v41: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-04-23T18:19:48.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:48 smithi177 ceph-mon[32308]: pgmap v41: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-04-23T18:19:48.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:48 smithi188 ceph-mon[32694]: pgmap v41: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-04-23T18:19:49.191 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.45 MB/sec warmup 23 sec latency 22852.210 ms 2024-04-23T18:19:50.192 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.44 MB/sec warmup 24 sec latency 23852.322 ms 2024-04-23T18:19:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:50 smithi050 ceph-mon[29187]: pgmap v42: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 6.4 KiB/s wr, 14 op/s 2024-04-23T18:19:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:50 smithi177 ceph-mon[32308]: pgmap v42: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 6.4 KiB/s wr, 14 op/s 2024-04-23T18:19:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:50 smithi188 ceph-mon[32694]: pgmap v42: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 6.4 KiB/s wr, 14 op/s 2024-04-23T18:19:51.192 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.42 MB/sec warmup 25 sec latency 24852.444 ms 2024-04-23T18:19:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.769+0000 7f169efc4640 1 -- 172.21.15.50:0/1183974913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16981011a0 msgr2=0x7f1698101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.769+0000 7f169efc4640 1 --2- 172.21.15.50:0/1183974913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16981011a0 0x7f1698101620 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1684009900 tx=0x7f168402a400 comp rx=0 tx=0).stop 2024-04-23T18:19:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.769+0000 7f169efc4640 1 -- 172.21.15.50:0/1183974913 shutdown_connections 2024-04-23T18:19:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 --2- 172.21.15.50:0/1183974913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1698101b60 0x7f169810dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 --2- 172.21.15.50:0/1183974913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16981011a0 0x7f1698101620 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 --2- 172.21.15.50:0/1183974913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1698106fb0 0x7f1698107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 -- 172.21.15.50:0/1183974913 >> 172.21.15.50:0/1183974913 conn(0x7f1698076c50 msgr2=0x7f1698077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 -- 172.21.15.50:0/1183974913 shutdown_connections 2024-04-23T18:19:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 -- 172.21.15.50:0/1183974913 wait complete. 2024-04-23T18:19:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 Processor -- start 2024-04-23T18:19:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 -- start start 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16981011a0 0x7f1698102b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1698101b60 0x7f1698103050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1698106fb0 0x7f1698103590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1698111090 con 0x7f1698101b60 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1698110f10 con 0x7f1698106fb0 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.770+0000 7f169efc4640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1698111210 con 0x7f16981011a0 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.771+0000 7f1697fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1698101b60 0x7f1698103050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.771+0000 7f1697fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1698101b60 0x7f1698103050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43122/0 (socket says 172.21.15.50:43122) 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.771+0000 7f1697fff640 1 -- 172.21.15.50:0/208286144 learned_addr learned my addr 172.21.15.50:0/208286144 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.771+0000 7f169cd60640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16981011a0 0x7f1698102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.771+0000 7f169d561640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1698106fb0 0x7f1698103590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.771+0000 7f1697fff640 1 -- 172.21.15.50:0/208286144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16981011a0 msgr2=0x7f1698102b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.771+0000 7f1697fff640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16981011a0 0x7f1698102b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.771+0000 7f1697fff640 1 -- 172.21.15.50:0/208286144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1698106fb0 msgr2=0x7f1698103590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:51.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.771+0000 7f1697fff640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1698106fb0 0x7f1698103590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.771+0000 7f1697fff640 1 -- 172.21.15.50:0/208286144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16981a6af0 con 0x7f1698101b60 2024-04-23T18:19:51.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.771+0000 7f1697fff640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1698101b60 0x7f1698103050 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1684005940 tx=0x7f16840040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:51.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.771+0000 7f1695ffb640 1 -- 172.21.15.50:0/208286144 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168403f030 con 0x7f1698101b60 2024-04-23T18:19:51.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.772+0000 7f1695ffb640 1 -- 172.21.15.50:0/208286144 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16840044c0 con 0x7f1698101b60 2024-04-23T18:19:51.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.772+0000 7f169efc4640 1 -- 172.21.15.50:0/208286144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16981a6d80 con 0x7f1698101b60 2024-04-23T18:19:51.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.772+0000 7f1695ffb640 1 -- 172.21.15.50:0/208286144 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1684035070 con 0x7f1698101b60 2024-04-23T18:19:51.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.772+0000 7f169efc4640 1 -- 172.21.15.50:0/208286144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16981a7210 con 0x7f1698101b60 2024-04-23T18:19:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.774+0000 7f1695ffb640 1 -- 172.21.15.50:0/208286144 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f1684042030 con 0x7f1698101b60 2024-04-23T18:19:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.774+0000 7f16a1db0740 1 Processor -- start 2024-04-23T18:19:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.774+0000 7f16a1db0740 1 -- start start 2024-04-23T18:19:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f16a1db0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78c620 0x5595cc78ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f16a1db0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78cf40 0x5595cc7972c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f16a1db0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc797800 0x5595cc799bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f16a1db0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5595cc6f1970 con 0x5595cc797800 2024-04-23T18:19:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f16a1db0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5595cc68e4a0 con 0x5595cc78c620 2024-04-23T18:19:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f16a1db0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5595cc6ed930 con 0x5595cc78cf40 2024-04-23T18:19:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f1695ffb640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f16880a3ee0 0x7f16880a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169cd60640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78cf40 0x5595cc7972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f1697fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc797800 0x5595cc799bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169d561640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78c620 0x5595cc78ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169cd60640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78cf40 0x5595cc7972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:59588/0 (socket says 172.21.15.50:59588) 2024-04-23T18:19:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169d561640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78c620 0x5595cc78ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:48710/0 (socket says 172.21.15.50:48710) 2024-04-23T18:19:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169cd60640 1 -- 172.21.15.50:0/559462305 learned_addr learned my addr 172.21.15.50:0/559462305 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169d561640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f16880a3ee0 0x7f16880a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f1695ffb640 1 -- 172.21.15.50:0/208286144 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f16840f0d70 con 0x7f1698101b60 2024-04-23T18:19:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169cd60640 1 -- 172.21.15.50:0/559462305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78c620 msgr2=0x5595cc78ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169cd60640 1 --2- 172.21.15.50:0/559462305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78c620 0x5595cc78ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169cd60640 1 -- 172.21.15.50:0/559462305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc797800 msgr2=0x5595cc799bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169cd60640 1 --2- 172.21.15.50:0/559462305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc797800 0x5595cc799bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169cd60640 1 -- 172.21.15.50:0/559462305 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595cc404210 con 0x5595cc78cf40 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169d561640 1 --2- 172.21.15.50:0/559462305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78c620 0x5595cc78ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169d561640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f16880a3ee0 0x7f16880a63a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1690005c40 tx=0x7f1690009500 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.775+0000 7f169cd60640 1 --2- 172.21.15.50:0/559462305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78cf40 0x5595cc7972c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f168c009c70 tx=0x7f168c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2111d88db0fa6ba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.776+0000 7f1682ffd640 1 -- 172.21.15.50:0/559462305 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c004230 con 0x5595cc78cf40 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.776+0000 7f1682ffd640 1 -- 172.21.15.50:0/559462305 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f168c009e20 con 0x5595cc78cf40 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.776+0000 7f1682ffd640 1 -- 172.21.15.50:0/559462305 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c019640 con 0x5595cc78cf40 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.776+0000 7f16a1db0740 1 -- 172.21.15.50:0/559462305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78cf40 msgr2=0x5595cc7972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.776+0000 7f16a1db0740 1 --2- 172.21.15.50:0/559462305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78cf40 0x5595cc7972c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f168c009c70 tx=0x7f168c011810 comp rx=0 tx=0).stop 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.776+0000 7f16a1db0740 1 -- 172.21.15.50:0/559462305 shutdown_connections 2024-04-23T18:19:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.776+0000 7f16a1db0740 1 --2- 172.21.15.50:0/559462305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc797800 0x5595cc799bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.776+0000 7f16a1db0740 1 --2- 172.21.15.50:0/559462305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78cf40 0x5595cc7972c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.776+0000 7f16a1db0740 1 --2- 172.21.15.50:0/559462305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78c620 0x5595cc78ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.777+0000 7f16a1db0740 1 -- 172.21.15.50:0/559462305 >> 172.21.15.50:0/559462305 conn(0x5595cc788b30 msgr2=0x5595cc788f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.777+0000 7f16a1db0740 1 -- 172.21.15.50:0/559462305 shutdown_connections 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.777+0000 7f16a1db0740 1 -- 172.21.15.50:0/559462305 wait complete. 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.777+0000 7f16a1db0740 1 -- start start 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.777+0000 7f16a1db0740 1 Processor -- start 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.777+0000 7f16a1db0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78c620 0x5595cc796ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.777+0000 7f16a1db0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78cf40 0x5595cc7942e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.777+0000 7f16a1db0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc797800 0x5595cc794820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f16a1db0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5595cc79a430 con 0x5595cc78cf40 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f16a1db0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5595cc68c0b0 con 0x5595cc797800 2024-04-23T18:19:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f16a1db0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5595cc79a5b0 con 0x5595cc78c620 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f16a1db0740 10 client.0 _ll_register_callbacks cb 0x5595cc679d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1697fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78cf40 0x5595cc7942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1697fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78cf40 0x5595cc7942e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43146/0 (socket says 172.21.15.50:43146) 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1697fff640 1 -- 172.21.15.50:0/1533926762 learned_addr learned my addr 172.21.15.50:0/1533926762 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f169cd60640 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78c620 0x5595cc796ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f169d561640 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc797800 0x5595cc794820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1697fff640 1 -- 172.21.15.50:0/1533926762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78c620 msgr2=0x5595cc796ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1697fff640 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78c620 0x5595cc796ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1697fff640 1 -- 172.21.15.50:0/1533926762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc797800 msgr2=0x5595cc794820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1697fff640 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc797800 0x5595cc794820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1697fff640 1 -- 172.21.15.50:0/1533926762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595cc4d0620 con 0x5595cc78cf40 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1697fff640 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78cf40 0x5595cc7942e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f16840bc8f0 tx=0x7f168403b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c627a55a111635c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1677fff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1677fff640 1 -- 172.21.15.50:0/1533926762 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1684049040 con 0x5595cc78cf40 2024-04-23T18:19:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1677fff640 1 -- 172.21.15.50:0/1533926762 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f16840b9260 con 0x5595cc78cf40 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f16a1db0740 1 -- 172.21.15.50:0/1533926762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f168c009e20 con 0x5595cc78cf40 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f1677fff640 1 -- 172.21.15.50:0/1533926762 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f16840c0ef0 con 0x5595cc78cf40 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f16a1db0740 10 client.15699 fetch_fsmap learned FSMap version 22 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.778+0000 7f16a1db0740 1 -- 172.21.15.50:0/1533926762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5595cc7a6220 con 0x5595cc78cf40 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.779+0000 7f1677fff640 1 -- 172.21.15.50:0/1533926762 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f168c009e20 con 0x5595cc78cf40 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.779+0000 7f1677fff640 1 -- 172.21.15.50:0/1533926762 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 22) ==== 8342+0+0 (secure 0 0 0) 0x7f16840eebc0 con 0x5595cc78cf40 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.779+0000 7f16a1db0740 10 client.15699 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.779+0000 7f16a1db0740 10 client.15699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.779+0000 7f16a1db0740 20 client.15699 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.779+0000 7f16a1db0740 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595cc7a7660 0x5595cc7a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.779+0000 7f16a1db0740 4 client.15699 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.779+0000 7f16a1db0740 1 -- 172.21.15.50:0/1533926762 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595cc6ff390 con 0x5595cc7a7660 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.779+0000 7f169cd60640 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595cc7a7660 0x5595cc7a9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.780+0000 7f1677fff640 10 client.15699 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.780+0000 7f169cd60640 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595cc7a7660 0x5595cc7a9ac0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e2f97d094733ca0 server_cookie=54ecf0a84ba527eb in_seq=0 out_seq=0 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f1677fff640 1 -- 172.21.15.50:0/1533926762 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5595cc6ff390 con 0x5595cc7a7660 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f1677fff640 10 client.15699 handle_command_reply: tid=0 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f16a1db0740 1 client.15699 shutdown 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f16a1db0740 1 -- 172.21.15.50:0/1533926762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78cf40 msgr2=0x5595cc7942e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f16a1db0740 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78cf40 0x5595cc7942e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f16840bc8f0 tx=0x7f168403b4f0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f16a1db0740 1 -- 172.21.15.50:0/1533926762 shutdown_connections 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f16a1db0740 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595cc7a7660 0x5595cc7a9ac0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f16a1db0740 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc797800 0x5595cc794820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f16a1db0740 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78cf40 0x5595cc7942e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f1677fff640 0 client.15699 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f16a1db0740 1 --2- 172.21.15.50:0/1533926762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78c620 0x5595cc796ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f16a1db0740 1 -- 172.21.15.50:0/1533926762 >> 172.21.15.50:0/1533926762 conn(0x5595cc788b30 msgr2=0x5595cc78e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:51.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f16a1db0740 1 -- 172.21.15.50:0/1533926762 shutdown_connections 2024-04-23T18:19:51.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f16a1db0740 1 -- 172.21.15.50:0/1533926762 wait complete. 2024-04-23T18:19:51.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.781+0000 7f16a1db0740 20 client.15699 trim_cache size 0 max 16384 2024-04-23T18:19:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.796+0000 7f16a1db0740 1 Processor -- start 2024-04-23T18:19:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.796+0000 7f16a1db0740 1 -- start start 2024-04-23T18:19:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f16a1db0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78c620 0x5595cc796b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f16a1db0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78cf40 0x5595cc7af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f16a1db0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc797800 0x5595cc7af910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f16a1db0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f16981b24b0 con 0x5595cc797800 2024-04-23T18:19:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f16a1db0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f16981afd80 con 0x5595cc78cf40 2024-04-23T18:19:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f16a1db0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f16981adbd0 con 0x5595cc78c620 2024-04-23T18:19:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f169d561640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc797800 0x5595cc7af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f169d561640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc797800 0x5595cc7af910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43148/0 (socket says 172.21.15.50:43148) 2024-04-23T18:19:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f169d561640 1 -- 172.21.15.50:0/3535014799 learned_addr learned my addr 172.21.15.50:0/3535014799 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f1697fff640 1 --2- 172.21.15.50:0/3535014799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78cf40 0x5595cc7af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f169cd60640 1 --2- 172.21.15.50:0/3535014799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78c620 0x5595cc796b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f169d561640 1 -- 172.21.15.50:0/3535014799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78c620 msgr2=0x5595cc796b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f169d561640 1 --2- 172.21.15.50:0/3535014799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78c620 0x5595cc796b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f169d561640 1 -- 172.21.15.50:0/3535014799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78cf40 msgr2=0x5595cc7af3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f169d561640 1 --2- 172.21.15.50:0/3535014799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78cf40 0x5595cc7af3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f169d561640 1 -- 172.21.15.50:0/3535014799 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595cc78d9f0 con 0x5595cc797800 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f1697fff640 1 --2- 172.21.15.50:0/3535014799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78cf40 0x5595cc7af3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f169d561640 1 --2- 172.21.15.50:0/3535014799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc797800 0x5595cc7af910 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f16900099e0 tx=0x7f1690008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d5c6b62c7e4c65 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f16827fc640 1 -- 172.21.15.50:0/3535014799 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16900045b0 con 0x5595cc797800 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.797+0000 7f16827fc640 1 -- 172.21.15.50:0/3535014799 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1690004750 con 0x5595cc797800 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.798+0000 7f16827fc640 1 -- 172.21.15.50:0/3535014799 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16900206a0 con 0x5595cc797800 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.798+0000 7f16a1db0740 1 -- 172.21.15.50:0/3535014799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc797800 msgr2=0x5595cc7af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.798+0000 7f16a1db0740 1 --2- 172.21.15.50:0/3535014799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc797800 0x5595cc7af910 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f16900099e0 tx=0x7f1690008000 comp rx=0 tx=0).stop 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.798+0000 7f16a1db0740 1 -- 172.21.15.50:0/3535014799 shutdown_connections 2024-04-23T18:19:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.798+0000 7f16a1db0740 1 --2- 172.21.15.50:0/3535014799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc797800 0x5595cc7af910 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.798+0000 7f16a1db0740 1 --2- 172.21.15.50:0/3535014799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78cf40 0x5595cc7af3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.798+0000 7f16a1db0740 1 --2- 172.21.15.50:0/3535014799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc78c620 0x5595cc796b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.798+0000 7f16a1db0740 1 -- 172.21.15.50:0/3535014799 >> 172.21.15.50:0/3535014799 conn(0x5595cc788b30 msgr2=0x5595cc7942a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.798+0000 7f16a1db0740 1 -- 172.21.15.50:0/3535014799 shutdown_connections 2024-04-23T18:19:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.798+0000 7f16a1db0740 1 -- 172.21.15.50:0/3535014799 wait complete. 2024-04-23T18:19:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.798+0000 7f16a1db0740 1 -- start start 2024-04-23T18:19:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f16a1db0740 1 Processor -- start 2024-04-23T18:19:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f16a1db0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78c620 0x5595cc79b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f16a1db0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78cf40 0x5595cc79b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f16a1db0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc797800 0x5595cc79be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f16a1db0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f16981ae240 con 0x5595cc78c620 2024-04-23T18:19:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f16a1db0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f16981af6b0 con 0x5595cc78cf40 2024-04-23T18:19:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f16a1db0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f16981ae910 con 0x5595cc797800 2024-04-23T18:19:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f16a1db0740 10 client.0 _ll_register_callbacks cb 0x5595cc679d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f169cd60640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78c620 0x5595cc79b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f169cd60640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78c620 0x5595cc79b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43154/0 (socket says 172.21.15.50:43154) 2024-04-23T18:19:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f169cd60640 1 -- 172.21.15.50:0/1350710714 learned_addr learned my addr 172.21.15.50:0/1350710714 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f1697fff640 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78cf40 0x5595cc79b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f169d561640 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc797800 0x5595cc79be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f169cd60640 1 -- 172.21.15.50:0/1350710714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc797800 msgr2=0x5595cc79be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f169cd60640 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc797800 0x5595cc79be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f169cd60640 1 -- 172.21.15.50:0/1350710714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78cf40 msgr2=0x5595cc79b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f169cd60640 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78cf40 0x5595cc79b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f169cd60640 1 -- 172.21.15.50:0/1350710714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595cc80ce40 con 0x5595cc78c620 2024-04-23T18:19:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.799+0000 7f169d561640 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc797800 0x5595cc79be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:19:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f169cd60640 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78c620 0x5595cc79b400 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f168c0041a0 tx=0x7f168c003d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d409bf3ec7b1ef33 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f1677fff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f1677fff640 1 -- 172.21.15.50:0/1350710714 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c003f60 con 0x5595cc78c620 2024-04-23T18:19:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f1677fff640 1 -- 172.21.15.50:0/1350710714 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f168c01f040 con 0x5595cc78c620 2024-04-23T18:19:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f16a1db0740 1 -- 172.21.15.50:0/1350710714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1690004750 con 0x5595cc78c620 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f1677fff640 1 -- 172.21.15.50:0/1350710714 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f168c011d40 con 0x5595cc78c620 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f16a1db0740 10 client.15705 fetch_fsmap learned FSMap version 22 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f16a1db0740 1 -- 172.21.15.50:0/1350710714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5595cc80e130 con 0x5595cc78c620 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f1677fff640 1 -- 172.21.15.50:0/1350710714 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1690004750 con 0x5595cc78c620 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f1677fff640 1 -- 172.21.15.50:0/1350710714 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 22) ==== 8342+0+0 (secure 0 0 0) 0x7f168c018070 con 0x5595cc78c620 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f16a1db0740 10 client.15705 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f16a1db0740 10 client.15705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f16a1db0740 20 client.15705 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f16a1db0740 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595cc80e650 0x5595cc810a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f16a1db0740 4 client.15705 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.800+0000 7f16a1db0740 1 -- 172.21.15.50:0/1350710714 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5595cc6f9200 con 0x5595cc80e650 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.801+0000 7f1697fff640 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595cc80e650 0x5595cc810a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.801+0000 7f1697fff640 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595cc80e650 0x5595cc810a40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16960f1d511dd68 server_cookie=2897a07452f06a77 in_seq=0 out_seq=0 2024-04-23T18:19:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.801+0000 7f1677fff640 10 client.15705 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.813+0000 7f1677fff640 1 -- 172.21.15.50:0/1350710714 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5595cc6f9200 con 0x5595cc80e650 2024-04-23T18:19:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.813+0000 7f1677fff640 10 client.15705 handle_command_reply: tid=0 2024-04-23T18:19:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f16a1db0740 1 client.15705 shutdown 2024-04-23T18:19:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f16a1db0740 1 -- 172.21.15.50:0/1350710714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78c620 msgr2=0x5595cc79b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f16a1db0740 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78c620 0x5595cc79b400 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f168c0041a0 tx=0x7f168c003d90 comp rx=0 tx=0).stop 2024-04-23T18:19:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f16a1db0740 1 -- 172.21.15.50:0/1350710714 shutdown_connections 2024-04-23T18:19:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f16a1db0740 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595cc80e650 0x5595cc810a40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f16a1db0740 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595cc797800 0x5595cc79be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f16a1db0740 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595cc78cf40 0x5595cc79b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f1677fff640 0 client.15705 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f16a1db0740 1 --2- 172.21.15.50:0/1350710714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595cc78c620 0x5595cc79b400 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f16a1db0740 1 -- 172.21.15.50:0/1350710714 >> 172.21.15.50:0/1350710714 conn(0x5595cc788b30 msgr2=0x5595cc787420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f16a1db0740 1 -- 172.21.15.50:0/1350710714 shutdown_connections 2024-04-23T18:19:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f16a1db0740 1 -- 172.21.15.50:0/1350710714 wait complete. 2024-04-23T18:19:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.814+0000 7f16a1db0740 20 client.15705 trim_cache size 0 max 16384 2024-04-23T18:19:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:19:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:19:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:19:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:19:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.815+0000 7f169efc4640 1 -- 172.21.15.50:0/208286144 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f16880a3ee0 msgr2=0x7f16880a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.815+0000 7f169efc4640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f16880a3ee0 0x7f16880a63a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1690005c40 tx=0x7f1690009500 comp rx=0 tx=0).stop 2024-04-23T18:19:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.815+0000 7f169efc4640 1 -- 172.21.15.50:0/208286144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1698101b60 msgr2=0x7f1698103050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.815+0000 7f169efc4640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1698101b60 0x7f1698103050 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1684005940 tx=0x7f16840040c0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.815+0000 7f169efc4640 1 -- 172.21.15.50:0/208286144 shutdown_connections 2024-04-23T18:19:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.815+0000 7f169efc4640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f16880a3ee0 0x7f16880a63a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.815+0000 7f169efc4640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1698106fb0 0x7f1698103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.815+0000 7f169efc4640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1698101b60 0x7f1698103050 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.815+0000 7f169efc4640 1 --2- 172.21.15.50:0/208286144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16981011a0 0x7f1698102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.815+0000 7f169efc4640 1 -- 172.21.15.50:0/208286144 >> 172.21.15.50:0/208286144 conn(0x7f1698076c50 msgr2=0x7f169806a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.815+0000 7f169efc4640 1 -- 172.21.15.50:0/208286144 shutdown_connections 2024-04-23T18:19:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:19:51.815+0000 7f169efc4640 1 -- 172.21.15.50:0/208286144 wait complete. 2024-04-23T18:19:52.192 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.40 MB/sec warmup 26 sec latency 25852.547 ms 2024-04-23T18:19:52.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:52 smithi050 ceph-mon[29187]: pgmap v43: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 6.4 KiB/s wr, 14 op/s 2024-04-23T18:19:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:52 smithi177 ceph-mon[32308]: pgmap v43: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 6.4 KiB/s wr, 14 op/s 2024-04-23T18:19:52.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:52 smithi188 ceph-mon[32694]: pgmap v43: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 6.4 KiB/s wr, 14 op/s 2024-04-23T18:19:53.192 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.39 MB/sec warmup 27 sec latency 26852.625 ms 2024-04-23T18:19:54.192 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.37 MB/sec warmup 28 sec latency 27852.755 ms 2024-04-23T18:19:54.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:54 smithi050 ceph-mon[29187]: pgmap v44: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.4 KiB/s wr, 17 op/s 2024-04-23T18:19:54.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:54 smithi177 ceph-mon[32308]: pgmap v44: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.4 KiB/s wr, 17 op/s 2024-04-23T18:19:54.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:54 smithi188 ceph-mon[32694]: pgmap v44: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.4 KiB/s wr, 17 op/s 2024-04-23T18:19:55.192 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.36 MB/sec warmup 29 sec latency 28852.873 ms 2024-04-23T18:19:56.192 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279 0.35 MB/sec warmup 30 sec latency 29853.004 ms 2024-04-23T18:19:56.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:19:56.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:56 smithi050 ceph-mon[29187]: pgmap v45: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-04-23T18:19:56.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.529+0000 7fa0f95e2640 1 -- 172.21.15.50:0/1487763924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0f4102240 msgr2=0x7fa0f4102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.529+0000 7fa0f95e2640 1 --2- 172.21.15.50:0/1487763924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0f4102240 0x7fa0f4102620 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4009900 tx=0x7fa0e402a410 comp rx=0 tx=0).stop 2024-04-23T18:19:56.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.529+0000 7fa0f95e2640 1 -- 172.21.15.50:0/1487763924 shutdown_connections 2024-04-23T18:19:56.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.529+0000 7fa0f95e2640 1 --2- 172.21.15.50:0/1487763924 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0f4107280 0x7fa0f410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.529+0000 7fa0f95e2640 1 --2- 172.21.15.50:0/1487763924 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0f4102bf0 0x7fa0f4106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.529+0000 7fa0f95e2640 1 --2- 172.21.15.50:0/1487763924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0f4102240 0x7fa0f4102620 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.529+0000 7fa0f95e2640 1 -- 172.21.15.50:0/1487763924 >> 172.21.15.50:0/1487763924 conn(0x7fa0f40790e0 msgr2=0x7fa0f40794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:56.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.530+0000 7fa0f95e2640 1 -- 172.21.15.50:0/1487763924 shutdown_connections 2024-04-23T18:19:56.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.530+0000 7fa0f95e2640 1 -- 172.21.15.50:0/1487763924 wait complete. 2024-04-23T18:19:56.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.530+0000 7fa0f95e2640 1 Processor -- start 2024-04-23T18:19:56.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.530+0000 7fa0f95e2640 1 -- start start 2024-04-23T18:19:56.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0f95e2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0f4102240 0x7fa0f419e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0f95e2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0f4107280 0x7fa0f419e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0f95e2640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0f4198600 0x7fa0f4198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0f95e2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f4112200 con 0x7fa0f4107280 2024-04-23T18:19:56.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0f95e2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f4112080 con 0x7fa0f4102240 2024-04-23T18:19:56.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0f95e2640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f4112380 con 0x7fa0f4198600 2024-04-23T18:19:56.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0ea7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0f4107280 0x7fa0f419e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0ea7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0f4107280 0x7fa0f419e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43176/0 (socket says 172.21.15.50:43176) 2024-04-23T18:19:56.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0ea7fc640 1 -- 172.21.15.50:0/3973980497 learned_addr learned my addr 172.21.15.50:0/3973980497 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:56.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0f2ffd640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0f4102240 0x7fa0f419e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0f37fe640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0f4198600 0x7fa0f4198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0ea7fc640 1 -- 172.21.15.50:0/3973980497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0f4198600 msgr2=0x7fa0f4198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0ea7fc640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0f4198600 0x7fa0f4198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0ea7fc640 1 -- 172.21.15.50:0/3973980497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0f4102240 msgr2=0x7fa0f419e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0ea7fc640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0f4102240 0x7fa0f419e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0ea7fc640 1 -- 172.21.15.50:0/3973980497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f4199120 con 0x7fa0f4107280 2024-04-23T18:19:56.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0f37fe640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0f4198600 0x7fa0f4198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:56.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.531+0000 7fa0ea7fc640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0f4107280 0x7fa0f419e990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00e9c0 tx=0x7fa0dc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:56.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.532+0000 7fa0f0ff9640 1 -- 172.21.15.50:0/3973980497 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc00ccc0 con 0x7fa0f4107280 2024-04-23T18:19:56.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.532+0000 7fa0f0ff9640 1 -- 172.21.15.50:0/3973980497 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0dc0054d0 con 0x7fa0f4107280 2024-04-23T18:19:56.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.532+0000 7fa0f95e2640 1 -- 172.21.15.50:0/3973980497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f4199410 con 0x7fa0f4107280 2024-04-23T18:19:56.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.532+0000 7fa0f0ff9640 1 -- 172.21.15.50:0/3973980497 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc010430 con 0x7fa0f4107280 2024-04-23T18:19:56.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.532+0000 7fa0f95e2640 1 -- 172.21.15.50:0/3973980497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f41a5610 con 0x7fa0f4107280 2024-04-23T18:19:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.533+0000 7fa0f95e2640 1 -- 172.21.15.50:0/3973980497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f4199b30 con 0x7fa0f4107280 2024-04-23T18:19:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.534+0000 7fa0f0ff9640 1 -- 172.21.15.50:0/3973980497 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fa0dc005020 con 0x7fa0f4107280 2024-04-23T18:19:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.535+0000 7fa0f0ff9640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0e00a3e40 0x7fa0e00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.535+0000 7fa0f2ffd640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0e00a3e40 0x7fa0e00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.535+0000 7fa0f0ff9640 1 -- 172.21.15.50:0/3973980497 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fa0dc0d0e60 con 0x7fa0f4107280 2024-04-23T18:19:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.535+0000 7fa0f2ffd640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0e00a3e40 0x7fa0e00a6300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4031440 tx=0x7fa0e4005b50 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.540+0000 7fa0f0ff9640 1 -- 172.21.15.50:0/3973980497 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa0dc09d2c0 con 0x7fa0f4107280 2024-04-23T18:19:56.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:56 smithi177 ceph-mon[32308]: pgmap v45: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-04-23T18:19:56.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:56 smithi188 ceph-mon[32694]: pgmap v45: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-04-23T18:19:56.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.767+0000 7fa0f95e2640 1 -- 172.21.15.50:0/3973980497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0f410a480 con 0x7fa0f4107280 2024-04-23T18:19:56.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.768+0000 7fa0f0ff9640 1 -- 172.21.15.50:0/3973980497 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11876 (secure 0 0 0) 0x7fa0dc09cbc0 con 0x7fa0f4107280 2024-04-23T18:19:56.761 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:19:56.761 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:19:38.805732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:19:56.761 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 22 2024-04-23T18:19:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.770+0000 7fa0f95e2640 1 -- 172.21.15.50:0/3973980497 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0e00a3e40 msgr2=0x7fa0e00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.770+0000 7fa0f95e2640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0e00a3e40 0x7fa0e00a6300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4031440 tx=0x7fa0e4005b50 comp rx=0 tx=0).stop 2024-04-23T18:19:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.770+0000 7fa0f95e2640 1 -- 172.21.15.50:0/3973980497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0f4107280 msgr2=0x7fa0f419e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.770+0000 7fa0f95e2640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0f4107280 0x7fa0f419e990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00e9c0 tx=0x7fa0dc00ee80 comp rx=0 tx=0).stop 2024-04-23T18:19:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.770+0000 7fa0f95e2640 1 -- 172.21.15.50:0/3973980497 shutdown_connections 2024-04-23T18:19:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.770+0000 7fa0f95e2640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0e00a3e40 0x7fa0e00a6300 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.770+0000 7fa0f95e2640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0f4198600 0x7fa0f4198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.770+0000 7fa0f95e2640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0f4107280 0x7fa0f419e990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.770+0000 7fa0f95e2640 1 --2- 172.21.15.50:0/3973980497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0f4102240 0x7fa0f419e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.770+0000 7fa0f95e2640 1 -- 172.21.15.50:0/3973980497 >> 172.21.15.50:0/3973980497 conn(0x7fa0f40790e0 msgr2=0x7fa0f41003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.770+0000 7fa0f95e2640 1 -- 172.21.15.50:0/3973980497 shutdown_connections 2024-04-23T18:19:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.770+0000 7fa0f95e2640 1 -- 172.21.15.50:0/3973980497 wait complete. 2024-04-23T18:19:56.774 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id bff1ca7e --release --await' 2024-04-23T18:19:56.774 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id bff1ca7e --release --await 2024-04-23T18:19:56.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.930+0000 7fdd0b944640 1 -- 172.21.15.50:0/2356809497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd041031d0 msgr2=0x7fdd041035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.930+0000 7fdd0b944640 1 --2- 172.21.15.50:0/2356809497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd041031d0 0x7fdd041035b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009900 tx=0x7fdd0002a510 comp rx=0 tx=0).stop 2024-04-23T18:19:56.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.930+0000 7fdd0b944640 1 -- 172.21.15.50:0/2356809497 shutdown_connections 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.930+0000 7fdd0b944640 1 --2- 172.21.15.50:0/2356809497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd0410c510 0x7fdd0410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.930+0000 7fdd0b944640 1 --2- 172.21.15.50:0/2356809497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd04103af0 0x7fdd0410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.930+0000 7fdd0b944640 1 --2- 172.21.15.50:0/2356809497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd041031d0 0x7fdd041035b0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.930+0000 7fdd0b944640 1 -- 172.21.15.50:0/2356809497 >> 172.21.15.50:0/2356809497 conn(0x7fdd04077f50 msgr2=0x7fdd04078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.930+0000 7fdd0b944640 1 -- 172.21.15.50:0/2356809497 shutdown_connections 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.931+0000 7fdd0b944640 1 -- 172.21.15.50:0/2356809497 wait complete. 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.931+0000 7fdd0b944640 1 Processor -- start 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.931+0000 7fdd0b944640 1 -- start start 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.931+0000 7fdd0b944640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd041031d0 0x7fdd04100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.931+0000 7fdd0b944640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd04103af0 0x7fdd04101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.931+0000 7fdd0b944640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd0410c510 0x7fdd041018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.931+0000 7fdd0b944640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041121a0 con 0x7fdd0410c510 2024-04-23T18:19:56.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.931+0000 7fdd0b944640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd04112020 con 0x7fdd041031d0 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.931+0000 7fdd0b944640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdd04112320 con 0x7fdd04103af0 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.931+0000 7fdd09ee1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd0410c510 0x7fdd041018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.931+0000 7fdd09ee1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd0410c510 0x7fdd041018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43188/0 (socket says 172.21.15.50:43188) 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.931+0000 7fdd09ee1640 1 -- 172.21.15.50:0/1620329433 learned_addr learned my addr 172.21.15.50:0/1620329433 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdd096e0640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd041031d0 0x7fdd04100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdd08edf640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd04103af0 0x7fdd04101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdd09ee1640 1 -- 172.21.15.50:0/1620329433 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd04103af0 msgr2=0x7fdd04101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdd09ee1640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd04103af0 0x7fdd04101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdd09ee1640 1 -- 172.21.15.50:0/1620329433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd041031d0 msgr2=0x7fdd04100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdd09ee1640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd041031d0 0x7fdd04100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdd09ee1640 1 -- 172.21.15.50:0/1620329433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd041a6c40 con 0x7fdd0410c510 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdd08edf640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd04103af0 0x7fdd04101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:19:56.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdd09ee1640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd0410c510 0x7fdd041018a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00eaa0 tx=0x7fdcfc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:56.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdcfa7fc640 1 -- 172.21.15.50:0/1620329433 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc00cdb0 con 0x7fdd0410c510 2024-04-23T18:19:56.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdcfa7fc640 1 -- 172.21.15.50:0/1620329433 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcfc005020 con 0x7fdd0410c510 2024-04-23T18:19:56.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdd0b944640 1 -- 172.21.15.50:0/1620329433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd041a6f30 con 0x7fdd0410c510 2024-04-23T18:19:56.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdcfa7fc640 1 -- 172.21.15.50:0/1620329433 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc0116d0 con 0x7fdd0410c510 2024-04-23T18:19:56.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.932+0000 7fdd0b944640 1 -- 172.21.15.50:0/1620329433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd041a74a0 con 0x7fdd0410c510 2024-04-23T18:19:56.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.934+0000 7fdcfa7fc640 1 -- 172.21.15.50:0/1620329433 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fdcfc0051d0 con 0x7fdd0410c510 2024-04-23T18:19:56.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd0e65e740 1 Processor -- start 2024-04-23T18:19:56.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd0e65e740 1 -- start start 2024-04-23T18:19:56.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd0e65e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 0x55e1b7143870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd0e65e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd0e65e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b714e480 0x55e1b7150870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd0e65e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e1b70a86d0 con 0x55e1b7143db0 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd0e65e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e1b70451a0 con 0x55e1b714e480 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd0e65e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e1b70a4690 con 0x55e1b7143490 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd096e0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd096e0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43192/0 (socket says 172.21.15.50:43192) 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd096e0640 1 -- 172.21.15.50:0/3963226511 learned_addr learned my addr 172.21.15.50:0/3963226511 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd08edf640 1 --2- 172.21.15.50:0/3963226511 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 0x55e1b7143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdcfa7fc640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdcf00a3ec0 0x7fdcf00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd08edf640 1 --2- 172.21.15.50:0/3963226511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b714e480 0x55e1b7150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd09ee1640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdcf00a3ec0 0x7fdcf00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd096e0640 1 -- 172.21.15.50:0/3963226511 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 msgr2=0x55e1b7143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd096e0640 1 --2- 172.21.15.50:0/3963226511 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 0x55e1b7143870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd096e0640 1 -- 172.21.15.50:0/3963226511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b714e480 msgr2=0x55e1b7150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd096e0640 1 --2- 172.21.15.50:0/3963226511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b714e480 0x55e1b7150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdd096e0640 1 -- 172.21.15.50:0/3963226511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1b704f5b0 con 0x55e1b7143db0 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.935+0000 7fdcfa7fc640 1 -- 172.21.15.50:0/1620329433 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fdcfc0d0bb0 con 0x7fdd0410c510 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.936+0000 7fdd096e0640 1 --2- 172.21.15.50:0/3963226511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714df40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009900 tx=0x7fdd00032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69dcfcb62f33e371 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.936+0000 7fdd09ee1640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdcf00a3ec0 0x7fdcf00a6380 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc0ce190 tx=0x7fdcfc0bd040 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.936+0000 7fdcef7fe640 1 -- 172.21.15.50:0/3963226511 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd00040070 con 0x55e1b7143db0 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.936+0000 7fdcef7fe640 1 -- 172.21.15.50:0/3963226511 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd00004030 con 0x55e1b7143db0 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.936+0000 7fdcef7fe640 1 -- 172.21.15.50:0/3963226511 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd00035040 con 0x55e1b7143db0 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.936+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3963226511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 msgr2=0x55e1b714df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.936+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/3963226511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714df40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009900 tx=0x7fdd00032bd0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3963226511 shutdown_connections 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/3963226511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b714e480 0x55e1b7150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/3963226511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714df40 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/3963226511 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 0x55e1b7143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3963226511 >> 172.21.15.50:0/3963226511 conn(0x55e1b713f7b0 msgr2=0x55e1b713fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3963226511 shutdown_connections 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3963226511 wait complete. 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 -- start start 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 Processor -- start 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 0x55e1b714dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b714e480 0x55e1b714b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e1b71512b0 con 0x55e1b7143db0 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e1b7042b20 con 0x55e1b714e480 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e1b7151430 con 0x55e1b7143490 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd0e65e740 10 client.0 _ll_register_callbacks cb 0x55e1b7042530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.937+0000 7fdd096e0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd096e0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43194/0 (socket says 172.21.15.50:43194) 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd096e0640 1 -- 172.21.15.50:0/3248351480 learned_addr learned my addr 172.21.15.50:0/3248351480 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd08edf640 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 0x55e1b714dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd08edf640 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b714e480 0x55e1b714b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd096e0640 1 -- 172.21.15.50:0/3248351480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 msgr2=0x55e1b714dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd096e0640 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 0x55e1b714dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd096e0640 1 -- 172.21.15.50:0/3248351480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b714e480 msgr2=0x55e1b714b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd096e0640 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b714e480 0x55e1b714b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd096e0640 1 -- 172.21.15.50:0/3248351480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1b6dba920 con 0x55e1b7143db0 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd08edf640 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b714e480 0x55e1b714b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd096e0640 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714b1b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009640 tx=0x7fdd00005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555b8fb78703cfb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdcecff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdcecff9640 1 -- 172.21.15.50:0/3248351480 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd00040070 con 0x55e1b7143db0 2024-04-23T18:19:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdcecff9640 1 -- 172.21.15.50:0/3248351480 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fdd00005460 con 0x55e1b7143db0 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3248351480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd00004030 con 0x55e1b7143db0 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdcecff9640 1 -- 172.21.15.50:0/3248351480 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fdd00056050 con 0x55e1b7143db0 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd0e65e740 10 client.15723 fetch_fsmap learned FSMap version 22 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3248351480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55e1b6e86d30 con 0x55e1b7143db0 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.938+0000 7fdcecff9640 1 -- 172.21.15.50:0/3248351480 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd00004030 con 0x55e1b7143db0 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.939+0000 7fdcecff9640 1 -- 172.21.15.50:0/3248351480 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 22) ==== 8342+0+0 (secure 0 0 0) 0x7fdd00004c90 con 0x55e1b7143db0 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.939+0000 7fdd0e65e740 10 client.15723 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.939+0000 7fdd0e65e740 10 client.15723 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.939+0000 7fdd0e65e740 20 client.15723 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.939+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e1b715e490 0x55e1b7160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.939+0000 7fdd0e65e740 4 client.15723 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.939+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3248351480 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1b7066500 con 0x55e1b715e490 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.939+0000 7fdd08edf640 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e1b715e490 0x55e1b7160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.939+0000 7fdd08edf640 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e1b715e490 0x55e1b7160880 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e707e8fbd4a61a1 server_cookie=3aa4971b2ce77fe7 in_seq=0 out_seq=0 2024-04-23T18:19:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.939+0000 7fdcecff9640 10 client.15723 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdcecff9640 1 -- 172.21.15.50:0/3248351480 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e1b7066500 con 0x55e1b715e490 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdcecff9640 10 client.15723 handle_command_reply: tid=0 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdd0e65e740 1 client.15723 shutdown 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3248351480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 msgr2=0x55e1b714b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714b1b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009640 tx=0x7fdd00005db0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3248351480 shutdown_connections 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e1b715e490 0x55e1b7160880 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b714e480 0x55e1b714b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714b1b0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/3248351480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 0x55e1b714dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3248351480 >> 172.21.15.50:0/3248351480 conn(0x55e1b713f7b0 msgr2=0x55e1b713e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdcecff9640 0 client.15723 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:56.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3248351480 shutdown_connections 2024-04-23T18:19:56.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdd0e65e740 1 -- 172.21.15.50:0/3248351480 wait complete. 2024-04-23T18:19:56.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.940+0000 7fdd0e65e740 20 client.15723 trim_cache size 0 max 16384 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd0e65e740 1 Processor -- start 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd0e65e740 1 -- start start 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd0e65e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b713f7b0 0x55e1b715a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd0e65e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b7143490 0x55e1b714d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd0e65e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd0e65e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041b26b0 con 0x55e1b7143db0 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd0e65e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041aff80 con 0x55e1b7143490 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd0e65e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041addd0 con 0x55e1b713f7b0 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd08edf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd08edf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43198/0 (socket says 172.21.15.50:43198) 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd08edf640 1 -- 172.21.15.50:0/2345180594 learned_addr learned my addr 172.21.15.50:0/2345180594 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd096e0640 1 --2- 172.21.15.50:0/2345180594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b7143490 0x55e1b714d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd08edf640 1 --2- 172.21.15.50:0/2345180594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b713f7b0 0x55e1b715a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd08edf640 1 -- 172.21.15.50:0/2345180594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b713f7b0 msgr2=0x55e1b715a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd08edf640 1 --2- 172.21.15.50:0/2345180594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b713f7b0 0x55e1b715a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd08edf640 1 -- 172.21.15.50:0/2345180594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b7143490 msgr2=0x55e1b714d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd08edf640 1 --2- 172.21.15.50:0/2345180594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b7143490 0x55e1b714d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd08edf640 1 -- 172.21.15.50:0/2345180594 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1b714a160 con 0x55e1b7143db0 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd096e0640 1 --2- 172.21.15.50:0/2345180594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b7143490 0x55e1b714d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.958+0000 7fdd08edf640 1 --2- 172.21.15.50:0/2345180594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714df30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4004700 tx=0x7fdcf4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4e4d40942c6198 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdceeffd640 1 -- 172.21.15.50:0/2345180594 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4004020 con 0x55e1b7143db0 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdceeffd640 1 -- 172.21.15.50:0/2345180594 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcf4026030 con 0x55e1b7143db0 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdceeffd640 1 -- 172.21.15.50:0/2345180594 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4004a00 con 0x55e1b7143db0 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdd0e65e740 1 -- 172.21.15.50:0/2345180594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 msgr2=0x55e1b714df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/2345180594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714df30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4004700 tx=0x7fdcf4004730 comp rx=0 tx=0).stop 2024-04-23T18:19:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdd0e65e740 1 -- 172.21.15.50:0/2345180594 shutdown_connections 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/2345180594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b7143db0 0x55e1b714df30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/2345180594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b7143490 0x55e1b714d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/2345180594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b713f7b0 0x55e1b715a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdd0e65e740 1 -- 172.21.15.50:0/2345180594 >> 172.21.15.50:0/2345180594 conn(0x55e1b7150b50 msgr2=0x55e1b7144be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdd0e65e740 1 -- 172.21.15.50:0/2345180594 shutdown_connections 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdd0e65e740 1 -- 172.21.15.50:0/2345180594 wait complete. 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.959+0000 7fdd0e65e740 1 -- start start 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd0e65e740 1 Processor -- start 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd0e65e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b713f7b0 0x55e1b71566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd0e65e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 0x55e1b7156c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd0e65e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b7143db0 0x55e1b7151670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd0e65e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041ae440 con 0x55e1b713f7b0 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd0e65e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041af8b0 con 0x55e1b7143db0 2024-04-23T18:19:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd0e65e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041aeb10 con 0x55e1b7143490 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd0e65e740 10 client.0 _ll_register_callbacks cb 0x7fdd00032b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd08edf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b713f7b0 0x55e1b71566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd08edf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b713f7b0 0x55e1b71566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43212/0 (socket says 172.21.15.50:43212) 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd08edf640 1 -- 172.21.15.50:0/1672580936 learned_addr learned my addr 172.21.15.50:0/1672580936 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd096e0640 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 0x55e1b7156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd08edf640 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b7143db0 0x55e1b7151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd08edf640 1 -- 172.21.15.50:0/1672580936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 msgr2=0x55e1b7156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd08edf640 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 0x55e1b7156c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd08edf640 1 -- 172.21.15.50:0/1672580936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b7143db0 msgr2=0x55e1b7151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd08edf640 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b7143db0 0x55e1b7151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.960+0000 7fdd08edf640 1 -- 172.21.15.50:0/1672580936 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1b7151ce0 con 0x55e1b713f7b0 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdd08edf640 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b713f7b0 0x55e1b71566c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4005d00 tx=0x7fdcf4005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6834bc2e1b8ba26e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:19:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdcecff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdcecff9640 1 -- 172.21.15.50:0/1672580936 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4033040 con 0x55e1b713f7b0 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdcecff9640 1 -- 172.21.15.50:0/1672580936 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fdcf4013d00 con 0x55e1b713f7b0 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdd0e65e740 1 -- 172.21.15.50:0/1672580936 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcf4026030 con 0x55e1b713f7b0 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdcecff9640 1 -- 172.21.15.50:0/1672580936 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fdcf403f050 con 0x55e1b713f7b0 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdd0e65e740 10 client.15729 fetch_fsmap learned FSMap version 22 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdd0e65e740 1 -- 172.21.15.50:0/1672580936 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55e1b71c5190 con 0x55e1b713f7b0 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdcecff9640 1 -- 172.21.15.50:0/1672580936 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcf4026030 con 0x55e1b713f7b0 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdcecff9640 1 -- 172.21.15.50:0/1672580936 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 22) ==== 8342+0+0 (secure 0 0 0) 0x7fdcf400f040 con 0x55e1b713f7b0 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdd0e65e740 10 client.15729 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdd0e65e740 10 client.15729 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdd0e65e740 20 client.15729 populate_metadata read hostname 'smithi050' 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e1b714e480 0x55e1b71c7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdd0e65e740 4 client.15729 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "bff1ca7e", "release": true, "await": true}] 2024-04-23T18:19:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdd0e65e740 1 -- 172.21.15.50:0/1672580936 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "bff1ca7e", "release": true, "await": true}) -- 0x55e1b70b60f0 con 0x55e1b714e480 2024-04-23T18:19:56.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.961+0000 7fdd096e0640 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e1b714e480 0x55e1b71c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:19:56.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.962+0000 7fdd096e0640 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e1b714e480 0x55e1b71c7810 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22f5d98be963ebfb server_cookie=acba54f9c75afafc in_seq=0 out_seq=0 2024-04-23T18:19:56.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.962+0000 7fdcecff9640 10 client.15729 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdcecff9640 1 -- 172.21.15.50:0/1672580936 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55e1b70b60f0 con 0x55e1b714e480 2024-04-23T18:19:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdcecff9640 10 client.15729 handle_command_reply: tid=0 2024-04-23T18:19:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdd0e65e740 1 client.15729 shutdown 2024-04-23T18:19:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdd0e65e740 1 -- 172.21.15.50:0/1672580936 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b713f7b0 msgr2=0x55e1b71566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b713f7b0 0x55e1b71566c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4005d00 tx=0x7fdcf4005c70 comp rx=0 tx=0).stop 2024-04-23T18:19:56.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdd0e65e740 1 -- 172.21.15.50:0/1672580936 shutdown_connections 2024-04-23T18:19:56.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e1b714e480 0x55e1b71c7810 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e1b7143db0 0x55e1b7151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e1b7143490 0x55e1b7156c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdd0e65e740 1 --2- 172.21.15.50:0/1672580936 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e1b713f7b0 0x55e1b71566c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdcecff9640 0 client.15729 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:19:56.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdd0e65e740 1 -- 172.21.15.50:0/1672580936 >> 172.21.15.50:0/1672580936 conn(0x55e1b7150b50 msgr2=0x55e1b7145470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:56.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdd0e65e740 1 -- 172.21.15.50:0/1672580936 shutdown_connections 2024-04-23T18:19:56.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdd0e65e740 1 -- 172.21.15.50:0/1672580936 wait complete. 2024-04-23T18:19:56.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.968+0000 7fdd0e65e740 20 client.15729 trim_cache size 0 max 16384 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 22, 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 8, 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bff1ca7e": { 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 8, 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:19:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 90.0, 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:19:56.964 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:19:56.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.969+0000 7fdd0b944640 1 -- 172.21.15.50:0/1620329433 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdcf00a3ec0 msgr2=0x7fdcf00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.969+0000 7fdd0b944640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdcf00a3ec0 0x7fdcf00a6380 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc0ce190 tx=0x7fdcfc0bd040 comp rx=0 tx=0).stop 2024-04-23T18:19:56.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.969+0000 7fdd0b944640 1 -- 172.21.15.50:0/1620329433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd0410c510 msgr2=0x7fdd041018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:19:56.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.969+0000 7fdd0b944640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd0410c510 0x7fdd041018a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00eaa0 tx=0x7fdcfc00ef60 comp rx=0 tx=0).stop 2024-04-23T18:19:56.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.969+0000 7fdd0b944640 1 -- 172.21.15.50:0/1620329433 shutdown_connections 2024-04-23T18:19:56.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.969+0000 7fdd0b944640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdcf00a3ec0 0x7fdcf00a6380 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.969+0000 7fdd0b944640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd0410c510 0x7fdd041018a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.969+0000 7fdd0b944640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd04103af0 0x7fdd04101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.969+0000 7fdd0b944640 1 --2- 172.21.15.50:0/1620329433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd041031d0 0x7fdd04100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:19:56.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.969+0000 7fdd0b944640 1 -- 172.21.15.50:0/1620329433 >> 172.21.15.50:0/1620329433 conn(0x7fdd04077f50 msgr2=0x7fdd040fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:19:56.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.969+0000 7fdd0b944640 1 -- 172.21.15.50:0/1620329433 shutdown_connections 2024-04-23T18:19:56.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:19:56.969+0000 7fdd0b944640 1 -- 172.21.15.50:0/1620329433 wait complete. 2024-04-23T18:19:56.973 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'bff1ca7e', seconds elapsed: 31.2 2024-04-23T18:19:57.192 INFO:tasks.workunit.client.0.smithi050.stdout: 1 711 0.68 MB/sec warmup 31 sec latency 30626.483 ms 2024-04-23T18:19:57.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:57 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3973980497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:57.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:57 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3973980497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:57.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:57 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3973980497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:19:58.192 INFO:tasks.workunit.client.0.smithi050.stdout: 1 3812 1.24 MB/sec warmup 32 sec latency 8.185 ms 2024-04-23T18:19:58.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:19:58 smithi050 ceph-mon[29187]: pgmap v46: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-23T18:19:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:19:58 smithi177 ceph-mon[32308]: pgmap v46: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-23T18:19:58.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:19:58 smithi188 ceph-mon[32694]: pgmap v46: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-23T18:19:59.192 INFO:tasks.workunit.client.0.smithi050.stdout: 1 6436 1.55 MB/sec warmup 33 sec latency 32.956 ms 2024-04-23T18:20:00.192 INFO:tasks.workunit.client.0.smithi050.stdout: 1 8527 1.89 MB/sec warmup 34 sec latency 22.589 ms 2024-04-23T18:20:00.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:00 smithi050 ceph-mon[29187]: pgmap v47: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-23T18:20:00.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:00 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:20:00.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:00 smithi050 ceph-mon[29187]: overall HEALTH_OK 2024-04-23T18:20:00.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:00 smithi177 ceph-mon[32308]: pgmap v47: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-23T18:20:00.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:00 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:20:00.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:00 smithi177 ceph-mon[32308]: overall HEALTH_OK 2024-04-23T18:20:00.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:00 smithi188 ceph-mon[32694]: pgmap v47: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-23T18:20:00.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:00 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:20:00.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:00 smithi188 ceph-mon[32694]: overall HEALTH_OK 2024-04-23T18:20:01.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 10902 2.23 MB/sec warmup 35 sec latency 20.140 ms 2024-04-23T18:20:02.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 13581 2.52 MB/sec warmup 36 sec latency 22.463 ms 2024-04-23T18:20:02.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:02 smithi050 ceph-mon[29187]: pgmap v48: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-23T18:20:02.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:02 smithi177 ceph-mon[32308]: pgmap v48: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-23T18:20:02.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:02 smithi188 ceph-mon[32694]: pgmap v48: 97 pgs: 97 active+clean; 12 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-23T18:20:03.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 15667 2.77 MB/sec warmup 37 sec latency 21.698 ms 2024-04-23T18:20:04.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 18476 3.08 MB/sec warmup 38 sec latency 18.543 ms 2024-04-23T18:20:04.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:04 smithi050 ceph-mon[29187]: pgmap v49: 97 pgs: 97 active+clean; 19 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 699 KiB/s wr, 125 op/s 2024-04-23T18:20:04.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:04 smithi050 ceph-mon[29187]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-23T18:20:04.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:04 smithi177 ceph-mon[32308]: pgmap v49: 97 pgs: 97 active+clean; 19 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 699 KiB/s wr, 125 op/s 2024-04-23T18:20:04.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:04 smithi177 ceph-mon[32308]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-23T18:20:04.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:04 smithi188 ceph-mon[32694]: pgmap v49: 97 pgs: 97 active+clean; 19 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 699 KiB/s wr, 125 op/s 2024-04-23T18:20:04.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:04 smithi188 ceph-mon[32694]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-23T18:20:04.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.905+0000 7f0357768640 1 -- 172.21.15.50:0/3397826363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0350103200 msgr2=0x7f03501035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.905+0000 7f0357768640 1 --2- 172.21.15.50:0/3397826363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0350103200 0x7f03501035e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f034c009900 tx=0x7f034c02a480 comp rx=0 tx=0).stop 2024-04-23T18:20:04.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.905+0000 7f0357768640 1 -- 172.21.15.50:0/3397826363 shutdown_connections 2024-04-23T18:20:04.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.905+0000 7f0357768640 1 --2- 172.21.15.50:0/3397826363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f035010c540 0x7f035010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.905+0000 7f0357768640 1 --2- 172.21.15.50:0/3397826363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0350103b20 0x7f035010c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.905+0000 7f0357768640 1 --2- 172.21.15.50:0/3397826363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0350103200 0x7f03501035e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.905+0000 7f0357768640 1 -- 172.21.15.50:0/3397826363 >> 172.21.15.50:0/3397826363 conn(0x7f0350077fd0 msgr2=0x7f03500783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:04.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.905+0000 7f0357768640 1 -- 172.21.15.50:0/3397826363 shutdown_connections 2024-04-23T18:20:04.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.905+0000 7f0357768640 1 -- 172.21.15.50:0/3397826363 wait complete. 2024-04-23T18:20:04.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0357768640 1 Processor -- start 2024-04-23T18:20:04.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0357768640 1 -- start start 2024-04-23T18:20:04.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0357768640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0350103200 0x7f0350100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0357768640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0350103b20 0x7f0350101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0357768640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f035010c540 0x7f03501018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0357768640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0350112180 con 0x7f035010c540 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0357768640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0350112000 con 0x7f0350103b20 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0357768640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0350112300 con 0x7f0350103200 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0355d05640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f035010c540 0x7f03501018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0355504640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0350103200 0x7f0350100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0355d05640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f035010c540 0x7f03501018d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42550/0 (socket says 172.21.15.50:42550) 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0355d05640 1 -- 172.21.15.50:0/3099491278 learned_addr learned my addr 172.21.15.50:0/3099491278 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.906+0000 7f0354d03640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0350103b20 0x7f0350101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f0354d03640 1 -- 172.21.15.50:0/3099491278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0350103200 msgr2=0x7f0350100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f0354d03640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0350103200 0x7f0350100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f0354d03640 1 -- 172.21.15.50:0/3099491278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f035010c540 msgr2=0x7f03501018d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f0354d03640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f035010c540 0x7f03501018d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f0354d03640 1 -- 172.21.15.50:0/3099491278 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03501a6b80 con 0x7f0350103b20 2024-04-23T18:20:04.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f0355d05640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f035010c540 0x7f03501018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:04.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f0355504640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0350103200 0x7f0350100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:04.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f0354d03640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0350103b20 0x7f0350101390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f034000d6d0 tx=0x7f034000db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:04.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f03467fc640 1 -- 172.21.15.50:0/3099491278 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340017070 con 0x7f0350103b20 2024-04-23T18:20:04.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f03467fc640 1 -- 172.21.15.50:0/3099491278 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0340009820 con 0x7f0350103b20 2024-04-23T18:20:04.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f0357768640 1 -- 172.21.15.50:0/3099491278 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03501a6e70 con 0x7f0350103b20 2024-04-23T18:20:04.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f03467fc640 1 -- 172.21.15.50:0/3099491278 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340004c40 con 0x7f0350103b20 2024-04-23T18:20:04.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.907+0000 7f0357768640 1 -- 172.21.15.50:0/3099491278 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03501a72e0 con 0x7f0350103b20 2024-04-23T18:20:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.909+0000 7f03467fc640 1 -- 172.21.15.50:0/3099491278 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0340005220 con 0x7f0350103b20 2024-04-23T18:20:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f035a482740 1 Processor -- start 2024-04-23T18:20:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f035a482740 1 -- start start 2024-04-23T18:20:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f03467fc640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f033c0a3df0 0x7f033c0a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f035a482740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804620 0x5615f2804a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f035a482740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804f40 0x5615f280f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f035a482740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f2811bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f035a482740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5615f2769970 con 0x5615f2804f40 2024-04-23T18:20:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f035a482740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5615f27064a0 con 0x5615f2804620 2024-04-23T18:20:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f035a482740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5615f2765930 con 0x5615f280f800 2024-04-23T18:20:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f0355d05640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f033c0a3df0 0x7f033c0a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f03467fc640 1 -- 172.21.15.50:0/3099491278 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f03400d0160 con 0x7f0350103b20 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f0355504640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804620 0x5615f2804a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f0354d03640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804f40 0x5615f280f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f0355d05640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f2811bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f0355504640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804620 0x5615f2804a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:42808/0 (socket says 172.21.15.50:42808) 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f0355504640 1 -- 172.21.15.50:0/551633407 learned_addr learned my addr 172.21.15.50:0/551633407 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f0355d05640 1 -- 172.21.15.50:0/551633407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804620 msgr2=0x5615f2804a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f0355d05640 1 --2- 172.21.15.50:0/551633407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804620 0x5615f2804a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f0355d05640 1 -- 172.21.15.50:0/551633407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804f40 msgr2=0x5615f280f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.910+0000 7f0355d05640 1 --2- 172.21.15.50:0/551633407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804f40 0x5615f280f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.911+0000 7f0355d05640 1 -- 172.21.15.50:0/551633407 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615f247c210 con 0x5615f280f800 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.911+0000 7f0355d05640 1 --2- 172.21.15.50:0/551633407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f2811bf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0348013560 tx=0x7f0348013a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55ea5907c14e330a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.911+0000 7f0355d05640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f033c0a3df0 0x7f033c0a62b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f03480099a0 tx=0x7f034801e000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.911+0000 7f033b7fe640 1 -- 172.21.15.50:0/551633407 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0348023070 con 0x5615f280f800 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.911+0000 7f033b7fe640 1 -- 172.21.15.50:0/551633407 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f034800ec00 con 0x5615f280f800 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.911+0000 7f033b7fe640 1 -- 172.21.15.50:0/551633407 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0348005210 con 0x5615f280f800 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.912+0000 7f035a482740 1 -- 172.21.15.50:0/551633407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 msgr2=0x5615f2811bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.912+0000 7f035a482740 1 --2- 172.21.15.50:0/551633407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f2811bf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0348013560 tx=0x7f0348013a20 comp rx=0 tx=0).stop 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.912+0000 7f035a482740 1 -- 172.21.15.50:0/551633407 shutdown_connections 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.912+0000 7f035a482740 1 --2- 172.21.15.50:0/551633407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f2811bf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.912+0000 7f035a482740 1 --2- 172.21.15.50:0/551633407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804f40 0x5615f280f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.912+0000 7f035a482740 1 --2- 172.21.15.50:0/551633407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804620 0x5615f2804a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.912+0000 7f035a482740 1 -- 172.21.15.50:0/551633407 >> 172.21.15.50:0/551633407 conn(0x5615f2800b30 msgr2=0x5615f2800f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.912+0000 7f035a482740 1 -- 172.21.15.50:0/551633407 shutdown_connections 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.912+0000 7f035a482740 1 -- 172.21.15.50:0/551633407 wait complete. 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.912+0000 7f035a482740 1 -- start start 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.912+0000 7f035a482740 1 Processor -- start 2024-04-23T18:20:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f035a482740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804620 0x5615f280f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f035a482740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804f40 0x5615f280c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f035a482740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f280cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f035a482740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5615f28123d0 con 0x5615f2804f40 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f035a482740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5615f27040b0 con 0x5615f2804620 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f035a482740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5615f2812550 con 0x5615f280f800 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f035a482740 10 client.0 _ll_register_callbacks cb 0x5615f26f1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0354d03640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804f40 0x5615f280c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0355504640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804620 0x5615f280f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0354d03640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804f40 0x5615f280c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42570/0 (socket says 172.21.15.50:42570) 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0354d03640 1 -- 172.21.15.50:0/2394550251 learned_addr learned my addr 172.21.15.50:0/2394550251 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0355d05640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f280cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0354d03640 1 -- 172.21.15.50:0/2394550251 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 msgr2=0x5615f280cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0354d03640 1 --2- 172.21.15.50:0/2394550251 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f280cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0354d03640 1 -- 172.21.15.50:0/2394550251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804620 msgr2=0x5615f280f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0354d03640 1 --2- 172.21.15.50:0/2394550251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804620 0x5615f280f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0354d03640 1 -- 172.21.15.50:0/2394550251 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615f2548620 con 0x5615f2804f40 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0354d03640 1 --2- 172.21.15.50:0/2394550251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804f40 0x5615f280c4c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0340009610 tx=0x7f034009c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fdc67983ebd15e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0338ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0338ff9640 1 -- 172.21.15.50:0/2394550251 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340020040 con 0x5615f2804f40 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0338ff9640 1 -- 172.21.15.50:0/2394550251 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f03400099d0 con 0x5615f2804f40 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f035a482740 1 -- 172.21.15.50:0/2394550251 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f034800ec00 con 0x5615f2804f40 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f0338ff9640 1 -- 172.21.15.50:0/2394550251 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f034007e030 con 0x5615f2804f40 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.913+0000 7f035a482740 10 client.15741 fetch_fsmap learned FSMap version 23 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.914+0000 7f035a482740 1 -- 172.21.15.50:0/2394550251 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5615f281e3c0 con 0x5615f2804f40 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.914+0000 7f0338ff9640 1 -- 172.21.15.50:0/2394550251 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f034800ec00 con 0x5615f2804f40 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.914+0000 7f0338ff9640 1 -- 172.21.15.50:0/2394550251 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f03400a0150 con 0x5615f2804f40 2024-04-23T18:20:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.914+0000 7f035a482740 10 client.15741 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.914+0000 7f035a482740 10 client.15741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.914+0000 7f035a482740 20 client.15741 populate_metadata read hostname 'smithi050' 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.914+0000 7f035a482740 1 --2- 172.21.15.50:0/2394550251 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f281f830 0x5615f2821c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.914+0000 7f035a482740 4 client.15741 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.914+0000 7f035a482740 1 -- 172.21.15.50:0/2394550251 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615f2777390 con 0x5615f281f830 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.914+0000 7f0355504640 1 --2- 172.21.15.50:0/2394550251 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f281f830 0x5615f2821c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.914+0000 7f0338ff9640 10 client.15741 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.914+0000 7f0355504640 1 --2- 172.21.15.50:0/2394550251 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f281f830 0x5615f2821c40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfee4c668bfdeee7 server_cookie=6ec6784214ea42f3 in_seq=0 out_seq=0 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f0338ff9640 1 -- 172.21.15.50:0/2394550251 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5615f2777390 con 0x5615f281f830 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f0338ff9640 10 client.15741 handle_command_reply: tid=0 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f035a482740 1 client.15741 shutdown 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f035a482740 1 -- 172.21.15.50:0/2394550251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804f40 msgr2=0x5615f280c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f035a482740 1 --2- 172.21.15.50:0/2394550251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804f40 0x5615f280c4c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0340009610 tx=0x7f034009c330 comp rx=0 tx=0).stop 2024-04-23T18:20:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f035a482740 1 -- 172.21.15.50:0/2394550251 shutdown_connections 2024-04-23T18:20:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f035a482740 1 --2- 172.21.15.50:0/2394550251 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f281f830 0x5615f2821c40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f035a482740 1 --2- 172.21.15.50:0/2394550251 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f280cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f035a482740 1 --2- 172.21.15.50:0/2394550251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804f40 0x5615f280c4c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f0338ff9640 0 client.15741 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f035a482740 1 --2- 172.21.15.50:0/2394550251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804620 0x5615f280f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f035a482740 1 -- 172.21.15.50:0/2394550251 >> 172.21.15.50:0/2394550251 conn(0x5615f2800b30 msgr2=0x5615f28068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f035a482740 1 -- 172.21.15.50:0/2394550251 shutdown_connections 2024-04-23T18:20:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.915+0000 7f035a482740 1 -- 172.21.15.50:0/2394550251 wait complete. 2024-04-23T18:20:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.916+0000 7f035a482740 20 client.15741 trim_cache size 0 max 16384 2024-04-23T18:20:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f035a482740 1 Processor -- start 2024-04-23T18:20:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f035a482740 1 -- start start 2024-04-23T18:20:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f035a482740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804620 0x5615f281b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f035a482740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804f40 0x5615f2800430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f035a482740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f2827550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f035a482740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f03501b2510 con 0x5615f2804620 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f035a482740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03501afde0 con 0x5615f2804f40 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f035a482740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f03501adc30 con 0x5615f280f800 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f0355504640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804620 0x5615f281b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f0355504640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804620 0x5615f281b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42584/0 (socket says 172.21.15.50:42584) 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f0355504640 1 -- 172.21.15.50:0/2315798772 learned_addr learned my addr 172.21.15.50:0/2315798772 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f0355d05640 1 --2- 172.21.15.50:0/2315798772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f2827550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f0354d03640 1 --2- 172.21.15.50:0/2315798772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804f40 0x5615f2800430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f0355504640 1 -- 172.21.15.50:0/2315798772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 msgr2=0x5615f2827550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f0355504640 1 --2- 172.21.15.50:0/2315798772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f2827550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f0355504640 1 -- 172.21.15.50:0/2315798772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804f40 msgr2=0x5615f2800430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f0355504640 1 --2- 172.21.15.50:0/2315798772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804f40 0x5615f2800430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f0355504640 1 -- 172.21.15.50:0/2315798772 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615f2811e20 con 0x5615f2804620 2024-04-23T18:20:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f0355504640 1 --2- 172.21.15.50:0/2315798772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804620 0x5615f281b220 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f034c02a9a0 tx=0x7f034c005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594b8243737d9434 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.931+0000 7f033affd640 1 -- 172.21.15.50:0/2315798772 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c035040 con 0x5615f2804620 2024-04-23T18:20:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f033affd640 1 -- 172.21.15.50:0/2315798772 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f034c04b070 con 0x5615f2804620 2024-04-23T18:20:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f033affd640 1 -- 172.21.15.50:0/2315798772 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c005570 con 0x5615f2804620 2024-04-23T18:20:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f035a482740 1 -- 172.21.15.50:0/2315798772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804620 msgr2=0x5615f281b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f035a482740 1 --2- 172.21.15.50:0/2315798772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804620 0x5615f281b220 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f034c02a9a0 tx=0x7f034c005b50 comp rx=0 tx=0).stop 2024-04-23T18:20:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f035a482740 1 -- 172.21.15.50:0/2315798772 shutdown_connections 2024-04-23T18:20:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f035a482740 1 --2- 172.21.15.50:0/2315798772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f280f800 0x5615f2827550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f035a482740 1 --2- 172.21.15.50:0/2315798772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804f40 0x5615f2800430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f035a482740 1 --2- 172.21.15.50:0/2315798772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f2804620 0x5615f281b220 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f035a482740 1 -- 172.21.15.50:0/2315798772 >> 172.21.15.50:0/2315798772 conn(0x5615f2800b30 msgr2=0x5615f281ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f035a482740 1 -- 172.21.15.50:0/2315798772 shutdown_connections 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f035a482740 1 -- 172.21.15.50:0/2315798772 wait complete. 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f035a482740 1 -- start start 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.932+0000 7f035a482740 1 Processor -- start 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f035a482740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f2804620 0x5615f2815450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f035a482740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804f40 0x5615f2815990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f035a482740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f280f800 0x5615f2815ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f035a482740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f03501ae2a0 con 0x5615f280f800 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f035a482740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03501af710 con 0x5615f2804f40 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f035a482740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f03501ae970 con 0x5615f2804620 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f035a482740 10 client.0 _ll_register_callbacks cb 0x7f034007ff50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f0355d05640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f280f800 0x5615f2815ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f0355504640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f2804620 0x5615f2815450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f0354d03640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804f40 0x5615f2815990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f0355d05640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f280f800 0x5615f2815ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42600/0 (socket says 172.21.15.50:42600) 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f0355504640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f2804620 0x5615f2815450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:50752/0 (socket says 172.21.15.50:50752) 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f0355d05640 1 -- 172.21.15.50:0/4271573310 learned_addr learned my addr 172.21.15.50:0/4271573310 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f0355d05640 1 -- 172.21.15.50:0/4271573310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f2804620 msgr2=0x5615f2815450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f0355d05640 1 --2- 172.21.15.50:0/4271573310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f2804620 0x5615f2815450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f0355d05640 1 -- 172.21.15.50:0/4271573310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804f40 msgr2=0x5615f2815990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f0355d05640 1 --2- 172.21.15.50:0/4271573310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804f40 0x5615f2815990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f0355d05640 1 -- 172.21.15.50:0/4271573310 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615f2885070 con 0x5615f280f800 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.933+0000 7f0354d03640 1 --2- 172.21.15.50:0/4271573310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804f40 0x5615f2815990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f0355d05640 1 --2- 172.21.15.50:0/4271573310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f280f800 0x5615f2815ed0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0348013f20 tx=0x7f0348013340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a440c82ea46d08 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f0338ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:20:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f0338ff9640 1 -- 172.21.15.50:0/4271573310 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034802c070 con 0x5615f280f800 2024-04-23T18:20:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f0338ff9640 1 -- 172.21.15.50:0/4271573310 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f0348011440 con 0x5615f280f800 2024-04-23T18:20:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f035a482740 1 -- 172.21.15.50:0/4271573310 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f034c04b070 con 0x5615f280f800 2024-04-23T18:20:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f0338ff9640 1 -- 172.21.15.50:0/4271573310 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0348023040 con 0x5615f280f800 2024-04-23T18:20:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f035a482740 10 client.15747 fetch_fsmap learned FSMap version 23 2024-04-23T18:20:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f035a482740 1 -- 172.21.15.50:0/4271573310 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5615f2886360 con 0x5615f280f800 2024-04-23T18:20:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f0338ff9640 1 -- 172.21.15.50:0/4271573310 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f034c04b070 con 0x5615f280f800 2024-04-23T18:20:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f0338ff9640 1 -- 172.21.15.50:0/4271573310 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f034801f420 con 0x5615f280f800 2024-04-23T18:20:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f035a482740 10 client.15747 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:20:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f035a482740 10 client.15747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f035a482740 20 client.15747 populate_metadata read hostname 'smithi050' 2024-04-23T18:20:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f035a482740 1 --2- 172.21.15.50:0/4271573310 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f28868a0 0x5615f2888cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f035a482740 4 client.15747 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.934+0000 7f035a482740 1 -- 172.21.15.50:0/4271573310 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5615f2771200 con 0x5615f28868a0 2024-04-23T18:20:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.935+0000 7f0355504640 1 --2- 172.21.15.50:0/4271573310 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f28868a0 0x5615f2888cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.935+0000 7f0355504640 1 --2- 172.21.15.50:0/4271573310 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f28868a0 0x5615f2888cc0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b82c3061d9973b9 server_cookie=bbb54795860aff72 in_seq=0 out_seq=0 2024-04-23T18:20:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.935+0000 7f0338ff9640 10 client.15747 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:04.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.989+0000 7f0338ff9640 1 -- 172.21.15.50:0/4271573310 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5615f2771200 con 0x5615f28868a0 2024-04-23T18:20:04.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.989+0000 7f0338ff9640 10 client.15747 handle_command_reply: tid=0 2024-04-23T18:20:04.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.989+0000 7f035a482740 1 client.15747 shutdown 2024-04-23T18:20:04.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f035a482740 1 -- 172.21.15.50:0/4271573310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f280f800 msgr2=0x5615f2815ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f035a482740 1 --2- 172.21.15.50:0/4271573310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f280f800 0x5615f2815ed0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0348013f20 tx=0x7f0348013340 comp rx=0 tx=0).stop 2024-04-23T18:20:04.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f035a482740 1 -- 172.21.15.50:0/4271573310 shutdown_connections 2024-04-23T18:20:04.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f035a482740 1 --2- 172.21.15.50:0/4271573310 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f28868a0 0x5615f2888cc0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f035a482740 1 --2- 172.21.15.50:0/4271573310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f280f800 0x5615f2815ed0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f035a482740 1 --2- 172.21.15.50:0/4271573310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f2804f40 0x5615f2815990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f035a482740 1 --2- 172.21.15.50:0/4271573310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f2804620 0x5615f2815450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f035a482740 1 -- 172.21.15.50:0/4271573310 >> 172.21.15.50:0/4271573310 conn(0x5615f2800b30 msgr2=0x5615f27ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f0338ff9640 0 client.15747 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f035a482740 1 -- 172.21.15.50:0/4271573310 shutdown_connections 2024-04-23T18:20:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f035a482740 1 -- 172.21.15.50:0/4271573310 wait complete. 2024-04-23T18:20:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f035a482740 20 client.15747 trim_cache size 0 max 16384 2024-04-23T18:20:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:20:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:20:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:20:04.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:20:04.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f0357768640 1 -- 172.21.15.50:0/3099491278 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f033c0a3df0 msgr2=0x7f033c0a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f0357768640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f033c0a3df0 0x7f033c0a62b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f03480099a0 tx=0x7f034801e000 comp rx=0 tx=0).stop 2024-04-23T18:20:04.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f0357768640 1 -- 172.21.15.50:0/3099491278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0350103b20 msgr2=0x7f0350101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:04.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.990+0000 7f0357768640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0350103b20 0x7f0350101390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f034000d6d0 tx=0x7f034000db90 comp rx=0 tx=0).stop 2024-04-23T18:20:04.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.991+0000 7f0357768640 1 -- 172.21.15.50:0/3099491278 shutdown_connections 2024-04-23T18:20:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.991+0000 7f0357768640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f033c0a3df0 0x7f033c0a62b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.991+0000 7f0357768640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f035010c540 0x7f03501018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.991+0000 7f0357768640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0350103b20 0x7f0350101390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.991+0000 7f0357768640 1 --2- 172.21.15.50:0/3099491278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0350103200 0x7f0350100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.991+0000 7f0357768640 1 -- 172.21.15.50:0/3099491278 >> 172.21.15.50:0/3099491278 conn(0x7f0350077fd0 msgr2=0x7f03500fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.991+0000 7f0357768640 1 -- 172.21.15.50:0/3099491278 shutdown_connections 2024-04-23T18:20:04.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:04.991+0000 7f0357768640 1 -- 172.21.15.50:0/3099491278 wait complete. 2024-04-23T18:20:05.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 21120 3.36 MB/sec warmup 39 sec latency 49.484 ms 2024-04-23T18:20:06.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 22983 3.52 MB/sec warmup 40 sec latency 19.045 ms 2024-04-23T18:20:06.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:06 smithi050 ceph-mon[29187]: pgmap v50: 97 pgs: 97 active+clean; 19 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 699 KiB/s wr, 122 op/s 2024-04-23T18:20:06.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:06 smithi177 ceph-mon[32308]: pgmap v50: 97 pgs: 97 active+clean; 19 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 699 KiB/s wr, 122 op/s 2024-04-23T18:20:06.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:06 smithi188 ceph-mon[32694]: pgmap v50: 97 pgs: 97 active+clean; 19 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 699 KiB/s wr, 122 op/s 2024-04-23T18:20:07.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 25463 3.78 MB/sec warmup 41 sec latency 10.220 ms 2024-04-23T18:20:08.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 28080 4.00 MB/sec warmup 42 sec latency 11.107 ms 2024-04-23T18:20:08.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:08 smithi050 ceph-mon[29187]: pgmap v51: 97 pgs: 97 active+clean; 21 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 272 op/s 2024-04-23T18:20:08.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:08 smithi177 ceph-mon[32308]: pgmap v51: 97 pgs: 97 active+clean; 21 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 272 op/s 2024-04-23T18:20:08.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:08 smithi188 ceph-mon[32694]: pgmap v51: 97 pgs: 97 active+clean; 21 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 272 op/s 2024-04-23T18:20:09.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 30039 4.17 MB/sec warmup 43 sec latency 20.109 ms 2024-04-23T18:20:10.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 32560 4.39 MB/sec warmup 44 sec latency 10.870 ms 2024-04-23T18:20:10.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:10 smithi050 ceph-mon[29187]: pgmap v52: 97 pgs: 97 active+clean; 21 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 267 op/s 2024-04-23T18:20:10.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:10 smithi177 ceph-mon[32308]: pgmap v52: 97 pgs: 97 active+clean; 21 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 267 op/s 2024-04-23T18:20:10.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:10 smithi188 ceph-mon[32694]: pgmap v52: 97 pgs: 97 active+clean; 21 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 267 op/s 2024-04-23T18:20:11.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 35322 4.61 MB/sec warmup 45 sec latency 12.891 ms 2024-04-23T18:20:12.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 37284 4.73 MB/sec warmup 46 sec latency 10.813 ms 2024-04-23T18:20:12.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:12 smithi050 ceph-mon[29187]: pgmap v53: 97 pgs: 97 active+clean; 21 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 267 op/s 2024-04-23T18:20:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:12 smithi177 ceph-mon[32308]: pgmap v53: 97 pgs: 97 active+clean; 21 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 267 op/s 2024-04-23T18:20:12.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:12 smithi188 ceph-mon[32694]: pgmap v53: 97 pgs: 97 active+clean; 21 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 267 op/s 2024-04-23T18:20:13.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 39777 4.93 MB/sec warmup 47 sec latency 16.296 ms 2024-04-23T18:20:14.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 42425 5.12 MB/sec warmup 48 sec latency 17.300 ms 2024-04-23T18:20:14.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:14 smithi050 ceph-mon[29187]: pgmap v54: 97 pgs: 97 active+clean; 27 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 367 op/s 2024-04-23T18:20:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:14 smithi177 ceph-mon[32308]: pgmap v54: 97 pgs: 97 active+clean; 27 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 367 op/s 2024-04-23T18:20:14.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:14 smithi188 ceph-mon[32694]: pgmap v54: 97 pgs: 97 active+clean; 27 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 367 op/s 2024-04-23T18:20:15.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 44458 5.22 MB/sec warmup 49 sec latency 9.637 ms 2024-04-23T18:20:16.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 46767 5.39 MB/sec warmup 50 sec latency 14.209 ms 2024-04-23T18:20:16.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:16 smithi050 ceph-mon[29187]: pgmap v55: 97 pgs: 97 active+clean; 27 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 250 op/s 2024-04-23T18:20:16.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:16 smithi177 ceph-mon[32308]: pgmap v55: 97 pgs: 97 active+clean; 27 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 250 op/s 2024-04-23T18:20:16.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:16 smithi188 ceph-mon[32694]: pgmap v55: 97 pgs: 97 active+clean; 27 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 250 op/s 2024-04-23T18:20:17.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 49514 5.57 MB/sec warmup 51 sec latency 11.531 ms 2024-04-23T18:20:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.081+0000 7f474204e640 1 -- 172.21.15.50:0/357447965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f473c1072a0 msgr2=0x7f473c10ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.081+0000 7f474204e640 1 --2- 172.21.15.50:0/357447965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f473c1072a0 0x7f473c10ee70 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4734009be0 tx=0x7f473402a6c0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.082+0000 7f474204e640 1 -- 172.21.15.50:0/357447965 shutdown_connections 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.082+0000 7f474204e640 1 --2- 172.21.15.50:0/357447965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f473c1072a0 0x7f473c10ee70 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.082+0000 7f474204e640 1 --2- 172.21.15.50:0/357447965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f473c102b80 0x7f473c106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.082+0000 7f474204e640 1 --2- 172.21.15.50:0/357447965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f473c1021d0 0x7f473c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.082+0000 7f474204e640 1 -- 172.21.15.50:0/357447965 >> 172.21.15.50:0/357447965 conn(0x7f473c0793e0 msgr2=0x7f473c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.082+0000 7f474204e640 1 -- 172.21.15.50:0/357447965 shutdown_connections 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.082+0000 7f474204e640 1 -- 172.21.15.50:0/357447965 wait complete. 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.082+0000 7f474204e640 1 Processor -- start 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.082+0000 7f474204e640 1 -- start start 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f474204e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f473c1021d0 0x7f473c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f474204e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f473c1072a0 0x7f473c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f474204e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f473c19cf00 0x7f473c19d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f474204e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f473c112210 con 0x7f473c1072a0 2024-04-23T18:20:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f474204e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f473c112090 con 0x7f473c19cf00 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f474204e640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f473c112390 con 0x7f473c1021d0 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f473affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f473c1072a0 0x7f473c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f473affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f473c1072a0 0x7f473c198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46392/0 (socket says 172.21.15.50:46392) 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f473b7fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f473c1021d0 0x7f473c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f473affd640 1 -- 172.21.15.50:0/538102112 learned_addr learned my addr 172.21.15.50:0/538102112 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f473bfff640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f473c19cf00 0x7f473c19d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f473affd640 1 -- 172.21.15.50:0/538102112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f473c1021d0 msgr2=0x7f473c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f473affd640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f473c1021d0 0x7f473c198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f473affd640 1 -- 172.21.15.50:0/538102112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f473c19cf00 msgr2=0x7f473c19d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f473affd640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f473c19cf00 0x7f473c19d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f473affd640 1 -- 172.21.15.50:0/538102112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f473c19d9f0 con 0x7f473c1072a0 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f473affd640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f473c1072a0 0x7f473c198bd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f473000e9f0 tx=0x7f473000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f4738ff9640 1 -- 172.21.15.50:0/538102112 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473000ccc0 con 0x7f473c1072a0 2024-04-23T18:20:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f4738ff9640 1 -- 172.21.15.50:0/538102112 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47300054d0 con 0x7f473c1072a0 2024-04-23T18:20:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f474204e640 1 -- 172.21.15.50:0/538102112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f473c19dce0 con 0x7f473c1072a0 2024-04-23T18:20:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f4738ff9640 1 -- 172.21.15.50:0/538102112 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4730010430 con 0x7f473c1072a0 2024-04-23T18:20:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.083+0000 7f474204e640 1 -- 172.21.15.50:0/538102112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f473c1a54d0 con 0x7f473c1072a0 2024-04-23T18:20:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.086+0000 7f4738ff9640 1 -- 172.21.15.50:0/538102112 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f47300046c0 con 0x7f473c1072a0 2024-04-23T18:20:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f4744d64740 1 Processor -- start 2024-04-23T18:20:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f4744d64740 1 -- start start 2024-04-23T18:20:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f4744d64740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c21103c620 0x55c21103ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f4738ff9640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f47240a3e40 0x7f47240a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f4744d64740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103cf40 0x55c2110472c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f4744d64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c211047800 0x55c211049bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f4744d64740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c210fa1970 con 0x55c21103cf40 2024-04-23T18:20:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f4744d64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c210f3e4a0 con 0x55c211047800 2024-04-23T18:20:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f4744d64740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c210f9d930 con 0x55c21103c620 2024-04-23T18:20:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f473bfff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c21103c620 0x55c21103ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f4738ff9640 1 -- 172.21.15.50:0/538102112 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f47300d0a50 con 0x7f473c1072a0 2024-04-23T18:20:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f473affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103cf40 0x55c2110472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f473bfff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c21103c620 0x55c21103ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:60244/0 (socket says 172.21.15.50:60244) 2024-04-23T18:20:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f473bfff640 1 -- 172.21.15.50:0/3699791487 learned_addr learned my addr 172.21.15.50:0/3699791487 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.087+0000 7f473bfff640 1 --2- 172.21.15.50:0/3699791487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c211047800 0x55c211049bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.088+0000 7f473b7fe640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f47240a3e40 0x7f47240a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.088+0000 7f473bfff640 1 -- 172.21.15.50:0/3699791487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c211047800 msgr2=0x55c211049bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.088+0000 7f473bfff640 1 --2- 172.21.15.50:0/3699791487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c211047800 0x55c211049bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.088+0000 7f473bfff640 1 -- 172.21.15.50:0/3699791487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103cf40 msgr2=0x55c2110472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.088+0000 7f473bfff640 1 --2- 172.21.15.50:0/3699791487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103cf40 0x55c2110472c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.088+0000 7f473bfff640 1 -- 172.21.15.50:0/3699791487 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c210cb4210 con 0x55c21103c620 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.088+0000 7f473affd640 1 --2- 172.21.15.50:0/3699791487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103cf40 0x55c2110472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.088+0000 7f473b7fe640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f47240a3e40 0x7f47240a6300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f472c005fd0 tx=0x7f472c009260 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.089+0000 7f473bfff640 1 --2- 172.21.15.50:0/3699791487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c21103c620 0x55c21103ca00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4734005c60 tx=0x7f4734005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dac65e84bfcd564 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.089+0000 7f4721ffb640 1 -- 172.21.15.50:0/3699791487 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4734036040 con 0x55c21103c620 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.089+0000 7f4721ffb640 1 -- 172.21.15.50:0/3699791487 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4734030e20 con 0x55c21103c620 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.089+0000 7f4721ffb640 1 -- 172.21.15.50:0/3699791487 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4734004320 con 0x55c21103c620 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 -- 172.21.15.50:0/3699791487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c21103c620 msgr2=0x55c21103ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 --2- 172.21.15.50:0/3699791487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c21103c620 0x55c21103ca00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4734005c60 tx=0x7f4734005bd0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 -- 172.21.15.50:0/3699791487 shutdown_connections 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 --2- 172.21.15.50:0/3699791487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c211047800 0x55c211049bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 --2- 172.21.15.50:0/3699791487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103cf40 0x55c2110472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 --2- 172.21.15.50:0/3699791487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c21103c620 0x55c21103ca00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.089 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 -- 172.21.15.50:0/3699791487 >> 172.21.15.50:0/3699791487 conn(0x55c211038b30 msgr2=0x55c211038f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:18.089 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 -- 172.21.15.50:0/3699791487 shutdown_connections 2024-04-23T18:20:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 -- 172.21.15.50:0/3699791487 wait complete. 2024-04-23T18:20:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 -- start start 2024-04-23T18:20:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 Processor -- start 2024-04-23T18:20:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c211047190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c21103cf40 0x55c211044480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.090+0000 7f4744d64740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c211047800 0x55c211044a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f4744d64740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c21104a3d0 con 0x55c21103c620 2024-04-23T18:20:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f4744d64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c210f3c0b0 con 0x55c21103cf40 2024-04-23T18:20:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f4744d64740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c21104a550 con 0x55c211047800 2024-04-23T18:20:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f4744d64740 10 client.0 _ll_register_callbacks cb 0x55c210f29d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c211047190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c211047190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46408/0 (socket says 172.21.15.50:46408) 2024-04-23T18:20:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473b7fe640 1 -- 172.21.15.50:0/1719449220 learned_addr learned my addr 172.21.15.50:0/1719449220 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473bfff640 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c211047800 0x55c211044a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473affd640 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c21103cf40 0x55c211044480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473b7fe640 1 -- 172.21.15.50:0/1719449220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c211047800 msgr2=0x55c211044a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473b7fe640 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c211047800 0x55c211044a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473b7fe640 1 -- 172.21.15.50:0/1719449220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c21103cf40 msgr2=0x55c211044480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:18.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473b7fe640 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c21103cf40 0x55c211044480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473b7fe640 1 -- 172.21.15.50:0/1719449220 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c210d80620 con 0x55c21103c620 2024-04-23T18:20:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473bfff640 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c211047800 0x55c211044a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473affd640 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c21103cf40 0x55c211044480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f473b7fe640 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c211047190 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f472c003970 tx=0x7f472c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684a9bd3d34ff8db server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.091+0000 7f471affd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:20:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.092+0000 7f471affd640 1 -- 172.21.15.50:0/1719449220 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472c00ecc0 con 0x55c21103c620 2024-04-23T18:20:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.092+0000 7f471affd640 1 -- 172.21.15.50:0/1719449220 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f472c01e420 con 0x55c21103c620 2024-04-23T18:20:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.092+0000 7f4744d64740 1 -- 172.21.15.50:0/1719449220 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4734030e20 con 0x55c21103c620 2024-04-23T18:20:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.093+0000 7f471affd640 1 -- 172.21.15.50:0/1719449220 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f472c00ee60 con 0x55c21103c620 2024-04-23T18:20:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.093+0000 7f4744d64740 10 client.15759 fetch_fsmap learned FSMap version 23 2024-04-23T18:20:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.093+0000 7f4744d64740 1 -- 172.21.15.50:0/1719449220 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c2110563d0 con 0x55c21103c620 2024-04-23T18:20:18.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.093+0000 7f471affd640 1 -- 172.21.15.50:0/1719449220 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4734030e20 con 0x55c21103c620 2024-04-23T18:20:18.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.093+0000 7f471affd640 1 -- 172.21.15.50:0/1719449220 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f472c0306b0 con 0x55c21103c620 2024-04-23T18:20:18.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.094+0000 7f4744d64740 10 client.15759 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:20:18.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.094+0000 7f4744d64740 10 client.15759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.094+0000 7f4744d64740 20 client.15759 populate_metadata read hostname 'smithi050' 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.094+0000 7f4744d64740 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c211057890 0x55c211059cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.094+0000 7f4744d64740 4 client.15759 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.094+0000 7f4744d64740 1 -- 172.21.15.50:0/1719449220 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c210faf390 con 0x55c211057890 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.094+0000 7f473affd640 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c211057890 0x55c211059cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.095+0000 7f471affd640 10 client.15759 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.095+0000 7f473affd640 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c211057890 0x55c211059cc0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6de9fc07819c69a8 server_cookie=7554a3743a1d9dc6 in_seq=0 out_seq=0 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.096+0000 7f471affd640 1 -- 172.21.15.50:0/1719449220 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c210faf390 con 0x55c211057890 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.096+0000 7f471affd640 10 client.15759 handle_command_reply: tid=0 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.096+0000 7f4744d64740 1 client.15759 shutdown 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.096+0000 7f4744d64740 1 -- 172.21.15.50:0/1719449220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 msgr2=0x55c211047190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.096+0000 7f4744d64740 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c211047190 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f472c003970 tx=0x7f472c004730 comp rx=0 tx=0).stop 2024-04-23T18:20:18.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.097+0000 7f4744d64740 1 -- 172.21.15.50:0/1719449220 shutdown_connections 2024-04-23T18:20:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.097+0000 7f4744d64740 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c211057890 0x55c211059cc0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.097+0000 7f4744d64740 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c211047800 0x55c211044a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.097+0000 7f4744d64740 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c21103cf40 0x55c211044480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.097+0000 7f4744d64740 1 --2- 172.21.15.50:0/1719449220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c211047190 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.097+0000 7f4744d64740 1 -- 172.21.15.50:0/1719449220 >> 172.21.15.50:0/1719449220 conn(0x55c211038b30 msgr2=0x55c21103e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.097+0000 7f471affd640 0 client.15759 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.097+0000 7f4744d64740 1 -- 172.21.15.50:0/1719449220 shutdown_connections 2024-04-23T18:20:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.097+0000 7f4744d64740 1 -- 172.21.15.50:0/1719449220 wait complete. 2024-04-23T18:20:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.097+0000 7f4744d64740 20 client.15759 trim_cache size 0 max 16384 2024-04-23T18:20:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.113+0000 7f4744d64740 1 Processor -- start 2024-04-23T18:20:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.113+0000 7f4744d64740 1 -- start start 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f4744d64740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c211053290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f4744d64740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c21103cf40 0x55c21103f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f4744d64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c211047800 0x55c211046cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f4744d64740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f473c1b02c0 con 0x55c21103c620 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f4744d64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f473c1adb90 con 0x55c211047800 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f4744d64740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f473c1ab9e0 con 0x55c21103cf40 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f473b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c211053290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f473b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c211053290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46422/0 (socket says 172.21.15.50:46422) 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f473b7fe640 1 -- 172.21.15.50:0/1163976390 learned_addr learned my addr 172.21.15.50:0/1163976390 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f473bfff640 1 --2- 172.21.15.50:0/1163976390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c211047800 0x55c211046cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f473affd640 1 --2- 172.21.15.50:0/1163976390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c21103cf40 0x55c21103f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f473b7fe640 1 -- 172.21.15.50:0/1163976390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c21103cf40 msgr2=0x55c21103f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f473b7fe640 1 --2- 172.21.15.50:0/1163976390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c21103cf40 0x55c21103f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f473b7fe640 1 -- 172.21.15.50:0/1163976390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c211047800 msgr2=0x55c211046cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f473b7fe640 1 --2- 172.21.15.50:0/1163976390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c211047800 0x55c211046cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f473b7fe640 1 -- 172.21.15.50:0/1163976390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c211043430 con 0x55c21103c620 2024-04-23T18:20:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.114+0000 7f473b7fe640 1 --2- 172.21.15.50:0/1163976390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c211053290 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f472c0118d0 tx=0x7f472c00e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8d7e3014794e243 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.115+0000 7f47217fa640 1 -- 172.21.15.50:0/1163976390 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472c035040 con 0x55c21103c620 2024-04-23T18:20:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.115+0000 7f47217fa640 1 -- 172.21.15.50:0/1163976390 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f472c0308d0 con 0x55c21103c620 2024-04-23T18:20:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.115+0000 7f47217fa640 1 -- 172.21.15.50:0/1163976390 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472c026750 con 0x55c21103c620 2024-04-23T18:20:18.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.115+0000 7f4744d64740 1 -- 172.21.15.50:0/1163976390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 msgr2=0x55c211053290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.115+0000 7f4744d64740 1 --2- 172.21.15.50:0/1163976390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c211053290 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f472c0118d0 tx=0x7f472c00e820 comp rx=0 tx=0).stop 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 -- 172.21.15.50:0/1163976390 shutdown_connections 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 --2- 172.21.15.50:0/1163976390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c211047800 0x55c211046cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 --2- 172.21.15.50:0/1163976390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c21103cf40 0x55c21103f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 --2- 172.21.15.50:0/1163976390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c211053290 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 -- 172.21.15.50:0/1163976390 >> 172.21.15.50:0/1163976390 conn(0x55c211038b30 msgr2=0x55c21103f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 -- 172.21.15.50:0/1163976390 shutdown_connections 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 -- 172.21.15.50:0/1163976390 wait complete. 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 -- start start 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 Processor -- start 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c21104f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c21103cf40 0x55c21104fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c211047800 0x55c211040bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f473c1ac050 con 0x55c21103c620 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f473c1ad4c0 con 0x55c21103cf40 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f473c1ac720 con 0x55c211047800 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.116+0000 7f4744d64740 10 client.0 _ll_register_callbacks cb 0x7f472c0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f473b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c21104f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f473b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c21104f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46438/0 (socket says 172.21.15.50:46438) 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f473affd640 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c21103cf40 0x55c21104fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f473b7fe640 1 -- 172.21.15.50:0/3992777304 learned_addr learned my addr 172.21.15.50:0/3992777304 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f473bfff640 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c211047800 0x55c211040bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f473affd640 1 -- 172.21.15.50:0/3992777304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c211047800 msgr2=0x55c211040bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f473affd640 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c211047800 0x55c211040bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f473affd640 1 -- 172.21.15.50:0/3992777304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 msgr2=0x55c21104f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f473affd640 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c21104f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f473affd640 1 -- 172.21.15.50:0/3992777304 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c211044d70 con 0x55c21103cf40 2024-04-23T18:20:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f473affd640 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c21103cf40 0x55c21104fce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47300ce1b0 tx=0x7f4730087040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd99d072a4243bfb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f471affd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:20:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f471affd640 1 -- 172.21.15.50:0/3992777304 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473009ca00 con 0x55c21103cf40 2024-04-23T18:20:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f471affd640 1 -- 172.21.15.50:0/3992777304 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f473009ad00 con 0x55c21103cf40 2024-04-23T18:20:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f4744d64740 1 -- 172.21.15.50:0/3992777304 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f472c0308d0 con 0x55c21103cf40 2024-04-23T18:20:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f471affd640 1 -- 172.21.15.50:0/3992777304 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f47300a0db0 con 0x55c21103cf40 2024-04-23T18:20:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f4744d64740 10 client.25385 fetch_fsmap learned FSMap version 23 2024-04-23T18:20:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.117+0000 7f4744d64740 1 -- 172.21.15.50:0/3992777304 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c2110bd8f0 con 0x55c21103cf40 2024-04-23T18:20:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.118+0000 7f471affd640 1 -- 172.21.15.50:0/3992777304 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f472c0308d0 con 0x55c21103cf40 2024-04-23T18:20:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.118+0000 7f471affd640 1 -- 172.21.15.50:0/3992777304 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f47300bbbc0 con 0x55c21103cf40 2024-04-23T18:20:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.118+0000 7f4744d64740 10 client.25385 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:20:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.118+0000 7f4744d64740 10 client.25385 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.118+0000 7f4744d64740 20 client.25385 populate_metadata read hostname 'smithi050' 2024-04-23T18:20:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.118+0000 7f4744d64740 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c2110be120 0x55c2110c0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.118+0000 7f4744d64740 4 client.25385 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.118+0000 7f4744d64740 1 -- 172.21.15.50:0/3992777304 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c210fa9200 con 0x55c2110be120 2024-04-23T18:20:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.118+0000 7f473b7fe640 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c2110be120 0x55c2110c0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.118+0000 7f471affd640 10 client.25385 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.118+0000 7f473b7fe640 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c2110be120 0x55c2110c0540 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f10abe242c8618e server_cookie=254b5bea57d1f931 in_seq=0 out_seq=0 2024-04-23T18:20:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f471affd640 1 -- 172.21.15.50:0/3992777304 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c210fa9200 con 0x55c2110be120 2024-04-23T18:20:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f471affd640 10 client.25385 handle_command_reply: tid=0 2024-04-23T18:20:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f4744d64740 1 client.25385 shutdown 2024-04-23T18:20:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f4744d64740 1 -- 172.21.15.50:0/3992777304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c21103cf40 msgr2=0x55c21104fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f4744d64740 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c21103cf40 0x55c21104fce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47300ce1b0 tx=0x7f4730087040 comp rx=0 tx=0).stop 2024-04-23T18:20:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f4744d64740 1 -- 172.21.15.50:0/3992777304 shutdown_connections 2024-04-23T18:20:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f4744d64740 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c2110be120 0x55c2110c0540 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f4744d64740 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c211047800 0x55c211040bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f4744d64740 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c21103cf40 0x55c21104fce0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f4744d64740 1 --2- 172.21.15.50:0/3992777304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c21103c620 0x55c21104f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f471affd640 0 client.25385 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f4744d64740 1 -- 172.21.15.50:0/3992777304 >> 172.21.15.50:0/3992777304 conn(0x55c211038b30 msgr2=0x55c21103e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f4744d64740 1 -- 172.21.15.50:0/3992777304 shutdown_connections 2024-04-23T18:20:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f4744d64740 1 -- 172.21.15.50:0/3992777304 wait complete. 2024-04-23T18:20:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.170+0000 7f4744d64740 20 client.25385 trim_cache size 0 max 16384 2024-04-23T18:20:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:20:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:20:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:20:18.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:20:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.171+0000 7f474204e640 1 -- 172.21.15.50:0/538102112 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f47240a3e40 msgr2=0x7f47240a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.171+0000 7f474204e640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f47240a3e40 0x7f47240a6300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f472c005fd0 tx=0x7f472c009260 comp rx=0 tx=0).stop 2024-04-23T18:20:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.171+0000 7f474204e640 1 -- 172.21.15.50:0/538102112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f473c1072a0 msgr2=0x7f473c198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.171+0000 7f474204e640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f473c1072a0 0x7f473c198bd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f473000e9f0 tx=0x7f473000edf0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.171+0000 7f474204e640 1 -- 172.21.15.50:0/538102112 shutdown_connections 2024-04-23T18:20:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.171+0000 7f474204e640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f47240a3e40 0x7f47240a6300 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.171+0000 7f474204e640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f473c19cf00 0x7f473c19d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.171+0000 7f474204e640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f473c1072a0 0x7f473c198bd0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.171+0000 7f474204e640 1 --2- 172.21.15.50:0/538102112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f473c1021d0 0x7f473c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.171+0000 7f474204e640 1 -- 172.21.15.50:0/538102112 >> 172.21.15.50:0/538102112 conn(0x7f473c0793e0 msgr2=0x7f473c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.171+0000 7f474204e640 1 -- 172.21.15.50:0/538102112 shutdown_connections 2024-04-23T18:20:18.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:18.171+0000 7f474204e640 1 -- 172.21.15.50:0/538102112 wait complete. 2024-04-23T18:20:18.193 INFO:tasks.workunit.client.0.smithi050.stdout: 1 51100 5.65 MB/sec warmup 52 sec latency 26.425 ms 2024-04-23T18:20:18.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:18 smithi050 ceph-mon[29187]: pgmap v56: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 353 op/s 2024-04-23T18:20:18.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:18 smithi177 ceph-mon[32308]: pgmap v56: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 353 op/s 2024-04-23T18:20:18.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:18 smithi188 ceph-mon[32694]: pgmap v56: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 353 op/s 2024-04-23T18:20:19.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 53526 5.80 MB/sec warmup 53 sec latency 75.531 ms 2024-04-23T18:20:20.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 55833 5.85 MB/sec warmup 54 sec latency 15.253 ms 2024-04-23T18:20:20.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:20 smithi050 ceph-mon[29187]: pgmap v57: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 203 op/s 2024-04-23T18:20:20.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:20 smithi177 ceph-mon[32308]: pgmap v57: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 203 op/s 2024-04-23T18:20:20.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:20 smithi188 ceph-mon[32694]: pgmap v57: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 203 op/s 2024-04-23T18:20:21.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 57694 5.96 MB/sec warmup 55 sec latency 60.590 ms 2024-04-23T18:20:22.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 60540 6.17 MB/sec warmup 56 sec latency 10.508 ms 2024-04-23T18:20:22.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:22 smithi050 ceph-mon[29187]: pgmap v58: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 203 op/s 2024-04-23T18:20:22.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:22 smithi177 ceph-mon[32308]: pgmap v58: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 203 op/s 2024-04-23T18:20:22.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:22 smithi188 ceph-mon[32694]: pgmap v58: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 203 op/s 2024-04-23T18:20:23.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 62733 6.19 MB/sec warmup 57 sec latency 12.123 ms 2024-04-23T18:20:24.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 64696 6.30 MB/sec warmup 58 sec latency 9.869 ms 2024-04-23T18:20:24.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:24 smithi050 ceph-mon[29187]: pgmap v59: 97 pgs: 97 active+clean; 31 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 342 op/s 2024-04-23T18:20:24.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:24 smithi177 ceph-mon[32308]: pgmap v59: 97 pgs: 97 active+clean; 31 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 342 op/s 2024-04-23T18:20:24.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:24 smithi188 ceph-mon[32694]: pgmap v59: 97 pgs: 97 active+clean; 31 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 342 op/s 2024-04-23T18:20:25.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 67563 6.49 MB/sec warmup 59 sec latency 12.129 ms 2024-04-23T18:20:26.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 69803 6.52 MB/sec warmup 60 sec latency 10.546 ms 2024-04-23T18:20:26.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:26 smithi050 ceph-mon[29187]: pgmap v60: 97 pgs: 97 active+clean; 31 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 242 op/s 2024-04-23T18:20:26.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:26 smithi177 ceph-mon[32308]: pgmap v60: 97 pgs: 97 active+clean; 31 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 242 op/s 2024-04-23T18:20:26.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:26 smithi188 ceph-mon[32694]: pgmap v60: 97 pgs: 97 active+clean; 31 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 242 op/s 2024-04-23T18:20:27.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 71789 6.62 MB/sec warmup 61 sec latency 12.116 ms 2024-04-23T18:20:28.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 74612 6.79 MB/sec warmup 62 sec latency 28.123 ms 2024-04-23T18:20:28.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:28 smithi050 ceph-mon[29187]: pgmap v61: 97 pgs: 97 active+clean; 37 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 347 op/s 2024-04-23T18:20:28.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:28 smithi177 ceph-mon[32308]: pgmap v61: 97 pgs: 97 active+clean; 37 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 347 op/s 2024-04-23T18:20:28.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:28 smithi188 ceph-mon[32694]: pgmap v61: 97 pgs: 97 active+clean; 37 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 347 op/s 2024-04-23T18:20:29.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 76236 6.79 MB/sec warmup 63 sec latency 14.455 ms 2024-04-23T18:20:30.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 78548 6.90 MB/sec warmup 64 sec latency 11.994 ms 2024-04-23T18:20:30.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:30 smithi050 ceph-mon[29187]: pgmap v62: 97 pgs: 97 active+clean; 37 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 244 op/s 2024-04-23T18:20:30.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:30 smithi177 ceph-mon[32308]: pgmap v62: 97 pgs: 97 active+clean; 37 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 244 op/s 2024-04-23T18:20:30.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:30 smithi188 ceph-mon[32694]: pgmap v62: 97 pgs: 97 active+clean; 37 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 244 op/s 2024-04-23T18:20:31.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 80852 6.96 MB/sec warmup 65 sec latency 14.689 ms 2024-04-23T18:20:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.264+0000 7f01b1243640 1 -- 172.21.15.50:0/1352012728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac1010f0 msgr2=0x7f01ac1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.264+0000 7f01b1243640 1 --2- 172.21.15.50:0/1352012728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac1010f0 0x7f01ac1014d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f019c009900 tx=0x7f019c02a410 comp rx=0 tx=0).stop 2024-04-23T18:20:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.264+0000 7f01b1243640 1 -- 172.21.15.50:0/1352012728 shutdown_connections 2024-04-23T18:20:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.264+0000 7f01b1243640 1 --2- 172.21.15.50:0/1352012728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01ac106170 0x7f01ac10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.264+0000 7f01b1243640 1 --2- 172.21.15.50:0/1352012728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01ac101aa0 0x7f01ac105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.264+0000 7f01b1243640 1 --2- 172.21.15.50:0/1352012728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac1010f0 0x7f01ac1014d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.264+0000 7f01b1243640 1 -- 172.21.15.50:0/1352012728 >> 172.21.15.50:0/1352012728 conn(0x7f01ac077fd0 msgr2=0x7f01ac0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.264+0000 7f01b1243640 1 -- 172.21.15.50:0/1352012728 shutdown_connections 2024-04-23T18:20:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.264+0000 7f01b1243640 1 -- 172.21.15.50:0/1352012728 wait complete. 2024-04-23T18:20:31.261 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01b1243640 1 Processor -- start 2024-04-23T18:20:31.261 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01b1243640 1 -- start start 2024-04-23T18:20:31.261 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01b1243640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01ac1010f0 0x7f01ac1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.261 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01b1243640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac106170 0x7f01ac1a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.261 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01b1243640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01ac19a6f0 0x7f01ac19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.261 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01b1243640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac111100 con 0x7f01ac106170 2024-04-23T18:20:31.261 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01b1243640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac110f80 con 0x7f01ac19a6f0 2024-04-23T18:20:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01b1243640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac111280 con 0x7f01ac1010f0 2024-04-23T18:20:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01a259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac106170 0x7f01ac1a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01aad9d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01ac1010f0 0x7f01ac1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01a259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac106170 0x7f01ac1a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50594/0 (socket says 172.21.15.50:50594) 2024-04-23T18:20:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01a259c640 1 -- 172.21.15.50:0/3474942056 learned_addr learned my addr 172.21.15.50:0/3474942056 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.265+0000 7f01ab59e640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01ac19a6f0 0x7f01ac19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01a259c640 1 -- 172.21.15.50:0/3474942056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01ac1010f0 msgr2=0x7f01ac1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01a259c640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01ac1010f0 0x7f01ac1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01a259c640 1 -- 172.21.15.50:0/3474942056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01ac19a6f0 msgr2=0x7f01ac19ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01a259c640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01ac19a6f0 0x7f01ac19ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01a259c640 1 -- 172.21.15.50:0/3474942056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ac19b2f0 con 0x7f01ac106170 2024-04-23T18:20:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01ab59e640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01ac19a6f0 0x7f01ac19ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01aad9d640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01ac1010f0 0x7f01ac1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01a259c640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac106170 0x7f01ac1a0b70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f0194005fd0 tx=0x7f019400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01a8d99640 1 -- 172.21.15.50:0/3474942056 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0194017070 con 0x7f01ac106170 2024-04-23T18:20:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01a8d99640 1 -- 172.21.15.50:0/3474942056 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0194009c30 con 0x7f01ac106170 2024-04-23T18:20:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01b1243640 1 -- 172.21.15.50:0/3474942056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01ac1a73d0 con 0x7f01ac106170 2024-04-23T18:20:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01a8d99640 1 -- 172.21.15.50:0/3474942056 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0194003d30 con 0x7f01ac106170 2024-04-23T18:20:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.266+0000 7f01b1243640 1 -- 172.21.15.50:0/3474942056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01ac1a7830 con 0x7f01ac106170 2024-04-23T18:20:31.269 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.268+0000 7f01a8d99640 1 -- 172.21.15.50:0/3474942056 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0194003ed0 con 0x7f01ac106170 2024-04-23T18:20:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01b4036740 1 Processor -- start 2024-04-23T18:20:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01b4036740 1 -- start start 2024-04-23T18:20:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01a8d99640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f01980a3e40 0x7f01980a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01a8d99640 1 -- 172.21.15.50:0/3474942056 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f01940d0370 con 0x7f01ac106170 2024-04-23T18:20:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01ab59e640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f01980a3e40 0x7f01980a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01b4036740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 0x559996374a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01b4036740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x55999637f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01b4036740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999637f800 0x559996381bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01b4036740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5599962d9970 con 0x559996374620 2024-04-23T18:20:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01b4036740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5599962764a0 con 0x55999637f800 2024-04-23T18:20:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01b4036740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5599962d5930 con 0x559996374f40 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01aad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 0x559996374a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01ab59e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999637f800 0x559996381bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01aad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 0x559996374a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50606/0 (socket says 172.21.15.50:50606) 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01aad9d640 1 -- 172.21.15.50:0/1920231319 learned_addr learned my addr 172.21.15.50:0/1920231319 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.269+0000 7f01a259c640 1 --2- 172.21.15.50:0/1920231319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x55999637f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.270+0000 7f01ab59e640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f01980a3e40 0x7f01980a6300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f01a4009fd0 tx=0x7f01a4011f60 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.270+0000 7f01ab59e640 1 -- 172.21.15.50:0/1920231319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 msgr2=0x55999637f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.270+0000 7f01ab59e640 1 --2- 172.21.15.50:0/1920231319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x55999637f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.270+0000 7f01ab59e640 1 -- 172.21.15.50:0/1920231319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 msgr2=0x559996374a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.270+0000 7f01ab59e640 1 --2- 172.21.15.50:0/1920231319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 0x559996374a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.270+0000 7f01ab59e640 1 -- 172.21.15.50:0/1920231319 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559995fec210 con 0x55999637f800 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.270+0000 7f01aad9d640 1 --2- 172.21.15.50:0/1920231319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 0x559996374a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.270+0000 7f01ab59e640 1 --2- 172.21.15.50:0/1920231319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999637f800 0x559996381bf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f01a400ff10 tx=0x7f01a401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6745c6060e8201e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.270+0000 7f01a159a640 1 -- 172.21.15.50:0/1920231319 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a401a040 con 0x55999637f800 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.270+0000 7f01a159a640 1 -- 172.21.15.50:0/1920231319 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01a4020450 con 0x55999637f800 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.270+0000 7f01a159a640 1 -- 172.21.15.50:0/1920231319 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a401f610 con 0x55999637f800 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.271+0000 7f01b4036740 1 -- 172.21.15.50:0/1920231319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999637f800 msgr2=0x559996381bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.271+0000 7f01b4036740 1 --2- 172.21.15.50:0/1920231319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999637f800 0x559996381bf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f01a400ff10 tx=0x7f01a401d000 comp rx=0 tx=0).stop 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.271+0000 7f01b4036740 1 -- 172.21.15.50:0/1920231319 shutdown_connections 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.271+0000 7f01b4036740 1 --2- 172.21.15.50:0/1920231319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999637f800 0x559996381bf0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.271+0000 7f01b4036740 1 --2- 172.21.15.50:0/1920231319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x55999637f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.271+0000 7f01b4036740 1 --2- 172.21.15.50:0/1920231319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 0x559996374a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.271+0000 7f01b4036740 1 -- 172.21.15.50:0/1920231319 >> 172.21.15.50:0/1920231319 conn(0x559996370b30 msgr2=0x559996370f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.271+0000 7f01b4036740 1 -- 172.21.15.50:0/1920231319 shutdown_connections 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.271+0000 7f01b4036740 1 -- 172.21.15.50:0/1920231319 wait complete. 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.271+0000 7f01b4036740 1 -- start start 2024-04-23T18:20:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.271+0000 7f01b4036740 1 Processor -- start 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01b4036740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559996374620 0x55999637f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01b4036740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x55999637c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01b4036740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55999637f800 0x55999637c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01b4036740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5599963823d0 con 0x55999637f800 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01b4036740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5599962740b0 con 0x559996374620 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01b4036740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559996382550 con 0x559996374f40 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01b4036740 10 client.0 _ll_register_callbacks cb 0x559996261d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01ab59e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55999637f800 0x55999637c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01a259c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x55999637c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01ab59e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55999637f800 0x55999637c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50618/0 (socket says 172.21.15.50:50618) 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01aad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559996374620 0x55999637f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01ab59e640 1 -- 172.21.15.50:0/3044389090 learned_addr learned my addr 172.21.15.50:0/3044389090 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01a259c640 1 -- 172.21.15.50:0/3044389090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559996374620 msgr2=0x55999637f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01a259c640 1 --2- 172.21.15.50:0/3044389090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559996374620 0x55999637f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01a259c640 1 -- 172.21.15.50:0/3044389090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55999637f800 msgr2=0x55999637c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01a259c640 1 --2- 172.21.15.50:0/3044389090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55999637f800 0x55999637c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01a259c640 1 -- 172.21.15.50:0/3044389090 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599960b8620 con 0x559996374f40 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01ab59e640 1 --2- 172.21.15.50:0/3044389090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55999637f800 0x55999637c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01a259c640 1 --2- 172.21.15.50:0/3044389090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x55999637c400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f01940cdad0 tx=0x7f019409c320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c584fd332605a4d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.272+0000 7f01927fc640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01927fc640 1 -- 172.21.15.50:0/3044389090 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01940a4f90 con 0x559996374f40 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01b4036740 1 -- 172.21.15.50:0/3044389090 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01a4020450 con 0x559996374f40 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01927fc640 1 -- 172.21.15.50:0/3044389090 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f01940cdd50 con 0x559996374f40 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01927fc640 1 -- 172.21.15.50:0/3044389090 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f01940bb170 con 0x559996374f40 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01b4036740 10 client.25471 fetch_fsmap learned FSMap version 23 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01b4036740 1 -- 172.21.15.50:0/3044389090 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55999638e340 con 0x559996374f40 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01927fc640 1 -- 172.21.15.50:0/3044389090 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01a4020450 con 0x559996374f40 2024-04-23T18:20:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01927fc640 1 -- 172.21.15.50:0/3044389090 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f019409b770 con 0x559996374f40 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01b4036740 10 client.25471 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01b4036740 10 client.25471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01b4036740 20 client.25471 populate_metadata read hostname 'smithi050' 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01b4036740 1 --2- 172.21.15.50:0/3044389090 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55999638f800 0x559996391bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01b4036740 4 client.25471 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01b4036740 1 -- 172.21.15.50:0/3044389090 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599962e7390 con 0x55999638f800 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.273+0000 7f01aad9d640 1 --2- 172.21.15.50:0/3044389090 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55999638f800 0x559996391bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.274+0000 7f01aad9d640 1 --2- 172.21.15.50:0/3044389090 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55999638f800 0x559996391bf0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cab505aaba3d4333 server_cookie=e6d32f74ca581414 in_seq=0 out_seq=0 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.274+0000 7f01927fc640 10 client.25471 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.274+0000 7f01927fc640 1 -- 172.21.15.50:0/3044389090 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5599962e7390 con 0x55999638f800 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.274+0000 7f01927fc640 10 client.25471 handle_command_reply: tid=0 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.274+0000 7f01b4036740 1 client.25471 shutdown 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.275+0000 7f01b4036740 1 -- 172.21.15.50:0/3044389090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 msgr2=0x55999637c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.275+0000 7f01b4036740 1 --2- 172.21.15.50:0/3044389090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x55999637c400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f01940cdad0 tx=0x7f019409c320 comp rx=0 tx=0).stop 2024-04-23T18:20:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.275+0000 7f01b4036740 1 -- 172.21.15.50:0/3044389090 shutdown_connections 2024-04-23T18:20:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.275+0000 7f01b4036740 1 --2- 172.21.15.50:0/3044389090 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55999638f800 0x559996391bf0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.275+0000 7f01b4036740 1 --2- 172.21.15.50:0/3044389090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55999637f800 0x55999637c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.275+0000 7f01b4036740 1 --2- 172.21.15.50:0/3044389090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x55999637c400 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.275+0000 7f01b4036740 1 --2- 172.21.15.50:0/3044389090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559996374620 0x55999637f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.275+0000 7f01b4036740 1 -- 172.21.15.50:0/3044389090 >> 172.21.15.50:0/3044389090 conn(0x559996370b30 msgr2=0x5599963768b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.275+0000 7f01927fc640 0 client.25471 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.275+0000 7f01b4036740 1 -- 172.21.15.50:0/3044389090 shutdown_connections 2024-04-23T18:20:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.275+0000 7f01b4036740 1 -- 172.21.15.50:0/3044389090 wait complete. 2024-04-23T18:20:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.275+0000 7f01b4036740 20 client.25471 trim_cache size 0 max 16384 2024-04-23T18:20:31.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.290+0000 7f01b4036740 1 Processor -- start 2024-04-23T18:20:31.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.290+0000 7f01b4036740 1 -- start start 2024-04-23T18:20:31.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.290+0000 7f01b4036740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 0x55999638b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.290+0000 7f01b4036740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x559996377850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.290+0000 7f01b4036740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999637f800 0x55999637ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.290+0000 7f01b4036740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac1b25d0 con 0x559996374620 2024-04-23T18:20:31.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.290+0000 7f01b4036740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac1afea0 con 0x55999637f800 2024-04-23T18:20:31.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.290+0000 7f01b4036740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac1adcf0 con 0x559996374f40 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01aad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 0x55999638b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01a259c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x559996377850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01aad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 0x55999638b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50628/0 (socket says 172.21.15.50:50628) 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01aad9d640 1 -- 172.21.15.50:0/3560569765 learned_addr learned my addr 172.21.15.50:0/3560569765 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01ab59e640 1 --2- 172.21.15.50:0/3560569765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999637f800 0x55999637ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01aad9d640 1 -- 172.21.15.50:0/3560569765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 msgr2=0x559996377850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01aad9d640 1 --2- 172.21.15.50:0/3560569765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x559996377850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01aad9d640 1 -- 172.21.15.50:0/3560569765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999637f800 msgr2=0x55999637ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01aad9d640 1 --2- 172.21.15.50:0/3560569765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999637f800 0x55999637ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01aad9d640 1 -- 172.21.15.50:0/3560569765 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55999636c2e0 con 0x559996374620 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01a259c640 1 --2- 172.21.15.50:0/3560569765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x559996377850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01aad9d640 1 --2- 172.21.15.50:0/3560569765 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 0x55999638b290 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f019c009230 tx=0x7f019c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2ef7dcce69d5b0e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01a0d99640 1 -- 172.21.15.50:0/3560569765 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c035040 con 0x559996374620 2024-04-23T18:20:31.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01a0d99640 1 -- 172.21.15.50:0/3560569765 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f019c04c070 con 0x559996374620 2024-04-23T18:20:31.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.291+0000 7f01a0d99640 1 -- 172.21.15.50:0/3560569765 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c004f00 con 0x559996374620 2024-04-23T18:20:31.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.292+0000 7f01b4036740 1 -- 172.21.15.50:0/3560569765 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 msgr2=0x55999638b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:31.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.292+0000 7f01b4036740 1 --2- 172.21.15.50:0/3560569765 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 0x55999638b290 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f019c009230 tx=0x7f019c009260 comp rx=0 tx=0).stop 2024-04-23T18:20:31.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.292+0000 7f01b4036740 1 -- 172.21.15.50:0/3560569765 shutdown_connections 2024-04-23T18:20:31.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.292+0000 7f01b4036740 1 --2- 172.21.15.50:0/3560569765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55999637f800 0x55999637ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.292+0000 7f01b4036740 1 --2- 172.21.15.50:0/3560569765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559996374f40 0x559996377850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.292+0000 7f01b4036740 1 --2- 172.21.15.50:0/3560569765 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374620 0x55999638b290 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.292+0000 7f01b4036740 1 -- 172.21.15.50:0/3560569765 >> 172.21.15.50:0/3560569765 conn(0x559996370b30 msgr2=0x559996377140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.292+0000 7f01b4036740 1 -- 172.21.15.50:0/3560569765 shutdown_connections 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.292+0000 7f01b4036740 1 -- 172.21.15.50:0/3560569765 wait complete. 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.292+0000 7f01b4036740 1 -- start start 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.292+0000 7f01b4036740 1 Processor -- start 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01b4036740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559996374620 0x55999637a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01b4036740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374f40 0x5599963877a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01b4036740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55999637f800 0x559996378670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01b4036740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac1ae360 con 0x559996374f40 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01b4036740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac1af7d0 con 0x559996374620 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01b4036740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac1aea30 con 0x55999637f800 2024-04-23T18:20:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01b4036740 10 client.0 _ll_register_callbacks cb 0x7f0194009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01a259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374f40 0x5599963877a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01a259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374f40 0x5599963877a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50632/0 (socket says 172.21.15.50:50632) 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01a259c640 1 -- 172.21.15.50:0/248556526 learned_addr learned my addr 172.21.15.50:0/248556526 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01aad9d640 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559996374620 0x55999637a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01ab59e640 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55999637f800 0x559996378670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01a259c640 1 -- 172.21.15.50:0/248556526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55999637f800 msgr2=0x559996378670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01a259c640 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55999637f800 0x559996378670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01a259c640 1 -- 172.21.15.50:0/248556526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559996374620 msgr2=0x55999637a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01a259c640 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559996374620 0x55999637a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01a259c640 1 -- 172.21.15.50:0/248556526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559996374d70 con 0x559996374f40 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01a259c640 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374f40 0x5599963877a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0194098780 tx=0x7f0194098ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea02b313066dfd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01927fc640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01927fc640 1 -- 172.21.15.50:0/248556526 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01940985c0 con 0x559996374f40 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01927fc640 1 -- 172.21.15.50:0/248556526 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f0194098160 con 0x559996374f40 2024-04-23T18:20:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.293+0000 7f01b4036740 1 -- 172.21.15.50:0/248556526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f019c04c070 con 0x559996374f40 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.294+0000 7f01927fc640 1 -- 172.21.15.50:0/248556526 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f01940a4f90 con 0x559996374f40 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.294+0000 7f01b4036740 10 client.15783 fetch_fsmap learned FSMap version 23 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.294+0000 7f01b4036740 1 -- 172.21.15.50:0/248556526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5599963f5280 con 0x559996374f40 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.294+0000 7f01927fc640 1 -- 172.21.15.50:0/248556526 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f019c04c070 con 0x559996374f40 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.294+0000 7f01927fc640 1 -- 172.21.15.50:0/248556526 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f019409b1e0 con 0x559996374f40 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.294+0000 7f01b4036740 10 client.15783 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.294+0000 7f01b4036740 10 client.15783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.294+0000 7f01b4036740 20 client.15783 populate_metadata read hostname 'smithi050' 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.294+0000 7f01b4036740 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5599963f5a00 0x5599963f7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.294+0000 7f01b4036740 4 client.15783 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.294+0000 7f01b4036740 1 -- 172.21.15.50:0/248556526 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5599962e1200 con 0x5599963f5a00 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.294+0000 7f01aad9d640 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5599963f5a00 0x5599963f7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.295+0000 7f01aad9d640 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5599963f5a00 0x5599963f7e20 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43d6869a62e502b2 server_cookie=6b88d3a499dc1199 in_seq=0 out_seq=0 2024-04-23T18:20:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.295+0000 7f01927fc640 10 client.15783 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01927fc640 1 -- 172.21.15.50:0/248556526 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5599962e1200 con 0x5599963f5a00 2024-04-23T18:20:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01927fc640 10 client.15783 handle_command_reply: tid=0 2024-04-23T18:20:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01b4036740 1 client.15783 shutdown 2024-04-23T18:20:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01b4036740 1 -- 172.21.15.50:0/248556526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374f40 msgr2=0x5599963877a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01b4036740 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374f40 0x5599963877a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0194098780 tx=0x7f0194098ba0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01b4036740 1 -- 172.21.15.50:0/248556526 shutdown_connections 2024-04-23T18:20:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01b4036740 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5599963f5a00 0x5599963f7e20 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01b4036740 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55999637f800 0x559996378670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01927fc640 0 client.15783 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01b4036740 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559996374f40 0x5599963877a0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01b4036740 1 --2- 172.21.15.50:0/248556526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559996374620 0x55999637a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01b4036740 1 -- 172.21.15.50:0/248556526 >> 172.21.15.50:0/248556526 conn(0x559996370b30 msgr2=0x559996376860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01b4036740 1 -- 172.21.15.50:0/248556526 shutdown_connections 2024-04-23T18:20:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01b4036740 1 -- 172.21.15.50:0/248556526 wait complete. 2024-04-23T18:20:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.332+0000 7f01b4036740 20 client.15783 trim_cache size 0 max 16384 2024-04-23T18:20:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:20:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:20:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:20:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:20:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.333+0000 7f01b1243640 1 -- 172.21.15.50:0/3474942056 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f01980a3e40 msgr2=0x7f01980a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.333+0000 7f01b1243640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f01980a3e40 0x7f01980a6300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f01a4009fd0 tx=0x7f01a4011f60 comp rx=0 tx=0).stop 2024-04-23T18:20:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.333+0000 7f01b1243640 1 -- 172.21.15.50:0/3474942056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac106170 msgr2=0x7f01ac1a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.333+0000 7f01b1243640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac106170 0x7f01ac1a0b70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f0194005fd0 tx=0x7f019400dde0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.333+0000 7f01b1243640 1 -- 172.21.15.50:0/3474942056 shutdown_connections 2024-04-23T18:20:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.333+0000 7f01b1243640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f01980a3e40 0x7f01980a6300 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.333+0000 7f01b1243640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01ac19a6f0 0x7f01ac19ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.333+0000 7f01b1243640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac106170 0x7f01ac1a0b70 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.333+0000 7f01b1243640 1 --2- 172.21.15.50:0/3474942056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01ac1010f0 0x7f01ac1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.333+0000 7f01b1243640 1 -- 172.21.15.50:0/3474942056 >> 172.21.15.50:0/3474942056 conn(0x7f01ac077fd0 msgr2=0x7f01ac0ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.333+0000 7f01b1243640 1 -- 172.21.15.50:0/3474942056 shutdown_connections 2024-04-23T18:20:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:31.333+0000 7f01b1243640 1 -- 172.21.15.50:0/3474942056 wait complete. 2024-04-23T18:20:32.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 82519 6.99 MB/sec warmup 66 sec latency 53.884 ms 2024-04-23T18:20:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:32 smithi050 ceph-mon[29187]: pgmap v63: 97 pgs: 97 active+clean; 37 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 244 op/s 2024-04-23T18:20:32.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:32 smithi177 ceph-mon[32308]: pgmap v63: 97 pgs: 97 active+clean; 37 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 244 op/s 2024-04-23T18:20:32.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:32 smithi188 ceph-mon[32694]: pgmap v63: 97 pgs: 97 active+clean; 37 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 244 op/s 2024-04-23T18:20:33.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 84549 7.06 MB/sec warmup 67 sec latency 128.560 ms 2024-04-23T18:20:34.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 86302 7.09 MB/sec warmup 68 sec latency 14.290 ms 2024-04-23T18:20:34.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:34 smithi050 ceph-mon[29187]: pgmap v64: 97 pgs: 97 active+clean; 44 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 382 op/s 2024-04-23T18:20:34.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:34 smithi177 ceph-mon[32308]: pgmap v64: 97 pgs: 97 active+clean; 44 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 382 op/s 2024-04-23T18:20:34.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:34 smithi188 ceph-mon[32694]: pgmap v64: 97 pgs: 97 active+clean; 44 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 382 op/s 2024-04-23T18:20:35.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 88818 7.21 MB/sec warmup 69 sec latency 10.930 ms 2024-04-23T18:20:36.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 90641 7.21 MB/sec warmup 70 sec latency 23.218 ms 2024-04-23T18:20:36.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:36 smithi050 ceph-mon[29187]: pgmap v65: 97 pgs: 97 active+clean; 44 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 243 op/s 2024-04-23T18:20:36.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:36 smithi177 ceph-mon[32308]: pgmap v65: 97 pgs: 97 active+clean; 44 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 243 op/s 2024-04-23T18:20:36.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:36 smithi188 ceph-mon[32694]: pgmap v65: 97 pgs: 97 active+clean; 44 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 243 op/s 2024-04-23T18:20:37.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 92754 7.30 MB/sec warmup 71 sec latency 13.314 ms 2024-04-23T18:20:37.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:37 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:20:37.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:37 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:20:37.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:37 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:20:38.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 94660 7.29 MB/sec warmup 72 sec latency 55.348 ms 2024-04-23T18:20:38.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:38 smithi177 ceph-mon[32308]: pgmap v66: 97 pgs: 97 active+clean; 48 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 376 op/s 2024-04-23T18:20:38.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:38 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:38.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:38 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:38.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:38 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:38.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:38 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:38.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:38 smithi188 ceph-mon[32694]: pgmap v66: 97 pgs: 97 active+clean; 48 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 376 op/s 2024-04-23T18:20:38.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:38 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:38.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:38 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:38.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:38 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:38.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:38 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:38.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:38 smithi050 ceph-mon[29187]: pgmap v66: 97 pgs: 97 active+clean; 48 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 376 op/s 2024-04-23T18:20:38.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:38 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:38.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:38 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:38.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:38 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:38.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:38 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:39.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 96493 7.36 MB/sec warmup 73 sec latency 16.785 ms 2024-04-23T18:20:39.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:39 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:20:39.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:39 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:20:39.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:39 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:39.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:39 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:20:39.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:39 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:20:39.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:39 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:39.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:39 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:20:39.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:39 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:20:39.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:39 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:20:40.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 98991 7.44 MB/sec warmup 74 sec latency 12.292 ms 2024-04-23T18:20:40.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:40 smithi177 ceph-mon[32308]: pgmap v67: 97 pgs: 97 active+clean; 48 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 271 op/s 2024-04-23T18:20:40.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:40 smithi188 ceph-mon[32694]: pgmap v67: 97 pgs: 97 active+clean; 48 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 271 op/s 2024-04-23T18:20:40.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:40 smithi050 ceph-mon[29187]: pgmap v67: 97 pgs: 97 active+clean; 48 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 271 op/s 2024-04-23T18:20:41.194 INFO:tasks.workunit.client.0.smithi050.stdout: 1 100764 7.49 MB/sec warmup 75 sec latency 18.849 ms 2024-04-23T18:20:42.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 103292 7.57 MB/sec warmup 76 sec latency 13.948 ms 2024-04-23T18:20:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:42 smithi177 ceph-mon[32308]: pgmap v68: 97 pgs: 97 active+clean; 48 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 271 op/s 2024-04-23T18:20:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:42 smithi188 ceph-mon[32694]: pgmap v68: 97 pgs: 97 active+clean; 48 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 271 op/s 2024-04-23T18:20:42.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:42 smithi050 ceph-mon[29187]: pgmap v68: 97 pgs: 97 active+clean; 48 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 271 op/s 2024-04-23T18:20:43.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 105306 7.57 MB/sec warmup 77 sec latency 39.300 ms 2024-04-23T18:20:44.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 107231 7.63 MB/sec warmup 78 sec latency 12.701 ms 2024-04-23T18:20:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.424+0000 7fca48736640 1 -- 172.21.15.50:0/3388707279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca40102b70 msgr2=0x7fca40106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.424+0000 7fca48736640 1 --2- 172.21.15.50:0/3388707279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca40102b70 0x7fca40106bb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fca30009900 tx=0x7fca3002a410 comp rx=0 tx=0).stop 2024-04-23T18:20:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.424+0000 7fca48736640 1 -- 172.21.15.50:0/3388707279 shutdown_connections 2024-04-23T18:20:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.424+0000 7fca48736640 1 --2- 172.21.15.50:0/3388707279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca40107290 0x7fca4010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.424+0000 7fca48736640 1 --2- 172.21.15.50:0/3388707279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca40102b70 0x7fca40106bb0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.424+0000 7fca48736640 1 --2- 172.21.15.50:0/3388707279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca401021c0 0x7fca401025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.424+0000 7fca48736640 1 -- 172.21.15.50:0/3388707279 >> 172.21.15.50:0/3388707279 conn(0x7fca400fda20 msgr2=0x7fca400ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.424+0000 7fca48736640 1 -- 172.21.15.50:0/3388707279 shutdown_connections 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.424+0000 7fca48736640 1 -- 172.21.15.50:0/3388707279 wait complete. 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca48736640 1 Processor -- start 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca48736640 1 -- start start 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca48736640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca401021c0 0x7fca40198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca48736640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca40107290 0x7fca40198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca48736640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca4019cf50 0x7fca4019d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca48736640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fca401121b0 con 0x7fca4019cf50 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca48736640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca40112030 con 0x7fca401021c0 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca48736640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fca40112330 con 0x7fca40107290 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca46cd3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca4019cf50 0x7fca4019d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca45cd1640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca40107290 0x7fca40198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca46cd3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca4019cf50 0x7fca4019d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41774/0 (socket says 172.21.15.50:41774) 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca46cd3640 1 -- 172.21.15.50:0/3854679127 learned_addr learned my addr 172.21.15.50:0/3854679127 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca45cd1640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca40107290 0x7fca40198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:35238/0 (socket says 172.21.15.50:35238) 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca464d2640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca401021c0 0x7fca40198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca46cd3640 1 -- 172.21.15.50:0/3854679127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca40107290 msgr2=0x7fca40198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca46cd3640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca40107290 0x7fca40198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca46cd3640 1 -- 172.21.15.50:0/3854679127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca401021c0 msgr2=0x7fca40198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca46cd3640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca401021c0 0x7fca40198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.425+0000 7fca46cd3640 1 -- 172.21.15.50:0/3854679127 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca4019da40 con 0x7fca4019cf50 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.426+0000 7fca46cd3640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca4019cf50 0x7fca4019d400 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fca3800eb10 tx=0x7fca3800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.426+0000 7fca377fe640 1 -- 172.21.15.50:0/3854679127 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38004b20 con 0x7fca4019cf50 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.426+0000 7fca377fe640 1 -- 172.21.15.50:0/3854679127 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca38004cc0 con 0x7fca4019cf50 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.426+0000 7fca48736640 1 -- 172.21.15.50:0/3854679127 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca4019dd30 con 0x7fca4019cf50 2024-04-23T18:20:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.426+0000 7fca377fe640 1 -- 172.21.15.50:0/3854679127 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38010670 con 0x7fca4019cf50 2024-04-23T18:20:44.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.426+0000 7fca48736640 1 -- 172.21.15.50:0/3854679127 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca401a54f0 con 0x7fca4019cf50 2024-04-23T18:20:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.428+0000 7fca377fe640 1 -- 172.21.15.50:0/3854679127 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fca38005020 con 0x7fca4019cf50 2024-04-23T18:20:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.428+0000 7fca4b454740 1 Processor -- start 2024-04-23T18:20:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.428+0000 7fca4b454740 1 -- start start 2024-04-23T18:20:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca4b454740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 0x55b9768f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca4b454740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 0x55b9768fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca4b454740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768fc800 0x55b9768febf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca4b454740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b976856970 con 0x55b9768fc800 2024-04-23T18:20:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca4b454740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b9767f34a0 con 0x55b9768f1620 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca4b454740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b976852930 con 0x55b9768f1f40 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca377fe640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca2c0a3e40 0x7fca2c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca45cd1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 0x55b9768f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca464d2640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 0x55b9768fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca45cd1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 0x55b9768f1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:38866/0 (socket says 172.21.15.50:38866) 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca464d2640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 0x55b9768fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:35244/0 (socket says 172.21.15.50:35244) 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca464d2640 1 -- 172.21.15.50:0/1535856386 learned_addr learned my addr 172.21.15.50:0/1535856386 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca46cd3640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca2c0a3e40 0x7fca2c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca45cd1640 1 --2- 172.21.15.50:0/1535856386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768fc800 0x55b9768febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca377fe640 1 -- 172.21.15.50:0/3854679127 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fca380d0cf0 con 0x7fca4019cf50 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca464d2640 1 -- 172.21.15.50:0/1535856386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 msgr2=0x55b9768f1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca464d2640 1 --2- 172.21.15.50:0/1535856386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 0x55b9768f1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca464d2640 1 -- 172.21.15.50:0/1535856386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768fc800 msgr2=0x55b9768febf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca464d2640 1 --2- 172.21.15.50:0/1535856386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768fc800 0x55b9768febf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca464d2640 1 -- 172.21.15.50:0/1535856386 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b976569210 con 0x55b9768f1f40 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca46cd3640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca2c0a3e40 0x7fca2c0a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fca3809c940 tx=0x7fca3806f730 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.429+0000 7fca464d2640 1 --2- 172.21.15.50:0/1535856386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 0x55b9768fc2c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009cf0 tx=0x7fca3c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=844b77cedc21a06f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.430+0000 7fca34ff9640 1 -- 172.21.15.50:0/1535856386 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3c01e070 con 0x55b9768f1f40 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.430+0000 7fca34ff9640 1 -- 172.21.15.50:0/1535856386 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca3c009de0 con 0x55b9768f1f40 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.430+0000 7fca34ff9640 1 -- 172.21.15.50:0/1535856386 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3c019450 con 0x55b9768f1f40 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.430+0000 7fca4b454740 1 -- 172.21.15.50:0/1535856386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 msgr2=0x55b9768fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.430+0000 7fca4b454740 1 --2- 172.21.15.50:0/1535856386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 0x55b9768fc2c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009cf0 tx=0x7fca3c011810 comp rx=0 tx=0).stop 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.430+0000 7fca4b454740 1 -- 172.21.15.50:0/1535856386 shutdown_connections 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.430+0000 7fca4b454740 1 --2- 172.21.15.50:0/1535856386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768fc800 0x55b9768febf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.430+0000 7fca4b454740 1 --2- 172.21.15.50:0/1535856386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 0x55b9768fc2c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.430+0000 7fca4b454740 1 --2- 172.21.15.50:0/1535856386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 0x55b9768f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.430+0000 7fca4b454740 1 -- 172.21.15.50:0/1535856386 >> 172.21.15.50:0/1535856386 conn(0x55b9768edb30 msgr2=0x55b9768edf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.430+0000 7fca4b454740 1 -- 172.21.15.50:0/1535856386 shutdown_connections 2024-04-23T18:20:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.430+0000 7fca4b454740 1 -- 172.21.15.50:0/1535856386 wait complete. 2024-04-23T18:20:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca4b454740 1 -- start start 2024-04-23T18:20:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca4b454740 1 Processor -- start 2024-04-23T18:20:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca4b454740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1620 0x55b9768fc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca4b454740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 0x55b9768f9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca4b454740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768fc800 0x55b9768f9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca4b454740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b9768ff430 con 0x55b9768f1620 2024-04-23T18:20:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca4b454740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b9767f10b0 con 0x55b9768fc800 2024-04-23T18:20:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca4b454740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b9768ff5b0 con 0x55b9768f1f40 2024-04-23T18:20:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca4b454740 10 client.0 _ll_register_callbacks cb 0x55b9767ded30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca45cd1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1620 0x55b9768fc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca464d2640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 0x55b9768f9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca464d2640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 0x55b9768f9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:35246/0 (socket says 172.21.15.50:35246) 2024-04-23T18:20:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca464d2640 1 -- 172.21.15.50:0/213872339 learned_addr learned my addr 172.21.15.50:0/213872339 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca45cd1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1620 0x55b9768fc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41796/0 (socket says 172.21.15.50:41796) 2024-04-23T18:20:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.431+0000 7fca45cd1640 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768fc800 0x55b9768f9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca464d2640 1 -- 172.21.15.50:0/213872339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768fc800 msgr2=0x55b9768f9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca464d2640 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768fc800 0x55b9768f9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca464d2640 1 -- 172.21.15.50:0/213872339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1620 msgr2=0x55b9768fc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca464d2640 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1620 0x55b9768fc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca464d2640 1 -- 172.21.15.50:0/213872339 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b976635620 con 0x55b9768f1f40 2024-04-23T18:20:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca45cd1640 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1620 0x55b9768fc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca464d2640 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 0x55b9768f9480 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fca3c000c00 tx=0x7fca3c0052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ec0b2343ee52458 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:20:44.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca29ffb640 1 -- 172.21.15.50:0/213872339 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3c01e040 con 0x55b9768f1f40 2024-04-23T18:20:44.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca29ffb640 1 -- 172.21.15.50:0/213872339 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fca3c005510 con 0x55b9768f1f40 2024-04-23T18:20:44.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca4b454740 1 -- 172.21.15.50:0/213872339 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca3c009de0 con 0x55b9768f1f40 2024-04-23T18:20:44.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca29ffb640 1 -- 172.21.15.50:0/213872339 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fca3c030050 con 0x55b9768f1f40 2024-04-23T18:20:44.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca4b454740 10 client.25489 fetch_fsmap learned FSMap version 23 2024-04-23T18:20:44.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca4b454740 1 -- 172.21.15.50:0/213872339 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b97690b380 con 0x55b9768f1f40 2024-04-23T18:20:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.432+0000 7fca29ffb640 1 -- 172.21.15.50:0/213872339 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca3c009de0 con 0x55b9768f1f40 2024-04-23T18:20:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.433+0000 7fca29ffb640 1 -- 172.21.15.50:0/213872339 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fca3c021070 con 0x55b9768f1f40 2024-04-23T18:20:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.433+0000 7fca4b454740 10 client.25489 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:20:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.433+0000 7fca4b454740 10 client.25489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.433+0000 7fca4b454740 20 client.25489 populate_metadata read hostname 'smithi050' 2024-04-23T18:20:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.433+0000 7fca4b454740 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b97690c780 0x55b97690eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.433+0000 7fca4b454740 4 client.25489 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.433+0000 7fca4b454740 1 -- 172.21.15.50:0/213872339 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b976864390 con 0x55b97690c780 2024-04-23T18:20:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.433+0000 7fca45cd1640 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b97690c780 0x55b97690eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.433+0000 7fca45cd1640 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b97690c780 0x55b97690eb70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=677d283f75c7d341 server_cookie=d2d18a1b2a41258a in_seq=0 out_seq=0 2024-04-23T18:20:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.433+0000 7fca29ffb640 10 client.25489 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca29ffb640 1 -- 172.21.15.50:0/213872339 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b976864390 con 0x55b97690c780 2024-04-23T18:20:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca29ffb640 10 client.25489 handle_command_reply: tid=0 2024-04-23T18:20:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca4b454740 1 client.25489 shutdown 2024-04-23T18:20:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca4b454740 1 -- 172.21.15.50:0/213872339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 msgr2=0x55b9768f9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca4b454740 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 0x55b9768f9480 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fca3c000c00 tx=0x7fca3c0052f0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca4b454740 1 -- 172.21.15.50:0/213872339 shutdown_connections 2024-04-23T18:20:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca4b454740 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b97690c780 0x55b97690eb70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca4b454740 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768fc800 0x55b9768f9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca4b454740 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1f40 0x55b9768f9480 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca29ffb640 0 client.25489 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:44.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca4b454740 1 --2- 172.21.15.50:0/213872339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1620 0x55b9768fc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca4b454740 1 -- 172.21.15.50:0/213872339 >> 172.21.15.50:0/213872339 conn(0x55b9768edb30 msgr2=0x55b9768eccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:44.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca4b454740 1 -- 172.21.15.50:0/213872339 shutdown_connections 2024-04-23T18:20:44.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca4b454740 1 -- 172.21.15.50:0/213872339 wait complete. 2024-04-23T18:20:44.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.434+0000 7fca4b454740 20 client.25489 trim_cache size 0 max 16384 2024-04-23T18:20:44.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.449+0000 7fca4b454740 1 Processor -- start 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.449+0000 7fca4b454740 1 -- start start 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.449+0000 7fca4b454740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768edb30 0x55b9769082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.449+0000 7fca4b454740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1620 0x55b9768ecb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.449+0000 7fca4b454740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1f40 0x55b9768fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.449+0000 7fca4b454740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fca401b03a0 con 0x55b9768f1f40 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.449+0000 7fca4b454740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca401adc70 con 0x55b9768edb30 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.449+0000 7fca4b454740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fca401abac0 con 0x55b9768f1620 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca45cd1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768edb30 0x55b9769082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca45cd1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768edb30 0x55b9769082f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:38880/0 (socket says 172.21.15.50:38880) 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca45cd1640 1 -- 172.21.15.50:0/2282017389 learned_addr learned my addr 172.21.15.50:0/2282017389 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca464d2640 1 --2- 172.21.15.50:0/2282017389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1620 0x55b9768ecb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca45cd1640 1 --2- 172.21.15.50:0/2282017389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1f40 0x55b9768fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca45cd1640 1 -- 172.21.15.50:0/2282017389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1620 msgr2=0x55b9768ecb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca45cd1640 1 --2- 172.21.15.50:0/2282017389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1620 0x55b9768ecb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca45cd1640 1 -- 172.21.15.50:0/2282017389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1f40 msgr2=0x55b9768fbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca45cd1640 1 --2- 172.21.15.50:0/2282017389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1f40 0x55b9768fbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca45cd1640 1 -- 172.21.15.50:0/2282017389 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9768f9d30 con 0x55b9768edb30 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca45cd1640 1 --2- 172.21.15.50:0/2282017389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768edb30 0x55b9769082f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fca4006b5f0 tx=0x7fca30031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1934575d1208b83 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca2bfff640 1 -- 172.21.15.50:0/2282017389 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca30052070 con 0x55b9768edb30 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca2bfff640 1 -- 172.21.15.50:0/2282017389 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca3004c7a0 con 0x55b9768edb30 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.450+0000 7fca2bfff640 1 -- 172.21.15.50:0/2282017389 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca30035040 con 0x55b9768edb30 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.451+0000 7fca4b454740 1 -- 172.21.15.50:0/2282017389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768edb30 msgr2=0x55b9769082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.451+0000 7fca4b454740 1 --2- 172.21.15.50:0/2282017389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768edb30 0x55b9769082f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fca4006b5f0 tx=0x7fca30031400 comp rx=0 tx=0).stop 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.451+0000 7fca4b454740 1 -- 172.21.15.50:0/2282017389 shutdown_connections 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.451+0000 7fca4b454740 1 --2- 172.21.15.50:0/2282017389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1f40 0x55b9768fbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.451+0000 7fca4b454740 1 --2- 172.21.15.50:0/2282017389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768f1620 0x55b9768ecb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.451+0000 7fca4b454740 1 --2- 172.21.15.50:0/2282017389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768edb30 0x55b9769082f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.451+0000 7fca4b454740 1 -- 172.21.15.50:0/2282017389 >> 172.21.15.50:0/2282017389 conn(0x55b97690eee0 msgr2=0x55b9768ec420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.451+0000 7fca4b454740 1 -- 172.21.15.50:0/2282017389 shutdown_connections 2024-04-23T18:20:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.451+0000 7fca4b454740 1 -- 172.21.15.50:0/2282017389 wait complete. 2024-04-23T18:20:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.451+0000 7fca4b454740 1 -- start start 2024-04-23T18:20:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.451+0000 7fca4b454740 1 Processor -- start 2024-04-23T18:20:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca4b454740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768edb30 0x55b9769049f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca4b454740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 0x55b9768f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca4b454740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1f40 0x55b9768f6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca4b454740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fca401ac130 con 0x55b9768f1f40 2024-04-23T18:20:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca4b454740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca401ad5a0 con 0x55b9768f1620 2024-04-23T18:20:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca4b454740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fca401ac800 con 0x55b9768edb30 2024-04-23T18:20:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca4b454740 10 client.0 _ll_register_callbacks cb 0x7fca380093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca45cd1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1f40 0x55b9768f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca464d2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 0x55b9768f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca464d2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 0x55b9768f5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:38890/0 (socket says 172.21.15.50:38890) 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca45cd1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1f40 0x55b9768f6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41812/0 (socket says 172.21.15.50:41812) 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca464d2640 1 -- 172.21.15.50:0/155806310 learned_addr learned my addr 172.21.15.50:0/155806310 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca45cd1640 1 --2- 172.21.15.50:0/155806310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768edb30 0x55b9769049f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca464d2640 1 -- 172.21.15.50:0/155806310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768edb30 msgr2=0x55b9769049f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca464d2640 1 --2- 172.21.15.50:0/155806310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768edb30 0x55b9769049f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca464d2640 1 -- 172.21.15.50:0/155806310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1f40 msgr2=0x55b9768f6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca464d2640 1 --2- 172.21.15.50:0/155806310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1f40 0x55b9768f6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca464d2640 1 -- 172.21.15.50:0/155806310 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9768f67b0 con 0x55b9768f1620 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca464d2640 1 --2- 172.21.15.50:0/155806310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 0x55b9768f5d30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55b9768f8ce0 tx=0x7fca3c011d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=598fa737e61960a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca29ffb640 1 -- 172.21.15.50:0/155806310 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3c039060 con 0x55b9768f1620 2024-04-23T18:20:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca4b454740 1 -- 172.21.15.50:0/155806310 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca3004c7a0 con 0x55b9768f1620 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.452+0000 7fca29ffb640 1 -- 172.21.15.50:0/155806310 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fca3c005360 con 0x55b9768f1620 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca29ffb640 1 -- 172.21.15.50:0/155806310 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fca3c01e040 con 0x55b9768f1620 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca4b454740 10 client.25421 fetch_fsmap learned FSMap version 23 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca4b454740 1 -- 172.21.15.50:0/155806310 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b976976cd0 con 0x55b9768f1620 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca29ffb640 1 -- 172.21.15.50:0/155806310 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca3004c7a0 con 0x55b9768f1620 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca29ffb640 1 -- 172.21.15.50:0/155806310 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fca3c021040 con 0x55b9768f1620 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca4b454740 10 client.25421 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca4b454740 10 client.25421 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca4b454740 20 client.25421 populate_metadata read hostname 'smithi050' 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca4b454740 1 --2- 172.21.15.50:0/155806310 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b9768fc800 0x55b976979430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca4b454740 4 client.25421 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca4b454740 1 -- 172.21.15.50:0/155806310 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b97685e200 con 0x55b9768fc800 2024-04-23T18:20:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca45cd1640 1 --2- 172.21.15.50:0/155806310 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b9768fc800 0x55b976979430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:44.453 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca45cd1640 1 --2- 172.21.15.50:0/155806310 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b9768fc800 0x55b976979430 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5112e44b9a49fa15 server_cookie=189740fb01f9946e in_seq=0 out_seq=0 2024-04-23T18:20:44.453 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.453+0000 7fca29ffb640 10 client.25421 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca29ffb640 1 -- 172.21.15.50:0/155806310 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b97685e200 con 0x55b9768fc800 2024-04-23T18:20:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca29ffb640 10 client.25421 handle_command_reply: tid=0 2024-04-23T18:20:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca4b454740 1 client.25421 shutdown 2024-04-23T18:20:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca4b454740 1 -- 172.21.15.50:0/155806310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 msgr2=0x55b9768f5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca4b454740 1 --2- 172.21.15.50:0/155806310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 0x55b9768f5d30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55b9768f8ce0 tx=0x7fca3c011d10 comp rx=0 tx=0).stop 2024-04-23T18:20:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca4b454740 1 -- 172.21.15.50:0/155806310 shutdown_connections 2024-04-23T18:20:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca4b454740 1 --2- 172.21.15.50:0/155806310 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b9768fc800 0x55b976979430 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca4b454740 1 --2- 172.21.15.50:0/155806310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b9768f1f40 0x55b9768f6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca4b454740 1 --2- 172.21.15.50:0/155806310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9768f1620 0x55b9768f5d30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca4b454740 1 --2- 172.21.15.50:0/155806310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b9768edb30 0x55b9769049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca4b454740 1 -- 172.21.15.50:0/155806310 >> 172.21.15.50:0/155806310 conn(0x55b97690eee0 msgr2=0x55b9768eccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca29ffb640 0 client.25421 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca4b454740 1 -- 172.21.15.50:0/155806310 shutdown_connections 2024-04-23T18:20:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca4b454740 1 -- 172.21.15.50:0/155806310 wait complete. 2024-04-23T18:20:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.492+0000 7fca4b454740 20 client.25421 trim_cache size 0 max 16384 2024-04-23T18:20:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:20:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:20:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:20:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:20:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.493+0000 7fca48736640 1 -- 172.21.15.50:0/3854679127 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca2c0a3e40 msgr2=0x7fca2c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.493+0000 7fca48736640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca2c0a3e40 0x7fca2c0a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fca3809c940 tx=0x7fca3806f730 comp rx=0 tx=0).stop 2024-04-23T18:20:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.493+0000 7fca48736640 1 -- 172.21.15.50:0/3854679127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca4019cf50 msgr2=0x7fca4019d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.493+0000 7fca48736640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca4019cf50 0x7fca4019d400 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fca3800eb10 tx=0x7fca3800c2d0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.493+0000 7fca48736640 1 -- 172.21.15.50:0/3854679127 shutdown_connections 2024-04-23T18:20:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.493+0000 7fca48736640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca2c0a3e40 0x7fca2c0a6300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.493+0000 7fca48736640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca4019cf50 0x7fca4019d400 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.493+0000 7fca48736640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca40107290 0x7fca40198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.493+0000 7fca48736640 1 --2- 172.21.15.50:0/3854679127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca401021c0 0x7fca40198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.493+0000 7fca48736640 1 -- 172.21.15.50:0/3854679127 >> 172.21.15.50:0/3854679127 conn(0x7fca400fda20 msgr2=0x7fca400ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.493+0000 7fca48736640 1 -- 172.21.15.50:0/3854679127 shutdown_connections 2024-04-23T18:20:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:44.493+0000 7fca48736640 1 -- 172.21.15.50:0/3854679127 wait complete. 2024-04-23T18:20:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:44 smithi177 ceph-mon[32308]: pgmap v69: 97 pgs: 97 active+clean; 51 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.2 MiB/s wr, 407 op/s 2024-04-23T18:20:44.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:44 smithi188 ceph-mon[32694]: pgmap v69: 97 pgs: 97 active+clean; 51 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.2 MiB/s wr, 407 op/s 2024-04-23T18:20:44.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:44 smithi050 ceph-mon[29187]: pgmap v69: 97 pgs: 97 active+clean; 51 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.2 MiB/s wr, 407 op/s 2024-04-23T18:20:45.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 109626 7.69 MB/sec warmup 79 sec latency 42.748 ms 2024-04-23T18:20:46.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 111410 7.74 MB/sec warmup 80 sec latency 12.731 ms 2024-04-23T18:20:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:46 smithi177 ceph-mon[32308]: pgmap v70: 97 pgs: 97 active+clean; 51 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 269 op/s 2024-04-23T18:20:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:46 smithi188 ceph-mon[32694]: pgmap v70: 97 pgs: 97 active+clean; 51 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 269 op/s 2024-04-23T18:20:46.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:46 smithi050 ceph-mon[29187]: pgmap v70: 97 pgs: 97 active+clean; 51 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 269 op/s 2024-04-23T18:20:47.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 113812 7.81 MB/sec warmup 81 sec latency 11.864 ms 2024-04-23T18:20:48.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 115898 7.80 MB/sec warmup 82 sec latency 35.411 ms 2024-04-23T18:20:48.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:48 smithi177 ceph-mon[32308]: pgmap v71: 97 pgs: 97 active+clean; 52 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-04-23T18:20:48.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:48 smithi188 ceph-mon[32694]: pgmap v71: 97 pgs: 97 active+clean; 52 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-04-23T18:20:48.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:48 smithi050 ceph-mon[29187]: pgmap v71: 97 pgs: 97 active+clean; 52 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-04-23T18:20:49.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 117709 7.86 MB/sec warmup 83 sec latency 24.560 ms 2024-04-23T18:20:50.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 120265 7.92 MB/sec warmup 84 sec latency 17.815 ms 2024-04-23T18:20:50.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:50 smithi050 ceph-mon[29187]: pgmap v72: 97 pgs: 97 active+clean; 52 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 295 op/s 2024-04-23T18:20:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:50 smithi177 ceph-mon[32308]: pgmap v72: 97 pgs: 97 active+clean; 52 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 295 op/s 2024-04-23T18:20:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:50 smithi188 ceph-mon[32694]: pgmap v72: 97 pgs: 97 active+clean; 52 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 295 op/s 2024-04-23T18:20:51.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 122059 7.96 MB/sec warmup 85 sec latency 15.796 ms 2024-04-23T18:20:52.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 124449 8.02 MB/sec warmup 86 sec latency 12.750 ms 2024-04-23T18:20:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:52 smithi177 ceph-mon[32308]: pgmap v73: 97 pgs: 97 active+clean; 52 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 295 op/s 2024-04-23T18:20:52.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:52 smithi188 ceph-mon[32694]: pgmap v73: 97 pgs: 97 active+clean; 52 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 295 op/s 2024-04-23T18:20:52.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:52 smithi050 ceph-mon[29187]: pgmap v73: 97 pgs: 97 active+clean; 52 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 295 op/s 2024-04-23T18:20:53.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 126391 8.00 MB/sec warmup 87 sec latency 17.008 ms 2024-04-23T18:20:53.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:53 smithi177 ceph-mon[32308]: pgmap v74: 97 pgs: 97 active+clean; 56 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.2 MiB/s wr, 430 op/s 2024-04-23T18:20:53.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:53 smithi188 ceph-mon[32694]: pgmap v74: 97 pgs: 97 active+clean; 56 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.2 MiB/s wr, 430 op/s 2024-04-23T18:20:53.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:53 smithi050 ceph-mon[29187]: pgmap v74: 97 pgs: 97 active+clean; 56 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.2 MiB/s wr, 430 op/s 2024-04-23T18:20:54.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 128265 8.06 MB/sec warmup 88 sec latency 28.215 ms 2024-04-23T18:20:55.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 130627 8.10 MB/sec warmup 89 sec latency 16.275 ms 2024-04-23T18:20:55.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:55 smithi050 ceph-mon[29187]: pgmap v75: 97 pgs: 97 active+clean; 56 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 293 op/s 2024-04-23T18:20:56.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:55 smithi177 ceph-mon[32308]: pgmap v75: 97 pgs: 97 active+clean; 56 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 293 op/s 2024-04-23T18:20:56.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:55 smithi188 ceph-mon[32694]: pgmap v75: 97 pgs: 97 active+clean; 56 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 293 op/s 2024-04-23T18:20:56.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 132419 8.12 MB/sec warmup 90 sec latency 17.723 ms 2024-04-23T18:20:57.195 INFO:tasks.workunit.client.0.smithi050.stdout: 1 135047 8.21 MB/sec warmup 91 sec latency 10.507 ms 2024-04-23T18:20:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.593+0000 7f875dd29640 1 -- 172.21.15.50:0/3934933155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8758106f90 msgr2=0x7f8758107370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.593+0000 7f875dd29640 1 --2- 172.21.15.50:0/3934933155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8758106f90 0x7f8758107370 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8748009900 tx=0x7f874802a410 comp rx=0 tx=0).stop 2024-04-23T18:20:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.593+0000 7f875dd29640 1 -- 172.21.15.50:0/3934933155 shutdown_connections 2024-04-23T18:20:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.593+0000 7f875dd29640 1 --2- 172.21.15.50:0/3934933155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f875806b120 0x7f875810ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.593+0000 7f875dd29640 1 --2- 172.21.15.50:0/3934933155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875806a760 0x7f875806abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.593+0000 7f875dd29640 1 --2- 172.21.15.50:0/3934933155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8758106f90 0x7f8758107370 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.593+0000 7f875dd29640 1 -- 172.21.15.50:0/3934933155 >> 172.21.15.50:0/3934933155 conn(0x7f8758077dc0 msgr2=0x7f87580781d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.593+0000 7f875dd29640 1 -- 172.21.15.50:0/3934933155 shutdown_connections 2024-04-23T18:20:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.593+0000 7f875dd29640 1 -- 172.21.15.50:0/3934933155 wait complete. 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f875dd29640 1 Processor -- start 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f875dd29640 1 -- start start 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f875dd29640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875806a760 0x7f8758101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f875dd29640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f875806b120 0x7f87581016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f875dd29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8758106f90 0x7f8758101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f875dd29640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f87581121c0 con 0x7f8758106f90 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f875dd29640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8758112040 con 0x7f875806a760 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f875dd29640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8758112340 con 0x7f875806b120 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f87577fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875806a760 0x7f8758101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f8757fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8758106f90 0x7f8758101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f87577fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875806a760 0x7f8758101190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:45090/0 (socket says 172.21.15.50:45090) 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f8756ffd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f875806b120 0x7f87581016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.594+0000 7f87577fe640 1 -- 172.21.15.50:0/857688880 learned_addr learned my addr 172.21.15.50:0/857688880 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.595+0000 7f87577fe640 1 -- 172.21.15.50:0/857688880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f875806b120 msgr2=0x7f87581016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.595+0000 7f87577fe640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f875806b120 0x7f87581016d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.595+0000 7f87577fe640 1 -- 172.21.15.50:0/857688880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8758106f90 msgr2=0x7f8758101c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.595+0000 7f87577fe640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8758106f90 0x7f8758101c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.595+0000 7f87577fe640 1 -- 172.21.15.50:0/857688880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87581a6a60 con 0x7f875806a760 2024-04-23T18:20:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.595+0000 7f8757fff640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8758106f90 0x7f8758101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.595+0000 7f87577fe640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875806a760 0x7f8758101190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8748006010 tx=0x7f87480042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.595+0000 7f8754ff9640 1 -- 172.21.15.50:0/857688880 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874803f030 con 0x7f875806a760 2024-04-23T18:20:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.595+0000 7f875dd29640 1 -- 172.21.15.50:0/857688880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87581a6cf0 con 0x7f875806a760 2024-04-23T18:20:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.595+0000 7f8754ff9640 1 -- 172.21.15.50:0/857688880 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8748003cb0 con 0x7f875806a760 2024-04-23T18:20:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.595+0000 7f8754ff9640 1 -- 172.21.15.50:0/857688880 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8748035070 con 0x7f875806a760 2024-04-23T18:20:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.595+0000 7f875dd29640 1 -- 172.21.15.50:0/857688880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87581a71d0 con 0x7f875806a760 2024-04-23T18:20:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.597+0000 7f8754ff9640 1 -- 172.21.15.50:0/857688880 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8748042030 con 0x7f875806a760 2024-04-23T18:20:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.597+0000 7f8760a44740 1 Processor -- start 2024-04-23T18:20:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f8760a44740 1 -- start start 2024-04-23T18:20:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f8760a44740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f8760a44740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30c02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f8760a44740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 0x55a0e30c2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f8760a44740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a0e301a970 con 0x55a0e30b5f40 2024-04-23T18:20:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f8760a44740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a0e2fb74a0 con 0x55a0e30b5620 2024-04-23T18:20:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f8760a44740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a0e3016930 con 0x55a0e30c0800 2024-04-23T18:20:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f8754ff9640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f87400a3dc0 0x7f87400a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f87577fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30c02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f8756ffd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 0x55a0e30c2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f87577fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30c02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59590/0 (socket says 172.21.15.50:59590) 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f87577fe640 1 -- 172.21.15.50:0/3076807790 learned_addr learned my addr 172.21.15.50:0/3076807790 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f8757fff640 1 --2- 172.21.15.50:0/3076807790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f8757fff640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f87400a3dc0 0x7f87400a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f8754ff9640 1 -- 172.21.15.50:0/857688880 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f87480f0bd0 con 0x7f875806a760 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f87577fe640 1 -- 172.21.15.50:0/3076807790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 msgr2=0x55a0e30c2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f87577fe640 1 --2- 172.21.15.50:0/3076807790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 0x55a0e30c2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f87577fe640 1 -- 172.21.15.50:0/3076807790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 msgr2=0x55a0e30b5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f87577fe640 1 --2- 172.21.15.50:0/3076807790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30b5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.598+0000 7f87577fe640 1 -- 172.21.15.50:0/3076807790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0e2d2d210 con 0x55a0e30b5f40 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f87577fe640 1 --2- 172.21.15.50:0/3076807790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30c02c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f87480b8fe0 tx=0x7f874803b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e56a5ca9be32e74 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f8757fff640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f87400a3dc0 0x7f87400a6280 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8750005c40 tx=0x7f8750009600 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f873dffb640 1 -- 172.21.15.50:0/3076807790 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8748049070 con 0x55a0e30b5f40 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f873dffb640 1 -- 172.21.15.50:0/3076807790 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87480bde50 con 0x55a0e30b5f40 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f873dffb640 1 -- 172.21.15.50:0/3076807790 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87480dd150 con 0x55a0e30b5f40 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f8760a44740 1 -- 172.21.15.50:0/3076807790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 msgr2=0x55a0e30c02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f8760a44740 1 --2- 172.21.15.50:0/3076807790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30c02c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f87480b8fe0 tx=0x7f874803b720 comp rx=0 tx=0).stop 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f8760a44740 1 -- 172.21.15.50:0/3076807790 shutdown_connections 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f8760a44740 1 --2- 172.21.15.50:0/3076807790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 0x55a0e30c2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f8760a44740 1 --2- 172.21.15.50:0/3076807790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30c02c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f8760a44740 1 --2- 172.21.15.50:0/3076807790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30b5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f8760a44740 1 -- 172.21.15.50:0/3076807790 >> 172.21.15.50:0/3076807790 conn(0x55a0e30b1b30 msgr2=0x55a0e30b1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f8760a44740 1 -- 172.21.15.50:0/3076807790 shutdown_connections 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.599+0000 7f8760a44740 1 -- 172.21.15.50:0/3076807790 wait complete. 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8760a44740 1 -- start start 2024-04-23T18:20:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8760a44740 1 Processor -- start 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8760a44740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30c0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8760a44740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30bd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8760a44740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 0x55a0e30bd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8760a44740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a0e301a970 con 0x55a0e30b5f40 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8760a44740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a0e2fb50b0 con 0x55a0e30b5620 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8760a44740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a0e30c31f0 con 0x55a0e30c0800 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8760a44740 10 client.0 _ll_register_callbacks cb 0x55a0e2fa2d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8756ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30bd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f87577fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30c0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8756ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30bd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59602/0 (socket says 172.21.15.50:59602) 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8756ffd640 1 -- 172.21.15.50:0/2589652417 learned_addr learned my addr 172.21.15.50:0/2589652417 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.600+0000 7f8757fff640 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 0x55a0e30bd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8756ffd640 1 -- 172.21.15.50:0/2589652417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 msgr2=0x55a0e30bd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 0x55a0e30bd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8756ffd640 1 -- 172.21.15.50:0/2589652417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 msgr2=0x55a0e30c0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30c0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8756ffd640 1 -- 172.21.15.50:0/2589652417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0e2df9620 con 0x55a0e30b5f40 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f87577fe640 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30c0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30bd410 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f874c00bed0 tx=0x7f874c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3215eb2bb1e7236 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8736ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8736ffd640 1 -- 172.21.15.50:0/2589652417 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c004020 con 0x55a0e30b5f40 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8760a44740 1 -- 172.21.15.50:0/2589652417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87480bde50 con 0x55a0e30b5f40 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8736ffd640 1 -- 172.21.15.50:0/2589652417 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f874c0041c0 con 0x55a0e30b5f40 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8736ffd640 1 -- 172.21.15.50:0/2589652417 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f874c004400 con 0x55a0e30b5f40 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8760a44740 10 client.15813 fetch_fsmap learned FSMap version 23 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.601+0000 7f8760a44740 1 -- 172.21.15.50:0/2589652417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a0e30cef50 con 0x55a0e30b5f40 2024-04-23T18:20:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.602+0000 7f8736ffd640 1 -- 172.21.15.50:0/2589652417 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87480bde50 con 0x55a0e30b5f40 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.602+0000 7f8736ffd640 1 -- 172.21.15.50:0/2589652417 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f874c003960 con 0x55a0e30b5f40 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.602+0000 7f8760a44740 10 client.15813 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.602+0000 7f8760a44740 10 client.15813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.602+0000 7f8760a44740 20 client.15813 populate_metadata read hostname 'smithi050' 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.602+0000 7f8760a44740 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0e30d0520 0x55a0e30d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.602+0000 7f8760a44740 4 client.15813 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.602+0000 7f8760a44740 1 -- 172.21.15.50:0/2589652417 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0e3028390 con 0x55a0e30d0520 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.602+0000 7f87577fe640 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0e30d0520 0x55a0e30d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.602+0000 7f8736ffd640 10 client.15813 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.602+0000 7f87577fe640 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0e30d0520 0x55a0e30d29e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e84a662437c9b2d0 server_cookie=67802fe1838fee18 in_seq=0 out_seq=0 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.603+0000 7f8736ffd640 1 -- 172.21.15.50:0/2589652417 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a0e3028390 con 0x55a0e30d0520 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.603+0000 7f8736ffd640 10 client.15813 handle_command_reply: tid=0 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.603+0000 7f8760a44740 1 client.15813 shutdown 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.603+0000 7f8760a44740 1 -- 172.21.15.50:0/2589652417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 msgr2=0x55a0e30bd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.603+0000 7f8760a44740 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30bd410 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f874c00bed0 tx=0x7f874c011810 comp rx=0 tx=0).stop 2024-04-23T18:20:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.604+0000 7f8760a44740 1 -- 172.21.15.50:0/2589652417 shutdown_connections 2024-04-23T18:20:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.604+0000 7f8760a44740 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0e30d0520 0x55a0e30d29e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.604+0000 7f8760a44740 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 0x55a0e30bd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.604+0000 7f8760a44740 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30bd410 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.604+0000 7f8760a44740 1 --2- 172.21.15.50:0/2589652417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30c0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.604+0000 7f8760a44740 1 -- 172.21.15.50:0/2589652417 >> 172.21.15.50:0/2589652417 conn(0x55a0e30b1b30 msgr2=0x55a0e30b78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.604+0000 7f8736ffd640 0 client.15813 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.604+0000 7f8760a44740 1 -- 172.21.15.50:0/2589652417 shutdown_connections 2024-04-23T18:20:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.604+0000 7f8760a44740 1 -- 172.21.15.50:0/2589652417 wait complete. 2024-04-23T18:20:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.604+0000 7f8760a44740 20 client.15813 trim_cache size 0 max 16384 2024-04-23T18:20:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f8760a44740 1 Processor -- start 2024-04-23T18:20:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f8760a44740 1 -- start start 2024-04-23T18:20:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f8760a44740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f8760a44740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30b5f40 0x55a0e30bc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f8760a44740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30c0800 0x55a0e30bfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f8760a44740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f87581b2580 con 0x55a0e30c0800 2024-04-23T18:20:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f8760a44740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f87581afe50 con 0x55a0e30b5620 2024-04-23T18:20:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f8760a44740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f87581adca0 con 0x55a0e30b5f40 2024-04-23T18:20:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f87577fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f8757fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30c0800 0x55a0e30bfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f87577fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30cbf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:45112/0 (socket says 172.21.15.50:45112) 2024-04-23T18:20:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f87577fe640 1 -- 172.21.15.50:0/3551112421 learned_addr learned my addr 172.21.15.50:0/3551112421 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f8756ffd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30b5f40 0x55a0e30bc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f87577fe640 1 -- 172.21.15.50:0/3551112421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30b5f40 msgr2=0x55a0e30bc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f87577fe640 1 --2- 172.21.15.50:0/3551112421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30b5f40 0x55a0e30bc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f87577fe640 1 -- 172.21.15.50:0/3551112421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30c0800 msgr2=0x55a0e30bfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f87577fe640 1 --2- 172.21.15.50:0/3551112421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30c0800 0x55a0e30bfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.621+0000 7f87577fe640 1 -- 172.21.15.50:0/3551112421 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0e30bb980 con 0x55a0e30b5620 2024-04-23T18:20:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.622+0000 7f8756ffd640 1 --2- 172.21.15.50:0/3551112421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30b5f40 0x55a0e30bc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.622+0000 7f87577fe640 1 --2- 172.21.15.50:0/3551112421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30cbf30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f87480c5260 tx=0x7f8748005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=939f68ed3ec7bce server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.622+0000 7f873d7fa640 1 -- 172.21.15.50:0/3551112421 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87480d5070 con 0x55a0e30b5620 2024-04-23T18:20:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.622+0000 7f873d7fa640 1 -- 172.21.15.50:0/3551112421 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87480c5790 con 0x55a0e30b5620 2024-04-23T18:20:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.622+0000 7f873d7fa640 1 -- 172.21.15.50:0/3551112421 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8748049040 con 0x55a0e30b5620 2024-04-23T18:20:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.623+0000 7f8760a44740 1 -- 172.21.15.50:0/3551112421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 msgr2=0x55a0e30cbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.623+0000 7f8760a44740 1 --2- 172.21.15.50:0/3551112421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30cbf30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f87480c5260 tx=0x7f8748005c90 comp rx=0 tx=0).stop 2024-04-23T18:20:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.623+0000 7f8760a44740 1 -- 172.21.15.50:0/3551112421 shutdown_connections 2024-04-23T18:20:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.623+0000 7f8760a44740 1 --2- 172.21.15.50:0/3551112421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30c0800 0x55a0e30bfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.623+0000 7f8760a44740 1 --2- 172.21.15.50:0/3551112421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30b5f40 0x55a0e30bc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.623+0000 7f8760a44740 1 --2- 172.21.15.50:0/3551112421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30cbf30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.623+0000 7f8760a44740 1 -- 172.21.15.50:0/3551112421 >> 172.21.15.50:0/3551112421 conn(0x55a0e30b1b30 msgr2=0x55a0e30bbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.623+0000 7f8760a44740 1 -- 172.21.15.50:0/3551112421 shutdown_connections 2024-04-23T18:20:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.623+0000 7f8760a44740 1 -- 172.21.15.50:0/3551112421 wait complete. 2024-04-23T18:20:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.623+0000 7f8760a44740 1 -- start start 2024-04-23T18:20:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.623+0000 7f8760a44740 1 Processor -- start 2024-04-23T18:20:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f8760a44740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f8760a44740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30b9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f8760a44740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 0x55a0e30ba140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f8760a44740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f87581ae310 con 0x55a0e30b5f40 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f8760a44740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f87581af780 con 0x55a0e30b5620 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f8760a44740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f87581ae9e0 con 0x55a0e30c0800 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f8760a44740 10 client.0 _ll_register_callbacks cb 0x7f874c00b9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f8756ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30b9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f87577fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f8757fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 0x55a0e30ba140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f87577fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30c8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:45118/0 (socket says 172.21.15.50:45118) 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f87577fe640 1 -- 172.21.15.50:0/3902984310 learned_addr learned my addr 172.21.15.50:0/3902984310 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f8756ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30b9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59614/0 (socket says 172.21.15.50:59614) 2024-04-23T18:20:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f87577fe640 1 -- 172.21.15.50:0/3902984310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 msgr2=0x55a0e30ba140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f87577fe640 1 --2- 172.21.15.50:0/3902984310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 0x55a0e30ba140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f87577fe640 1 -- 172.21.15.50:0/3902984310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 msgr2=0x55a0e30b9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f87577fe640 1 --2- 172.21.15.50:0/3902984310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30b9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f87577fe640 1 -- 172.21.15.50:0/3902984310 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0e30ba780 con 0x55a0e30b5620 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f8756ffd640 1 --2- 172.21.15.50:0/3902984310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30b9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f87577fe640 1 --2- 172.21.15.50:0/3902984310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30c8630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8748005ab0 tx=0x7f8748004460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e9d92203b54a904 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.624+0000 7f8736ffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.625+0000 7f8736ffd640 1 -- 172.21.15.50:0/3902984310 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87480d5070 con 0x55a0e30b5620 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.625+0000 7f8760a44740 1 -- 172.21.15.50:0/3902984310 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87480c5790 con 0x55a0e30b5620 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.625+0000 7f8736ffd640 1 -- 172.21.15.50:0/3902984310 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f874803b460 con 0x55a0e30b5620 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.625+0000 7f8736ffd640 1 -- 172.21.15.50:0/3902984310 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8748049050 con 0x55a0e30b5620 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.625+0000 7f8760a44740 10 client.25439 fetch_fsmap learned FSMap version 23 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.625+0000 7f8760a44740 1 -- 172.21.15.50:0/3902984310 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a0e31369a0 con 0x55a0e30b5620 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.625+0000 7f8736ffd640 1 -- 172.21.15.50:0/3902984310 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87480c5790 con 0x55a0e30b5620 2024-04-23T18:20:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.625+0000 7f8736ffd640 1 -- 172.21.15.50:0/3902984310 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f87480dbce0 con 0x55a0e30b5620 2024-04-23T18:20:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.625+0000 7f8760a44740 10 client.25439 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:20:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.625+0000 7f8760a44740 10 client.25439 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:20:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.626+0000 7f8760a44740 20 client.25439 populate_metadata read hostname 'smithi050' 2024-04-23T18:20:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.626+0000 7f8760a44740 1 --2- 172.21.15.50:0/3902984310 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0e3136ce0 0x55a0e31390d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:20:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.626+0000 7f8760a44740 4 client.25439 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:20:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.626+0000 7f8760a44740 1 -- 172.21.15.50:0/3902984310 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a0e3022200 con 0x55a0e3136ce0 2024-04-23T18:20:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.626+0000 7f8756ffd640 1 --2- 172.21.15.50:0/3902984310 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0e3136ce0 0x55a0e31390d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:20:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.626+0000 7f8736ffd640 10 client.25439 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.626+0000 7f8756ffd640 1 --2- 172.21.15.50:0/3902984310 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0e3136ce0 0x55a0e31390d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e51182fcad1acf97 server_cookie=66067134c9cc4792 in_seq=0 out_seq=0 2024-04-23T18:20:57.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8736ffd640 1 -- 172.21.15.50:0/3902984310 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a0e3022200 con 0x55a0e3136ce0 2024-04-23T18:20:57.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8736ffd640 10 client.25439 handle_command_reply: tid=0 2024-04-23T18:20:57.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8760a44740 1 client.25439 shutdown 2024-04-23T18:20:57.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8760a44740 1 -- 172.21.15.50:0/3902984310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 msgr2=0x55a0e30c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8760a44740 1 --2- 172.21.15.50:0/3902984310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30c8630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8748005ab0 tx=0x7f8748004460 comp rx=0 tx=0).stop 2024-04-23T18:20:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8760a44740 1 -- 172.21.15.50:0/3902984310 shutdown_connections 2024-04-23T18:20:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8760a44740 1 --2- 172.21.15.50:0/3902984310 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0e3136ce0 0x55a0e31390d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8760a44740 1 --2- 172.21.15.50:0/3902984310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0e30c0800 0x55a0e30ba140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8760a44740 1 --2- 172.21.15.50:0/3902984310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0e30b5f40 0x55a0e30b9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8760a44740 1 --2- 172.21.15.50:0/3902984310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0e30b5620 0x55a0e30c8630 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8736ffd640 0 client.25439 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:20:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8760a44740 1 -- 172.21.15.50:0/3902984310 >> 172.21.15.50:0/3902984310 conn(0x55a0e30b1b30 msgr2=0x55a0e30b0420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8760a44740 1 -- 172.21.15.50:0/3902984310 shutdown_connections 2024-04-23T18:20:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8760a44740 1 -- 172.21.15.50:0/3902984310 wait complete. 2024-04-23T18:20:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.659+0000 7f8760a44740 20 client.25439 trim_cache size 0 max 16384 2024-04-23T18:20:57.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:20:57.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:20:57.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:20:57.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:20:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.660+0000 7f875dd29640 1 -- 172.21.15.50:0/857688880 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f87400a3dc0 msgr2=0x7f87400a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.660+0000 7f875dd29640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f87400a3dc0 0x7f87400a6280 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8750005c40 tx=0x7f8750009600 comp rx=0 tx=0).stop 2024-04-23T18:20:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.660+0000 7f875dd29640 1 -- 172.21.15.50:0/857688880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875806a760 msgr2=0x7f8758101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:20:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.660+0000 7f875dd29640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875806a760 0x7f8758101190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8748006010 tx=0x7f87480042f0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.660+0000 7f875dd29640 1 -- 172.21.15.50:0/857688880 shutdown_connections 2024-04-23T18:20:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.660+0000 7f875dd29640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f87400a3dc0 0x7f87400a6280 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.660+0000 7f875dd29640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8758106f90 0x7f8758101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.660+0000 7f875dd29640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f875806b120 0x7f87581016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.660+0000 7f875dd29640 1 --2- 172.21.15.50:0/857688880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f875806a760 0x7f8758101190 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:20:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.660+0000 7f875dd29640 1 -- 172.21.15.50:0/857688880 >> 172.21.15.50:0/857688880 conn(0x7f8758077dc0 msgr2=0x7f87580ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:20:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.660+0000 7f875dd29640 1 -- 172.21.15.50:0/857688880 shutdown_connections 2024-04-23T18:20:57.657 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:20:57.660+0000 7f875dd29640 1 -- 172.21.15.50:0/857688880 wait complete. 2024-04-23T18:20:57.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:57 smithi050 ceph-mon[29187]: pgmap v76: 97 pgs: 97 active+clean; 60 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.3 MiB/s wr, 430 op/s 2024-04-23T18:20:58.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:57 smithi177 ceph-mon[32308]: pgmap v76: 97 pgs: 97 active+clean; 60 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.3 MiB/s wr, 430 op/s 2024-04-23T18:20:58.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:57 smithi188 ceph-mon[32694]: pgmap v76: 97 pgs: 97 active+clean; 60 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.3 MiB/s wr, 430 op/s 2024-04-23T18:20:58.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 136596 8.18 MB/sec warmup 92 sec latency 76.107 ms 2024-04-23T18:20:59.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 138637 8.24 MB/sec warmup 93 sec latency 33.301 ms 2024-04-23T18:20:59.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:20:59 smithi050 ceph-mon[29187]: pgmap v77: 97 pgs: 97 active+clean; 60 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 271 op/s 2024-04-23T18:21:00.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:20:59 smithi177 ceph-mon[32308]: pgmap v77: 97 pgs: 97 active+clean; 60 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 271 op/s 2024-04-23T18:21:00.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:20:59 smithi188 ceph-mon[32694]: pgmap v77: 97 pgs: 97 active+clean; 60 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 271 op/s 2024-04-23T18:21:00.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 140769 8.23 MB/sec warmup 94 sec latency 15.790 ms 2024-04-23T18:21:01.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 142234 8.27 MB/sec warmup 95 sec latency 22.409 ms 2024-04-23T18:21:01.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:01 smithi050 ceph-mon[29187]: pgmap v78: 97 pgs: 97 active+clean; 60 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 271 op/s 2024-04-23T18:21:02.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:01 smithi177 ceph-mon[32308]: pgmap v78: 97 pgs: 97 active+clean; 60 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 271 op/s 2024-04-23T18:21:02.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:01 smithi188 ceph-mon[32694]: pgmap v78: 97 pgs: 97 active+clean; 60 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 271 op/s 2024-04-23T18:21:02.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 144328 8.26 MB/sec warmup 96 sec latency 14.060 ms 2024-04-23T18:21:03.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 146170 8.30 MB/sec warmup 97 sec latency 16.768 ms 2024-04-23T18:21:03.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:03 smithi050 ceph-mon[29187]: pgmap v79: 97 pgs: 97 active+clean; 57 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 455 op/s 2024-04-23T18:21:04.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:03 smithi177 ceph-mon[32308]: pgmap v79: 97 pgs: 97 active+clean; 57 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 455 op/s 2024-04-23T18:21:04.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:03 smithi188 ceph-mon[32694]: pgmap v79: 97 pgs: 97 active+clean; 57 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 455 op/s 2024-04-23T18:21:04.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 148359 8.34 MB/sec warmup 98 sec latency 52.583 ms 2024-04-23T18:21:05.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 150164 8.35 MB/sec warmup 99 sec latency 18.629 ms 2024-04-23T18:21:05.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:05 smithi050 ceph-mon[29187]: pgmap v80: 97 pgs: 97 active+clean; 57 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 320 op/s 2024-04-23T18:21:06.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:05 smithi177 ceph-mon[32308]: pgmap v80: 97 pgs: 97 active+clean; 57 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 320 op/s 2024-04-23T18:21:06.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:05 smithi188 ceph-mon[32694]: pgmap v80: 97 pgs: 97 active+clean; 57 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 320 op/s 2024-04-23T18:21:06.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 152766 8.43 MB/sec warmup 100 sec latency 8.966 ms 2024-04-23T18:21:07.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 154896 8.42 MB/sec warmup 101 sec latency 13.733 ms 2024-04-23T18:21:07.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:07 smithi050 ceph-mon[29187]: pgmap v81: 97 pgs: 97 active+clean; 62 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.4 MiB/s wr, 453 op/s 2024-04-23T18:21:08.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:07 smithi177 ceph-mon[32308]: pgmap v81: 97 pgs: 97 active+clean; 62 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.4 MiB/s wr, 453 op/s 2024-04-23T18:21:08.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:07 smithi188 ceph-mon[32694]: pgmap v81: 97 pgs: 97 active+clean; 62 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.4 MiB/s wr, 453 op/s 2024-04-23T18:21:08.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 156769 8.46 MB/sec warmup 102 sec latency 14.203 ms 2024-04-23T18:21:09.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 158969 8.48 MB/sec warmup 103 sec latency 42.817 ms 2024-04-23T18:21:09.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:09 smithi050 ceph-mon[29187]: pgmap v82: 97 pgs: 97 active+clean; 62 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-04-23T18:21:10.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:09 smithi177 ceph-mon[32308]: pgmap v82: 97 pgs: 97 active+clean; 62 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-04-23T18:21:10.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:09 smithi188 ceph-mon[32694]: pgmap v82: 97 pgs: 97 active+clean; 62 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-04-23T18:21:10.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 160699 8.49 MB/sec warmup 104 sec latency 35.241 ms 2024-04-23T18:21:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.755+0000 7f7767b29640 1 -- 172.21.15.50:0/668005820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7760101aa0 msgr2=0x7f7760105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.755+0000 7f7767b29640 1 --2- 172.21.15.50:0/668005820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7760101aa0 0x7f7760105a90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7750009900 tx=0x7f775002a410 comp rx=0 tx=0).stop 2024-04-23T18:21:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.755+0000 7f7767b29640 1 -- 172.21.15.50:0/668005820 shutdown_connections 2024-04-23T18:21:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.755+0000 7f7767b29640 1 --2- 172.21.15.50:0/668005820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7760106170 0x7f776010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.755+0000 7f7767b29640 1 --2- 172.21.15.50:0/668005820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7760101aa0 0x7f7760105a90 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.755+0000 7f7767b29640 1 --2- 172.21.15.50:0/668005820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77601010f0 0x7f77601014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.755+0000 7f7767b29640 1 -- 172.21.15.50:0/668005820 >> 172.21.15.50:0/668005820 conn(0x7f7760077fd0 msgr2=0x7f77600783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.756+0000 7f7767b29640 1 -- 172.21.15.50:0/668005820 shutdown_connections 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.756+0000 7f7767b29640 1 -- 172.21.15.50:0/668005820 wait complete. 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.756+0000 7f7767b29640 1 Processor -- start 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.756+0000 7f7767b29640 1 -- start start 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.756+0000 7f7767b29640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77601010f0 0x7f77601995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.756+0000 7f7767b29640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7760106170 0x7f7760199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.756+0000 7f7767b29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f776019a080 0x7f77601a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.756+0000 7f7767b29640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f77601110b0 con 0x7f776019a080 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.756+0000 7f7767b29640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7760110f30 con 0x7f77601010f0 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.756+0000 7f7767b29640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7760111230 con 0x7f7760106170 2024-04-23T18:21:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f77660c6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f776019a080 0x7f77601a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f77650c4640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7760106170 0x7f7760199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f77660c6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f776019a080 0x7f77601a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35944/0 (socket says 172.21.15.50:35944) 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f77660c6640 1 -- 172.21.15.50:0/3301968339 learned_addr learned my addr 172.21.15.50:0/3301968339 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f77658c5640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77601010f0 0x7f77601995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f77660c6640 1 -- 172.21.15.50:0/3301968339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7760106170 msgr2=0x7f7760199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f77660c6640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7760106170 0x7f7760199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f77660c6640 1 -- 172.21.15.50:0/3301968339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77601010f0 msgr2=0x7f77601995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f77660c6640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77601010f0 0x7f77601995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f77660c6640 1 -- 172.21.15.50:0/3301968339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77601a4e00 con 0x7f776019a080 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f77660c6640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f776019a080 0x7f77601a4630 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f775800eb10 tx=0x7f775800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f7756ffd640 1 -- 172.21.15.50:0/3301968339 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7758004b20 con 0x7f776019a080 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f7756ffd640 1 -- 172.21.15.50:0/3301968339 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7758004cc0 con 0x7f776019a080 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f7767b29640 1 -- 172.21.15.50:0/3301968339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77601a50f0 con 0x7f776019a080 2024-04-23T18:21:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f7756ffd640 1 -- 172.21.15.50:0/3301968339 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7758010670 con 0x7f776019a080 2024-04-23T18:21:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.757+0000 7f7767b29640 1 -- 172.21.15.50:0/3301968339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77601a5660 con 0x7f776019a080 2024-04-23T18:21:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.760+0000 7f7756ffd640 1 -- 172.21.15.50:0/3301968339 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7758005020 con 0x7f776019a080 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.760+0000 7f776a843740 1 Processor -- start 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.760+0000 7f776a843740 1 -- start start 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f776a843740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf22500 0x563e7bf228e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f776a843740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f776a843740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf3e810 0x563e7bf40c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f776a843740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563e7be87970 con 0x563e7bf22500 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f776a843740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563e7be244a0 con 0x563e7bf22e20 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f776a843740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563e7be83930 con 0x563e7bf3e810 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f7756ffd640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f774c0a3e40 0x7f774c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f77650c4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf22500 0x563e7bf228e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f77650c4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf22500 0x563e7bf228e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35954/0 (socket says 172.21.15.50:35954) 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f77650c4640 1 -- 172.21.15.50:0/2544212956 learned_addr learned my addr 172.21.15.50:0/2544212956 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f77658c5640 1 --2- 172.21.15.50:0/2544212956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f77650c4640 1 --2- 172.21.15.50:0/2544212956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf3e810 0x563e7bf40c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f77660c6640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f774c0a3e40 0x7f774c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f77650c4640 1 -- 172.21.15.50:0/2544212956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf3e810 msgr2=0x563e7bf40c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f77650c4640 1 --2- 172.21.15.50:0/2544212956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf3e810 0x563e7bf40c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f7756ffd640 1 -- 172.21.15.50:0/3301968339 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f77580d0cd0 con 0x7f776019a080 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f77650c4640 1 -- 172.21.15.50:0/2544212956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 msgr2=0x563e7bf3e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f77650c4640 1 --2- 172.21.15.50:0/2544212956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f77650c4640 1 -- 172.21.15.50:0/2544212956 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e7bb9a210 con 0x563e7bf22500 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.761+0000 7f77658c5640 1 --2- 172.21.15.50:0/2544212956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.762+0000 7f77650c4640 1 --2- 172.21.15.50:0/2544212956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf22500 0x563e7bf228e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7750033000 tx=0x7f7750005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca3566c7c72388b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.762+0000 7f77660c6640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f774c0a3e40 0x7f774c0a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f775809c980 tx=0x7f7758067280 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.762+0000 7f774bfff640 1 -- 172.21.15.50:0/2544212956 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7750035070 con 0x563e7bf22500 2024-04-23T18:21:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.762+0000 7f774bfff640 1 -- 172.21.15.50:0/2544212956 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77500317e0 con 0x563e7bf22500 2024-04-23T18:21:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.762+0000 7f774bfff640 1 -- 172.21.15.50:0/2544212956 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77500035f0 con 0x563e7bf22500 2024-04-23T18:21:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.763+0000 7f776a843740 1 -- 172.21.15.50:0/2544212956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf22500 msgr2=0x563e7bf228e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.763+0000 7f776a843740 1 --2- 172.21.15.50:0/2544212956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf22500 0x563e7bf228e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7750033000 tx=0x7f7750005780 comp rx=0 tx=0).stop 2024-04-23T18:21:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.763+0000 7f776a843740 1 -- 172.21.15.50:0/2544212956 shutdown_connections 2024-04-23T18:21:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.763+0000 7f776a843740 1 --2- 172.21.15.50:0/2544212956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf3e810 0x563e7bf40c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.763+0000 7f776a843740 1 --2- 172.21.15.50:0/2544212956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.763+0000 7f776a843740 1 --2- 172.21.15.50:0/2544212956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf22500 0x563e7bf228e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.763+0000 7f776a843740 1 -- 172.21.15.50:0/2544212956 >> 172.21.15.50:0/2544212956 conn(0x563e7bf1eb30 msgr2=0x563e7bf1ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.763+0000 7f776a843740 1 -- 172.21.15.50:0/2544212956 shutdown_connections 2024-04-23T18:21:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.763+0000 7f776a843740 1 -- 172.21.15.50:0/2544212956 wait complete. 2024-04-23T18:21:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.763+0000 7f776a843740 1 -- start start 2024-04-23T18:21:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.763+0000 7f776a843740 1 Processor -- start 2024-04-23T18:21:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f776a843740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf22500 0x563e7bf3e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f776a843740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f776a843740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf3e810 0x563e7bf3b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f776a843740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563e7bf413e0 con 0x563e7bf3e810 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f776a843740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563e7be220b0 con 0x563e7bf22e20 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f776a843740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563e7bf41560 con 0x563e7bf22500 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f776a843740 10 client.0 _ll_register_callbacks cb 0x563e7be0fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77658c5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77650c4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf3e810 0x563e7bf3b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77658c5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:36458/0 (socket says 172.21.15.50:36458) 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77658c5640 1 -- 172.21.15.50:0/1707474290 learned_addr learned my addr 172.21.15.50:0/1707474290 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77650c4640 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf22500 0x563e7bf3e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77650c4640 1 -- 172.21.15.50:0/1707474290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf22500 msgr2=0x563e7bf3e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77650c4640 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf22500 0x563e7bf3e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77650c4640 1 -- 172.21.15.50:0/1707474290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 msgr2=0x563e7bf3b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77650c4640 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77650c4640 1 -- 172.21.15.50:0/1707474290 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e7bc66620 con 0x563e7bf3e810 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77658c5640 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77650c4640 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf3e810 0x563e7bf3b940 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f775002a930 tx=0x7f7750045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac7244c8527041a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.764+0000 7f77497fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f77497fa640 1 -- 172.21.15.50:0/1707474290 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7750035070 con 0x563e7bf3e810 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f77497fa640 1 -- 172.21.15.50:0/1707474290 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f77500315e0 con 0x563e7bf3e810 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f776a843740 1 -- 172.21.15.50:0/1707474290 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77500317e0 con 0x563e7bf3e810 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f77497fa640 1 -- 172.21.15.50:0/1707474290 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7750005020 con 0x563e7bf3e810 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f776a843740 10 client.15831 fetch_fsmap learned FSMap version 23 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f776a843740 1 -- 172.21.15.50:0/1707474290 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563e7bf4d330 con 0x563e7bf3e810 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f77497fa640 1 -- 172.21.15.50:0/1707474290 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77500317e0 con 0x563e7bf3e810 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f77497fa640 1 -- 172.21.15.50:0/1707474290 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f775003c440 con 0x563e7bf3e810 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f776a843740 10 client.15831 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f776a843740 10 client.15831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f776a843740 20 client.15831 populate_metadata read hostname 'smithi050' 2024-04-23T18:21:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f776a843740 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563e7bf4e680 0x563e7bf50a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f776a843740 4 client.15831 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.765+0000 7f776a843740 1 -- 172.21.15.50:0/1707474290 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e7be95390 con 0x563e7bf4e680 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.766+0000 7f77658c5640 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563e7bf4e680 0x563e7bf50a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.766+0000 7f77497fa640 10 client.15831 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.766+0000 7f77658c5640 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563e7bf4e680 0x563e7bf50a70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9de4fa36503ca4c2 server_cookie=aaef277f4a5a243c in_seq=0 out_seq=0 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f77497fa640 1 -- 172.21.15.50:0/1707474290 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e7be95390 con 0x563e7bf4e680 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f77497fa640 10 client.15831 handle_command_reply: tid=0 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f776a843740 1 client.15831 shutdown 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f776a843740 1 -- 172.21.15.50:0/1707474290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf3e810 msgr2=0x563e7bf3b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f776a843740 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf3e810 0x563e7bf3b940 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f775002a930 tx=0x7f7750045040 comp rx=0 tx=0).stop 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f776a843740 1 -- 172.21.15.50:0/1707474290 shutdown_connections 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f776a843740 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563e7bf4e680 0x563e7bf50a70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f776a843740 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf3e810 0x563e7bf3b940 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f776a843740 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f77497fa640 0 client.15831 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f776a843740 1 --2- 172.21.15.50:0/1707474290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf22500 0x563e7bf3e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f776a843740 1 -- 172.21.15.50:0/1707474290 >> 172.21.15.50:0/1707474290 conn(0x563e7bf1eb30 msgr2=0x563e7bf24940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f776a843740 1 -- 172.21.15.50:0/1707474290 shutdown_connections 2024-04-23T18:21:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f776a843740 1 -- 172.21.15.50:0/1707474290 wait complete. 2024-04-23T18:21:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.767+0000 7f776a843740 20 client.15831 trim_cache size 0 max 16384 2024-04-23T18:21:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f776a843740 1 Processor -- start 2024-04-23T18:21:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f776a843740 1 -- start start 2024-04-23T18:21:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f776a843740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf1eb30 0x563e7bf36b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f776a843740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf22500 0x563e7bf1db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f776a843740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f776a843740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f77601a5a40 con 0x563e7bf1eb30 2024-04-23T18:21:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f776a843740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7760071b70 con 0x563e7bf22e20 2024-04-23T18:21:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f776a843740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f776006f9e0 con 0x563e7bf22500 2024-04-23T18:21:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f77650c4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf1eb30 0x563e7bf36b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f77658c5640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf22500 0x563e7bf1db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f77658c5640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf22500 0x563e7bf1db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:60184/0 (socket says 172.21.15.50:60184) 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f77650c4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf1eb30 0x563e7bf36b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35974/0 (socket says 172.21.15.50:35974) 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f77658c5640 1 -- 172.21.15.50:0/3224561961 learned_addr learned my addr 172.21.15.50:0/3224561961 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.784+0000 7f77650c4640 1 --2- 172.21.15.50:0/3224561961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.785+0000 7f77658c5640 1 -- 172.21.15.50:0/3224561961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 msgr2=0x563e7bf3dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.785+0000 7f77658c5640 1 --2- 172.21.15.50:0/3224561961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.785+0000 7f77658c5640 1 -- 172.21.15.50:0/3224561961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf1eb30 msgr2=0x563e7bf36b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.785+0000 7f77658c5640 1 --2- 172.21.15.50:0/3224561961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf1eb30 0x563e7bf36b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.785+0000 7f77658c5640 1 -- 172.21.15.50:0/3224561961 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e7bf1e960 con 0x563e7bf22500 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.785+0000 7f77650c4640 1 --2- 172.21.15.50:0/3224561961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.785+0000 7f77658c5640 1 --2- 172.21.15.50:0/3224561961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf22500 0x563e7bf1db30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x563e7bf3ac60 tx=0x7f775c014e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=330eafa546212e03 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.785+0000 7f774b7fe640 1 -- 172.21.15.50:0/3224561961 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c004030 con 0x563e7bf22500 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.785+0000 7f774b7fe640 1 -- 172.21.15.50:0/3224561961 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f775c024020 con 0x563e7bf22500 2024-04-23T18:21:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.785+0000 7f774b7fe640 1 -- 172.21.15.50:0/3224561961 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c011d10 con 0x563e7bf22500 2024-04-23T18:21:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.786+0000 7f776a843740 1 -- 172.21.15.50:0/3224561961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf22500 msgr2=0x563e7bf1db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.786+0000 7f776a843740 1 --2- 172.21.15.50:0/3224561961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf22500 0x563e7bf1db30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x563e7bf3ac60 tx=0x7f775c014e90 comp rx=0 tx=0).stop 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.786+0000 7f776a843740 1 -- 172.21.15.50:0/3224561961 shutdown_connections 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.786+0000 7f776a843740 1 --2- 172.21.15.50:0/3224561961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf3dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.786+0000 7f776a843740 1 --2- 172.21.15.50:0/3224561961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf22500 0x563e7bf1db30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.786+0000 7f776a843740 1 --2- 172.21.15.50:0/3224561961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf1eb30 0x563e7bf36b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.786+0000 7f776a843740 1 -- 172.21.15.50:0/3224561961 >> 172.21.15.50:0/3224561961 conn(0x563e7bf3baa0 msgr2=0x563e7bf1d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.786+0000 7f776a843740 1 -- 172.21.15.50:0/3224561961 shutdown_connections 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.786+0000 7f776a843740 1 -- 172.21.15.50:0/3224561961 wait complete. 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.786+0000 7f776a843740 1 -- start start 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.786+0000 7f776a843740 1 Processor -- start 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f776a843740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf1eb30 0x563e7bf46c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f776a843740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf22500 0x563e7bf42000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f776a843740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf42610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f776a843740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7760070030 con 0x563e7bf22500 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f776a843740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f77600714a0 con 0x563e7bf22e20 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f776a843740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7760070700 con 0x563e7bf1eb30 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f776a843740 10 client.0 _ll_register_callbacks cb 0x7f7750032e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f77650c4640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf1eb30 0x563e7bf46c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f77650c4640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf1eb30 0x563e7bf46c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:60186/0 (socket says 172.21.15.50:60186) 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f77650c4640 1 -- 172.21.15.50:0/1934407445 learned_addr learned my addr 172.21.15.50:0/1934407445 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f77658c5640 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf22500 0x563e7bf42000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f77650c4640 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf42610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f77650c4640 1 -- 172.21.15.50:0/1934407445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 msgr2=0x563e7bf42610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f77650c4640 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf42610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f77650c4640 1 -- 172.21.15.50:0/1934407445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf22500 msgr2=0x563e7bf42000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f77650c4640 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf22500 0x563e7bf42000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f77650c4640 1 -- 172.21.15.50:0/1934407445 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e7bf42de0 con 0x563e7bf1eb30 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f77650c4640 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf1eb30 0x563e7bf46c90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7750009900 tx=0x7f7750000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c46fa8436b644659 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.787+0000 7f77497fa640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:21:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f77497fa640 1 -- 172.21.15.50:0/1934407445 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7750035070 con 0x563e7bf1eb30 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f77497fa640 1 -- 172.21.15.50:0/1934407445 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f7750031530 con 0x563e7bf1eb30 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f776a843740 1 -- 172.21.15.50:0/1934407445 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f775c024020 con 0x563e7bf1eb30 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f77497fa640 1 -- 172.21.15.50:0/1934407445 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7750009520 con 0x563e7bf1eb30 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f776a843740 10 client.25531 fetch_fsmap learned FSMap version 23 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f776a843740 1 -- 172.21.15.50:0/1934407445 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563e7bfb4990 con 0x563e7bf1eb30 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f77497fa640 1 -- 172.21.15.50:0/1934407445 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f775c024020 con 0x563e7bf1eb30 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f77497fa640 1 -- 172.21.15.50:0/1934407445 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f77500046c0 con 0x563e7bf1eb30 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f776a843740 10 client.25531 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f776a843740 10 client.25531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f776a843740 20 client.25531 populate_metadata read hostname 'smithi050' 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f776a843740 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563e7bf3e810 0x563e7bfb6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f776a843740 4 client.25531 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f776a843740 1 -- 172.21.15.50:0/1934407445 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563e7be8f200 con 0x563e7bf3e810 2024-04-23T18:21:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.788+0000 7f77658c5640 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563e7bf3e810 0x563e7bfb6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:10.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.789+0000 7f77497fa640 10 client.25531 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:10.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.789+0000 7f77658c5640 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563e7bf3e810 0x563e7bfb6f60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8421f571126c18be server_cookie=f31228648deee604 in_seq=0 out_seq=0 2024-04-23T18:21:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f77497fa640 1 -- 172.21.15.50:0/1934407445 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563e7be8f200 con 0x563e7bf3e810 2024-04-23T18:21:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f77497fa640 10 client.25531 handle_command_reply: tid=0 2024-04-23T18:21:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f776a843740 1 client.25531 shutdown 2024-04-23T18:21:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f776a843740 1 -- 172.21.15.50:0/1934407445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf1eb30 msgr2=0x563e7bf46c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f776a843740 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf1eb30 0x563e7bf46c90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7750009900 tx=0x7f7750000bc0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f776a843740 1 -- 172.21.15.50:0/1934407445 shutdown_connections 2024-04-23T18:21:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f776a843740 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563e7bf3e810 0x563e7bfb6f60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f776a843740 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e7bf22e20 0x563e7bf42610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f776a843740 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563e7bf22500 0x563e7bf42000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f776a843740 1 --2- 172.21.15.50:0/1934407445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e7bf1eb30 0x563e7bf46c90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f77497fa640 0 client.25531 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f776a843740 1 -- 172.21.15.50:0/1934407445 >> 172.21.15.50:0/1934407445 conn(0x563e7bf3baa0 msgr2=0x563e7bf36980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.820+0000 7f776a843740 1 -- 172.21.15.50:0/1934407445 shutdown_connections 2024-04-23T18:21:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f776a843740 1 -- 172.21.15.50:0/1934407445 wait complete. 2024-04-23T18:21:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f776a843740 20 client.25531 trim_cache size 0 max 16384 2024-04-23T18:21:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:21:10.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:21:10.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:21:10.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:21:10.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f7767b29640 1 -- 172.21.15.50:0/3301968339 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f774c0a3e40 msgr2=0x7f774c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f7767b29640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f774c0a3e40 0x7f774c0a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f775809c980 tx=0x7f7758067280 comp rx=0 tx=0).stop 2024-04-23T18:21:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f7767b29640 1 -- 172.21.15.50:0/3301968339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f776019a080 msgr2=0x7f77601a4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f7767b29640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f776019a080 0x7f77601a4630 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f775800eb10 tx=0x7f775800c2d0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f7767b29640 1 -- 172.21.15.50:0/3301968339 shutdown_connections 2024-04-23T18:21:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f7767b29640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f774c0a3e40 0x7f774c0a6300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f7767b29640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f776019a080 0x7f77601a4630 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f7767b29640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7760106170 0x7f7760199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f7767b29640 1 --2- 172.21.15.50:0/3301968339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77601010f0 0x7f77601995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f7767b29640 1 -- 172.21.15.50:0/3301968339 >> 172.21.15.50:0/3301968339 conn(0x7f7760077fd0 msgr2=0x7f77600ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f7767b29640 1 -- 172.21.15.50:0/3301968339 shutdown_connections 2024-04-23T18:21:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:10.821+0000 7f7767b29640 1 -- 172.21.15.50:0/3301968339 wait complete. 2024-04-23T18:21:11.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 163273 8.56 MB/sec warmup 105 sec latency 48.713 ms 2024-04-23T18:21:11.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:11 smithi050 ceph-mon[29187]: pgmap v83: 97 pgs: 97 active+clean; 62 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-04-23T18:21:12.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:11 smithi177 ceph-mon[32308]: pgmap v83: 97 pgs: 97 active+clean; 62 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-04-23T18:21:12.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:11 smithi188 ceph-mon[32694]: pgmap v83: 97 pgs: 97 active+clean; 62 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-04-23T18:21:12.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 165139 8.55 MB/sec warmup 106 sec latency 15.022 ms 2024-04-23T18:21:13.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 167340 8.60 MB/sec warmup 107 sec latency 12.315 ms 2024-04-23T18:21:13.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:13 smithi050 ceph-mon[29187]: pgmap v84: 97 pgs: 97 active+clean; 63 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 468 op/s 2024-04-23T18:21:14.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:13 smithi177 ceph-mon[32308]: pgmap v84: 97 pgs: 97 active+clean; 63 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 468 op/s 2024-04-23T18:21:14.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:13 smithi188 ceph-mon[32694]: pgmap v84: 97 pgs: 97 active+clean; 63 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 468 op/s 2024-04-23T18:21:14.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 169509 8.62 MB/sec warmup 108 sec latency 43.923 ms 2024-04-23T18:21:15.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 171456 8.63 MB/sec warmup 109 sec latency 20.577 ms 2024-04-23T18:21:15.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:15 smithi050 ceph-mon[29187]: pgmap v85: 97 pgs: 97 active+clean; 63 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 285 op/s 2024-04-23T18:21:16.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:15 smithi177 ceph-mon[32308]: pgmap v85: 97 pgs: 97 active+clean; 63 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 285 op/s 2024-04-23T18:21:16.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:15 smithi188 ceph-mon[32694]: pgmap v85: 97 pgs: 97 active+clean; 63 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 285 op/s 2024-04-23T18:21:16.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 174100 8.71 MB/sec warmup 110 sec latency 10.707 ms 2024-04-23T18:21:17.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 176253 8.69 MB/sec warmup 111 sec latency 16.815 ms 2024-04-23T18:21:17.881 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:17 smithi188 ceph-mon[32694]: pgmap v86: 97 pgs: 97 active+clean; 69 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.2 MiB/s wr, 428 op/s 2024-04-23T18:21:17.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:17 smithi050 ceph-mon[29187]: pgmap v86: 97 pgs: 97 active+clean; 69 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.2 MiB/s wr, 428 op/s 2024-04-23T18:21:18.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:17 smithi177 ceph-mon[32308]: pgmap v86: 97 pgs: 97 active+clean; 69 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.2 MiB/s wr, 428 op/s 2024-04-23T18:21:18.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 178188 8.73 MB/sec warmup 112 sec latency 16.208 ms 2024-04-23T18:21:19.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 180374 8.76 MB/sec warmup 113 sec latency 71.263 ms 2024-04-23T18:21:19.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:19 smithi050 ceph-mon[29187]: pgmap v87: 97 pgs: 97 active+clean; 69 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 295 op/s 2024-04-23T18:21:20.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:19 smithi177 ceph-mon[32308]: pgmap v87: 97 pgs: 97 active+clean; 69 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 295 op/s 2024-04-23T18:21:20.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:19 smithi188 ceph-mon[32694]: pgmap v87: 97 pgs: 97 active+clean; 69 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 295 op/s 2024-04-23T18:21:20.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 182164 8.76 MB/sec warmup 114 sec latency 23.184 ms 2024-04-23T18:21:21.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 184658 8.83 MB/sec warmup 115 sec latency 11.951 ms 2024-04-23T18:21:21.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:21 smithi050 ceph-mon[29187]: pgmap v88: 97 pgs: 97 active+clean; 69 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 295 op/s 2024-04-23T18:21:22.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:21 smithi177 ceph-mon[32308]: pgmap v88: 97 pgs: 97 active+clean; 69 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 295 op/s 2024-04-23T18:21:22.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:21 smithi188 ceph-mon[32694]: pgmap v88: 97 pgs: 97 active+clean; 69 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 295 op/s 2024-04-23T18:21:22.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 186350 8.80 MB/sec warmup 116 sec latency 13.662 ms 2024-04-23T18:21:23.196 INFO:tasks.workunit.client.0.smithi050.stdout: 1 188483 8.84 MB/sec warmup 117 sec latency 9.160 ms 2024-04-23T18:21:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.916+0000 7fa8618da640 1 -- 172.21.15.50:0/2635673792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa85c1021c0 msgr2=0x7fa85c1025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.916+0000 7fa8618da640 1 --2- 172.21.15.50:0/2635673792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa85c1021c0 0x7fa85c1025a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa84c009900 tx=0x7fa84c02a410 comp rx=0 tx=0).stop 2024-04-23T18:21:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.916+0000 7fa8618da640 1 -- 172.21.15.50:0/2635673792 shutdown_connections 2024-04-23T18:21:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.916+0000 7fa8618da640 1 --2- 172.21.15.50:0/2635673792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa85c107290 0x7fa85c10ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.916+0000 7fa8618da640 1 --2- 172.21.15.50:0/2635673792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa85c102b70 0x7fa85c106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.916+0000 7fa8618da640 1 --2- 172.21.15.50:0/2635673792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa85c1021c0 0x7fa85c1025a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.916+0000 7fa8618da640 1 -- 172.21.15.50:0/2635673792 >> 172.21.15.50:0/2635673792 conn(0x7fa85c0fda20 msgr2=0x7fa85c0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.916+0000 7fa8618da640 1 -- 172.21.15.50:0/2635673792 shutdown_connections 2024-04-23T18:21:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.916+0000 7fa8618da640 1 -- 172.21.15.50:0/2635673792 wait complete. 2024-04-23T18:21:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa8618da640 1 Processor -- start 2024-04-23T18:21:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa8618da640 1 -- start start 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa8618da640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa85c1021c0 0x7fa85c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa8618da640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa85c107290 0x7fa85c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa8618da640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa85c19cfb0 0x7fa85c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa8618da640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1121f0 con 0x7fa85c1021c0 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa8618da640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c112070 con 0x7fa85c107290 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa8618da640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c112370 con 0x7fa85c19cfb0 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa85affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa85c1021c0 0x7fa85c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa85b7fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa85c19cfb0 0x7fa85c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa85affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa85c1021c0 0x7fa85c1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49784/0 (socket says 172.21.15.50:49784) 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa85affd640 1 -- 172.21.15.50:0/2918903838 learned_addr learned my addr 172.21.15.50:0/2918903838 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa853fff640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa85c107290 0x7fa85c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa85affd640 1 -- 172.21.15.50:0/2918903838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa85c19cfb0 msgr2=0x7fa85c19d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa85affd640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa85c19cfb0 0x7fa85c19d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa85affd640 1 -- 172.21.15.50:0/2918903838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa85c107290 msgr2=0x7fa85c198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa85affd640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa85c107290 0x7fa85c198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.917+0000 7fa85affd640 1 -- 172.21.15.50:0/2918903838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85c19daa0 con 0x7fa85c1021c0 2024-04-23T18:21:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.918+0000 7fa85affd640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa85c1021c0 0x7fa85c1986f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa84c005940 tx=0x7fa84c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.918+0000 7fa8608d8640 1 -- 172.21.15.50:0/2918903838 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c03f030 con 0x7fa85c1021c0 2024-04-23T18:21:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.918+0000 7fa8618da640 1 -- 172.21.15.50:0/2918903838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa85c19dd30 con 0x7fa85c1021c0 2024-04-23T18:21:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.918+0000 7fa8608d8640 1 -- 172.21.15.50:0/2918903838 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa84c0044c0 con 0x7fa85c1021c0 2024-04-23T18:21:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.918+0000 7fa8618da640 1 -- 172.21.15.50:0/2918903838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa85c1a5550 con 0x7fa85c1021c0 2024-04-23T18:21:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.918+0000 7fa8608d8640 1 -- 172.21.15.50:0/2918903838 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c035070 con 0x7fa85c1021c0 2024-04-23T18:21:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.920+0000 7fa8608d8640 1 -- 172.21.15.50:0/2918903838 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa84c042030 con 0x7fa85c1021c0 2024-04-23T18:21:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.920+0000 7fa8645f2740 1 Processor -- start 2024-04-23T18:21:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa8645f2740 1 -- start start 2024-04-23T18:21:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa8645f2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 0x556cad925920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa8645f2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925e60 0x556cad9346d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa8645f2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad934c10 0x556cad937000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa8645f2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556cad88a970 con 0x556cad925e60 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa8645f2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556cad8274a0 con 0x556cad925540 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa8645f2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556cad886930 con 0x556cad934c10 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa85affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925e60 0x556cad9346d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa85b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 0x556cad925920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa85b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 0x556cad925920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:55184/0 (socket says 172.21.15.50:55184) 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa85b7fe640 1 -- 172.21.15.50:0/1190559742 learned_addr learned my addr 172.21.15.50:0/1190559742 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa85affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925e60 0x556cad9346d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49786/0 (socket says 172.21.15.50:49786) 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa853fff640 1 --2- 172.21.15.50:0/1190559742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad934c10 0x556cad937000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa8608d8640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa8480a3dc0 0x7fa8480a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa85b7fe640 1 -- 172.21.15.50:0/1190559742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad934c10 msgr2=0x556cad937000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa85b7fe640 1 --2- 172.21.15.50:0/1190559742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad934c10 0x556cad937000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa85b7fe640 1 -- 172.21.15.50:0/1190559742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925e60 msgr2=0x556cad9346d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa85b7fe640 1 --2- 172.21.15.50:0/1190559742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925e60 0x556cad9346d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa85b7fe640 1 -- 172.21.15.50:0/1190559742 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cad59d210 con 0x556cad925540 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa8608d8640 1 -- 172.21.15.50:0/2918903838 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fa84c0f0790 con 0x7fa85c1021c0 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa85b7fe640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa8480a3dc0 0x7fa8480a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.921+0000 7fa85b7fe640 1 --2- 172.21.15.50:0/1190559742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 0x556cad925920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa854011590 tx=0x7fa854011a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0b96891961e04bc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.922+0000 7fa851ffb640 1 -- 172.21.15.50:0/1190559742 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa854017050 con 0x556cad925540 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.922+0000 7fa851ffb640 1 -- 172.21.15.50:0/1190559742 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa85401bb90 con 0x556cad925540 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.922+0000 7fa851ffb640 1 -- 172.21.15.50:0/1190559742 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa854025e10 con 0x556cad925540 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.922+0000 7fa8645f2740 1 -- 172.21.15.50:0/1190559742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 msgr2=0x556cad925920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.922+0000 7fa8645f2740 1 --2- 172.21.15.50:0/1190559742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 0x556cad925920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa854011590 tx=0x7fa854011a50 comp rx=0 tx=0).stop 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.922+0000 7fa85b7fe640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa8480a3dc0 0x7fa8480a6280 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa854022000 tx=0x7fa854023000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.922+0000 7fa8645f2740 1 -- 172.21.15.50:0/1190559742 shutdown_connections 2024-04-23T18:21:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.922+0000 7fa8645f2740 1 --2- 172.21.15.50:0/1190559742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad934c10 0x556cad937000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.922+0000 7fa8645f2740 1 --2- 172.21.15.50:0/1190559742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925e60 0x556cad9346d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.922+0000 7fa8645f2740 1 --2- 172.21.15.50:0/1190559742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 0x556cad925920 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.922+0000 7fa8645f2740 1 -- 172.21.15.50:0/1190559742 >> 172.21.15.50:0/1190559742 conn(0x556cad921b30 msgr2=0x556cad921f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa8645f2740 1 -- 172.21.15.50:0/1190559742 shutdown_connections 2024-04-23T18:21:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa8645f2740 1 -- 172.21.15.50:0/1190559742 wait complete. 2024-04-23T18:21:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa8645f2740 1 -- start start 2024-04-23T18:21:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa8645f2740 1 Processor -- start 2024-04-23T18:21:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa8645f2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 0x556cad9344e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa8645f2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 0x556cad931840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa8645f2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad934c10 0x556cad931d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa8645f2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556cad937840 con 0x556cad934c10 2024-04-23T18:21:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa8645f2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556cad8250b0 con 0x556cad925540 2024-04-23T18:21:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa8645f2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556cad9379c0 con 0x556cad925e60 2024-04-23T18:21:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa8645f2740 10 client.0 _ll_register_callbacks cb 0x556cad812d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa85b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad934c10 0x556cad931d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa85affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 0x556cad9344e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa853fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 0x556cad931840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa85affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 0x556cad9344e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:55196/0 (socket says 172.21.15.50:55196) 2024-04-23T18:21:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa853fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 0x556cad931840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:42604/0 (socket says 172.21.15.50:42604) 2024-04-23T18:21:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.923+0000 7fa85affd640 1 -- 172.21.15.50:0/4107521395 learned_addr learned my addr 172.21.15.50:0/4107521395 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa85b7fe640 1 -- 172.21.15.50:0/4107521395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 msgr2=0x556cad931840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa85b7fe640 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 0x556cad931840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa85b7fe640 1 -- 172.21.15.50:0/4107521395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 msgr2=0x556cad9344e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa85b7fe640 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 0x556cad9344e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa85b7fe640 1 -- 172.21.15.50:0/4107521395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cad669620 con 0x556cad934c10 2024-04-23T18:21:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa85affd640 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 0x556cad9344e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa853fff640 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 0x556cad931840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa85b7fe640 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad934c10 0x556cad931d80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa854005c10 tx=0x7fa854025f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a3bf8c805b9432 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa83affd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:21:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa83affd640 1 -- 172.21.15.50:0/4107521395 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85402a040 con 0x556cad934c10 2024-04-23T18:21:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa83affd640 1 -- 172.21.15.50:0/4107521395 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fa8540049c0 con 0x556cad934c10 2024-04-23T18:21:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa8645f2740 1 -- 172.21.15.50:0/4107521395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa85401bb90 con 0x556cad934c10 2024-04-23T18:21:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa83affd640 1 -- 172.21.15.50:0/4107521395 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa854017020 con 0x556cad934c10 2024-04-23T18:21:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa8645f2740 10 client.15849 fetch_fsmap learned FSMap version 23 2024-04-23T18:21:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.924+0000 7fa8645f2740 1 -- 172.21.15.50:0/4107521395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556cad943770 con 0x556cad934c10 2024-04-23T18:21:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.925+0000 7fa83affd640 1 -- 172.21.15.50:0/4107521395 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa85401bb90 con 0x556cad934c10 2024-04-23T18:21:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.925+0000 7fa83affd640 1 -- 172.21.15.50:0/4107521395 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fa854036030 con 0x556cad934c10 2024-04-23T18:21:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.925+0000 7fa8645f2740 10 client.15849 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:21:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.925+0000 7fa8645f2740 10 client.15849 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.925+0000 7fa8645f2740 20 client.15849 populate_metadata read hostname 'smithi050' 2024-04-23T18:21:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.925+0000 7fa8645f2740 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cad944c80 0x556cad947090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.925+0000 7fa8645f2740 4 client.15849 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.925+0000 7fa8645f2740 1 -- 172.21.15.50:0/4107521395 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cad898390 con 0x556cad944c80 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.925+0000 7fa85affd640 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cad944c80 0x556cad947090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.925+0000 7fa83affd640 10 client.15849 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.925+0000 7fa85affd640 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cad944c80 0x556cad947090 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5ed3f85fe18b689 server_cookie=fe65abaf9043d04 in_seq=0 out_seq=0 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.926+0000 7fa83affd640 1 -- 172.21.15.50:0/4107521395 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556cad898390 con 0x556cad944c80 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.926+0000 7fa83affd640 10 client.15849 handle_command_reply: tid=0 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.926+0000 7fa8645f2740 1 client.15849 shutdown 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.926+0000 7fa8645f2740 1 -- 172.21.15.50:0/4107521395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad934c10 msgr2=0x556cad931d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.926+0000 7fa8645f2740 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad934c10 0x556cad931d80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa854005c10 tx=0x7fa854025f40 comp rx=0 tx=0).stop 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.926+0000 7fa8645f2740 1 -- 172.21.15.50:0/4107521395 shutdown_connections 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.926+0000 7fa8645f2740 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cad944c80 0x556cad947090 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.926+0000 7fa8645f2740 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad934c10 0x556cad931d80 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.926+0000 7fa8645f2740 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 0x556cad931840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.926+0000 7fa83affd640 0 client.15849 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.926+0000 7fa8645f2740 1 --2- 172.21.15.50:0/4107521395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925540 0x556cad9344e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.927+0000 7fa8645f2740 1 -- 172.21.15.50:0/4107521395 >> 172.21.15.50:0/4107521395 conn(0x556cad921b30 msgr2=0x556cad9279a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.927+0000 7fa8645f2740 1 -- 172.21.15.50:0/4107521395 shutdown_connections 2024-04-23T18:21:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.927+0000 7fa8645f2740 1 -- 172.21.15.50:0/4107521395 wait complete. 2024-04-23T18:21:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.927+0000 7fa8645f2740 20 client.15849 trim_cache size 0 max 16384 2024-04-23T18:21:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.942+0000 7fa8645f2740 1 Processor -- start 2024-04-23T18:21:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.942+0000 7fa8645f2740 1 -- start start 2024-04-23T18:21:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.942+0000 7fa8645f2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925540 0x556cad940700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.942+0000 7fa8645f2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 0x556cad94c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.942+0000 7fa8645f2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad934c10 0x556cad94cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.942+0000 7fa8645f2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1ae240 con 0x556cad925540 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.942+0000 7fa8645f2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1abb10 con 0x556cad934c10 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.942+0000 7fa8645f2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1a9980 con 0x556cad925e60 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.942+0000 7fa85affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925540 0x556cad940700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa85affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925540 0x556cad940700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49804/0 (socket says 172.21.15.50:49804) 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa85affd640 1 -- 172.21.15.50:0/149349025 learned_addr learned my addr 172.21.15.50:0/149349025 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa853fff640 1 --2- 172.21.15.50:0/149349025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 0x556cad94c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa85b7fe640 1 --2- 172.21.15.50:0/149349025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad934c10 0x556cad94cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa853fff640 1 -- 172.21.15.50:0/149349025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad934c10 msgr2=0x556cad94cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa853fff640 1 --2- 172.21.15.50:0/149349025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad934c10 0x556cad94cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa853fff640 1 -- 172.21.15.50:0/149349025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925540 msgr2=0x556cad940700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa853fff640 1 --2- 172.21.15.50:0/149349025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925540 0x556cad940700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa853fff640 1 -- 172.21.15.50:0/149349025 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cad92fd90 con 0x556cad925e60 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa85affd640 1 --2- 172.21.15.50:0/149349025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925540 0x556cad940700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa853fff640 1 --2- 172.21.15.50:0/149349025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 0x556cad94c9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa85c1087a0 tx=0x7fa844005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5ac602099072206 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa8517fa640 1 -- 172.21.15.50:0/149349025 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844022070 con 0x556cad925e60 2024-04-23T18:21:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa8517fa640 1 -- 172.21.15.50:0/149349025 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8440035f0 con 0x556cad925e60 2024-04-23T18:21:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.943+0000 7fa8517fa640 1 -- 172.21.15.50:0/149349025 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84401d6a0 con 0x556cad925e60 2024-04-23T18:21:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.944+0000 7fa8645f2740 1 -- 172.21.15.50:0/149349025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 msgr2=0x556cad94c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.944+0000 7fa8645f2740 1 --2- 172.21.15.50:0/149349025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 0x556cad94c9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa85c1087a0 tx=0x7fa844005f90 comp rx=0 tx=0).stop 2024-04-23T18:21:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.944+0000 7fa8645f2740 1 -- 172.21.15.50:0/149349025 shutdown_connections 2024-04-23T18:21:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.944+0000 7fa8645f2740 1 --2- 172.21.15.50:0/149349025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad934c10 0x556cad94cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.944+0000 7fa8645f2740 1 --2- 172.21.15.50:0/149349025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925e60 0x556cad94c9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.944+0000 7fa8645f2740 1 --2- 172.21.15.50:0/149349025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad925540 0x556cad940700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.944+0000 7fa8645f2740 1 -- 172.21.15.50:0/149349025 >> 172.21.15.50:0/149349025 conn(0x556cad921b30 msgr2=0x556cad930110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.944+0000 7fa8645f2740 1 -- 172.21.15.50:0/149349025 shutdown_connections 2024-04-23T18:21:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.944+0000 7fa8645f2740 1 -- 172.21.15.50:0/149349025 wait complete. 2024-04-23T18:21:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.944+0000 7fa8645f2740 1 -- start start 2024-04-23T18:21:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa8645f2740 1 Processor -- start 2024-04-23T18:21:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa8645f2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925540 0x556cad93d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa8645f2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925e60 0x556cad92efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa8645f2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad934c10 0x556cad92dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa8645f2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1a9fd0 con 0x556cad934c10 2024-04-23T18:21:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa8645f2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1ab440 con 0x556cad925e60 2024-04-23T18:21:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa8645f2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1aa6a0 con 0x556cad925540 2024-04-23T18:21:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa8645f2740 10 client.0 _ll_register_callbacks cb 0x7fa844009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa85affd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925540 0x556cad93d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa85b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad934c10 0x556cad92dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa85affd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925540 0x556cad93d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:42630/0 (socket says 172.21.15.50:42630) 2024-04-23T18:21:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa85affd640 1 -- 172.21.15.50:0/3432936987 learned_addr learned my addr 172.21.15.50:0/3432936987 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa853fff640 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925e60 0x556cad92efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa85affd640 1 -- 172.21.15.50:0/3432936987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925e60 msgr2=0x556cad92efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa85affd640 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925e60 0x556cad92efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa85affd640 1 -- 172.21.15.50:0/3432936987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad934c10 msgr2=0x556cad92dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa85affd640 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad934c10 0x556cad92dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa85affd640 1 -- 172.21.15.50:0/3432936987 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cad92f880 con 0x556cad925540 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa853fff640 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925e60 0x556cad92efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa85affd640 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925540 0x556cad93d0e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa84c0b91c0 tx=0x7fa84c0bc210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=135501621647e1cf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.945+0000 7fa83affd640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa83affd640 1 -- 172.21.15.50:0/3432936987 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c0f6040 con 0x556cad925540 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa83affd640 1 -- 172.21.15.50:0/3432936987 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fa84c0a70c0 con 0x556cad925540 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa8645f2740 1 -- 172.21.15.50:0/3432936987 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8440035f0 con 0x556cad925540 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa83affd640 1 -- 172.21.15.50:0/3432936987 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa84c049040 con 0x556cad925540 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa8645f2740 10 client.25549 fetch_fsmap learned FSMap version 23 2024-04-23T18:21:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa8645f2740 1 -- 172.21.15.50:0/3432936987 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556cad9aa880 con 0x556cad925540 2024-04-23T18:21:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa83affd640 1 -- 172.21.15.50:0/3432936987 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8440035f0 con 0x556cad925540 2024-04-23T18:21:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa83affd640 1 -- 172.21.15.50:0/3432936987 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fa84c0cbce0 con 0x556cad925540 2024-04-23T18:21:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa8645f2740 10 client.25549 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:21:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa8645f2740 10 client.25549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa8645f2740 20 client.25549 populate_metadata read hostname 'smithi050' 2024-04-23T18:21:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa8645f2740 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cad9ab010 0x556cad9ad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa8645f2740 4 client.25549 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:21:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa8645f2740 1 -- 172.21.15.50:0/3432936987 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556cad892200 con 0x556cad9ab010 2024-04-23T18:21:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.946+0000 7fa853fff640 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cad9ab010 0x556cad9ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:23.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.947+0000 7fa83affd640 10 client.25549 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:23.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:23.947+0000 7fa853fff640 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cad9ab010 0x556cad9ad430 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26fc89357f36b541 server_cookie=723228e1ad6cf310 in_seq=0 out_seq=0 2024-04-23T18:21:23.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:23 smithi050 ceph-mon[29187]: pgmap v89: 97 pgs: 97 active+clean; 73 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 428 op/s 2024-04-23T18:21:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.004+0000 7fa83affd640 1 -- 172.21.15.50:0/3432936987 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556cad892200 con 0x556cad9ab010 2024-04-23T18:21:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.004+0000 7fa83affd640 10 client.25549 handle_command_reply: tid=0 2024-04-23T18:21:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.004+0000 7fa8645f2740 1 client.25549 shutdown 2024-04-23T18:21:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.004+0000 7fa8645f2740 1 -- 172.21.15.50:0/3432936987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925540 msgr2=0x556cad93d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.004+0000 7fa8645f2740 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925540 0x556cad93d0e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa84c0b91c0 tx=0x7fa84c0bc210 comp rx=0 tx=0).stop 2024-04-23T18:21:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.005+0000 7fa8645f2740 1 -- 172.21.15.50:0/3432936987 shutdown_connections 2024-04-23T18:21:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.005+0000 7fa8645f2740 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cad9ab010 0x556cad9ad430 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.005+0000 7fa8645f2740 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cad934c10 0x556cad92dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.005+0000 7fa8645f2740 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cad925e60 0x556cad92efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.005+0000 7fa83affd640 0 client.25549 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.005+0000 7fa8645f2740 1 --2- 172.21.15.50:0/3432936987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cad925540 0x556cad93d0e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.005+0000 7fa8645f2740 1 -- 172.21.15.50:0/3432936987 >> 172.21.15.50:0/3432936987 conn(0x556cad921b30 msgr2=0x556cad92c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.005+0000 7fa8645f2740 1 -- 172.21.15.50:0/3432936987 shutdown_connections 2024-04-23T18:21:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.005+0000 7fa8645f2740 1 -- 172.21.15.50:0/3432936987 wait complete. 2024-04-23T18:21:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.005+0000 7fa8645f2740 20 client.25549 trim_cache size 0 max 16384 2024-04-23T18:21:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:21:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:21:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:21:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:21:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.006+0000 7fa8618da640 1 -- 172.21.15.50:0/2918903838 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa8480a3dc0 msgr2=0x7fa8480a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.006+0000 7fa8618da640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa8480a3dc0 0x7fa8480a6280 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa854022000 tx=0x7fa854023000 comp rx=0 tx=0).stop 2024-04-23T18:21:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.006+0000 7fa8618da640 1 -- 172.21.15.50:0/2918903838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa85c1021c0 msgr2=0x7fa85c1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.006+0000 7fa8618da640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa85c1021c0 0x7fa85c1986f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa84c005940 tx=0x7fa84c0040c0 comp rx=0 tx=0).stop 2024-04-23T18:21:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.006+0000 7fa8618da640 1 -- 172.21.15.50:0/2918903838 shutdown_connections 2024-04-23T18:21:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.006+0000 7fa8618da640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa8480a3dc0 0x7fa8480a6280 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.006+0000 7fa8618da640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa85c19cfb0 0x7fa85c19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.006+0000 7fa8618da640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa85c107290 0x7fa85c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.006+0000 7fa8618da640 1 --2- 172.21.15.50:0/2918903838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa85c1021c0 0x7fa85c1986f0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.006+0000 7fa8618da640 1 -- 172.21.15.50:0/2918903838 >> 172.21.15.50:0/2918903838 conn(0x7fa85c0fda20 msgr2=0x7fa85c0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.006+0000 7fa8618da640 1 -- 172.21.15.50:0/2918903838 shutdown_connections 2024-04-23T18:21:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:24.006+0000 7fa8618da640 1 -- 172.21.15.50:0/2918903838 wait complete. 2024-04-23T18:21:24.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:23 smithi177 ceph-mon[32308]: pgmap v89: 97 pgs: 97 active+clean; 73 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 428 op/s 2024-04-23T18:21:24.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:23 smithi188 ceph-mon[32694]: pgmap v89: 97 pgs: 97 active+clean; 73 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 428 op/s 2024-04-23T18:21:24.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 190414 8.83 MB/sec warmup 118 sec latency 37.638 ms 2024-04-23T18:21:25.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 192144 8.85 MB/sec warmup 119 sec latency 50.719 ms 2024-04-23T18:21:25.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:25 smithi050 ceph-mon[29187]: pgmap v90: 97 pgs: 97 active+clean; 73 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 276 op/s 2024-04-23T18:21:26.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:25 smithi177 ceph-mon[32308]: pgmap v90: 97 pgs: 97 active+clean; 73 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 276 op/s 2024-04-23T18:21:26.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:25 smithi188 ceph-mon[32694]: pgmap v90: 97 pgs: 97 active+clean; 73 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 276 op/s 2024-04-23T18:21:27.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 196506 11.01 MB/sec execute 1 sec latency 16.363 ms 2024-04-23T18:21:27.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:27 smithi050 ceph-mon[29187]: pgmap v91: 97 pgs: 97 active+clean; 70 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 430 op/s 2024-04-23T18:21:28.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:27 smithi177 ceph-mon[32308]: pgmap v91: 97 pgs: 97 active+clean; 70 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 430 op/s 2024-04-23T18:21:28.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:27 smithi188 ceph-mon[32694]: pgmap v91: 97 pgs: 97 active+clean; 70 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 430 op/s 2024-04-23T18:21:28.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 198897 12.38 MB/sec execute 2 sec latency 19.657 ms 2024-04-23T18:21:29.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 200781 10.26 MB/sec execute 3 sec latency 70.272 ms 2024-04-23T18:21:29.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:29 smithi050 ceph-mon[29187]: pgmap v92: 97 pgs: 97 active+clean; 70 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 287 op/s 2024-04-23T18:21:30.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:29 smithi177 ceph-mon[32308]: pgmap v92: 97 pgs: 97 active+clean; 70 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 287 op/s 2024-04-23T18:21:30.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:29 smithi188 ceph-mon[32694]: pgmap v92: 97 pgs: 97 active+clean; 70 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 287 op/s 2024-04-23T18:21:30.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 202753 11.05 MB/sec execute 4 sec latency 25.477 ms 2024-04-23T18:21:31.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 205081 11.21 MB/sec execute 5 sec latency 18.300 ms 2024-04-23T18:21:31.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:31 smithi050 ceph-mon[29187]: pgmap v93: 97 pgs: 97 active+clean; 70 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 287 op/s 2024-04-23T18:21:32.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:31 smithi177 ceph-mon[32308]: pgmap v93: 97 pgs: 97 active+clean; 70 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 287 op/s 2024-04-23T18:21:32.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:31 smithi188 ceph-mon[32694]: pgmap v93: 97 pgs: 97 active+clean; 70 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 287 op/s 2024-04-23T18:21:32.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 206837 10.88 MB/sec execute 6 sec latency 14.365 ms 2024-04-23T18:21:33.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 209396 11.58 MB/sec execute 7 sec latency 12.260 ms 2024-04-23T18:21:33.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:33 smithi050 ceph-mon[29187]: pgmap v94: 97 pgs: 97 active+clean; 76 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 420 op/s 2024-04-23T18:21:34.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:33 smithi177 ceph-mon[32308]: pgmap v94: 97 pgs: 97 active+clean; 76 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 420 op/s 2024-04-23T18:21:34.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:33 smithi188 ceph-mon[32694]: pgmap v94: 97 pgs: 97 active+clean; 76 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 420 op/s 2024-04-23T18:21:34.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 211339 11.01 MB/sec execute 8 sec latency 14.406 ms 2024-04-23T18:21:35.198 INFO:tasks.workunit.client.0.smithi050.stdout: 1 213269 11.28 MB/sec execute 9 sec latency 67.352 ms 2024-04-23T18:21:35.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:35 smithi050 ceph-mon[29187]: pgmap v95: 97 pgs: 97 active+clean; 76 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 287 op/s 2024-04-23T18:21:36.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:35 smithi177 ceph-mon[32308]: pgmap v95: 97 pgs: 97 active+clean; 76 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 287 op/s 2024-04-23T18:21:36.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:35 smithi188 ceph-mon[32694]: pgmap v95: 97 pgs: 97 active+clean; 76 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 287 op/s 2024-04-23T18:21:36.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 215342 10.93 MB/sec execute 10 sec latency 29.396 ms 2024-04-23T18:21:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.096+0000 7fb2445ef640 1 -- 172.21.15.50:0/35801193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb23c1021c0 msgr2=0x7fb23c1025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.096+0000 7fb2445ef640 1 --2- 172.21.15.50:0/35801193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb23c1021c0 0x7fb23c1025a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb238009900 tx=0x7fb23802a410 comp rx=0 tx=0).stop 2024-04-23T18:21:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.096+0000 7fb2445ef640 1 -- 172.21.15.50:0/35801193 shutdown_connections 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.096+0000 7fb2445ef640 1 --2- 172.21.15.50:0/35801193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb23c107290 0x7fb23c10ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.096+0000 7fb2445ef640 1 --2- 172.21.15.50:0/35801193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb23c102b70 0x7fb23c106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.096+0000 7fb2445ef640 1 --2- 172.21.15.50:0/35801193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb23c1021c0 0x7fb23c1025a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.096+0000 7fb2445ef640 1 -- 172.21.15.50:0/35801193 >> 172.21.15.50:0/35801193 conn(0x7fb23c0fda20 msgr2=0x7fb23c0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb2445ef640 1 -- 172.21.15.50:0/35801193 shutdown_connections 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb2445ef640 1 -- 172.21.15.50:0/35801193 wait complete. 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb2445ef640 1 Processor -- start 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb2445ef640 1 -- start start 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb2445ef640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb23c1021c0 0x7fb23c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb2445ef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb23c107290 0x7fb23c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb2445ef640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb23c19cfb0 0x7fb23c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb2445ef640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c1121f0 con 0x7fb23c19cfb0 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb2445ef640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c112070 con 0x7fb23c107290 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb2445ef640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c112370 con 0x7fb23c1021c0 2024-04-23T18:21:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb242b8c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb23c19cfb0 0x7fb23c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb242b8c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb23c19cfb0 0x7fb23c19d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60556/0 (socket says 172.21.15.50:60556) 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb241b8a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb23c107290 0x7fb23c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.097+0000 7fb242b8c640 1 -- 172.21.15.50:0/3040146104 learned_addr learned my addr 172.21.15.50:0/3040146104 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.098+0000 7fb24238b640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb23c1021c0 0x7fb23c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.098+0000 7fb242b8c640 1 -- 172.21.15.50:0/3040146104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb23c1021c0 msgr2=0x7fb23c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.098+0000 7fb242b8c640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb23c1021c0 0x7fb23c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.098+0000 7fb242b8c640 1 -- 172.21.15.50:0/3040146104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb23c107290 msgr2=0x7fb23c198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.098+0000 7fb242b8c640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb23c107290 0x7fb23c198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.098+0000 7fb242b8c640 1 -- 172.21.15.50:0/3040146104 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23c19daa0 con 0x7fb23c19cfb0 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.098+0000 7fb242b8c640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb23c19cfb0 0x7fb23c19d460 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb23400eb10 tx=0x7fb23400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.098+0000 7fb2337fe640 1 -- 172.21.15.50:0/3040146104 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb234004b20 con 0x7fb23c19cfb0 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.098+0000 7fb2445ef640 1 -- 172.21.15.50:0/3040146104 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb23c19dd90 con 0x7fb23c19cfb0 2024-04-23T18:21:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.098+0000 7fb2337fe640 1 -- 172.21.15.50:0/3040146104 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb234004cc0 con 0x7fb23c19cfb0 2024-04-23T18:21:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.098+0000 7fb2337fe640 1 -- 172.21.15.50:0/3040146104 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb234010670 con 0x7fb23c19cfb0 2024-04-23T18:21:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.098+0000 7fb2445ef640 1 -- 172.21.15.50:0/3040146104 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb23c1a5530 con 0x7fb23c19cfb0 2024-04-23T18:21:37.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.100+0000 7fb2337fe640 1 -- 172.21.15.50:0/3040146104 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb234005020 con 0x7fb23c19cfb0 2024-04-23T18:21:37.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.101+0000 7fb2473d7740 1 Processor -- start 2024-04-23T18:21:37.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.101+0000 7fb2473d7740 1 -- start start 2024-04-23T18:21:37.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb2473d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2620 0x5611f72c2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb2473d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72c2f40 0x5611f72cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb2473d7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72cd800 0x5611f72cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb2473d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5611f7227970 con 0x5611f72c2f40 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb2473d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5611f71c44a0 con 0x5611f72c2620 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb2473d7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5611f7223930 con 0x5611f72cd800 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb2337fe640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2280a3dc0 0x7fb2280a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb241b8a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2620 0x5611f72c2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb241b8a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2620 0x5611f72c2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:57442/0 (socket says 172.21.15.50:57442) 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb241b8a640 1 -- 172.21.15.50:0/780195366 learned_addr learned my addr 172.21.15.50:0/780195366 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb241b8a640 1 --2- 172.21.15.50:0/780195366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72cd800 0x5611f72cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb242b8c640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2280a3dc0 0x7fb2280a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb24238b640 1 --2- 172.21.15.50:0/780195366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72c2f40 0x5611f72cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb2337fe640 1 -- 172.21.15.50:0/3040146104 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fb2340d0cf0 con 0x7fb23c19cfb0 2024-04-23T18:21:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb241b8a640 1 -- 172.21.15.50:0/780195366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72cd800 msgr2=0x5611f72cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb241b8a640 1 --2- 172.21.15.50:0/780195366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72cd800 0x5611f72cfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb241b8a640 1 -- 172.21.15.50:0/780195366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72c2f40 msgr2=0x5611f72cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb241b8a640 1 --2- 172.21.15.50:0/780195366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72c2f40 0x5611f72cd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb241b8a640 1 -- 172.21.15.50:0/780195366 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611f6f3a210 con 0x5611f72c2620 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb241b8a640 1 --2- 172.21.15.50:0/780195366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2620 0x5611f72c2a00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009fd0 tx=0x7fb22c005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=735e1deb8285e728 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb242b8c640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2280a3dc0 0x7fb2280a6280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb2340ce500 tx=0x7fb23406e6e0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.102+0000 7fb230bfe640 1 -- 172.21.15.50:0/780195366 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c004030 con 0x5611f72c2620 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.103+0000 7fb230bfe640 1 -- 172.21.15.50:0/780195366 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb22c00fc40 con 0x5611f72c2620 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.103+0000 7fb230bfe640 1 -- 172.21.15.50:0/780195366 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c003830 con 0x5611f72c2620 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.103+0000 7fb2473d7740 1 -- 172.21.15.50:0/780195366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2620 msgr2=0x5611f72c2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.103+0000 7fb2473d7740 1 --2- 172.21.15.50:0/780195366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2620 0x5611f72c2a00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009fd0 tx=0x7fb22c005850 comp rx=0 tx=0).stop 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.103+0000 7fb2473d7740 1 -- 172.21.15.50:0/780195366 shutdown_connections 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.103+0000 7fb2473d7740 1 --2- 172.21.15.50:0/780195366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72cd800 0x5611f72cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.103+0000 7fb2473d7740 1 --2- 172.21.15.50:0/780195366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72c2f40 0x5611f72cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.103+0000 7fb2473d7740 1 --2- 172.21.15.50:0/780195366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2620 0x5611f72c2a00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.103+0000 7fb2473d7740 1 -- 172.21.15.50:0/780195366 >> 172.21.15.50:0/780195366 conn(0x5611f72beb30 msgr2=0x5611f72bef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.103+0000 7fb2473d7740 1 -- 172.21.15.50:0/780195366 shutdown_connections 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.103+0000 7fb2473d7740 1 -- 172.21.15.50:0/780195366 wait complete. 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb2473d7740 1 -- start start 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb2473d7740 1 Processor -- start 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb2473d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2620 0x5611f72cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb2473d7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2f40 0x5611f72ca470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb2473d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 0x5611f72ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb2473d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5611f72d0430 con 0x5611f72cd800 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb2473d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5611f71c20b0 con 0x5611f72c2620 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb2473d7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5611f72d05b0 con 0x5611f72c2f40 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb2473d7740 10 client.0 _ll_register_callbacks cb 0x5611f71afd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb241b8a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2620 0x5611f72cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb241b8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 0x5611f72ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb24238b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2f40 0x5611f72ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb241b8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 0x5611f72ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60572/0 (socket says 172.21.15.50:60572) 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb24238b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2f40 0x5611f72ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:57888/0 (socket says 172.21.15.50:57888) 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.104+0000 7fb241b8a640 1 -- 172.21.15.50:0/2316346051 learned_addr learned my addr 172.21.15.50:0/2316346051 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb241b8a640 1 -- 172.21.15.50:0/2316346051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2f40 msgr2=0x5611f72ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb241b8a640 1 --2- 172.21.15.50:0/2316346051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2f40 0x5611f72ca470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb241b8a640 1 -- 172.21.15.50:0/2316346051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2620 msgr2=0x5611f72cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb241b8a640 1 --2- 172.21.15.50:0/2316346051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2620 0x5611f72cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb241b8a640 1 -- 172.21.15.50:0/2316346051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611f7006620 con 0x5611f72cd800 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb24238b640 1 --2- 172.21.15.50:0/2316346051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2f40 0x5611f72ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb241b8a640 1 --2- 172.21.15.50:0/2316346051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 0x5611f72ca9f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009e50 tx=0x7fb22c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6468f2a93a844f17 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb225ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb225ffb640 1 -- 172.21.15.50:0/2316346051 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c026070 con 0x5611f72cd800 2024-04-23T18:21:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb225ffb640 1 -- 172.21.15.50:0/2316346051 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fb22c020540 con 0x5611f72cd800 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb2473d7740 1 -- 172.21.15.50:0/2316346051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb22c00fc40 con 0x5611f72cd800 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb225ffb640 1 -- 172.21.15.50:0/2316346051 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb22c029040 con 0x5611f72cd800 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb2473d7740 10 client.15867 fetch_fsmap learned FSMap version 23 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb2473d7740 1 -- 172.21.15.50:0/2316346051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5611f72dc380 con 0x5611f72cd800 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb225ffb640 1 -- 172.21.15.50:0/2316346051 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb22c00fc40 con 0x5611f72cd800 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb225ffb640 1 -- 172.21.15.50:0/2316346051 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fb22c038850 con 0x5611f72cd800 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb2473d7740 10 client.15867 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.105+0000 7fb2473d7740 10 client.15867 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.106+0000 7fb2473d7740 20 client.15867 populate_metadata read hostname 'smithi050' 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.106+0000 7fb2473d7740 1 --2- 172.21.15.50:0/2316346051 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5611f72dd6d0 0x5611f72dfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.106+0000 7fb2473d7740 4 client.15867 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.106+0000 7fb2473d7740 1 -- 172.21.15.50:0/2316346051 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611f7235390 con 0x5611f72dd6d0 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.106+0000 7fb24238b640 1 --2- 172.21.15.50:0/2316346051 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5611f72dd6d0 0x5611f72dfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.106+0000 7fb24238b640 1 --2- 172.21.15.50:0/2316346051 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5611f72dd6d0 0x5611f72dfac0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e223843d0af32abb server_cookie=914355d67265f40d in_seq=0 out_seq=0 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.106+0000 7fb225ffb640 10 client.15867 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb225ffb640 1 -- 172.21.15.50:0/2316346051 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5611f7235390 con 0x5611f72dd6d0 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb225ffb640 10 client.15867 handle_command_reply: tid=0 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb2473d7740 1 client.15867 shutdown 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb2473d7740 1 -- 172.21.15.50:0/2316346051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 msgr2=0x5611f72ca9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb2473d7740 1 --2- 172.21.15.50:0/2316346051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 0x5611f72ca9f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009e50 tx=0x7fb22c005f90 comp rx=0 tx=0).stop 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb2473d7740 1 -- 172.21.15.50:0/2316346051 shutdown_connections 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb2473d7740 1 --2- 172.21.15.50:0/2316346051 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5611f72dd6d0 0x5611f72dfac0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb2473d7740 1 --2- 172.21.15.50:0/2316346051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 0x5611f72ca9f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb2473d7740 1 --2- 172.21.15.50:0/2316346051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2f40 0x5611f72ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb225ffb640 0 client.15867 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb2473d7740 1 --2- 172.21.15.50:0/2316346051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2620 0x5611f72cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb2473d7740 1 -- 172.21.15.50:0/2316346051 >> 172.21.15.50:0/2316346051 conn(0x5611f72beb30 msgr2=0x5611f72c48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb2473d7740 1 -- 172.21.15.50:0/2316346051 shutdown_connections 2024-04-23T18:21:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb2473d7740 1 -- 172.21.15.50:0/2316346051 wait complete. 2024-04-23T18:21:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.107+0000 7fb2473d7740 20 client.15867 trim_cache size 0 max 16384 2024-04-23T18:21:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.123+0000 7fb2473d7740 1 Processor -- start 2024-04-23T18:21:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb2473d7740 1 -- start start 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb2473d7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2620 0x5611f72d92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb2473d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2f40 0x5611f72cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb2473d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 0x5611f72e51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb2473d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c1b0470 con 0x5611f72cd800 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb2473d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c1add40 con 0x5611f72c2f40 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb2473d7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c1abb90 con 0x5611f72c2620 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb241b8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 0x5611f72e51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb241b8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 0x5611f72e51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60574/0 (socket says 172.21.15.50:60574) 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb241b8a640 1 -- 172.21.15.50:0/894047189 learned_addr learned my addr 172.21.15.50:0/894047189 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb241b8a640 1 --2- 172.21.15.50:0/894047189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2620 0x5611f72d92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb24238b640 1 --2- 172.21.15.50:0/894047189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2f40 0x5611f72cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb241b8a640 1 -- 172.21.15.50:0/894047189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2620 msgr2=0x5611f72d92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb241b8a640 1 --2- 172.21.15.50:0/894047189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2620 0x5611f72d92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb241b8a640 1 -- 172.21.15.50:0/894047189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2f40 msgr2=0x5611f72cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb241b8a640 1 --2- 172.21.15.50:0/894047189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2f40 0x5611f72cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.124+0000 7fb241b8a640 1 -- 172.21.15.50:0/894047189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611f72d0310 con 0x5611f72cd800 2024-04-23T18:21:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.125+0000 7fb241b8a640 1 --2- 172.21.15.50:0/894047189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 0x5611f72e51e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0059f0 tx=0x7fb22c009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89323089575a7adf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.125+0000 7fb227fff640 1 -- 172.21.15.50:0/894047189 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c026070 con 0x5611f72cd800 2024-04-23T18:21:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.125+0000 7fb227fff640 1 -- 172.21.15.50:0/894047189 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb22c00fe50 con 0x5611f72cd800 2024-04-23T18:21:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.125+0000 7fb227fff640 1 -- 172.21.15.50:0/894047189 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c038a70 con 0x5611f72cd800 2024-04-23T18:21:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.126+0000 7fb2473d7740 1 -- 172.21.15.50:0/894047189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 msgr2=0x5611f72e51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.126+0000 7fb2473d7740 1 --2- 172.21.15.50:0/894047189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 0x5611f72e51e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0059f0 tx=0x7fb22c009de0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.126+0000 7fb2473d7740 1 -- 172.21.15.50:0/894047189 shutdown_connections 2024-04-23T18:21:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.126+0000 7fb2473d7740 1 --2- 172.21.15.50:0/894047189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72cd800 0x5611f72e51e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.126+0000 7fb2473d7740 1 --2- 172.21.15.50:0/894047189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72c2f40 0x5611f72cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.126+0000 7fb2473d7740 1 --2- 172.21.15.50:0/894047189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2620 0x5611f72d92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.126+0000 7fb2473d7740 1 -- 172.21.15.50:0/894047189 >> 172.21.15.50:0/894047189 conn(0x5611f72beb30 msgr2=0x5611f72c59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.126+0000 7fb2473d7740 1 -- 172.21.15.50:0/894047189 shutdown_connections 2024-04-23T18:21:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.126+0000 7fb2473d7740 1 -- 172.21.15.50:0/894047189 wait complete. 2024-04-23T18:21:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.126+0000 7fb2473d7740 1 -- start start 2024-04-23T18:21:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.126+0000 7fb2473d7740 1 Processor -- start 2024-04-23T18:21:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb2473d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72c2620 0x5611f72d5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb2473d7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2f40 0x5611f72d5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb2473d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72cd800 0x5611f72c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb2473d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c1ac200 con 0x5611f72c2620 2024-04-23T18:21:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb2473d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c1ad670 con 0x5611f72cd800 2024-04-23T18:21:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb2473d7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c1ac8d0 con 0x5611f72c2f40 2024-04-23T18:21:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb2473d7740 10 client.0 _ll_register_callbacks cb 0x7fb22c020400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb241b8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72c2620 0x5611f72d5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb241b8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72c2620 0x5611f72d5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60586/0 (socket says 172.21.15.50:60586) 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb241b8a640 1 -- 172.21.15.50:0/1553563248 learned_addr learned my addr 172.21.15.50:0/1553563248 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb24238b640 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2f40 0x5611f72d5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb241b8a640 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72cd800 0x5611f72c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb241b8a640 1 -- 172.21.15.50:0/1553563248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2f40 msgr2=0x5611f72d5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb241b8a640 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2f40 0x5611f72d5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb241b8a640 1 -- 172.21.15.50:0/1553563248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72cd800 msgr2=0x5611f72c6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb241b8a640 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72cd800 0x5611f72c6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb241b8a640 1 -- 172.21.15.50:0/1553563248 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611f72c7320 con 0x5611f72c2620 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb241b8a640 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72c2620 0x5611f72d5800 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009db0 tx=0x7fb22c009b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e840613c1598a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.127+0000 7fb225ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb225ffb640 1 -- 172.21.15.50:0/1553563248 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c026070 con 0x5611f72c2620 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb225ffb640 1 -- 172.21.15.50:0/1553563248 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fb22c0386f0 con 0x5611f72c2620 2024-04-23T18:21:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb2473d7740 1 -- 172.21.15.50:0/1553563248 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb22c00fe50 con 0x5611f72c2620 2024-04-23T18:21:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb225ffb640 1 -- 172.21.15.50:0/1553563248 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb22c0388f0 con 0x5611f72c2620 2024-04-23T18:21:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb2473d7740 10 client.15873 fetch_fsmap learned FSMap version 23 2024-04-23T18:21:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb2473d7740 1 -- 172.21.15.50:0/1553563248 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5611f734bd70 con 0x5611f72c2620 2024-04-23T18:21:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb225ffb640 1 -- 172.21.15.50:0/1553563248 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb22c00fe50 con 0x5611f72c2620 2024-04-23T18:21:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb225ffb640 1 -- 172.21.15.50:0/1553563248 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fb22c005420 con 0x5611f72c2620 2024-04-23T18:21:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb2473d7740 10 client.15873 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:21:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb2473d7740 10 client.15873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb2473d7740 20 client.15873 populate_metadata read hostname 'smithi050' 2024-04-23T18:21:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb2473d7740 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5611f734c430 0x5611f734e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb2473d7740 4 client.15873 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:21:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb2473d7740 1 -- 172.21.15.50:0/1553563248 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5611f722f200 con 0x5611f734c430 2024-04-23T18:21:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.128+0000 7fb24238b640 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5611f734c430 0x5611f734e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.129+0000 7fb225ffb640 10 client.15873 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.129+0000 7fb24238b640 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5611f734c430 0x5611f734e850 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b050d2ac532ded69 server_cookie=4aaea0567287efb5 in_seq=0 out_seq=0 2024-04-23T18:21:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.174+0000 7fb225ffb640 1 -- 172.21.15.50:0/1553563248 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5611f722f200 con 0x5611f734c430 2024-04-23T18:21:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.174+0000 7fb225ffb640 10 client.15873 handle_command_reply: tid=0 2024-04-23T18:21:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.174+0000 7fb2473d7740 1 client.15873 shutdown 2024-04-23T18:21:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.174+0000 7fb2473d7740 1 -- 172.21.15.50:0/1553563248 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72c2620 msgr2=0x5611f72d5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.174+0000 7fb2473d7740 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72c2620 0x5611f72d5800 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009db0 tx=0x7fb22c009b10 comp rx=0 tx=0).stop 2024-04-23T18:21:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.174+0000 7fb2473d7740 1 -- 172.21.15.50:0/1553563248 shutdown_connections 2024-04-23T18:21:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.174+0000 7fb2473d7740 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5611f734c430 0x5611f734e850 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.175+0000 7fb2473d7740 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611f72cd800 0x5611f72c6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.175+0000 7fb2473d7740 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5611f72c2f40 0x5611f72d5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.175+0000 7fb225ffb640 0 client.15873 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.175+0000 7fb2473d7740 1 --2- 172.21.15.50:0/1553563248 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5611f72c2620 0x5611f72d5800 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.175+0000 7fb2473d7740 1 -- 172.21.15.50:0/1553563248 >> 172.21.15.50:0/1553563248 conn(0x5611f72beb30 msgr2=0x5611f72c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.175+0000 7fb2473d7740 1 -- 172.21.15.50:0/1553563248 shutdown_connections 2024-04-23T18:21:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.175+0000 7fb2473d7740 1 -- 172.21.15.50:0/1553563248 wait complete. 2024-04-23T18:21:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.175+0000 7fb2473d7740 20 client.15873 trim_cache size 0 max 16384 2024-04-23T18:21:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:21:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:21:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:21:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:21:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.176+0000 7fb2445ef640 1 -- 172.21.15.50:0/3040146104 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2280a3dc0 msgr2=0x7fb2280a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.176+0000 7fb2445ef640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2280a3dc0 0x7fb2280a6280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb2340ce500 tx=0x7fb23406e6e0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.176+0000 7fb2445ef640 1 -- 172.21.15.50:0/3040146104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb23c19cfb0 msgr2=0x7fb23c19d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.176+0000 7fb2445ef640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb23c19cfb0 0x7fb23c19d460 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb23400eb10 tx=0x7fb23400c2d0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.177+0000 7fb2445ef640 1 -- 172.21.15.50:0/3040146104 shutdown_connections 2024-04-23T18:21:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.177+0000 7fb2445ef640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2280a3dc0 0x7fb2280a6280 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.177+0000 7fb2445ef640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb23c19cfb0 0x7fb23c19d460 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.177+0000 7fb2445ef640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb23c107290 0x7fb23c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.177+0000 7fb2445ef640 1 --2- 172.21.15.50:0/3040146104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb23c1021c0 0x7fb23c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.177+0000 7fb2445ef640 1 -- 172.21.15.50:0/3040146104 >> 172.21.15.50:0/3040146104 conn(0x7fb23c0fda20 msgr2=0x7fb23c0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.177+0000 7fb2445ef640 1 -- 172.21.15.50:0/3040146104 shutdown_connections 2024-04-23T18:21:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:37.177+0000 7fb2445ef640 1 -- 172.21.15.50:0/3040146104 wait complete. 2024-04-23T18:21:37.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 217155 11.01 MB/sec execute 11 sec latency 37.011 ms 2024-04-23T18:21:37.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:37 smithi050 ceph-mon[29187]: pgmap v96: 97 pgs: 97 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.2 MiB/s wr, 419 op/s 2024-04-23T18:21:38.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:37 smithi177 ceph-mon[32308]: pgmap v96: 97 pgs: 97 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.2 MiB/s wr, 419 op/s 2024-04-23T18:21:38.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:37 smithi188 ceph-mon[32694]: pgmap v96: 97 pgs: 97 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.2 MiB/s wr, 419 op/s 2024-04-23T18:21:38.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 219342 11.11 MB/sec execute 12 sec latency 64.706 ms 2024-04-23T18:21:38.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:38 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:21:39.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:38 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:21:39.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:38 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:21:39.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 221038 10.91 MB/sec execute 13 sec latency 18.823 ms 2024-04-23T18:21:39.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:39 smithi050 ceph-mon[29187]: pgmap v97: 97 pgs: 97 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 264 op/s 2024-04-23T18:21:39.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:39 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:21:39.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:39 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:21:39.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:39 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:21:40.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:39 smithi177 ceph-mon[32308]: pgmap v97: 97 pgs: 97 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 264 op/s 2024-04-23T18:21:40.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:39 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:21:40.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:39 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:21:40.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:39 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:21:40.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:39 smithi188 ceph-mon[32694]: pgmap v97: 97 pgs: 97 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 264 op/s 2024-04-23T18:21:40.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:39 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:21:40.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:39 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:21:40.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:39 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:21:40.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 223340 11.03 MB/sec execute 14 sec latency 26.063 ms 2024-04-23T18:21:41.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 225047 10.94 MB/sec execute 15 sec latency 21.204 ms 2024-04-23T18:21:41.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:41 smithi050 ceph-mon[29187]: pgmap v98: 97 pgs: 97 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 264 op/s 2024-04-23T18:21:42.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:41 smithi177 ceph-mon[32308]: pgmap v98: 97 pgs: 97 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 264 op/s 2024-04-23T18:21:42.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:41 smithi188 ceph-mon[32694]: pgmap v98: 97 pgs: 97 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 264 op/s 2024-04-23T18:21:42.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 227459 11.13 MB/sec execute 16 sec latency 10.437 ms 2024-04-23T18:21:43.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 229629 10.95 MB/sec execute 17 sec latency 17.784 ms 2024-04-23T18:21:43.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:43 smithi050 ceph-mon[29187]: pgmap v99: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.1 MiB/s wr, 421 op/s 2024-04-23T18:21:44.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:43 smithi177 ceph-mon[32308]: pgmap v99: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.1 MiB/s wr, 421 op/s 2024-04-23T18:21:44.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:43 smithi188 ceph-mon[32694]: pgmap v99: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.1 MiB/s wr, 421 op/s 2024-04-23T18:21:44.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 231594 11.01 MB/sec execute 18 sec latency 13.335 ms 2024-04-23T18:21:45.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 233954 11.14 MB/sec execute 19 sec latency 43.332 ms 2024-04-23T18:21:45.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:45 smithi050 ceph-mon[29187]: pgmap v100: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 288 op/s 2024-04-23T18:21:46.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:45 smithi177 ceph-mon[32308]: pgmap v100: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 288 op/s 2024-04-23T18:21:46.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:45 smithi188 ceph-mon[32694]: pgmap v100: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 288 op/s 2024-04-23T18:21:46.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 235635 11.07 MB/sec execute 20 sec latency 13.704 ms 2024-04-23T18:21:47.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 238005 11.20 MB/sec execute 21 sec latency 10.781 ms 2024-04-23T18:21:47.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:47 smithi050 ceph-mon[29187]: pgmap v101: 97 pgs: 97 active+clean; 83 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 417 op/s 2024-04-23T18:21:48.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:47 smithi177 ceph-mon[32308]: pgmap v101: 97 pgs: 97 active+clean; 83 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 417 op/s 2024-04-23T18:21:48.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:47 smithi188 ceph-mon[32694]: pgmap v101: 97 pgs: 97 active+clean; 83 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 417 op/s 2024-04-23T18:21:48.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 240086 11.02 MB/sec execute 22 sec latency 18.929 ms 2024-04-23T18:21:48.402 INFO:tasks.quiescer.fs.[cephfs]:Total duration (55.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:21:48.402 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.85 2024-04-23T18:21:48.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:21:48.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.513+0000 7f7e58c10640 1 -- 172.21.15.50:0/3999769322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e5410c570 msgr2=0x7f7e5410e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.513+0000 7f7e58c10640 1 --2- 172.21.15.50:0/3999769322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e5410c570 0x7f7e5410e9e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009ca0 tx=0x7f7e4c02a950 comp rx=0 tx=0).stop 2024-04-23T18:21:48.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.513+0000 7f7e58c10640 1 -- 172.21.15.50:0/3999769322 shutdown_connections 2024-04-23T18:21:48.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.513+0000 7f7e58c10640 1 --2- 172.21.15.50:0/3999769322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e5410c570 0x7f7e5410e9e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.513+0000 7f7e58c10640 1 --2- 172.21.15.50:0/3999769322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e54103b50 0x7f7e5410c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.513+0000 7f7e58c10640 1 --2- 172.21.15.50:0/3999769322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e54103230 0x7f7e54103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.513+0000 7f7e58c10640 1 -- 172.21.15.50:0/3999769322 >> 172.21.15.50:0/3999769322 conn(0x7f7e54077fb0 msgr2=0x7f7e540783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.513+0000 7f7e58c10640 1 -- 172.21.15.50:0/3999769322 shutdown_connections 2024-04-23T18:21:48.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.513+0000 7f7e58c10640 1 -- 172.21.15.50:0/3999769322 wait complete. 2024-04-23T18:21:48.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e58c10640 1 Processor -- start 2024-04-23T18:21:48.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e58c10640 1 -- start start 2024-04-23T18:21:48.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e58c10640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e54103230 0x7f7e54100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e58c10640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e54103b50 0x7f7e541013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e58c10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5410c570 0x7f7e54101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e58c10640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7e541121d0 con 0x7f7e54103230 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e58c10640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7e54112050 con 0x7f7e5410c570 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e58c10640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7e54112350 con 0x7f7e54103b50 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e5259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e54103230 0x7f7e54100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e5259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e54103230 0x7f7e54100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49912/0 (socket says 172.21.15.50:49912) 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e5259c640 1 -- 172.21.15.50:0/4270923297 learned_addr learned my addr 172.21.15.50:0/4270923297 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e52d9d640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5410c570 0x7f7e54101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e51d9b640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e54103b50 0x7f7e541013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e5259c640 1 -- 172.21.15.50:0/4270923297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e54103b50 msgr2=0x7f7e541013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e5259c640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e54103b50 0x7f7e541013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e5259c640 1 -- 172.21.15.50:0/4270923297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5410c570 msgr2=0x7f7e54101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e5259c640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5410c570 0x7f7e54101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.514+0000 7f7e5259c640 1 -- 172.21.15.50:0/4270923297 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e541a6bb0 con 0x7f7e54103230 2024-04-23T18:21:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.515+0000 7f7e5259c640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e54103230 0x7f7e54100e80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400e9f0 tx=0x7f7e4400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.515+0000 7f7e437fe640 1 -- 172.21.15.50:0/4270923297 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4400cd80 con 0x7f7e54103230 2024-04-23T18:21:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.515+0000 7f7e58c10640 1 -- 172.21.15.50:0/4270923297 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e541a6ea0 con 0x7f7e54103230 2024-04-23T18:21:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.515+0000 7f7e437fe640 1 -- 172.21.15.50:0/4270923297 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e440054d0 con 0x7f7e54103230 2024-04-23T18:21:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.515+0000 7f7e437fe640 1 -- 172.21.15.50:0/4270923297 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44010690 con 0x7f7e54103230 2024-04-23T18:21:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.515+0000 7f7e58c10640 1 -- 172.21.15.50:0/4270923297 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e541a7360 con 0x7f7e54103230 2024-04-23T18:21:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.516+0000 7f7e58c10640 1 -- 172.21.15.50:0/4270923297 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e540691c0 con 0x7f7e54103230 2024-04-23T18:21:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.517+0000 7f7e437fe640 1 -- 172.21.15.50:0/4270923297 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7e44004e50 con 0x7f7e54103230 2024-04-23T18:21:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.518+0000 7f7e437fe640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e3c0a3d50 0x7f7e3c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.518+0000 7f7e51d9b640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e3c0a3d50 0x7f7e3c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.518+0000 7f7e437fe640 1 -- 172.21.15.50:0/4270923297 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7f7e440d0ba0 con 0x7f7e54103230 2024-04-23T18:21:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.518+0000 7f7e51d9b640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e3c0a3d50 0x7f7e3c0a6210 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7e541028c0 tx=0x7f7e48005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.523+0000 7f7e437fe640 1 -- 172.21.15.50:0/4270923297 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e4409d250 con 0x7f7e54103230 2024-04-23T18:21:48.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.762+0000 7f7e58c10640 1 -- 172.21.15.50:0/4270923297 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e54107160 con 0x7f7e54103230 2024-04-23T18:21:48.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.762+0000 7f7e437fe640 1 -- 172.21.15.50:0/4270923297 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11878 (secure 0 0 0) 0x7f7e4409cb50 con 0x7f7e54103230 2024-04-23T18:21:48.756 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:21:48.756 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:19:58.806076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1048893636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1048893636},{"type":"v1","addr":"172.21.15.177:6837","nonce":1048893636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:21:48.756 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 23 2024-04-23T18:21:48.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.764+0000 7f7e58c10640 1 -- 172.21.15.50:0/4270923297 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e3c0a3d50 msgr2=0x7f7e3c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.764+0000 7f7e58c10640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e3c0a3d50 0x7f7e3c0a6210 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7e541028c0 tx=0x7f7e48005f90 comp rx=0 tx=0).stop 2024-04-23T18:21:48.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.764+0000 7f7e58c10640 1 -- 172.21.15.50:0/4270923297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e54103230 msgr2=0x7f7e54100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.764+0000 7f7e58c10640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e54103230 0x7f7e54100e80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400e9f0 tx=0x7f7e4400edf0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.764+0000 7f7e58c10640 1 -- 172.21.15.50:0/4270923297 shutdown_connections 2024-04-23T18:21:48.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.764+0000 7f7e58c10640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e3c0a3d50 0x7f7e3c0a6210 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.764+0000 7f7e58c10640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5410c570 0x7f7e54101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.764+0000 7f7e58c10640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e54103b50 0x7f7e541013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.764+0000 7f7e58c10640 1 --2- 172.21.15.50:0/4270923297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e54103230 0x7f7e54100e80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.765+0000 7f7e58c10640 1 -- 172.21.15.50:0/4270923297 >> 172.21.15.50:0/4270923297 conn(0x7f7e54077fb0 msgr2=0x7f7e540fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.765+0000 7f7e58c10640 1 -- 172.21.15.50:0/4270923297 shutdown_connections 2024-04-23T18:21:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.765+0000 7f7e58c10640 1 -- 172.21.15.50:0/4270923297 wait complete. 2024-04-23T18:21:48.770 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 87.85 --await-for 42.0' 2024-04-23T18:21:48.770 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 87.85 --await-for 42.0 2024-04-23T18:21:48.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 -- 172.21.15.50:0/539560654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4ac106190 msgr2=0x7fd4ac10dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 --2- 172.21.15.50:0/539560654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4ac106190 0x7fd4ac10dd60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009ca0 tx=0x7fd4a402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 -- 172.21.15.50:0/539560654 shutdown_connections 2024-04-23T18:21:48.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 --2- 172.21.15.50:0/539560654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4ac106190 0x7fd4ac10dd60 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 --2- 172.21.15.50:0/539560654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4ac101ac0 0x7fd4ac105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 --2- 172.21.15.50:0/539560654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4ac101110 0x7fd4ac1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 -- 172.21.15.50:0/539560654 >> 172.21.15.50:0/539560654 conn(0x7fd4ac077fb0 msgr2=0x7fd4ac0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 -- 172.21.15.50:0/539560654 shutdown_connections 2024-04-23T18:21:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 -- 172.21.15.50:0/539560654 wait complete. 2024-04-23T18:21:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 Processor -- start 2024-04-23T18:21:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 -- start start 2024-04-23T18:21:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4ac101110 0x7fd4ac198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.893+0000 7fd4b138e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4ac106190 0x7fd4ac198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4b138e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4ac19cff0 0x7fd4ac19d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4b138e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac111100 con 0x7fd4ac106190 2024-04-23T18:21:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4b138e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac110f80 con 0x7fd4ac101110 2024-04-23T18:21:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4b138e640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac111280 con 0x7fd4ac19cff0 2024-04-23T18:21:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4a3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4ac106190 0x7fd4ac198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4a3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4ac106190 0x7fd4ac198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49938/0 (socket says 172.21.15.50:49938) 2024-04-23T18:21:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4aaffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4ac101110 0x7fd4ac198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4a3fff640 1 -- 172.21.15.50:0/592277947 learned_addr learned my addr 172.21.15.50:0/592277947 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4ab7fe640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4ac19cff0 0x7fd4ac19d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4a3fff640 1 -- 172.21.15.50:0/592277947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4ac19cff0 msgr2=0x7fd4ac19d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4a3fff640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4ac19cff0 0x7fd4ac19d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4a3fff640 1 -- 172.21.15.50:0/592277947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4ac101110 msgr2=0x7fd4ac198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4a3fff640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4ac101110 0x7fd4ac198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4a3fff640 1 -- 172.21.15.50:0/592277947 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ac19dae0 con 0x7fd4ac106190 2024-04-23T18:21:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4ab7fe640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4ac19cff0 0x7fd4ac19d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4a3fff640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4ac106190 0x7fd4ac198c70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd49400e9f0 tx=0x7fd49400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4a8ff9640 1 -- 172.21.15.50:0/592277947 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49400ccc0 con 0x7fd4ac106190 2024-04-23T18:21:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4b138e640 1 -- 172.21.15.50:0/592277947 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4ac19ddd0 con 0x7fd4ac106190 2024-04-23T18:21:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4a8ff9640 1 -- 172.21.15.50:0/592277947 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4940054d0 con 0x7fd4ac106190 2024-04-23T18:21:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4a8ff9640 1 -- 172.21.15.50:0/592277947 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd494010430 con 0x7fd4ac106190 2024-04-23T18:21:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.894+0000 7fd4b138e640 1 -- 172.21.15.50:0/592277947 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4ac1a55c0 con 0x7fd4ac106190 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.897+0000 7fd4a8ff9640 1 -- 172.21.15.50:0/592277947 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd4940046c0 con 0x7fd4ac106190 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.897+0000 7fd4b417f740 1 Processor -- start 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.897+0000 7fd4b417f740 1 -- start start 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.897+0000 7fd4b417f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee858600a0 0x55ee85860480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.897+0000 7fd4b417f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee8586ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.897+0000 7fd4b417f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee8586b090 0x55ee8586d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.897+0000 7fd4b417f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ee857c5370 con 0x55ee858600a0 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.897+0000 7fd4b417f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ee857622e0 con 0x55ee858609c0 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.897+0000 7fd4b417f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ee857c1330 con 0x55ee8586b090 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4ab7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee858600a0 0x55ee85860480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4aaffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee8586ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4a3fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee8586b090 0x55ee8586d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4a8ff9640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd4980a3d20 0x7fd4980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4aaffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee8586ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:52424/0 (socket says 172.21.15.50:52424) 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4aaffd640 1 -- 172.21.15.50:0/494089184 learned_addr learned my addr 172.21.15.50:0/494089184 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4ab7fe640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd4980a3d20 0x7fd4980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4aaffd640 1 -- 172.21.15.50:0/494089184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee8586b090 msgr2=0x55ee8586d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4aaffd640 1 --2- 172.21.15.50:0/494089184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee8586b090 0x55ee8586d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4aaffd640 1 -- 172.21.15.50:0/494089184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee858600a0 msgr2=0x55ee85860480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4aaffd640 1 --2- 172.21.15.50:0/494089184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee858600a0 0x55ee85860480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4aaffd640 1 -- 172.21.15.50:0/494089184 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee855a3d80 con 0x55ee858609c0 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4a8ff9640 1 -- 172.21.15.50:0/592277947 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fd4940d0af0 con 0x7fd4ac106190 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4aaffd640 1 --2- 172.21.15.50:0/494089184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee8586ab50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009cf0 tx=0x7fd49c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e40f89bf5ffe4a47 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4ab7fe640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd4980a3d20 0x7fd4980a61e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4005fe0 tx=0x7fd4a402a8c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4a17fa640 1 -- 172.21.15.50:0/494089184 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49c004230 con 0x55ee858609c0 2024-04-23T18:21:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4a17fa640 1 -- 172.21.15.50:0/494089184 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd49c0043d0 con 0x55ee858609c0 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.898+0000 7fd4a17fa640 1 -- 172.21.15.50:0/494089184 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49c019880 con 0x55ee858609c0 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.899+0000 7fd4b417f740 1 -- 172.21.15.50:0/494089184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 msgr2=0x55ee8586ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.899+0000 7fd4b417f740 1 --2- 172.21.15.50:0/494089184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee8586ab50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009cf0 tx=0x7fd49c011810 comp rx=0 tx=0).stop 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.899+0000 7fd4b417f740 1 -- 172.21.15.50:0/494089184 shutdown_connections 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.899+0000 7fd4b417f740 1 --2- 172.21.15.50:0/494089184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee8586b090 0x55ee8586d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.899+0000 7fd4b417f740 1 --2- 172.21.15.50:0/494089184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee8586ab50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.899+0000 7fd4b417f740 1 --2- 172.21.15.50:0/494089184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee858600a0 0x55ee85860480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.899+0000 7fd4b417f740 1 -- 172.21.15.50:0/494089184 >> 172.21.15.50:0/494089184 conn(0x55ee8585c3c0 msgr2=0x55ee8585c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.899+0000 7fd4b417f740 1 -- 172.21.15.50:0/494089184 shutdown_connections 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.899+0000 7fd4b417f740 1 -- 172.21.15.50:0/494089184 wait complete. 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4b417f740 1 -- start start 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4b417f740 1 Processor -- start 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4b417f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 0x55ee85868010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4b417f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85868550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4b417f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee85868a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4b417f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ee8586de80 con 0x55ee8586b090 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4b417f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ee8575ffb0 con 0x55ee858609c0 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4b417f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ee8586e000 con 0x55ee858600a0 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4b417f740 10 client.0 _ll_register_callbacks cb 0x55ee8575f9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4a3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85868550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4ab7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee85868a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4a3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85868550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:52436/0 (socket says 172.21.15.50:52436) 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4a3fff640 1 -- 172.21.15.50:0/1054256743 learned_addr learned my addr 172.21.15.50:0/1054256743 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4aaffd640 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 0x55ee85868010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4ab7fe640 1 -- 172.21.15.50:0/1054256743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 msgr2=0x55ee85868010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4ab7fe640 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 0x55ee85868010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4ab7fe640 1 -- 172.21.15.50:0/1054256743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 msgr2=0x55ee85868550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4ab7fe640 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85868550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4ab7fe640 1 -- 172.21.15.50:0/1054256743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee85604510 con 0x55ee8586b090 2024-04-23T18:21:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.900+0000 7fd4a3fff640 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85868550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4ab7fe640 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee85868a90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4006010 tx=0x7fd4a40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f8d83db099a7e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4927fc640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:21:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4927fc640 1 -- 172.21.15.50:0/1054256743 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4003e30 con 0x55ee8586b090 2024-04-23T18:21:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4927fc640 1 -- 172.21.15.50:0/1054256743 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fd4a4004620 con 0x55ee8586b090 2024-04-23T18:21:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4b417f740 1 -- 172.21.15.50:0/1054256743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd49c0043d0 con 0x55ee8586b090 2024-04-23T18:21:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4927fc640 1 -- 172.21.15.50:0/1054256743 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd4a4004820 con 0x55ee8586b090 2024-04-23T18:21:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4b417f740 10 client.15891 fetch_fsmap learned FSMap version 23 2024-04-23T18:21:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4b417f740 1 -- 172.21.15.50:0/1054256743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ee85497670 con 0x55ee8586b090 2024-04-23T18:21:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4927fc640 1 -- 172.21.15.50:0/1054256743 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd49c0043d0 con 0x55ee8586b090 2024-04-23T18:21:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4927fc640 1 -- 172.21.15.50:0/1054256743 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fd4a40305c0 con 0x55ee8586b090 2024-04-23T18:21:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4b417f740 10 client.15891 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:21:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4b417f740 10 client.15891 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4b417f740 20 client.15891 populate_metadata read hostname 'smithi050' 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4b417f740 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ee8587b390 0x55ee8587d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4b417f740 4 client.15891 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.901+0000 7fd4b417f740 1 -- 172.21.15.50:0/1054256743 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee857d2d90 con 0x55ee8587b390 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.902+0000 7fd4aaffd640 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ee8587b390 0x55ee8587d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.902+0000 7fd4927fc640 10 client.15891 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.902+0000 7fd4aaffd640 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ee8587b390 0x55ee8587d7f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=addb9f9d242c39e8 server_cookie=58864f57516e1732 in_seq=0 out_seq=0 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4927fc640 1 -- 172.21.15.50:0/1054256743 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee857d2d90 con 0x55ee8587b390 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4927fc640 10 client.15891 handle_command_reply: tid=0 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4b417f740 1 client.15891 shutdown 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4b417f740 1 -- 172.21.15.50:0/1054256743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 msgr2=0x55ee85868a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4b417f740 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee85868a90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4006010 tx=0x7fd4a40095c0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4b417f740 1 -- 172.21.15.50:0/1054256743 shutdown_connections 2024-04-23T18:21:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4b417f740 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ee8587b390 0x55ee8587d7f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4b417f740 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee85868a90 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4927fc640 0 client.15891 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4b417f740 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85868550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4b417f740 1 --2- 172.21.15.50:0/1054256743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 0x55ee85868010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4b417f740 1 -- 172.21.15.50:0/1054256743 >> 172.21.15.50:0/1054256743 conn(0x55ee8585c3c0 msgr2=0x55ee8585b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4b417f740 1 -- 172.21.15.50:0/1054256743 shutdown_connections 2024-04-23T18:21:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4b417f740 1 -- 172.21.15.50:0/1054256743 wait complete. 2024-04-23T18:21:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.903+0000 7fd4b417f740 20 client.15891 trim_cache size 0 max 16384 2024-04-23T18:21:48.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.922+0000 7fd4b417f740 1 Processor -- start 2024-04-23T18:21:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.922+0000 7fd4b417f740 1 -- start start 2024-04-23T18:21:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.922+0000 7fd4b417f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 0x55ee85869eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.922+0000 7fd4b417f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85862800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.922+0000 7fd4b417f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee858830c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.922+0000 7fd4b417f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac1b0470 con 0x55ee8586b090 2024-04-23T18:21:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.922+0000 7fd4b417f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac1add40 con 0x55ee858609c0 2024-04-23T18:21:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.922+0000 7fd4b417f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac1abb90 con 0x55ee858600a0 2024-04-23T18:21:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.922+0000 7fd4ab7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee858830c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.922+0000 7fd4a3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85862800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.922+0000 7fd4ab7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee858830c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49972/0 (socket says 172.21.15.50:49972) 2024-04-23T18:21:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.923+0000 7fd4ab7fe640 1 -- 172.21.15.50:0/3514832688 learned_addr learned my addr 172.21.15.50:0/3514832688 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.923+0000 7fd4aaffd640 1 --2- 172.21.15.50:0/3514832688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 0x55ee85869eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.923+0000 7fd4ab7fe640 1 -- 172.21.15.50:0/3514832688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 msgr2=0x55ee85869eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.923+0000 7fd4ab7fe640 1 --2- 172.21.15.50:0/3514832688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 0x55ee85869eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.923+0000 7fd4ab7fe640 1 -- 172.21.15.50:0/3514832688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 msgr2=0x55ee85862800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.923+0000 7fd4ab7fe640 1 --2- 172.21.15.50:0/3514832688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85862800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.923+0000 7fd4ab7fe640 1 -- 172.21.15.50:0/3514832688 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee85860da0 con 0x55ee8586b090 2024-04-23T18:21:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.923+0000 7fd4ab7fe640 1 --2- 172.21.15.50:0/3514832688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee858830c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4030990 tx=0x7fd4a4009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6fde704329f923b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.923+0000 7fd4a0ff9640 1 -- 172.21.15.50:0/3514832688 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4009b20 con 0x55ee8586b090 2024-04-23T18:21:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.923+0000 7fd4a0ff9640 1 -- 172.21.15.50:0/3514832688 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4a4063020 con 0x55ee8586b090 2024-04-23T18:21:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.923+0000 7fd4a0ff9640 1 -- 172.21.15.50:0/3514832688 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4030d60 con 0x55ee8586b090 2024-04-23T18:21:48.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.924+0000 7fd4b417f740 1 -- 172.21.15.50:0/3514832688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 msgr2=0x55ee858830c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.924+0000 7fd4b417f740 1 --2- 172.21.15.50:0/3514832688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee858830c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4030990 tx=0x7fd4a4009360 comp rx=0 tx=0).stop 2024-04-23T18:21:48.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.924+0000 7fd4b417f740 1 -- 172.21.15.50:0/3514832688 shutdown_connections 2024-04-23T18:21:48.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.924+0000 7fd4b417f740 1 --2- 172.21.15.50:0/3514832688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee858830c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.924+0000 7fd4b417f740 1 --2- 172.21.15.50:0/3514832688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85862800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.924+0000 7fd4b417f740 1 --2- 172.21.15.50:0/3514832688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 0x55ee85869eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.924+0000 7fd4b417f740 1 -- 172.21.15.50:0/3514832688 >> 172.21.15.50:0/3514832688 conn(0x55ee8585c3c0 msgr2=0x55ee85868860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.924+0000 7fd4b417f740 1 -- 172.21.15.50:0/3514832688 shutdown_connections 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.924+0000 7fd4b417f740 1 -- 172.21.15.50:0/3514832688 wait complete. 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.924+0000 7fd4b417f740 1 -- start start 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.924+0000 7fd4b417f740 1 Processor -- start 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4b417f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 0x55ee85873440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4b417f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85870220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4b417f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee85870760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4b417f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac1ac200 con 0x55ee8586b090 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4b417f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac1ad670 con 0x55ee858609c0 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4b417f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac1ac8d0 con 0x55ee858600a0 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4b417f740 10 client.0 _ll_register_callbacks cb 0x7fd4a4031920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4a3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85870220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4ab7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee85870760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4a3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85870220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:52446/0 (socket says 172.21.15.50:52446) 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4a3fff640 1 -- 172.21.15.50:0/2885777506 learned_addr learned my addr 172.21.15.50:0/2885777506 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4aaffd640 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 0x55ee85873440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4a3fff640 1 -- 172.21.15.50:0/2885777506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 msgr2=0x55ee85873440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4a3fff640 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 0x55ee85873440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4a3fff640 1 -- 172.21.15.50:0/2885777506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 msgr2=0x55ee85870760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4a3fff640 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee85870760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4a3fff640 1 -- 172.21.15.50:0/2885777506 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee8586d480 con 0x55ee858609c0 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4ab7fe640 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee85870760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4a3fff640 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85870220 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd49400c6a0 tx=0x7fd4940bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b68861f98d91c70b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4927fc640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.925+0000 7fd4927fc640 1 -- 172.21.15.50:0/2885777506 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4940b5020 con 0x55ee858609c0 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4927fc640 1 -- 172.21.15.50:0/2885777506 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fd4940a5480 con 0x55ee858609c0 2024-04-23T18:21:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4b417f740 1 -- 172.21.15.50:0/2885777506 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4a4063020 con 0x55ee858609c0 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4927fc640 1 -- 172.21.15.50:0/2885777506 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd4940a0e50 con 0x55ee858609c0 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4b417f740 10 client.25517 fetch_fsmap learned FSMap version 23 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4b417f740 1 -- 172.21.15.50:0/2885777506 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ee8586dd60 con 0x55ee858609c0 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4927fc640 1 -- 172.21.15.50:0/2885777506 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4a4063020 con 0x55ee858609c0 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4927fc640 1 -- 172.21.15.50:0/2885777506 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fd4940987f0 con 0x55ee858609c0 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4b417f740 10 client.25517 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4b417f740 10 client.25517 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4b417f740 20 client.25517 populate_metadata read hostname 'smithi050' 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4b417f740 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ee858e2cb0 0x55ee858e50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4b417f740 4 client.25517 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.85, "await_for": 42.0}] 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4b417f740 1 -- 172.21.15.50:0/2885777506 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.85, "await_for": 42.0}) -- 0x55ee857ccc00 con 0x55ee858e2cb0 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.926+0000 7fd4aaffd640 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ee858e2cb0 0x55ee858e50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.927+0000 7fd4927fc640 10 client.25517 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.927+0000 7fd4aaffd640 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ee858e2cb0 0x55ee858e50d0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf90b2768b299f27 server_cookie=9c38826a13149e13 in_seq=0 out_seq=0 2024-04-23T18:21:48.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.996+0000 7fd4927fc640 1 -- 172.21.15.50:0/2885777506 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55ee857ccc00 con 0x55ee858e2cb0 2024-04-23T18:21:48.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.996+0000 7fd4927fc640 10 client.25517 handle_command_reply: tid=0 2024-04-23T18:21:48.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b417f740 1 client.25517 shutdown 2024-04-23T18:21:48.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b417f740 1 -- 172.21.15.50:0/2885777506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 msgr2=0x55ee85870220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b417f740 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85870220 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd49400c6a0 tx=0x7fd4940bd040 comp rx=0 tx=0).stop 2024-04-23T18:21:48.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b417f740 1 -- 172.21.15.50:0/2885777506 shutdown_connections 2024-04-23T18:21:48.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b417f740 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ee858e2cb0 0x55ee858e50d0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b417f740 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ee8586b090 0x55ee85870760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b417f740 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee858609c0 0x55ee85870220 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b417f740 1 --2- 172.21.15.50:0/2885777506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ee858600a0 0x55ee85873440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4927fc640 0 client.25517 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:48.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b417f740 1 -- 172.21.15.50:0/2885777506 >> 172.21.15.50:0/2885777506 conn(0x55ee8585c3c0 msgr2=0x55ee8585ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b417f740 1 -- 172.21.15.50:0/2885777506 shutdown_connections 2024-04-23T18:21:48.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b417f740 1 -- 172.21.15.50:0/2885777506 wait complete. 2024-04-23T18:21:48.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b417f740 20 client.25517 trim_cache size 0 max 16384 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 23, 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 10, 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "cf8448b3": { 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 10, 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 87.8, 2024-04-23T18:21:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:21:48.994 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:21:48.994 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:21:48.994 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:21:48.994 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:21:48.994 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:21:48.994 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:21:48.994 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:21:48.994 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:21:48.994 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:21:48.994 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:21:48.994 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:21:48.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b138e640 1 -- 172.21.15.50:0/592277947 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd4980a3d20 msgr2=0x7fd4980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b138e640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd4980a3d20 0x7fd4980a61e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4005fe0 tx=0x7fd4a402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b138e640 1 -- 172.21.15.50:0/592277947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4ac106190 msgr2=0x7fd4ac198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.997+0000 7fd4b138e640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4ac106190 0x7fd4ac198c70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd49400e9f0 tx=0x7fd49400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.998+0000 7fd4b138e640 1 -- 172.21.15.50:0/592277947 shutdown_connections 2024-04-23T18:21:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.998+0000 7fd4b138e640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd4980a3d20 0x7fd4980a61e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.998+0000 7fd4b138e640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4ac19cff0 0x7fd4ac19d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.998+0000 7fd4b138e640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4ac106190 0x7fd4ac198c70 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.998+0000 7fd4b138e640 1 --2- 172.21.15.50:0/592277947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4ac101110 0x7fd4ac198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.998+0000 7fd4b138e640 1 -- 172.21.15.50:0/592277947 >> 172.21.15.50:0/592277947 conn(0x7fd4ac077fb0 msgr2=0x7fd4ac06a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.998+0000 7fd4b138e640 1 -- 172.21.15.50:0/592277947 shutdown_connections 2024-04-23T18:21:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:21:48.998+0000 7fd4b138e640 1 -- 172.21.15.50:0/592277947 wait complete. 2024-04-23T18:21:49.002 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'cf8448b3', quiesce took 0.6 seconds. Will release after: 27.85 2024-04-23T18:21:49.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 11.05 MB/sec execute 23 sec latency 209.858 ms 2024-04-23T18:21:49.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:49 smithi050 ceph-mon[29187]: pgmap v102: 97 pgs: 97 active+clean; 83 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 286 op/s 2024-04-23T18:21:49.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:49 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4270923297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:21:50.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:49 smithi177 ceph-mon[32308]: pgmap v102: 97 pgs: 97 active+clean; 83 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 286 op/s 2024-04-23T18:21:50.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:49 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4270923297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:21:50.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:49 smithi188 ceph-mon[32694]: pgmap v102: 97 pgs: 97 active+clean; 83 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 286 op/s 2024-04-23T18:21:50.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:49 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4270923297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:21:50.197 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 10.59 MB/sec execute 24 sec latency 1209.894 ms 2024-04-23T18:21:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.278+0000 7fe874ccf640 1 -- 172.21.15.50:0/3086750617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8701021c0 msgr2=0x7fe8701025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.278+0000 7fe874ccf640 1 --2- 172.21.15.50:0/3086750617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8701021c0 0x7fe8701025a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe860009900 tx=0x7fe86002a410 comp rx=0 tx=0).stop 2024-04-23T18:21:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.278+0000 7fe874ccf640 1 -- 172.21.15.50:0/3086750617 shutdown_connections 2024-04-23T18:21:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.278+0000 7fe874ccf640 1 --2- 172.21.15.50:0/3086750617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe870107290 0x7fe87010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.278+0000 7fe874ccf640 1 --2- 172.21.15.50:0/3086750617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe870102b70 0x7fe870106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.278+0000 7fe874ccf640 1 --2- 172.21.15.50:0/3086750617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8701021c0 0x7fe8701025a0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.278+0000 7fe874ccf640 1 -- 172.21.15.50:0/3086750617 >> 172.21.15.50:0/3086750617 conn(0x7fe8700fda40 msgr2=0x7fe8700ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.278+0000 7fe874ccf640 1 -- 172.21.15.50:0/3086750617 shutdown_connections 2024-04-23T18:21:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.278+0000 7fe874ccf640 1 -- 172.21.15.50:0/3086750617 wait complete. 2024-04-23T18:21:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.279+0000 7fe874ccf640 1 Processor -- start 2024-04-23T18:21:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.279+0000 7fe874ccf640 1 -- start start 2024-04-23T18:21:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.279+0000 7fe874ccf640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8701021c0 0x7fe8701986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.279+0000 7fe874ccf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe870107290 0x7fe870198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.279+0000 7fe874ccf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe87019cf40 0x7fe87019d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.279+0000 7fe874ccf640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701121d0 con 0x7fe87019cf40 2024-04-23T18:21:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.279+0000 7fe874ccf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe870112050 con 0x7fe870107290 2024-04-23T18:21:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.279+0000 7fe874ccf640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe870112350 con 0x7fe8701021c0 2024-04-23T18:21:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe86ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe87019cf40 0x7fe87019d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe86ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe87019cf40 0x7fe87019d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50014/0 (socket says 172.21.15.50:50014) 2024-04-23T18:21:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe86ed9d640 1 -- 172.21.15.50:0/3112047482 learned_addr learned my addr 172.21.15.50:0/3112047482 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe867fff640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe870107290 0x7fe870198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe86e59c640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8701021c0 0x7fe8701986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe86ed9d640 1 -- 172.21.15.50:0/3112047482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8701021c0 msgr2=0x7fe8701986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe86ed9d640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8701021c0 0x7fe8701986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe86ed9d640 1 -- 172.21.15.50:0/3112047482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe870107290 msgr2=0x7fe870198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe86ed9d640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe870107290 0x7fe870198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe86ed9d640 1 -- 172.21.15.50:0/3112047482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87019da30 con 0x7fe87019cf40 2024-04-23T18:21:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe86ed9d640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe87019cf40 0x7fe87019d3f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe86800eaa0 tx=0x7fe86800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe8677fe640 1 -- 172.21.15.50:0/3112047482 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868004d20 con 0x7fe87019cf40 2024-04-23T18:21:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe8677fe640 1 -- 172.21.15.50:0/3112047482 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe868005020 con 0x7fe87019cf40 2024-04-23T18:21:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe874ccf640 1 -- 172.21.15.50:0/3112047482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87019dd20 con 0x7fe87019cf40 2024-04-23T18:21:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe8677fe640 1 -- 172.21.15.50:0/3112047482 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8680116b0 con 0x7fe87019cf40 2024-04-23T18:21:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.280+0000 7fe874ccf640 1 -- 172.21.15.50:0/3112047482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8701a5510 con 0x7fe87019cf40 2024-04-23T18:21:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.283+0000 7fe8677fe640 1 -- 172.21.15.50:0/3112047482 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe8680051d0 con 0x7fe87019cf40 2024-04-23T18:21:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.283+0000 7fe8779e6740 1 Processor -- start 2024-04-23T18:21:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.283+0000 7fe8779e6740 1 -- start start 2024-04-23T18:21:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.283+0000 7fe8677fe640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe85c0a3e40 0x7fe85c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.283+0000 7fe8779e6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fc620 0x55e3257fca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.283+0000 7fe8779e6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fcf40 0x55e3258072c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.283+0000 7fe8779e6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e325809bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.283+0000 7fe8779e6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e325761970 con 0x55e3257fc620 2024-04-23T18:21:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.283+0000 7fe8779e6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e3256fe4a0 con 0x55e3257fcf40 2024-04-23T18:21:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.283+0000 7fe8779e6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e32575d930 con 0x55e325807800 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe86e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fc620 0x55e3257fca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe867fff640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe85c0a3e40 0x7fe85c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe86e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fc620 0x55e3257fca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50026/0 (socket says 172.21.15.50:50026) 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe86e59c640 1 -- 172.21.15.50:0/569410762 learned_addr learned my addr 172.21.15.50:0/569410762 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe8677fe640 1 -- 172.21.15.50:0/3112047482 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fe8680d0920 con 0x7fe87019cf40 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe86ed9d640 1 --2- 172.21.15.50:0/569410762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e325809bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe867fff640 1 --2- 172.21.15.50:0/569410762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fcf40 0x55e3258072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe86e59c640 1 -- 172.21.15.50:0/569410762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 msgr2=0x55e325809bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe86e59c640 1 --2- 172.21.15.50:0/569410762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e325809bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe86e59c640 1 -- 172.21.15.50:0/569410762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fcf40 msgr2=0x55e3258072c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe86e59c640 1 --2- 172.21.15.50:0/569410762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fcf40 0x55e3258072c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe86e59c640 1 -- 172.21.15.50:0/569410762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e325474210 con 0x55e3257fc620 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe86ed9d640 1 --2- 172.21.15.50:0/569410762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e325809bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:21:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe867fff640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe85c0a3e40 0x7fe85c0a6300 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe858005e10 tx=0x7fe85800ab10 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe86e59c640 1 --2- 172.21.15.50:0/569410762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fc620 0x55e3257fca00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe860009900 tx=0x7fe860032e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b8e8613260284a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.284+0000 7fe864ff9640 1 -- 172.21.15.50:0/569410762 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860040070 con 0x55e3257fc620 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.285+0000 7fe864ff9640 1 -- 172.21.15.50:0/569410762 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8600316a0 con 0x55e3257fc620 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.285+0000 7fe864ff9640 1 -- 172.21.15.50:0/569410762 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860035040 con 0x55e3257fc620 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.285+0000 7fe8779e6740 1 -- 172.21.15.50:0/569410762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fc620 msgr2=0x55e3257fca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.285+0000 7fe8779e6740 1 --2- 172.21.15.50:0/569410762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fc620 0x55e3257fca00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe860009900 tx=0x7fe860032e60 comp rx=0 tx=0).stop 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.285+0000 7fe8779e6740 1 -- 172.21.15.50:0/569410762 shutdown_connections 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.285+0000 7fe8779e6740 1 --2- 172.21.15.50:0/569410762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e325809bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.285+0000 7fe8779e6740 1 --2- 172.21.15.50:0/569410762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fcf40 0x55e3258072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.285+0000 7fe8779e6740 1 --2- 172.21.15.50:0/569410762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fc620 0x55e3257fca00 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.285+0000 7fe8779e6740 1 -- 172.21.15.50:0/569410762 >> 172.21.15.50:0/569410762 conn(0x55e3257f8b30 msgr2=0x55e3257f8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.285+0000 7fe8779e6740 1 -- 172.21.15.50:0/569410762 shutdown_connections 2024-04-23T18:21:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.285+0000 7fe8779e6740 1 -- 172.21.15.50:0/569410762 wait complete. 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe8779e6740 1 -- start start 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe8779e6740 1 Processor -- start 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe8779e6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e3258070e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe8779e6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 0x55e3258043d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe8779e6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e325804910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe8779e6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e32580a3d0 con 0x55e3257fcf40 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe8779e6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e3256fc0b0 con 0x55e3257fc620 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe8779e6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e32580a550 con 0x55e325807800 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe8779e6740 10 client.0 _ll_register_callbacks cb 0x55e3256e9d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe867fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 0x55e3258043d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe867fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 0x55e3258043d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50040/0 (socket says 172.21.15.50:50040) 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe867fff640 1 -- 172.21.15.50:0/3707149835 learned_addr learned my addr 172.21.15.50:0/3707149835 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe86ed9d640 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e325804910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe86e59c640 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e3258070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.286+0000 7fe867fff640 1 -- 172.21.15.50:0/3707149835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 msgr2=0x55e325804910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe867fff640 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e325804910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe867fff640 1 -- 172.21.15.50:0/3707149835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 msgr2=0x55e3258070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe867fff640 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e3258070e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe867fff640 1 -- 172.21.15.50:0/3707149835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e325540620 con 0x55e3257fcf40 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe867fff640 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 0x55e3258043d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe85800fbc0 tx=0x7fe85800a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4fb2e3f794c099 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe855ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe855ffb640 1 -- 172.21.15.50:0/3707149835 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85800ae60 con 0x55e3257fcf40 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe8779e6740 1 -- 172.21.15.50:0/3707149835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8600316a0 con 0x55e3257fcf40 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe855ffb640 1 -- 172.21.15.50:0/3707149835 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fe858010b90 con 0x55e3257fcf40 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe855ffb640 1 -- 172.21.15.50:0/3707149835 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe858010d90 con 0x55e3257fcf40 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe8779e6740 10 client.15909 fetch_fsmap learned FSMap version 23 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe8779e6740 1 -- 172.21.15.50:0/3707149835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e3258162c0 con 0x55e3257fcf40 2024-04-23T18:21:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.287+0000 7fe855ffb640 1 -- 172.21.15.50:0/3707149835 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8600316a0 con 0x55e3257fcf40 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.288+0000 7fe855ffb640 1 -- 172.21.15.50:0/3707149835 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fe85801e050 con 0x55e3257fcf40 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.288+0000 7fe8779e6740 10 client.15909 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.288+0000 7fe8779e6740 10 client.15909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.288+0000 7fe8779e6740 20 client.15909 populate_metadata read hostname 'smithi050' 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.288+0000 7fe8779e6740 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e325817c20 0x55e32581a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.288+0000 7fe8779e6740 4 client.15909 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.288+0000 7fe8779e6740 1 -- 172.21.15.50:0/3707149835 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e32576f390 con 0x55e325817c20 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.288+0000 7fe86e59c640 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e325817c20 0x55e32581a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.288+0000 7fe86e59c640 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e325817c20 0x55e32581a0e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=639b59b749d24250 server_cookie=963504cc0cca2c96 in_seq=0 out_seq=0 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.288+0000 7fe855ffb640 10 client.15909 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe855ffb640 1 -- 172.21.15.50:0/3707149835 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e32576f390 con 0x55e325817c20 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe855ffb640 10 client.15909 handle_command_reply: tid=0 2024-04-23T18:21:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe8779e6740 1 client.15909 shutdown 2024-04-23T18:21:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe8779e6740 1 -- 172.21.15.50:0/3707149835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 msgr2=0x55e3258043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe8779e6740 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 0x55e3258043d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe85800fbc0 tx=0x7fe85800a2d0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe8779e6740 1 -- 172.21.15.50:0/3707149835 shutdown_connections 2024-04-23T18:21:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe8779e6740 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e325817c20 0x55e32581a0e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe8779e6740 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e325804910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe8779e6740 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 0x55e3258043d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe855ffb640 0 client.15909 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe8779e6740 1 --2- 172.21.15.50:0/3707149835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e3258070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe8779e6740 1 -- 172.21.15.50:0/3707149835 >> 172.21.15.50:0/3707149835 conn(0x55e3257f8b30 msgr2=0x55e3257fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe8779e6740 1 -- 172.21.15.50:0/3707149835 shutdown_connections 2024-04-23T18:21:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.289+0000 7fe8779e6740 1 -- 172.21.15.50:0/3707149835 wait complete. 2024-04-23T18:21:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.290+0000 7fe8779e6740 20 client.15909 trim_cache size 0 max 16384 2024-04-23T18:21:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.304+0000 7fe8779e6740 1 Processor -- start 2024-04-23T18:21:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.304+0000 7fe8779e6740 1 -- start start 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe8779e6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e325813290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe8779e6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e3257fcf40 0x55e325806c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe8779e6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e325807800 0x55e32581f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe8779e6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701b0480 con 0x55e325807800 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe8779e6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701add50 con 0x55e3257fc620 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe8779e6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701abba0 con 0x55e3257fcf40 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe86ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e325807800 0x55e32581f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe86ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e325807800 0x55e32581f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50056/0 (socket says 172.21.15.50:50056) 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe86ed9d640 1 -- 172.21.15.50:0/4175537236 learned_addr learned my addr 172.21.15.50:0/4175537236 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe86ed9d640 1 -- 172.21.15.50:0/4175537236 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e3257fcf40 msgr2=0x55e325806c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe86ed9d640 1 --2- 172.21.15.50:0/4175537236 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e3257fcf40 0x55e325806c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe86e59c640 1 --2- 172.21.15.50:0/4175537236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e325813290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe86ed9d640 1 -- 172.21.15.50:0/4175537236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 msgr2=0x55e325813290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe86ed9d640 1 --2- 172.21.15.50:0/4175537236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e325813290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe86ed9d640 1 -- 172.21.15.50:0/4175537236 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e325802920 con 0x55e325807800 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe86ed9d640 1 --2- 172.21.15.50:0/4175537236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e325807800 0x55e32581f9f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe868099240 tx=0x7fe86809c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d04550adac5580c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe857fff640 1 -- 172.21.15.50:0/4175537236 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86801f040 con 0x55e325807800 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe857fff640 1 -- 172.21.15.50:0/4175537236 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe86809c870 con 0x55e325807800 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.305+0000 7fe857fff640 1 -- 172.21.15.50:0/4175537236 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868099070 con 0x55e325807800 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.306+0000 7fe8779e6740 1 -- 172.21.15.50:0/4175537236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e325807800 msgr2=0x55e32581f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.306+0000 7fe8779e6740 1 --2- 172.21.15.50:0/4175537236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e325807800 0x55e32581f9f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe868099240 tx=0x7fe86809c300 comp rx=0 tx=0).stop 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.306+0000 7fe8779e6740 1 -- 172.21.15.50:0/4175537236 shutdown_connections 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.306+0000 7fe8779e6740 1 --2- 172.21.15.50:0/4175537236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e325807800 0x55e32581f9f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.306+0000 7fe8779e6740 1 --2- 172.21.15.50:0/4175537236 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e3257fcf40 0x55e325806c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.306+0000 7fe8779e6740 1 --2- 172.21.15.50:0/4175537236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e325813290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.306+0000 7fe8779e6740 1 -- 172.21.15.50:0/4175537236 >> 172.21.15.50:0/4175537236 conn(0x55e3257f8b30 msgr2=0x55e3257f3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.306+0000 7fe8779e6740 1 -- 172.21.15.50:0/4175537236 shutdown_connections 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.306+0000 7fe8779e6740 1 -- 172.21.15.50:0/4175537236 wait complete. 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.306+0000 7fe8779e6740 1 -- start start 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.306+0000 7fe8779e6740 1 Processor -- start 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe8779e6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e32580f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe8779e6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 0x55e32580fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe8779e6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e32580ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe8779e6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701ac210 con 0x55e3257fcf40 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe8779e6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701ad680 con 0x55e3257fc620 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe8779e6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe8701ac8e0 con 0x55e325807800 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe8779e6740 10 client.0 _ll_register_callbacks cb 0x7fe858011260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe867fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 0x55e32580fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe867fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 0x55e32580fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50064/0 (socket says 172.21.15.50:50064) 2024-04-23T18:21:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe867fff640 1 -- 172.21.15.50:0/597693091 learned_addr learned my addr 172.21.15.50:0/597693091 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe86ed9d640 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e32580ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe86e59c640 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e32580f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe867fff640 1 -- 172.21.15.50:0/597693091 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 msgr2=0x55e32580ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe867fff640 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e32580ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe867fff640 1 -- 172.21.15.50:0/597693091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 msgr2=0x55e32580f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe867fff640 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e32580f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe867fff640 1 -- 172.21.15.50:0/597693091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e32580b550 con 0x55e3257fcf40 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe86e59c640 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e32580f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe867fff640 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 0x55e32580fcd0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x55e3258059a0 tx=0x7fe858031910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72284ac31de922b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe855ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe855ffb640 1 -- 172.21.15.50:0/597693091 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85800f790 con 0x55e3257fcf40 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe855ffb640 1 -- 172.21.15.50:0/597693091 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7fe858010b90 con 0x55e3257fcf40 2024-04-23T18:21:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.307+0000 7fe8779e6740 1 -- 172.21.15.50:0/597693091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe86809c870 con 0x55e3257fcf40 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.308+0000 7fe855ffb640 1 -- 172.21.15.50:0/597693091 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe858031c60 con 0x55e3257fcf40 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.308+0000 7fe8779e6740 10 client.15915 fetch_fsmap learned FSMap version 23 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.308+0000 7fe8779e6740 1 -- 172.21.15.50:0/597693091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e3258822b0 con 0x55e3257fcf40 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.308+0000 7fe855ffb640 1 -- 172.21.15.50:0/597693091 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe86809c870 con 0x55e3257fcf40 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.308+0000 7fe855ffb640 1 -- 172.21.15.50:0/597693091 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fe858038020 con 0x55e3257fcf40 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.308+0000 7fe8779e6740 10 client.15915 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.308+0000 7fe8779e6740 10 client.15915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.308+0000 7fe8779e6740 20 client.15915 populate_metadata read hostname 'smithi050' 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.308+0000 7fe8779e6740 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e325882a40 0x55e325884e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.308+0000 7fe8779e6740 4 client.15915 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.308+0000 7fe8779e6740 1 -- 172.21.15.50:0/597693091 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e325769200 con 0x55e325882a40 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.308+0000 7fe86e59c640 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e325882a40 0x55e325884e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.309+0000 7fe855ffb640 10 client.15915 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.309+0000 7fe86e59c640 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e325882a40 0x55e325884e60 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dd5a22051b9dbed server_cookie=471ac80fc21f1283 in_seq=0 out_seq=0 2024-04-23T18:21:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe855ffb640 1 -- 172.21.15.50:0/597693091 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e325769200 con 0x55e325882a40 2024-04-23T18:21:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe855ffb640 10 client.15915 handle_command_reply: tid=0 2024-04-23T18:21:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe8779e6740 1 client.15915 shutdown 2024-04-23T18:21:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe8779e6740 1 -- 172.21.15.50:0/597693091 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 msgr2=0x55e32580fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe8779e6740 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 0x55e32580fcd0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x55e3258059a0 tx=0x7fe858031910 comp rx=0 tx=0).stop 2024-04-23T18:21:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe8779e6740 1 -- 172.21.15.50:0/597693091 shutdown_connections 2024-04-23T18:21:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe8779e6740 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e325882a40 0x55e325884e60 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe8779e6740 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e325807800 0x55e32580ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe8779e6740 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e3257fcf40 0x55e32580fcd0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe855ffb640 0 client.15915 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:21:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe8779e6740 1 --2- 172.21.15.50:0/597693091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e3257fc620 0x55e32580f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe8779e6740 1 -- 172.21.15.50:0/597693091 >> 172.21.15.50:0/597693091 conn(0x55e3257f8b30 msgr2=0x55e3257f7420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe8779e6740 1 -- 172.21.15.50:0/597693091 shutdown_connections 2024-04-23T18:21:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe8779e6740 1 -- 172.21.15.50:0/597693091 wait complete. 2024-04-23T18:21:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.336+0000 7fe8779e6740 20 client.15915 trim_cache size 0 max 16384 2024-04-23T18:21:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:21:50.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:21:50.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:21:50.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:21:50.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.337+0000 7fe874ccf640 1 -- 172.21.15.50:0/3112047482 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe85c0a3e40 msgr2=0x7fe85c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:50.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.337+0000 7fe874ccf640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe85c0a3e40 0x7fe85c0a6300 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe858005e10 tx=0x7fe85800ab10 comp rx=0 tx=0).stop 2024-04-23T18:21:50.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.337+0000 7fe874ccf640 1 -- 172.21.15.50:0/3112047482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe87019cf40 msgr2=0x7fe87019d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:21:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.337+0000 7fe874ccf640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe87019cf40 0x7fe87019d3f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe86800eaa0 tx=0x7fe86800eea0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.337+0000 7fe874ccf640 1 -- 172.21.15.50:0/3112047482 shutdown_connections 2024-04-23T18:21:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.337+0000 7fe874ccf640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe85c0a3e40 0x7fe85c0a6300 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.337+0000 7fe874ccf640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe87019cf40 0x7fe87019d3f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.337+0000 7fe874ccf640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe870107290 0x7fe870198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.337+0000 7fe874ccf640 1 --2- 172.21.15.50:0/3112047482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8701021c0 0x7fe8701986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:21:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.337+0000 7fe874ccf640 1 -- 172.21.15.50:0/3112047482 >> 172.21.15.50:0/3112047482 conn(0x7fe8700fda40 msgr2=0x7fe8700ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:21:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.337+0000 7fe874ccf640 1 -- 172.21.15.50:0/3112047482 shutdown_connections 2024-04-23T18:21:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:21:50.337+0000 7fe874ccf640 1 -- 172.21.15.50:0/3112047482 wait complete. 2024-04-23T18:21:51.198 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 10.17 MB/sec execute 25 sec latency 2209.973 ms 2024-04-23T18:21:51.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:51 smithi050 ceph-mon[29187]: pgmap v103: 97 pgs: 97 active+clean; 83 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 286 op/s 2024-04-23T18:21:52.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:51 smithi177 ceph-mon[32308]: pgmap v103: 97 pgs: 97 active+clean; 83 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 286 op/s 2024-04-23T18:21:52.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:51 smithi188 ceph-mon[32694]: pgmap v103: 97 pgs: 97 active+clean; 83 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 286 op/s 2024-04-23T18:21:52.198 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 9.78 MB/sec execute 26 sec latency 3210.064 ms 2024-04-23T18:21:53.198 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 9.42 MB/sec execute 27 sec latency 4210.191 ms 2024-04-23T18:21:53.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:53 smithi050 ceph-mon[29187]: pgmap v104: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 382 op/s 2024-04-23T18:21:54.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:53 smithi177 ceph-mon[32308]: pgmap v104: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 382 op/s 2024-04-23T18:21:54.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:53 smithi188 ceph-mon[32694]: pgmap v104: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 382 op/s 2024-04-23T18:21:54.198 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 9.08 MB/sec execute 28 sec latency 5210.318 ms 2024-04-23T18:21:55.198 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 8.77 MB/sec execute 29 sec latency 6210.374 ms 2024-04-23T18:21:55.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:55 smithi050 ceph-mon[29187]: pgmap v105: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 226 op/s 2024-04-23T18:21:56.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:55 smithi177 ceph-mon[32308]: pgmap v105: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 226 op/s 2024-04-23T18:21:56.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:55 smithi188 ceph-mon[32694]: pgmap v105: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 226 op/s 2024-04-23T18:21:56.198 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 8.48 MB/sec execute 30 sec latency 7210.412 ms 2024-04-23T18:21:57.198 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 8.20 MB/sec execute 31 sec latency 8210.523 ms 2024-04-23T18:21:57.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:57 smithi050 ceph-mon[29187]: pgmap v106: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 256 op/s 2024-04-23T18:21:58.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:57 smithi177 ceph-mon[32308]: pgmap v106: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 256 op/s 2024-04-23T18:21:58.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:57 smithi188 ceph-mon[32694]: pgmap v106: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 256 op/s 2024-04-23T18:21:58.198 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 7.95 MB/sec execute 32 sec latency 9210.619 ms 2024-04-23T18:21:59.198 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 7.70 MB/sec execute 33 sec latency 10210.740 ms 2024-04-23T18:21:59.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:21:59 smithi050 ceph-mon[29187]: pgmap v107: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1.8 MiB/s wr, 126 op/s 2024-04-23T18:22:00.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:21:59 smithi177 ceph-mon[32308]: pgmap v107: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1.8 MiB/s wr, 126 op/s 2024-04-23T18:22:00.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:21:59 smithi188 ceph-mon[32694]: pgmap v107: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1.8 MiB/s wr, 126 op/s 2024-04-23T18:22:00.198 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 7.48 MB/sec execute 34 sec latency 11210.821 ms 2024-04-23T18:22:01.198 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 7.26 MB/sec execute 35 sec latency 12210.875 ms 2024-04-23T18:22:01.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:01 smithi050 ceph-mon[29187]: pgmap v108: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1.8 MiB/s wr, 126 op/s 2024-04-23T18:22:02.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:01 smithi177 ceph-mon[32308]: pgmap v108: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1.8 MiB/s wr, 126 op/s 2024-04-23T18:22:02.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:01 smithi188 ceph-mon[32694]: pgmap v108: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1.8 MiB/s wr, 126 op/s 2024-04-23T18:22:02.199 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 7.06 MB/sec execute 36 sec latency 13211.017 ms 2024-04-23T18:22:03.199 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 6.87 MB/sec execute 37 sec latency 14211.144 ms 2024-04-23T18:22:03.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.436+0000 7faa72b51640 1 -- 172.21.15.50:0/3002985994 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa6c101aa0 msgr2=0x7faa6c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:03.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.436+0000 7faa72b51640 1 --2- 172.21.15.50:0/3002985994 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa6c101aa0 0x7faa6c105a90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7faa58009900 tx=0x7faa5802a410 comp rx=0 tx=0).stop 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.437+0000 7faa72b51640 1 -- 172.21.15.50:0/3002985994 shutdown_connections 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.437+0000 7faa72b51640 1 --2- 172.21.15.50:0/3002985994 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa6c106170 0x7faa6c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.437+0000 7faa72b51640 1 --2- 172.21.15.50:0/3002985994 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa6c101aa0 0x7faa6c105a90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.437+0000 7faa72b51640 1 --2- 172.21.15.50:0/3002985994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa6c1010f0 0x7faa6c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.437+0000 7faa72b51640 1 -- 172.21.15.50:0/3002985994 >> 172.21.15.50:0/3002985994 conn(0x7faa6c077fd0 msgr2=0x7faa6c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.437+0000 7faa72b51640 1 -- 172.21.15.50:0/3002985994 shutdown_connections 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.437+0000 7faa72b51640 1 -- 172.21.15.50:0/3002985994 wait complete. 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.437+0000 7faa72b51640 1 Processor -- start 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.437+0000 7faa72b51640 1 -- start start 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa72b51640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa6c1010f0 0x7faa6c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa72b51640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa6c106170 0x7faa6c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa72b51640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa6c19cf80 0x7faa6c19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa72b51640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1110b0 con 0x7faa6c19cf80 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa72b51640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c110f30 con 0x7faa6c1010f0 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa72b51640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c111230 con 0x7faa6c106170 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa710ee640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa6c19cf80 0x7faa6c19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa710ee640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa6c19cf80 0x7faa6c19d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49258/0 (socket says 172.21.15.50:49258) 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa710ee640 1 -- 172.21.15.50:0/1022799769 learned_addr learned my addr 172.21.15.50:0/1022799769 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa6bfff640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa6c106170 0x7faa6c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa708ed640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa6c1010f0 0x7faa6c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa710ee640 1 -- 172.21.15.50:0/1022799769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa6c106170 msgr2=0x7faa6c198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa710ee640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa6c106170 0x7faa6c198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa710ee640 1 -- 172.21.15.50:0/1022799769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa6c1010f0 msgr2=0x7faa6c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa710ee640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa6c1010f0 0x7faa6c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa710ee640 1 -- 172.21.15.50:0/1022799769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa6c19da70 con 0x7faa6c19cf80 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa710ee640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa6c19cf80 0x7faa6c19d430 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7faa6400eaa0 tx=0x7faa6400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:03.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.438+0000 7faa69ffb640 1 -- 172.21.15.50:0/1022799769 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6400cdb0 con 0x7faa6c19cf80 2024-04-23T18:22:03.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.439+0000 7faa69ffb640 1 -- 172.21.15.50:0/1022799769 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa64005020 con 0x7faa6c19cf80 2024-04-23T18:22:03.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.439+0000 7faa72b51640 1 -- 172.21.15.50:0/1022799769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa6c19dd60 con 0x7faa6c19cf80 2024-04-23T18:22:03.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.439+0000 7faa69ffb640 1 -- 172.21.15.50:0/1022799769 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa640116d0 con 0x7faa6c19cf80 2024-04-23T18:22:03.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.439+0000 7faa72b51640 1 -- 172.21.15.50:0/1022799769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa6c1a5520 con 0x7faa6c19cf80 2024-04-23T18:22:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.441+0000 7faa69ffb640 1 -- 172.21.15.50:0/1022799769 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faa640051d0 con 0x7faa6c19cf80 2024-04-23T18:22:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa7586a740 1 Processor -- start 2024-04-23T18:22:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa7586a740 1 -- start start 2024-04-23T18:22:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa7586a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa7586a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 0x5623d17c22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa7586a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa7586a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5623d171c970 con 0x5623d17b7f40 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa7586a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5623d16b94a0 con 0x5623d17c2800 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa7586a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5623d1718930 con 0x5623d17b7620 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa69ffb640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faa5c0a3e40 0x7faa5c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa708ed640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 0x5623d17c22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa710ee640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faa5c0a3e40 0x7faa5c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa6bfff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa708ed640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 0x5623d17c22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49268/0 (socket says 172.21.15.50:49268) 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa708ed640 1 -- 172.21.15.50:0/4198069441 learned_addr learned my addr 172.21.15.50:0/4198069441 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa6bfff640 1 --2- 172.21.15.50:0/4198069441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.442+0000 7faa69ffb640 1 -- 172.21.15.50:0/1022799769 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7faa640d08a0 con 0x7faa6c19cf80 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.443+0000 7faa708ed640 1 -- 172.21.15.50:0/4198069441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 msgr2=0x5623d17b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.443+0000 7faa708ed640 1 --2- 172.21.15.50:0/4198069441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17b7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.443+0000 7faa708ed640 1 -- 172.21.15.50:0/4198069441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 msgr2=0x5623d17c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.443+0000 7faa708ed640 1 --2- 172.21.15.50:0/4198069441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.443+0000 7faa708ed640 1 -- 172.21.15.50:0/4198069441 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623d142f210 con 0x5623d17b7f40 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.443+0000 7faa6bfff640 1 --2- 172.21.15.50:0/4198069441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.443+0000 7faa710ee640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faa5c0a3e40 0x7faa5c0a6300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faa64099270 tx=0x7faa6408e040 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.443+0000 7faa708ed640 1 --2- 172.21.15.50:0/4198069441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 0x5623d17c22c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7faa60009d20 tx=0x7faa60011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4354c627d81d5b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.443+0000 7faa56ffd640 1 -- 172.21.15.50:0/4198069441 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6001e070 con 0x5623d17b7f40 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.443+0000 7faa56ffd640 1 -- 172.21.15.50:0/4198069441 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa60019ae0 con 0x5623d17b7f40 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.443+0000 7faa56ffd640 1 -- 172.21.15.50:0/4198069441 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6000ea50 con 0x5623d17b7f40 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.444+0000 7faa7586a740 1 -- 172.21.15.50:0/4198069441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 msgr2=0x5623d17c22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.444+0000 7faa7586a740 1 --2- 172.21.15.50:0/4198069441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 0x5623d17c22c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7faa60009d20 tx=0x7faa60011810 comp rx=0 tx=0).stop 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.444+0000 7faa7586a740 1 -- 172.21.15.50:0/4198069441 shutdown_connections 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.444+0000 7faa7586a740 1 --2- 172.21.15.50:0/4198069441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.444+0000 7faa7586a740 1 --2- 172.21.15.50:0/4198069441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 0x5623d17c22c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.444+0000 7faa7586a740 1 --2- 172.21.15.50:0/4198069441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.444+0000 7faa7586a740 1 -- 172.21.15.50:0/4198069441 >> 172.21.15.50:0/4198069441 conn(0x5623d17b3b30 msgr2=0x5623d17b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.444+0000 7faa7586a740 1 -- 172.21.15.50:0/4198069441 shutdown_connections 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.444+0000 7faa7586a740 1 -- 172.21.15.50:0/4198069441 wait complete. 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.444+0000 7faa7586a740 1 -- start start 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.444+0000 7faa7586a740 1 Processor -- start 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa7586a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17c2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa7586a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17b7f40 0x5623d17bf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa7586a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17c2800 0x5623d17bf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa7586a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5623d17c5430 con 0x5623d17c2800 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa7586a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5623d16b70b0 con 0x5623d17b7f40 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa7586a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5623d17c55b0 con 0x5623d17b7620 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa7586a740 10 client.0 _ll_register_callbacks cb 0x5623d16a4d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa6bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17c2800 0x5623d17bf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa6bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17c2800 0x5623d17bf9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49274/0 (socket says 172.21.15.50:49274) 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa6bfff640 1 -- 172.21.15.50:0/3310360528 learned_addr learned my addr 172.21.15.50:0/3310360528 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa6bfff640 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa708ed640 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17b7f40 0x5623d17bf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa6bfff640 1 -- 172.21.15.50:0/3310360528 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 msgr2=0x5623d17c2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa6bfff640 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17c2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa6bfff640 1 -- 172.21.15.50:0/3310360528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17b7f40 msgr2=0x5623d17bf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa6bfff640 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17b7f40 0x5623d17bf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa6bfff640 1 -- 172.21.15.50:0/3310360528 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623d14fb620 con 0x5623d17c2800 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa708ed640 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17b7f40 0x5623d17bf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa6bfff640 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17c2800 0x5623d17bf9c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7faa58009900 tx=0x7faa5802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b3299132129545 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa4bfff640 1 -- 172.21.15.50:0/3310360528 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa58035040 con 0x5623d17c2800 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.445+0000 7faa4bfff640 1 -- 172.21.15.50:0/3310360528 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7faa58004390 con 0x5623d17c2800 2024-04-23T18:22:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.446+0000 7faa7586a740 1 -- 172.21.15.50:0/3310360528 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa60019ae0 con 0x5623d17c2800 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.446+0000 7faa4bfff640 1 -- 172.21.15.50:0/3310360528 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7faa58054050 con 0x5623d17c2800 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.446+0000 7faa7586a740 10 client.15927 fetch_fsmap learned FSMap version 23 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.446+0000 7faa7586a740 1 -- 172.21.15.50:0/3310360528 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5623d17d1360 con 0x5623d17c2800 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.446+0000 7faa4bfff640 1 -- 172.21.15.50:0/3310360528 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa60019ae0 con 0x5623d17c2800 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.446+0000 7faa4bfff640 1 -- 172.21.15.50:0/3310360528 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7faa5804b070 con 0x5623d17c2800 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.446+0000 7faa7586a740 10 client.15927 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.446+0000 7faa7586a740 10 client.15927 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.446+0000 7faa7586a740 20 client.15927 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.446+0000 7faa7586a740 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623d17d2890 0x5623d17d4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.446+0000 7faa7586a740 4 client.15927 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.446+0000 7faa7586a740 1 -- 172.21.15.50:0/3310360528 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623d172a390 con 0x5623d17d2890 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.447+0000 7faa708ed640 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623d17d2890 0x5623d17d4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.447+0000 7faa4bfff640 10 client.15927 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.447+0000 7faa708ed640 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623d17d2890 0x5623d17d4d10 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=306cf595c12bf7b server_cookie=8e5bf72d3613d793 in_seq=0 out_seq=0 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa4bfff640 1 -- 172.21.15.50:0/3310360528 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623d172a390 con 0x5623d17d2890 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa4bfff640 10 client.15927 handle_command_reply: tid=0 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa7586a740 1 client.15927 shutdown 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa7586a740 1 -- 172.21.15.50:0/3310360528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17c2800 msgr2=0x5623d17bf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa7586a740 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17c2800 0x5623d17bf9c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7faa58009900 tx=0x7faa5802a410 comp rx=0 tx=0).stop 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa7586a740 1 -- 172.21.15.50:0/3310360528 shutdown_connections 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa7586a740 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623d17d2890 0x5623d17d4d10 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa7586a740 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17c2800 0x5623d17bf9c0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa4bfff640 0 client.15927 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa7586a740 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17b7f40 0x5623d17bf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa7586a740 1 --2- 172.21.15.50:0/3310360528 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17c2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa7586a740 1 -- 172.21.15.50:0/3310360528 >> 172.21.15.50:0/3310360528 conn(0x5623d17b3b30 msgr2=0x5623d17b9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa7586a740 1 -- 172.21.15.50:0/3310360528 shutdown_connections 2024-04-23T18:22:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa7586a740 1 -- 172.21.15.50:0/3310360528 wait complete. 2024-04-23T18:22:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.448+0000 7faa7586a740 20 client.15927 trim_cache size 0 max 16384 2024-04-23T18:22:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.463+0000 7faa7586a740 1 Processor -- start 2024-04-23T18:22:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.463+0000 7faa7586a740 1 -- start start 2024-04-23T18:22:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.463+0000 7faa7586a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.463+0000 7faa7586a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 0x5623d17be600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.463+0000 7faa7586a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.463+0000 7faa7586a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1b03c0 con 0x5623d17b7f40 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.463+0000 7faa7586a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1adc90 con 0x5623d17c2800 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.463+0000 7faa7586a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1abae0 con 0x5623d17b7620 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.463+0000 7faa708ed640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 0x5623d17be600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.463+0000 7faa708ed640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 0x5623d17be600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49280/0 (socket says 172.21.15.50:49280) 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.463+0000 7faa708ed640 1 -- 172.21.15.50:0/2307136930 learned_addr learned my addr 172.21.15.50:0/2307136930 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.464+0000 7faa6bfff640 1 --2- 172.21.15.50:0/2307136930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.464+0000 7faa6bfff640 1 --2- 172.21.15.50:0/2307136930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.464+0000 7faa708ed640 1 -- 172.21.15.50:0/2307136930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 msgr2=0x5623d17ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.464+0000 7faa708ed640 1 --2- 172.21.15.50:0/2307136930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.464+0000 7faa708ed640 1 -- 172.21.15.50:0/2307136930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 msgr2=0x5623d17c1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.464+0000 7faa708ed640 1 --2- 172.21.15.50:0/2307136930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17c1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.464+0000 7faa708ed640 1 -- 172.21.15.50:0/2307136930 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623d17c4f60 con 0x5623d17b7f40 2024-04-23T18:22:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.464+0000 7faa6bfff640 1 --2- 172.21.15.50:0/2307136930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17c1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.464+0000 7faa708ed640 1 --2- 172.21.15.50:0/2307136930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 0x5623d17be600 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7faa60013010 tx=0x7faa60005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a6dbc132b1a8b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.464+0000 7faa567fc640 1 -- 172.21.15.50:0/2307136930 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6001e040 con 0x5623d17b7f40 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.464+0000 7faa567fc640 1 -- 172.21.15.50:0/2307136930 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa6001f040 con 0x5623d17b7f40 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.464+0000 7faa567fc640 1 -- 172.21.15.50:0/2307136930 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6000eda0 con 0x5623d17b7f40 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.465+0000 7faa7586a740 1 -- 172.21.15.50:0/2307136930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 msgr2=0x5623d17be600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.465+0000 7faa7586a740 1 --2- 172.21.15.50:0/2307136930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 0x5623d17be600 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7faa60013010 tx=0x7faa60005d50 comp rx=0 tx=0).stop 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.465+0000 7faa7586a740 1 -- 172.21.15.50:0/2307136930 shutdown_connections 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.465+0000 7faa7586a740 1 --2- 172.21.15.50:0/2307136930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17c1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.465+0000 7faa7586a740 1 --2- 172.21.15.50:0/2307136930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7f40 0x5623d17be600 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.465+0000 7faa7586a740 1 --2- 172.21.15.50:0/2307136930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7620 0x5623d17ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.465+0000 7faa7586a740 1 -- 172.21.15.50:0/2307136930 >> 172.21.15.50:0/2307136930 conn(0x5623d17b3b30 msgr2=0x5623d17ba140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.465+0000 7faa7586a740 1 -- 172.21.15.50:0/2307136930 shutdown_connections 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.465+0000 7faa7586a740 1 -- 172.21.15.50:0/2307136930 wait complete. 2024-04-23T18:22:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.465+0000 7faa7586a740 1 -- start start 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.465+0000 7faa7586a740 1 Processor -- start 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa7586a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7620 0x5623d17caa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa7586a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7f40 0x5623d17bbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa7586a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17bc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa7586a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1ac150 con 0x5623d17b7620 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa7586a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1ad5c0 con 0x5623d17c2800 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa7586a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1ac820 con 0x5623d17b7f40 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa7586a740 10 client.0 _ll_register_callbacks cb 0x7faa58031700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa6bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7620 0x5623d17caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa6bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7620 0x5623d17caa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49286/0 (socket says 172.21.15.50:49286) 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa6bfff640 1 -- 172.21.15.50:0/2161721444 learned_addr learned my addr 172.21.15.50:0/2161721444 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa708ed640 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7f40 0x5623d17bbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa6bfff640 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17bc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa6bfff640 1 -- 172.21.15.50:0/2161721444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7f40 msgr2=0x5623d17bbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa6bfff640 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7f40 0x5623d17bbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa6bfff640 1 -- 172.21.15.50:0/2161721444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 msgr2=0x5623d17bc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa6bfff640 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17bc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa6bfff640 1 -- 172.21.15.50:0/2161721444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623d17bc8c0 con 0x5623d17b7620 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa708ed640 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7f40 0x5623d17bbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa6bfff640 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7620 0x5623d17caa60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7faa58031a90 tx=0x7faa5802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20edc509308f34ad server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa4bfff640 1 -- 172.21.15.50:0/2161721444 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa58035070 con 0x5623d17b7620 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa4bfff640 1 -- 172.21.15.50:0/2161721444 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8498+0+0 (secure 0 0 0) 0x7faa58031b10 con 0x5623d17b7620 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa7586a740 1 -- 172.21.15.50:0/2161721444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa6001f040 con 0x5623d17b7620 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.466+0000 7faa4bfff640 1 -- 172.21.15.50:0/2161721444 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7faa58031d10 con 0x5623d17b7620 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.467+0000 7faa7586a740 10 client.15933 fetch_fsmap learned FSMap version 23 2024-04-23T18:22:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.467+0000 7faa7586a740 1 -- 172.21.15.50:0/2161721444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5623d1838ca0 con 0x5623d17b7620 2024-04-23T18:22:03.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.467+0000 7faa4bfff640 1 -- 172.21.15.50:0/2161721444 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa6001f040 con 0x5623d17b7620 2024-04-23T18:22:03.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.467+0000 7faa4bfff640 1 -- 172.21.15.50:0/2161721444 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7faa5804b040 con 0x5623d17b7620 2024-04-23T18:22:03.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.467+0000 7faa7586a740 10 client.15933 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T18:22:03.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.467+0000 7faa7586a740 10 client.15933 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:03.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.467+0000 7faa7586a740 20 client.15933 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:03.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.467+0000 7faa7586a740 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623d18393d0 0x5623d183b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:03.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.467+0000 7faa7586a740 4 client.15933 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:03.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.467+0000 7faa7586a740 1 -- 172.21.15.50:0/2161721444 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5623d1724200 con 0x5623d18393d0 2024-04-23T18:22:03.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.467+0000 7faa708ed640 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623d18393d0 0x5623d183b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:03.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.468+0000 7faa708ed640 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623d18393d0 0x5623d183b7f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fdc86472e75d9ea server_cookie=4545df37c76d3a32 in_seq=0 out_seq=0 2024-04-23T18:22:03.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.468+0000 7faa4bfff640 10 client.15933 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.828+0000 7faa4bfff640 1 -- 172.21.15.50:0/2161721444 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5623d1724200 con 0x5623d18393d0 2024-04-23T18:22:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.828+0000 7faa4bfff640 10 client.15933 handle_command_reply: tid=0 2024-04-23T18:22:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.828+0000 7faa7586a740 1 client.15933 shutdown 2024-04-23T18:22:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.828+0000 7faa7586a740 1 -- 172.21.15.50:0/2161721444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7620 msgr2=0x5623d17caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.828+0000 7faa7586a740 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7620 0x5623d17caa60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7faa58031a90 tx=0x7faa5802a410 comp rx=0 tx=0).stop 2024-04-23T18:22:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.828+0000 7faa7586a740 1 -- 172.21.15.50:0/2161721444 shutdown_connections 2024-04-23T18:22:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.828+0000 7faa7586a740 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623d18393d0 0x5623d183b7f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.828+0000 7faa7586a740 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623d17c2800 0x5623d17bc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.828+0000 7faa7586a740 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623d17b7f40 0x5623d17bbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.828+0000 7faa4bfff640 0 client.15933 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.828+0000 7faa7586a740 1 --2- 172.21.15.50:0/2161721444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623d17b7620 0x5623d17caa60 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.828+0000 7faa7586a740 1 -- 172.21.15.50:0/2161721444 >> 172.21.15.50:0/2161721444 conn(0x5623d17b3b30 msgr2=0x5623d17b9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa7586a740 1 -- 172.21.15.50:0/2161721444 shutdown_connections 2024-04-23T18:22:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa7586a740 1 -- 172.21.15.50:0/2161721444 wait complete. 2024-04-23T18:22:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa7586a740 20 client.15933 trim_cache size 0 max 16384 2024-04-23T18:22:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:22:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:22:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:22:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:22:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa72b51640 1 -- 172.21.15.50:0/1022799769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faa5c0a3e40 msgr2=0x7faa5c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa72b51640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faa5c0a3e40 0x7faa5c0a6300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faa64099270 tx=0x7faa6408e040 comp rx=0 tx=0).stop 2024-04-23T18:22:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa72b51640 1 -- 172.21.15.50:0/1022799769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa6c19cf80 msgr2=0x7faa6c19d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa72b51640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa6c19cf80 0x7faa6c19d430 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7faa6400eaa0 tx=0x7faa6400ef60 comp rx=0 tx=0).stop 2024-04-23T18:22:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa72b51640 1 -- 172.21.15.50:0/1022799769 shutdown_connections 2024-04-23T18:22:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa72b51640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faa5c0a3e40 0x7faa5c0a6300 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa72b51640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa6c19cf80 0x7faa6c19d430 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa72b51640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa6c106170 0x7faa6c198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa72b51640 1 --2- 172.21.15.50:0/1022799769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa6c1010f0 0x7faa6c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.829+0000 7faa72b51640 1 -- 172.21.15.50:0/1022799769 >> 172.21.15.50:0/1022799769 conn(0x7faa6c077fd0 msgr2=0x7faa6c069a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.830+0000 7faa72b51640 1 -- 172.21.15.50:0/1022799769 shutdown_connections 2024-04-23T18:22:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:03.830+0000 7faa72b51640 1 -- 172.21.15.50:0/1022799769 wait complete. 2024-04-23T18:22:03.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:03 smithi050 ceph-mon[29187]: pgmap v109: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 2.6 MiB/s wr, 172 op/s 2024-04-23T18:22:04.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:03 smithi177 ceph-mon[32308]: pgmap v109: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 2.6 MiB/s wr, 172 op/s 2024-04-23T18:22:04.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:03 smithi188 ceph-mon[32694]: pgmap v109: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 2.6 MiB/s wr, 172 op/s 2024-04-23T18:22:04.199 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 6.69 MB/sec execute 38 sec latency 15211.287 ms 2024-04-23T18:22:05.199 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 6.52 MB/sec execute 39 sec latency 16211.408 ms 2024-04-23T18:22:05.288 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: *** Caught signal (Segmentation fault) ** 2024-04-23T18:22:05.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: in thread 7f91dfdb4700 thread_name:md_log_replay 2024-04-23T18:22:05.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:22:05.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f91f1175d20] 2024-04-23T18:22:05.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 2: /usr/bin/ceph-mds(+0x549372) [0x5616ea198372] 2024-04-23T18:22:05.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x5616ea19bb23] 2024-04-23T18:22:05.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 4: (MDLog::_replay_thread()+0x617) [0x5616ea1f6647] 2024-04-23T18:22:05.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 5: (MDLog::ReplayThread::entry()+0x11) [0x5616ea1ffd31] 2024-04-23T18:22:05.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f91f116b1ca] 2024-04-23T18:22:05.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 7: clone() 2024-04-23T18:22:05.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 2024-04-23T18:22:04.799+0000 7f91dfdb4700 -1 *** Caught signal (Segmentation fault) ** 2024-04-23T18:22:05.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: in thread 7f91dfdb4700 thread_name:md_log_replay 2024-04-23T18:22:05.289 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f91f1175d20] 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 2: /usr/bin/ceph-mds(+0x549372) [0x5616ea198372] 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x5616ea19bb23] 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 4: (MDLog::_replay_thread()+0x617) [0x5616ea1f6647] 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 5: (MDLog::ReplayThread::entry()+0x11) [0x5616ea1ffd31] 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f91f116b1ca] 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 7: clone() 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 0> 2024-04-23T18:22:04.799+0000 7f91dfdb4700 -1 *** Caught signal (Segmentation fault) ** 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: in thread 7f91dfdb4700 thread_name:md_log_replay 2024-04-23T18:22:05.290 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 2024-04-23T18:22:05.291 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:22:05.291 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f91f1175d20] 2024-04-23T18:22:05.291 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 2: /usr/bin/ceph-mds(+0x549372) [0x5616ea198372] 2024-04-23T18:22:05.291 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x5616ea19bb23] 2024-04-23T18:22:05.291 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 4: (MDLog::_replay_thread()+0x617) [0x5616ea1f6647] 2024-04-23T18:22:05.291 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 5: (MDLog::ReplayThread::entry()+0x11) [0x5616ea1ffd31] 2024-04-23T18:22:05.291 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f91f116b1ca] 2024-04-23T18:22:05.291 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 7: clone() 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 0> 2024-04-23T18:22:04.799+0000 7f91dfdb4700 -1 *** Caught signal (Segmentation fault) ** 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: in thread 7f91dfdb4700 thread_name:md_log_replay 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f91f1175d20] 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 2: /usr/bin/ceph-mds(+0x549372) [0x5616ea198372] 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x5616ea19bb23] 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 4: (MDLog::_replay_thread()+0x617) [0x5616ea1f6647] 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 5: (MDLog::ReplayThread::entry()+0x11) [0x5616ea1ffd31] 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f91f116b1ca] 2024-04-23T18:22:05.292 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 7: clone() 2024-04-23T18:22:05.293 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-23T18:22:05.293 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[62704]: 2024-04-23T18:22:05.293 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:04 smithi177 podman[66627]: 2024-04-23 18:22:04.860040757 +0000 UTC m=+0.020709421 container died 9b09d288c4b24034e1b47bac5ae3eeed90f9a0097a3ebd8fca5eeeff13a72236 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:22:05.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:05 smithi050 ceph-mon[29187]: pgmap v110: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-23T18:22:05.958 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:05 smithi177 ceph-mon[32308]: pgmap v110: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-23T18:22:05.959 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:05 smithi177 podman[66627]: 2024-04-23 18:22:05.743265478 +0000 UTC m=+0.903934190 container remove 9b09d288c4b24034e1b47bac5ae3eeed90f9a0097a3ebd8fca5eeeff13a72236 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-23T18:22:05.959 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:05 smithi177 systemd[1]: ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=139/n/a 2024-04-23T18:22:06.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:05 smithi188 ceph-mon[32694]: pgmap v110: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-23T18:22:06.199 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 6.36 MB/sec execute 40 sec latency 17211.445 ms 2024-04-23T18:22:06.287 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:05 smithi177 systemd[1]: ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-23T18:22:06.287 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:05 smithi177 systemd[1]: ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.h.service: Consumed 4.818s CPU time. 2024-04-23T18:22:07.199 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 6.20 MB/sec execute 41 sec latency 18211.580 ms 2024-04-23T18:22:07.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:07 smithi050 ceph-mon[29187]: pgmap v111: 97 pgs: 97 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 2.3 MiB/s wr, 106 op/s 2024-04-23T18:22:08.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:07 smithi177 ceph-mon[32308]: pgmap v111: 97 pgs: 97 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 2.3 MiB/s wr, 106 op/s 2024-04-23T18:22:08.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:07 smithi188 ceph-mon[32694]: pgmap v111: 97 pgs: 97 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 2.3 MiB/s wr, 106 op/s 2024-04-23T18:22:08.199 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 6.05 MB/sec execute 42 sec latency 19211.695 ms 2024-04-23T18:22:09.199 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 5.91 MB/sec execute 43 sec latency 20211.811 ms 2024-04-23T18:22:10.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:09 smithi177 ceph-mon[32308]: pgmap v112: 97 pgs: 97 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-23T18:22:10.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:09 smithi188 ceph-mon[32694]: pgmap v112: 97 pgs: 97 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-23T18:22:10.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:09 smithi050 ceph-mon[29187]: pgmap v112: 97 pgs: 97 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-23T18:22:10.199 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 5.78 MB/sec execute 44 sec latency 21211.869 ms 2024-04-23T18:22:11.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 5.65 MB/sec execute 45 sec latency 22211.912 ms 2024-04-23T18:22:12.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:11 smithi177 ceph-mon[32308]: pgmap v113: 97 pgs: 97 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-23T18:22:12.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:11 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:12.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:11 smithi188 ceph-mon[32694]: pgmap v113: 97 pgs: 97 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-23T18:22:12.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:11 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:12.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:11 smithi050 ceph-mon[29187]: pgmap v113: 97 pgs: 97 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-23T18:22:12.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:11 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:12.199 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 5.53 MB/sec execute 46 sec latency 23212.029 ms 2024-04-23T18:22:13.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 5.41 MB/sec execute 47 sec latency 24212.136 ms 2024-04-23T18:22:14.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:13 smithi177 ceph-mon[32308]: pgmap v114: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 104 op/s 2024-04-23T18:22:14.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:13 smithi188 ceph-mon[32694]: pgmap v114: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 104 op/s 2024-04-23T18:22:14.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:13 smithi050 ceph-mon[29187]: pgmap v114: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 2.3 MiB/s wr, 104 op/s 2024-04-23T18:22:14.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 5.30 MB/sec execute 48 sec latency 25212.246 ms 2024-04-23T18:22:15.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 5.19 MB/sec execute 49 sec latency 26212.317 ms 2024-04-23T18:22:16.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:15 smithi177 ceph-mon[32308]: pgmap v115: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.4 MiB/s wr, 58 op/s 2024-04-23T18:22:16.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:15 smithi188 ceph-mon[32694]: pgmap v115: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.4 MiB/s wr, 58 op/s 2024-04-23T18:22:16.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:15 smithi050 ceph-mon[29187]: pgmap v115: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.4 MiB/s wr, 58 op/s 2024-04-23T18:22:16.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 5.09 MB/sec execute 50 sec latency 27212.356 ms 2024-04-23T18:22:16.413 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:16 smithi177 systemd[1]: ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-04-23T18:22:16.413 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:16 smithi177 systemd[1]: Stopped Ceph mds.h for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:22:16.413 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:16 smithi177 systemd[1]: ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.h.service: Consumed 4.818s CPU time. 2024-04-23T18:22:16.413 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:16 smithi177 systemd[1]: Starting Ceph mds.h for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:22:16.413 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:16 smithi177 podman[66727]: 2024-04-23 18:22:16.182027019 +0000 UTC m=+0.018352956 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:22:16.413 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:16 smithi177 podman[66727]: 2024-04-23 18:22:16.29418653 +0000 UTC m=+0.130512470 container create e561ba7acda0a088a906bee368ba6b777c2306fce39be4c43cfcefc9c56c9b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-23T18:22:16.715 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:16 smithi177 podman[66727]: 2024-04-23 18:22:16.496899981 +0000 UTC m=+0.333225922 container init e561ba7acda0a088a906bee368ba6b777c2306fce39be4c43cfcefc9c56c9b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:22:16.716 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:16 smithi177 podman[66727]: 2024-04-23 18:22:16.499406384 +0000 UTC m=+0.335732360 container start e561ba7acda0a088a906bee368ba6b777c2306fce39be4c43cfcefc9c56c9b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-23T18:22:16.716 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:16 smithi177 ceph-mds[66745]: -- 172.21.15.177:0/3041528639 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d3f1723a40 con 0x55d3f16f6800 2024-04-23T18:22:16.716 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:16 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: starting mds.h at 2024-04-23T18:22:16.716 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:16 smithi177 bash[66727]: e561ba7acda0a088a906bee368ba6b777c2306fce39be4c43cfcefc9c56c9b29 2024-04-23T18:22:16.716 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:22:16 smithi177 systemd[1]: Started Ceph mds.h for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:22:16.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:22:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.930+0000 7fe3a2337640 1 -- 172.21.15.50:0/2086589614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe39c106170 msgr2=0x7fe39c10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.930+0000 7fe3a2337640 1 --2- 172.21.15.50:0/2086589614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe39c106170 0x7fe39c10dd40 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe394009ca0 tx=0x7fe39402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.930+0000 7fe3a2337640 1 -- 172.21.15.50:0/2086589614 shutdown_connections 2024-04-23T18:22:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.930+0000 7fe3a2337640 1 --2- 172.21.15.50:0/2086589614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe39c106170 0x7fe39c10dd40 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.930+0000 7fe3a2337640 1 --2- 172.21.15.50:0/2086589614 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe39c101aa0 0x7fe39c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.930+0000 7fe3a2337640 1 --2- 172.21.15.50:0/2086589614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe39c1010f0 0x7fe39c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.930+0000 7fe3a2337640 1 -- 172.21.15.50:0/2086589614 >> 172.21.15.50:0/2086589614 conn(0x7fe39c077fd0 msgr2=0x7fe39c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.930+0000 7fe3a2337640 1 -- 172.21.15.50:0/2086589614 shutdown_connections 2024-04-23T18:22:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.930+0000 7fe3a2337640 1 -- 172.21.15.50:0/2086589614 wait complete. 2024-04-23T18:22:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.931+0000 7fe3a2337640 1 Processor -- start 2024-04-23T18:22:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.931+0000 7fe3a2337640 1 -- start start 2024-04-23T18:22:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.931+0000 7fe3a2337640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe39c1010f0 0x7fe39c102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.931+0000 7fe3a2337640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe39c106170 0x7fe39c103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.931+0000 7fe3a2337640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe39c1035b0 0x7fe39c1a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.931+0000 7fe3a2337640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c111130 con 0x7fe39c1035b0 2024-04-23T18:22:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.931+0000 7fe3a2337640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c110fb0 con 0x7fe39c106170 2024-04-23T18:22:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.931+0000 7fe3a2337640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c1112b0 con 0x7fe39c1010f0 2024-04-23T18:22:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.931+0000 7fe3a08d4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe39c1035b0 0x7fe39c1a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.931+0000 7fe3a08d4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe39c1035b0 0x7fe39c1a6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34464/0 (socket says 172.21.15.50:34464) 2024-04-23T18:22:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.931+0000 7fe39b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe39c106170 0x7fe39c103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.931+0000 7fe3a08d4640 1 -- 172.21.15.50:0/2713606458 learned_addr learned my addr 172.21.15.50:0/2713606458 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe39bfff640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe39c1010f0 0x7fe39c102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe39b7fe640 1 -- 172.21.15.50:0/2713606458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe39c1010f0 msgr2=0x7fe39c102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe39b7fe640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe39c1010f0 0x7fe39c102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe39b7fe640 1 -- 172.21.15.50:0/2713606458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe39c1035b0 msgr2=0x7fe39c1a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe39b7fe640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe39c1035b0 0x7fe39c1a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe39b7fe640 1 -- 172.21.15.50:0/2713606458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39c1a6f20 con 0x7fe39c106170 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe3a08d4640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe39c1035b0 0x7fe39c1a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe39b7fe640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe39c106170 0x7fe39c103070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe39000e9c0 tx=0x7fe39000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe3997fa640 1 -- 172.21.15.50:0/2713606458 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39000ccc0 con 0x7fe39c106170 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe3a2337640 1 -- 172.21.15.50:0/2713606458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe39c1a7210 con 0x7fe39c106170 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe3997fa640 1 -- 172.21.15.50:0/2713606458 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe390005020 con 0x7fe39c106170 2024-04-23T18:22:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe3997fa640 1 -- 172.21.15.50:0/2713606458 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe390004a00 con 0x7fe39c106170 2024-04-23T18:22:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.932+0000 7fe3a2337640 1 -- 172.21.15.50:0/2713606458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe39c1a7710 con 0x7fe39c106170 2024-04-23T18:22:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.935+0000 7fe3997fa640 1 -- 172.21.15.50:0/2713606458 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe3900051d0 con 0x7fe39c106170 2024-04-23T18:22:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.935+0000 7fe3a504f740 1 Processor -- start 2024-04-23T18:22:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.935+0000 7fe3a504f740 1 -- start start 2024-04-23T18:22:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.935+0000 7fe3997fa640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe3840a3dc0 0x7fe3840a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.935+0000 7fe3a504f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf23620 0x5622bdf23a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.935+0000 7fe3a504f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 0x5622bdf2e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.935+0000 7fe3a504f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf2e800 0x5622bdf30bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe3a08d4640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe3840a3dc0 0x7fe3840a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe3a504f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5622bde88970 con 0x5622bdf23f40 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe3a504f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5622bde254a0 con 0x5622bdf2e800 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe3a504f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5622bde84930 con 0x5622bdf23620 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe3997fa640 1 -- 172.21.15.50:0/2713606458 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fe3900d0710 con 0x7fe39c106170 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe39b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 0x5622bdf2e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe39b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 0x5622bdf2e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34476/0 (socket says 172.21.15.50:34476) 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe39b7fe640 1 -- 172.21.15.50:0/2640080694 learned_addr learned my addr 172.21.15.50:0/2640080694 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe3a08d4640 1 --2- 172.21.15.50:0/2640080694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf2e800 0x5622bdf30bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe39bfff640 1 --2- 172.21.15.50:0/2640080694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf23620 0x5622bdf23a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe39b7fe640 1 -- 172.21.15.50:0/2640080694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf23620 msgr2=0x5622bdf23a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe39b7fe640 1 --2- 172.21.15.50:0/2640080694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf23620 0x5622bdf23a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe39b7fe640 1 -- 172.21.15.50:0/2640080694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf2e800 msgr2=0x5622bdf30bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe39b7fe640 1 --2- 172.21.15.50:0/2640080694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf2e800 0x5622bdf30bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe39b7fe640 1 -- 172.21.15.50:0/2640080694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622bdb9b210 con 0x5622bdf23f40 2024-04-23T18:22:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe3a08d4640 1 --2- 172.21.15.50:0/2640080694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf2e800 0x5622bdf30bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe3a08d4640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe3840a3dc0 0x7fe3840a6280 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe394005fe0 tx=0x7fe39402f450 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe39b7fe640 1 --2- 172.21.15.50:0/2640080694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 0x5622bdf2e2c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe390099090 tx=0x7fe39009ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603c71343f67fffd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.936+0000 7fe3827fc640 1 -- 172.21.15.50:0/2640080694 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39001c040 con 0x5622bdf23f40 2024-04-23T18:22:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.937+0000 7fe3827fc640 1 -- 172.21.15.50:0/2640080694 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3900a5330 con 0x5622bdf23f40 2024-04-23T18:22:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.937+0000 7fe3827fc640 1 -- 172.21.15.50:0/2640080694 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3900a4ea0 con 0x5622bdf23f40 2024-04-23T18:22:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.937+0000 7fe3a504f740 1 -- 172.21.15.50:0/2640080694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 msgr2=0x5622bdf2e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.937+0000 7fe3a504f740 1 --2- 172.21.15.50:0/2640080694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 0x5622bdf2e2c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe390099090 tx=0x7fe39009ce80 comp rx=0 tx=0).stop 2024-04-23T18:22:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.937+0000 7fe3a504f740 1 -- 172.21.15.50:0/2640080694 shutdown_connections 2024-04-23T18:22:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.937+0000 7fe3a504f740 1 --2- 172.21.15.50:0/2640080694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf2e800 0x5622bdf30bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.937+0000 7fe3a504f740 1 --2- 172.21.15.50:0/2640080694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 0x5622bdf2e2c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.937+0000 7fe3a504f740 1 --2- 172.21.15.50:0/2640080694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf23620 0x5622bdf23a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.937+0000 7fe3a504f740 1 -- 172.21.15.50:0/2640080694 >> 172.21.15.50:0/2640080694 conn(0x5622bdf1fb30 msgr2=0x5622bdf1ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.937+0000 7fe3a504f740 1 -- 172.21.15.50:0/2640080694 shutdown_connections 2024-04-23T18:22:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.937+0000 7fe3a504f740 1 -- 172.21.15.50:0/2640080694 wait complete. 2024-04-23T18:22:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe3a504f740 1 -- start start 2024-04-23T18:22:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe3a504f740 1 Processor -- start 2024-04-23T18:22:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe3a504f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 0x5622bdf2e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe3a504f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf23f40 0x5622bdf2b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe3a504f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf2e800 0x5622bdf2bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe3a504f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5622bdf31430 con 0x5622bdf2e800 2024-04-23T18:22:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe3a504f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5622bde230b0 con 0x5622bdf23620 2024-04-23T18:22:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe3a504f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5622bdf315b0 con 0x5622bdf23f40 2024-04-23T18:22:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe3a504f740 10 client.0 _ll_register_callbacks cb 0x5622bde10d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe3a08d4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf2e800 0x5622bdf2bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe3a08d4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf2e800 0x5622bdf2bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34482/0 (socket says 172.21.15.50:34482) 2024-04-23T18:22:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe3a08d4640 1 -- 172.21.15.50:0/4079244583 learned_addr learned my addr 172.21.15.50:0/4079244583 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.938+0000 7fe39bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 0x5622bdf2e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe39b7fe640 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf23f40 0x5622bdf2b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe3a08d4640 1 -- 172.21.15.50:0/4079244583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf23f40 msgr2=0x5622bdf2b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe3a08d4640 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf23f40 0x5622bdf2b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe3a08d4640 1 -- 172.21.15.50:0/4079244583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 msgr2=0x5622bdf2e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe3a08d4640 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 0x5622bdf2e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe3a08d4640 1 -- 172.21.15.50:0/4079244583 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622bdc67620 con 0x5622bdf2e800 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe39bfff640 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 0x5622bdf2e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe3a08d4640 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf2e800 0x5622bdf2bad0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe394003ba0 tx=0x7fe39402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d9ceda53d0f431 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe37b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe37b7fe640 1 -- 172.21.15.50:0/4079244583 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394050020 con 0x5622bdf2e800 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe37b7fe640 1 -- 172.21.15.50:0/4079244583 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fe394004630 con 0x5622bdf2e800 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe3a504f740 1 -- 172.21.15.50:0/4079244583 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3900a5330 con 0x5622bdf2e800 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe37b7fe640 1 -- 172.21.15.50:0/4079244583 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe39405b050 con 0x5622bdf2e800 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe3a504f740 10 client.15948 fetch_fsmap learned FSMap version 26 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.939+0000 7fe3a504f740 1 -- 172.21.15.50:0/4079244583 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5622bdf3d3c0 con 0x5622bdf2e800 2024-04-23T18:22:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.940+0000 7fe37b7fe640 1 -- 172.21.15.50:0/4079244583 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3900a5330 con 0x5622bdf2e800 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.940+0000 7fe37b7fe640 1 -- 172.21.15.50:0/4079244583 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fe3940305c0 con 0x5622bdf2e800 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.940+0000 7fe3a504f740 10 client.15948 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.940+0000 7fe3a504f740 10 client.15948 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.940+0000 7fe3a504f740 20 client.15948 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.940+0000 7fe3a504f740 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5622bdf3e920 0x5622bdf40dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.940+0000 7fe3a504f740 4 client.15948 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.940+0000 7fe3a504f740 1 -- 172.21.15.50:0/4079244583 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622bde96390 con 0x5622bdf3e920 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.940+0000 7fe39bfff640 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5622bdf3e920 0x5622bdf40dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.940+0000 7fe39bfff640 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5622bdf3e920 0x5622bdf40dc0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=626f637df41da7fd server_cookie=a728993a18498c in_seq=0 out_seq=0 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.940+0000 7fe37b7fe640 10 client.15948 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe37b7fe640 1 -- 172.21.15.50:0/4079244583 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5622bde96390 con 0x5622bdf3e920 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe37b7fe640 10 client.15948 handle_command_reply: tid=0 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe3a504f740 1 client.15948 shutdown 2024-04-23T18:22:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe3a504f740 1 -- 172.21.15.50:0/4079244583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf2e800 msgr2=0x5622bdf2bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe3a504f740 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf2e800 0x5622bdf2bad0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe394003ba0 tx=0x7fe39402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe3a504f740 1 -- 172.21.15.50:0/4079244583 shutdown_connections 2024-04-23T18:22:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe3a504f740 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5622bdf3e920 0x5622bdf40dc0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe3a504f740 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf2e800 0x5622bdf2bad0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe3a504f740 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf23f40 0x5622bdf2b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe3a504f740 1 --2- 172.21.15.50:0/4079244583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 0x5622bdf2e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe37b7fe640 0 client.15948 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe3a504f740 1 -- 172.21.15.50:0/4079244583 >> 172.21.15.50:0/4079244583 conn(0x5622bdf1fb30 msgr2=0x5622bdf1ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.941+0000 7fe3a504f740 1 -- 172.21.15.50:0/4079244583 shutdown_connections 2024-04-23T18:22:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.942+0000 7fe3a504f740 1 -- 172.21.15.50:0/4079244583 wait complete. 2024-04-23T18:22:16.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.942+0000 7fe3a504f740 20 client.15948 trim_cache size 0 max 16384 2024-04-23T18:22:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe3a504f740 1 Processor -- start 2024-04-23T18:22:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe3a504f740 1 -- start start 2024-04-23T18:22:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe3a504f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 0x5622bdf3a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe3a504f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 0x5622bdf2a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe3a504f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf2e800 0x5622bdf1eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe3a504f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c1b25f0 con 0x5622bdf23f40 2024-04-23T18:22:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe3a504f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c1afe60 con 0x5622bdf23620 2024-04-23T18:22:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe3a504f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c1adcb0 con 0x5622bdf2e800 2024-04-23T18:22:16.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe39bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 0x5622bdf3a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe39b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 0x5622bdf2a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe39bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 0x5622bdf3a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:48816/0 (socket says 172.21.15.50:48816) 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe39b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 0x5622bdf2a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34492/0 (socket says 172.21.15.50:34492) 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe39bfff640 1 -- 172.21.15.50:0/2492845638 learned_addr learned my addr 172.21.15.50:0/2492845638 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe3a08d4640 1 --2- 172.21.15.50:0/2492845638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf2e800 0x5622bdf1eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe39bfff640 1 -- 172.21.15.50:0/2492845638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf2e800 msgr2=0x5622bdf1eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.957+0000 7fe39bfff640 1 --2- 172.21.15.50:0/2492845638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf2e800 0x5622bdf1eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.958+0000 7fe39bfff640 1 -- 172.21.15.50:0/2492845638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 msgr2=0x5622bdf2a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.958+0000 7fe39bfff640 1 --2- 172.21.15.50:0/2492845638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 0x5622bdf2a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.958+0000 7fe39bfff640 1 -- 172.21.15.50:0/2492845638 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622bdf2c0d0 con 0x5622bdf23620 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.958+0000 7fe39bfff640 1 --2- 172.21.15.50:0/2492845638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 0x5622bdf3a2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x5622bdf41630 tx=0x7fe38c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac4e5f36e96368d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.958+0000 7fe381ffb640 1 -- 172.21.15.50:0/2492845638 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38c0203d0 con 0x5622bdf23620 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.958+0000 7fe381ffb640 1 -- 172.21.15.50:0/2492845638 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe38c010430 con 0x5622bdf23620 2024-04-23T18:22:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.958+0000 7fe381ffb640 1 -- 172.21.15.50:0/2492845638 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38c01e680 con 0x5622bdf23620 2024-04-23T18:22:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.959+0000 7fe3a504f740 1 -- 172.21.15.50:0/2492845638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 msgr2=0x5622bdf3a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.959+0000 7fe3a504f740 1 --2- 172.21.15.50:0/2492845638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 0x5622bdf3a2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x5622bdf41630 tx=0x7fe38c005f90 comp rx=0 tx=0).stop 2024-04-23T18:22:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.959+0000 7fe3a504f740 1 -- 172.21.15.50:0/2492845638 shutdown_connections 2024-04-23T18:22:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.959+0000 7fe3a504f740 1 --2- 172.21.15.50:0/2492845638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf2e800 0x5622bdf1eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.959+0000 7fe3a504f740 1 --2- 172.21.15.50:0/2492845638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23f40 0x5622bdf2a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.959+0000 7fe3a504f740 1 --2- 172.21.15.50:0/2492845638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23620 0x5622bdf3a2f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.959+0000 7fe3a504f740 1 -- 172.21.15.50:0/2492845638 >> 172.21.15.50:0/2492845638 conn(0x5622bdf1fb30 msgr2=0x5622bdf1e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.959+0000 7fe3a504f740 1 -- 172.21.15.50:0/2492845638 shutdown_connections 2024-04-23T18:22:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.959+0000 7fe3a504f740 1 -- 172.21.15.50:0/2492845638 wait complete. 2024-04-23T18:22:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.959+0000 7fe3a504f740 1 -- start start 2024-04-23T18:22:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.959+0000 7fe3a504f740 1 Processor -- start 2024-04-23T18:22:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe3a504f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23620 0x5622bdf36bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe3a504f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23f40 0x5622bdf31f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe3a504f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf2e800 0x5622bdf32480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe3a504f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c1ae320 con 0x5622bdf23620 2024-04-23T18:22:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe3a504f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c1af790 con 0x5622bdf23f40 2024-04-23T18:22:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe3a504f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c1ae9f0 con 0x5622bdf2e800 2024-04-23T18:22:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe3a504f740 10 client.0 _ll_register_callbacks cb 0x7fe3940319f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe39bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23620 0x5622bdf36bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe39bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23620 0x5622bdf36bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34498/0 (socket says 172.21.15.50:34498) 2024-04-23T18:22:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe39b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23f40 0x5622bdf31f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe39bfff640 1 -- 172.21.15.50:0/417118451 learned_addr learned my addr 172.21.15.50:0/417118451 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe3a08d4640 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf2e800 0x5622bdf32480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe39bfff640 1 -- 172.21.15.50:0/417118451 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf2e800 msgr2=0x5622bdf32480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe39bfff640 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf2e800 0x5622bdf32480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe39bfff640 1 -- 172.21.15.50:0/417118451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23f40 msgr2=0x5622bdf31f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe39bfff640 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23f40 0x5622bdf31f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe39bfff640 1 -- 172.21.15.50:0/417118451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622bdf1a090 con 0x5622bdf23620 2024-04-23T18:22:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe3a08d4640 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf2e800 0x5622bdf32480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe39b7fe640 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23f40 0x5622bdf31f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe39bfff640 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23620 0x5622bdf36bd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fe38c010d60 tx=0x7fe38c0120f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6f031b04ae6ac2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.960+0000 7fe37b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.961+0000 7fe37b7fe640 1 -- 172.21.15.50:0/417118451 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38c010af0 con 0x5622bdf23620 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.961+0000 7fe37b7fe640 1 -- 172.21.15.50:0/417118451 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fe38c004020 con 0x5622bdf23620 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.961+0000 7fe3a504f740 1 -- 172.21.15.50:0/417118451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe38c010430 con 0x5622bdf23620 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.961+0000 7fe37b7fe640 1 -- 172.21.15.50:0/417118451 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe38c004220 con 0x5622bdf23620 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.961+0000 7fe3a504f740 10 client.15954 fetch_fsmap learned FSMap version 26 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.961+0000 7fe3a504f740 1 -- 172.21.15.50:0/417118451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5622bdfa4d60 con 0x5622bdf23620 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.961+0000 7fe37b7fe640 1 -- 172.21.15.50:0/417118451 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe38c010430 con 0x5622bdf23620 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.961+0000 7fe37b7fe640 1 -- 172.21.15.50:0/417118451 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fe38c020650 con 0x5622bdf23620 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.961+0000 7fe3a504f740 10 client.15954 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.962+0000 7fe3a504f740 10 client.15954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.962+0000 7fe3a504f740 20 client.15954 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.962+0000 7fe3a504f740 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5622bdfa5470 0x5622bdfa7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.962+0000 7fe3a504f740 4 client.15954 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.962+0000 7fe3a504f740 1 -- 172.21.15.50:0/417118451 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5622bde90200 con 0x5622bdfa5470 2024-04-23T18:22:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.962+0000 7fe39b7fe640 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5622bdfa5470 0x5622bdfa7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.962+0000 7fe37b7fe640 10 client.15954 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.962+0000 7fe39b7fe640 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5622bdfa5470 0x5622bdfa7890 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73fcef5f759ccf53 server_cookie=33315a6d0d8aa251 in_seq=0 out_seq=0 2024-04-23T18:22:16.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.979+0000 7f6d79bf7640 1 -- 172.21.15.50:0/1078031787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d74106170 msgr2=0x7f6d7410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:16.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.979+0000 7f6d79bf7640 1 --2- 172.21.15.50:0/1078031787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d74106170 0x7f6d7410dd40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c009be0 tx=0x7f6d6c02a6c0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.979+0000 7f6d79bf7640 1 -- 172.21.15.50:0/1078031787 shutdown_connections 2024-04-23T18:22:16.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.979+0000 7f6d79bf7640 1 --2- 172.21.15.50:0/1078031787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d74106170 0x7f6d7410dd40 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.979+0000 7f6d79bf7640 1 --2- 172.21.15.50:0/1078031787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d74101a50 0x7f6d74105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.979+0000 7f6d79bf7640 1 --2- 172.21.15.50:0/1078031787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d741010a0 0x7f6d74101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.979+0000 7f6d79bf7640 1 -- 172.21.15.50:0/1078031787 >> 172.21.15.50:0/1078031787 conn(0x7f6d74077f50 msgr2=0x7f6d74078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:16.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.979+0000 7f6d79bf7640 1 -- 172.21.15.50:0/1078031787 shutdown_connections 2024-04-23T18:22:16.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.979+0000 7f6d79bf7640 1 -- 172.21.15.50:0/1078031787 wait complete. 2024-04-23T18:22:16.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.979+0000 7f6d79bf7640 1 Processor -- start 2024-04-23T18:22:16.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d79bf7640 1 -- start start 2024-04-23T18:22:16.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d79bf7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d741010a0 0x7f6d741986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d79bf7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d74106170 0x7f6d74198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d79bf7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d7419cf80 0x7f6d7419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d79bf7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6d741110e0 con 0x7f6d74106170 2024-04-23T18:22:16.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d79bf7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6d74110f60 con 0x7f6d741010a0 2024-04-23T18:22:16.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d79bf7640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6d74111260 con 0x7f6d7419cf80 2024-04-23T18:22:16.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d72ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d74106170 0x7f6d74198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d737fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d741010a0 0x7f6d741986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d72ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d74106170 0x7f6d74198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34504/0 (socket says 172.21.15.50:34504) 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d72ffd640 1 -- 172.21.15.50:0/452994111 learned_addr learned my addr 172.21.15.50:0/452994111 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d73fff640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d7419cf80 0x7f6d7419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d72ffd640 1 -- 172.21.15.50:0/452994111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d7419cf80 msgr2=0x7f6d7419d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d72ffd640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d7419cf80 0x7f6d7419d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d72ffd640 1 -- 172.21.15.50:0/452994111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d741010a0 msgr2=0x7f6d741986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.980+0000 7f6d72ffd640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d741010a0 0x7f6d741986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.981+0000 7f6d72ffd640 1 -- 172.21.15.50:0/452994111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7419da70 con 0x7f6d74106170 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.981+0000 7f6d72ffd640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d74106170 0x7f6d74198c00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6d6800e9f0 tx=0x7f6d6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.981+0000 7f6d70ff9640 1 -- 172.21.15.50:0/452994111 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6800ccc0 con 0x7f6d74106170 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.981+0000 7f6d70ff9640 1 -- 172.21.15.50:0/452994111 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d680054d0 con 0x7f6d74106170 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.981+0000 7f6d79bf7640 1 -- 172.21.15.50:0/452994111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d7419dd60 con 0x7f6d74106170 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.981+0000 7f6d70ff9640 1 -- 172.21.15.50:0/452994111 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d68010430 con 0x7f6d74106170 2024-04-23T18:22:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.981+0000 7f6d79bf7640 1 -- 172.21.15.50:0/452994111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d741a5550 con 0x7f6d74106170 2024-04-23T18:22:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.981+0000 7fe37b7fe640 1 -- 172.21.15.50:0/417118451 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5622bde90200 con 0x5622bdfa5470 2024-04-23T18:22:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.981+0000 7fe37b7fe640 10 client.15954 handle_command_reply: tid=0 2024-04-23T18:22:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.981+0000 7fe3a504f740 1 client.15954 shutdown 2024-04-23T18:22:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.982+0000 7fe3a504f740 1 -- 172.21.15.50:0/417118451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23620 msgr2=0x5622bdf36bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.982+0000 7fe3a504f740 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23620 0x5622bdf36bd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fe38c010d60 tx=0x7fe38c0120f0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.982+0000 7fe3a504f740 1 -- 172.21.15.50:0/417118451 shutdown_connections 2024-04-23T18:22:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.982+0000 7fe3a504f740 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5622bdfa5470 0x5622bdfa7890 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.982+0000 7fe3a504f740 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5622bdf2e800 0x5622bdf32480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.982+0000 7fe3a504f740 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622bdf23f40 0x5622bdf31f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.982+0000 7fe37b7fe640 0 client.15954 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.982+0000 7fe3a504f740 1 --2- 172.21.15.50:0/417118451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5622bdf23620 0x5622bdf36bd0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.982+0000 7fe3a504f740 1 -- 172.21.15.50:0/417118451 >> 172.21.15.50:0/417118451 conn(0x5622bdf1fb30 msgr2=0x5622bdf29d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.982+0000 7fe3a504f740 1 -- 172.21.15.50:0/417118451 shutdown_connections 2024-04-23T18:22:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.982+0000 7fe3a504f740 1 -- 172.21.15.50:0/417118451 wait complete. 2024-04-23T18:22:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.982+0000 7fe3a504f740 20 client.15954 trim_cache size 0 max 16384 2024-04-23T18:22:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:22:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:22:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:22:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:22:16.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.983+0000 7f6d79bf7640 1 -- 172.21.15.50:0/452994111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d74068d70 con 0x7f6d74106170 2024-04-23T18:22:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.983+0000 7fe3a2337640 1 -- 172.21.15.50:0/2713606458 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe3840a3dc0 msgr2=0x7fe3840a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.983+0000 7fe3a2337640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe3840a3dc0 0x7fe3840a6280 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe394005fe0 tx=0x7fe39402f450 comp rx=0 tx=0).stop 2024-04-23T18:22:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.983+0000 7fe3a2337640 1 -- 172.21.15.50:0/2713606458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe39c106170 msgr2=0x7fe39c103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.983+0000 7fe3a2337640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe39c106170 0x7fe39c103070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe39000e9c0 tx=0x7fe39000ee80 comp rx=0 tx=0).stop 2024-04-23T18:22:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.983+0000 7fe3a2337640 1 -- 172.21.15.50:0/2713606458 shutdown_connections 2024-04-23T18:22:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.983+0000 7fe3a2337640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe3840a3dc0 0x7fe3840a6280 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.983+0000 7fe3a2337640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe39c1035b0 0x7fe39c1a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.983+0000 7fe3a2337640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe39c106170 0x7fe39c103070 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.983+0000 7fe3a2337640 1 --2- 172.21.15.50:0/2713606458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe39c1010f0 0x7fe39c102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.983+0000 7fe3a2337640 1 -- 172.21.15.50:0/2713606458 >> 172.21.15.50:0/2713606458 conn(0x7fe39c077fd0 msgr2=0x7fe39c069880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.983+0000 7fe3a2337640 1 -- 172.21.15.50:0/2713606458 shutdown_connections 2024-04-23T18:22:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:16.983+0000 7fe3a2337640 1 -- 172.21.15.50:0/2713606458 wait complete. 2024-04-23T18:22:16.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.983+0000 7f6d70ff9640 1 -- 172.21.15.50:0/452994111 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6d680046c0 con 0x7f6d74106170 2024-04-23T18:22:16.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.984+0000 7f6d70ff9640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6d5c0a3da0 0x7f6d5c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:16.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.984+0000 7f6d70ff9640 1 -- 172.21.15.50:0/452994111 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6d680d0aa0 con 0x7f6d74106170 2024-04-23T18:22:16.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.984+0000 7f6d737fe640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6d5c0a3da0 0x7f6d5c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:16.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.985+0000 7f6d737fe640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6d5c0a3da0 0x7f6d5c0a6260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6d64009530 tx=0x7f6d64009260 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:16.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:16.989+0000 7f6d70ff9640 1 -- 172.21.15.50:0/452994111 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d6809d150 con 0x7f6d74106170 2024-04-23T18:22:17.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:16 smithi177 ceph-mon[32308]: daemon mds.h restarted 2024-04-23T18:22:17.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:16 smithi188 ceph-mon[32694]: daemon mds.h restarted 2024-04-23T18:22:17.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:16 smithi050 ceph-mon[29187]: daemon mds.h restarted 2024-04-23T18:22:17.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 241513 4.99 MB/sec execute 51 sec latency 28212.388 ms 2024-04-23T18:22:17.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.217+0000 7f6d79bf7640 1 -- 172.21.15.50:0/452994111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d7410a410 con 0x7f6d74106170 2024-04-23T18:22:17.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.218+0000 7f6d70ff9640 1 -- 172.21.15.50:0/452994111 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11879 (secure 0 0 0) 0x7f6d6809ca50 con 0x7f6d74106170 2024-04-23T18:22:17.211 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:22:17.211 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:22:16.725864+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25547":{"gid":25547,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1939040460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1939040460},{"type":"v1","addr":"172.21.15.177:6837","nonce":1939040460}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:22:17.212 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 26 2024-04-23T18:22:17.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.219+0000 7f6d79bf7640 1 -- 172.21.15.50:0/452994111 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6d5c0a3da0 msgr2=0x7f6d5c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.219+0000 7f6d79bf7640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6d5c0a3da0 0x7f6d5c0a6260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6d64009530 tx=0x7f6d64009260 comp rx=0 tx=0).stop 2024-04-23T18:22:17.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.220+0000 7f6d79bf7640 1 -- 172.21.15.50:0/452994111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d74106170 msgr2=0x7f6d74198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.220+0000 7f6d79bf7640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d74106170 0x7f6d74198c00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6d6800e9f0 tx=0x7f6d6800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.220+0000 7f6d79bf7640 1 -- 172.21.15.50:0/452994111 shutdown_connections 2024-04-23T18:22:17.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.220+0000 7f6d79bf7640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6d5c0a3da0 0x7f6d5c0a6260 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.220+0000 7f6d79bf7640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d7419cf80 0x7f6d7419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.220+0000 7f6d79bf7640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d74106170 0x7f6d74198c00 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.220+0000 7f6d79bf7640 1 --2- 172.21.15.50:0/452994111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d741010a0 0x7f6d741986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.220+0000 7f6d79bf7640 1 -- 172.21.15.50:0/452994111 >> 172.21.15.50:0/452994111 conn(0x7f6d74077f50 msgr2=0x7f6d74069440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:17.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.220+0000 7f6d79bf7640 1 -- 172.21.15.50:0/452994111 shutdown_connections 2024-04-23T18:22:17.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.220+0000 7f6d79bf7640 1 -- 172.21.15.50:0/452994111 wait complete. 2024-04-23T18:22:17.224 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id cf8448b3 --release --await' 2024-04-23T18:22:17.224 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id cf8448b3 --release --await 2024-04-23T18:22:17.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.342+0000 7f791b674640 1 -- 172.21.15.50:0/1782833376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f791406b1f0 msgr2=0x7f7914106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.342+0000 7f791b674640 1 --2- 172.21.15.50:0/1782833376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f791406b1f0 0x7f7914106b90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7904009900 tx=0x7f790402a410 comp rx=0 tx=0).stop 2024-04-23T18:22:17.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.342+0000 7f791b674640 1 -- 172.21.15.50:0/1782833376 shutdown_connections 2024-04-23T18:22:17.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.342+0000 7f791b674640 1 --2- 172.21.15.50:0/1782833376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7914107270 0x7f791410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.342+0000 7f791b674640 1 --2- 172.21.15.50:0/1782833376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f791406b1f0 0x7f7914106b90 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.342+0000 7f791b674640 1 --2- 172.21.15.50:0/1782833376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f791406a840 0x7f791406ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.342+0000 7f791b674640 1 -- 172.21.15.50:0/1782833376 >> 172.21.15.50:0/1782833376 conn(0x7f7914077eb0 msgr2=0x7f79140782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.342+0000 7f791b674640 1 -- 172.21.15.50:0/1782833376 shutdown_connections 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.342+0000 7f791b674640 1 -- 172.21.15.50:0/1782833376 wait complete. 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f791b674640 1 Processor -- start 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f791b674640 1 -- start start 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f791b674640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f791406a840 0x7f7914198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f791b674640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7914107270 0x7f7914198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f791b674640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f791419cfd0 0x7f791419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f791b674640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f79141121b0 con 0x7f791406a840 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f791b674640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7914112030 con 0x7f791419cfd0 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f791b674640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7914112330 con 0x7f7914107270 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f7919410640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f791406a840 0x7f7914198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f7919410640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f791406a840 0x7f7914198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34512/0 (socket says 172.21.15.50:34512) 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f7919410640 1 -- 172.21.15.50:0/2584070437 learned_addr learned my addr 172.21.15.50:0/2584070437 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:17.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f7918c0f640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7914107270 0x7f7914198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f7919c11640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f791419cfd0 0x7f791419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f7919410640 1 -- 172.21.15.50:0/2584070437 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7914107270 msgr2=0x7f7914198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.343+0000 7f7919410640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7914107270 0x7f7914198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.344+0000 7f7919410640 1 -- 172.21.15.50:0/2584070437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f791419cfd0 msgr2=0x7f791419d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.344+0000 7f7919410640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f791419cfd0 0x7f791419d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.344+0000 7f7919410640 1 -- 172.21.15.50:0/2584070437 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791419dac0 con 0x7f791406a840 2024-04-23T18:22:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.344+0000 7f7919410640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f791406a840 0x7f7914198710 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f791000e9f0 tx=0x7f791000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.344+0000 7f790a7fc640 1 -- 172.21.15.50:0/2584070437 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791000cd80 con 0x7f791406a840 2024-04-23T18:22:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.344+0000 7f790a7fc640 1 -- 172.21.15.50:0/2584070437 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79100108e0 con 0x7f791406a840 2024-04-23T18:22:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.344+0000 7f791b674640 1 -- 172.21.15.50:0/2584070437 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f791419ddb0 con 0x7f791406a840 2024-04-23T18:22:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.344+0000 7f790a7fc640 1 -- 172.21.15.50:0/2584070437 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7910018980 con 0x7f791406a840 2024-04-23T18:22:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.344+0000 7f791b674640 1 -- 172.21.15.50:0/2584070437 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79141a55c0 con 0x7f791406a840 2024-04-23T18:22:17.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.346+0000 7f790a7fc640 1 -- 172.21.15.50:0/2584070437 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7910010430 con 0x7f791406a840 2024-04-23T18:22:17.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f791e381740 1 Processor -- start 2024-04-23T18:22:17.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f791e381740 1 -- start start 2024-04-23T18:22:17.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f791e381740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73490 0x5623f9c73870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f791e381740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c73db0 0x5623f9c7df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f791e381740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 0x5623f9c80870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f791e381740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5623f9bd86d0 con 0x5623f9c7e480 2024-04-23T18:22:17.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f791e381740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5623f9b751a0 con 0x5623f9c73db0 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f791e381740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5623f9bd4690 con 0x5623f9c73490 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f790a7fc640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f79000a3da0 0x7f79000a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7918c0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 0x5623f9c80870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7919410640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c73db0 0x5623f9c7df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7918c0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 0x5623f9c80870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34522/0 (socket says 172.21.15.50:34522) 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7918c0f640 1 -- 172.21.15.50:0/4050332720 learned_addr learned my addr 172.21.15.50:0/4050332720 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7919c11640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f79000a3da0 0x7f79000a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7919c11640 1 --2- 172.21.15.50:0/4050332720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73490 0x5623f9c73870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f790a7fc640 1 -- 172.21.15.50:0/2584070437 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f79100d0ac0 con 0x7f791406a840 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7918c0f640 1 -- 172.21.15.50:0/4050332720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73490 msgr2=0x5623f9c73870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7918c0f640 1 --2- 172.21.15.50:0/4050332720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73490 0x5623f9c73870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7918c0f640 1 -- 172.21.15.50:0/4050332720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c73db0 msgr2=0x5623f9c7df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7918c0f640 1 --2- 172.21.15.50:0/4050332720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c73db0 0x5623f9c7df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7918c0f640 1 -- 172.21.15.50:0/4050332720 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623f9b7f5b0 con 0x5623f9c7e480 2024-04-23T18:22:17.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7919c11640 1 --2- 172.21.15.50:0/4050332720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73490 0x5623f9c73870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.347+0000 7f7919410640 1 --2- 172.21.15.50:0/4050332720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c73db0 0x5623f9c7df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.348+0000 7f7918c0f640 1 --2- 172.21.15.50:0/4050332720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 0x5623f9c80870 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7904009900 tx=0x7f7904032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e4f24e9e2e9869 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.348+0000 7f7919c11640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f79000a3da0 0x7f79000a6260 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f790c0098b0 tx=0x7f790c009840 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.348+0000 7f78ff7fe640 1 -- 172.21.15.50:0/4050332720 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904040070 con 0x5623f9c7e480 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.348+0000 7f78ff7fe640 1 -- 172.21.15.50:0/4050332720 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7904004030 con 0x5623f9c7e480 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.348+0000 7f78ff7fe640 1 -- 172.21.15.50:0/4050332720 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904035040 con 0x5623f9c7e480 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 -- 172.21.15.50:0/4050332720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 msgr2=0x5623f9c80870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 --2- 172.21.15.50:0/4050332720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 0x5623f9c80870 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7904009900 tx=0x7f7904032b20 comp rx=0 tx=0).stop 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 -- 172.21.15.50:0/4050332720 shutdown_connections 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 --2- 172.21.15.50:0/4050332720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 0x5623f9c80870 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 --2- 172.21.15.50:0/4050332720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c73db0 0x5623f9c7df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 --2- 172.21.15.50:0/4050332720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73490 0x5623f9c73870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 -- 172.21.15.50:0/4050332720 >> 172.21.15.50:0/4050332720 conn(0x5623f9c6f7b0 msgr2=0x5623f9c6fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 -- 172.21.15.50:0/4050332720 shutdown_connections 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 -- 172.21.15.50:0/4050332720 wait complete. 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 -- start start 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 Processor -- start 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73490 0x5623f9c7de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73db0 0x5623f9c7b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c7e480 0x5623f9c7b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.349+0000 7f791e381740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5623f9c812b0 con 0x5623f9c73db0 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f791e381740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5623f9b72b20 con 0x5623f9c7e480 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f791e381740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5623f9c81430 con 0x5623f9c73490 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f791e381740 10 client.0 _ll_register_callbacks cb 0x5623f9b72530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f7918c0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73db0 0x5623f9c7b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f7918c0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73db0 0x5623f9c7b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34526/0 (socket says 172.21.15.50:34526) 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f7918c0f640 1 -- 172.21.15.50:0/225800301 learned_addr learned my addr 172.21.15.50:0/225800301 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f7919410640 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73490 0x5623f9c7de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f7919c11640 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c7e480 0x5623f9c7b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f7918c0f640 1 -- 172.21.15.50:0/225800301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73490 msgr2=0x5623f9c7de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f7918c0f640 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73490 0x5623f9c7de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f7918c0f640 1 -- 172.21.15.50:0/225800301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c7e480 msgr2=0x5623f9c7b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f7918c0f640 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c7e480 0x5623f9c7b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f7918c0f640 1 -- 172.21.15.50:0/225800301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623f98ea920 con 0x5623f9c73db0 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f7918c0f640 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73db0 0x5623f9c7b150 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7904033000 tx=0x7f7904009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebffb37610ed565 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f78fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f78fcff9640 1 -- 172.21.15.50:0/225800301 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904040070 con 0x5623f9c73db0 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f78fcff9640 1 -- 172.21.15.50:0/225800301 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f79040316a0 con 0x5623f9c73db0 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f791e381740 1 -- 172.21.15.50:0/225800301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7904004030 con 0x5623f9c73db0 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f78fcff9640 1 -- 172.21.15.50:0/225800301 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7904056050 con 0x5623f9c73db0 2024-04-23T18:22:17.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f791e381740 10 client.15972 fetch_fsmap learned FSMap version 26 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.350+0000 7f791e381740 1 -- 172.21.15.50:0/225800301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5623f99b6d30 con 0x5623f9c73db0 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.351+0000 7f78fcff9640 1 -- 172.21.15.50:0/225800301 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7904004030 con 0x5623f9c73db0 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.351+0000 7f78fcff9640 1 -- 172.21.15.50:0/225800301 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f7904004d20 con 0x5623f9c73db0 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.351+0000 7f791e381740 10 client.15972 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.351+0000 7f791e381740 10 client.15972 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.351+0000 7f791e381740 20 client.15972 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.351+0000 7f791e381740 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623f9c8e420 0x5623f9c90810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.351+0000 7f791e381740 4 client.15972 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.351+0000 7f791e381740 1 -- 172.21.15.50:0/225800301 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623f9b96500 con 0x5623f9c8e420 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.351+0000 7f7919410640 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623f9c8e420 0x5623f9c90810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.351+0000 7f78fcff9640 10 client.15972 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.351+0000 7f7919410640 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623f9c8e420 0x5623f9c90810 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58cce0fa825a9998 server_cookie=98e808bc6f8d6460 in_seq=0 out_seq=0 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f78fcff9640 1 -- 172.21.15.50:0/225800301 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623f9b96500 con 0x5623f9c8e420 2024-04-23T18:22:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f78fcff9640 10 client.15972 handle_command_reply: tid=0 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f791e381740 1 client.15972 shutdown 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f791e381740 1 -- 172.21.15.50:0/225800301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73db0 msgr2=0x5623f9c7b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f791e381740 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73db0 0x5623f9c7b150 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7904033000 tx=0x7f7904009660 comp rx=0 tx=0).stop 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f791e381740 1 -- 172.21.15.50:0/225800301 shutdown_connections 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f791e381740 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623f9c8e420 0x5623f9c90810 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f791e381740 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c7e480 0x5623f9c7b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f791e381740 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73db0 0x5623f9c7b150 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f78fcff9640 0 client.15972 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f791e381740 1 --2- 172.21.15.50:0/225800301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73490 0x5623f9c7de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f791e381740 1 -- 172.21.15.50:0/225800301 >> 172.21.15.50:0/225800301 conn(0x5623f9c6f7b0 msgr2=0x5623f9c755b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f791e381740 1 -- 172.21.15.50:0/225800301 shutdown_connections 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f791e381740 1 -- 172.21.15.50:0/225800301 wait complete. 2024-04-23T18:22:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.352+0000 7f791e381740 20 client.15972 trim_cache size 0 max 16384 2024-04-23T18:22:17.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f791e381740 1 Processor -- start 2024-04-23T18:22:17.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f791e381740 1 -- start start 2024-04-23T18:22:17.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f791e381740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c73490 0x5623f9c8a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f791e381740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73db0 0x5623f9c7d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f791e381740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 0x5623f9c95f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f791e381740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f79141b03c0 con 0x5623f9c7e480 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f791e381740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f79141adc30 con 0x5623f9c73490 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f791e381740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f79141abaa0 con 0x5623f9c73db0 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f7919c11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 0x5623f9c95f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f7919c11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 0x5623f9c95f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34534/0 (socket says 172.21.15.50:34534) 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f7919c11640 1 -- 172.21.15.50:0/930365430 learned_addr learned my addr 172.21.15.50:0/930365430 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f7919410640 1 --2- 172.21.15.50:0/930365430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c73490 0x5623f9c8a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.370+0000 7f7918c0f640 1 --2- 172.21.15.50:0/930365430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73db0 0x5623f9c7d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.371+0000 7f7919c11640 1 -- 172.21.15.50:0/930365430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73db0 msgr2=0x5623f9c7d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.371+0000 7f7919c11640 1 --2- 172.21.15.50:0/930365430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73db0 0x5623f9c7d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.371+0000 7f7919c11640 1 -- 172.21.15.50:0/930365430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c73490 msgr2=0x5623f9c8a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.371+0000 7f7919c11640 1 --2- 172.21.15.50:0/930365430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c73490 0x5623f9c8a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.371+0000 7f7919c11640 1 -- 172.21.15.50:0/930365430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623f9c73be0 con 0x5623f9c7e480 2024-04-23T18:22:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.371+0000 7f7919c11640 1 --2- 172.21.15.50:0/930365430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 0x5623f9c95f80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f790c003c30 tx=0x7f790c009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532a4d2eafba76d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.371+0000 7f78feffd640 1 -- 172.21.15.50:0/930365430 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c012e60 con 0x5623f9c7e480 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.371+0000 7f78feffd640 1 -- 172.21.15.50:0/930365430 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f790c003c60 con 0x5623f9c7e480 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.371+0000 7f78feffd640 1 -- 172.21.15.50:0/930365430 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c0053e0 con 0x5623f9c7e480 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.371+0000 7f791e381740 1 -- 172.21.15.50:0/930365430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 msgr2=0x5623f9c95f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 --2- 172.21.15.50:0/930365430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 0x5623f9c95f80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f790c003c30 tx=0x7f790c009600 comp rx=0 tx=0).stop 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 -- 172.21.15.50:0/930365430 shutdown_connections 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 --2- 172.21.15.50:0/930365430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c7e480 0x5623f9c95f80 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 --2- 172.21.15.50:0/930365430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73db0 0x5623f9c7d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 --2- 172.21.15.50:0/930365430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c73490 0x5623f9c8a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 -- 172.21.15.50:0/930365430 >> 172.21.15.50:0/930365430 conn(0x5623f9c6f7b0 msgr2=0x5623f9c76480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 -- 172.21.15.50:0/930365430 shutdown_connections 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 -- 172.21.15.50:0/930365430 wait complete. 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 -- start start 2024-04-23T18:22:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 Processor -- start 2024-04-23T18:22:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73490 0x5623f9c86680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73db0 0x5623f9c86bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c7e480 0x5623f9c7f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f79141ac0f0 con 0x5623f9c73490 2024-04-23T18:22:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f79141ad560 con 0x5623f9c7e480 2024-04-23T18:22:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f79141ac7c0 con 0x5623f9c73db0 2024-04-23T18:22:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f791e381740 10 client.0 _ll_register_callbacks cb 0x7f7904009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.372+0000 7f7919410640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73490 0x5623f9c86680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f7919410640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73490 0x5623f9c86680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34546/0 (socket says 172.21.15.50:34546) 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f7919410640 1 -- 172.21.15.50:0/4210439559 learned_addr learned my addr 172.21.15.50:0/4210439559 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f7919c11640 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c7e480 0x5623f9c7f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f7918c0f640 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73db0 0x5623f9c86bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f7919410640 1 -- 172.21.15.50:0/4210439559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73db0 msgr2=0x5623f9c86bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f7919410640 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73db0 0x5623f9c86bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f7919410640 1 -- 172.21.15.50:0/4210439559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c7e480 msgr2=0x5623f9c7f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f7919410640 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c7e480 0x5623f9c7f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f7919410640 1 -- 172.21.15.50:0/4210439559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623f9c7fbc0 con 0x5623f9c73490 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f7918c0f640 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73db0 0x5623f9c86bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f7919410640 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73490 0x5623f9c86680 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7910098f80 tx=0x7f791006c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ed1a771317574d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f78fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f78fcff9640 1 -- 172.21.15.50:0/4210439559 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79100ce4a0 con 0x5623f9c73490 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f78fcff9640 1 -- 172.21.15.50:0/4210439559 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f7910099190 con 0x5623f9c73490 2024-04-23T18:22:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f791e381740 1 -- 172.21.15.50:0/4210439559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f790c003c60 con 0x5623f9c73490 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f78fcff9640 1 -- 172.21.15.50:0/4210439559 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f79100a5250 con 0x5623f9c73490 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f791e381740 10 client.15978 fetch_fsmap learned FSMap version 26 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f791e381740 1 -- 172.21.15.50:0/4210439559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5623f9cf5030 con 0x5623f9c73490 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.373+0000 7f78fcff9640 1 -- 172.21.15.50:0/4210439559 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f790c003c60 con 0x5623f9c73490 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.374+0000 7f78fcff9640 1 -- 172.21.15.50:0/4210439559 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f791006b200 con 0x5623f9c73490 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.374+0000 7f791e381740 10 client.15978 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.374+0000 7f791e381740 10 client.15978 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.374+0000 7f791e381740 20 client.15978 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.374+0000 7f791e381740 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623f9cf5740 0x5623f9cf7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.374+0000 7f791e381740 4 client.15978 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "cf8448b3", "release": true, "await": true}] 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.374+0000 7f791e381740 1 -- 172.21.15.50:0/4210439559 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "cf8448b3", "release": true, "await": true}) -- 0x5623f9be60f0 con 0x5623f9cf5740 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.374+0000 7f7918c0f640 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623f9cf5740 0x5623f9cf7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.374+0000 7f7918c0f640 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623f9cf5740 0x5623f9cf7b60 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e53223e4e8e00ac8 server_cookie=f759db80fb504625 in_seq=0 out_seq=0 2024-04-23T18:22:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.374+0000 7f78fcff9640 10 client.15978 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:17.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f78fcff9640 1 -- 172.21.15.50:0/4210439559 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5623f9be60f0 con 0x5623f9cf5740 2024-04-23T18:22:17.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f78fcff9640 10 client.15978 handle_command_reply: tid=0 2024-04-23T18:22:17.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f791e381740 1 client.15978 shutdown 2024-04-23T18:22:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f791e381740 1 -- 172.21.15.50:0/4210439559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73490 msgr2=0x5623f9c86680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f791e381740 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73490 0x5623f9c86680 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7910098f80 tx=0x7f791006c4c0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f791e381740 1 -- 172.21.15.50:0/4210439559 shutdown_connections 2024-04-23T18:22:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f791e381740 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623f9cf5740 0x5623f9cf7b60 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f791e381740 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623f9c7e480 0x5623f9c7f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f791e381740 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623f9c73db0 0x5623f9c86bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f791e381740 1 --2- 172.21.15.50:0/4210439559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623f9c73490 0x5623f9c86680 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f78fcff9640 0 client.15978 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f791e381740 1 -- 172.21.15.50:0/4210439559 >> 172.21.15.50:0/4210439559 conn(0x5623f9c6f7b0 msgr2=0x5623f9c6e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f791e381740 1 -- 172.21.15.50:0/4210439559 shutdown_connections 2024-04-23T18:22:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f791e381740 1 -- 172.21.15.50:0/4210439559 wait complete. 2024-04-23T18:22:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.728+0000 7f791e381740 20 client.15978 trim_cache size 0 max 16384 2024-04-23T18:22:17.724 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:22:17.724 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 26, 2024-04-23T18:22:17.724 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:22:17.724 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 12, 2024-04-23T18:22:17.724 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:22:17.724 INFO:teuthology.orchestra.run.smithi050.stdout: "cf8448b3": { 2024-04-23T18:22:17.724 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 12, 2024-04-23T18:22:17.724 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:22:17.724 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:22:17.724 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:22:17.724 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 87.8, 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:17.725 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:22:17.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.729+0000 7f791b674640 1 -- 172.21.15.50:0/2584070437 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f79000a3da0 msgr2=0x7f79000a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.729+0000 7f791b674640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f79000a3da0 0x7f79000a6260 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f790c0098b0 tx=0x7f790c009840 comp rx=0 tx=0).stop 2024-04-23T18:22:17.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.729+0000 7f791b674640 1 -- 172.21.15.50:0/2584070437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f791406a840 msgr2=0x7f7914198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.729+0000 7f791b674640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f791406a840 0x7f7914198710 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f791000e9f0 tx=0x7f791000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.729+0000 7f791b674640 1 -- 172.21.15.50:0/2584070437 shutdown_connections 2024-04-23T18:22:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.729+0000 7f791b674640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f79000a3da0 0x7f79000a6260 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.729+0000 7f791b674640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f791419cfd0 0x7f791419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.729+0000 7f791b674640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7914107270 0x7f7914198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.729+0000 7f791b674640 1 --2- 172.21.15.50:0/2584070437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f791406a840 0x7f7914198710 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.729+0000 7f791b674640 1 -- 172.21.15.50:0/2584070437 >> 172.21.15.50:0/2584070437 conn(0x7f7914077eb0 msgr2=0x7f7914108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.729+0000 7f791b674640 1 -- 172.21.15.50:0/2584070437 shutdown_connections 2024-04-23T18:22:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.729+0000 7f791b674640 1 -- 172.21.15.50:0/2584070437 wait complete. 2024-04-23T18:22:17.734 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'cf8448b3', seconds elapsed: 29.3 2024-04-23T18:22:17.734 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.85 2024-04-23T18:22:17.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:22:17.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.856+0000 7f86d4c64640 1 -- 172.21.15.50:0/2688372763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d010c520 msgr2=0x7f86d010e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.856+0000 7f86d4c64640 1 --2- 172.21.15.50:0/2688372763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d010c520 0x7f86d010e9c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009ca0 tx=0x7f86c802a9e0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.856+0000 7f86d4c64640 1 -- 172.21.15.50:0/2688372763 shutdown_connections 2024-04-23T18:22:17.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.856+0000 7f86d4c64640 1 --2- 172.21.15.50:0/2688372763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d010c520 0x7f86d010e9c0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.856+0000 7f86d4c64640 1 --2- 172.21.15.50:0/2688372763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86d0103b00 0x7f86d010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.856+0000 7f86d4c64640 1 --2- 172.21.15.50:0/2688372763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86d01031e0 0x7f86d01035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.856+0000 7f86d4c64640 1 -- 172.21.15.50:0/2688372763 >> 172.21.15.50:0/2688372763 conn(0x7f86d0077f50 msgr2=0x7f86d0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:17.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.856+0000 7f86d4c64640 1 -- 172.21.15.50:0/2688372763 shutdown_connections 2024-04-23T18:22:17.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.856+0000 7f86d4c64640 1 -- 172.21.15.50:0/2688372763 wait complete. 2024-04-23T18:22:17.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86d4c64640 1 Processor -- start 2024-04-23T18:22:17.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86d4c64640 1 -- start start 2024-04-23T18:22:17.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86d4c64640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86d01031e0 0x7f86d0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86d4c64640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d0103b00 0x7f86d0198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86d4c64640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86d010c520 0x7f86d019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86d4c64640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f86d0112190 con 0x7f86d0103b00 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86d4c64640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f86d0112010 con 0x7f86d01031e0 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86d4c64640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f86d0112310 con 0x7f86d010c520 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86ce59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86d01031e0 0x7f86d0198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86cdd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d0103b00 0x7f86d0198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86ce59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86d01031e0 0x7f86d0198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:48926/0 (socket says 172.21.15.50:48926) 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86ce59c640 1 -- 172.21.15.50:0/3173399836 learned_addr learned my addr 172.21.15.50:0/3173399836 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86ced9d640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86d010c520 0x7f86d019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86cdd9b640 1 -- 172.21.15.50:0/3173399836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86d010c520 msgr2=0x7f86d019cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86cdd9b640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86d010c520 0x7f86d019cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86cdd9b640 1 -- 172.21.15.50:0/3173399836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86d01031e0 msgr2=0x7f86d0198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:17.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86cdd9b640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86d01031e0 0x7f86d0198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:17.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86cdd9b640 1 -- 172.21.15.50:0/3173399836 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d019d6d0 con 0x7f86d0103b00 2024-04-23T18:22:17.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86ced9d640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86d010c520 0x7f86d019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:17.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.857+0000 7f86ce59c640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86d01031e0 0x7f86d0198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:17.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.858+0000 7f86cdd9b640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d0103b00 0x7f86d0198c70 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f86c400e9f0 tx=0x7f86c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:17.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.858+0000 7f86bf7fe640 1 -- 172.21.15.50:0/3173399836 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c400ccc0 con 0x7f86d0103b00 2024-04-23T18:22:17.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.858+0000 7f86bf7fe640 1 -- 172.21.15.50:0/3173399836 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86c40108e0 con 0x7f86d0103b00 2024-04-23T18:22:17.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.858+0000 7f86d4c64640 1 -- 172.21.15.50:0/3173399836 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86d019d9c0 con 0x7f86d0103b00 2024-04-23T18:22:17.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.858+0000 7f86bf7fe640 1 -- 172.21.15.50:0/3173399836 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c4018880 con 0x7f86d0103b00 2024-04-23T18:22:17.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.858+0000 7f86d4c64640 1 -- 172.21.15.50:0/3173399836 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86d01a5250 con 0x7f86d0103b00 2024-04-23T18:22:17.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.859+0000 7f86d4c64640 1 -- 172.21.15.50:0/3173399836 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86d0068d70 con 0x7f86d0103b00 2024-04-23T18:22:17.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.860+0000 7f86bf7fe640 1 -- 172.21.15.50:0/3173399836 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f86c4010430 con 0x7f86d0103b00 2024-04-23T18:22:17.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.861+0000 7f86bf7fe640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f86b80a3d20 0x7f86b80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:17.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.861+0000 7f86bf7fe640 1 -- 172.21.15.50:0/3173399836 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f86c40d0f40 con 0x7f86d0103b00 2024-04-23T18:22:17.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.861+0000 7f86ce59c640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f86b80a3d20 0x7f86b80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:17.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.861+0000 7f86ce59c640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f86b80a3d20 0x7f86b80a61e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f86c0011550 tx=0x7f86c0005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:17.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:17.864+0000 7f86bf7fe640 1 -- 172.21.15.50:0/3173399836 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f86c409cfa0 con 0x7f86d0103b00 2024-04-23T18:22:17.980 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:17 smithi188 ceph-mon[32694]: pgmap v116: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 2.2 MiB/s wr, 86 op/s 2024-04-23T18:22:17.980 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:17 smithi188 ceph-mon[32694]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T18:22:17.980 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:17 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6836/1939040460,v1:172.21.15.177:6837/1939040460] up:boot 2024-04-23T18:22:17.980 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:17 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:22:17.980 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:17 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:22:17.980 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:17 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:17.980 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:17 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/452994111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:18.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:17 smithi177 ceph-mon[32308]: pgmap v116: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 2.2 MiB/s wr, 86 op/s 2024-04-23T18:22:18.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:17 smithi177 ceph-mon[32308]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T18:22:18.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:17 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6836/1939040460,v1:172.21.15.177:6837/1939040460] up:boot 2024-04-23T18:22:18.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:17 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:22:18.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:17 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:22:18.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:17 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:18.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:17 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/452994111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:18.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.108+0000 7f86d4c64640 1 -- 172.21.15.50:0/3173399836 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86d0199a30 con 0x7f86d0103b00 2024-04-23T18:22:18.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.109+0000 7f86bf7fe640 1 -- 172.21.15.50:0/3173399836 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11879 (secure 0 0 0) 0x7f86d0199a30 con 0x7f86d0103b00 2024-04-23T18:22:18.102 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:22:18.102 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:22:16.725864+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25547":{"gid":25547,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1939040460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1939040460},{"type":"v1","addr":"172.21.15.177:6837","nonce":1939040460}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:22:18.102 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 26 2024-04-23T18:22:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.111+0000 7f86d4c64640 1 -- 172.21.15.50:0/3173399836 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f86b80a3d20 msgr2=0x7f86b80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.111+0000 7f86d4c64640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f86b80a3d20 0x7f86b80a61e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f86c0011550 tx=0x7f86c0005f90 comp rx=0 tx=0).stop 2024-04-23T18:22:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.111+0000 7f86d4c64640 1 -- 172.21.15.50:0/3173399836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d0103b00 msgr2=0x7f86d0198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.111+0000 7f86d4c64640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d0103b00 0x7f86d0198c70 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f86c400e9f0 tx=0x7f86c400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.111+0000 7f86d4c64640 1 -- 172.21.15.50:0/3173399836 shutdown_connections 2024-04-23T18:22:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.111+0000 7f86d4c64640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f86b80a3d20 0x7f86b80a61e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.111+0000 7f86d4c64640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86d010c520 0x7f86d019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.111+0000 7f86d4c64640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86d0103b00 0x7f86d0198c70 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.111+0000 7f86d4c64640 1 --2- 172.21.15.50:0/3173399836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86d01031e0 0x7f86d0198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.111+0000 7f86d4c64640 1 -- 172.21.15.50:0/3173399836 >> 172.21.15.50:0/3173399836 conn(0x7f86d0077f50 msgr2=0x7f86d00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:18.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.111+0000 7f86d4c64640 1 -- 172.21.15.50:0/3173399836 shutdown_connections 2024-04-23T18:22:18.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.111+0000 7f86d4c64640 1 -- 172.21.15.50:0/3173399836 wait complete. 2024-04-23T18:22:18.116 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:17 smithi050 ceph-mon[29187]: pgmap v116: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 2.2 MiB/s wr, 86 op/s 2024-04-23T18:22:18.116 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:17 smithi050 ceph-mon[29187]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T18:22:18.116 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:17 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6836/1939040460,v1:172.21.15.177:6837/1939040460] up:boot 2024-04-23T18:22:18.116 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:17 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:22:18.116 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:17 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:22:18.117 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:17 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:18.117 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:17 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/452994111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:18.118 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 87.85 --await-for 42.0' 2024-04-23T18:22:18.118 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 87.85 --await-for 42.0 2024-04-23T18:22:18.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242218 4.91 MB/sec execute 52 sec latency 28738.226 ms 2024-04-23T18:22:18.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.241+0000 7fbb17cae640 1 -- 172.21.15.50:0/3489808180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb10106170 msgr2=0x7fbb1010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.241+0000 7fbb17cae640 1 --2- 172.21.15.50:0/3489808180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb10106170 0x7fbb1010dd40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009ca0 tx=0x7fbb0802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.241+0000 7fbb17cae640 1 -- 172.21.15.50:0/3489808180 shutdown_connections 2024-04-23T18:22:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.241+0000 7fbb17cae640 1 --2- 172.21.15.50:0/3489808180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb10106170 0x7fbb1010dd40 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.241+0000 7fbb17cae640 1 --2- 172.21.15.50:0/3489808180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb10101a50 0x7fbb10105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.241+0000 7fbb17cae640 1 --2- 172.21.15.50:0/3489808180 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb101010a0 0x7fbb10101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.241+0000 7fbb17cae640 1 -- 172.21.15.50:0/3489808180 >> 172.21.15.50:0/3489808180 conn(0x7fbb10077f50 msgr2=0x7fbb10078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.241+0000 7fbb17cae640 1 -- 172.21.15.50:0/3489808180 shutdown_connections 2024-04-23T18:22:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.241+0000 7fbb17cae640 1 -- 172.21.15.50:0/3489808180 wait complete. 2024-04-23T18:22:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.241+0000 7fbb17cae640 1 Processor -- start 2024-04-23T18:22:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb17cae640 1 -- start start 2024-04-23T18:22:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb17cae640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb101010a0 0x7fbb10198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb17cae640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb10106170 0x7fbb10198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb17cae640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb1019cfd0 0x7fbb1019d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb17cae640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101110e0 con 0x7fbb1019cfd0 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb17cae640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbb10110f60 con 0x7fbb101010a0 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb17cae640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbb10111260 con 0x7fbb10106170 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb1624b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb1019cfd0 0x7fbb1019d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb1624b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb1019cfd0 0x7fbb1019d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34576/0 (socket says 172.21.15.50:34576) 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb15249640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb10106170 0x7fbb10198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb1624b640 1 -- 172.21.15.50:0/4157751613 learned_addr learned my addr 172.21.15.50:0/4157751613 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb15a4a640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb101010a0 0x7fbb10198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb1624b640 1 -- 172.21.15.50:0/4157751613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb10106170 msgr2=0x7fbb10198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb1624b640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb10106170 0x7fbb10198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb1624b640 1 -- 172.21.15.50:0/4157751613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb101010a0 msgr2=0x7fbb10198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb1624b640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb101010a0 0x7fbb10198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb1624b640 1 -- 172.21.15.50:0/4157751613 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1019dac0 con 0x7fbb1019cfd0 2024-04-23T18:22:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb15a4a640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb101010a0 0x7fbb10198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.242+0000 7fbb1624b640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb1019cfd0 0x7fbb1019d480 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbb08005cc0 tx=0x7fbb08031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.243+0000 7fbb06ffd640 1 -- 172.21.15.50:0/4157751613 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb08003b70 con 0x7fbb1019cfd0 2024-04-23T18:22:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.243+0000 7fbb06ffd640 1 -- 172.21.15.50:0/4157751613 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb080318f0 con 0x7fbb1019cfd0 2024-04-23T18:22:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.243+0000 7fbb17cae640 1 -- 172.21.15.50:0/4157751613 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb1019dd50 con 0x7fbb1019cfd0 2024-04-23T18:22:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.243+0000 7fbb06ffd640 1 -- 172.21.15.50:0/4157751613 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0802f950 con 0x7fbb1019cfd0 2024-04-23T18:22:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.243+0000 7fbb17cae640 1 -- 172.21.15.50:0/4157751613 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb101a55a0 con 0x7fbb1019cfd0 2024-04-23T18:22:18.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.245+0000 7fbb06ffd640 1 -- 172.21.15.50:0/4157751613 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbb0802fb50 con 0x7fbb1019cfd0 2024-04-23T18:22:18.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb1a9c9740 1 Processor -- start 2024-04-23T18:22:18.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb1a9c9740 1 -- start start 2024-04-23T18:22:18.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb1a9c9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3cf80 0x5631d4e3d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb1a9c9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e4bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb1a9c9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e4c4a0 0x5631d4e4e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb1a9c9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5631d4da2370 con 0x5631d4e3d8a0 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb1a9c9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5631d4d3f2e0 con 0x5631d4e4c4a0 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb1a9c9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5631d4d9e330 con 0x5631d4e3cf80 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb15249640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3cf80 0x5631d4e3d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb15a4a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e4bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb15249640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3cf80 0x5631d4e3d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:59656/0 (socket says 172.21.15.50:59656) 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb15249640 1 -- 172.21.15.50:0/3852112232 learned_addr learned my addr 172.21.15.50:0/3852112232 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb15249640 1 --2- 172.21.15.50:0/3852112232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e4c4a0 0x5631d4e4e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb06ffd640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbafc0a3da0 0x7fbafc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb15249640 1 -- 172.21.15.50:0/3852112232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e4c4a0 msgr2=0x5631d4e4e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb15249640 1 --2- 172.21.15.50:0/3852112232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e4c4a0 0x5631d4e4e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb1624b640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbafc0a3da0 0x7fbafc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb15249640 1 -- 172.21.15.50:0/3852112232 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 msgr2=0x5631d4e4bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb15249640 1 --2- 172.21.15.50:0/3852112232 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e4bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.246+0000 7fbb15249640 1 -- 172.21.15.50:0/3852112232 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631d4b80d80 con 0x5631d4e3cf80 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.247+0000 7fbb06ffd640 1 -- 172.21.15.50:0/4157751613 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbb080f1980 con 0x7fbb1019cfd0 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.247+0000 7fbb15249640 1 --2- 172.21.15.50:0/3852112232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3cf80 0x5631d4e3d360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009560 tx=0x7fbb00004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a832783f7221fa93 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.247+0000 7fbb1624b640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbafc0a3da0 0x7fbafc0a6260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbb080b9ea0 tx=0x7fbb08090530 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.247+0000 7fbafbfff640 1 -- 172.21.15.50:0/3852112232 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00004050 con 0x5631d4e3cf80 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.247+0000 7fbafbfff640 1 -- 172.21.15.50:0/3852112232 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb000054d0 con 0x5631d4e3cf80 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.247+0000 7fbafbfff640 1 -- 172.21.15.50:0/3852112232 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00004c10 con 0x5631d4e3cf80 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.248+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/3852112232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3cf80 msgr2=0x5631d4e3d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.248+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/3852112232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3cf80 0x5631d4e3d360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009560 tx=0x7fbb00004730 comp rx=0 tx=0).stop 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.248+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/3852112232 shutdown_connections 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.248+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/3852112232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e4c4a0 0x5631d4e4e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.248+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/3852112232 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e4bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.248+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/3852112232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3cf80 0x5631d4e3d360 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.248+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/3852112232 >> 172.21.15.50:0/3852112232 conn(0x5631d4e393c0 msgr2=0x5631d4e397a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.248+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/3852112232 shutdown_connections 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.248+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/3852112232 wait complete. 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.248+0000 7fbb1a9c9740 1 -- start start 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.248+0000 7fbb1a9c9740 1 Processor -- start 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb1a9c9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e3cf80 0x5631d4e493b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb1a9c9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e498f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb1a9c9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e4c4a0 0x5631d4e49e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb1a9c9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5631d4e4f2e0 con 0x5631d4e3d8a0 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb1a9c9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5631d4d3cfb0 con 0x5631d4e3cf80 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb1a9c9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5631d4e4f460 con 0x5631d4e4c4a0 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb1a9c9740 10 client.0 _ll_register_callbacks cb 0x5631d4d3c9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb15a4a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb15a4a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e498f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34594/0 (socket says 172.21.15.50:34594) 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb15a4a640 1 -- 172.21.15.50:0/1398136918 learned_addr learned my addr 172.21.15.50:0/1398136918 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb15249640 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e3cf80 0x5631d4e493b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb15249640 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e4c4a0 0x5631d4e49e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb15a4a640 1 -- 172.21.15.50:0/1398136918 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e4c4a0 msgr2=0x5631d4e49e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb15a4a640 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e4c4a0 0x5631d4e49e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb15a4a640 1 -- 172.21.15.50:0/1398136918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e3cf80 msgr2=0x5631d4e493b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb15a4a640 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e3cf80 0x5631d4e493b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb15a4a640 1 -- 172.21.15.50:0/1398136918 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631d4be1510 con 0x5631d4e3d8a0 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb15a4a640 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e498f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c012d50 tx=0x7fbb0c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f41fd6a2e1322 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbaf97fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbaf97fa640 1 -- 172.21.15.50:0/1398136918 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c00fe20 con 0x5631d4e3d8a0 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbaf97fa640 1 -- 172.21.15.50:0/1398136918 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbb0c010430 con 0x5631d4e3d8a0 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.249+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1398136918 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb000054d0 con 0x5631d4e3d8a0 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.250+0000 7fbaf97fa640 1 -- 172.21.15.50:0/1398136918 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbb0c010630 con 0x5631d4e3d8a0 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.250+0000 7fbb1a9c9740 10 client.15996 fetch_fsmap learned FSMap version 26 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.250+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1398136918 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5631d4a74670 con 0x5631d4e3d8a0 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.250+0000 7fbaf97fa640 1 -- 172.21.15.50:0/1398136918 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb000054d0 con 0x5631d4e3d8a0 2024-04-23T18:22:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.250+0000 7fbaf97fa640 1 -- 172.21.15.50:0/1398136918 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fbb0c01c7d0 con 0x5631d4e3d8a0 2024-04-23T18:22:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.250+0000 7fbb1a9c9740 10 client.15996 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:22:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.250+0000 7fbb1a9c9740 10 client.15996 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:22:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.250+0000 7fbb1a9c9740 20 client.15996 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.250+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631d4e5ca10 0x5631d4e5ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.250+0000 7fbb1a9c9740 4 client.15996 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.250+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1398136918 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631d4dafd90 con 0x5631d4e5ca10 2024-04-23T18:22:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.250+0000 7fbb15249640 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631d4e5ca10 0x5631d4e5ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.251+0000 7fbb15249640 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631d4e5ca10 0x5631d4e5ee60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=909dd4df11900558 server_cookie=22b54823cc51aba9 in_seq=0 out_seq=0 2024-04-23T18:22:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.251+0000 7fbaf97fa640 10 client.15996 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.251+0000 7fbaf97fa640 1 -- 172.21.15.50:0/1398136918 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631d4dafd90 con 0x5631d4e5ca10 2024-04-23T18:22:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.251+0000 7fbaf97fa640 10 client.15996 handle_command_reply: tid=0 2024-04-23T18:22:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.251+0000 7fbb1a9c9740 1 client.15996 shutdown 2024-04-23T18:22:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.252+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1398136918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 msgr2=0x5631d4e498f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.252+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e498f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c012d50 tx=0x7fbb0c00f790 comp rx=0 tx=0).stop 2024-04-23T18:22:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.252+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1398136918 shutdown_connections 2024-04-23T18:22:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.252+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631d4e5ca10 0x5631d4e5ee60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.252+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e4c4a0 0x5631d4e49e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.252+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e498f0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.252+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/1398136918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e3cf80 0x5631d4e493b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.252+0000 7fbaf97fa640 0 client.15996 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.252+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1398136918 >> 172.21.15.50:0/1398136918 conn(0x5631d4e393c0 msgr2=0x5631d4e3f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.252+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1398136918 shutdown_connections 2024-04-23T18:22:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.252+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1398136918 wait complete. 2024-04-23T18:22:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.252+0000 7fbb1a9c9740 20 client.15996 trim_cache size 0 max 16384 2024-04-23T18:22:18.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.269+0000 7fbb1a9c9740 1 Processor -- start 2024-04-23T18:22:18.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.269+0000 7fbb1a9c9740 1 -- start start 2024-04-23T18:22:18.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb1a9c9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e3cf80 0x5631d4e38d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb1a9c9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3d8a0 0x5631d4e49a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb1a9c9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e4c4a0 0x5631d4e64770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb1a9c9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101ae1e0 con 0x5631d4e4c4a0 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb1a9c9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101aba50 con 0x5631d4e3cf80 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb1a9c9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101a98a0 con 0x5631d4e3d8a0 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb15249640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e4c4a0 0x5631d4e64770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb15249640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e4c4a0 0x5631d4e64770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34606/0 (socket says 172.21.15.50:34606) 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb15249640 1 -- 172.21.15.50:0/1235429224 learned_addr learned my addr 172.21.15.50:0/1235429224 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb15a4a640 1 --2- 172.21.15.50:0/1235429224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3d8a0 0x5631d4e49a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb15249640 1 --2- 172.21.15.50:0/1235429224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e3cf80 0x5631d4e38d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb15249640 1 -- 172.21.15.50:0/1235429224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3d8a0 msgr2=0x5631d4e49a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb15249640 1 --2- 172.21.15.50:0/1235429224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3d8a0 0x5631d4e49a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb15249640 1 -- 172.21.15.50:0/1235429224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e3cf80 msgr2=0x5631d4e38d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb15249640 1 --2- 172.21.15.50:0/1235429224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e3cf80 0x5631d4e38d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb15249640 1 -- 172.21.15.50:0/1235429224 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631d4e3d6d0 con 0x5631d4e4c4a0 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbb15249640 1 --2- 172.21.15.50:0/1235429224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e4c4a0 0x5631d4e64770 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbb00004690 tx=0x7fbb00009ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f189d57b5a19b218 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbafb7fe640 1 -- 172.21.15.50:0/1235429224 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00011110 con 0x5631d4e4c4a0 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbafb7fe640 1 -- 172.21.15.50:0/1235429224 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb000112b0 con 0x5631d4e4c4a0 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.270+0000 7fbafb7fe640 1 -- 172.21.15.50:0/1235429224 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0000fde0 con 0x5631d4e4c4a0 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.271+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1235429224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e4c4a0 msgr2=0x5631d4e64770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.271+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/1235429224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e4c4a0 0x5631d4e64770 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbb00004690 tx=0x7fbb00009ae0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.271+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1235429224 shutdown_connections 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.271+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/1235429224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e4c4a0 0x5631d4e64770 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.271+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/1235429224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3d8a0 0x5631d4e49a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.271+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/1235429224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e3cf80 0x5631d4e38d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.271+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1235429224 >> 172.21.15.50:0/1235429224 conn(0x5631d4e393c0 msgr2=0x5631d4e49370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.271+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1235429224 shutdown_connections 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.271+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/1235429224 wait complete. 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.271+0000 7fbb1a9c9740 1 -- start start 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.271+0000 7fbb1a9c9740 1 Processor -- start 2024-04-23T18:22:18.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb1a9c9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3cf80 0x5631d4e54780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb1a9c9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e54cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb1a9c9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e4c4a0 0x5631d4e51750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb1a9c9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101a9f10 con 0x5631d4e3d8a0 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb1a9c9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101ab380 con 0x5631d4e4c4a0 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb1a9c9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101aa5e0 con 0x5631d4e3cf80 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb15a4a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e54cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb1a9c9740 10 client.0 _ll_register_callbacks cb 0x7fbb0c012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb15a4a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e54cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34622/0 (socket says 172.21.15.50:34622) 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb15a4a640 1 -- 172.21.15.50:0/202023582 learned_addr learned my addr 172.21.15.50:0/202023582 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb15249640 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3cf80 0x5631d4e54780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb15249640 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e4c4a0 0x5631d4e51750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb15a4a640 1 -- 172.21.15.50:0/202023582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3cf80 msgr2=0x5631d4e54780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb15a4a640 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3cf80 0x5631d4e54780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb15a4a640 1 -- 172.21.15.50:0/202023582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e4c4a0 msgr2=0x5631d4e51750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb15a4a640 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e4c4a0 0x5631d4e51750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb15a4a640 1 -- 172.21.15.50:0/202023582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631d4e51d40 con 0x5631d4e3d8a0 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb15249640 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e4c4a0 0x5631d4e51750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbb15a4a640 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e54cc0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x5631d4e48c10 tx=0x7fbb0c005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2833d1522318243d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.272+0000 7fbaf97fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbaf97fa640 1 -- 172.21.15.50:0/202023582 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c010430 con 0x5631d4e3d8a0 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbaf97fa640 1 -- 172.21.15.50:0/202023582 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbb0c0107d0 con 0x5631d4e3d8a0 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/202023582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb000112b0 con 0x5631d4e3d8a0 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbaf97fa640 1 -- 172.21.15.50:0/202023582 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbb0c0109d0 con 0x5631d4e3d8a0 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbb1a9c9740 10 client.16002 fetch_fsmap learned FSMap version 26 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/202023582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5631d4ec38e0 con 0x5631d4e3d8a0 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbaf97fa640 1 -- 172.21.15.50:0/202023582 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb000112b0 con 0x5631d4e3d8a0 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbaf97fa640 1 -- 172.21.15.50:0/202023582 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fbb0c02c030 con 0x5631d4e3d8a0 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbb1a9c9740 10 client.16002 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:22:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbb1a9c9740 10 client.16002 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:22:18.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbb1a9c9740 20 client.16002 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:18.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631d4ec4160 0x5631d4ec6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:18.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbb1a9c9740 4 client.16002 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.85, "await_for": 42.0}] 2024-04-23T18:22:18.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/202023582 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.85, "await_for": 42.0}) -- 0x5631d4e441f0 con 0x5631d4ec4160 2024-04-23T18:22:18.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.273+0000 7fbb15249640 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631d4ec4160 0x5631d4ec6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:18.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.274+0000 7fbaf97fa640 10 client.16002 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:18.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.274+0000 7fbb15249640 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631d4ec4160 0x5631d4ec6580 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6567bcc7e598586d server_cookie=9473f112bde2cc50 in_seq=0 out_seq=0 2024-04-23T18:22:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbaf97fa640 1 -- 172.21.15.50:0/202023582 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5631d4e441f0 con 0x5631d4ec4160 2024-04-23T18:22:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbaf97fa640 10 client.16002 handle_command_reply: tid=0 2024-04-23T18:22:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbb1a9c9740 1 client.16002 shutdown 2024-04-23T18:22:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/202023582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 msgr2=0x5631d4e54cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e54cc0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x5631d4e48c10 tx=0x7fbb0c005d30 comp rx=0 tx=0).stop 2024-04-23T18:22:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/202023582 shutdown_connections 2024-04-23T18:22:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631d4ec4160 0x5631d4ec6580 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631d4e4c4a0 0x5631d4e51750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631d4e3d8a0 0x5631d4e54cc0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbb1a9c9740 1 --2- 172.21.15.50:0/202023582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631d4e3cf80 0x5631d4e54780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbaf97fa640 0 client.16002 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/202023582 >> 172.21.15.50:0/202023582 conn(0x5631d4e393c0 msgr2=0x5631d4e49be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:18.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/202023582 shutdown_connections 2024-04-23T18:22:18.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbb1a9c9740 1 -- 172.21.15.50:0/202023582 wait complete. 2024-04-23T18:22:18.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.321+0000 7fbb1a9c9740 20 client.16002 trim_cache size 0 max 16384 2024-04-23T18:22:18.316 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:22:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 26, 2024-04-23T18:22:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:22:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 14, 2024-04-23T18:22:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:22:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: "d13aa7a2": { 2024-04-23T18:22:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 14, 2024-04-23T18:22:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 87.8, 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:18.318 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:18.318 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:18.318 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:22:18.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.322+0000 7fbb17cae640 1 -- 172.21.15.50:0/4157751613 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbafc0a3da0 msgr2=0x7fbafc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:18.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.322+0000 7fbb17cae640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbafc0a3da0 0x7fbafc0a6260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbb080b9ea0 tx=0x7fbb08090530 comp rx=0 tx=0).stop 2024-04-23T18:22:18.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.322+0000 7fbb17cae640 1 -- 172.21.15.50:0/4157751613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb1019cfd0 msgr2=0x7fbb1019d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:18.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.322+0000 7fbb17cae640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb1019cfd0 0x7fbb1019d480 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbb08005cc0 tx=0x7fbb08031700 comp rx=0 tx=0).stop 2024-04-23T18:22:18.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.322+0000 7fbb17cae640 1 -- 172.21.15.50:0/4157751613 shutdown_connections 2024-04-23T18:22:18.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.322+0000 7fbb17cae640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbafc0a3da0 0x7fbafc0a6260 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.322+0000 7fbb17cae640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb1019cfd0 0x7fbb1019d480 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.322+0000 7fbb17cae640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb10106170 0x7fbb10198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.322+0000 7fbb17cae640 1 --2- 172.21.15.50:0/4157751613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb101010a0 0x7fbb10198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:18.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.322+0000 7fbb17cae640 1 -- 172.21.15.50:0/4157751613 >> 172.21.15.50:0/4157751613 conn(0x7fbb10077f50 msgr2=0x7fbb10104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:18.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.322+0000 7fbb17cae640 1 -- 172.21.15.50:0/4157751613 shutdown_connections 2024-04-23T18:22:18.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:18.322+0000 7fbb17cae640 1 -- 172.21.15.50:0/4157751613 wait complete. 2024-04-23T18:22:18.326 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'd13aa7a2', quiesce took 0.6 seconds. Will release after: 27.85 2024-04-23T18:22:19.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:18 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3173399836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:19.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:18 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3173399836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:19.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:18 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3173399836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:19.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 4.84 MB/sec execute 53 sec latency 887.540 ms 2024-04-23T18:22:20.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:19 smithi177 ceph-mon[32308]: pgmap v118: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 1.7 MiB/s wr, 67 op/s 2024-04-23T18:22:20.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:19 smithi188 ceph-mon[32694]: pgmap v118: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 1.7 MiB/s wr, 67 op/s 2024-04-23T18:22:20.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:19 smithi050 ceph-mon[29187]: pgmap v118: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 1.7 MiB/s wr, 67 op/s 2024-04-23T18:22:20.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 4.75 MB/sec execute 54 sec latency 1887.601 ms 2024-04-23T18:22:21.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 4.67 MB/sec execute 55 sec latency 2887.703 ms 2024-04-23T18:22:21.996 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:21 smithi050 ceph-mon[29187]: pgmap v119: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 1.7 MiB/s wr, 67 op/s 2024-04-23T18:22:22.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:21 smithi177 ceph-mon[32308]: pgmap v119: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 1.7 MiB/s wr, 67 op/s 2024-04-23T18:22:22.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:21 smithi188 ceph-mon[32694]: pgmap v119: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 1.7 MiB/s wr, 67 op/s 2024-04-23T18:22:22.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 4.58 MB/sec execute 56 sec latency 3887.766 ms 2024-04-23T18:22:23.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 4.50 MB/sec execute 57 sec latency 4887.909 ms 2024-04-23T18:22:24.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:23 smithi177 ceph-mon[32308]: pgmap v120: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-04-23T18:22:24.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:23 smithi188 ceph-mon[32694]: pgmap v120: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-04-23T18:22:24.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:23 smithi050 ceph-mon[29187]: pgmap v120: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-04-23T18:22:24.200 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 4.42 MB/sec execute 58 sec latency 5887.991 ms 2024-04-23T18:22:25.201 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 4.35 MB/sec execute 59 sec latency 6888.103 ms 2024-04-23T18:22:26.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:25 smithi177 ceph-mon[32308]: pgmap v121: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-04-23T18:22:26.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:25 smithi188 ceph-mon[32694]: pgmap v121: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-04-23T18:22:26.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:25 smithi050 ceph-mon[29187]: pgmap v121: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-04-23T18:22:26.201 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 4.28 MB/sec execute 60 sec latency 7888.202 ms 2024-04-23T18:22:27.201 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 4.21 MB/sec execute 61 sec latency 8888.300 ms 2024-04-23T18:22:28.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:27 smithi177 ceph-mon[32308]: pgmap v122: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 835 KiB/s wr, 69 op/s 2024-04-23T18:22:28.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:27 smithi188 ceph-mon[32694]: pgmap v122: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 835 KiB/s wr, 69 op/s 2024-04-23T18:22:28.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:27 smithi050 ceph-mon[29187]: pgmap v122: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 835 KiB/s wr, 69 op/s 2024-04-23T18:22:28.201 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 4.14 MB/sec execute 62 sec latency 9888.430 ms 2024-04-23T18:22:29.201 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 4.07 MB/sec execute 63 sec latency 10888.481 ms 2024-04-23T18:22:30.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:29 smithi177 ceph-mon[32308]: pgmap v123: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 705 KiB/s wr, 58 op/s 2024-04-23T18:22:30.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.084+0000 7fa0d4064640 1 -- 172.21.15.50:0/541720408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0cc074250 msgr2=0x7fa0cc074630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.084+0000 7fa0d4064640 1 --2- 172.21.15.50:0/541720408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0cc074250 0x7fa0cc074630 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8009ca0 tx=0x7fa0c802a950 comp rx=0 tx=0).stop 2024-04-23T18:22:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.084+0000 7fa0d4064640 1 -- 172.21.15.50:0/541720408 shutdown_connections 2024-04-23T18:22:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.084+0000 7fa0d4064640 1 --2- 172.21.15.50:0/541720408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0cc10e840 0x7fa0cc110c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.084+0000 7fa0d4064640 1 --2- 172.21.15.50:0/541720408 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0cc074b70 0x7fa0cc10e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.084+0000 7fa0d4064640 1 --2- 172.21.15.50:0/541720408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0cc074250 0x7fa0cc074630 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.084+0000 7fa0d4064640 1 -- 172.21.15.50:0/541720408 >> 172.21.15.50:0/541720408 conn(0x7fa0cc0793e0 msgr2=0x7fa0cc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.084+0000 7fa0d4064640 1 -- 172.21.15.50:0/541720408 shutdown_connections 2024-04-23T18:22:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.084+0000 7fa0d4064640 1 -- 172.21.15.50:0/541720408 wait complete. 2024-04-23T18:22:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.085+0000 7fa0d4064640 1 Processor -- start 2024-04-23T18:22:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.085+0000 7fa0d4064640 1 -- start start 2024-04-23T18:22:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.085+0000 7fa0d4064640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0cc074250 0x7fa0cc1a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.085+0000 7fa0d4064640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0cc074b70 0x7fa0cc1a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.085+0000 7fa0d4064640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0cc10e840 0x7fa0cc19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.085+0000 7fa0d4064640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc114400 con 0x7fa0cc10e840 2024-04-23T18:22:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.085+0000 7fa0d4064640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc114280 con 0x7fa0cc074250 2024-04-23T18:22:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.085+0000 7fa0d4064640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc114580 con 0x7fa0cc074b70 2024-04-23T18:22:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.085+0000 7fa0d2601640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0cc10e840 0x7fa0cc19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.085+0000 7fa0d2601640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0cc10e840 0x7fa0cc19a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37260/0 (socket says 172.21.15.50:37260) 2024-04-23T18:22:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.085+0000 7fa0d2601640 1 -- 172.21.15.50:0/4016678400 learned_addr learned my addr 172.21.15.50:0/4016678400 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.085+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0cc074b70 0x7fa0cc1a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0d1e00640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0cc074250 0x7fa0cc1a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0d2601640 1 -- 172.21.15.50:0/4016678400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0cc074b70 msgr2=0x7fa0cc1a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0d2601640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0cc074b70 0x7fa0cc1a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0d2601640 1 -- 172.21.15.50:0/4016678400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0cc074250 msgr2=0x7fa0cc1a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0d2601640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0cc074250 0x7fa0cc1a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0d2601640 1 -- 172.21.15.50:0/4016678400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0cc19af80 con 0x7fa0cc10e840 2024-04-23T18:22:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0d1e00640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0cc074250 0x7fa0cc1a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0cc074b70 0x7fa0cc1a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0d2601640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0cc10e840 0x7fa0cc19a6c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00d680 tx=0x7fa0bc00db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0c2ffd640 1 -- 172.21.15.50:0/4016678400 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc017070 con 0x7fa0cc10e840 2024-04-23T18:22:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0d4064640 1 -- 172.21.15.50:0/4016678400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0cc19b270 con 0x7fa0cc10e840 2024-04-23T18:22:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0c2ffd640 1 -- 172.21.15.50:0/4016678400 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0bc009820 con 0x7fa0cc10e840 2024-04-23T18:22:30.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0c2ffd640 1 -- 172.21.15.50:0/4016678400 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc004c40 con 0x7fa0cc10e840 2024-04-23T18:22:30.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.086+0000 7fa0d4064640 1 -- 172.21.15.50:0/4016678400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0cc1a7530 con 0x7fa0cc10e840 2024-04-23T18:22:30.085 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:29 smithi188 ceph-mon[32694]: pgmap v123: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 705 KiB/s wr, 58 op/s 2024-04-23T18:22:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.089+0000 7fa0d6d79740 1 Processor -- start 2024-04-23T18:22:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.089+0000 7fa0c2ffd640 1 -- 172.21.15.50:0/4016678400 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa0bc0099d0 con 0x7fa0cc10e840 2024-04-23T18:22:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d6d79740 1 -- start start 2024-04-23T18:22:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d6d79740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67e620 0x55839b67ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d6d79740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 0x55839b6892c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d6d79740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b689800 0x55839b68bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d6d79740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55839b5e3970 con 0x55839b67ef40 2024-04-23T18:22:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d6d79740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55839b5804a0 con 0x55839b67e620 2024-04-23T18:22:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d6d79740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55839b5df930 con 0x55839b689800 2024-04-23T18:22:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d1e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 0x55839b6892c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d1e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 0x55839b6892c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37266/0 (socket says 172.21.15.50:37266) 2024-04-23T18:22:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d1e00640 1 -- 172.21.15.50:0/1917535201 learned_addr learned my addr 172.21.15.50:0/1917535201 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0c2ffd640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0b80a3dc0 0x7fa0b80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/1917535201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67e620 0x55839b67ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/1917535201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b689800 0x55839b68bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d1e00640 1 -- 172.21.15.50:0/1917535201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b689800 msgr2=0x55839b68bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d1e00640 1 --2- 172.21.15.50:0/1917535201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b689800 0x55839b68bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d1e00640 1 -- 172.21.15.50:0/1917535201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67e620 msgr2=0x55839b67ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d1e00640 1 --2- 172.21.15.50:0/1917535201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67e620 0x55839b67ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d2601640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0b80a3dc0 0x7fa0b80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.090+0000 7fa0d1e00640 1 -- 172.21.15.50:0/1917535201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55839b2f6210 con 0x55839b67ef40 2024-04-23T18:22:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.091+0000 7fa0c2ffd640 1 -- 172.21.15.50:0/4016678400 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa0bc0d0270 con 0x7fa0cc10e840 2024-04-23T18:22:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.091+0000 7fa0d1e00640 1 --2- 172.21.15.50:0/1917535201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 0x55839b6892c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa0c802fdc0 tx=0x7fa0c80093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b48ad23539bf6df server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.091+0000 7fa0b7fff640 1 -- 172.21.15.50:0/1917535201 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8034040 con 0x55839b67ef40 2024-04-23T18:22:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.091+0000 7fa0d2601640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0b80a3dc0 0x7fa0b80a6280 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc09c150 tx=0x7fa0bc06ab70 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.091+0000 7fa0b7fff640 1 -- 172.21.15.50:0/1917535201 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c8003e20 con 0x55839b67ef40 2024-04-23T18:22:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.091+0000 7fa0b7fff640 1 -- 172.21.15.50:0/1917535201 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8004410 con 0x55839b67ef40 2024-04-23T18:22:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.092+0000 7fa0d6d79740 1 -- 172.21.15.50:0/1917535201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 msgr2=0x55839b6892c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.092+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/1917535201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 0x55839b6892c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa0c802fdc0 tx=0x7fa0c80093f0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.092+0000 7fa0d6d79740 1 -- 172.21.15.50:0/1917535201 shutdown_connections 2024-04-23T18:22:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.092+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/1917535201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b689800 0x55839b68bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.092+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/1917535201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 0x55839b6892c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.092+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/1917535201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67e620 0x55839b67ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.092+0000 7fa0d6d79740 1 -- 172.21.15.50:0/1917535201 >> 172.21.15.50:0/1917535201 conn(0x55839b67ab30 msgr2=0x55839b67af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.092+0000 7fa0d6d79740 1 -- 172.21.15.50:0/1917535201 shutdown_connections 2024-04-23T18:22:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.092+0000 7fa0d6d79740 1 -- 172.21.15.50:0/1917535201 wait complete. 2024-04-23T18:22:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.092+0000 7fa0d6d79740 1 -- start start 2024-04-23T18:22:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.092+0000 7fa0d6d79740 1 Processor -- start 2024-04-23T18:22:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d6d79740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67e620 0x55839b689170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d6d79740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67ef40 0x55839b686460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d6d79740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 0x55839b6869c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d6d79740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55839b68c3d0 con 0x55839b689800 2024-04-23T18:22:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d6d79740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55839b57e0b0 con 0x55839b67ef40 2024-04-23T18:22:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d6d79740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55839b68c550 con 0x55839b67e620 2024-04-23T18:22:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d6d79740 10 client.0 _ll_register_callbacks cb 0x55839b56bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d15ff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 0x55839b6869c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d15ff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 0x55839b6869c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37280/0 (socket says 172.21.15.50:37280) 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d15ff640 1 -- 172.21.15.50:0/2228750808 learned_addr learned my addr 172.21.15.50:0/2228750808 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d1e00640 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67ef40 0x55839b686460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67e620 0x55839b689170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d15ff640 1 -- 172.21.15.50:0/2228750808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67e620 msgr2=0x55839b689170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67e620 0x55839b689170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d15ff640 1 -- 172.21.15.50:0/2228750808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67ef40 msgr2=0x55839b686460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67ef40 0x55839b686460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d15ff640 1 -- 172.21.15.50:0/2228750808 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55839b3c2620 con 0x55839b689800 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 0x55839b6869c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4011de0 tx=0x7fa0c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55c52aa2b00f29a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.093+0000 7fa0b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0b57fa640 1 -- 172.21.15.50:0/2228750808 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c401b040 con 0x55839b689800 2024-04-23T18:22:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0b57fa640 1 -- 172.21.15.50:0/2228750808 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa0c400edc0 con 0x55839b689800 2024-04-23T18:22:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2228750808 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0c8003e20 con 0x55839b689800 2024-04-23T18:22:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0b57fa640 1 -- 172.21.15.50:0/2228750808 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa0c4020420 con 0x55839b689800 2024-04-23T18:22:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0d6d79740 10 client.16014 fetch_fsmap learned FSMap version 26 2024-04-23T18:22:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2228750808 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55839b6983a0 con 0x55839b689800 2024-04-23T18:22:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0b57fa640 1 -- 172.21.15.50:0/2228750808 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c8003e20 con 0x55839b689800 2024-04-23T18:22:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0b57fa640 1 -- 172.21.15.50:0/2228750808 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fa0c4029570 con 0x55839b689800 2024-04-23T18:22:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0d6d79740 10 client.16014 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:22:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0d6d79740 10 client.16014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0d6d79740 20 client.16014 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55839b699860 0x55839b69bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0d6d79740 4 client.16014 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.094+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2228750808 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55839b5f1390 con 0x55839b699860 2024-04-23T18:22:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.095+0000 7fa0d1e00640 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55839b699860 0x55839b69bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.095+0000 7fa0d1e00640 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55839b699860 0x55839b69bc50 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0e6c1ff17ff152 server_cookie=3383f51cea936986 in_seq=0 out_seq=0 2024-04-23T18:22:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.095+0000 7fa0b57fa640 10 client.16014 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0b57fa640 1 -- 172.21.15.50:0/2228750808 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55839b5f1390 con 0x55839b699860 2024-04-23T18:22:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0b57fa640 10 client.16014 handle_command_reply: tid=0 2024-04-23T18:22:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0d6d79740 1 client.16014 shutdown 2024-04-23T18:22:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2228750808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 msgr2=0x55839b6869c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 0x55839b6869c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4011de0 tx=0x7fa0c4004730 comp rx=0 tx=0).stop 2024-04-23T18:22:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2228750808 shutdown_connections 2024-04-23T18:22:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55839b699860 0x55839b69bc50 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 0x55839b6869c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67ef40 0x55839b686460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0b57fa640 0 client.16014 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/2228750808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67e620 0x55839b689170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2228750808 >> 172.21.15.50:0/2228750808 conn(0x55839b67ab30 msgr2=0x55839b6808b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2228750808 shutdown_connections 2024-04-23T18:22:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2228750808 wait complete. 2024-04-23T18:22:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.096+0000 7fa0d6d79740 20 client.16014 trim_cache size 0 max 16384 2024-04-23T18:22:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.111+0000 7fa0d6d79740 1 Processor -- start 2024-04-23T18:22:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.111+0000 7fa0d6d79740 1 -- start start 2024-04-23T18:22:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.111+0000 7fa0d6d79740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67e620 0x55839b695290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.111+0000 7fa0d6d79740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67ef40 0x55839b679b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.111+0000 7fa0d6d79740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 0x55839b688ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.111+0000 7fa0d6d79740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc1b26d0 con 0x55839b689800 2024-04-23T18:22:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.111+0000 7fa0d6d79740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc1aff40 con 0x55839b67e620 2024-04-23T18:22:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.111+0000 7fa0d6d79740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc1add90 con 0x55839b67ef40 2024-04-23T18:22:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0d15ff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 0x55839b688ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0d15ff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 0x55839b688ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37296/0 (socket says 172.21.15.50:37296) 2024-04-23T18:22:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0d15ff640 1 -- 172.21.15.50:0/2424357262 learned_addr learned my addr 172.21.15.50:0/2424357262 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/2424357262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67e620 0x55839b695290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0d1e00640 1 --2- 172.21.15.50:0/2424357262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67ef40 0x55839b679b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0d15ff640 1 -- 172.21.15.50:0/2424357262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67ef40 msgr2=0x55839b679b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/2424357262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67ef40 0x55839b679b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0d15ff640 1 -- 172.21.15.50:0/2424357262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67e620 msgr2=0x55839b695290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/2424357262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67e620 0x55839b695290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0d15ff640 1 -- 172.21.15.50:0/2424357262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55839b686d30 con 0x55839b689800 2024-04-23T18:22:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/2424357262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 0x55839b688ca0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4005eb0 tx=0x7fa0c4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1423759ffb26cf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0b77fe640 1 -- 172.21.15.50:0/2424357262 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c400ee10 con 0x55839b689800 2024-04-23T18:22:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0b77fe640 1 -- 172.21.15.50:0/2424357262 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c4030050 con 0x55839b689800 2024-04-23T18:22:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.112+0000 7fa0b77fe640 1 -- 172.21.15.50:0/2424357262 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4020c80 con 0x55839b689800 2024-04-23T18:22:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.113+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2424357262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 msgr2=0x55839b688ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.113+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/2424357262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 0x55839b688ca0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4005eb0 tx=0x7fa0c4009f90 comp rx=0 tx=0).stop 2024-04-23T18:22:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.113+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2424357262 shutdown_connections 2024-04-23T18:22:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.113+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/2424357262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b689800 0x55839b688ca0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.113+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/2424357262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67ef40 0x55839b679b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.113+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/2424357262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b67e620 0x55839b695290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.113+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2424357262 >> 172.21.15.50:0/2424357262 conn(0x55839b67ab30 msgr2=0x55839b679420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.113+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2424357262 shutdown_connections 2024-04-23T18:22:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.113+0000 7fa0d6d79740 1 -- 172.21.15.50:0/2424357262 wait complete. 2024-04-23T18:22:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.113+0000 7fa0d6d79740 1 -- start start 2024-04-23T18:22:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.113+0000 7fa0d6d79740 1 Processor -- start 2024-04-23T18:22:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d6d79740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67e620 0x55839b691b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d6d79740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 0x55839b68cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d6d79740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b689800 0x55839b68d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d6d79740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc1ae400 con 0x55839b67ef40 2024-04-23T18:22:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d6d79740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc1af870 con 0x55839b689800 2024-04-23T18:22:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d6d79740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc1aead0 con 0x55839b67e620 2024-04-23T18:22:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d6d79740 10 client.0 _ll_register_callbacks cb 0x7fa0c40119f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d1e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 0x55839b68cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d1e00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 0x55839b68cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37300/0 (socket says 172.21.15.50:37300) 2024-04-23T18:22:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d1e00640 1 -- 172.21.15.50:0/3201692133 learned_addr learned my addr 172.21.15.50:0/3201692133 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67e620 0x55839b691b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b689800 0x55839b68d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d1e00640 1 -- 172.21.15.50:0/3201692133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67e620 msgr2=0x55839b691b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d1e00640 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67e620 0x55839b691b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d1e00640 1 -- 172.21.15.50:0/3201692133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b689800 msgr2=0x55839b68d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d1e00640 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b689800 0x55839b68d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d1e00640 1 -- 172.21.15.50:0/3201692133 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55839b68dbf0 con 0x55839b67ef40 2024-04-23T18:22:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b689800 0x55839b68d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0d1e00640 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 0x55839b68cea0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa0c802fdf0 tx=0x7fa0c802fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab06293416e041a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.114+0000 7fa0b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0b57fa640 1 -- 172.21.15.50:0/3201692133 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8034040 con 0x55839b67ef40 2024-04-23T18:22:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0b57fa640 1 -- 172.21.15.50:0/3201692133 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa0c802f5b0 con 0x55839b67ef40 2024-04-23T18:22:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0d6d79740 1 -- 172.21.15.50:0/3201692133 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0c4030050 con 0x55839b67ef40 2024-04-23T18:22:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0b57fa640 1 -- 172.21.15.50:0/3201692133 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa0c805d050 con 0x55839b67ef40 2024-04-23T18:22:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0d6d79740 10 client.16020 fetch_fsmap learned FSMap version 26 2024-04-23T18:22:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0d6d79740 1 -- 172.21.15.50:0/3201692133 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55839b6ffdb0 con 0x55839b67ef40 2024-04-23T18:22:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0b57fa640 1 -- 172.21.15.50:0/3201692133 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c4030050 con 0x55839b67ef40 2024-04-23T18:22:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0b57fa640 1 -- 172.21.15.50:0/3201692133 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fa0c8058030 con 0x55839b67ef40 2024-04-23T18:22:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0d6d79740 10 client.16020 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:22:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0d6d79740 10 client.16020 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0d6d79740 20 client.16020 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55839b700250 0x55839b702640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0d6d79740 4 client.16020 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0d6d79740 1 -- 172.21.15.50:0/3201692133 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55839b5eb200 con 0x55839b700250 2024-04-23T18:22:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.115+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55839b700250 0x55839b702640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.116+0000 7fa0d15ff640 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55839b700250 0x55839b702640 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22f59dba4ab66cec server_cookie=f2e9d2765e4d450f in_seq=0 out_seq=0 2024-04-23T18:22:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.116+0000 7fa0b57fa640 10 client.16020 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0b57fa640 1 -- 172.21.15.50:0/3201692133 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55839b5eb200 con 0x55839b700250 2024-04-23T18:22:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0b57fa640 10 client.16020 handle_command_reply: tid=0 2024-04-23T18:22:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0d6d79740 1 client.16020 shutdown 2024-04-23T18:22:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0d6d79740 1 -- 172.21.15.50:0/3201692133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 msgr2=0x55839b68cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 0x55839b68cea0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa0c802fdf0 tx=0x7fa0c802fa60 comp rx=0 tx=0).stop 2024-04-23T18:22:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0d6d79740 1 -- 172.21.15.50:0/3201692133 shutdown_connections 2024-04-23T18:22:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55839b700250 0x55839b702640 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55839b689800 0x55839b68d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55839b67ef40 0x55839b68cea0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0b57fa640 0 client.16020 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0d6d79740 1 --2- 172.21.15.50:0/3201692133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55839b67e620 0x55839b691b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0d6d79740 1 -- 172.21.15.50:0/3201692133 >> 172.21.15.50:0/3201692133 conn(0x55839b67ab30 msgr2=0x55839b679cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0d6d79740 1 -- 172.21.15.50:0/3201692133 shutdown_connections 2024-04-23T18:22:30.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0d6d79740 1 -- 172.21.15.50:0/3201692133 wait complete. 2024-04-23T18:22:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.136+0000 7fa0d6d79740 20 client.16020 trim_cache size 0 max 16384 2024-04-23T18:22:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:22:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:22:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:22:30.132 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:22:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.137+0000 7fa0d4064640 1 -- 172.21.15.50:0/4016678400 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0b80a3dc0 msgr2=0x7fa0b80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.137+0000 7fa0d4064640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0b80a3dc0 0x7fa0b80a6280 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc09c150 tx=0x7fa0bc06ab70 comp rx=0 tx=0).stop 2024-04-23T18:22:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.137+0000 7fa0d4064640 1 -- 172.21.15.50:0/4016678400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0cc10e840 msgr2=0x7fa0cc19a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.137+0000 7fa0d4064640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0cc10e840 0x7fa0cc19a6c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00d680 tx=0x7fa0bc00db40 comp rx=0 tx=0).stop 2024-04-23T18:22:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.137+0000 7fa0d4064640 1 -- 172.21.15.50:0/4016678400 shutdown_connections 2024-04-23T18:22:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.137+0000 7fa0d4064640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0b80a3dc0 0x7fa0b80a6280 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.137+0000 7fa0d4064640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0cc10e840 0x7fa0cc19a6c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.137+0000 7fa0d4064640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0cc074b70 0x7fa0cc1a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.137+0000 7fa0d4064640 1 --2- 172.21.15.50:0/4016678400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0cc074250 0x7fa0cc1a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.137+0000 7fa0d4064640 1 -- 172.21.15.50:0/4016678400 >> 172.21.15.50:0/4016678400 conn(0x7fa0cc0793e0 msgr2=0x7fa0cc0fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.137+0000 7fa0d4064640 1 -- 172.21.15.50:0/4016678400 shutdown_connections 2024-04-23T18:22:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:30.137+0000 7fa0d4064640 1 -- 172.21.15.50:0/4016678400 wait complete. 2024-04-23T18:22:30.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:29 smithi050 ceph-mon[29187]: pgmap v123: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 705 KiB/s wr, 58 op/s 2024-04-23T18:22:30.201 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 4.01 MB/sec execute 64 sec latency 11888.542 ms 2024-04-23T18:22:31.201 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.95 MB/sec execute 65 sec latency 12888.630 ms 2024-04-23T18:22:32.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:31 smithi177 ceph-mon[32308]: pgmap v124: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 696 KiB/s wr, 57 op/s 2024-04-23T18:22:32.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:31 smithi188 ceph-mon[32694]: pgmap v124: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 696 KiB/s wr, 57 op/s 2024-04-23T18:22:32.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:31 smithi050 ceph-mon[29187]: pgmap v124: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 696 KiB/s wr, 57 op/s 2024-04-23T18:22:32.201 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.89 MB/sec execute 66 sec latency 13888.688 ms 2024-04-23T18:22:33.201 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.83 MB/sec execute 67 sec latency 14888.723 ms 2024-04-23T18:22:34.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:33 smithi177 ceph-mon[32308]: pgmap v125: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 793 KiB/s wr, 68 op/s 2024-04-23T18:22:34.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:33 smithi188 ceph-mon[32694]: pgmap v125: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 793 KiB/s wr, 68 op/s 2024-04-23T18:22:34.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:33 smithi050 ceph-mon[29187]: pgmap v125: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 793 KiB/s wr, 68 op/s 2024-04-23T18:22:34.201 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.77 MB/sec execute 68 sec latency 15888.827 ms 2024-04-23T18:22:35.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.72 MB/sec execute 69 sec latency 16888.935 ms 2024-04-23T18:22:36.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:35 smithi177 ceph-mon[32308]: pgmap v126: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 122 KiB/s wr, 19 op/s 2024-04-23T18:22:36.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:35 smithi188 ceph-mon[32694]: pgmap v126: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 122 KiB/s wr, 19 op/s 2024-04-23T18:22:36.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:35 smithi050 ceph-mon[29187]: pgmap v126: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 122 KiB/s wr, 19 op/s 2024-04-23T18:22:36.201 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.67 MB/sec execute 70 sec latency 17889.032 ms 2024-04-23T18:22:37.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.61 MB/sec execute 71 sec latency 18889.107 ms 2024-04-23T18:22:38.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:37 smithi177 ceph-mon[32308]: pgmap v127: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 124 KiB/s wr, 21 op/s 2024-04-23T18:22:38.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:37 smithi188 ceph-mon[32694]: pgmap v127: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 124 KiB/s wr, 21 op/s 2024-04-23T18:22:38.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:37 smithi050 ceph-mon[29187]: pgmap v127: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 124 KiB/s wr, 21 op/s 2024-04-23T18:22:38.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.56 MB/sec execute 72 sec latency 19889.222 ms 2024-04-23T18:22:39.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.52 MB/sec execute 73 sec latency 20889.346 ms 2024-04-23T18:22:40.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:39 smithi177 ceph-mon[32308]: pgmap v128: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 98 KiB/s wr, 13 op/s 2024-04-23T18:22:40.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:39 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:22:40.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:39 smithi188 ceph-mon[32694]: pgmap v128: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 98 KiB/s wr, 13 op/s 2024-04-23T18:22:40.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:39 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:22:40.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:39 smithi050 ceph-mon[29187]: pgmap v128: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 98 KiB/s wr, 13 op/s 2024-04-23T18:22:40.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:39 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:22:40.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.47 MB/sec execute 74 sec latency 21889.433 ms 2024-04-23T18:22:41.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:40 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:22:41.037 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:40 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:22:41.038 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:40 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:22:41.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:40 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:22:41.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:40 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:22:41.079 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:40 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:22:41.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:40 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:22:41.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:40 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:22:41.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:40 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:22:41.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.42 MB/sec execute 75 sec latency 22889.463 ms 2024-04-23T18:22:42.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.38 MB/sec execute 76 sec latency 23889.498 ms 2024-04-23T18:22:42.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:41 smithi177 ceph-mon[32308]: pgmap v129: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 98 KiB/s wr, 13 op/s 2024-04-23T18:22:42.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:41 smithi188 ceph-mon[32694]: pgmap v129: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 98 KiB/s wr, 13 op/s 2024-04-23T18:22:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:41 smithi050 ceph-mon[29187]: pgmap v129: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 98 KiB/s wr, 13 op/s 2024-04-23T18:22:43.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.33 MB/sec execute 77 sec latency 24889.533 ms 2024-04-23T18:22:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.238+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1819175861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b94101aa0 msgr2=0x7f3b94105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.238+0000 7f3b9b58a640 1 --2- 172.21.15.50:0/1819175861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b94101aa0 0x7f3b94105a90 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3b84009900 tx=0x7f3b8402a410 comp rx=0 tx=0).stop 2024-04-23T18:22:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.238+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1819175861 shutdown_connections 2024-04-23T18:22:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.238+0000 7f3b9b58a640 1 --2- 172.21.15.50:0/1819175861 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b94106170 0x7f3b9410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.238+0000 7f3b9b58a640 1 --2- 172.21.15.50:0/1819175861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b94101aa0 0x7f3b94105a90 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.238+0000 7f3b9b58a640 1 --2- 172.21.15.50:0/1819175861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b941010f0 0x7f3b941014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.238+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1819175861 >> 172.21.15.50:0/1819175861 conn(0x7f3b94077fd0 msgr2=0x7f3b940783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.238+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1819175861 shutdown_connections 2024-04-23T18:22:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.238+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1819175861 wait complete. 2024-04-23T18:22:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b9b58a640 1 Processor -- start 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b9b58a640 1 -- start start 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b9b58a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b941010f0 0x7f3b94198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b9b58a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b94106170 0x7f3b94198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b9b58a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b9419cf30 0x7f3b9419d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b9b58a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3b941121f0 con 0x7f3b9419cf30 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b9b58a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b94112070 con 0x7f3b941010f0 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b9b58a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3b94112370 con 0x7f3b94106170 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b99b27640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b9419cf30 0x7f3b9419d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b99b27640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b9419cf30 0x7f3b9419d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40774/0 (socket says 172.21.15.50:40774) 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b99b27640 1 -- 172.21.15.50:0/1295859438 learned_addr learned my addr 172.21.15.50:0/1295859438 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b98b25640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b94106170 0x7f3b94198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b99326640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b941010f0 0x7f3b94198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b99b27640 1 -- 172.21.15.50:0/1295859438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b94106170 msgr2=0x7f3b94198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b99b27640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b94106170 0x7f3b94198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b99b27640 1 -- 172.21.15.50:0/1295859438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b941010f0 msgr2=0x7f3b94198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b99b27640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b941010f0 0x7f3b94198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.239+0000 7f3b99b27640 1 -- 172.21.15.50:0/1295859438 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b9419da20 con 0x7f3b9419cf30 2024-04-23T18:22:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.240+0000 7f3b99b27640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b9419cf30 0x7f3b9419d3e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00eb10 tx=0x7f3b8c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.240+0000 7f3b8a7fc640 1 -- 172.21.15.50:0/1295859438 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c004b20 con 0x7f3b9419cf30 2024-04-23T18:22:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.240+0000 7f3b8a7fc640 1 -- 172.21.15.50:0/1295859438 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b8c004cc0 con 0x7f3b9419cf30 2024-04-23T18:22:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.240+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1295859438 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b9419dd10 con 0x7f3b9419cf30 2024-04-23T18:22:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.240+0000 7f3b8a7fc640 1 -- 172.21.15.50:0/1295859438 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c010670 con 0x7f3b9419cf30 2024-04-23T18:22:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.240+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1295859438 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b941a5520 con 0x7f3b9419cf30 2024-04-23T18:22:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.242+0000 7f3b8a7fc640 1 -- 172.21.15.50:0/1295859438 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3b8c005020 con 0x7f3b9419cf30 2024-04-23T18:22:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b9e377740 1 Processor -- start 2024-04-23T18:22:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b9e377740 1 -- start start 2024-04-23T18:22:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b9e377740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbc520 0x560db4dbc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b9e377740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 0x560db4dcb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b9e377740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dce000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b9e377740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560db4d21970 con 0x560db4dbce40 2024-04-23T18:22:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b9e377740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560db4cbe4a0 con 0x560db4dcbc10 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b9e377740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560db4d1d930 con 0x560db4dbc520 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b8a7fc640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3b800a3e40 0x7f3b800a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b99326640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 0x560db4dcb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b99326640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 0x560db4dcb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40778/0 (socket says 172.21.15.50:40778) 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b99326640 1 -- 172.21.15.50:0/1957097155 learned_addr learned my addr 172.21.15.50:0/1957097155 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.243+0000 7f3b98b25640 1 --2- 172.21.15.50:0/1957097155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbc520 0x560db4dbc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b99b27640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3b800a3e40 0x7f3b800a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b98b25640 1 --2- 172.21.15.50:0/1957097155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b8a7fc640 1 -- 172.21.15.50:0/1295859438 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3b8c0d0c80 con 0x7f3b9419cf30 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b99326640 1 -- 172.21.15.50:0/1957097155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbc520 msgr2=0x560db4dbc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b99326640 1 --2- 172.21.15.50:0/1957097155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbc520 0x560db4dbc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b99326640 1 -- 172.21.15.50:0/1957097155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 msgr2=0x560db4dce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b99326640 1 --2- 172.21.15.50:0/1957097155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dce000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b99326640 1 -- 172.21.15.50:0/1957097155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560db4a34210 con 0x560db4dbce40 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b98b25640 1 --2- 172.21.15.50:0/1957097155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbc520 0x560db4dbc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b99326640 1 --2- 172.21.15.50:0/1957097155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 0x560db4dcb6d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3b9000bd10 tx=0x7f3b900107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ae84440023574 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b99b27640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3b800a3e40 0x7f3b800a6300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c09cb60 tx=0x7f3b8c00efd0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b7f7fe640 1 -- 172.21.15.50:0/1957097155 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90004030 con 0x560db4dbce40 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b7f7fe640 1 -- 172.21.15.50:0/1957097155 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b900035f0 con 0x560db4dbce40 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.244+0000 7f3b7f7fe640 1 -- 172.21.15.50:0/1957097155 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90011640 con 0x560db4dbce40 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.245+0000 7f3b9e377740 1 -- 172.21.15.50:0/1957097155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 msgr2=0x560db4dcb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.245+0000 7f3b9e377740 1 --2- 172.21.15.50:0/1957097155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 0x560db4dcb6d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3b9000bd10 tx=0x7f3b900107a0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.245+0000 7f3b9e377740 1 -- 172.21.15.50:0/1957097155 shutdown_connections 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.245+0000 7f3b9e377740 1 --2- 172.21.15.50:0/1957097155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.245+0000 7f3b9e377740 1 --2- 172.21.15.50:0/1957097155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 0x560db4dcb6d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.245+0000 7f3b9e377740 1 --2- 172.21.15.50:0/1957097155 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbc520 0x560db4dbc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.245+0000 7f3b9e377740 1 -- 172.21.15.50:0/1957097155 >> 172.21.15.50:0/1957097155 conn(0x560db4db8b30 msgr2=0x560db4db8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.245+0000 7f3b9e377740 1 -- 172.21.15.50:0/1957097155 shutdown_connections 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.245+0000 7f3b9e377740 1 -- 172.21.15.50:0/1957097155 wait complete. 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.245+0000 7f3b9e377740 1 -- start start 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b9e377740 1 Processor -- start 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b9e377740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbc520 0x560db4dcb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b9e377740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 0x560db4dc87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b9e377740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dc8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b9e377740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560db4dce840 con 0x560db4dbce40 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b9e377740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560db4cbc0b0 con 0x560db4dcbc10 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b9e377740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560db4dce9c0 con 0x560db4dbc520 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b9e377740 10 client.0 _ll_register_callbacks cb 0x560db4ca9d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b99326640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 0x560db4dc87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b99326640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 0x560db4dc87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40780/0 (socket says 172.21.15.50:40780) 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b99326640 1 -- 172.21.15.50:0/126914880 learned_addr learned my addr 172.21.15.50:0/126914880 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b98b25640 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbc520 0x560db4dcb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b98b25640 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dc8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b99326640 1 -- 172.21.15.50:0/126914880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbc520 msgr2=0x560db4dcb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b99326640 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbc520 0x560db4dcb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b99326640 1 -- 172.21.15.50:0/126914880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 msgr2=0x560db4dc8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b99326640 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dc8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.246+0000 7f3b99326640 1 -- 172.21.15.50:0/126914880 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560db4b00620 con 0x560db4dbce40 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.247+0000 7f3b98b25640 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dc8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.247+0000 7f3b99326640 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 0x560db4dc87f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3b90010ca0 tx=0x7f3b9000ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b001ec53dc800e03 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.247+0000 7f3b7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.247+0000 7f3b7cff9640 1 -- 172.21.15.50:0/126914880 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90029060 con 0x560db4dbce40 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.247+0000 7f3b7cff9640 1 -- 172.21.15.50:0/126914880 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3b90004370 con 0x560db4dbce40 2024-04-23T18:22:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.247+0000 7f3b9e377740 1 -- 172.21.15.50:0/126914880 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b900035f0 con 0x560db4dbce40 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.247+0000 7f3b7cff9640 1 -- 172.21.15.50:0/126914880 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3b900037a0 con 0x560db4dbce40 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.247+0000 7f3b9e377740 10 client.16032 fetch_fsmap learned FSMap version 26 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.247+0000 7f3b9e377740 1 -- 172.21.15.50:0/126914880 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560db4dda790 con 0x560db4dbce40 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.247+0000 7f3b7cff9640 1 -- 172.21.15.50:0/126914880 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b900035f0 con 0x560db4dbce40 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.248+0000 7f3b7cff9640 1 -- 172.21.15.50:0/126914880 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f3b900046c0 con 0x560db4dbce40 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.248+0000 7f3b9e377740 10 client.16032 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.248+0000 7f3b9e377740 10 client.16032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.248+0000 7f3b9e377740 20 client.16032 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.248+0000 7f3b9e377740 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560db4ddbb90 0x560db4dddf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.248+0000 7f3b9e377740 4 client.16032 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.248+0000 7f3b9e377740 1 -- 172.21.15.50:0/126914880 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560db4d2f390 con 0x560db4ddbb90 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.248+0000 7f3b98b25640 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560db4ddbb90 0x560db4dddf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.248+0000 7f3b98b25640 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560db4ddbb90 0x560db4dddf80 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fabe229d46b06d5d server_cookie=fb5a1886c29bfffc in_seq=0 out_seq=0 2024-04-23T18:22:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.248+0000 7f3b7cff9640 10 client.16032 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.249+0000 7f3b7cff9640 1 -- 172.21.15.50:0/126914880 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560db4d2f390 con 0x560db4ddbb90 2024-04-23T18:22:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.249+0000 7f3b7cff9640 10 client.16032 handle_command_reply: tid=0 2024-04-23T18:22:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.249+0000 7f3b9e377740 1 client.16032 shutdown 2024-04-23T18:22:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.249+0000 7f3b9e377740 1 -- 172.21.15.50:0/126914880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 msgr2=0x560db4dc87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.249+0000 7f3b9e377740 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 0x560db4dc87f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3b90010ca0 tx=0x7f3b9000ba00 comp rx=0 tx=0).stop 2024-04-23T18:22:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.249+0000 7f3b9e377740 1 -- 172.21.15.50:0/126914880 shutdown_connections 2024-04-23T18:22:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.249+0000 7f3b9e377740 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560db4ddbb90 0x560db4dddf80 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.249+0000 7f3b9e377740 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dc8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.249+0000 7f3b9e377740 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbce40 0x560db4dc87f0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.249+0000 7f3b7cff9640 0 client.16032 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.249+0000 7f3b9e377740 1 --2- 172.21.15.50:0/126914880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbc520 0x560db4dcb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.249+0000 7f3b9e377740 1 -- 172.21.15.50:0/126914880 >> 172.21.15.50:0/126914880 conn(0x560db4db8b30 msgr2=0x560db4dbe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.250+0000 7f3b9e377740 1 -- 172.21.15.50:0/126914880 shutdown_connections 2024-04-23T18:22:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.250+0000 7f3b9e377740 1 -- 172.21.15.50:0/126914880 wait complete. 2024-04-23T18:22:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.250+0000 7f3b9e377740 20 client.16032 trim_cache size 0 max 16384 2024-04-23T18:22:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.265+0000 7f3b9e377740 1 Processor -- start 2024-04-23T18:22:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.265+0000 7f3b9e377740 1 -- start start 2024-04-23T18:22:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.265+0000 7f3b9e377740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dbc520 0x560db4dd7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.265+0000 7f3b9e377740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbce40 0x560db4dc3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.265+0000 7f3b9e377740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dcbc10 0x560db4dcb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.265+0000 7f3b9e377740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3b941ae2e0 con 0x560db4dcbc10 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.265+0000 7f3b9e377740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b941abb50 con 0x560db4dbc520 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.265+0000 7f3b9e377740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3b941a99a0 con 0x560db4dbce40 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.265+0000 7f3b98b25640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dcbc10 0x560db4dcb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.265+0000 7f3b98b25640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dcbc10 0x560db4dcb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40792/0 (socket says 172.21.15.50:40792) 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.265+0000 7f3b98b25640 1 -- 172.21.15.50:0/2609771316 learned_addr learned my addr 172.21.15.50:0/2609771316 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.266+0000 7f3b99326640 1 --2- 172.21.15.50:0/2609771316 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbce40 0x560db4dc3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.266+0000 7f3b98b25640 1 --2- 172.21.15.50:0/2609771316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dbc520 0x560db4dd7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.266+0000 7f3b98b25640 1 -- 172.21.15.50:0/2609771316 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbce40 msgr2=0x560db4dc3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.266+0000 7f3b98b25640 1 --2- 172.21.15.50:0/2609771316 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbce40 0x560db4dc3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.266+0000 7f3b98b25640 1 -- 172.21.15.50:0/2609771316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dbc520 msgr2=0x560db4dd7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.266+0000 7f3b98b25640 1 --2- 172.21.15.50:0/2609771316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dbc520 0x560db4dd7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.266+0000 7f3b98b25640 1 -- 172.21.15.50:0/2609771316 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560db4dbd380 con 0x560db4dcbc10 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.266+0000 7f3b98b25640 1 --2- 172.21.15.50:0/2609771316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dcbc10 0x560db4dcb030 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3b84032fd0 tx=0x7f3b84005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c2e26202315738 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.266+0000 7f3b7effd640 1 -- 172.21.15.50:0/2609771316 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b84035040 con 0x560db4dcbc10 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.266+0000 7f3b7effd640 1 -- 172.21.15.50:0/2609771316 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b84052070 con 0x560db4dcbc10 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.266+0000 7f3b7effd640 1 -- 172.21.15.50:0/2609771316 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b84031c10 con 0x560db4dcbc10 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.267+0000 7f3b9e377740 1 -- 172.21.15.50:0/2609771316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dcbc10 msgr2=0x560db4dcb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.267+0000 7f3b9e377740 1 --2- 172.21.15.50:0/2609771316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dcbc10 0x560db4dcb030 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3b84032fd0 tx=0x7f3b84005780 comp rx=0 tx=0).stop 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.267+0000 7f3b9e377740 1 -- 172.21.15.50:0/2609771316 shutdown_connections 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.267+0000 7f3b9e377740 1 --2- 172.21.15.50:0/2609771316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dcbc10 0x560db4dcb030 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.267+0000 7f3b9e377740 1 --2- 172.21.15.50:0/2609771316 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbce40 0x560db4dc3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.267+0000 7f3b9e377740 1 --2- 172.21.15.50:0/2609771316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dbc520 0x560db4dd7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.267+0000 7f3b9e377740 1 -- 172.21.15.50:0/2609771316 >> 172.21.15.50:0/2609771316 conn(0x560db4db8b30 msgr2=0x560db4dc3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.267+0000 7f3b9e377740 1 -- 172.21.15.50:0/2609771316 shutdown_connections 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.267+0000 7f3b9e377740 1 -- 172.21.15.50:0/2609771316 wait complete. 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.267+0000 7f3b9e377740 1 -- start start 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.267+0000 7f3b9e377740 1 Processor -- start 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b9e377740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbc520 0x560db4dd4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b9e377740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbce40 0x560db4dcf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b9e377740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dcfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b9e377740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3b941aa010 con 0x560db4dbc520 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b9e377740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b941ab480 con 0x560db4dcbc10 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b9e377740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3b941aa6e0 con 0x560db4dbce40 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b9e377740 10 client.0 _ll_register_callbacks cb 0x7f3b9000bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b98b25640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbc520 0x560db4dd4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b98b25640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbc520 0x560db4dd4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40808/0 (socket says 172.21.15.50:40808) 2024-04-23T18:22:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b98b25640 1 -- 172.21.15.50:0/3848931981 learned_addr learned my addr 172.21.15.50:0/3848931981 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b99326640 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbce40 0x560db4dcf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b98b25640 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dcfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b98b25640 1 -- 172.21.15.50:0/3848931981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbce40 msgr2=0x560db4dcf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b98b25640 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbce40 0x560db4dcf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b98b25640 1 -- 172.21.15.50:0/3848931981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 msgr2=0x560db4dcfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b98b25640 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dcfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b98b25640 1 -- 172.21.15.50:0/3848931981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560db4dce000 con 0x560db4dbc520 2024-04-23T18:22:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b98b25640 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbc520 0x560db4dd4100 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3b8402a930 tx=0x7f3b84032bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60711c5a4235eb2f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.268+0000 7f3b7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b7cff9640 1 -- 172.21.15.50:0/3848931981 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b84035030 con 0x560db4dbc520 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b7cff9640 1 -- 172.21.15.50:0/3848931981 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3b8404c5f0 con 0x560db4dbc520 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b9e377740 1 -- 172.21.15.50:0/3848931981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b84052070 con 0x560db4dbc520 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b7cff9640 1 -- 172.21.15.50:0/3848931981 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3b84004490 con 0x560db4dbc520 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b9e377740 10 client.16038 fetch_fsmap learned FSMap version 26 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b9e377740 1 -- 172.21.15.50:0/3848931981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560db4db8960 con 0x560db4dbc520 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b7cff9640 1 -- 172.21.15.50:0/3848931981 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b84052070 con 0x560db4dbc520 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b7cff9640 1 -- 172.21.15.50:0/3848931981 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f3b84003820 con 0x560db4dbc520 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b9e377740 10 client.16038 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b9e377740 10 client.16038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b9e377740 20 client.16038 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b9e377740 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560db4e42200 0x560db4e44620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b9e377740 4 client.16038 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b9e377740 1 -- 172.21.15.50:0/3848931981 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560db4dc90c0 con 0x560db4e42200 2024-04-23T18:22:43.268 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.269+0000 7f3b99326640 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560db4e42200 0x560db4e44620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:43.268 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.270+0000 7f3b7cff9640 10 client.16038 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:43.268 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.270+0000 7f3b99326640 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560db4e42200 0x560db4e44620 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d08862db92064c57 server_cookie=561ae4b9a6d2690a in_seq=0 out_seq=0 2024-04-23T18:22:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b7cff9640 1 -- 172.21.15.50:0/3848931981 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560db4dc90c0 con 0x560db4e42200 2024-04-23T18:22:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b7cff9640 10 client.16038 handle_command_reply: tid=0 2024-04-23T18:22:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b9e377740 1 client.16038 shutdown 2024-04-23T18:22:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b9e377740 1 -- 172.21.15.50:0/3848931981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbc520 msgr2=0x560db4dd4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b9e377740 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbc520 0x560db4dd4100 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3b8402a930 tx=0x7f3b84032bb0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b9e377740 1 -- 172.21.15.50:0/3848931981 shutdown_connections 2024-04-23T18:22:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b9e377740 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560db4e42200 0x560db4e44620 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b9e377740 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560db4dcbc10 0x560db4dcfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b9e377740 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560db4dbce40 0x560db4dcf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b7cff9640 0 client.16038 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b9e377740 1 --2- 172.21.15.50:0/3848931981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560db4dbc520 0x560db4dd4100 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b9e377740 1 -- 172.21.15.50:0/3848931981 >> 172.21.15.50:0/3848931981 conn(0x560db4db8b30 msgr2=0x560db4dbe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b9e377740 1 -- 172.21.15.50:0/3848931981 shutdown_connections 2024-04-23T18:22:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b9e377740 1 -- 172.21.15.50:0/3848931981 wait complete. 2024-04-23T18:22:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.286+0000 7f3b9e377740 20 client.16038 trim_cache size 0 max 16384 2024-04-23T18:22:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:22:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:22:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:22:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:22:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.287+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1295859438 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3b800a3e40 msgr2=0x7f3b800a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.287+0000 7f3b9b58a640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3b800a3e40 0x7f3b800a6300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c09cb60 tx=0x7f3b8c00efd0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.287+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1295859438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b9419cf30 msgr2=0x7f3b9419d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.287+0000 7f3b9b58a640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b9419cf30 0x7f3b9419d3e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00eb10 tx=0x7f3b8c00c2d0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.287+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1295859438 shutdown_connections 2024-04-23T18:22:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.287+0000 7f3b9b58a640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3b800a3e40 0x7f3b800a6300 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.287+0000 7f3b9b58a640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b9419cf30 0x7f3b9419d3e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.287+0000 7f3b9b58a640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b94106170 0x7f3b94198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.287+0000 7f3b9b58a640 1 --2- 172.21.15.50:0/1295859438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b941010f0 0x7f3b94198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.287+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1295859438 >> 172.21.15.50:0/1295859438 conn(0x7f3b94077fd0 msgr2=0x7f3b940fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.287+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1295859438 shutdown_connections 2024-04-23T18:22:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:43.287+0000 7f3b9b58a640 1 -- 172.21.15.50:0/1295859438 wait complete. 2024-04-23T18:22:44.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.29 MB/sec execute 78 sec latency 25889.592 ms 2024-04-23T18:22:44.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:44 smithi177 ceph-mon[32308]: pgmap v130: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 133 KiB/s wr, 17 op/s 2024-04-23T18:22:44.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:44 smithi188 ceph-mon[32694]: pgmap v130: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 133 KiB/s wr, 17 op/s 2024-04-23T18:22:44.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:44 smithi050 ceph-mon[29187]: pgmap v130: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 133 KiB/s wr, 17 op/s 2024-04-23T18:22:45.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.25 MB/sec execute 79 sec latency 26889.675 ms 2024-04-23T18:22:45.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:45 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:45.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:45 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:45.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:45 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:46.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:22:46.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 242380 3.21 MB/sec execute 80 sec latency 27889.743 ms 2024-04-23T18:22:46.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:46 smithi177 ceph-mon[32308]: pgmap v131: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 37 KiB/s wr, 7 op/s 2024-04-23T18:22:46.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.303+0000 7fb7b79b4640 1 -- 172.21.15.50:0/2565468747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7b0074320 msgr2=0x7fb7b0074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.303+0000 7fb7b79b4640 1 --2- 172.21.15.50:0/2565468747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7b0074320 0x7fb7b0074700 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009ca0 tx=0x7fb7ac02a950 comp rx=0 tx=0).stop 2024-04-23T18:22:46.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.303+0000 7fb7b79b4640 1 -- 172.21.15.50:0/2565468747 shutdown_connections 2024-04-23T18:22:46.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.303+0000 7fb7b79b4640 1 --2- 172.21.15.50:0/2565468747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7b010e800 0x7fb7b0110c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.303+0000 7fb7b79b4640 1 --2- 172.21.15.50:0/2565468747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7b0074c40 0x7fb7b010e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.303+0000 7fb7b79b4640 1 --2- 172.21.15.50:0/2565468747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7b0074320 0x7fb7b0074700 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.303+0000 7fb7b79b4640 1 -- 172.21.15.50:0/2565468747 >> 172.21.15.50:0/2565468747 conn(0x7fb7b00793e0 msgr2=0x7fb7b00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:46.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.303+0000 7fb7b79b4640 1 -- 172.21.15.50:0/2565468747 shutdown_connections 2024-04-23T18:22:46.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.303+0000 7fb7b79b4640 1 -- 172.21.15.50:0/2565468747 wait complete. 2024-04-23T18:22:46.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b79b4640 1 Processor -- start 2024-04-23T18:22:46.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b79b4640 1 -- start start 2024-04-23T18:22:46.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b79b4640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7b0074320 0x7fb7b01a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b79b4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7b0074c40 0x7fb7b01a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b79b4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7b010e800 0x7fb7b019a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b79b4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b0114420 con 0x7fb7b010e800 2024-04-23T18:22:46.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b79b4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01142a0 con 0x7fb7b0074c40 2024-04-23T18:22:46.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b79b4640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01145a0 con 0x7fb7b0074320 2024-04-23T18:22:46.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b5f51640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7b010e800 0x7fb7b019a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b4f4f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7b0074c40 0x7fb7b01a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b5f51640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7b010e800 0x7fb7b019a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40822/0 (socket says 172.21.15.50:40822) 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b5f51640 1 -- 172.21.15.50:0/4061766078 learned_addr learned my addr 172.21.15.50:0/4061766078 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b4f4f640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7b0074c40 0x7fb7b01a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:41646/0 (socket says 172.21.15.50:41646) 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b5750640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7b0074320 0x7fb7b01a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b5f51640 1 -- 172.21.15.50:0/4061766078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7b0074320 msgr2=0x7fb7b01a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b5f51640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7b0074320 0x7fb7b01a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b5f51640 1 -- 172.21.15.50:0/4061766078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7b0074c40 msgr2=0x7fb7b01a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b5f51640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7b0074c40 0x7fb7b01a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b5f51640 1 -- 172.21.15.50:0/4061766078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b019afb0 con 0x7fb7b010e800 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.304+0000 7fb7b4f4f640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7b0074c40 0x7fb7b01a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.305+0000 7fb7b5f51640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7b010e800 0x7fb7b019a7e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800d5e0 tx=0x7fb7a800daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.305+0000 7fb7a67fc640 1 -- 172.21.15.50:0/4061766078 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8017070 con 0x7fb7b010e800 2024-04-23T18:22:46.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.305+0000 7fb7b79b4640 1 -- 172.21.15.50:0/4061766078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b019b2a0 con 0x7fb7b010e800 2024-04-23T18:22:46.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.305+0000 7fb7a67fc640 1 -- 172.21.15.50:0/4061766078 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7a80047a0 con 0x7fb7b010e800 2024-04-23T18:22:46.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.305+0000 7fb7a67fc640 1 -- 172.21.15.50:0/4061766078 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8009b10 con 0x7fb7b010e800 2024-04-23T18:22:46.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.305+0000 7fb7b79b4640 1 -- 172.21.15.50:0/4061766078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b01a74c0 con 0x7fb7b010e800 2024-04-23T18:22:46.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.306+0000 7fb7b79b4640 1 -- 172.21.15.50:0/4061766078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7b00759d0 con 0x7fb7b010e800 2024-04-23T18:22:46.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.307+0000 7fb7a67fc640 1 -- 172.21.15.50:0/4061766078 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb7a8004950 con 0x7fb7b010e800 2024-04-23T18:22:46.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.308+0000 7fb7a67fc640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb79c0a3d20 0x7fb79c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.308+0000 7fb7a67fc640 1 -- 172.21.15.50:0/4061766078 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb7a80d0040 con 0x7fb7b010e800 2024-04-23T18:22:46.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.308+0000 7fb7b5750640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb79c0a3d20 0x7fb79c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.308+0000 7fb7b5750640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb79c0a3d20 0x7fb79c0a61e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac004ad0 tx=0x7fb7ac004a60 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:46.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.313+0000 7fb7a67fc640 1 -- 172.21.15.50:0/4061766078 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7a809c0a0 con 0x7fb7b010e800 2024-04-23T18:22:46.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:46 smithi188 ceph-mon[32694]: pgmap v131: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 37 KiB/s wr, 7 op/s 2024-04-23T18:22:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:46 smithi050 ceph-mon[29187]: pgmap v131: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 37 KiB/s wr, 7 op/s 2024-04-23T18:22:46.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.538+0000 7fb7b79b4640 1 -- 172.21.15.50:0/4061766078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7b002d5c0 con 0x7fb7b010e800 2024-04-23T18:22:46.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.539+0000 7fb7a67fc640 1 -- 172.21.15.50:0/4061766078 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11877 (secure 0 0 0) 0x7fb7a80bc030 con 0x7fb7b010e800 2024-04-23T18:22:46.533 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:22:46.533 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:22:43.811482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25547":{"gid":25547,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1939040460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1939040460},{"type":"v1","addr":"172.21.15.177:6837","nonce":1939040460}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:22:46.533 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 27 2024-04-23T18:22:46.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.541+0000 7fb7b79b4640 1 -- 172.21.15.50:0/4061766078 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb79c0a3d20 msgr2=0x7fb79c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.541+0000 7fb7b79b4640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb79c0a3d20 0x7fb79c0a61e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac004ad0 tx=0x7fb7ac004a60 comp rx=0 tx=0).stop 2024-04-23T18:22:46.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.541+0000 7fb7b79b4640 1 -- 172.21.15.50:0/4061766078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7b010e800 msgr2=0x7fb7b019a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.541+0000 7fb7b79b4640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7b010e800 0x7fb7b019a7e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800d5e0 tx=0x7fb7a800daa0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.541+0000 7fb7b79b4640 1 -- 172.21.15.50:0/4061766078 shutdown_connections 2024-04-23T18:22:46.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.541+0000 7fb7b79b4640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb79c0a3d20 0x7fb79c0a61e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.541+0000 7fb7b79b4640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb7b010e800 0x7fb7b019a7e0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.541+0000 7fb7b79b4640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7b0074c40 0x7fb7b01a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.541+0000 7fb7b79b4640 1 --2- 172.21.15.50:0/4061766078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7b0074320 0x7fb7b01a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.541+0000 7fb7b79b4640 1 -- 172.21.15.50:0/4061766078 >> 172.21.15.50:0/4061766078 conn(0x7fb7b00793e0 msgr2=0x7fb7b00fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:46.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.541+0000 7fb7b79b4640 1 -- 172.21.15.50:0/4061766078 shutdown_connections 2024-04-23T18:22:46.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.541+0000 7fb7b79b4640 1 -- 172.21.15.50:0/4061766078 wait complete. 2024-04-23T18:22:46.546 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id d13aa7a2 --release --await' 2024-04-23T18:22:46.546 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id d13aa7a2 --release --await 2024-04-23T18:22:46.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.703+0000 7fafa6e2a640 1 -- 172.21.15.50:0/2077503808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa01072b0 msgr2=0x7fafa010eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.703+0000 7fafa6e2a640 1 --2- 172.21.15.50:0/2077503808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa01072b0 0x7fafa010eec0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7faf88009900 tx=0x7faf8802a410 comp rx=0 tx=0).stop 2024-04-23T18:22:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.703+0000 7fafa6e2a640 1 -- 172.21.15.50:0/2077503808 shutdown_connections 2024-04-23T18:22:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.703+0000 7fafa6e2a640 1 --2- 172.21.15.50:0/2077503808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa01072b0 0x7fafa010eec0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.703+0000 7fafa6e2a640 1 --2- 172.21.15.50:0/2077503808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafa01014a0 0x7fafa0106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.703+0000 7fafa6e2a640 1 --2- 172.21.15.50:0/2077503808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafa0100af0 0x7fafa0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.703+0000 7fafa6e2a640 1 -- 172.21.15.50:0/2077503808 >> 172.21.15.50:0/2077503808 conn(0x7fafa00760e0 msgr2=0x7fafa00764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.703+0000 7fafa6e2a640 1 -- 172.21.15.50:0/2077503808 shutdown_connections 2024-04-23T18:22:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.703+0000 7fafa6e2a640 1 -- 172.21.15.50:0/2077503808 wait complete. 2024-04-23T18:22:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa6e2a640 1 Processor -- start 2024-04-23T18:22:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa6e2a640 1 -- start start 2024-04-23T18:22:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa6e2a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafa0100af0 0x7fafa0101a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa6e2a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafa01072b0 0x7fafa0101fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa6e2a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa01024e0 0x7fafa01a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa6e2a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fafa0112250 con 0x7fafa01024e0 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa6e2a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01120d0 con 0x7fafa0100af0 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa6e2a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01123d0 con 0x7fafa01072b0 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa53c7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa01024e0 0x7fafa01a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa53c7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa01024e0 0x7fafa01a6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40834/0 (socket says 172.21.15.50:40834) 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa53c7640 1 -- 172.21.15.50:0/1442285494 learned_addr learned my addr 172.21.15.50:0/1442285494 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa4bc6640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafa0100af0 0x7fafa0101a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7faf9ffff640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafa01072b0 0x7fafa0101fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa53c7640 1 -- 172.21.15.50:0/1442285494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafa01072b0 msgr2=0x7fafa0101fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa53c7640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafa01072b0 0x7fafa0101fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa53c7640 1 -- 172.21.15.50:0/1442285494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafa0100af0 msgr2=0x7fafa0101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa53c7640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafa0100af0 0x7fafa0101a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.704+0000 7fafa53c7640 1 -- 172.21.15.50:0/1442285494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa01a7040 con 0x7fafa01024e0 2024-04-23T18:22:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.705+0000 7fafa53c7640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa01024e0 0x7fafa01a6940 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7faf88004770 tx=0x7faf88004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.705+0000 7faf9dffb640 1 -- 172.21.15.50:0/1442285494 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8803f030 con 0x7fafa01024e0 2024-04-23T18:22:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.705+0000 7fafa6e2a640 1 -- 172.21.15.50:0/1442285494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa01a72d0 con 0x7fafa01024e0 2024-04-23T18:22:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.705+0000 7faf9dffb640 1 -- 172.21.15.50:0/1442285494 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf88003aa0 con 0x7fafa01024e0 2024-04-23T18:22:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.705+0000 7faf9dffb640 1 -- 172.21.15.50:0/1442285494 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf88035070 con 0x7fafa01024e0 2024-04-23T18:22:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.705+0000 7fafa6e2a640 1 -- 172.21.15.50:0/1442285494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa01a77c0 con 0x7fafa01024e0 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7faf9dffb640 1 -- 172.21.15.50:0/1442285494 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faf88042030 con 0x7fafa01024e0 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7fafa9c1d740 1 Processor -- start 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7fafa9c1d740 1 -- start start 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7fafa9c1d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438490 0x562281438870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7fafa9c1d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438db0 0x562281442f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7fafa9c1d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 0x562281445870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7fafa9c1d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56228139d6d0 con 0x562281443480 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7fafa9c1d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56228133a1a0 con 0x562281438490 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7fafa9c1d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562281399690 con 0x562281438db0 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7faf9ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 0x562281445870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7faf9ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 0x562281445870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40846/0 (socket says 172.21.15.50:40846) 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7faf9ffff640 1 -- 172.21.15.50:0/302325048 learned_addr learned my addr 172.21.15.50:0/302325048 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7faf9dffb640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faf8c0a3d50 0x7faf8c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7faf9ffff640 1 --2- 172.21.15.50:0/302325048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438490 0x562281438870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7fafa4bc6640 1 --2- 172.21.15.50:0/302325048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438db0 0x562281442f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7faf9dffb640 1 -- 172.21.15.50:0/1442285494 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faf880f0e60 con 0x7fafa01024e0 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7fafa53c7640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faf8c0a3d50 0x7faf8c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7faf9ffff640 1 -- 172.21.15.50:0/302325048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438db0 msgr2=0x562281442f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7faf9ffff640 1 --2- 172.21.15.50:0/302325048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438db0 0x562281442f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7faf9ffff640 1 -- 172.21.15.50:0/302325048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438490 msgr2=0x562281438870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7faf9ffff640 1 --2- 172.21.15.50:0/302325048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438490 0x562281438870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.708+0000 7faf9ffff640 1 -- 172.21.15.50:0/302325048 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622813445b0 con 0x562281443480 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.709+0000 7faf9ffff640 1 --2- 172.21.15.50:0/302325048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 0x562281445870 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7faf900126f0 tx=0x7faf9000b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4f4faa38b14d428 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.709+0000 7fafa53c7640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faf8c0a3d50 0x7faf8c0a6210 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faf88032fd0 tx=0x7faf8803b5d0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.709+0000 7faf96ffd640 1 -- 172.21.15.50:0/302325048 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90012c70 con 0x562281443480 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.709+0000 7faf96ffd640 1 -- 172.21.15.50:0/302325048 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf90012e10 con 0x562281443480 2024-04-23T18:22:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.709+0000 7faf96ffd640 1 -- 172.21.15.50:0/302325048 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf900049c0 con 0x562281443480 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.709+0000 7fafa9c1d740 1 -- 172.21.15.50:0/302325048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 msgr2=0x562281445870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.709+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/302325048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 0x562281445870 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7faf900126f0 tx=0x7faf9000b6e0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 -- 172.21.15.50:0/302325048 shutdown_connections 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/302325048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 0x562281445870 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/302325048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438db0 0x562281442f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/302325048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438490 0x562281438870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 -- 172.21.15.50:0/302325048 >> 172.21.15.50:0/302325048 conn(0x5622814347b0 msgr2=0x562281434b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 -- 172.21.15.50:0/302325048 shutdown_connections 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 -- 172.21.15.50:0/302325048 wait complete. 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 -- start start 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 Processor -- start 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438490 0x562281442f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438db0 0x562281440220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281443480 0x562281440960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562281446310 con 0x562281438db0 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562281337b20 con 0x562281443480 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.710+0000 7fafa9c1d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562281446490 con 0x562281438490 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa9c1d740 10 client.0 _ll_register_callbacks cb 0x562281337530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa4bc6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438db0 0x562281440220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa4bc6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438db0 0x562281440220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40850/0 (socket says 172.21.15.50:40850) 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa4bc6640 1 -- 172.21.15.50:0/3796830449 learned_addr learned my addr 172.21.15.50:0/3796830449 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7faf9ffff640 1 --2- 172.21.15.50:0/3796830449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438490 0x562281442f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa4bc6640 1 -- 172.21.15.50:0/3796830449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438490 msgr2=0x562281442f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa4bc6640 1 --2- 172.21.15.50:0/3796830449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438490 0x562281442f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa4bc6640 1 -- 172.21.15.50:0/3796830449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281443480 msgr2=0x562281440960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa4bc6640 1 --2- 172.21.15.50:0/3796830449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281443480 0x562281440960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa4bc6640 1 -- 172.21.15.50:0/3796830449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622810af920 con 0x562281438db0 2024-04-23T18:22:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa4bc6640 1 --2- 172.21.15.50:0/3796830449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438db0 0x562281440220 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7faf98004070 tx=0x7faf98004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d750959a79e845b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7faf7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7faf7ffff640 1 -- 172.21.15.50:0/3796830449 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98013c40 con 0x562281438db0 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7faf7ffff640 1 -- 172.21.15.50:0/3796830449 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faf98013de0 con 0x562281438db0 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa9c1d740 1 -- 172.21.15.50:0/3796830449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf90012e10 con 0x562281438db0 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7faf7ffff640 1 -- 172.21.15.50:0/3796830449 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faf9801c440 con 0x562281438db0 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa9c1d740 10 client.16056 fetch_fsmap learned FSMap version 27 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7fafa9c1d740 1 -- 172.21.15.50:0/3796830449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56228117bd30 con 0x562281438db0 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.711+0000 7faf7ffff640 1 -- 172.21.15.50:0/3796830449 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf90012e10 con 0x562281438db0 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.712+0000 7faf7ffff640 1 -- 172.21.15.50:0/3796830449 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7faf9801a070 con 0x562281438db0 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.712+0000 7fafa9c1d740 10 client.16056 fetch_fsmap finished waiting for FSMap version 27 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.712+0000 7fafa9c1d740 10 client.16056 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.712+0000 7fafa9c1d740 20 client.16056 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.712+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/3796830449 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562281453800 0x562281455c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.712+0000 7fafa9c1d740 4 client.16056 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.712+0000 7fafa9c1d740 1 -- 172.21.15.50:0/3796830449 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56228135b500 con 0x562281453800 2024-04-23T18:22:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.712+0000 7faf9ffff640 1 --2- 172.21.15.50:0/3796830449 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562281453800 0x562281455c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.712+0000 7faf7ffff640 10 client.16056 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.712+0000 7faf9ffff640 1 --2- 172.21.15.50:0/3796830449 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562281453800 0x562281455c40 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a671f2f3ec45ce74 server_cookie=a1921730f61b3870 in_seq=0 out_seq=0 2024-04-23T18:22:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.713+0000 7faf7ffff640 1 -- 172.21.15.50:0/3796830449 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56228135b500 con 0x562281453800 2024-04-23T18:22:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.713+0000 7faf7ffff640 10 client.16056 handle_command_reply: tid=0 2024-04-23T18:22:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.713+0000 7fafa9c1d740 1 client.16056 shutdown 2024-04-23T18:22:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.713+0000 7fafa9c1d740 1 -- 172.21.15.50:0/3796830449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438db0 msgr2=0x562281440220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.713+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/3796830449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438db0 0x562281440220 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7faf98004070 tx=0x7faf98004530 comp rx=0 tx=0).stop 2024-04-23T18:22:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.713+0000 7fafa9c1d740 1 -- 172.21.15.50:0/3796830449 shutdown_connections 2024-04-23T18:22:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.713+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/3796830449 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562281453800 0x562281455c40 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.713+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/3796830449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281443480 0x562281440960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.713+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/3796830449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438db0 0x562281440220 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.713+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/3796830449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438490 0x562281442f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.713+0000 7faf7ffff640 0 client.16056 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:46.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.713+0000 7fafa9c1d740 1 -- 172.21.15.50:0/3796830449 >> 172.21.15.50:0/3796830449 conn(0x5622814347b0 msgr2=0x56228143a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:46.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.714+0000 7fafa9c1d740 1 -- 172.21.15.50:0/3796830449 shutdown_connections 2024-04-23T18:22:46.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.714+0000 7fafa9c1d740 1 -- 172.21.15.50:0/3796830449 wait complete. 2024-04-23T18:22:46.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.714+0000 7fafa9c1d740 20 client.16056 trim_cache size 0 max 16384 2024-04-23T18:22:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.731+0000 7fafa9c1d740 1 Processor -- start 2024-04-23T18:22:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.731+0000 7fafa9c1d740 1 -- start start 2024-04-23T18:22:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.731+0000 7fafa9c1d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438490 0x56228144f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.731+0000 7fafa9c1d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438db0 0x562281442a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.731+0000 7fafa9c1d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281443480 0x56228145b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.731+0000 7fafa9c1d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01b25e0 con 0x562281438490 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.731+0000 7fafa9c1d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01afe80 con 0x562281438db0 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.731+0000 7fafa9c1d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01adcd0 con 0x562281443480 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.731+0000 7faf9ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438490 0x56228144f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.731+0000 7faf9ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438490 0x56228144f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40862/0 (socket says 172.21.15.50:40862) 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.731+0000 7faf9ffff640 1 -- 172.21.15.50:0/2583263362 learned_addr learned my addr 172.21.15.50:0/2583263362 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.732+0000 7fafa4bc6640 1 --2- 172.21.15.50:0/2583263362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438db0 0x562281442a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.732+0000 7faf9ffff640 1 -- 172.21.15.50:0/2583263362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281443480 msgr2=0x56228145b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.732+0000 7faf9ffff640 1 --2- 172.21.15.50:0/2583263362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281443480 0x56228145b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.732+0000 7faf9ffff640 1 -- 172.21.15.50:0/2583263362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438db0 msgr2=0x562281442a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.732+0000 7faf9ffff640 1 --2- 172.21.15.50:0/2583263362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438db0 0x562281442a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.732+0000 7faf9ffff640 1 -- 172.21.15.50:0/2583263362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562281440cd0 con 0x562281438490 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.732+0000 7fafa4bc6640 1 --2- 172.21.15.50:0/2583263362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438db0 0x562281442a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.732+0000 7faf9ffff640 1 --2- 172.21.15.50:0/2583263362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438490 0x56228144f1d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7faf90005590 tx=0x7faf900055c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2852c0da00c584c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.732+0000 7faf967fc640 1 -- 172.21.15.50:0/2583263362 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9002e070 con 0x562281438490 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.732+0000 7faf967fc640 1 -- 172.21.15.50:0/2583263362 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf90029870 con 0x562281438490 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.732+0000 7faf967fc640 1 -- 172.21.15.50:0/2583263362 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90010ac0 con 0x562281438490 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 -- 172.21.15.50:0/2583263362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438490 msgr2=0x56228144f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/2583263362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438490 0x56228144f1d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7faf90005590 tx=0x7faf900055c0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 -- 172.21.15.50:0/2583263362 shutdown_connections 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/2583263362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281443480 0x56228145b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/2583263362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438db0 0x562281442a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/2583263362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281438490 0x56228144f1d0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 -- 172.21.15.50:0/2583263362 >> 172.21.15.50:0/2583263362 conn(0x5622814347b0 msgr2=0x56228143a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 -- 172.21.15.50:0/2583263362 shutdown_connections 2024-04-23T18:22:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 -- 172.21.15.50:0/2583263362 wait complete. 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 -- start start 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 Processor -- start 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438490 0x56228144b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438db0 0x56228143d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 0x56228143da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01ae340 con 0x562281443480 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01af7b0 con 0x562281438db0 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01aea10 con 0x562281438490 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.733+0000 7fafa9c1d740 10 client.0 _ll_register_callbacks cb 0x7faf98003e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf9ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 0x56228143da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf9ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 0x56228143da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40874/0 (socket says 172.21.15.50:40874) 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf9ffff640 1 -- 172.21.15.50:0/912686621 learned_addr learned my addr 172.21.15.50:0/912686621 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf9ffff640 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438490 0x56228144b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7fafa4bc6640 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438db0 0x56228143d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf9ffff640 1 -- 172.21.15.50:0/912686621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438490 msgr2=0x56228144b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf9ffff640 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438490 0x56228144b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf9ffff640 1 -- 172.21.15.50:0/912686621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438db0 msgr2=0x56228143d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf9ffff640 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438db0 0x56228143d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf9ffff640 1 -- 172.21.15.50:0/912686621 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562281438be0 con 0x562281443480 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf9ffff640 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 0x56228143da50 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7faf900107a0 tx=0x7faf90005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff47d7dd8e736a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf7ffff640 1 -- 172.21.15.50:0/912686621 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9002e040 con 0x562281443480 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7fafa9c1d740 1 -- 172.21.15.50:0/912686621 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf90029870 con 0x562281443480 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf7ffff640 1 -- 172.21.15.50:0/912686621 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faf900050b0 con 0x562281443480 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7faf7ffff640 1 -- 172.21.15.50:0/912686621 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faf900052b0 con 0x562281443480 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7fafa9c1d740 10 client.16062 fetch_fsmap learned FSMap version 27 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.734+0000 7fafa9c1d740 1 -- 172.21.15.50:0/912686621 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56228143d280 con 0x562281443480 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.735+0000 7faf7ffff640 1 -- 172.21.15.50:0/912686621 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf90029870 con 0x562281443480 2024-04-23T18:22:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.735+0000 7faf7ffff640 1 -- 172.21.15.50:0/912686621 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7faf90034070 con 0x562281443480 2024-04-23T18:22:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.735+0000 7fafa9c1d740 10 client.16062 fetch_fsmap finished waiting for FSMap version 27 2024-04-23T18:22:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.735+0000 7fafa9c1d740 10 client.16062 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:22:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.735+0000 7fafa9c1d740 20 client.16062 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.735+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5622814bae90 0x5622814bd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.735+0000 7fafa9c1d740 4 client.16062 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "d13aa7a2", "release": true, "await": true}] 2024-04-23T18:22:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.735+0000 7fafa9c1d740 1 -- 172.21.15.50:0/912686621 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "d13aa7a2", "release": true, "await": true}) -- 0x5622813ab0f0 con 0x5622814bae90 2024-04-23T18:22:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.735+0000 7fafa4bc6640 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5622814bae90 0x5622814bd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.735+0000 7faf7ffff640 10 client.16062 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.735+0000 7fafa4bc6640 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5622814bae90 0x5622814bd280 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b19315bd18a6c2f8 server_cookie=f3e72e05cd642c0e in_seq=0 out_seq=0 2024-04-23T18:22:46.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7faf7ffff640 1 -- 172.21.15.50:0/912686621 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5622813ab0f0 con 0x5622814bae90 2024-04-23T18:22:46.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7faf7ffff640 10 client.16062 handle_command_reply: tid=0 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7fafa9c1d740 1 client.16062 shutdown 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7fafa9c1d740 1 -- 172.21.15.50:0/912686621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 msgr2=0x56228143da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 0x56228143da50 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7faf900107a0 tx=0x7faf90005f30 comp rx=0 tx=0).stop 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7fafa9c1d740 1 -- 172.21.15.50:0/912686621 shutdown_connections 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5622814bae90 0x5622814bd280 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562281443480 0x56228143da50 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562281438db0 0x56228143d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7faf7ffff640 0 client.16062 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7fafa9c1d740 1 --2- 172.21.15.50:0/912686621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562281438490 0x56228144b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7fafa9c1d740 1 -- 172.21.15.50:0/912686621 >> 172.21.15.50:0/912686621 conn(0x5622814347b0 msgr2=0x56228144e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7fafa9c1d740 1 -- 172.21.15.50:0/912686621 shutdown_connections 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7fafa9c1d740 1 -- 172.21.15.50:0/912686621 wait complete. 2024-04-23T18:22:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.740+0000 7fafa9c1d740 20 client.16062 trim_cache size 0 max 16384 2024-04-23T18:22:46.737 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:22:46.737 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 27, 2024-04-23T18:22:46.737 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:22:46.737 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 16, 2024-04-23T18:22:46.737 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:22:46.737 INFO:teuthology.orchestra.run.smithi050.stdout: "d13aa7a2": { 2024-04-23T18:22:46.737 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 16, 2024-04-23T18:22:46.737 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:22:46.737 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:22:46.737 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:22:46.737 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 87.8, 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:22:46.738 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:22:46.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.741+0000 7fafa6e2a640 1 -- 172.21.15.50:0/1442285494 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faf8c0a3d50 msgr2=0x7faf8c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.741+0000 7fafa6e2a640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faf8c0a3d50 0x7faf8c0a6210 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faf88032fd0 tx=0x7faf8803b5d0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.741+0000 7fafa6e2a640 1 -- 172.21.15.50:0/1442285494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa01024e0 msgr2=0x7fafa01a6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:46.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.741+0000 7fafa6e2a640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa01024e0 0x7fafa01a6940 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7faf88004770 tx=0x7faf88004270 comp rx=0 tx=0).stop 2024-04-23T18:22:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.741+0000 7fafa6e2a640 1 -- 172.21.15.50:0/1442285494 shutdown_connections 2024-04-23T18:22:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.741+0000 7fafa6e2a640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faf8c0a3d50 0x7faf8c0a6210 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.741+0000 7fafa6e2a640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafa01024e0 0x7fafa01a6940 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.741+0000 7fafa6e2a640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafa01072b0 0x7fafa0101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.741+0000 7fafa6e2a640 1 --2- 172.21.15.50:0/1442285494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafa0100af0 0x7fafa0101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.741+0000 7fafa6e2a640 1 -- 172.21.15.50:0/1442285494 >> 172.21.15.50:0/1442285494 conn(0x7fafa00760e0 msgr2=0x7fafa00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.741+0000 7fafa6e2a640 1 -- 172.21.15.50:0/1442285494 shutdown_connections 2024-04-23T18:22:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:22:46.741+0000 7fafa6e2a640 1 -- 172.21.15.50:0/1442285494 wait complete. 2024-04-23T18:22:46.745 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'd13aa7a2', seconds elapsed: 29.0 2024-04-23T18:22:47.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 243647 3.23 MB/sec execute 81 sec latency 28429.237 ms 2024-04-23T18:22:47.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:47 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4061766078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:47.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:47 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4061766078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:47.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:47 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4061766078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:22:48.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 245463 3.34 MB/sec execute 82 sec latency 15.003 ms 2024-04-23T18:22:48.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:48 smithi177 ceph-mon[32308]: pgmap v132: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 49 KiB/s wr, 15 op/s 2024-04-23T18:22:48.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:48 smithi188 ceph-mon[32694]: pgmap v132: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 49 KiB/s wr, 15 op/s 2024-04-23T18:22:48.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:48 smithi050 ceph-mon[29187]: pgmap v132: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 49 KiB/s wr, 15 op/s 2024-04-23T18:22:49.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 247800 3.45 MB/sec execute 83 sec latency 15.606 ms 2024-04-23T18:22:50.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 249729 3.55 MB/sec execute 84 sec latency 13.213 ms 2024-04-23T18:22:50.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:50 smithi177 ceph-mon[32308]: pgmap v133: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 KiB/s wr, 12 op/s 2024-04-23T18:22:50.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:50 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:50.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:50 smithi188 ceph-mon[32694]: pgmap v133: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 KiB/s wr, 12 op/s 2024-04-23T18:22:50.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:50 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:50 smithi050 ceph-mon[29187]: pgmap v133: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 KiB/s wr, 12 op/s 2024-04-23T18:22:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:50 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:22:51.202 INFO:tasks.workunit.client.0.smithi050.stdout: 1 252166 3.67 MB/sec execute 85 sec latency 10.233 ms 2024-04-23T18:22:52.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 254283 3.71 MB/sec execute 86 sec latency 15.121 ms 2024-04-23T18:22:52.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:52 smithi177 ceph-mon[32308]: pgmap v134: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 KiB/s wr, 12 op/s 2024-04-23T18:22:52.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:52 smithi188 ceph-mon[32694]: pgmap v134: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 KiB/s wr, 12 op/s 2024-04-23T18:22:52.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:52 smithi050 ceph-mon[29187]: pgmap v134: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 KiB/s wr, 12 op/s 2024-04-23T18:22:53.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 256114 3.81 MB/sec execute 87 sec latency 14.228 ms 2024-04-23T18:22:54.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 258321 3.90 MB/sec execute 88 sec latency 12.131 ms 2024-04-23T18:22:54.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:54 smithi177 ceph-mon[32308]: pgmap v135: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 722 KiB/s wr, 117 op/s 2024-04-23T18:22:54.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:54 smithi188 ceph-mon[32694]: pgmap v135: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 722 KiB/s wr, 117 op/s 2024-04-23T18:22:54.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:54 smithi050 ceph-mon[29187]: pgmap v135: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 722 KiB/s wr, 117 op/s 2024-04-23T18:22:55.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 260028 3.96 MB/sec execute 89 sec latency 29.261 ms 2024-04-23T18:22:56.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 262602 4.09 MB/sec execute 90 sec latency 9.223 ms 2024-04-23T18:22:56.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:56 smithi177 ceph-mon[32308]: pgmap v136: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 687 KiB/s wr, 113 op/s 2024-04-23T18:22:56.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:56 smithi188 ceph-mon[32694]: pgmap v136: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 687 KiB/s wr, 113 op/s 2024-04-23T18:22:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.378+0000 7f0499351640 1 -- 172.21.15.50:0/3365945703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0494102b80 msgr2=0x7f0494106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.378+0000 7f0499351640 1 --2- 172.21.15.50:0/3365945703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0494102b80 0x7f0494106bc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f047c009860 tx=0x7f047c02a340 comp rx=0 tx=0).stop 2024-04-23T18:22:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.378+0000 7f0499351640 1 -- 172.21.15.50:0/3365945703 shutdown_connections 2024-04-23T18:22:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.378+0000 7f0499351640 1 --2- 172.21.15.50:0/3365945703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04941072a0 0x7f049410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.378+0000 7f0499351640 1 --2- 172.21.15.50:0/3365945703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0494102b80 0x7f0494106bc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.378+0000 7f0499351640 1 --2- 172.21.15.50:0/3365945703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04941021d0 0x7f04941025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.378+0000 7f0499351640 1 -- 172.21.15.50:0/3365945703 >> 172.21.15.50:0/3365945703 conn(0x7f04940793e0 msgr2=0x7f04940797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.378+0000 7f0499351640 1 -- 172.21.15.50:0/3365945703 shutdown_connections 2024-04-23T18:22:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.378+0000 7f0499351640 1 -- 172.21.15.50:0/3365945703 wait complete. 2024-04-23T18:22:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f0499351640 1 Processor -- start 2024-04-23T18:22:56.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f0499351640 1 -- start start 2024-04-23T18:22:56.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f0499351640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04941021d0 0x7f0494074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f0499351640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04941072a0 0x7f0494074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f0499351640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0494074cd0 0x7f04941a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f0499351640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f04941121e0 con 0x7f04941021d0 2024-04-23T18:22:56.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f0499351640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0494112060 con 0x7f0494074cd0 2024-04-23T18:22:56.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f0499351640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0494112360 con 0x7f04941072a0 2024-04-23T18:22:56.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f0492ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04941021d0 0x7f0494074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f04927fc640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04941072a0 0x7f0494074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f0492ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04941021d0 0x7f0494074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47596/0 (socket says 172.21.15.50:47596) 2024-04-23T18:22:56.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f0492ffd640 1 -- 172.21.15.50:0/1864810629 learned_addr learned my addr 172.21.15.50:0/1864810629 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:56.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.379+0000 7f04937fe640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0494074cd0 0x7f04941a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.380+0000 7f0492ffd640 1 -- 172.21.15.50:0/1864810629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04941072a0 msgr2=0x7f0494074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:56.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.380+0000 7f0492ffd640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04941072a0 0x7f0494074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.380+0000 7f0492ffd640 1 -- 172.21.15.50:0/1864810629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0494074cd0 msgr2=0x7f04941a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:56.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.380+0000 7f0492ffd640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0494074cd0 0x7f04941a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.380+0000 7f0492ffd640 1 -- 172.21.15.50:0/1864810629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04941a6f90 con 0x7f04941021d0 2024-04-23T18:22:56.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.380+0000 7f0492ffd640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04941021d0 0x7f0494074250 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f048400e9c0 tx=0x7f048400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:56.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.380+0000 7f048bfff640 1 -- 172.21.15.50:0/1864810629 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048400cde0 con 0x7f04941021d0 2024-04-23T18:22:56.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.380+0000 7f0499351640 1 -- 172.21.15.50:0/1864810629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04941a7280 con 0x7f04941021d0 2024-04-23T18:22:56.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.380+0000 7f048bfff640 1 -- 172.21.15.50:0/1864810629 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04840035f0 con 0x7f04941021d0 2024-04-23T18:22:56.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.380+0000 7f048bfff640 1 -- 172.21.15.50:0/1864810629 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0484010690 con 0x7f04941021d0 2024-04-23T18:22:56.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.380+0000 7f0499351640 1 -- 172.21.15.50:0/1864810629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04941a77a0 con 0x7f04941021d0 2024-04-23T18:22:56.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.382+0000 7f048bfff640 1 -- 172.21.15.50:0/1864810629 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0484004000 con 0x7f04941021d0 2024-04-23T18:22:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f049c060740 1 Processor -- start 2024-04-23T18:22:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f049c060740 1 -- start start 2024-04-23T18:22:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f049c060740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60c620 0x55fa2b60ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f049c060740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60cf40 0x55fa2b6172c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f049c060740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 0x55fa2b619bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f049c060740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55fa2b571970 con 0x55fa2b617800 2024-04-23T18:22:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f049c060740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fa2b50e4a0 con 0x55fa2b60cf40 2024-04-23T18:22:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f049c060740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fa2b56d930 con 0x55fa2b60c620 2024-04-23T18:22:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f04927fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 0x55fa2b619bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f0492ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60cf40 0x55fa2b6172c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f04937fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60c620 0x55fa2b60ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f048bfff640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f04800a3df0 0x7f04800a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f04927fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 0x55fa2b619bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47612/0 (socket says 172.21.15.50:47612) 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.383+0000 7f04927fc640 1 -- 172.21.15.50:0/4119705957 learned_addr learned my addr 172.21.15.50:0/4119705957 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.384+0000 7f04937fe640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f04800a3df0 0x7f04800a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.384+0000 7f04927fc640 1 -- 172.21.15.50:0/4119705957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60c620 msgr2=0x55fa2b60ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.384+0000 7f04927fc640 1 --2- 172.21.15.50:0/4119705957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60c620 0x55fa2b60ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.384+0000 7f04927fc640 1 -- 172.21.15.50:0/4119705957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60cf40 msgr2=0x55fa2b6172c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.384+0000 7f04927fc640 1 --2- 172.21.15.50:0/4119705957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60cf40 0x55fa2b6172c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.384+0000 7f04927fc640 1 -- 172.21.15.50:0/4119705957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa2b284210 con 0x55fa2b617800 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.384+0000 7f048bfff640 1 -- 172.21.15.50:0/1864810629 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f04840d0b40 con 0x7f04941021d0 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.384+0000 7f04927fc640 1 --2- 172.21.15.50:0/4119705957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 0x55fa2b619bf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f049406b630 tx=0x7f047c005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97bb652940f1e1ae server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.384+0000 7f04897fa640 1 -- 172.21.15.50:0/4119705957 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c037040 con 0x55fa2b617800 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.384+0000 7f04937fe640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f04800a3df0 0x7f04800a62b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f048c005fd0 tx=0x7f048c004ab0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.384+0000 7f04897fa640 1 -- 172.21.15.50:0/4119705957 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f047c030450 con 0x55fa2b617800 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.384+0000 7f04897fa640 1 -- 172.21.15.50:0/4119705957 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c003970 con 0x55fa2b617800 2024-04-23T18:22:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 -- 172.21.15.50:0/4119705957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 msgr2=0x55fa2b619bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 --2- 172.21.15.50:0/4119705957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 0x55fa2b619bf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f049406b630 tx=0x7f047c005b10 comp rx=0 tx=0).stop 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 -- 172.21.15.50:0/4119705957 shutdown_connections 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 --2- 172.21.15.50:0/4119705957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 0x55fa2b619bf0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 --2- 172.21.15.50:0/4119705957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60cf40 0x55fa2b6172c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 --2- 172.21.15.50:0/4119705957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60c620 0x55fa2b60ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 -- 172.21.15.50:0/4119705957 >> 172.21.15.50:0/4119705957 conn(0x55fa2b608b30 msgr2=0x55fa2b608f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 -- 172.21.15.50:0/4119705957 shutdown_connections 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 -- 172.21.15.50:0/4119705957 wait complete. 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 -- start start 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 Processor -- start 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60c620 0x55fa2b617150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 0x55fa2b614440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b617800 0x55fa2b6149a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55fa2b61a3d0 con 0x55fa2b60cf40 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fa2b50c0b0 con 0x55fa2b617800 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.385+0000 7f049c060740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fa2b61a550 con 0x55fa2b60c620 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f04927fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 0x55fa2b614440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f049c060740 10 client.0 _ll_register_callbacks cb 0x55fa2b4f9d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f0492ffd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60c620 0x55fa2b617150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f04927fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 0x55fa2b614440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47622/0 (socket says 172.21.15.50:47622) 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f04927fc640 1 -- 172.21.15.50:0/1976799157 learned_addr learned my addr 172.21.15.50:0/1976799157 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f0492ffd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60c620 0x55fa2b617150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:36650/0 (socket says 172.21.15.50:36650) 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f04937fe640 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b617800 0x55fa2b6149a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f0492ffd640 1 -- 172.21.15.50:0/1976799157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b617800 msgr2=0x55fa2b6149a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f0492ffd640 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b617800 0x55fa2b6149a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f0492ffd640 1 -- 172.21.15.50:0/1976799157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 msgr2=0x55fa2b614440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f0492ffd640 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 0x55fa2b614440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f0492ffd640 1 -- 172.21.15.50:0/1976799157 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa2b350620 con 0x55fa2b60c620 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f04927fc640 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 0x55fa2b614440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f0492ffd640 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60c620 0x55fa2b617150 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04840ce240 tx=0x7f0484085fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df18c9e62c7ec576 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f047a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f047a7fc640 1 -- 172.21.15.50:0/1976799157 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04840a5700 con 0x55fa2b60c620 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f047a7fc640 1 -- 172.21.15.50:0/1976799157 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f0484003aa0 con 0x55fa2b60c620 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f049c060740 1 -- 172.21.15.50:0/1976799157 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f047c030450 con 0x55fa2b60c620 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f047a7fc640 1 -- 172.21.15.50:0/1976799157 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f04840a0e40 con 0x55fa2b60c620 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.386+0000 7f049c060740 10 client.25753 fetch_fsmap learned FSMap version 28 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.387+0000 7f049c060740 1 -- 172.21.15.50:0/1976799157 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fa2b626300 con 0x55fa2b60c620 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.387+0000 7f047a7fc640 1 -- 172.21.15.50:0/1976799157 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f047c030450 con 0x55fa2b60c620 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.387+0000 7f047a7fc640 1 -- 172.21.15.50:0/1976799157 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f048409ccf0 con 0x55fa2b60c620 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.387+0000 7f049c060740 10 client.25753 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:22:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.387+0000 7f049c060740 10 client.25753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.387+0000 7f049c060740 20 client.25753 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.387+0000 7f049c060740 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fa2b6277c0 0x55fa2b629c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.387+0000 7f049c060740 4 client.25753 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:22:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.387+0000 7f049c060740 1 -- 172.21.15.50:0/1976799157 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa2b57f390 con 0x55fa2b6277c0 2024-04-23T18:22:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.387+0000 7f04927fc640 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fa2b6277c0 0x55fa2b629c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.388+0000 7f047a7fc640 10 client.25753 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.388+0000 7f04927fc640 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fa2b6277c0 0x55fa2b629c50 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ad67b6663277916 server_cookie=4469a2275484210b in_seq=0 out_seq=0 2024-04-23T18:22:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f047a7fc640 1 -- 172.21.15.50:0/1976799157 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fa2b57f390 con 0x55fa2b6277c0 2024-04-23T18:22:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f047a7fc640 10 client.25753 handle_command_reply: tid=0 2024-04-23T18:22:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f049c060740 1 client.25753 shutdown 2024-04-23T18:22:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f049c060740 1 -- 172.21.15.50:0/1976799157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60c620 msgr2=0x55fa2b617150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f049c060740 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60c620 0x55fa2b617150 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04840ce240 tx=0x7f0484085fd0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f049c060740 1 -- 172.21.15.50:0/1976799157 shutdown_connections 2024-04-23T18:22:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f049c060740 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fa2b6277c0 0x55fa2b629c50 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f049c060740 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b617800 0x55fa2b6149a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f047a7fc640 0 client.25753 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f049c060740 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 0x55fa2b614440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f049c060740 1 --2- 172.21.15.50:0/1976799157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60c620 0x55fa2b617150 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f049c060740 1 -- 172.21.15.50:0/1976799157 >> 172.21.15.50:0/1976799157 conn(0x55fa2b608b30 msgr2=0x55fa2b60e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f049c060740 1 -- 172.21.15.50:0/1976799157 shutdown_connections 2024-04-23T18:22:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f049c060740 1 -- 172.21.15.50:0/1976799157 wait complete. 2024-04-23T18:22:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.389+0000 7f049c060740 20 client.25753 trim_cache size 0 max 16384 2024-04-23T18:22:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.404+0000 7f049c060740 1 Processor -- start 2024-04-23T18:22:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f049c060740 1 -- start start 2024-04-23T18:22:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f049c060740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60c620 0x55fa2b623290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f049c060740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 0x55fa2b60f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f049c060740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b617800 0x55fa2b616c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f049c060740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f04941b26c0 con 0x55fa2b60cf40 2024-04-23T18:22:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f049c060740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f04941aff30 con 0x55fa2b60c620 2024-04-23T18:22:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f049c060740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f04941add80 con 0x55fa2b617800 2024-04-23T18:22:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f04927fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 0x55fa2b60f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f0492ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60c620 0x55fa2b623290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f04927fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 0x55fa2b60f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47634/0 (socket says 172.21.15.50:47634) 2024-04-23T18:22:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f04927fc640 1 -- 172.21.15.50:0/3496525423 learned_addr learned my addr 172.21.15.50:0/3496525423 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f04937fe640 1 --2- 172.21.15.50:0/3496525423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b617800 0x55fa2b616c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f04927fc640 1 -- 172.21.15.50:0/3496525423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b617800 msgr2=0x55fa2b616c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f04927fc640 1 --2- 172.21.15.50:0/3496525423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b617800 0x55fa2b616c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f04927fc640 1 -- 172.21.15.50:0/3496525423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60c620 msgr2=0x55fa2b623290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f04927fc640 1 --2- 172.21.15.50:0/3496525423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60c620 0x55fa2b623290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f04927fc640 1 -- 172.21.15.50:0/3496525423 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa2b60cd70 con 0x55fa2b60cf40 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.405+0000 7f0492ffd640 1 --2- 172.21.15.50:0/3496525423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60c620 0x55fa2b623290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f04927fc640 1 --2- 172.21.15.50:0/3496525423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 0x55fa2b60f970 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f047c009860 tx=0x7f047c005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78b0925b3e1e072 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f0488ff9640 1 -- 172.21.15.50:0/3496525423 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c037040 con 0x55fa2b60cf40 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f0488ff9640 1 -- 172.21.15.50:0/3496525423 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f047c050bd0 con 0x55fa2b60cf40 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f0488ff9640 1 -- 172.21.15.50:0/3496525423 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c0506a0 con 0x55fa2b60cf40 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f049c060740 1 -- 172.21.15.50:0/3496525423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 msgr2=0x55fa2b60f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f049c060740 1 --2- 172.21.15.50:0/3496525423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 0x55fa2b60f970 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f047c009860 tx=0x7f047c005fb0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f049c060740 1 -- 172.21.15.50:0/3496525423 shutdown_connections 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f049c060740 1 --2- 172.21.15.50:0/3496525423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b617800 0x55fa2b616c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f049c060740 1 --2- 172.21.15.50:0/3496525423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b60cf40 0x55fa2b60f970 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f049c060740 1 --2- 172.21.15.50:0/3496525423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60c620 0x55fa2b623290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f049c060740 1 -- 172.21.15.50:0/3496525423 >> 172.21.15.50:0/3496525423 conn(0x55fa2b608b30 msgr2=0x55fa2b60f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f049c060740 1 -- 172.21.15.50:0/3496525423 shutdown_connections 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.406+0000 7f049c060740 1 -- 172.21.15.50:0/3496525423 wait complete. 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f049c060740 1 -- start start 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f049c060740 1 Processor -- start 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f049c060740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60c620 0x55fa2b61f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f049c060740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60cf40 0x55fa2b61ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f049c060740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 0x55fa2b61b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f049c060740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f04941ae3f0 con 0x55fa2b617800 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f049c060740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f04941af860 con 0x55fa2b60c620 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f049c060740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f04941aeac0 con 0x55fa2b60cf40 2024-04-23T18:22:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f049c060740 10 client.0 _ll_register_callbacks cb 0x7f04840046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:22:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f04937fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 0x55fa2b61b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f0492ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60c620 0x55fa2b61f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f04937fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 0x55fa2b61b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47648/0 (socket says 172.21.15.50:47648) 2024-04-23T18:22:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f04937fe640 1 -- 172.21.15.50:0/3311900567 learned_addr learned my addr 172.21.15.50:0/3311900567 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:22:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f04927fc640 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60cf40 0x55fa2b61ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f04937fe640 1 -- 172.21.15.50:0/3311900567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60cf40 msgr2=0x55fa2b61ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f04937fe640 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60cf40 0x55fa2b61ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f04937fe640 1 -- 172.21.15.50:0/3311900567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60c620 msgr2=0x55fa2b61f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:22:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f04937fe640 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60c620 0x55fa2b61f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.407+0000 7f04937fe640 1 -- 172.21.15.50:0/3311900567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa2b61b900 con 0x55fa2b617800 2024-04-23T18:22:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f0492ffd640 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60c620 0x55fa2b61f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f04937fe640 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 0x55fa2b61b2c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f048c00fac0 tx=0x7f048c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129f7fe4008cf8a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f047a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f047a7fc640 1 -- 172.21.15.50:0/3311900567 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c030060 con 0x55fa2b617800 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f047a7fc640 1 -- 172.21.15.50:0/3311900567 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f048c022c20 con 0x55fa2b617800 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f049c060740 1 -- 172.21.15.50:0/3311900567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f047c050bd0 con 0x55fa2b617800 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f047a7fc640 1 -- 172.21.15.50:0/3311900567 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f048c022e20 con 0x55fa2b617800 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f049c060740 10 client.16080 fetch_fsmap learned FSMap version 28 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f049c060740 1 -- 172.21.15.50:0/3311900567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fa2b68dbf0 con 0x55fa2b617800 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f047a7fc640 1 -- 172.21.15.50:0/3311900567 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f047c050bd0 con 0x55fa2b617800 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f047a7fc640 1 -- 172.21.15.50:0/3311900567 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f048c02a070 con 0x55fa2b617800 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f049c060740 10 client.16080 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f049c060740 10 client.16080 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f049c060740 20 client.16080 populate_metadata read hostname 'smithi050' 2024-04-23T18:22:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f049c060740 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fa2b68e400 0x55fa2b690820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:22:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f049c060740 4 client.16080 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:22:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.408+0000 7f049c060740 1 -- 172.21.15.50:0/3311900567 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fa2b579200 con 0x55fa2b68e400 2024-04-23T18:22:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.409+0000 7f0492ffd640 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fa2b68e400 0x55fa2b690820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:22:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.409+0000 7f0492ffd640 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fa2b68e400 0x55fa2b690820 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fa6183a846a48b5 server_cookie=adceea761fcce795 in_seq=0 out_seq=0 2024-04-23T18:22:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.409+0000 7f047a7fc640 10 client.16080 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f047a7fc640 1 -- 172.21.15.50:0/3311900567 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fa2b579200 con 0x55fa2b68e400 2024-04-23T18:22:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f047a7fc640 10 client.16080 handle_command_reply: tid=0 2024-04-23T18:22:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f049c060740 1 client.16080 shutdown 2024-04-23T18:22:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f049c060740 1 -- 172.21.15.50:0/3311900567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 msgr2=0x55fa2b61b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f049c060740 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 0x55fa2b61b2c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f048c00fac0 tx=0x7f048c004a40 comp rx=0 tx=0).stop 2024-04-23T18:22:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f049c060740 1 -- 172.21.15.50:0/3311900567 shutdown_connections 2024-04-23T18:22:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f049c060740 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fa2b68e400 0x55fa2b690820 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f049c060740 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fa2b617800 0x55fa2b61b2c0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f049c060740 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fa2b60cf40 0x55fa2b61ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f047a7fc640 0 client.16080 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:22:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f049c060740 1 --2- 172.21.15.50:0/3311900567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa2b60c620 0x55fa2b61f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f049c060740 1 -- 172.21.15.50:0/3311900567 >> 172.21.15.50:0/3311900567 conn(0x55fa2b608b30 msgr2=0x55fa2b607cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:56.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f049c060740 1 -- 172.21.15.50:0/3311900567 shutdown_connections 2024-04-23T18:22:56.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f049c060740 1 -- 172.21.15.50:0/3311900567 wait complete. 2024-04-23T18:22:56.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.449+0000 7f049c060740 20 client.16080 trim_cache size 0 max 16384 2024-04-23T18:22:56.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:22:56.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:22:56.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:22:56.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:22:56.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.450+0000 7f0499351640 1 -- 172.21.15.50:0/1864810629 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f04800a3df0 msgr2=0x7f04800a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:56.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.450+0000 7f0499351640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f04800a3df0 0x7f04800a62b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f048c005fd0 tx=0x7f048c004ab0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.450+0000 7f0499351640 1 -- 172.21.15.50:0/1864810629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04941021d0 msgr2=0x7f0494074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:22:56.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.450+0000 7f0499351640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04941021d0 0x7f0494074250 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f048400e9c0 tx=0x7f048400ee80 comp rx=0 tx=0).stop 2024-04-23T18:22:56.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.450+0000 7f0499351640 1 -- 172.21.15.50:0/1864810629 shutdown_connections 2024-04-23T18:22:56.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.450+0000 7f0499351640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f04800a3df0 0x7f04800a62b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.450+0000 7f0499351640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0494074cd0 0x7f04941a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.450+0000 7f0499351640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04941072a0 0x7f0494074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.450+0000 7f0499351640 1 --2- 172.21.15.50:0/1864810629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04941021d0 0x7f0494074250 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:22:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.450+0000 7f0499351640 1 -- 172.21.15.50:0/1864810629 >> 172.21.15.50:0/1864810629 conn(0x7f04940793e0 msgr2=0x7f04940ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:22:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.450+0000 7f0499351640 1 -- 172.21.15.50:0/1864810629 shutdown_connections 2024-04-23T18:22:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:22:56.450+0000 7f0499351640 1 -- 172.21.15.50:0/1864810629 wait complete. 2024-04-23T18:22:56.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:56 smithi050 ceph-mon[29187]: pgmap v136: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 687 KiB/s wr, 113 op/s 2024-04-23T18:22:57.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 263922 4.11 MB/sec execute 91 sec latency 113.182 ms 2024-04-23T18:22:58.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 266320 4.22 MB/sec execute 92 sec latency 10.333 ms 2024-04-23T18:22:58.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:22:58 smithi177 ceph-mon[32308]: pgmap v137: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 235 op/s 2024-04-23T18:22:58.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:22:58 smithi188 ceph-mon[32694]: pgmap v137: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 235 op/s 2024-04-23T18:22:58.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:22:58 smithi050 ceph-mon[29187]: pgmap v137: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 235 op/s 2024-04-23T18:22:59.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 268395 4.25 MB/sec execute 93 sec latency 19.092 ms 2024-04-23T18:23:00.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 270173 4.34 MB/sec execute 94 sec latency 13.293 ms 2024-04-23T18:23:00.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:00 smithi177 ceph-mon[32308]: pgmap v138: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 227 op/s 2024-04-23T18:23:00.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:00 smithi188 ceph-mon[32694]: pgmap v138: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 227 op/s 2024-04-23T18:23:00.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:00 smithi050 ceph-mon[29187]: pgmap v138: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 227 op/s 2024-04-23T18:23:01.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 271694 4.35 MB/sec execute 95 sec latency 28.752 ms 2024-04-23T18:23:02.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 273613 4.44 MB/sec execute 96 sec latency 10.844 ms 2024-04-23T18:23:02.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:02 smithi177 ceph-mon[32308]: pgmap v139: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 227 op/s 2024-04-23T18:23:02.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:02 smithi188 ceph-mon[32694]: pgmap v139: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 227 op/s 2024-04-23T18:23:02.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:02 smithi050 ceph-mon[29187]: pgmap v139: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.4 MiB/s wr, 227 op/s 2024-04-23T18:23:03.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 275919 4.51 MB/sec execute 97 sec latency 13.424 ms 2024-04-23T18:23:04.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.55 MB/sec execute 98 sec latency 235.036 ms 2024-04-23T18:23:04.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:04 smithi177 ceph-mon[32308]: pgmap v140: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 366 op/s 2024-04-23T18:23:04.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:04 smithi188 ceph-mon[32694]: pgmap v140: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 366 op/s 2024-04-23T18:23:04.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:04 smithi050 ceph-mon[29187]: pgmap v140: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 366 op/s 2024-04-23T18:23:05.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.50 MB/sec execute 99 sec latency 1235.133 ms 2024-04-23T18:23:06.203 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.46 MB/sec execute 100 sec latency 2235.257 ms 2024-04-23T18:23:06.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:06 smithi177 ceph-mon[32308]: pgmap v141: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 261 op/s 2024-04-23T18:23:06.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:06 smithi188 ceph-mon[32694]: pgmap v141: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 261 op/s 2024-04-23T18:23:06.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:06 smithi050 ceph-mon[29187]: pgmap v141: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 261 op/s 2024-04-23T18:23:07.204 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.41 MB/sec execute 101 sec latency 3235.397 ms 2024-04-23T18:23:08.204 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.37 MB/sec execute 102 sec latency 4235.482 ms 2024-04-23T18:23:08.287 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:08 smithi177 ceph-mon[32308]: pgmap v142: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 318 op/s 2024-04-23T18:23:08.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:08 smithi188 ceph-mon[32694]: pgmap v142: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 318 op/s 2024-04-23T18:23:08.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:08 smithi050 ceph-mon[29187]: pgmap v142: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 318 op/s 2024-04-23T18:23:09.204 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.33 MB/sec execute 103 sec latency 5235.595 ms 2024-04-23T18:23:09.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.549+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1494481824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa9c107290 msgr2=0x7ffa9c10ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:09.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.549+0000 7ffaa3f08640 1 --2- 172.21.15.50:0/1494481824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa9c107290 0x7ffa9c10ee60 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009ca0 tx=0x7ffa9402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.549+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1494481824 shutdown_connections 2024-04-23T18:23:09.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.549+0000 7ffaa3f08640 1 --2- 172.21.15.50:0/1494481824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa9c107290 0x7ffa9c10ee60 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.549+0000 7ffaa3f08640 1 --2- 172.21.15.50:0/1494481824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa9c102b70 0x7ffa9c106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.549+0000 7ffaa3f08640 1 --2- 172.21.15.50:0/1494481824 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa9c1021c0 0x7ffa9c1025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.549+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1494481824 >> 172.21.15.50:0/1494481824 conn(0x7ffa9c0fda20 msgr2=0x7ffa9c0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.549+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1494481824 shutdown_connections 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.550+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1494481824 wait complete. 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.550+0000 7ffaa3f08640 1 Processor -- start 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.550+0000 7ffaa3f08640 1 -- start start 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.550+0000 7ffaa3f08640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa9c1021c0 0x7ffa9c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.550+0000 7ffaa3f08640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa9c107290 0x7ffa9c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.550+0000 7ffaa3f08640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa9c074cd0 0x7ffa9c1a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.550+0000 7ffaa3f08640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c112200 con 0x7ffa9c1021c0 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.550+0000 7ffaa3f08640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c112080 con 0x7ffa9c107290 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.550+0000 7ffaa3f08640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c112380 con 0x7ffa9c074cd0 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.550+0000 7ffaa1ca4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa9c1021c0 0x7ffa9c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffaa1ca4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa9c1021c0 0x7ffa9c074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58446/0 (socket says 172.21.15.50:58446) 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffaa1ca4640 1 -- 172.21.15.50:0/1726708096 learned_addr learned my addr 172.21.15.50:0/1726708096 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffaa24a5640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa9c074cd0 0x7ffa9c1a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffaa14a3640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa9c107290 0x7ffa9c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffaa1ca4640 1 -- 172.21.15.50:0/1726708096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa9c074cd0 msgr2=0x7ffa9c1a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffaa1ca4640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa9c074cd0 0x7ffa9c1a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffaa1ca4640 1 -- 172.21.15.50:0/1726708096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa9c107290 msgr2=0x7ffa9c074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffaa1ca4640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa9c107290 0x7ffa9c074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffaa1ca4640 1 -- 172.21.15.50:0/1726708096 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa9c1a6fb0 con 0x7ffa9c1021c0 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffaa1ca4640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa9c1021c0 0x7ffa9c074250 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800d770 tx=0x7ffa9800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffa92ffd640 1 -- 172.21.15.50:0/1726708096 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98017070 con 0x7ffa9c1021c0 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1726708096 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa9c1a72a0 con 0x7ffa9c1021c0 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffa92ffd640 1 -- 172.21.15.50:0/1726708096 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa980054d0 con 0x7ffa9c1021c0 2024-04-23T18:23:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffa92ffd640 1 -- 172.21.15.50:0/1726708096 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98004c30 con 0x7ffa9c1021c0 2024-04-23T18:23:09.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.551+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1726708096 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa9c1a77f0 con 0x7ffa9c1021c0 2024-04-23T18:23:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.554+0000 7ffa92ffd640 1 -- 172.21.15.50:0/1726708096 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffa98004e10 con 0x7ffa9c1021c0 2024-04-23T18:23:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.554+0000 7ffaa6c27740 1 Processor -- start 2024-04-23T18:23:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.554+0000 7ffaa6c27740 1 -- start start 2024-04-23T18:23:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.554+0000 7ffaa6c27740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 0x55b2c0ecea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.554+0000 7ffaa6c27740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.554+0000 7ffaa6c27740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0edbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.554+0000 7ffaa6c27740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b2c0e33970 con 0x55b2c0ece620 2024-04-23T18:23:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.554+0000 7ffaa6c27740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b2c0dd04a0 con 0x55b2c0ecef40 2024-04-23T18:23:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.554+0000 7ffaa6c27740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b2c0e2f930 con 0x55b2c0ed9800 2024-04-23T18:23:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffa92ffd640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ffa880a3dc0 0x7ffa880a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa24a5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 0x55b2c0ecea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa24a5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 0x55b2c0ecea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58452/0 (socket says 172.21.15.50:58452) 2024-04-23T18:23:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa24a5640 1 -- 172.21.15.50:0/342901379 learned_addr learned my addr 172.21.15.50:0/342901379 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa14a3640 1 --2- 172.21.15.50:0/342901379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0edbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa1ca4640 1 --2- 172.21.15.50:0/342901379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffa92ffd640 1 -- 172.21.15.50:0/1726708096 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7ffa980d0210 con 0x7ffa9c1021c0 2024-04-23T18:23:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa24a5640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ffa880a3dc0 0x7ffa880a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa24a5640 1 -- 172.21.15.50:0/342901379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 msgr2=0x55b2c0edbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa24a5640 1 --2- 172.21.15.50:0/342901379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0edbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa24a5640 1 -- 172.21.15.50:0/342901379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 msgr2=0x55b2c0ed92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa24a5640 1 --2- 172.21.15.50:0/342901379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa24a5640 1 -- 172.21.15.50:0/342901379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2c0b46210 con 0x55b2c0ece620 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa14a3640 1 --2- 172.21.15.50:0/342901379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0edbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffaa24a5640 1 --2- 172.21.15.50:0/342901379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 0x55b2c0ecea00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ffa94035000 tx=0x7ffa94031e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aecd2eb9865307f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.555+0000 7ffa87fff640 1 -- 172.21.15.50:0/342901379 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa94004b40 con 0x55b2c0ece620 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.556+0000 7ffa87fff640 1 -- 172.21.15.50:0/342901379 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9404f070 con 0x55b2c0ece620 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.556+0000 7ffa87fff640 1 -- 172.21.15.50:0/342901379 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa94004570 con 0x55b2c0ece620 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.556+0000 7ffaa6c27740 1 -- 172.21.15.50:0/342901379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 msgr2=0x55b2c0ecea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.556+0000 7ffaa24a5640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ffa880a3dc0 0x7ffa880a6280 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ffa9404e000 tx=0x7ffa9404c000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.556+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/342901379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 0x55b2c0ecea00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ffa94035000 tx=0x7ffa94031e40 comp rx=0 tx=0).stop 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.556+0000 7ffaa6c27740 1 -- 172.21.15.50:0/342901379 shutdown_connections 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.556+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/342901379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0edbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.556+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/342901379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.556+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/342901379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 0x55b2c0ecea00 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.556+0000 7ffaa6c27740 1 -- 172.21.15.50:0/342901379 >> 172.21.15.50:0/342901379 conn(0x55b2c0ecab30 msgr2=0x55b2c0ecaf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.556+0000 7ffaa6c27740 1 -- 172.21.15.50:0/342901379 shutdown_connections 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.557+0000 7ffaa6c27740 1 -- 172.21.15.50:0/342901379 wait complete. 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.557+0000 7ffaa6c27740 1 -- start start 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.557+0000 7ffaa6c27740 1 Processor -- start 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.557+0000 7ffaa6c27740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ece620 0x55b2c0ed9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.557+0000 7ffaa6c27740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.557+0000 7ffaa6c27740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0ed6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.557+0000 7ffaa6c27740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b2c0edc2b0 con 0x55b2c0ecef40 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.557+0000 7ffaa6c27740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b2c0dce0b0 con 0x55b2c0ece620 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.557+0000 7ffaa6c27740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b2c0edc430 con 0x55b2c0ed9800 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa6c27740 10 client.0 _ll_register_callbacks cb 0x55b2c0dbbd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa14a3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa14a3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58456/0 (socket says 172.21.15.50:58456) 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa14a3640 1 -- 172.21.15.50:0/2967697195 learned_addr learned my addr 172.21.15.50:0/2967697195 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa1ca4640 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ece620 0x55b2c0ed9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa24a5640 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0ed6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa14a3640 1 -- 172.21.15.50:0/2967697195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 msgr2=0x55b2c0ed6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa14a3640 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0ed6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa14a3640 1 -- 172.21.15.50:0/2967697195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ece620 msgr2=0x55b2c0ed9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa14a3640 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ece620 0x55b2c0ed9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa14a3640 1 -- 172.21.15.50:0/2967697195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2c0c12620 con 0x55b2c0ecef40 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa24a5640 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0ed6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa14a3640 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed6420 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0126e0 tx=0x7ffa8c012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c06bb1c4bb37334 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffa857fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:23:09.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffa857fa640 1 -- 172.21.15.50:0/2967697195 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c004050 con 0x55b2c0ecef40 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa6c27740 1 -- 172.21.15.50:0/2967697195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa9404f070 con 0x55b2c0ecef40 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffa857fa640 1 -- 172.21.15.50:0/2967697195 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7ffa8c0041f0 con 0x55b2c0ecef40 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffa857fa640 1 -- 172.21.15.50:0/2967697195 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffa8c0043f0 con 0x55b2c0ecef40 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa6c27740 10 client.16092 fetch_fsmap learned FSMap version 28 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.558+0000 7ffaa6c27740 1 -- 172.21.15.50:0/2967697195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b2c0ee8270 con 0x55b2c0ecef40 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.559+0000 7ffa857fa640 1 -- 172.21.15.50:0/2967697195 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9404f070 con 0x55b2c0ecef40 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.559+0000 7ffa857fa640 1 -- 172.21.15.50:0/2967697195 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7ffa8c003920 con 0x55b2c0ecef40 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.559+0000 7ffaa6c27740 10 client.16092 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.559+0000 7ffaa6c27740 10 client.16092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.559+0000 7ffaa6c27740 20 client.16092 populate_metadata read hostname 'smithi050' 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.559+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b2c0ee97f0 0x55b2c0eebc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.559+0000 7ffaa6c27740 4 client.16092 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.559+0000 7ffaa6c27740 1 -- 172.21.15.50:0/2967697195 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2c0e41390 con 0x55b2c0ee97f0 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.559+0000 7ffaa1ca4640 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b2c0ee97f0 0x55b2c0eebc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.560+0000 7ffaa1ca4640 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b2c0ee97f0 0x55b2c0eebc70 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21399b2a150dfd85 server_cookie=d2a3b6e329ec6f2b in_seq=0 out_seq=0 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.560+0000 7ffa857fa640 10 client.16092 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.560+0000 7ffa857fa640 1 -- 172.21.15.50:0/2967697195 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b2c0e41390 con 0x55b2c0ee97f0 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.560+0000 7ffa857fa640 10 client.16092 handle_command_reply: tid=0 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.560+0000 7ffaa6c27740 1 client.16092 shutdown 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.561+0000 7ffaa6c27740 1 -- 172.21.15.50:0/2967697195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ecef40 msgr2=0x55b2c0ed6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.561+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed6420 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0126e0 tx=0x7ffa8c012ae0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.561+0000 7ffaa6c27740 1 -- 172.21.15.50:0/2967697195 shutdown_connections 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.561+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b2c0ee97f0 0x55b2c0eebc70 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.561+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0ed6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.561+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed6420 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.561+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/2967697195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ece620 0x55b2c0ed9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.561+0000 7ffa857fa640 0 client.16092 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:09.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.561+0000 7ffaa6c27740 1 -- 172.21.15.50:0/2967697195 >> 172.21.15.50:0/2967697195 conn(0x55b2c0ecab30 msgr2=0x55b2c0ed08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:09.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.561+0000 7ffaa6c27740 1 -- 172.21.15.50:0/2967697195 shutdown_connections 2024-04-23T18:23:09.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.561+0000 7ffaa6c27740 1 -- 172.21.15.50:0/2967697195 wait complete. 2024-04-23T18:23:09.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.561+0000 7ffaa6c27740 20 client.16092 trim_cache size 0 max 16384 2024-04-23T18:23:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa6c27740 1 Processor -- start 2024-04-23T18:23:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa6c27740 1 -- start start 2024-04-23T18:23:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa6c27740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ece620 0x55b2c0ee5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa6c27740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa6c27740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ed9800 0x55b2c0ed6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa6c27740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c1b25d0 con 0x55b2c0ed9800 2024-04-23T18:23:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa6c27740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c1afe40 con 0x55b2c0ecef40 2024-04-23T18:23:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa6c27740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c1adcb0 con 0x55b2c0ece620 2024-04-23T18:23:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa24a5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ed9800 0x55b2c0ed6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa24a5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ed9800 0x55b2c0ed6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58460/0 (socket says 172.21.15.50:58460) 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa24a5640 1 -- 172.21.15.50:0/1870869550 learned_addr learned my addr 172.21.15.50:0/1870869550 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa14a3640 1 --2- 172.21.15.50:0/1870869550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa24a5640 1 -- 172.21.15.50:0/1870869550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ece620 msgr2=0x55b2c0ee5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa1ca4640 1 --2- 172.21.15.50:0/1870869550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ece620 0x55b2c0ee5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa24a5640 1 --2- 172.21.15.50:0/1870869550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ece620 0x55b2c0ee5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa24a5640 1 -- 172.21.15.50:0/1870869550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 msgr2=0x55b2c0ed5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa24a5640 1 --2- 172.21.15.50:0/1870869550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa24a5640 1 -- 172.21.15.50:0/1870869550 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2c0ed2030 con 0x55b2c0ed9800 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa14a3640 1 --2- 172.21.15.50:0/1870869550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.576+0000 7ffaa1ca4640 1 --2- 172.21.15.50:0/1870869550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ece620 0x55b2c0ee5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.577+0000 7ffaa24a5640 1 --2- 172.21.15.50:0/1870869550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ed9800 0x55b2c0ed6840 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ffa9402ade0 tx=0x7ffa94036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a0fdea027b7ed9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.577+0000 7ffa877fe640 1 -- 172.21.15.50:0/1870869550 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa940047f0 con 0x55b2c0ed9800 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.577+0000 7ffa877fe640 1 -- 172.21.15.50:0/1870869550 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9404f040 con 0x55b2c0ed9800 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.577+0000 7ffa877fe640 1 -- 172.21.15.50:0/1870869550 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa94004e10 con 0x55b2c0ed9800 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.577+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1870869550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ed9800 msgr2=0x55b2c0ed6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.577+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/1870869550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ed9800 0x55b2c0ed6840 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ffa9402ade0 tx=0x7ffa94036000 comp rx=0 tx=0).stop 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.577+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1870869550 shutdown_connections 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.577+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/1870869550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ed9800 0x55b2c0ed6840 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.577+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/1870869550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0ed5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.577+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/1870869550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ece620 0x55b2c0ee5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.577+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1870869550 >> 172.21.15.50:0/1870869550 conn(0x55b2c0ecab30 msgr2=0x55b2c0ec9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1870869550 shutdown_connections 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1870869550 wait complete. 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa6c27740 1 -- start start 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa6c27740 1 Processor -- start 2024-04-23T18:23:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa6c27740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 0x55b2c0edf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa6c27740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0edf930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa6c27740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0edfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa6c27740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c1ae300 con 0x55b2c0ece620 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa6c27740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c1af770 con 0x55b2c0ecef40 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa6c27740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c1ae9d0 con 0x55b2c0ed9800 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa6c27740 10 client.0 _ll_register_callbacks cb 0x7ffa8c009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa1ca4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 0x55b2c0edf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa1ca4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 0x55b2c0edf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58464/0 (socket says 172.21.15.50:58464) 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.578+0000 7ffaa1ca4640 1 -- 172.21.15.50:0/1720066293 learned_addr learned my addr 172.21.15.50:0/1720066293 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffaa24a5640 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0edfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffaa14a3640 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0edf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffaa1ca4640 1 -- 172.21.15.50:0/1720066293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 msgr2=0x55b2c0edfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffaa1ca4640 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0edfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffaa1ca4640 1 -- 172.21.15.50:0/1720066293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 msgr2=0x55b2c0edf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffaa1ca4640 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0edf930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffaa1ca4640 1 -- 172.21.15.50:0/1720066293 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2c0f4f0f0 con 0x55b2c0ece620 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffaa14a3640 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0edf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffaa1ca4640 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 0x55b2c0edf3f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ffa980cd9d0 tx=0x7ffa98018ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a19f91e1602bfe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffa857fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffa857fa640 1 -- 172.21.15.50:0/1720066293 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9809c270 con 0x55b2c0ece620 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffa857fa640 1 -- 172.21.15.50:0/1720066293 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7ffa980cdbf0 con 0x55b2c0ece620 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1720066293 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa9404f040 con 0x55b2c0ece620 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffa857fa640 1 -- 172.21.15.50:0/1720066293 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffa980a0570 con 0x55b2c0ece620 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffaa6c27740 10 client.16098 fetch_fsmap learned FSMap version 28 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.579+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1720066293 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b2c0f503d0 con 0x55b2c0ece620 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.580+0000 7ffa857fa640 1 -- 172.21.15.50:0/1720066293 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9404f040 con 0x55b2c0ece620 2024-04-23T18:23:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.580+0000 7ffa857fa640 1 -- 172.21.15.50:0/1720066293 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7ffa980a0170 con 0x55b2c0ece620 2024-04-23T18:23:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.580+0000 7ffaa6c27740 10 client.16098 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:23:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.580+0000 7ffaa6c27740 10 client.16098 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.580+0000 7ffaa6c27740 20 client.16098 populate_metadata read hostname 'smithi050' 2024-04-23T18:23:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.580+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b2c0f50ad0 0x55b2c0f52ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.580+0000 7ffaa6c27740 4 client.16098 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.580+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1720066293 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b2c0e3b200 con 0x55b2c0f50ad0 2024-04-23T18:23:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.580+0000 7ffaa14a3640 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b2c0f50ad0 0x55b2c0f52ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.580+0000 7ffaa14a3640 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b2c0f50ad0 0x55b2c0f52ef0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e9c79934b74805 server_cookie=aaaaf652d17a2799 in_seq=0 out_seq=0 2024-04-23T18:23:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.580+0000 7ffa857fa640 10 client.16098 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffa857fa640 1 -- 172.21.15.50:0/1720066293 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b2c0e3b200 con 0x55b2c0f50ad0 2024-04-23T18:23:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffa857fa640 10 client.16098 handle_command_reply: tid=0 2024-04-23T18:23:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffaa6c27740 1 client.16098 shutdown 2024-04-23T18:23:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1720066293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 msgr2=0x55b2c0edf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 0x55b2c0edf3f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ffa980cd9d0 tx=0x7ffa98018ad0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1720066293 shutdown_connections 2024-04-23T18:23:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b2c0f50ad0 0x55b2c0f52ef0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b2c0ed9800 0x55b2c0edfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffa857fa640 0 client.16098 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b2c0ecef40 0x55b2c0edf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffaa6c27740 1 --2- 172.21.15.50:0/1720066293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b2c0ece620 0x55b2c0edf3f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1720066293 >> 172.21.15.50:0/1720066293 conn(0x55b2c0ecab30 msgr2=0x55b2c0ee4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1720066293 shutdown_connections 2024-04-23T18:23:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffaa6c27740 1 -- 172.21.15.50:0/1720066293 wait complete. 2024-04-23T18:23:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.604+0000 7ffaa6c27740 20 client.16098 trim_cache size 0 max 16384 2024-04-23T18:23:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:23:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:23:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:23:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:23:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.605+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1726708096 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ffa880a3dc0 msgr2=0x7ffa880a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.605+0000 7ffaa3f08640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ffa880a3dc0 0x7ffa880a6280 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ffa9404e000 tx=0x7ffa9404c000 comp rx=0 tx=0).stop 2024-04-23T18:23:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.605+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1726708096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa9c1021c0 msgr2=0x7ffa9c074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.605+0000 7ffaa3f08640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa9c1021c0 0x7ffa9c074250 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800d770 tx=0x7ffa9800dc30 comp rx=0 tx=0).stop 2024-04-23T18:23:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.605+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1726708096 shutdown_connections 2024-04-23T18:23:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.605+0000 7ffaa3f08640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ffa880a3dc0 0x7ffa880a6280 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.605+0000 7ffaa3f08640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa9c074cd0 0x7ffa9c1a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.605+0000 7ffaa3f08640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa9c107290 0x7ffa9c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.605+0000 7ffaa3f08640 1 --2- 172.21.15.50:0/1726708096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa9c1021c0 0x7ffa9c074250 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.605+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1726708096 >> 172.21.15.50:0/1726708096 conn(0x7ffa9c0fda20 msgr2=0x7ffa9c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.605+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1726708096 shutdown_connections 2024-04-23T18:23:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:09.605+0000 7ffaa3f08640 1 -- 172.21.15.50:0/1726708096 wait complete. 2024-04-23T18:23:10.204 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.29 MB/sec execute 104 sec latency 6235.699 ms 2024-04-23T18:23:10.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:10 smithi188 ceph-mon[32694]: pgmap v143: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 195 op/s 2024-04-23T18:23:10.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:10 smithi050 ceph-mon[29187]: pgmap v143: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 195 op/s 2024-04-23T18:23:10.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:10 smithi177 ceph-mon[32308]: pgmap v143: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 195 op/s 2024-04-23T18:23:11.204 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.25 MB/sec execute 105 sec latency 7235.822 ms 2024-04-23T18:23:12.204 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.21 MB/sec execute 106 sec latency 8235.947 ms 2024-04-23T18:23:12.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:12 smithi188 ceph-mon[32694]: pgmap v144: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 195 op/s 2024-04-23T18:23:12.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:12 smithi050 ceph-mon[29187]: pgmap v144: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 195 op/s 2024-04-23T18:23:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:12 smithi177 ceph-mon[32308]: pgmap v144: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 195 op/s 2024-04-23T18:23:13.204 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.17 MB/sec execute 107 sec latency 9236.042 ms 2024-04-23T18:23:14.204 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.13 MB/sec execute 108 sec latency 10236.167 ms 2024-04-23T18:23:14.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:14 smithi188 ceph-mon[32694]: pgmap v145: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 208 op/s 2024-04-23T18:23:14.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:14 smithi050 ceph-mon[29187]: pgmap v145: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 208 op/s 2024-04-23T18:23:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:14 smithi177 ceph-mon[32308]: pgmap v145: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 208 op/s 2024-04-23T18:23:15.204 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.09 MB/sec execute 109 sec latency 11236.298 ms 2024-04-23T18:23:16.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.05 MB/sec execute 110 sec latency 12236.411 ms 2024-04-23T18:23:16.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:16 smithi188 ceph-mon[32694]: pgmap v146: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 411 KiB/s wr, 69 op/s 2024-04-23T18:23:16.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:16 smithi050 ceph-mon[29187]: pgmap v146: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 411 KiB/s wr, 69 op/s 2024-04-23T18:23:16.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:16 smithi177 ceph-mon[32308]: pgmap v146: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 411 KiB/s wr, 69 op/s 2024-04-23T18:23:17.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 4.02 MB/sec execute 111 sec latency 13236.503 ms 2024-04-23T18:23:18.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 3.98 MB/sec execute 112 sec latency 14236.568 ms 2024-04-23T18:23:18.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:18 smithi188 ceph-mon[32694]: pgmap v147: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 413 KiB/s wr, 73 op/s 2024-04-23T18:23:18.338 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:18 smithi177 ceph-mon[32308]: pgmap v147: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 413 KiB/s wr, 73 op/s 2024-04-23T18:23:18.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:18 smithi050 ceph-mon[29187]: pgmap v147: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 413 KiB/s wr, 73 op/s 2024-04-23T18:23:19.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277298 3.94 MB/sec execute 113 sec latency 15236.667 ms 2024-04-23T18:23:20.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 277637 3.92 MB/sec execute 114 sec latency 15973.760 ms 2024-04-23T18:23:20.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:20 smithi188 ceph-mon[32694]: pgmap v148: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 KiB/s wr, 16 op/s 2024-04-23T18:23:20.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:20 smithi050 ceph-mon[29187]: pgmap v148: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 KiB/s wr, 16 op/s 2024-04-23T18:23:20.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:20 smithi177 ceph-mon[32308]: pgmap v148: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 KiB/s wr, 16 op/s 2024-04-23T18:23:21.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 279715 3.99 MB/sec execute 115 sec latency 23.865 ms 2024-04-23T18:23:22.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 280311 3.99 MB/sec execute 116 sec latency 825.249 ms 2024-04-23T18:23:22.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:22 smithi188 ceph-mon[32694]: pgmap v149: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 KiB/s wr, 16 op/s 2024-04-23T18:23:22.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:22 smithi050 ceph-mon[29187]: pgmap v149: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 KiB/s wr, 16 op/s 2024-04-23T18:23:22.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:22 smithi177 ceph-mon[32308]: pgmap v149: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 KiB/s wr, 16 op/s 2024-04-23T18:23:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.704+0000 7fee534fa640 1 -- 172.21.15.50:0/2453494738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c100ff0 msgr2=0x7fee4c1013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.704+0000 7fee534fa640 1 --2- 172.21.15.50:0/2453494738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c100ff0 0x7fee4c1013d0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fee48009930 tx=0x7fee4802a4d0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.705+0000 7fee534fa640 1 -- 172.21.15.50:0/2453494738 shutdown_connections 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.705+0000 7fee534fa640 1 --2- 172.21.15.50:0/2453494738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee4c1063f0 0x7fee4c1068a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.705+0000 7fee534fa640 1 --2- 172.21.15.50:0/2453494738 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee4c101910 0x7fee4c105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.705+0000 7fee534fa640 1 --2- 172.21.15.50:0/2453494738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c100ff0 0x7fee4c1013d0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.705+0000 7fee534fa640 1 -- 172.21.15.50:0/2453494738 >> 172.21.15.50:0/2453494738 conn(0x7fee4c0771d0 msgr2=0x7fee4c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.705+0000 7fee534fa640 1 -- 172.21.15.50:0/2453494738 shutdown_connections 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.705+0000 7fee534fa640 1 -- 172.21.15.50:0/2453494738 wait complete. 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.705+0000 7fee534fa640 1 Processor -- start 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.705+0000 7fee534fa640 1 -- start start 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee534fa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee4c100ff0 0x7fee4c19ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee534fa640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee4c101910 0x7fee4c19f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee534fa640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c1063f0 0x7fee4c1a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee534fa640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1187f0 con 0x7fee4c1063f0 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee534fa640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c118670 con 0x7fee4c100ff0 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee534fa640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c118970 con 0x7fee4c101910 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee51a97640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c1063f0 0x7fee4c1a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee51a97640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c1063f0 0x7fee4c1a3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35922/0 (socket says 172.21.15.50:35922) 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee51a97640 1 -- 172.21.15.50:0/3744122005 learned_addr learned my addr 172.21.15.50:0/3744122005 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee51296640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee4c100ff0 0x7fee4c19ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee50a95640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee4c101910 0x7fee4c19f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee51a97640 1 -- 172.21.15.50:0/3744122005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee4c101910 msgr2=0x7fee4c19f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee51a97640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee4c101910 0x7fee4c19f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee51a97640 1 -- 172.21.15.50:0/3744122005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee4c100ff0 msgr2=0x7fee4c19ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee51a97640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee4c100ff0 0x7fee4c19ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee51a97640 1 -- 172.21.15.50:0/3744122005 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4c1a3d60 con 0x7fee4c1063f0 2024-04-23T18:23:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee51296640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee4c100ff0 0x7fee4c19ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:23:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee50a95640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee4c101910 0x7fee4c19f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:23:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee51a97640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c1063f0 0x7fee4c1a3630 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fee44009fd0 tx=0x7fee4400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.706+0000 7fee427fc640 1 -- 172.21.15.50:0/3744122005 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee44017070 con 0x7fee4c1063f0 2024-04-23T18:23:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.707+0000 7fee427fc640 1 -- 172.21.15.50:0/3744122005 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee44003e20 con 0x7fee4c1063f0 2024-04-23T18:23:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.707+0000 7fee534fa640 1 -- 172.21.15.50:0/3744122005 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee4c1a4050 con 0x7fee4c1063f0 2024-04-23T18:23:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.707+0000 7fee427fc640 1 -- 172.21.15.50:0/3744122005 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee44010460 con 0x7fee4c1063f0 2024-04-23T18:23:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.707+0000 7fee534fa640 1 -- 172.21.15.50:0/3744122005 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee4c1ab8e0 con 0x7fee4c1063f0 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.709+0000 7fee427fc640 1 -- 172.21.15.50:0/3744122005 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fee44003970 con 0x7fee4c1063f0 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee56215740 1 Processor -- start 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee56215740 1 -- start start 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee56215740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 0x55df50746a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee56215740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746f40 0x55df507512c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee56215740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50751800 0x55df50753bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee56215740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55df506ab970 con 0x55df50746620 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee56215740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55df506484a0 con 0x55df50751800 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee56215740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55df506a7930 con 0x55df50746f40 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee427fc640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee380a3dc0 0x7fee380a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee50a95640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 0x55df50746a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee50a95640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 0x55df50746a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35934/0 (socket says 172.21.15.50:35934) 2024-04-23T18:23:22.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee50a95640 1 -- 172.21.15.50:0/8964077 learned_addr learned my addr 172.21.15.50:0/8964077 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee51296640 1 --2- 172.21.15.50:0/8964077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746f40 0x55df507512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee51a97640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee380a3dc0 0x7fee380a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.710+0000 7fee50a95640 1 --2- 172.21.15.50:0/8964077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50751800 0x55df50753bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.711+0000 7fee427fc640 1 -- 172.21.15.50:0/3744122005 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fee440d05c0 con 0x7fee4c1063f0 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.711+0000 7fee50a95640 1 -- 172.21.15.50:0/8964077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746f40 msgr2=0x55df507512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.711+0000 7fee50a95640 1 --2- 172.21.15.50:0/8964077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746f40 0x55df507512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.711+0000 7fee50a95640 1 -- 172.21.15.50:0/8964077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50751800 msgr2=0x55df50753bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.711+0000 7fee50a95640 1 --2- 172.21.15.50:0/8964077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50751800 0x55df50753bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.711+0000 7fee50a95640 1 -- 172.21.15.50:0/8964077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df503be210 con 0x55df50746620 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.711+0000 7fee50a95640 1 --2- 172.21.15.50:0/8964077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 0x55df50746a00 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fee3c005fd0 tx=0x7fee3c009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6dd839436761ea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.711+0000 7fee51a97640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee380a3dc0 0x7fee380a6280 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fee4409c500 tx=0x7fee44087040 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.711+0000 7fee377fe640 1 -- 172.21.15.50:0/8964077 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c0104d0 con 0x55df50746620 2024-04-23T18:23:22.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.711+0000 7fee377fe640 1 -- 172.21.15.50:0/8964077 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee3c01b070 con 0x55df50746620 2024-04-23T18:23:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.711+0000 7fee377fe640 1 -- 172.21.15.50:0/8964077 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c013780 con 0x55df50746620 2024-04-23T18:23:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.712+0000 7fee56215740 1 -- 172.21.15.50:0/8964077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 msgr2=0x55df50746a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.712+0000 7fee56215740 1 --2- 172.21.15.50:0/8964077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 0x55df50746a00 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fee3c005fd0 tx=0x7fee3c009650 comp rx=0 tx=0).stop 2024-04-23T18:23:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.712+0000 7fee56215740 1 -- 172.21.15.50:0/8964077 shutdown_connections 2024-04-23T18:23:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.712+0000 7fee56215740 1 --2- 172.21.15.50:0/8964077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50751800 0x55df50753bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.712+0000 7fee56215740 1 --2- 172.21.15.50:0/8964077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746f40 0x55df507512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.712+0000 7fee56215740 1 --2- 172.21.15.50:0/8964077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 0x55df50746a00 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.712+0000 7fee56215740 1 -- 172.21.15.50:0/8964077 >> 172.21.15.50:0/8964077 conn(0x55df50742b30 msgr2=0x55df50742f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.712+0000 7fee56215740 1 -- 172.21.15.50:0/8964077 shutdown_connections 2024-04-23T18:23:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.712+0000 7fee56215740 1 -- 172.21.15.50:0/8964077 wait complete. 2024-04-23T18:23:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.712+0000 7fee56215740 1 -- start start 2024-04-23T18:23:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.712+0000 7fee56215740 1 Processor -- start 2024-04-23T18:23:22.711 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee56215740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 0x55df507510b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.711 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee56215740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746f40 0x55df5074e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.711 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee56215740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50751800 0x55df5074e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.711 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee56215740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55df507543d0 con 0x55df50746620 2024-04-23T18:23:22.711 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee56215740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55df506460b0 con 0x55df50751800 2024-04-23T18:23:22.711 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee56215740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55df50754550 con 0x55df50746f40 2024-04-23T18:23:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee56215740 10 client.0 _ll_register_callbacks cb 0x55df50633d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee50a95640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 0x55df507510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee50a95640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 0x55df507510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35950/0 (socket says 172.21.15.50:35950) 2024-04-23T18:23:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee50a95640 1 -- 172.21.15.50:0/1575636394 learned_addr learned my addr 172.21.15.50:0/1575636394 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee51296640 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746f40 0x55df5074e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee50a95640 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50751800 0x55df5074e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee50a95640 1 -- 172.21.15.50:0/1575636394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746f40 msgr2=0x55df5074e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee50a95640 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746f40 0x55df5074e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee50a95640 1 -- 172.21.15.50:0/1575636394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50751800 msgr2=0x55df5074e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee50a95640 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50751800 0x55df5074e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee50a95640 1 -- 172.21.15.50:0/1575636394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df5048a620 con 0x55df50746620 2024-04-23T18:23:22.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.713+0000 7fee51296640 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746f40 0x55df5074e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee50a95640 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 0x55df507510b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fee3c013460 tx=0x7fee3c0095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b89856fd50a46de server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee34ff9640 1 -- 172.21.15.50:0/1575636394 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c010b80 con 0x55df50746620 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee34ff9640 1 -- 172.21.15.50:0/1575636394 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fee3c010d20 con 0x55df50746620 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee56215740 1 -- 172.21.15.50:0/1575636394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee3c01b070 con 0x55df50746620 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee34ff9640 1 -- 172.21.15.50:0/1575636394 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fee3c005020 con 0x55df50746620 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee56215740 10 client.16110 fetch_fsmap learned FSMap version 28 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee56215740 1 -- 172.21.15.50:0/1575636394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55df50760280 con 0x55df50746620 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee34ff9640 1 -- 172.21.15.50:0/1575636394 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee3c01b070 con 0x55df50746620 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee34ff9640 1 -- 172.21.15.50:0/1575636394 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fee3c02d030 con 0x55df50746620 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee56215740 10 client.16110 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee56215740 10 client.16110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee56215740 20 client.16110 populate_metadata read hostname 'smithi050' 2024-04-23T18:23:22.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee56215740 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55df50761740 0x55df50763b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee56215740 4 client.16110 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.714+0000 7fee56215740 1 -- 172.21.15.50:0/1575636394 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df506b9390 con 0x55df50761740 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.715+0000 7fee51296640 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55df50761740 0x55df50763b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.715+0000 7fee34ff9640 10 client.16110 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.715+0000 7fee51296640 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55df50761740 0x55df50763b30 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6d2a801e4e1036c server_cookie=51378cf4e73177ba in_seq=0 out_seq=0 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee34ff9640 1 -- 172.21.15.50:0/1575636394 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55df506b9390 con 0x55df50761740 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee34ff9640 10 client.16110 handle_command_reply: tid=0 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee56215740 1 client.16110 shutdown 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee56215740 1 -- 172.21.15.50:0/1575636394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 msgr2=0x55df507510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee56215740 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 0x55df507510b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fee3c013460 tx=0x7fee3c0095b0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee56215740 1 -- 172.21.15.50:0/1575636394 shutdown_connections 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee56215740 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55df50761740 0x55df50763b30 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee56215740 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50751800 0x55df5074e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee56215740 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746f40 0x55df5074e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee56215740 1 --2- 172.21.15.50:0/1575636394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50746620 0x55df507510b0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee34ff9640 0 client.16110 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:22.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee56215740 1 -- 172.21.15.50:0/1575636394 >> 172.21.15.50:0/1575636394 conn(0x55df50742b30 msgr2=0x55df50748860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:22.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee56215740 1 -- 172.21.15.50:0/1575636394 shutdown_connections 2024-04-23T18:23:22.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee56215740 1 -- 172.21.15.50:0/1575636394 wait complete. 2024-04-23T18:23:22.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.716+0000 7fee56215740 20 client.16110 trim_cache size 0 max 16384 2024-04-23T18:23:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.731+0000 7fee56215740 1 Processor -- start 2024-04-23T18:23:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.731+0000 7fee56215740 1 -- start start 2024-04-23T18:23:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee56215740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746620 0x55df5075d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee56215740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50746f40 0x55df50741b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee56215740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 0x55df50750be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee56215740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1b6ab0 con 0x55df50751800 2024-04-23T18:23:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee56215740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1b4320 con 0x55df50746f40 2024-04-23T18:23:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee56215740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1b2170 con 0x55df50746620 2024-04-23T18:23:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee50a95640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 0x55df50750be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee50a95640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 0x55df50750be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35964/0 (socket says 172.21.15.50:35964) 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee50a95640 1 -- 172.21.15.50:0/1753442923 learned_addr learned my addr 172.21.15.50:0/1753442923 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee51296640 1 --2- 172.21.15.50:0/1753442923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50746f40 0x55df50741b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee50a95640 1 --2- 172.21.15.50:0/1753442923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746620 0x55df5075d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee50a95640 1 -- 172.21.15.50:0/1753442923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746620 msgr2=0x55df5075d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee50a95640 1 --2- 172.21.15.50:0/1753442923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746620 0x55df5075d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee50a95640 1 -- 172.21.15.50:0/1753442923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50746f40 msgr2=0x55df50741b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee50a95640 1 --2- 172.21.15.50:0/1753442923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50746f40 0x55df50741b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee50a95640 1 -- 172.21.15.50:0/1753442923 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df50743b30 con 0x55df50751800 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee51296640 1 --2- 172.21.15.50:0/1753442923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50746f40 0x55df50741b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee50a95640 1 --2- 172.21.15.50:0/1753442923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 0x55df50750be0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fee3c0155a0 tx=0x7fee3c0104d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bca5125f0033b53 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee36ffd640 1 -- 172.21.15.50:0/1753442923 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c004680 con 0x55df50751800 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee36ffd640 1 -- 172.21.15.50:0/1753442923 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee3c01b040 con 0x55df50751800 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.732+0000 7fee36ffd640 1 -- 172.21.15.50:0/1753442923 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c010980 con 0x55df50751800 2024-04-23T18:23:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.733+0000 7fee56215740 1 -- 172.21.15.50:0/1753442923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 msgr2=0x55df50750be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.733+0000 7fee56215740 1 --2- 172.21.15.50:0/1753442923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 0x55df50750be0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fee3c0155a0 tx=0x7fee3c0104d0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.733+0000 7fee56215740 1 -- 172.21.15.50:0/1753442923 shutdown_connections 2024-04-23T18:23:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.733+0000 7fee56215740 1 --2- 172.21.15.50:0/1753442923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 0x55df50750be0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.733+0000 7fee56215740 1 --2- 172.21.15.50:0/1753442923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50746f40 0x55df50741b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.733+0000 7fee56215740 1 --2- 172.21.15.50:0/1753442923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746620 0x55df5075d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.733+0000 7fee56215740 1 -- 172.21.15.50:0/1753442923 >> 172.21.15.50:0/1753442923 conn(0x55df50742b30 msgr2=0x55df50741420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.733+0000 7fee56215740 1 -- 172.21.15.50:0/1753442923 shutdown_connections 2024-04-23T18:23:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.733+0000 7fee56215740 1 -- 172.21.15.50:0/1753442923 wait complete. 2024-04-23T18:23:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.733+0000 7fee56215740 1 -- start start 2024-04-23T18:23:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.733+0000 7fee56215740 1 Processor -- start 2024-04-23T18:23:22.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee56215740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746620 0x55df507523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee56215740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50746f40 0x55df50752910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee56215740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 0x55df50758780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee56215740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1b27e0 con 0x55df50751800 2024-04-23T18:23:22.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee56215740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1b3c50 con 0x55df50746f40 2024-04-23T18:23:22.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee56215740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1b2eb0 con 0x55df50746620 2024-04-23T18:23:22.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee56215740 10 client.0 _ll_register_callbacks cb 0x7fee3c015350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:22.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee50a95640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 0x55df50758780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee50a95640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 0x55df50758780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35976/0 (socket says 172.21.15.50:35976) 2024-04-23T18:23:22.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee50a95640 1 -- 172.21.15.50:0/1062528916 learned_addr learned my addr 172.21.15.50:0/1062528916 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:22.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee50a95640 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746620 0x55df507523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee51296640 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50746f40 0x55df50752910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee50a95640 1 -- 172.21.15.50:0/1062528916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746620 msgr2=0x55df507523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:22.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee50a95640 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746620 0x55df507523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee50a95640 1 -- 172.21.15.50:0/1062528916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50746f40 msgr2=0x55df50752910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee50a95640 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50746f40 0x55df50752910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee50a95640 1 -- 172.21.15.50:0/1062528916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df50758d40 con 0x55df50751800 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.734+0000 7fee51296640 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50746f40 0x55df50752910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee50a95640 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 0x55df50758780 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fee3c001eb0 tx=0x7fee3c004c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b3c8010f53c002 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee34ff9640 1 -- 172.21.15.50:0/1062528916 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c0150f0 con 0x55df50751800 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee34ff9640 1 -- 172.21.15.50:0/1062528916 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fee3c004960 con 0x55df50751800 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee56215740 1 -- 172.21.15.50:0/1062528916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee3c01b040 con 0x55df50751800 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee34ff9640 1 -- 172.21.15.50:0/1062528916 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fee3c0053d0 con 0x55df50751800 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee56215740 10 client.16116 fetch_fsmap learned FSMap version 28 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee56215740 1 -- 172.21.15.50:0/1062528916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55df50759ff0 con 0x55df50751800 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee34ff9640 1 -- 172.21.15.50:0/1062528916 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee3c01b040 con 0x55df50751800 2024-04-23T18:23:22.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee34ff9640 1 -- 172.21.15.50:0/1062528916 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fee3c02d030 con 0x55df50751800 2024-04-23T18:23:22.734 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee56215740 10 client.16116 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:23:22.734 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee56215740 10 client.16116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:22.734 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee56215740 20 client.16116 populate_metadata read hostname 'smithi050' 2024-04-23T18:23:22.734 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee56215740 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55df507c7e70 0x55df507ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:22.734 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee56215740 4 client.16116 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:22.734 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.735+0000 7fee56215740 1 -- 172.21.15.50:0/1062528916 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55df506b3200 con 0x55df507c7e70 2024-04-23T18:23:22.734 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.736+0000 7fee51296640 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55df507c7e70 0x55df507ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:22.734 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.736+0000 7fee51296640 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55df507c7e70 0x55df507ca290 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4e0ae6d2236e8d2 server_cookie=8cbba62c3448e89e in_seq=0 out_seq=0 2024-04-23T18:23:22.734 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.736+0000 7fee34ff9640 10 client.16116 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:22.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee34ff9640 1 -- 172.21.15.50:0/1062528916 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55df506b3200 con 0x55df507c7e70 2024-04-23T18:23:22.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee34ff9640 10 client.16116 handle_command_reply: tid=0 2024-04-23T18:23:22.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee56215740 1 client.16116 shutdown 2024-04-23T18:23:22.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee56215740 1 -- 172.21.15.50:0/1062528916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 msgr2=0x55df50758780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee56215740 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 0x55df50758780 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fee3c001eb0 tx=0x7fee3c004c50 comp rx=0 tx=0).stop 2024-04-23T18:23:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee56215740 1 -- 172.21.15.50:0/1062528916 shutdown_connections 2024-04-23T18:23:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee56215740 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55df507c7e70 0x55df507ca290 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee56215740 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55df50751800 0x55df50758780 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee56215740 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df50746f40 0x55df50752910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee34ff9640 0 client.16116 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee56215740 1 --2- 172.21.15.50:0/1062528916 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55df50746620 0x55df507523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee56215740 1 -- 172.21.15.50:0/1062528916 >> 172.21.15.50:0/1062528916 conn(0x55df50742b30 msgr2=0x55df50741cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee56215740 1 -- 172.21.15.50:0/1062528916 shutdown_connections 2024-04-23T18:23:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.772+0000 7fee56215740 1 -- 172.21.15.50:0/1062528916 wait complete. 2024-04-23T18:23:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee56215740 20 client.16116 trim_cache size 0 max 16384 2024-04-23T18:23:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:23:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:23:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:23:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:23:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee534fa640 1 -- 172.21.15.50:0/3744122005 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee380a3dc0 msgr2=0x7fee380a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee534fa640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee380a3dc0 0x7fee380a6280 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fee4409c500 tx=0x7fee44087040 comp rx=0 tx=0).stop 2024-04-23T18:23:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee534fa640 1 -- 172.21.15.50:0/3744122005 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c1063f0 msgr2=0x7fee4c1a3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee534fa640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c1063f0 0x7fee4c1a3630 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fee44009fd0 tx=0x7fee4400b260 comp rx=0 tx=0).stop 2024-04-23T18:23:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee534fa640 1 -- 172.21.15.50:0/3744122005 shutdown_connections 2024-04-23T18:23:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee534fa640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee380a3dc0 0x7fee380a6280 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee534fa640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c1063f0 0x7fee4c1a3630 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee534fa640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee4c101910 0x7fee4c19f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee534fa640 1 --2- 172.21.15.50:0/3744122005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee4c100ff0 0x7fee4c19ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee534fa640 1 -- 172.21.15.50:0/3744122005 >> 172.21.15.50:0/3744122005 conn(0x7fee4c0771d0 msgr2=0x7fee4c103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee534fa640 1 -- 172.21.15.50:0/3744122005 shutdown_connections 2024-04-23T18:23:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:22.773+0000 7fee534fa640 1 -- 172.21.15.50:0/3744122005 wait complete. 2024-04-23T18:23:23.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 280378 3.96 MB/sec execute 117 sec latency 1354.172 ms 2024-04-23T18:23:24.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 280397 3.93 MB/sec execute 118 sec latency 1011.990 ms 2024-04-23T18:23:24.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:24 smithi188 ceph-mon[32694]: pgmap v150: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 2.8 MiB/s wr, 139 op/s 2024-04-23T18:23:24.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:24 smithi050 ceph-mon[29187]: pgmap v150: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 2.8 MiB/s wr, 139 op/s 2024-04-23T18:23:24.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:24 smithi177 ceph-mon[32308]: pgmap v150: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 2.8 MiB/s wr, 139 op/s 2024-04-23T18:23:25.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 280681 3.91 MB/sec execute 119 sec latency 1106.675 ms 2024-04-23T18:23:26.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 282114 3.92 MB/sec execute 120 sec latency 17.162 ms 2024-04-23T18:23:26.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:26 smithi188 ceph-mon[32694]: pgmap v151: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 2.8 MiB/s wr, 126 op/s 2024-04-23T18:23:26.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:26 smithi050 ceph-mon[29187]: pgmap v151: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 2.8 MiB/s wr, 126 op/s 2024-04-23T18:23:26.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:26 smithi177 ceph-mon[32308]: pgmap v151: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 2.8 MiB/s wr, 126 op/s 2024-04-23T18:23:27.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 284249 4.00 MB/sec execute 121 sec latency 13.876 ms 2024-04-23T18:23:28.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 286221 4.03 MB/sec execute 122 sec latency 17.473 ms 2024-04-23T18:23:28.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:28 smithi188 ceph-mon[32694]: pgmap v152: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 162 op/s 2024-04-23T18:23:28.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:28 smithi050 ceph-mon[29187]: pgmap v152: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 162 op/s 2024-04-23T18:23:28.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:28 smithi177 ceph-mon[32308]: pgmap v152: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 162 op/s 2024-04-23T18:23:29.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 288216 4.10 MB/sec execute 123 sec latency 15.436 ms 2024-04-23T18:23:30.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 290507 4.16 MB/sec execute 124 sec latency 14.565 ms 2024-04-23T18:23:30.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:30 smithi188 ceph-mon[32694]: pgmap v153: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 158 op/s 2024-04-23T18:23:30.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:30 smithi050 ceph-mon[29187]: pgmap v153: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 158 op/s 2024-04-23T18:23:30.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:30 smithi177 ceph-mon[32308]: pgmap v153: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 158 op/s 2024-04-23T18:23:31.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 292311 4.22 MB/sec execute 125 sec latency 14.859 ms 2024-04-23T18:23:32.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 294578 4.29 MB/sec execute 126 sec latency 12.667 ms 2024-04-23T18:23:32.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:32 smithi188 ceph-mon[32694]: pgmap v154: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 158 op/s 2024-04-23T18:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:32 smithi050 ceph-mon[29187]: pgmap v154: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 158 op/s 2024-04-23T18:23:32.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:32 smithi177 ceph-mon[32308]: pgmap v154: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 158 op/s 2024-04-23T18:23:33.205 INFO:tasks.workunit.client.0.smithi050.stdout: 1 296093 4.30 MB/sec execute 127 sec latency 20.157 ms 2024-04-23T18:23:34.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 298337 4.38 MB/sec execute 128 sec latency 16.643 ms 2024-04-23T18:23:34.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:34 smithi188 ceph-mon[32694]: pgmap v155: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 258 op/s 2024-04-23T18:23:34.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:34 smithi050 ceph-mon[29187]: pgmap v155: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 258 op/s 2024-04-23T18:23:34.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:34 smithi177 ceph-mon[32308]: pgmap v155: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 258 op/s 2024-04-23T18:23:35.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 300161 4.39 MB/sec execute 129 sec latency 20.446 ms 2024-04-23T18:23:35.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.864+0000 7f380eac2640 1 -- 172.21.15.50:0/70388058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3808106170 msgr2=0x7f380810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:35.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.864+0000 7f380eac2640 1 --2- 172.21.15.50:0/70388058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3808106170 0x7f380810dd40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3800009ca0 tx=0x7f380002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.864+0000 7f380eac2640 1 -- 172.21.15.50:0/70388058 shutdown_connections 2024-04-23T18:23:35.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.864+0000 7f380eac2640 1 --2- 172.21.15.50:0/70388058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3808106170 0x7f380810dd40 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.864+0000 7f380eac2640 1 --2- 172.21.15.50:0/70388058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3808101aa0 0x7f3808105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.864+0000 7f380eac2640 1 --2- 172.21.15.50:0/70388058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38081010f0 0x7f38081014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.864+0000 7f380eac2640 1 -- 172.21.15.50:0/70388058 >> 172.21.15.50:0/70388058 conn(0x7f3808077fd0 msgr2=0x7f38080783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.864+0000 7f380eac2640 1 -- 172.21.15.50:0/70388058 shutdown_connections 2024-04-23T18:23:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.864+0000 7f380eac2640 1 -- 172.21.15.50:0/70388058 wait complete. 2024-04-23T18:23:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f380eac2640 1 Processor -- start 2024-04-23T18:23:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f380eac2640 1 -- start start 2024-04-23T18:23:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f380eac2640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38081010f0 0x7f38081a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f380eac2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3808106170 0x7f38081a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f380eac2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380819a700 0x7f380819ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f380eac2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3808111130 con 0x7f380819a700 2024-04-23T18:23:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f380eac2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3808110fb0 con 0x7f3808106170 2024-04-23T18:23:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f380eac2640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f38081112b0 con 0x7f38081010f0 2024-04-23T18:23:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f380d05f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380819a700 0x7f380819ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f380c85e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38081010f0 0x7f38081a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f3807fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3808106170 0x7f38081a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f380c85e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38081010f0 0x7f38081a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:35662/0 (socket says 172.21.15.50:35662) 2024-04-23T18:23:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.865+0000 7f380c85e640 1 -- 172.21.15.50:0/3051367039 learned_addr learned my addr 172.21.15.50:0/3051367039 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f380d05f640 1 -- 172.21.15.50:0/3051367039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38081010f0 msgr2=0x7f38081a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f380d05f640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38081010f0 0x7f38081a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f380d05f640 1 -- 172.21.15.50:0/3051367039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3808106170 msgr2=0x7f38081a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f380d05f640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3808106170 0x7f38081a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f380d05f640 1 -- 172.21.15.50:0/3051367039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f380819b300 con 0x7f380819a700 2024-04-23T18:23:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f3807fff640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3808106170 0x7f38081a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f380c85e640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38081010f0 0x7f38081a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f380d05f640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380819a700 0x7f380819ab80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3800005e30 tx=0x7f38000317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f3805ffb640 1 -- 172.21.15.50:0/3051367039 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800003b70 con 0x7f380819a700 2024-04-23T18:23:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f3805ffb640 1 -- 172.21.15.50:0/3051367039 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3800003d10 con 0x7f380819a700 2024-04-23T18:23:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f3805ffb640 1 -- 172.21.15.50:0/3051367039 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380002f970 con 0x7f380819a700 2024-04-23T18:23:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f380eac2640 1 -- 172.21.15.50:0/3051367039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f380819b590 con 0x7f380819a700 2024-04-23T18:23:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.866+0000 7f380eac2640 1 -- 172.21.15.50:0/3051367039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38081a76e0 con 0x7f380819a700 2024-04-23T18:23:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.868+0000 7f3805ffb640 1 -- 172.21.15.50:0/3051367039 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f380002fb10 con 0x7f380819a700 2024-04-23T18:23:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f38117d5740 1 Processor -- start 2024-04-23T18:23:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f38117d5740 1 -- start start 2024-04-23T18:23:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f38117d5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851c620 0x561ee851ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f38117d5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851cf40 0x561ee85272c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f3805ffb640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f37f80a3dc0 0x7f37f80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f38117d5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 0x561ee8529bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f38117d5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561ee8481970 con 0x561ee8527800 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f38117d5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561ee841e4a0 con 0x561ee851cf40 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f38117d5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561ee847d930 con 0x561ee851c620 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f3807fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851c620 0x561ee851ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f380d05f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 0x561ee8529bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f380c85e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851cf40 0x561ee85272c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f3807fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851c620 0x561ee851ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:35678/0 (socket says 172.21.15.50:35678) 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f380d05f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 0x561ee8529bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49628/0 (socket says 172.21.15.50:49628) 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f3807fff640 1 -- 172.21.15.50:0/2257831263 learned_addr learned my addr 172.21.15.50:0/2257831263 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f3807fff640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f37f80a3dc0 0x7f37f80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.869+0000 7f3805ffb640 1 -- 172.21.15.50:0/3051367039 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f38000f1920 con 0x7f380819a700 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.870+0000 7f380c85e640 1 -- 172.21.15.50:0/2257831263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851c620 msgr2=0x561ee851ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.870+0000 7f380c85e640 1 --2- 172.21.15.50:0/2257831263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851c620 0x561ee851ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.870+0000 7f380c85e640 1 -- 172.21.15.50:0/2257831263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 msgr2=0x561ee8529bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.870+0000 7f380c85e640 1 --2- 172.21.15.50:0/2257831263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 0x561ee8529bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.870+0000 7f380c85e640 1 -- 172.21.15.50:0/2257831263 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ee8194210 con 0x561ee851cf40 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.870+0000 7f3807fff640 1 --2- 172.21.15.50:0/2257831263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851c620 0x561ee851ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.870+0000 7f3807fff640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f37f80a3dc0 0x7f37f80a6280 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f37f40058c0 tx=0x7f37f4005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.870+0000 7f380c85e640 1 --2- 172.21.15.50:0/2257831263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851cf40 0x561ee85272c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f37fc011510 tx=0x7f37fc0119d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37a55a1db2ea230d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.870+0000 7f37f2ffd640 1 -- 172.21.15.50:0/2257831263 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc005020 con 0x561ee851cf40 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.870+0000 7f37f2ffd640 1 -- 172.21.15.50:0/2257831263 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37fc0047a0 con 0x561ee851cf40 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.870+0000 7f37f2ffd640 1 -- 172.21.15.50:0/2257831263 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc00f8e0 con 0x561ee851cf40 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.871+0000 7f38117d5740 1 -- 172.21.15.50:0/2257831263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851cf40 msgr2=0x561ee85272c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.871+0000 7f38117d5740 1 --2- 172.21.15.50:0/2257831263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851cf40 0x561ee85272c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f37fc011510 tx=0x7f37fc0119d0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.871+0000 7f38117d5740 1 -- 172.21.15.50:0/2257831263 shutdown_connections 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.871+0000 7f38117d5740 1 --2- 172.21.15.50:0/2257831263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 0x561ee8529bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.871+0000 7f38117d5740 1 --2- 172.21.15.50:0/2257831263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851cf40 0x561ee85272c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.871+0000 7f38117d5740 1 --2- 172.21.15.50:0/2257831263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851c620 0x561ee851ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.871+0000 7f38117d5740 1 -- 172.21.15.50:0/2257831263 >> 172.21.15.50:0/2257831263 conn(0x561ee8518b30 msgr2=0x561ee8518f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.871+0000 7f38117d5740 1 -- 172.21.15.50:0/2257831263 shutdown_connections 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.871+0000 7f38117d5740 1 -- 172.21.15.50:0/2257831263 wait complete. 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.871+0000 7f38117d5740 1 -- start start 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.871+0000 7f38117d5740 1 Processor -- start 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f38117d5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851c620 0x561ee85270e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f38117d5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851cf40 0x561ee85243d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f38117d5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee8527800 0x561ee8524910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f38117d5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561ee852a430 con 0x561ee851cf40 2024-04-23T18:23:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f38117d5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561ee841c0b0 con 0x561ee8527800 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f38117d5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561ee852a5b0 con 0x561ee851c620 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f38117d5740 10 client.0 _ll_register_callbacks cb 0x561ee8409d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f3807fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851cf40 0x561ee85243d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f3807fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851cf40 0x561ee85243d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49630/0 (socket says 172.21.15.50:49630) 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f3807fff640 1 -- 172.21.15.50:0/2974437901 learned_addr learned my addr 172.21.15.50:0/2974437901 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f380d05f640 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee8527800 0x561ee8524910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f380c85e640 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851c620 0x561ee85270e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f3807fff640 1 -- 172.21.15.50:0/2974437901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851c620 msgr2=0x561ee85270e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f3807fff640 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851c620 0x561ee85270e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f3807fff640 1 -- 172.21.15.50:0/2974437901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee8527800 msgr2=0x561ee8524910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f3807fff640 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee8527800 0x561ee8524910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f3807fff640 1 -- 172.21.15.50:0/2974437901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ee8260620 con 0x561ee851cf40 2024-04-23T18:23:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f3807fff640 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851cf40 0x561ee85243d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f37f4009ee0 tx=0x7f37f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83cc6893329bc63a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f37e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.872+0000 7f37e7fff640 1 -- 172.21.15.50:0/2974437901 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4003e00 con 0x561ee851cf40 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f37e7fff640 1 -- 172.21.15.50:0/2974437901 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f37f4005020 con 0x561ee851cf40 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f38117d5740 1 -- 172.21.15.50:0/2974437901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37fc0047a0 con 0x561ee851cf40 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f37e7fff640 1 -- 172.21.15.50:0/2974437901 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f37f40044d0 con 0x561ee851cf40 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f38117d5740 10 client.16128 fetch_fsmap learned FSMap version 28 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f38117d5740 1 -- 172.21.15.50:0/2974437901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561ee8536330 con 0x561ee851cf40 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f37e7fff640 1 -- 172.21.15.50:0/2974437901 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37fc0047a0 con 0x561ee851cf40 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f37e7fff640 1 -- 172.21.15.50:0/2974437901 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f37f4028070 con 0x561ee851cf40 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f38117d5740 10 client.16128 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f38117d5740 10 client.16128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f38117d5740 20 client.16128 populate_metadata read hostname 'smithi050' 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f38117d5740 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561ee85378b0 0x561ee8539ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f38117d5740 4 client.16128 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f38117d5740 1 -- 172.21.15.50:0/2974437901 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ee848f390 con 0x561ee85378b0 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.873+0000 7f380c85e640 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561ee85378b0 0x561ee8539ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.874+0000 7f37e7fff640 10 client.16128 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.874+0000 7f380c85e640 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561ee85378b0 0x561ee8539ce0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54d5f4c9021cfab8 server_cookie=9328327d1b76f19f in_seq=0 out_seq=0 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f37e7fff640 1 -- 172.21.15.50:0/2974437901 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ee848f390 con 0x561ee85378b0 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f37e7fff640 10 client.16128 handle_command_reply: tid=0 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f38117d5740 1 client.16128 shutdown 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f38117d5740 1 -- 172.21.15.50:0/2974437901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851cf40 msgr2=0x561ee85243d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f38117d5740 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851cf40 0x561ee85243d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f37f4009ee0 tx=0x7f37f4004730 comp rx=0 tx=0).stop 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f38117d5740 1 -- 172.21.15.50:0/2974437901 shutdown_connections 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f38117d5740 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561ee85378b0 0x561ee8539ce0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f38117d5740 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee8527800 0x561ee8524910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f38117d5740 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851cf40 0x561ee85243d0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f38117d5740 1 --2- 172.21.15.50:0/2974437901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851c620 0x561ee85270e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f37e7fff640 0 client.16128 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:35.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f38117d5740 1 -- 172.21.15.50:0/2974437901 >> 172.21.15.50:0/2974437901 conn(0x561ee8518b30 msgr2=0x561ee851e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:35.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.875+0000 7f38117d5740 1 -- 172.21.15.50:0/2974437901 shutdown_connections 2024-04-23T18:23:35.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.876+0000 7f38117d5740 1 -- 172.21.15.50:0/2974437901 wait complete. 2024-04-23T18:23:35.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.876+0000 7f38117d5740 20 client.16128 trim_cache size 0 max 16384 2024-04-23T18:23:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f38117d5740 1 Processor -- start 2024-04-23T18:23:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f38117d5740 1 -- start start 2024-04-23T18:23:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f38117d5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851c620 0x561ee85332f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f38117d5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851cf40 0x561ee8517b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f38117d5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee8527800 0x561ee853f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f38117d5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f38081b2600 con 0x561ee851c620 2024-04-23T18:23:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f38117d5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f38081afe70 con 0x561ee851cf40 2024-04-23T18:23:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f38117d5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f38081adcc0 con 0x561ee8527800 2024-04-23T18:23:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f380c85e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851c620 0x561ee85332f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f380c85e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851c620 0x561ee85332f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49646/0 (socket says 172.21.15.50:49646) 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f380c85e640 1 -- 172.21.15.50:0/653450961 learned_addr learned my addr 172.21.15.50:0/653450961 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f3807fff640 1 --2- 172.21.15.50:0/653450961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851cf40 0x561ee8517b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f380d05f640 1 --2- 172.21.15.50:0/653450961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee8527800 0x561ee853f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f380c85e640 1 -- 172.21.15.50:0/653450961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee8527800 msgr2=0x561ee853f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f380c85e640 1 --2- 172.21.15.50:0/653450961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee8527800 0x561ee853f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f380c85e640 1 -- 172.21.15.50:0/653450961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851cf40 msgr2=0x561ee8517b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f380c85e640 1 --2- 172.21.15.50:0/653450961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851cf40 0x561ee8517b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f380c85e640 1 -- 172.21.15.50:0/653450961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ee8522960 con 0x561ee851c620 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f3807fff640 1 --2- 172.21.15.50:0/653450961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851cf40 0x561ee8517b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.891+0000 7f380c85e640 1 --2- 172.21.15.50:0/653450961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851c620 0x561ee85332f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f37fc004a90 tx=0x7f37fc012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571292472c3a3f15 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.892+0000 7f37f27fc640 1 -- 172.21.15.50:0/653450961 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc0047a0 con 0x561ee851c620 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.892+0000 7f37f27fc640 1 -- 172.21.15.50:0/653450961 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37fc004bc0 con 0x561ee851c620 2024-04-23T18:23:35.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.892+0000 7f37f27fc640 1 -- 172.21.15.50:0/653450961 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc004f80 con 0x561ee851c620 2024-04-23T18:23:35.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.892+0000 7f38117d5740 1 -- 172.21.15.50:0/653450961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851c620 msgr2=0x561ee85332f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:35.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.892+0000 7f38117d5740 1 --2- 172.21.15.50:0/653450961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851c620 0x561ee85332f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f37fc004a90 tx=0x7f37fc012f90 comp rx=0 tx=0).stop 2024-04-23T18:23:35.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.892+0000 7f38117d5740 1 -- 172.21.15.50:0/653450961 shutdown_connections 2024-04-23T18:23:35.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.892+0000 7f38117d5740 1 --2- 172.21.15.50:0/653450961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee8527800 0x561ee853f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.892+0000 7f38117d5740 1 --2- 172.21.15.50:0/653450961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851cf40 0x561ee8517b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.892+0000 7f38117d5740 1 --2- 172.21.15.50:0/653450961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee851c620 0x561ee85332f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.892+0000 7f38117d5740 1 -- 172.21.15.50:0/653450961 >> 172.21.15.50:0/653450961 conn(0x561ee8518b30 msgr2=0x561ee8517420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:35.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.892+0000 7f38117d5740 1 -- 172.21.15.50:0/653450961 shutdown_connections 2024-04-23T18:23:35.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.892+0000 7f38117d5740 1 -- 172.21.15.50:0/653450961 wait complete. 2024-04-23T18:23:35.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f38117d5740 1 -- start start 2024-04-23T18:23:35.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f38117d5740 1 Processor -- start 2024-04-23T18:23:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f38117d5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851c620 0x561ee852fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f38117d5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851cf40 0x561ee852d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f38117d5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 0x561ee852d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f38117d5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f38081ae330 con 0x561ee8527800 2024-04-23T18:23:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f38117d5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f38081af7a0 con 0x561ee851c620 2024-04-23T18:23:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f38117d5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f38081aea00 con 0x561ee851cf40 2024-04-23T18:23:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f38117d5740 10 client.0 _ll_register_callbacks cb 0x7f37f4009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f380c85e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851c620 0x561ee852fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f380d05f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 0x561ee852d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f380c85e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851c620 0x561ee852fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:44250/0 (socket says 172.21.15.50:44250) 2024-04-23T18:23:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f380c85e640 1 -- 172.21.15.50:0/944912271 learned_addr learned my addr 172.21.15.50:0/944912271 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f380d05f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 0x561ee852d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49650/0 (socket says 172.21.15.50:49650) 2024-04-23T18:23:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.893+0000 7f3807fff640 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851cf40 0x561ee852d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f380c85e640 1 -- 172.21.15.50:0/944912271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851cf40 msgr2=0x561ee852d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f380c85e640 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851cf40 0x561ee852d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f380c85e640 1 -- 172.21.15.50:0/944912271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 msgr2=0x561ee852d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f380c85e640 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 0x561ee852d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f380c85e640 1 -- 172.21.15.50:0/944912271 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ee852de60 con 0x561ee851c620 2024-04-23T18:23:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f380d05f640 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 0x561ee852d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f380c85e640 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851c620 0x561ee852fcc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f37fc009250 tx=0x7f37fc0047a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1de9a17b3364b1d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f37e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f37e7fff640 1 -- 172.21.15.50:0/944912271 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc039070 con 0x561ee851c620 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f38117d5740 1 -- 172.21.15.50:0/944912271 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37fc004bc0 con 0x561ee851c620 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f37e7fff640 1 -- 172.21.15.50:0/944912271 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f37fc00f680 con 0x561ee851c620 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f37e7fff640 1 -- 172.21.15.50:0/944912271 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f37fc00f880 con 0x561ee851c620 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f38117d5740 10 client.25736 fetch_fsmap learned FSMap version 28 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f38117d5740 1 -- 172.21.15.50:0/944912271 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561ee859df80 con 0x561ee851c620 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.894+0000 7f37e7fff640 1 -- 172.21.15.50:0/944912271 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37fc004bc0 con 0x561ee851c620 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.895+0000 7f37e7fff640 1 -- 172.21.15.50:0/944912271 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f37fc005470 con 0x561ee851c620 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.895+0000 7f38117d5740 10 client.25736 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.895+0000 7f38117d5740 10 client.25736 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.895+0000 7f38117d5740 20 client.25736 populate_metadata read hostname 'smithi050' 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.895+0000 7f38117d5740 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561ee859e420 0x561ee85a0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.895+0000 7f38117d5740 4 client.25736 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.895+0000 7f38117d5740 1 -- 172.21.15.50:0/944912271 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561ee8524ca0 con 0x561ee859e420 2024-04-23T18:23:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.895+0000 7f3807fff640 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561ee859e420 0x561ee85a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.895+0000 7f3807fff640 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561ee859e420 0x561ee85a0840 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5163dda3a187e32 server_cookie=918283ed593f0bf4 in_seq=0 out_seq=0 2024-04-23T18:23:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.895+0000 7f37e7fff640 10 client.25736 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f37e7fff640 1 -- 172.21.15.50:0/944912271 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561ee8524ca0 con 0x561ee859e420 2024-04-23T18:23:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f37e7fff640 10 client.25736 handle_command_reply: tid=0 2024-04-23T18:23:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f38117d5740 1 client.25736 shutdown 2024-04-23T18:23:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f38117d5740 1 -- 172.21.15.50:0/944912271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851c620 msgr2=0x561ee852fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f38117d5740 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851c620 0x561ee852fcc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f37fc009250 tx=0x7f37fc0047a0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f38117d5740 1 -- 172.21.15.50:0/944912271 shutdown_connections 2024-04-23T18:23:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f38117d5740 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561ee859e420 0x561ee85a0840 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f38117d5740 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561ee8527800 0x561ee852d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f38117d5740 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561ee851cf40 0x561ee852d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f37e7fff640 0 client.25736 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f38117d5740 1 --2- 172.21.15.50:0/944912271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ee851c620 0x561ee852fcc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f38117d5740 1 -- 172.21.15.50:0/944912271 >> 172.21.15.50:0/944912271 conn(0x561ee8518b30 msgr2=0x561ee8517cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f38117d5740 1 -- 172.21.15.50:0/944912271 shutdown_connections 2024-04-23T18:23:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f38117d5740 1 -- 172.21.15.50:0/944912271 wait complete. 2024-04-23T18:23:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.953+0000 7f38117d5740 20 client.25736 trim_cache size 0 max 16384 2024-04-23T18:23:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:23:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:23:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:23:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:23:35.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.954+0000 7f380eac2640 1 -- 172.21.15.50:0/3051367039 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f37f80a3dc0 msgr2=0x7f37f80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:35.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.954+0000 7f380eac2640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f37f80a3dc0 0x7f37f80a6280 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f37f40058c0 tx=0x7f37f4005f90 comp rx=0 tx=0).stop 2024-04-23T18:23:35.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.954+0000 7f380eac2640 1 -- 172.21.15.50:0/3051367039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380819a700 msgr2=0x7f380819ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:35.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.954+0000 7f380eac2640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380819a700 0x7f380819ab80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3800005e30 tx=0x7f38000317c0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.954+0000 7f380eac2640 1 -- 172.21.15.50:0/3051367039 shutdown_connections 2024-04-23T18:23:35.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.954+0000 7f380eac2640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f37f80a3dc0 0x7f37f80a6280 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.954+0000 7f380eac2640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380819a700 0x7f380819ab80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.954+0000 7f380eac2640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3808106170 0x7f38081a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.954+0000 7f380eac2640 1 --2- 172.21.15.50:0/3051367039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38081010f0 0x7f38081a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:35.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.954+0000 7f380eac2640 1 -- 172.21.15.50:0/3051367039 >> 172.21.15.50:0/3051367039 conn(0x7f3808077fd0 msgr2=0x7f38080ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:35.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.954+0000 7f380eac2640 1 -- 172.21.15.50:0/3051367039 shutdown_connections 2024-04-23T18:23:35.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:35.954+0000 7f380eac2640 1 -- 172.21.15.50:0/3051367039 wait complete. 2024-04-23T18:23:36.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 301962 4.46 MB/sec execute 130 sec latency 66.257 ms 2024-04-23T18:23:36.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:36 smithi188 ceph-mon[32694]: pgmap v156: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1001 KiB/s wr, 135 op/s 2024-04-23T18:23:36.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:36 smithi050 ceph-mon[29187]: pgmap v156: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1001 KiB/s wr, 135 op/s 2024-04-23T18:23:36.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:36 smithi177 ceph-mon[32308]: pgmap v156: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1001 KiB/s wr, 135 op/s 2024-04-23T18:23:37.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 303437 4.47 MB/sec execute 131 sec latency 20.139 ms 2024-04-23T18:23:38.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 305542 4.54 MB/sec execute 132 sec latency 32.085 ms 2024-04-23T18:23:38.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:38 smithi188 ceph-mon[32694]: pgmap v157: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 261 op/s 2024-04-23T18:23:38.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:38 smithi050 ceph-mon[29187]: pgmap v157: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 261 op/s 2024-04-23T18:23:38.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:38 smithi177 ceph-mon[32308]: pgmap v157: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.7 MiB/s wr, 261 op/s 2024-04-23T18:23:39.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 307514 4.56 MB/sec execute 133 sec latency 21.806 ms 2024-04-23T18:23:40.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 309432 4.61 MB/sec execute 134 sec latency 21.773 ms 2024-04-23T18:23:40.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:40 smithi188 ceph-mon[32694]: pgmap v158: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 225 op/s 2024-04-23T18:23:40.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:40 smithi050 ceph-mon[29187]: pgmap v158: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 225 op/s 2024-04-23T18:23:40.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:40 smithi177 ceph-mon[32308]: pgmap v158: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 225 op/s 2024-04-23T18:23:41.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 311387 4.66 MB/sec execute 135 sec latency 14.305 ms 2024-04-23T18:23:41.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:41 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:23:41.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:41 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:23:41.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:41 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:23:41.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:41 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:23:41.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:41 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:23:41.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:41 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:23:41.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:41 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:23:41.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:41 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:23:41.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:41 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:23:41.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:41 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:23:41.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:41 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:23:41.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:41 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:23:42.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 313228 4.70 MB/sec execute 136 sec latency 18.542 ms 2024-04-23T18:23:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:42 smithi050 ceph-mon[29187]: pgmap v159: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 225 op/s 2024-04-23T18:23:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:42 smithi177 ceph-mon[32308]: pgmap v159: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 225 op/s 2024-04-23T18:23:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:42 smithi188 ceph-mon[32694]: pgmap v159: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 225 op/s 2024-04-23T18:23:43.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 315178 4.74 MB/sec execute 137 sec latency 15.952 ms 2024-04-23T18:23:44.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 317114 4.79 MB/sec execute 138 sec latency 30.706 ms 2024-04-23T18:23:44.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:44 smithi050 ceph-mon[29187]: pgmap v160: 97 pgs: 97 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.1 MiB/s wr, 325 op/s 2024-04-23T18:23:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:44 smithi177 ceph-mon[32308]: pgmap v160: 97 pgs: 97 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.1 MiB/s wr, 325 op/s 2024-04-23T18:23:44.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:44 smithi188 ceph-mon[32694]: pgmap v160: 97 pgs: 97 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.1 MiB/s wr, 325 op/s 2024-04-23T18:23:45.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 319470 4.86 MB/sec execute 139 sec latency 17.612 ms 2024-04-23T18:23:46.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 320886 4.86 MB/sec execute 140 sec latency 17.379 ms 2024-04-23T18:23:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:46 smithi050 ceph-mon[29187]: pgmap v161: 97 pgs: 97 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 225 op/s 2024-04-23T18:23:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:46 smithi177 ceph-mon[32308]: pgmap v161: 97 pgs: 97 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 225 op/s 2024-04-23T18:23:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:46 smithi188 ceph-mon[32694]: pgmap v161: 97 pgs: 97 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 225 op/s 2024-04-23T18:23:47.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 323183 4.92 MB/sec execute 141 sec latency 15.339 ms 2024-04-23T18:23:48.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 324487 4.93 MB/sec execute 142 sec latency 86.865 ms 2024-04-23T18:23:48.338 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:48 smithi177 ceph-mon[32308]: pgmap v162: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 319 op/s 2024-04-23T18:23:48.380 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:48 smithi188 ceph-mon[32694]: pgmap v162: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 319 op/s 2024-04-23T18:23:48.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:48 smithi050 ceph-mon[29187]: pgmap v162: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 319 op/s 2024-04-23T18:23:49.042 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.046+0000 7efc4b2a0640 1 -- 172.21.15.50:0/1150125707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc4410c520 msgr2=0x7efc4410e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:49.042 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.046+0000 7efc4b2a0640 1 --2- 172.21.15.50:0/1150125707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc4410c520 0x7efc4410e9c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7efc3c009ca0 tx=0x7efc3c02a9e0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.042 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.046+0000 7efc4b2a0640 1 -- 172.21.15.50:0/1150125707 shutdown_connections 2024-04-23T18:23:49.042 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.046+0000 7efc4b2a0640 1 --2- 172.21.15.50:0/1150125707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc4410c520 0x7efc4410e9c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.042 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.046+0000 7efc4b2a0640 1 --2- 172.21.15.50:0/1150125707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc44103b00 0x7efc4410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.042 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.046+0000 7efc4b2a0640 1 --2- 172.21.15.50:0/1150125707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc441031e0 0x7efc441035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.042 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.046+0000 7efc4b2a0640 1 -- 172.21.15.50:0/1150125707 >> 172.21.15.50:0/1150125707 conn(0x7efc44077fd0 msgr2=0x7efc440783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:49.042 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.046+0000 7efc4b2a0640 1 -- 172.21.15.50:0/1150125707 shutdown_connections 2024-04-23T18:23:49.042 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.046+0000 7efc4b2a0640 1 -- 172.21.15.50:0/1150125707 wait complete. 2024-04-23T18:23:49.042 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.046+0000 7efc4b2a0640 1 Processor -- start 2024-04-23T18:23:49.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4b2a0640 1 -- start start 2024-04-23T18:23:49.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4b2a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc441031e0 0x7efc441986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4b2a0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc44103b00 0x7efc44198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4b2a0640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc4410c520 0x7efc4419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4b2a0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efc44112190 con 0x7efc441031e0 2024-04-23T18:23:49.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4b2a0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efc44112010 con 0x7efc44103b00 2024-04-23T18:23:49.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4b2a0640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efc44112310 con 0x7efc4410c520 2024-04-23T18:23:49.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4903c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc441031e0 0x7efc441986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4903c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc441031e0 0x7efc441986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59124/0 (socket says 172.21.15.50:59124) 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4903c640 1 -- 172.21.15.50:0/2985303544 learned_addr learned my addr 172.21.15.50:0/2985303544 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4983d640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc4410c520 0x7efc4419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4883b640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc44103b00 0x7efc44198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4903c640 1 -- 172.21.15.50:0/2985303544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc4410c520 msgr2=0x7efc4419cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4903c640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc4410c520 0x7efc4419cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4903c640 1 -- 172.21.15.50:0/2985303544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc44103b00 msgr2=0x7efc44198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4903c640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc44103b00 0x7efc44198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4903c640 1 -- 172.21.15.50:0/2985303544 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc4419d670 con 0x7efc441031e0 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc4903c640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc441031e0 0x7efc441986d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7efc4000e9f0 tx=0x7efc4000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.047+0000 7efc3a7fc640 1 -- 172.21.15.50:0/2985303544 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4000cd80 con 0x7efc441031e0 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.048+0000 7efc3a7fc640 1 -- 172.21.15.50:0/2985303544 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc400108e0 con 0x7efc441031e0 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.048+0000 7efc4b2a0640 1 -- 172.21.15.50:0/2985303544 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc4419d960 con 0x7efc441031e0 2024-04-23T18:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.048+0000 7efc3a7fc640 1 -- 172.21.15.50:0/2985303544 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc40018980 con 0x7efc441031e0 2024-04-23T18:23:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.048+0000 7efc4b2a0640 1 -- 172.21.15.50:0/2985303544 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc441a51f0 con 0x7efc441031e0 2024-04-23T18:23:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.050+0000 7efc3a7fc640 1 -- 172.21.15.50:0/2985303544 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7efc40010430 con 0x7efc441031e0 2024-04-23T18:23:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc3a7fc640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efc300a3dc0 0x7efc300a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc4dfbb740 1 Processor -- start 2024-04-23T18:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc4dfbb740 1 -- start start 2024-04-23T18:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc4883b640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efc300a3dc0 0x7efc300a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc3a7fc640 1 -- 172.21.15.50:0/2985303544 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7efc400d0ab0 con 0x7efc441031e0 2024-04-23T18:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc4dfbb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd642a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc4dfbb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd642f40 0x55e6cd64d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc4dfbb740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd64d800 0x55e6cd64fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc4dfbb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e6cd5a7970 con 0x55e6cd642f40 2024-04-23T18:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc4dfbb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e6cd5444a0 con 0x55e6cd642620 2024-04-23T18:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc4dfbb740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e6cd5a3930 con 0x55e6cd64d800 2024-04-23T18:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc4983d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd64d800 0x55e6cd64fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.051+0000 7efc4903c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd642a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc4903c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd642a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:47342/0 (socket says 172.21.15.50:47342) 2024-04-23T18:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc4903c640 1 -- 172.21.15.50:0/738810988 learned_addr learned my addr 172.21.15.50:0/738810988 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc4883b640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efc300a3dc0 0x7efc300a6280 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efc44100860 tx=0x7efc34005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc4883b640 1 --2- 172.21.15.50:0/738810988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd642f40 0x55e6cd64d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc4903c640 1 -- 172.21.15.50:0/738810988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd64d800 msgr2=0x55e6cd64fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc4903c640 1 --2- 172.21.15.50:0/738810988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd64d800 0x55e6cd64fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc4903c640 1 -- 172.21.15.50:0/738810988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd642f40 msgr2=0x55e6cd64d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc4903c640 1 --2- 172.21.15.50:0/738810988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd642f40 0x55e6cd64d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc4903c640 1 -- 172.21.15.50:0/738810988 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6cd2ba210 con 0x55e6cd642620 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc4903c640 1 --2- 172.21.15.50:0/738810988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd642a00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efc400994e0 tx=0x7efc400107e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d972aace88a855 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc2f7fe640 1 -- 172.21.15.50:0/738810988 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc400a56d0 con 0x55e6cd642620 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc2f7fe640 1 -- 172.21.15.50:0/738810988 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc40010a90 con 0x55e6cd642620 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.052+0000 7efc2f7fe640 1 -- 172.21.15.50:0/738810988 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc40099320 con 0x55e6cd642620 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.053+0000 7efc4dfbb740 1 -- 172.21.15.50:0/738810988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 msgr2=0x55e6cd642a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.053+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/738810988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd642a00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efc400994e0 tx=0x7efc400107e0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.053+0000 7efc4dfbb740 1 -- 172.21.15.50:0/738810988 shutdown_connections 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.053+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/738810988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd64d800 0x55e6cd64fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.053+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/738810988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd642f40 0x55e6cd64d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.053+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/738810988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd642a00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.053+0000 7efc4dfbb740 1 -- 172.21.15.50:0/738810988 >> 172.21.15.50:0/738810988 conn(0x55e6cd63eb30 msgr2=0x55e6cd63ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.053+0000 7efc4dfbb740 1 -- 172.21.15.50:0/738810988 shutdown_connections 2024-04-23T18:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.053+0000 7efc4dfbb740 1 -- 172.21.15.50:0/738810988 wait complete. 2024-04-23T18:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.053+0000 7efc4dfbb740 1 -- start start 2024-04-23T18:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.053+0000 7efc4dfbb740 1 Processor -- start 2024-04-23T18:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4dfbb740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd642620 0x55e6cd64d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4dfbb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642f40 0x55e6cd64a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4dfbb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd64d800 0x55e6cd64aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4dfbb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e6cd6503d0 con 0x55e6cd64d800 2024-04-23T18:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4dfbb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e6cd5420b0 con 0x55e6cd642f40 2024-04-23T18:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4dfbb740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e6cd650550 con 0x55e6cd642620 2024-04-23T18:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4dfbb740 10 client.0 _ll_register_callbacks cb 0x55e6cd52fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4883b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642f40 0x55e6cd64a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4983d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd64d800 0x55e6cd64aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4883b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642f40 0x55e6cd64a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:47344/0 (socket says 172.21.15.50:47344) 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4883b640 1 -- 172.21.15.50:0/204738722 learned_addr learned my addr 172.21.15.50:0/204738722 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4903c640 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd642620 0x55e6cd64d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4883b640 1 -- 172.21.15.50:0/204738722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd642620 msgr2=0x55e6cd64d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4883b640 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd642620 0x55e6cd64d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4883b640 1 -- 172.21.15.50:0/204738722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd64d800 msgr2=0x55e6cd64aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4883b640 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd64d800 0x55e6cd64aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.054+0000 7efc4883b640 1 -- 172.21.15.50:0/204738722 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6cd386620 con 0x55e6cd642f40 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.055+0000 7efc4903c640 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd642620 0x55e6cd64d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.055+0000 7efc4883b640 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642f40 0x55e6cd64a4e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efc340038a0 tx=0x7efc34004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc9af38a0f6db850 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.055+0000 7efc2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.055+0000 7efc2cff9640 1 -- 172.21.15.50:0/204738722 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34027420 con 0x55e6cd642f40 2024-04-23T18:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.055+0000 7efc2cff9640 1 -- 172.21.15.50:0/204738722 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7efc3401d410 con 0x55e6cd642f40 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.055+0000 7efc4dfbb740 1 -- 172.21.15.50:0/204738722 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc40010a90 con 0x55e6cd642f40 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.055+0000 7efc2cff9640 1 -- 172.21.15.50:0/204738722 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efc3401d610 con 0x55e6cd642f40 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.055+0000 7efc4dfbb740 10 client.25748 fetch_fsmap learned FSMap version 28 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.055+0000 7efc4dfbb740 1 -- 172.21.15.50:0/204738722 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e6cd65c3e0 con 0x55e6cd642f40 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.056+0000 7efc2cff9640 1 -- 172.21.15.50:0/204738722 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc40010a90 con 0x55e6cd642f40 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.056+0000 7efc2cff9640 1 -- 172.21.15.50:0/204738722 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7efc34027a50 con 0x55e6cd642f40 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.056+0000 7efc4dfbb740 10 client.25748 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.056+0000 7efc4dfbb740 10 client.25748 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.056+0000 7efc4dfbb740 20 client.25748 populate_metadata read hostname 'smithi050' 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.056+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e6cd65d9b0 0x55e6cd65fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.056+0000 7efc4dfbb740 4 client.25748 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.056+0000 7efc4dfbb740 1 -- 172.21.15.50:0/204738722 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6cd5b5390 con 0x55e6cd65d9b0 2024-04-23T18:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.056+0000 7efc4903c640 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e6cd65d9b0 0x55e6cd65fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.056+0000 7efc2cff9640 10 client.25748 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.056+0000 7efc4903c640 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e6cd65d9b0 0x55e6cd65fdd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43ffd452119aa20d server_cookie=833f22093eacc3c5 in_seq=0 out_seq=0 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc2cff9640 1 -- 172.21.15.50:0/204738722 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e6cd5b5390 con 0x55e6cd65d9b0 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc2cff9640 10 client.25748 handle_command_reply: tid=0 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc4dfbb740 1 client.25748 shutdown 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc4dfbb740 1 -- 172.21.15.50:0/204738722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642f40 msgr2=0x55e6cd64a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642f40 0x55e6cd64a4e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efc340038a0 tx=0x7efc34004730 comp rx=0 tx=0).stop 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc4dfbb740 1 -- 172.21.15.50:0/204738722 shutdown_connections 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e6cd65d9b0 0x55e6cd65fdd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd64d800 0x55e6cd64aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc2cff9640 0 client.25748 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642f40 0x55e6cd64a4e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/204738722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd642620 0x55e6cd64d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc4dfbb740 1 -- 172.21.15.50:0/204738722 >> 172.21.15.50:0/204738722 conn(0x55e6cd63eb30 msgr2=0x55e6cd6448b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:49.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc4dfbb740 1 -- 172.21.15.50:0/204738722 shutdown_connections 2024-04-23T18:23:49.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc4dfbb740 1 -- 172.21.15.50:0/204738722 wait complete. 2024-04-23T18:23:49.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.058+0000 7efc4dfbb740 20 client.25748 trim_cache size 0 max 16384 2024-04-23T18:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4dfbb740 1 Processor -- start 2024-04-23T18:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4dfbb740 1 -- start start 2024-04-23T18:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4dfbb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd63e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4dfbb740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd642f40 0x55e6cd645970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4dfbb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd64d800 0x55e6cd64cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4dfbb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efc441b0510 con 0x55e6cd64d800 2024-04-23T18:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4dfbb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efc441add80 con 0x55e6cd642620 2024-04-23T18:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4dfbb740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efc441abbd0 con 0x55e6cd642f40 2024-04-23T18:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4903c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd63e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4903c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd63e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:47354/0 (socket says 172.21.15.50:47354) 2024-04-23T18:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4983d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd64d800 0x55e6cd64cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4903c640 1 -- 172.21.15.50:0/246176134 learned_addr learned my addr 172.21.15.50:0/246176134 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4883b640 1 --2- 172.21.15.50:0/246176134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd642f40 0x55e6cd645970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4903c640 1 -- 172.21.15.50:0/246176134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd642f40 msgr2=0x55e6cd645970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4903c640 1 --2- 172.21.15.50:0/246176134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd642f40 0x55e6cd645970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4903c640 1 -- 172.21.15.50:0/246176134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd64d800 msgr2=0x55e6cd64cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4903c640 1 --2- 172.21.15.50:0/246176134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd64d800 0x55e6cd64cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4903c640 1 -- 172.21.15.50:0/246176134 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6cd644fe0 con 0x55e6cd642620 2024-04-23T18:23:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.074+0000 7efc4983d640 1 --2- 172.21.15.50:0/246176134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd64d800 0x55e6cd64cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.075+0000 7efc4903c640 1 --2- 172.21.15.50:0/246176134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd63e430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efc40098f70 tx=0x7efc40009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3389044902cd5043 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.075+0000 7efc2effd640 1 -- 172.21.15.50:0/246176134 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc400ce490 con 0x55e6cd642620 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.075+0000 7efc2effd640 1 -- 172.21.15.50:0/246176134 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc40099180 con 0x55e6cd642620 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.075+0000 7efc2effd640 1 -- 172.21.15.50:0/246176134 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc40018d80 con 0x55e6cd642620 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.075+0000 7efc4dfbb740 1 -- 172.21.15.50:0/246176134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 msgr2=0x55e6cd63e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.075+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/246176134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd63e430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efc40098f70 tx=0x7efc40009310 comp rx=0 tx=0).stop 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 -- 172.21.15.50:0/246176134 shutdown_connections 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/246176134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd64d800 0x55e6cd64cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/246176134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd642f40 0x55e6cd645970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/246176134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd63e430 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 -- 172.21.15.50:0/246176134 >> 172.21.15.50:0/246176134 conn(0x55e6cd63eb30 msgr2=0x55e6cd645260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 -- 172.21.15.50:0/246176134 shutdown_connections 2024-04-23T18:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 -- 172.21.15.50:0/246176134 wait complete. 2024-04-23T18:23:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 -- start start 2024-04-23T18:23:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 Processor -- start 2024-04-23T18:23:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd655a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd642f40 0x55e6cd646d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd64d800 0x55e6cd647290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efc441ac240 con 0x55e6cd642f40 2024-04-23T18:23:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efc441ad6b0 con 0x55e6cd642620 2024-04-23T18:23:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.076+0000 7efc4dfbb740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efc441ac910 con 0x55e6cd64d800 2024-04-23T18:23:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4dfbb740 10 client.0 _ll_register_callbacks cb 0x7efc340044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:23:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4883b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd642f40 0x55e6cd646d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4903c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd655a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4903c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd655a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:47358/0 (socket says 172.21.15.50:47358) 2024-04-23T18:23:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4903c640 1 -- 172.21.15.50:0/3155347480 learned_addr learned my addr 172.21.15.50:0/3155347480 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:23:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4983d640 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd64d800 0x55e6cd647290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4883b640 1 -- 172.21.15.50:0/3155347480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd64d800 msgr2=0x55e6cd647290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4883b640 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd64d800 0x55e6cd647290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4883b640 1 -- 172.21.15.50:0/3155347480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 msgr2=0x55e6cd655a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:23:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4883b640 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd655a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4883b640 1 -- 172.21.15.50:0/3155347480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6cd647930 con 0x55e6cd642f40 2024-04-23T18:23:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4903c640 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd655a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:23:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4883b640 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd642f40 0x55e6cd646d50 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7efc34011810 tx=0x7efc3401df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=485ae198ebb7d20b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:23:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:23:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc2cff9640 1 -- 172.21.15.50:0/3155347480 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34033070 con 0x55e6cd642f40 2024-04-23T18:23:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc2cff9640 1 -- 172.21.15.50:0/3155347480 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7efc34009040 con 0x55e6cd642f40 2024-04-23T18:23:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4dfbb740 1 -- 172.21.15.50:0/3155347480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc40099180 con 0x55e6cd642f40 2024-04-23T18:23:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc2cff9640 1 -- 172.21.15.50:0/3155347480 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efc3401d410 con 0x55e6cd642f40 2024-04-23T18:23:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4dfbb740 10 client.16152 fetch_fsmap learned FSMap version 28 2024-04-23T18:23:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.077+0000 7efc4dfbb740 1 -- 172.21.15.50:0/3155347480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e6cd6c3df0 con 0x55e6cd642f40 2024-04-23T18:23:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.078+0000 7efc2cff9640 1 -- 172.21.15.50:0/3155347480 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc40099180 con 0x55e6cd642f40 2024-04-23T18:23:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.078+0000 7efc2cff9640 1 -- 172.21.15.50:0/3155347480 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7efc3401c710 con 0x55e6cd642f40 2024-04-23T18:23:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.078+0000 7efc4dfbb740 10 client.16152 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T18:23:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.078+0000 7efc4dfbb740 10 client.16152 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:23:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.078+0000 7efc4dfbb740 20 client.16152 populate_metadata read hostname 'smithi050' 2024-04-23T18:23:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.078+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e6cd6c45d0 0x55e6cd6c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:23:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.078+0000 7efc4dfbb740 4 client.16152 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:23:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.078+0000 7efc4dfbb740 1 -- 172.21.15.50:0/3155347480 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e6cd5af200 con 0x55e6cd6c45d0 2024-04-23T18:23:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.078+0000 7efc4903c640 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e6cd6c45d0 0x55e6cd6c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:23:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.078+0000 7efc2cff9640 10 client.16152 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.078+0000 7efc4903c640 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e6cd6c45d0 0x55e6cd6c69f0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=484721734aaed4fa server_cookie=d57f7199edfc1000 in_seq=0 out_seq=0 2024-04-23T18:23:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc2cff9640 1 -- 172.21.15.50:0/3155347480 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e6cd5af200 con 0x55e6cd6c45d0 2024-04-23T18:23:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc2cff9640 10 client.16152 handle_command_reply: tid=0 2024-04-23T18:23:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc4dfbb740 1 client.16152 shutdown 2024-04-23T18:23:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc4dfbb740 1 -- 172.21.15.50:0/3155347480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd642f40 msgr2=0x55e6cd646d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd642f40 0x55e6cd646d50 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7efc34011810 tx=0x7efc3401df60 comp rx=0 tx=0).stop 2024-04-23T18:23:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc4dfbb740 1 -- 172.21.15.50:0/3155347480 shutdown_connections 2024-04-23T18:23:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e6cd6c45d0 0x55e6cd6c69f0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cd64d800 0x55e6cd647290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc2cff9640 0 client.16152 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:23:49.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e6cd642f40 0x55e6cd646d50 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc4dfbb740 1 --2- 172.21.15.50:0/3155347480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6cd642620 0x55e6cd655a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc4dfbb740 1 -- 172.21.15.50:0/3155347480 >> 172.21.15.50:0/3155347480 conn(0x55e6cd63eb30 msgr2=0x55e6cd644860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc4dfbb740 1 -- 172.21.15.50:0/3155347480 shutdown_connections 2024-04-23T18:23:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc4dfbb740 1 -- 172.21.15.50:0/3155347480 wait complete. 2024-04-23T18:23:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.150+0000 7efc4dfbb740 20 client.16152 trim_cache size 0 max 16384 2024-04-23T18:23:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:23:49.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:23:49.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:23:49.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:23:49.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.151+0000 7efc4b2a0640 1 -- 172.21.15.50:0/2985303544 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efc300a3dc0 msgr2=0x7efc300a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.151+0000 7efc4b2a0640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efc300a3dc0 0x7efc300a6280 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efc44100860 tx=0x7efc34005f90 comp rx=0 tx=0).stop 2024-04-23T18:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.151+0000 7efc4b2a0640 1 -- 172.21.15.50:0/2985303544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc441031e0 msgr2=0x7efc441986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.151+0000 7efc4b2a0640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc441031e0 0x7efc441986d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7efc4000e9f0 tx=0x7efc4000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.151+0000 7efc4b2a0640 1 -- 172.21.15.50:0/2985303544 shutdown_connections 2024-04-23T18:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.151+0000 7efc4b2a0640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efc300a3dc0 0x7efc300a6280 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.151+0000 7efc4b2a0640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc4410c520 0x7efc4419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.151+0000 7efc4b2a0640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc44103b00 0x7efc44198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.151+0000 7efc4b2a0640 1 --2- 172.21.15.50:0/2985303544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc441031e0 0x7efc441986d0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.151+0000 7efc4b2a0640 1 -- 172.21.15.50:0/2985303544 >> 172.21.15.50:0/2985303544 conn(0x7efc44077fd0 msgr2=0x7efc440fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.151+0000 7efc4b2a0640 1 -- 172.21.15.50:0/2985303544 shutdown_connections 2024-04-23T18:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:23:49.151+0000 7efc4b2a0640 1 -- 172.21.15.50:0/2985303544 wait complete. 2024-04-23T18:23:49.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 326577 4.99 MB/sec execute 143 sec latency 63.470 ms 2024-04-23T18:23:50.037 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:23:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: ignoring --setuser ceph since I am not root 2024-04-23T18:23:50.037 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:23:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: ignoring --setgroup ceph since I am not root 2024-04-23T18:23:50.038 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:23:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: starting mds.h at 2024-04-23T18:23:50.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 328023 4.99 MB/sec execute 144 sec latency 80.388 ms 2024-04-23T18:23:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:50 smithi050 ceph-mon[29187]: pgmap v163: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 193 op/s 2024-04-23T18:23:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:50 smithi050 ceph-mon[29187]: daemon mds.h restarted 2024-04-23T18:23:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:50 smithi177 ceph-mon[32308]: pgmap v163: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 193 op/s 2024-04-23T18:23:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:50 smithi177 ceph-mon[32308]: daemon mds.h restarted 2024-04-23T18:23:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:50 smithi188 ceph-mon[32694]: pgmap v163: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 193 op/s 2024-04-23T18:23:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:50 smithi188 ceph-mon[32694]: daemon mds.h restarted 2024-04-23T18:23:51.206 INFO:tasks.workunit.client.0.smithi050.stdout: 1 330262 5.05 MB/sec execute 145 sec latency 13.450 ms 2024-04-23T18:23:51.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:51 smithi050 ceph-mon[29187]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T18:23:51.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:51 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6836/911343542,v1:172.21.15.177:6837/911343542] up:boot 2024-04-23T18:23:51.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:51 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:23:51.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:51 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:23:51.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:51 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:23:51.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:51 smithi177 ceph-mon[32308]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T18:23:51.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:51 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6836/911343542,v1:172.21.15.177:6837/911343542] up:boot 2024-04-23T18:23:51.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:51 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:23:51.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:51 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:23:51.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:51 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:23:51.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:51 smithi188 ceph-mon[32694]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T18:23:51.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:51 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6836/911343542,v1:172.21.15.177:6837/911343542] up:boot 2024-04-23T18:23:51.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:51 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:23:51.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:51 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:23:51.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:51 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:23:52.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 331779 5.05 MB/sec execute 146 sec latency 25.388 ms 2024-04-23T18:23:52.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:52 smithi050 ceph-mon[29187]: pgmap v165: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 232 op/s 2024-04-23T18:23:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:52 smithi177 ceph-mon[32308]: pgmap v165: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 232 op/s 2024-04-23T18:23:52.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:52 smithi188 ceph-mon[32694]: pgmap v165: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 232 op/s 2024-04-23T18:23:53.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 333969 5.11 MB/sec execute 147 sec latency 15.693 ms 2024-04-23T18:23:54.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 335835 5.13 MB/sec execute 148 sec latency 38.060 ms 2024-04-23T18:23:54.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:54 smithi050 ceph-mon[29187]: pgmap v166: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 282 op/s 2024-04-23T18:23:54.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:54 smithi177 ceph-mon[32308]: pgmap v166: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 282 op/s 2024-04-23T18:23:54.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:54 smithi188 ceph-mon[32694]: pgmap v166: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 282 op/s 2024-04-23T18:23:55.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 337643 5.17 MB/sec execute 149 sec latency 18.689 ms 2024-04-23T18:23:56.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 339445 5.19 MB/sec execute 150 sec latency 27.462 ms 2024-04-23T18:23:56.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:56 smithi050 ceph-mon[29187]: pgmap v167: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 282 op/s 2024-04-23T18:23:56.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:56 smithi177 ceph-mon[32308]: pgmap v167: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 282 op/s 2024-04-23T18:23:56.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:56 smithi188 ceph-mon[32694]: pgmap v167: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 282 op/s 2024-04-23T18:23:57.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 341462 5.24 MB/sec execute 151 sec latency 22.861 ms 2024-04-23T18:23:58.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 343492 5.28 MB/sec execute 152 sec latency 15.213 ms 2024-04-23T18:23:58.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:23:58 smithi050 ceph-mon[29187]: pgmap v168: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 291 op/s 2024-04-23T18:23:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:23:58 smithi177 ceph-mon[32308]: pgmap v168: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 291 op/s 2024-04-23T18:23:58.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:23:58 smithi188 ceph-mon[32694]: pgmap v168: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 291 op/s 2024-04-23T18:23:59.209 INFO:tasks.workunit.client.0.smithi050.stdout: 1 345468 5.32 MB/sec execute 153 sec latency 15.717 ms 2024-04-23T18:24:00.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 347406 5.35 MB/sec execute 154 sec latency 29.098 ms 2024-04-23T18:24:00.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:00 smithi050 ceph-mon[29187]: pgmap v169: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 291 op/s 2024-04-23T18:24:00.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:00 smithi177 ceph-mon[32308]: pgmap v169: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 291 op/s 2024-04-23T18:24:00.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:00 smithi188 ceph-mon[32694]: pgmap v169: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.0 MiB/s wr, 291 op/s 2024-04-23T18:24:01.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 348362 5.35 MB/sec execute 155 sec latency 31.800 ms 2024-04-23T18:24:02.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 350369 5.38 MB/sec execute 156 sec latency 34.717 ms 2024-04-23T18:24:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.249+0000 7ffb1e660640 1 -- 172.21.15.50:0/1287507336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb181094b0 msgr2=0x7ffb18111080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.249+0000 7ffb1e660640 1 --2- 172.21.15.50:0/1287507336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb181094b0 0x7ffb18111080 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009ca0 tx=0x7ffb1002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.249+0000 7ffb1e660640 1 -- 172.21.15.50:0/1287507336 shutdown_connections 2024-04-23T18:24:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.249+0000 7ffb1e660640 1 --2- 172.21.15.50:0/1287507336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb181094b0 0x7ffb18111080 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.249+0000 7ffb1e660640 1 --2- 172.21.15.50:0/1287507336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb18104d90 0x7ffb18108dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.249+0000 7ffb1e660640 1 --2- 172.21.15.50:0/1287507336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb181043e0 0x7ffb181047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.249+0000 7ffb1e660640 1 -- 172.21.15.50:0/1287507336 >> 172.21.15.50:0/1287507336 conn(0x7ffb180ffc60 msgr2=0x7ffb18102080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.249+0000 7ffb1e660640 1 -- 172.21.15.50:0/1287507336 shutdown_connections 2024-04-23T18:24:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.249+0000 7ffb1e660640 1 -- 172.21.15.50:0/1287507336 wait complete. 2024-04-23T18:24:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb1e660640 1 Processor -- start 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb1e660640 1 -- start start 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb1e660640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb181043e0 0x7ffb181a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb1e660640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb181094b0 0x7ffb181a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb1e660640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb1819a6e0 0x7ffb1819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb1e660640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffb18114420 con 0x7ffb181094b0 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb1e660640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181142a0 con 0x7ffb1819a6e0 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb1e660640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181145a0 con 0x7ffb181043e0 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb181094b0 0x7ffb181a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb17fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb181043e0 0x7ffb181a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb1cbfd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb1819a6e0 0x7ffb1819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb181094b0 0x7ffb181a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59960/0 (socket says 172.21.15.50:59960) 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb177fe640 1 -- 172.21.15.50:0/1960903762 learned_addr learned my addr 172.21.15.50:0/1960903762 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:02.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb177fe640 1 -- 172.21.15.50:0/1960903762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb181043e0 msgr2=0x7ffb181a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb177fe640 1 --2- 172.21.15.50:0/1960903762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb181043e0 0x7ffb181a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb177fe640 1 -- 172.21.15.50:0/1960903762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb1819a6e0 msgr2=0x7ffb1819ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb177fe640 1 --2- 172.21.15.50:0/1960903762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb1819a6e0 0x7ffb1819ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.250+0000 7ffb177fe640 1 -- 172.21.15.50:0/1960903762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb1819b2e0 con 0x7ffb181094b0 2024-04-23T18:24:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.251+0000 7ffb177fe640 1 --2- 172.21.15.50:0/1960903762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb181094b0 0x7ffb181a0b60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c00e9c0 tx=0x7ffb0c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.251+0000 7ffb157fa640 1 -- 172.21.15.50:0/1960903762 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c00cd80 con 0x7ffb181094b0 2024-04-23T18:24:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.251+0000 7ffb157fa640 1 -- 172.21.15.50:0/1960903762 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb0c0054d0 con 0x7ffb181094b0 2024-04-23T18:24:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.251+0000 7ffb1e660640 1 -- 172.21.15.50:0/1960903762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb181a73c0 con 0x7ffb181094b0 2024-04-23T18:24:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.251+0000 7ffb157fa640 1 -- 172.21.15.50:0/1960903762 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c010430 con 0x7ffb181094b0 2024-04-23T18:24:02.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.251+0000 7ffb1e660640 1 -- 172.21.15.50:0/1960903762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb181a7820 con 0x7ffb181094b0 2024-04-23T18:24:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb157fa640 1 -- 172.21.15.50:0/1960903762 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffb0c004fd0 con 0x7ffb181094b0 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb21374740 1 Processor -- start 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb21374740 1 -- start start 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb21374740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b3f9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb21374740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b4042c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb21374740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 0x56042b406bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb21374740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56042b35e970 con 0x56042b3f9f40 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb21374740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56042b2fb4a0 con 0x56042b3f9620 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb21374740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56042b35a930 con 0x56042b404800 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb157fa640 1 --2- 172.21.15.50:0/1960903762 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ffb000a3e40 0x7ffb000a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb17fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b4042c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb17fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b4042c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59964/0 (socket says 172.21.15.50:59964) 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb1cbfd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b3f9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb17fff640 1 -- 172.21.15.50:0/2477628837 learned_addr learned my addr 172.21.15.50:0/2477628837 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb177fe640 1 --2- 172.21.15.50:0/2477628837 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 0x56042b406bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.254+0000 7ffb1cbfd640 1 --2- 172.21.15.50:0/1960903762 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ffb000a3e40 0x7ffb000a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.255+0000 7ffb157fa640 1 -- 172.21.15.50:0/1960903762 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7ffb0c0d0ab0 con 0x7ffb181094b0 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.255+0000 7ffb17fff640 1 -- 172.21.15.50:0/2477628837 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 msgr2=0x56042b406bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.255+0000 7ffb17fff640 1 --2- 172.21.15.50:0/2477628837 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 0x56042b406bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.255+0000 7ffb17fff640 1 -- 172.21.15.50:0/2477628837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 msgr2=0x56042b3f9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.255+0000 7ffb17fff640 1 --2- 172.21.15.50:0/2477628837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b3f9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.255+0000 7ffb17fff640 1 -- 172.21.15.50:0/2477628837 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56042b071210 con 0x56042b3f9f40 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.255+0000 7ffb17fff640 1 --2- 172.21.15.50:0/2477628837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b4042c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009d20 tx=0x7ffb08011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d2734e17d6f4f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.255+0000 7ffb1cbfd640 1 --2- 172.21.15.50:0/1960903762 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ffb000a3e40 0x7ffb000a6300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009a60 tx=0x7ffb100095c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.255+0000 7ffafe7fc640 1 -- 172.21.15.50:0/2477628837 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0801e070 con 0x56042b3f9f40 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.255+0000 7ffafe7fc640 1 -- 172.21.15.50:0/2477628837 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb08019ae0 con 0x56042b3f9f40 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.255+0000 7ffafe7fc640 1 -- 172.21.15.50:0/2477628837 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0800ea60 con 0x56042b3f9f40 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.256+0000 7ffb21374740 1 -- 172.21.15.50:0/2477628837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 msgr2=0x56042b4042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.256+0000 7ffb21374740 1 --2- 172.21.15.50:0/2477628837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b4042c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009d20 tx=0x7ffb08011810 comp rx=0 tx=0).stop 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.256+0000 7ffb21374740 1 -- 172.21.15.50:0/2477628837 shutdown_connections 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.256+0000 7ffb21374740 1 --2- 172.21.15.50:0/2477628837 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 0x56042b406bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.256+0000 7ffb21374740 1 --2- 172.21.15.50:0/2477628837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b4042c0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.256+0000 7ffb21374740 1 --2- 172.21.15.50:0/2477628837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b3f9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.256+0000 7ffb21374740 1 -- 172.21.15.50:0/2477628837 >> 172.21.15.50:0/2477628837 conn(0x56042b3f5b30 msgr2=0x56042b3f5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.256+0000 7ffb21374740 1 -- 172.21.15.50:0/2477628837 shutdown_connections 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.256+0000 7ffb21374740 1 -- 172.21.15.50:0/2477628837 wait complete. 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.256+0000 7ffb21374740 1 -- start start 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.256+0000 7ffb21374740 1 Processor -- start 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb21374740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b3f9620 0x56042b404100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb21374740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9f40 0x56042b4013f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb21374740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b404800 0x56042b401930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb21374740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56042b407430 con 0x56042b404800 2024-04-23T18:24:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb21374740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56042b2f90b0 con 0x56042b3f9f40 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb21374740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56042b4075b0 con 0x56042b3f9620 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb21374740 10 client.0 _ll_register_callbacks cb 0x56042b2e6d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb1cbfd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b404800 0x56042b401930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb177fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9f40 0x56042b4013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb17fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b3f9620 0x56042b404100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb177fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9f40 0x56042b4013f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:40218/0 (socket says 172.21.15.50:40218) 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb17fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b3f9620 0x56042b404100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:39400/0 (socket says 172.21.15.50:39400) 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb1cbfd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b404800 0x56042b401930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59974/0 (socket says 172.21.15.50:59974) 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb177fe640 1 -- 172.21.15.50:0/177416417 learned_addr learned my addr 172.21.15.50:0/177416417 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb177fe640 1 -- 172.21.15.50:0/177416417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b3f9620 msgr2=0x56042b404100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb177fe640 1 --2- 172.21.15.50:0/177416417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b3f9620 0x56042b404100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb177fe640 1 -- 172.21.15.50:0/177416417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b404800 msgr2=0x56042b401930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb177fe640 1 --2- 172.21.15.50:0/177416417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b404800 0x56042b401930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb177fe640 1 -- 172.21.15.50:0/177416417 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56042b13d620 con 0x56042b3f9f40 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb1cbfd640 1 --2- 172.21.15.50:0/177416417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b404800 0x56042b401930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffb177fe640 1 --2- 172.21.15.50:0/177416417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9f40 0x56042b4013f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0097b0 tx=0x7ffb0c06c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f89491a6617e1ebf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.257+0000 7ffaf77fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffaf77fe640 1 -- 172.21.15.50:0/177416417 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c0046c0 con 0x56042b3f9f40 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffb21374740 1 -- 172.21.15.50:0/177416417 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb08019ae0 con 0x56042b3f9f40 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffaf77fe640 1 -- 172.21.15.50:0/177416417 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7ffb0c0a5240 con 0x56042b3f9f40 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffaf77fe640 1 -- 172.21.15.50:0/177416417 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ffb0c0bd0b0 con 0x56042b3f9f40 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffb21374740 10 client.25772 fetch_fsmap learned FSMap version 30 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffb21374740 1 -- 172.21.15.50:0/177416417 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56042b413360 con 0x56042b3f9f40 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffaf77fe640 1 -- 172.21.15.50:0/177416417 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb08019ae0 con 0x56042b3f9f40 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffaf77fe640 1 -- 172.21.15.50:0/177416417 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 30) ==== 8350+0+0 (secure 0 0 0) 0x7ffb0c06e020 con 0x56042b3f9f40 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffb21374740 10 client.25772 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffb21374740 10 client.25772 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffb21374740 20 client.25772 populate_metadata read hostname 'smithi050' 2024-04-23T18:24:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffb21374740 1 --2- 172.21.15.50:0/177416417 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56042b4148e0 0x56042b416d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffb21374740 4 client.25772 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffb21374740 1 -- 172.21.15.50:0/177416417 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56042b36c390 con 0x56042b4148e0 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.258+0000 7ffb17fff640 1 --2- 172.21.15.50:0/177416417 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56042b4148e0 0x56042b416d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.259+0000 7ffaf77fe640 10 client.25772 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.259+0000 7ffb17fff640 1 --2- 172.21.15.50:0/177416417 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56042b4148e0 0x56042b416d10 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=855e0ff2af5c07f1 server_cookie=9359157cadc14e2c in_seq=0 out_seq=0 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.259+0000 7ffaf77fe640 1 -- 172.21.15.50:0/177416417 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56042b36c390 con 0x56042b4148e0 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.259+0000 7ffaf77fe640 10 client.25772 handle_command_reply: tid=0 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.259+0000 7ffb21374740 1 client.25772 shutdown 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.260+0000 7ffb21374740 1 -- 172.21.15.50:0/177416417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9f40 msgr2=0x56042b4013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.260+0000 7ffb21374740 1 --2- 172.21.15.50:0/177416417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9f40 0x56042b4013f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0097b0 tx=0x7ffb0c06c340 comp rx=0 tx=0).stop 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.260+0000 7ffb21374740 1 -- 172.21.15.50:0/177416417 shutdown_connections 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.260+0000 7ffb21374740 1 --2- 172.21.15.50:0/177416417 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56042b4148e0 0x56042b416d10 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.260+0000 7ffb21374740 1 --2- 172.21.15.50:0/177416417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b404800 0x56042b401930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.260+0000 7ffaf77fe640 0 client.25772 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.260+0000 7ffb21374740 1 --2- 172.21.15.50:0/177416417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9f40 0x56042b4013f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.260+0000 7ffb21374740 1 --2- 172.21.15.50:0/177416417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b3f9620 0x56042b404100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.260+0000 7ffb21374740 1 -- 172.21.15.50:0/177416417 >> 172.21.15.50:0/177416417 conn(0x56042b3f5b30 msgr2=0x56042b3fb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.260+0000 7ffb21374740 1 -- 172.21.15.50:0/177416417 shutdown_connections 2024-04-23T18:24:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.260+0000 7ffb21374740 1 -- 172.21.15.50:0/177416417 wait complete. 2024-04-23T18:24:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.260+0000 7ffb21374740 20 client.25772 trim_cache size 0 max 16384 2024-04-23T18:24:02.269 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.275+0000 7ffb21374740 1 Processor -- start 2024-04-23T18:24:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.275+0000 7ffb21374740 1 -- start start 2024-04-23T18:24:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.275+0000 7ffb21374740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b4102f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.275+0000 7ffb21374740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b41c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.275+0000 7ffb21374740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 0x56042b41cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.275+0000 7ffb21374740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181b25c0 con 0x56042b3f9f40 2024-04-23T18:24:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.275+0000 7ffb21374740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181afe30 con 0x56042b3f9620 2024-04-23T18:24:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.275+0000 7ffb21374740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181adca0 con 0x56042b404800 2024-04-23T18:24:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.275+0000 7ffb177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b41c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffb177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b41c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59978/0 (socket says 172.21.15.50:59978) 2024-04-23T18:24:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffb1cbfd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 0x56042b41cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffb177fe640 1 -- 172.21.15.50:0/2377488136 learned_addr learned my addr 172.21.15.50:0/2377488136 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffb17fff640 1 --2- 172.21.15.50:0/2377488136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b4102f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffb177fe640 1 -- 172.21.15.50:0/2377488136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 msgr2=0x56042b41cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffb177fe640 1 --2- 172.21.15.50:0/2377488136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 0x56042b41cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffb177fe640 1 -- 172.21.15.50:0/2377488136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 msgr2=0x56042b4102f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffb177fe640 1 --2- 172.21.15.50:0/2377488136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b4102f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffb177fe640 1 -- 172.21.15.50:0/2377488136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56042b3f6140 con 0x56042b3f9f40 2024-04-23T18:24:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffb177fe640 1 --2- 172.21.15.50:0/2377488136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b41c620 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0bd250 tx=0x7ffb0c06e430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28941615021bcdfe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffafdffb640 1 -- 172.21.15.50:0/2377488136 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c09c3c0 con 0x56042b3f9f40 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffafdffb640 1 -- 172.21.15.50:0/2377488136 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb0c0046c0 con 0x56042b3f9f40 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.276+0000 7ffafdffb640 1 -- 172.21.15.50:0/2377488136 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c0987b0 con 0x56042b3f9f40 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.277+0000 7ffb21374740 1 -- 172.21.15.50:0/2377488136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 msgr2=0x56042b41c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.277+0000 7ffb21374740 1 --2- 172.21.15.50:0/2377488136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b41c620 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0bd250 tx=0x7ffb0c06e430 comp rx=0 tx=0).stop 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.277+0000 7ffb21374740 1 -- 172.21.15.50:0/2377488136 shutdown_connections 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.277+0000 7ffb21374740 1 --2- 172.21.15.50:0/2377488136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 0x56042b41cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.277+0000 7ffb21374740 1 --2- 172.21.15.50:0/2377488136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b41c620 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.277+0000 7ffb21374740 1 --2- 172.21.15.50:0/2377488136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b4102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.277+0000 7ffb21374740 1 -- 172.21.15.50:0/2377488136 >> 172.21.15.50:0/2377488136 conn(0x56042b3f5b30 msgr2=0x56042b3f4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.277+0000 7ffb21374740 1 -- 172.21.15.50:0/2377488136 shutdown_connections 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.277+0000 7ffb21374740 1 -- 172.21.15.50:0/2377488136 wait complete. 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.277+0000 7ffb21374740 1 -- start start 2024-04-23T18:24:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb21374740 1 Processor -- start 2024-04-23T18:24:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb21374740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b40c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb21374740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b40c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb21374740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 0x56042b409990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb21374740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181ae2f0 con 0x56042b3f9f40 2024-04-23T18:24:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb21374740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181af760 con 0x56042b3f9620 2024-04-23T18:24:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb21374740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181ae9c0 con 0x56042b404800 2024-04-23T18:24:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb21374740 10 client.0 _ll_register_callbacks cb 0x7ffb0c09c830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb17fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b40c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b40c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb17fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b40c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:40240/0 (socket says 172.21.15.50:40240) 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb17fff640 1 -- 172.21.15.50:0/990195601 learned_addr learned my addr 172.21.15.50:0/990195601 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb1cbfd640 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 0x56042b409990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb17fff640 1 -- 172.21.15.50:0/990195601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 msgr2=0x56042b409990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb17fff640 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 0x56042b409990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb17fff640 1 -- 172.21.15.50:0/990195601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 msgr2=0x56042b40c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb17fff640 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b40c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb17fff640 1 -- 172.21.15.50:0/990195601 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56042b40a080 con 0x56042b3f9620 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb177fe640 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b40c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffb17fff640 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b40c470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009d20 tx=0x7ffb08009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1a6ebe604fa4a06 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.278+0000 7ffaf77fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:24:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffaf77fe640 1 -- 172.21.15.50:0/990195601 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0801e040 con 0x56042b3f9620 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffaf77fe640 1 -- 172.21.15.50:0/990195601 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7ffb08019430 con 0x56042b3f9620 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffb21374740 1 -- 172.21.15.50:0/990195601 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56042b366200 con 0x56042b3f9620 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffaf77fe640 1 -- 172.21.15.50:0/990195601 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ffb0803b050 con 0x56042b3f9620 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffb21374740 10 client.25778 fetch_fsmap learned FSMap version 30 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffb21374740 1 -- 172.21.15.50:0/990195601 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56042b47a900 con 0x56042b3f9620 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffaf77fe640 1 -- 172.21.15.50:0/990195601 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56042b366200 con 0x56042b3f9620 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffaf77fe640 1 -- 172.21.15.50:0/990195601 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 30) ==== 8350+0+0 (secure 0 0 0) 0x7ffb08003a00 con 0x56042b3f9620 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffb21374740 10 client.25778 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffb21374740 10 client.25778 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffb21374740 20 client.25778 populate_metadata read hostname 'smithi050' 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffb21374740 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56042b47af20 0x56042b47d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffb21374740 4 client.25778 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffb21374740 1 -- 172.21.15.50:0/990195601 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56042b2e25f0 con 0x56042b47af20 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.279+0000 7ffb177fe640 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56042b47af20 0x56042b47d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.280+0000 7ffaf77fe640 10 client.25778 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.280+0000 7ffb177fe640 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56042b47af20 0x56042b47d340 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83e68e4d1b78dc19 server_cookie=cecb79f0663ab5a1 in_seq=0 out_seq=0 2024-04-23T18:24:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.321+0000 7ffaf77fe640 1 -- 172.21.15.50:0/990195601 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56042b2e25f0 con 0x56042b47af20 2024-04-23T18:24:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.321+0000 7ffaf77fe640 10 client.25778 handle_command_reply: tid=0 2024-04-23T18:24:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.321+0000 7ffb21374740 1 client.25778 shutdown 2024-04-23T18:24:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb21374740 1 -- 172.21.15.50:0/990195601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 msgr2=0x56042b40c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb21374740 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b40c470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009d20 tx=0x7ffb08009ed0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb21374740 1 -- 172.21.15.50:0/990195601 shutdown_connections 2024-04-23T18:24:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb21374740 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56042b47af20 0x56042b47d340 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb21374740 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56042b404800 0x56042b409990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb21374740 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56042b3f9f40 0x56042b40c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb21374740 1 --2- 172.21.15.50:0/990195601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56042b3f9620 0x56042b40c470 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffaf77fe640 0 client.25778 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb21374740 1 -- 172.21.15.50:0/990195601 >> 172.21.15.50:0/990195601 conn(0x56042b3f5b30 msgr2=0x56042b3f4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb21374740 1 -- 172.21.15.50:0/990195601 shutdown_connections 2024-04-23T18:24:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb21374740 1 -- 172.21.15.50:0/990195601 wait complete. 2024-04-23T18:24:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb21374740 20 client.25778 trim_cache size 0 max 16384 2024-04-23T18:24:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:24:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:24:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:24:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:24:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb1e660640 1 -- 172.21.15.50:0/1960903762 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ffb000a3e40 msgr2=0x7ffb000a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb1e660640 1 --2- 172.21.15.50:0/1960903762 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ffb000a3e40 0x7ffb000a6300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009a60 tx=0x7ffb100095c0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb1e660640 1 -- 172.21.15.50:0/1960903762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb181094b0 msgr2=0x7ffb181a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb1e660640 1 --2- 172.21.15.50:0/1960903762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb181094b0 0x7ffb181a0b60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c00e9c0 tx=0x7ffb0c00ee80 comp rx=0 tx=0).stop 2024-04-23T18:24:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb1e660640 1 -- 172.21.15.50:0/1960903762 shutdown_connections 2024-04-23T18:24:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb1e660640 1 --2- 172.21.15.50:0/1960903762 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ffb000a3e40 0x7ffb000a6300 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb1e660640 1 --2- 172.21.15.50:0/1960903762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb1819a6e0 0x7ffb1819ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb1e660640 1 --2- 172.21.15.50:0/1960903762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb181094b0 0x7ffb181a0b60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb1e660640 1 --2- 172.21.15.50:0/1960903762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb181043e0 0x7ffb181a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.322+0000 7ffb1e660640 1 -- 172.21.15.50:0/1960903762 >> 172.21.15.50:0/1960903762 conn(0x7ffb180ffc60 msgr2=0x7ffb181007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.323+0000 7ffb1e660640 1 -- 172.21.15.50:0/1960903762 shutdown_connections 2024-04-23T18:24:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:02.323+0000 7ffb1e660640 1 -- 172.21.15.50:0/1960903762 wait complete. 2024-04-23T18:24:02.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:02 smithi050 ceph-mon[29187]: pgmap v170: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 277 op/s 2024-04-23T18:24:02.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:02 smithi177 ceph-mon[32308]: pgmap v170: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 277 op/s 2024-04-23T18:24:02.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:02 smithi188 ceph-mon[32694]: pgmap v170: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 277 op/s 2024-04-23T18:24:03.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 351984 5.40 MB/sec execute 157 sec latency 43.264 ms 2024-04-23T18:24:04.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 353918 5.43 MB/sec execute 158 sec latency 15.273 ms 2024-04-23T18:24:04.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:04 smithi050 ceph-mon[29187]: pgmap v171: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 383 op/s 2024-04-23T18:24:04.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:04 smithi177 ceph-mon[32308]: pgmap v171: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 383 op/s 2024-04-23T18:24:04.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:04 smithi188 ceph-mon[32694]: pgmap v171: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 383 op/s 2024-04-23T18:24:05.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 355738 5.46 MB/sec execute 159 sec latency 18.513 ms 2024-04-23T18:24:06.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 357713 5.49 MB/sec execute 160 sec latency 14.830 ms 2024-04-23T18:24:06.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:06 smithi050 ceph-mon[29187]: pgmap v172: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2024-04-23T18:24:06.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:06 smithi177 ceph-mon[32308]: pgmap v172: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2024-04-23T18:24:06.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:06 smithi188 ceph-mon[32694]: pgmap v172: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2024-04-23T18:24:07.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 359500 5.51 MB/sec execute 161 sec latency 28.319 ms 2024-04-23T18:24:08.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 361951 5.58 MB/sec execute 162 sec latency 36.067 ms 2024-04-23T18:24:08.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:08 smithi050 ceph-mon[29187]: pgmap v173: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 374 op/s 2024-04-23T18:24:08.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:08 smithi177 ceph-mon[32308]: pgmap v173: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 374 op/s 2024-04-23T18:24:08.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:08 smithi188 ceph-mon[32694]: pgmap v173: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 374 op/s 2024-04-23T18:24:09.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 363360 5.58 MB/sec execute 163 sec latency 26.534 ms 2024-04-23T18:24:10.207 INFO:tasks.workunit.client.0.smithi050.stdout: 1 365719 5.63 MB/sec execute 164 sec latency 15.218 ms 2024-04-23T18:24:10.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:10 smithi050 ceph-mon[29187]: pgmap v174: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-04-23T18:24:10.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:10 smithi177 ceph-mon[32308]: pgmap v174: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-04-23T18:24:10.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:10 smithi188 ceph-mon[32694]: pgmap v174: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-04-23T18:24:11.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 367035 5.63 MB/sec execute 165 sec latency 80.831 ms 2024-04-23T18:24:12.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 369253 5.68 MB/sec execute 166 sec latency 41.549 ms 2024-04-23T18:24:12.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:12 smithi050 ceph-mon[29187]: pgmap v175: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-04-23T18:24:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:12 smithi177 ceph-mon[32308]: pgmap v175: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-04-23T18:24:12.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:12 smithi188 ceph-mon[32694]: pgmap v175: 97 pgs: 97 active+clean; 121 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-04-23T18:24:13.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 371181 5.69 MB/sec execute 167 sec latency 21.430 ms 2024-04-23T18:24:14.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 373098 5.73 MB/sec execute 168 sec latency 22.904 ms 2024-04-23T18:24:14.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:14 smithi050 ceph-mon[29187]: pgmap v176: 97 pgs: 97 active+clean; 129 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 384 op/s 2024-04-23T18:24:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:14 smithi177 ceph-mon[32308]: pgmap v176: 97 pgs: 97 active+clean; 129 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 384 op/s 2024-04-23T18:24:14.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:14 smithi188 ceph-mon[32694]: pgmap v176: 97 pgs: 97 active+clean; 129 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 384 op/s 2024-04-23T18:24:15.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 374933 5.74 MB/sec execute 169 sec latency 26.620 ms 2024-04-23T18:24:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.413+0000 7fa92a70f640 1 -- 172.21.15.50:0/2208125416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9241031e0 msgr2=0x7fa9241035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.413+0000 7fa92a70f640 1 --2- 172.21.15.50:0/2208125416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9241031e0 0x7fa9241035c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa914009900 tx=0x7fa91402a510 comp rx=0 tx=0).stop 2024-04-23T18:24:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.413+0000 7fa92a70f640 1 -- 172.21.15.50:0/2208125416 shutdown_connections 2024-04-23T18:24:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.413+0000 7fa92a70f640 1 --2- 172.21.15.50:0/2208125416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa92410c520 0x7fa92410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.413+0000 7fa92a70f640 1 --2- 172.21.15.50:0/2208125416 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa924103b00 0x7fa92410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.413+0000 7fa92a70f640 1 --2- 172.21.15.50:0/2208125416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9241031e0 0x7fa9241035c0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.413+0000 7fa92a70f640 1 -- 172.21.15.50:0/2208125416 >> 172.21.15.50:0/2208125416 conn(0x7fa924077fd0 msgr2=0x7fa9240783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:15.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.414+0000 7fa92a70f640 1 -- 172.21.15.50:0/2208125416 shutdown_connections 2024-04-23T18:24:15.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.414+0000 7fa92a70f640 1 -- 172.21.15.50:0/2208125416 wait complete. 2024-04-23T18:24:15.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.414+0000 7fa92a70f640 1 Processor -- start 2024-04-23T18:24:15.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.414+0000 7fa92a70f640 1 -- start start 2024-04-23T18:24:15.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.414+0000 7fa92a70f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9241031e0 0x7fa92419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa92a70f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa924103b00 0x7fa92419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa92a70f640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa92410c520 0x7fa9241985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa92a70f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241121b0 con 0x7fa9241031e0 2024-04-23T18:24:15.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa92a70f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa924112030 con 0x7fa924103b00 2024-04-23T18:24:15.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa92a70f640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa924112330 con 0x7fa92410c520 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa923fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9241031e0 0x7fa92419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa923fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9241031e0 0x7fa92419e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53960/0 (socket says 172.21.15.50:53960) 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa9237fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa924103b00 0x7fa92419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa923fff640 1 -- 172.21.15.50:0/4064136714 learned_addr learned my addr 172.21.15.50:0/4064136714 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa928cac640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa92410c520 0x7fa9241985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa923fff640 1 -- 172.21.15.50:0/4064136714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa92410c520 msgr2=0x7fa9241985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa923fff640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa92410c520 0x7fa9241985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa923fff640 1 -- 172.21.15.50:0/4064136714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa924103b00 msgr2=0x7fa92419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa923fff640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa924103b00 0x7fa92419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa923fff640 1 -- 172.21.15.50:0/4064136714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa924198d40 con 0x7fa9241031e0 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa9237fe640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa924103b00 0x7fa92419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa923fff640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9241031e0 0x7fa92419e410 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa914005940 tx=0x7fa9140040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa9217fa640 1 -- 172.21.15.50:0/4064136714 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91403f030 con 0x7fa9241031e0 2024-04-23T18:24:15.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa92a70f640 1 -- 172.21.15.50:0/4064136714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa924198fd0 con 0x7fa9241031e0 2024-04-23T18:24:15.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa9217fa640 1 -- 172.21.15.50:0/4064136714 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9140044c0 con 0x7fa9241031e0 2024-04-23T18:24:15.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.415+0000 7fa9217fa640 1 -- 172.21.15.50:0/4064136714 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914035070 con 0x7fa9241031e0 2024-04-23T18:24:15.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.416+0000 7fa92a70f640 1 -- 172.21.15.50:0/4064136714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9241a52d0 con 0x7fa9241031e0 2024-04-23T18:24:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.418+0000 7fa9217fa640 1 -- 172.21.15.50:0/4064136714 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa914042030 con 0x7fa9241031e0 2024-04-23T18:24:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa92d421740 1 Processor -- start 2024-04-23T18:24:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa92d421740 1 -- start start 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa9217fa640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa90c0a3dc0 0x7fa90c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa92d421740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456efda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa92d421740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456efdf40 0x55d456f082c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa92d421740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456f08800 0x55d456f0abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa92d421740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d456e62970 con 0x55d456efd620 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa92d421740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d456dff4a0 con 0x55d456f08800 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa928cac640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa90c0a3dc0 0x7fa90c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa92d421740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d456e5e930 con 0x55d456efdf40 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa9217fa640 1 -- 172.21.15.50:0/4064136714 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7fa9140f0b00 con 0x7fa9241031e0 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa923fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456efda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa923fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456efda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53964/0 (socket says 172.21.15.50:53964) 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa923fff640 1 -- 172.21.15.50:0/2975899490 learned_addr learned my addr 172.21.15.50:0/2975899490 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa9237fe640 1 --2- 172.21.15.50:0/2975899490 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456efdf40 0x55d456f082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa928cac640 1 --2- 172.21.15.50:0/2975899490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456f08800 0x55d456f0abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa923fff640 1 -- 172.21.15.50:0/2975899490 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456efdf40 msgr2=0x55d456f082c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa923fff640 1 --2- 172.21.15.50:0/2975899490 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456efdf40 0x55d456f082c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa923fff640 1 -- 172.21.15.50:0/2975899490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456f08800 msgr2=0x55d456f0abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa923fff640 1 --2- 172.21.15.50:0/2975899490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456f08800 0x55d456f0abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa923fff640 1 -- 172.21.15.50:0/2975899490 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d456b75210 con 0x55d456efd620 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.419+0000 7fa928cac640 1 --2- 172.21.15.50:0/2975899490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456f08800 0x55d456f0abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.420+0000 7fa928cac640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa90c0a3dc0 0x7fa90c0a6280 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa91c005fd0 tx=0x7fa91c013b00 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.420+0000 7fa923fff640 1 --2- 172.21.15.50:0/2975899490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456efda00 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa9140c51d0 tx=0x7fa9140bc4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7590ac64040ac1be server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.420+0000 7fa90a7fc640 1 -- 172.21.15.50:0/2975899490 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914049070 con 0x55d456efd620 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.420+0000 7fa90a7fc640 1 -- 172.21.15.50:0/2975899490 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9140bca90 con 0x55d456efd620 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.420+0000 7fa90a7fc640 1 -- 172.21.15.50:0/2975899490 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91403b9f0 con 0x55d456efd620 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 -- 172.21.15.50:0/2975899490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 msgr2=0x55d456efda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 --2- 172.21.15.50:0/2975899490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456efda00 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa9140c51d0 tx=0x7fa9140bc4b0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 -- 172.21.15.50:0/2975899490 shutdown_connections 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 --2- 172.21.15.50:0/2975899490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456f08800 0x55d456f0abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 --2- 172.21.15.50:0/2975899490 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456efdf40 0x55d456f082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 --2- 172.21.15.50:0/2975899490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456efda00 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 -- 172.21.15.50:0/2975899490 >> 172.21.15.50:0/2975899490 conn(0x55d456ef9b30 msgr2=0x55d456ef9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 -- 172.21.15.50:0/2975899490 shutdown_connections 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 -- 172.21.15.50:0/2975899490 wait complete. 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 -- start start 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 Processor -- start 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456f081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 0x55d456f054a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456f08800 0x55d456f05a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d456f0b1f0 con 0x55d456efd620 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d456dfd0b0 con 0x55d456efdf40 2024-04-23T18:24:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.421+0000 7fa92d421740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d456f0b370 con 0x55d456f08800 2024-04-23T18:24:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa92d421740 10 client.0 _ll_register_callbacks cb 0x55d456dead30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa923fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456f081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa9237fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 0x55d456f054a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa923fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456f081b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53970/0 (socket says 172.21.15.50:53970) 2024-04-23T18:24:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa923fff640 1 -- 172.21.15.50:0/1347342478 learned_addr learned my addr 172.21.15.50:0/1347342478 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa928cac640 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456f08800 0x55d456f05a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa923fff640 1 -- 172.21.15.50:0/1347342478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456f08800 msgr2=0x55d456f05a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa923fff640 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456f08800 0x55d456f05a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa923fff640 1 -- 172.21.15.50:0/1347342478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 msgr2=0x55d456f054a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa923fff640 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 0x55d456f054a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa923fff640 1 -- 172.21.15.50:0/1347342478 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d456c41620 con 0x55d456efd620 2024-04-23T18:24:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa9237fe640 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 0x55d456f054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa923fff640 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456f081b0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa9140b9460 tx=0x7fa9140ee230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65faea81e089bec8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa9037fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa9037fe640 1 -- 172.21.15.50:0/1347342478 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914049070 con 0x55d456efd620 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa9037fe640 1 -- 172.21.15.50:0/1347342478 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7fa9140b9100 con 0x55d456efd620 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.422+0000 7fa92d421740 1 -- 172.21.15.50:0/1347342478 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9140bca90 con 0x55d456efd620 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.423+0000 7fa9037fe640 1 -- 172.21.15.50:0/1347342478 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa9140c5700 con 0x55d456efd620 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.423+0000 7fa92d421740 10 client.16188 fetch_fsmap learned FSMap version 30 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.423+0000 7fa92d421740 1 -- 172.21.15.50:0/1347342478 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d456f0b1f0 con 0x55d456efd620 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.423+0000 7fa9037fe640 1 -- 172.21.15.50:0/1347342478 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9140bca90 con 0x55d456efd620 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.423+0000 7fa9037fe640 1 -- 172.21.15.50:0/1347342478 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 30) ==== 8350+0+0 (secure 0 0 0) 0x7fa9140a70e0 con 0x55d456efd620 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.423+0000 7fa92d421740 10 client.16188 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.423+0000 7fa92d421740 10 client.16188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.423+0000 7fa92d421740 20 client.16188 populate_metadata read hostname 'smithi050' 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.423+0000 7fa92d421740 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d456f183a0 0x55d456f1a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.423+0000 7fa92d421740 4 client.16188 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.423+0000 7fa92d421740 1 -- 172.21.15.50:0/1347342478 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d456e70390 con 0x55d456f183a0 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.423+0000 7fa9237fe640 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d456f183a0 0x55d456f1a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa9237fe640 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d456f183a0 0x55d456f1a790 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db2dc398465504e5 server_cookie=db2b2cb4725ebafa in_seq=0 out_seq=0 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa9037fe640 10 client.16188 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa9037fe640 1 -- 172.21.15.50:0/1347342478 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d456e70390 con 0x55d456f183a0 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa9037fe640 10 client.16188 handle_command_reply: tid=0 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa92d421740 1 client.16188 shutdown 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa92d421740 1 -- 172.21.15.50:0/1347342478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 msgr2=0x55d456f081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa92d421740 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456f081b0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa9140b9460 tx=0x7fa9140ee230 comp rx=0 tx=0).stop 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa92d421740 1 -- 172.21.15.50:0/1347342478 shutdown_connections 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa92d421740 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d456f183a0 0x55d456f1a790 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa92d421740 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456f08800 0x55d456f05a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa92d421740 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 0x55d456f054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa9037fe640 0 client.16188 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa92d421740 1 --2- 172.21.15.50:0/1347342478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456f081b0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa92d421740 1 -- 172.21.15.50:0/1347342478 >> 172.21.15.50:0/1347342478 conn(0x55d456ef9b30 msgr2=0x55d456eff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.424+0000 7fa92d421740 1 -- 172.21.15.50:0/1347342478 shutdown_connections 2024-04-23T18:24:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.425+0000 7fa92d421740 1 -- 172.21.15.50:0/1347342478 wait complete. 2024-04-23T18:24:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.425+0000 7fa92d421740 20 client.16188 trim_cache size 0 max 16384 2024-04-23T18:24:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa92d421740 1 Processor -- start 2024-04-23T18:24:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa92d421740 1 -- start start 2024-04-23T18:24:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa92d421740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456efd620 0x55d456f14000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa92d421740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efdf40 0x55d456ef8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa92d421740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456f08800 0x55d456f07ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa92d421740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241b0490 con 0x55d456efdf40 2024-04-23T18:24:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa92d421740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241add00 con 0x55d456f08800 2024-04-23T18:24:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa92d421740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241abb50 con 0x55d456efd620 2024-04-23T18:24:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa9237fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efdf40 0x55d456ef8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa9237fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efdf40 0x55d456ef8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53980/0 (socket says 172.21.15.50:53980) 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa9237fe640 1 -- 172.21.15.50:0/1374800240 learned_addr learned my addr 172.21.15.50:0/1374800240 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa923fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456efd620 0x55d456f14000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa928cac640 1 --2- 172.21.15.50:0/1374800240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456f08800 0x55d456f07ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa9237fe640 1 -- 172.21.15.50:0/1374800240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456efd620 msgr2=0x55d456f14000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa9237fe640 1 --2- 172.21.15.50:0/1374800240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456efd620 0x55d456f14000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa9237fe640 1 -- 172.21.15.50:0/1374800240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456f08800 msgr2=0x55d456f07ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa9237fe640 1 --2- 172.21.15.50:0/1374800240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456f08800 0x55d456f07ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.440+0000 7fa9237fe640 1 -- 172.21.15.50:0/1374800240 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d456f04450 con 0x55d456efdf40 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.441+0000 7fa9237fe640 1 --2- 172.21.15.50:0/1374800240 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efdf40 0x55d456ef8b30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x55d456efebc0 tx=0x7fa918005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf7102bacb34b36 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.441+0000 7fa909ffb640 1 -- 172.21.15.50:0/1374800240 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918028070 con 0x55d456efdf40 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.441+0000 7fa909ffb640 1 -- 172.21.15.50:0/1374800240 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9180120d0 con 0x55d456efdf40 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.441+0000 7fa909ffb640 1 -- 172.21.15.50:0/1374800240 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9180227b0 con 0x55d456efdf40 2024-04-23T18:24:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.441+0000 7fa92d421740 1 -- 172.21.15.50:0/1374800240 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efdf40 msgr2=0x55d456ef8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:15.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.441+0000 7fa92d421740 1 --2- 172.21.15.50:0/1374800240 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efdf40 0x55d456ef8b30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x55d456efebc0 tx=0x7fa918005f90 comp rx=0 tx=0).stop 2024-04-23T18:24:15.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 -- 172.21.15.50:0/1374800240 shutdown_connections 2024-04-23T18:24:15.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 --2- 172.21.15.50:0/1374800240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456f08800 0x55d456f07ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 --2- 172.21.15.50:0/1374800240 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efdf40 0x55d456ef8b30 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 --2- 172.21.15.50:0/1374800240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456efd620 0x55d456f14000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 -- 172.21.15.50:0/1374800240 >> 172.21.15.50:0/1374800240 conn(0x55d456ef9b30 msgr2=0x55d456ef8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:15.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 -- 172.21.15.50:0/1374800240 shutdown_connections 2024-04-23T18:24:15.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 -- 172.21.15.50:0/1374800240 wait complete. 2024-04-23T18:24:15.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 -- start start 2024-04-23T18:24:15.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 Processor -- start 2024-04-23T18:24:15.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456f10610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 0x55d456f0b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456f08800 0x55d456f0bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241ac1c0 con 0x55d456efd620 2024-04-23T18:24:15.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241ad630 con 0x55d456efdf40 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241ac890 con 0x55d456f08800 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa92d421740 10 client.0 _ll_register_callbacks cb 0x7fa914009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa923fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456f10610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.442+0000 7fa9237fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 0x55d456f0b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa9237fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 0x55d456f0b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:36910/0 (socket says 172.21.15.50:36910) 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa923fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456f10610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53988/0 (socket says 172.21.15.50:53988) 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa9237fe640 1 -- 172.21.15.50:0/3218135484 learned_addr learned my addr 172.21.15.50:0/3218135484 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa928cac640 1 --2- 172.21.15.50:0/3218135484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456f08800 0x55d456f0bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa9237fe640 1 -- 172.21.15.50:0/3218135484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456f08800 msgr2=0x55d456f0bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa9237fe640 1 --2- 172.21.15.50:0/3218135484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456f08800 0x55d456f0bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa9237fe640 1 -- 172.21.15.50:0/3218135484 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 msgr2=0x55d456f10610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa9237fe640 1 --2- 172.21.15.50:0/3218135484 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456f10610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa9237fe640 1 -- 172.21.15.50:0/3218135484 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d456efdd70 con 0x55d456efdf40 2024-04-23T18:24:15.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa9237fe640 1 --2- 172.21.15.50:0/3218135484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 0x55d456f0b980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa918000c00 tx=0x7fa9180122a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f070f17ecc3d947b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa9037fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:24:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa9037fe640 1 -- 172.21.15.50:0/3218135484 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918028040 con 0x55d456efdf40 2024-04-23T18:24:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa9037fe640 1 -- 172.21.15.50:0/3218135484 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7fa91800fd00 con 0x55d456efdf40 2024-04-23T18:24:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa92d421740 1 -- 172.21.15.50:0/3218135484 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9180120d0 con 0x55d456efdf40 2024-04-23T18:24:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa9037fe640 1 -- 172.21.15.50:0/3218135484 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa91803a050 con 0x55d456efdf40 2024-04-23T18:24:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa92d421740 10 client.25796 fetch_fsmap learned FSMap version 30 2024-04-23T18:24:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.443+0000 7fa92d421740 1 -- 172.21.15.50:0/3218135484 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d456f7e440 con 0x55d456efdf40 2024-04-23T18:24:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.444+0000 7fa9037fe640 1 -- 172.21.15.50:0/3218135484 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9180120d0 con 0x55d456efdf40 2024-04-23T18:24:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.444+0000 7fa9037fe640 1 -- 172.21.15.50:0/3218135484 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 30) ==== 8350+0+0 (secure 0 0 0) 0x7fa91802b070 con 0x55d456efdf40 2024-04-23T18:24:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.444+0000 7fa92d421740 10 client.25796 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:24:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.444+0000 7fa92d421740 10 client.25796 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:15.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.444+0000 7fa92d421740 20 client.25796 populate_metadata read hostname 'smithi050' 2024-04-23T18:24:15.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.444+0000 7fa92d421740 1 --2- 172.21.15.50:0/3218135484 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d456f7eb90 0x55d456f80fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:15.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.444+0000 7fa92d421740 4 client.25796 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:15.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.444+0000 7fa92d421740 1 -- 172.21.15.50:0/3218135484 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d456e6a200 con 0x55d456f7eb90 2024-04-23T18:24:15.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.444+0000 7fa923fff640 1 --2- 172.21.15.50:0/3218135484 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d456f7eb90 0x55d456f80fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:15.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.444+0000 7fa923fff640 1 --2- 172.21.15.50:0/3218135484 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d456f7eb90 0x55d456f80fb0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa4bf21636d950e2 server_cookie=332ae1323ba2a7e2 in_seq=0 out_seq=0 2024-04-23T18:24:15.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.444+0000 7fa9037fe640 10 client.25796 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.498+0000 7fa9037fe640 1 -- 172.21.15.50:0/3218135484 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d456e6a200 con 0x55d456f7eb90 2024-04-23T18:24:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.498+0000 7fa9037fe640 10 client.25796 handle_command_reply: tid=0 2024-04-23T18:24:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.498+0000 7fa92d421740 1 client.25796 shutdown 2024-04-23T18:24:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.499+0000 7fa92d421740 1 -- 172.21.15.50:0/3218135484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 msgr2=0x55d456f0b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.499+0000 7fa92d421740 1 --2- 172.21.15.50:0/3218135484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 0x55d456f0b980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa918000c00 tx=0x7fa9180122a0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.499+0000 7fa92d421740 1 -- 172.21.15.50:0/3218135484 shutdown_connections 2024-04-23T18:24:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.499+0000 7fa92d421740 1 --2- 172.21.15.50:0/3218135484 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d456f7eb90 0x55d456f80fb0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.499+0000 7fa92d421740 1 --2- 172.21.15.50:0/3218135484 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d456f08800 0x55d456f0bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.499+0000 7fa9037fe640 0 client.25796 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.499+0000 7fa92d421740 1 --2- 172.21.15.50:0/3218135484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d456efdf40 0x55d456f0b980 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.499+0000 7fa92d421740 1 --2- 172.21.15.50:0/3218135484 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d456efd620 0x55d456f10610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.499+0000 7fa92d421740 1 -- 172.21.15.50:0/3218135484 >> 172.21.15.50:0/3218135484 conn(0x55d456ef9b30 msgr2=0x55d456ef8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.499+0000 7fa92d421740 1 -- 172.21.15.50:0/3218135484 shutdown_connections 2024-04-23T18:24:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.499+0000 7fa92d421740 1 -- 172.21.15.50:0/3218135484 wait complete. 2024-04-23T18:24:15.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.499+0000 7fa92d421740 20 client.25796 trim_cache size 0 max 16384 2024-04-23T18:24:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:24:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:24:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:24:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:24:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.500+0000 7fa92a70f640 1 -- 172.21.15.50:0/4064136714 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa90c0a3dc0 msgr2=0x7fa90c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.500+0000 7fa92a70f640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa90c0a3dc0 0x7fa90c0a6280 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa91c005fd0 tx=0x7fa91c013b00 comp rx=0 tx=0).stop 2024-04-23T18:24:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.500+0000 7fa92a70f640 1 -- 172.21.15.50:0/4064136714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9241031e0 msgr2=0x7fa92419e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.500+0000 7fa92a70f640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9241031e0 0x7fa92419e410 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa914005940 tx=0x7fa9140040c0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.500+0000 7fa92a70f640 1 -- 172.21.15.50:0/4064136714 shutdown_connections 2024-04-23T18:24:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.500+0000 7fa92a70f640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa90c0a3dc0 0x7fa90c0a6280 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.500+0000 7fa92a70f640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa92410c520 0x7fa9241985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.500+0000 7fa92a70f640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa924103b00 0x7fa92419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.500+0000 7fa92a70f640 1 --2- 172.21.15.50:0/4064136714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9241031e0 0x7fa92419e410 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.500+0000 7fa92a70f640 1 -- 172.21.15.50:0/4064136714 >> 172.21.15.50:0/4064136714 conn(0x7fa924077fd0 msgr2=0x7fa9240ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.500+0000 7fa92a70f640 1 -- 172.21.15.50:0/4064136714 shutdown_connections 2024-04-23T18:24:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:15.500+0000 7fa92a70f640 1 -- 172.21.15.50:0/4064136714 wait complete. 2024-04-23T18:24:16.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 376663 5.77 MB/sec execute 170 sec latency 52.066 ms 2024-04-23T18:24:16.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:16 smithi050 ceph-mon[29187]: pgmap v177: 97 pgs: 97 active+clean; 129 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 243 op/s 2024-04-23T18:24:16.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:16 smithi177 ceph-mon[32308]: pgmap v177: 97 pgs: 97 active+clean; 129 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 243 op/s 2024-04-23T18:24:16.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:16 smithi188 ceph-mon[32694]: pgmap v177: 97 pgs: 97 active+clean; 129 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 243 op/s 2024-04-23T18:24:17.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 378468 5.78 MB/sec execute 171 sec latency 46.221 ms 2024-04-23T18:24:18.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 380389 5.82 MB/sec execute 172 sec latency 26.417 ms 2024-04-23T18:24:18.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:18 smithi050 ceph-mon[29187]: pgmap v178: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 396 op/s 2024-04-23T18:24:18.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:18 smithi177 ceph-mon[32308]: pgmap v178: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 396 op/s 2024-04-23T18:24:18.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:18 smithi188 ceph-mon[32694]: pgmap v178: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 396 op/s 2024-04-23T18:24:19.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 382440 5.85 MB/sec execute 173 sec latency 14.553 ms 2024-04-23T18:24:20.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 384288 5.87 MB/sec execute 174 sec latency 21.428 ms 2024-04-23T18:24:20.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:20 smithi050 ceph-mon[29187]: pgmap v179: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 264 op/s 2024-04-23T18:24:20.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:20 smithi177 ceph-mon[32308]: pgmap v179: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 264 op/s 2024-04-23T18:24:20.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:20 smithi188 ceph-mon[32694]: pgmap v179: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 264 op/s 2024-04-23T18:24:21.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 386116 5.90 MB/sec execute 175 sec latency 13.880 ms 2024-04-23T18:24:22.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 387864 5.91 MB/sec execute 176 sec latency 42.072 ms 2024-04-23T18:24:22.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:22 smithi050 ceph-mon[29187]: pgmap v180: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 264 op/s 2024-04-23T18:24:22.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:22 smithi177 ceph-mon[32308]: pgmap v180: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 264 op/s 2024-04-23T18:24:22.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:22 smithi188 ceph-mon[32694]: pgmap v180: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 264 op/s 2024-04-23T18:24:23.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 389233 5.91 MB/sec execute 177 sec latency 44.268 ms 2024-04-23T18:24:24.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 391188 5.95 MB/sec execute 178 sec latency 22.555 ms 2024-04-23T18:24:24.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:24 smithi050 ceph-mon[29187]: pgmap v181: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-04-23T18:24:24.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:24 smithi177 ceph-mon[32308]: pgmap v181: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-04-23T18:24:24.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:24 smithi188 ceph-mon[32694]: pgmap v181: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-04-23T18:24:25.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 392650 5.95 MB/sec execute 179 sec latency 40.791 ms 2024-04-23T18:24:26.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 394488 5.98 MB/sec execute 180 sec latency 80.829 ms 2024-04-23T18:24:26.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:26 smithi050 ceph-mon[29187]: pgmap v182: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 252 op/s 2024-04-23T18:24:26.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:26 smithi177 ceph-mon[32308]: pgmap v182: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 252 op/s 2024-04-23T18:24:26.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:26 smithi188 ceph-mon[32694]: pgmap v182: 97 pgs: 97 active+clean; 124 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 252 op/s 2024-04-23T18:24:27.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 396175 5.99 MB/sec execute 181 sec latency 62.515 ms 2024-04-23T18:24:28.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 397995 6.02 MB/sec execute 182 sec latency 43.632 ms 2024-04-23T18:24:28.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:28 smithi050 ceph-mon[29187]: pgmap v183: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 356 op/s 2024-04-23T18:24:28.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:28 smithi177 ceph-mon[32308]: pgmap v183: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 356 op/s 2024-04-23T18:24:28.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:28 smithi188 ceph-mon[32694]: pgmap v183: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 356 op/s 2024-04-23T18:24:28.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.591+0000 7f104a0d0640 1 -- 172.21.15.50:0/3780261521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10441031c0 msgr2=0x7f10441035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:28.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.591+0000 7f104a0d0640 1 --2- 172.21.15.50:0/3780261521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10441031c0 0x7f10441035a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1034009900 tx=0x7f103402a480 comp rx=0 tx=0).stop 2024-04-23T18:24:28.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.591+0000 7f104a0d0640 1 -- 172.21.15.50:0/3780261521 shutdown_connections 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.591+0000 7f104a0d0640 1 --2- 172.21.15.50:0/3780261521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f104410c590 0x7f104410e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.591+0000 7f104a0d0640 1 --2- 172.21.15.50:0/3780261521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1044103ae0 0x7f104410bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.591+0000 7f104a0d0640 1 --2- 172.21.15.50:0/3780261521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10441031c0 0x7f10441035a0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.591+0000 7f104a0d0640 1 -- 172.21.15.50:0/3780261521 >> 172.21.15.50:0/3780261521 conn(0x7f1044077fa0 msgr2=0x7f10440783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.591+0000 7f104a0d0640 1 -- 172.21.15.50:0/3780261521 shutdown_connections 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.591+0000 7f104a0d0640 1 -- 172.21.15.50:0/3780261521 wait complete. 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.591+0000 7f104a0d0640 1 Processor -- start 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.591+0000 7f104a0d0640 1 -- start start 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f104a0d0640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10441031c0 0x7f10441a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f104a0d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1044103ae0 0x7f10441a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f104a0d0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f104410c590 0x7f104419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f104a0d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f10441121d0 con 0x7f1044103ae0 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f104a0d0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1044112050 con 0x7f104410c590 2024-04-23T18:24:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f104a0d0640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1044112350 con 0x7f10441031c0 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f1042ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1044103ae0 0x7f10441a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f10437fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10441031c0 0x7f10441a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f1042ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1044103ae0 0x7f10441a0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56858/0 (socket says 172.21.15.50:56858) 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f1042ffd640 1 -- 172.21.15.50:0/2721836978 learned_addr learned my addr 172.21.15.50:0/2721836978 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f1043fff640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f104410c590 0x7f104419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f1042ffd640 1 -- 172.21.15.50:0/2721836978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10441031c0 msgr2=0x7f10441a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f1042ffd640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10441031c0 0x7f10441a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f1042ffd640 1 -- 172.21.15.50:0/2721836978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f104410c590 msgr2=0x7f104419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f1042ffd640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f104410c590 0x7f104419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f1042ffd640 1 -- 172.21.15.50:0/2721836978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104419afd0 con 0x7f1044103ae0 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f1042ffd640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1044103ae0 0x7f10441a0b40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f103800d770 tx=0x7f103800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f1040ff9640 1 -- 172.21.15.50:0/2721836978 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038017070 con 0x7f1044103ae0 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f1040ff9640 1 -- 172.21.15.50:0/2721836978 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10380054d0 con 0x7f1044103ae0 2024-04-23T18:24:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.592+0000 7f104a0d0640 1 -- 172.21.15.50:0/2721836978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f104419b2c0 con 0x7f1044103ae0 2024-04-23T18:24:28.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.593+0000 7f1040ff9640 1 -- 172.21.15.50:0/2721836978 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038004c30 con 0x7f1044103ae0 2024-04-23T18:24:28.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.593+0000 7f104a0d0640 1 -- 172.21.15.50:0/2721836978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10441a7580 con 0x7f1044103ae0 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.595+0000 7f1040ff9640 1 -- 172.21.15.50:0/2721836978 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1038004e10 con 0x7f1044103ae0 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.595+0000 7f104cde2740 1 Processor -- start 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.595+0000 7f104cde2740 1 -- start start 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.595+0000 7f104cde2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291148f620 0x56291148fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.595+0000 7f104cde2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148ff40 0x56291149a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.595+0000 7f104cde2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291149a800 0x56291149cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.595+0000 7f104cde2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5629113f4970 con 0x56291149a800 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.595+0000 7f104cde2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5629113914a0 con 0x56291148f620 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.595+0000 7f104cde2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5629113f0930 con 0x56291148ff40 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.595+0000 7f1042ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291149a800 0x56291149cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f1040ff9640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f102c0a3dc0 0x7f102c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f1042ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291149a800 0x56291149cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56864/0 (socket says 172.21.15.50:56864) 2024-04-23T18:24:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f1042ffd640 1 -- 172.21.15.50:0/207506154 learned_addr learned my addr 172.21.15.50:0/207506154 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f10437fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148ff40 0x56291149a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f1043fff640 1 --2- 172.21.15.50:0/207506154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291148f620 0x56291148fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f1043fff640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f102c0a3dc0 0x7f102c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f10437fe640 1 -- 172.21.15.50:0/207506154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291148f620 msgr2=0x56291148fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f10437fe640 1 --2- 172.21.15.50:0/207506154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291148f620 0x56291148fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f10437fe640 1 -- 172.21.15.50:0/207506154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291149a800 msgr2=0x56291149cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f10437fe640 1 --2- 172.21.15.50:0/207506154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291149a800 0x56291149cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f10437fe640 1 -- 172.21.15.50:0/207506154 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562911107210 con 0x56291148ff40 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f1040ff9640 1 -- 172.21.15.50:0/2721836978 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f10380d0270 con 0x7f1044103ae0 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f10437fe640 1 --2- 172.21.15.50:0/207506154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148ff40 0x56291149a2c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1034009230 tx=0x7f1034032bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3da5a97597cce08 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f1043fff640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f102c0a3dc0 0x7f102c0a6280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f103c009fd0 tx=0x7f103c004ab0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f1029ffb640 1 -- 172.21.15.50:0/207506154 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1034040070 con 0x56291148ff40 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f1029ffb640 1 -- 172.21.15.50:0/207506154 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1034004030 con 0x56291148ff40 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.596+0000 7f1029ffb640 1 -- 172.21.15.50:0/207506154 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1034035040 con 0x56291148ff40 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.597+0000 7f104cde2740 1 -- 172.21.15.50:0/207506154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148ff40 msgr2=0x56291149a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.597+0000 7f104cde2740 1 --2- 172.21.15.50:0/207506154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148ff40 0x56291149a2c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1034009230 tx=0x7f1034032bd0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.597+0000 7f104cde2740 1 -- 172.21.15.50:0/207506154 shutdown_connections 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.597+0000 7f104cde2740 1 --2- 172.21.15.50:0/207506154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291149a800 0x56291149cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.597+0000 7f104cde2740 1 --2- 172.21.15.50:0/207506154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148ff40 0x56291149a2c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.597+0000 7f104cde2740 1 --2- 172.21.15.50:0/207506154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291148f620 0x56291148fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.597+0000 7f104cde2740 1 -- 172.21.15.50:0/207506154 >> 172.21.15.50:0/207506154 conn(0x56291148bb30 msgr2=0x56291148bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.597+0000 7f104cde2740 1 -- 172.21.15.50:0/207506154 shutdown_connections 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.597+0000 7f104cde2740 1 -- 172.21.15.50:0/207506154 wait complete. 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.597+0000 7f104cde2740 1 -- start start 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f104cde2740 1 Processor -- start 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f104cde2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 0x56291149a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f104cde2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148ff40 0x562911497480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f104cde2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291149a800 0x562911497a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f104cde2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56291149d3d0 con 0x56291148ff40 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f104cde2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56291138f0b0 con 0x56291149a800 2024-04-23T18:24:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f104cde2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56291149d550 con 0x56291148f620 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f104cde2740 10 client.0 _ll_register_callbacks cb 0x56291137cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f10437fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 0x56291149a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f1042ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148ff40 0x562911497480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f1043fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291149a800 0x562911497a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f1042ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148ff40 0x562911497480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56876/0 (socket says 172.21.15.50:56876) 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f10437fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 0x56291149a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:53658/0 (socket says 172.21.15.50:53658) 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f1042ffd640 1 -- 172.21.15.50:0/3425272641 learned_addr learned my addr 172.21.15.50:0/3425272641 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f1042ffd640 1 -- 172.21.15.50:0/3425272641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 msgr2=0x56291149a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f1042ffd640 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 0x56291149a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f1042ffd640 1 -- 172.21.15.50:0/3425272641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291149a800 msgr2=0x562911497a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f1042ffd640 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291149a800 0x562911497a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f1042ffd640 1 -- 172.21.15.50:0/3425272641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629111d3620 con 0x56291148ff40 2024-04-23T18:24:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f10437fe640 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 0x56291149a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f1043fff640 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291149a800 0x562911497a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f1042ffd640 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148ff40 0x562911497480 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1038009610 tx=0x7f103809bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817546e0ef2daeba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.598+0000 7f1022ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f1022ffd640 1 -- 172.21.15.50:0/3425272641 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038020040 con 0x56291148ff40 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f1022ffd640 1 -- 172.21.15.50:0/3425272641 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f10380cdbf0 con 0x56291148ff40 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f104cde2740 1 -- 172.21.15.50:0/3425272641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1034004030 con 0x56291148ff40 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f1022ffd640 1 -- 172.21.15.50:0/3425272641 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f10380cddf0 con 0x56291148ff40 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f104cde2740 10 client.16206 fetch_fsmap learned FSMap version 30 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f104cde2740 1 -- 172.21.15.50:0/3425272641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5629114a9320 con 0x56291148ff40 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f1022ffd640 1 -- 172.21.15.50:0/3425272641 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1034004030 con 0x56291148ff40 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f1022ffd640 1 -- 172.21.15.50:0/3425272641 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 30) ==== 8350+0+0 (secure 0 0 0) 0x7f10380a49a0 con 0x56291148ff40 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f104cde2740 10 client.16206 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f104cde2740 10 client.16206 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f104cde2740 20 client.16206 populate_metadata read hostname 'smithi050' 2024-04-23T18:24:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f104cde2740 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5629114aa830 0x5629114acc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f104cde2740 4 client.16206 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f104cde2740 1 -- 172.21.15.50:0/3425272641 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562911402390 con 0x5629114aa830 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.599+0000 7f10437fe640 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5629114aa830 0x5629114acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.600+0000 7f10437fe640 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5629114aa830 0x5629114acc60 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a568b96267422bd server_cookie=a12193efc26b0433 in_seq=0 out_seq=0 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.600+0000 7f1022ffd640 10 client.16206 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.600+0000 7f1022ffd640 1 -- 172.21.15.50:0/3425272641 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562911402390 con 0x5629114aa830 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.600+0000 7f1022ffd640 10 client.16206 handle_command_reply: tid=0 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.600+0000 7f104cde2740 1 client.16206 shutdown 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.601+0000 7f104cde2740 1 -- 172.21.15.50:0/3425272641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148ff40 msgr2=0x562911497480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.601+0000 7f104cde2740 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148ff40 0x562911497480 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1038009610 tx=0x7f103809bc90 comp rx=0 tx=0).stop 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.601+0000 7f104cde2740 1 -- 172.21.15.50:0/3425272641 shutdown_connections 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.601+0000 7f104cde2740 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5629114aa830 0x5629114acc60 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.601+0000 7f104cde2740 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291149a800 0x562911497a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.601+0000 7f104cde2740 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148ff40 0x562911497480 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.601+0000 7f104cde2740 1 --2- 172.21.15.50:0/3425272641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 0x56291149a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.601+0000 7f1022ffd640 0 client.16206 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.601+0000 7f104cde2740 1 -- 172.21.15.50:0/3425272641 >> 172.21.15.50:0/3425272641 conn(0x56291148bb30 msgr2=0x562911491860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.601+0000 7f104cde2740 1 -- 172.21.15.50:0/3425272641 shutdown_connections 2024-04-23T18:24:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.601+0000 7f104cde2740 1 -- 172.21.15.50:0/3425272641 wait complete. 2024-04-23T18:24:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.601+0000 7f104cde2740 20 client.16206 trim_cache size 0 max 16384 2024-04-23T18:24:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f104cde2740 1 Processor -- start 2024-04-23T18:24:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f104cde2740 1 -- start start 2024-04-23T18:24:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f104cde2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148f620 0x5629114a6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f104cde2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148ff40 0x56291148acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f104cde2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291149a800 0x5629114b2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f104cde2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f10441b2640 con 0x56291148f620 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f104cde2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10441afeb0 con 0x56291149a800 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f104cde2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f10441add00 con 0x56291148ff40 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f10437fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148f620 0x5629114a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f10437fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148f620 0x5629114a6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56892/0 (socket says 172.21.15.50:56892) 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f10437fe640 1 -- 172.21.15.50:0/899693059 learned_addr learned my addr 172.21.15.50:0/899693059 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f1042ffd640 1 --2- 172.21.15.50:0/899693059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148ff40 0x56291148acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f1043fff640 1 --2- 172.21.15.50:0/899693059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291149a800 0x5629114b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f1042ffd640 1 -- 172.21.15.50:0/899693059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291149a800 msgr2=0x5629114b2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f1042ffd640 1 --2- 172.21.15.50:0/899693059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291149a800 0x5629114b2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f1042ffd640 1 -- 172.21.15.50:0/899693059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148f620 msgr2=0x5629114a6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f1042ffd640 1 --2- 172.21.15.50:0/899693059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148f620 0x5629114a6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f1042ffd640 1 -- 172.21.15.50:0/899693059 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562911496430 con 0x56291148ff40 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.616+0000 7f10437fe640 1 --2- 172.21.15.50:0/899693059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148f620 0x5629114a6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.617+0000 7f1042ffd640 1 --2- 172.21.15.50:0/899693059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148ff40 0x56291148acb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1038009f40 tx=0x7f103809a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bb0cbc49961246a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.617+0000 7f10297fa640 1 -- 172.21.15.50:0/899693059 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10380988e0 con 0x56291148ff40 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.617+0000 7f10297fa640 1 -- 172.21.15.50:0/899693059 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10380a0120 con 0x56291148ff40 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.617+0000 7f10297fa640 1 -- 172.21.15.50:0/899693059 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038098510 con 0x56291148ff40 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.617+0000 7f104cde2740 1 -- 172.21.15.50:0/899693059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148ff40 msgr2=0x56291148acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.617+0000 7f104cde2740 1 --2- 172.21.15.50:0/899693059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148ff40 0x56291148acb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1038009f40 tx=0x7f103809a780 comp rx=0 tx=0).stop 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 -- 172.21.15.50:0/899693059 shutdown_connections 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 --2- 172.21.15.50:0/899693059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291149a800 0x5629114b2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 --2- 172.21.15.50:0/899693059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148ff40 0x56291148acb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 --2- 172.21.15.50:0/899693059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291148f620 0x5629114a6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 -- 172.21.15.50:0/899693059 >> 172.21.15.50:0/899693059 conn(0x56291148bb30 msgr2=0x56291148bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 -- 172.21.15.50:0/899693059 shutdown_connections 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 -- 172.21.15.50:0/899693059 wait complete. 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 -- start start 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 Processor -- start 2024-04-23T18:24:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 0x5629114a0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291148ff40 0x5629114a0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291149a800 0x5629114a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f10441ae370 con 0x56291149a800 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10441af7e0 con 0x56291148ff40 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f10441aea40 con 0x56291148f620 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f104cde2740 10 client.0 _ll_register_callbacks cb 0x7f103809bdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f1042ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291148ff40 0x5629114a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f10437fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 0x5629114a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.618+0000 7f1043fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291149a800 0x5629114a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f10437fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 0x5629114a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:53676/0 (socket says 172.21.15.50:53676) 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f10437fe640 1 -- 172.21.15.50:0/1979726824 learned_addr learned my addr 172.21.15.50:0/1979726824 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f1042ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291148ff40 0x5629114a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:58128/0 (socket says 172.21.15.50:58128) 2024-04-23T18:24:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f10437fe640 1 -- 172.21.15.50:0/1979726824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291148ff40 msgr2=0x5629114a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f10437fe640 1 --2- 172.21.15.50:0/1979726824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291148ff40 0x5629114a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f10437fe640 1 -- 172.21.15.50:0/1979726824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291149a800 msgr2=0x5629114a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f10437fe640 1 --2- 172.21.15.50:0/1979726824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291149a800 0x5629114a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f10437fe640 1 -- 172.21.15.50:0/1979726824 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56291148fd70 con 0x56291148f620 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f1043fff640 1 --2- 172.21.15.50:0/1979726824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291149a800 0x5629114a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f10437fe640 1 --2- 172.21.15.50:0/1979726824 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 0x5629114a0150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1034009230 tx=0x7f103402a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=239d334e3cb26660 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f1022ffd640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f1022ffd640 1 -- 172.21.15.50:0/1979726824 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1034054030 con 0x56291148f620 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f104cde2740 1 -- 172.21.15.50:0/1979726824 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10380a0120 con 0x56291148f620 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f1022ffd640 1 -- 172.21.15.50:0/1979726824 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f1034004470 con 0x56291148f620 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f1022ffd640 1 -- 172.21.15.50:0/1979726824 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1034040040 con 0x56291148f620 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f104cde2740 10 client.25876 fetch_fsmap learned FSMap version 30 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.619+0000 7f104cde2740 1 -- 172.21.15.50:0/1979726824 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562911510e90 con 0x56291148f620 2024-04-23T18:24:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.620+0000 7f1022ffd640 1 -- 172.21.15.50:0/1979726824 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10380a0120 con 0x56291148f620 2024-04-23T18:24:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.620+0000 7f1022ffd640 1 -- 172.21.15.50:0/1979726824 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 30) ==== 8350+0+0 (secure 0 0 0) 0x7f1034050480 con 0x56291148f620 2024-04-23T18:24:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.620+0000 7f104cde2740 10 client.25876 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:24:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.620+0000 7f104cde2740 10 client.25876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.620+0000 7f104cde2740 20 client.25876 populate_metadata read hostname 'smithi050' 2024-04-23T18:24:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.620+0000 7f104cde2740 1 --2- 172.21.15.50:0/1979726824 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5629115112e0 0x5629115136d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.620+0000 7f104cde2740 4 client.25876 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.620+0000 7f104cde2740 1 -- 172.21.15.50:0/1979726824 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5629113fc200 con 0x5629115112e0 2024-04-23T18:24:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.620+0000 7f1042ffd640 1 --2- 172.21.15.50:0/1979726824 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5629115112e0 0x5629115136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.620+0000 7f1042ffd640 1 --2- 172.21.15.50:0/1979726824 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5629115112e0 0x5629115136d0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99d764d130dcc268 server_cookie=31aee9f55e48f7ba in_seq=0 out_seq=0 2024-04-23T18:24:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.620+0000 7f1022ffd640 10 client.25876 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.673+0000 7f1022ffd640 1 -- 172.21.15.50:0/1979726824 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5629113fc200 con 0x5629115112e0 2024-04-23T18:24:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.673+0000 7f1022ffd640 10 client.25876 handle_command_reply: tid=0 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.673+0000 7f104cde2740 1 client.25876 shutdown 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.673+0000 7f104cde2740 1 -- 172.21.15.50:0/1979726824 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 msgr2=0x5629114a0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.673+0000 7f104cde2740 1 --2- 172.21.15.50:0/1979726824 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 0x5629114a0150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1034009230 tx=0x7f103402a480 comp rx=0 tx=0).stop 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.673+0000 7f104cde2740 1 -- 172.21.15.50:0/1979726824 shutdown_connections 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.673+0000 7f104cde2740 1 --2- 172.21.15.50:0/1979726824 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5629115112e0 0x5629115136d0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.673+0000 7f104cde2740 1 --2- 172.21.15.50:0/1979726824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56291149a800 0x5629114a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.673+0000 7f104cde2740 1 --2- 172.21.15.50:0/1979726824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56291148ff40 0x5629114a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104cde2740 1 --2- 172.21.15.50:0/1979726824 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56291148f620 0x5629114a0150 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f1022ffd640 0 client.25876 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104cde2740 1 -- 172.21.15.50:0/1979726824 >> 172.21.15.50:0/1979726824 conn(0x56291148bb30 msgr2=0x56291148acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104cde2740 1 -- 172.21.15.50:0/1979726824 shutdown_connections 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104cde2740 1 -- 172.21.15.50:0/1979726824 wait complete. 2024-04-23T18:24:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104cde2740 20 client.25876 trim_cache size 0 max 16384 2024-04-23T18:24:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:24:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:24:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:24:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:24:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104a0d0640 1 -- 172.21.15.50:0/2721836978 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f102c0a3dc0 msgr2=0x7f102c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104a0d0640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f102c0a3dc0 0x7f102c0a6280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f103c009fd0 tx=0x7f103c004ab0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104a0d0640 1 -- 172.21.15.50:0/2721836978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1044103ae0 msgr2=0x7f10441a0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104a0d0640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1044103ae0 0x7f10441a0b40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f103800d770 tx=0x7f103800dc30 comp rx=0 tx=0).stop 2024-04-23T18:24:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104a0d0640 1 -- 172.21.15.50:0/2721836978 shutdown_connections 2024-04-23T18:24:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104a0d0640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f102c0a3dc0 0x7f102c0a6280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104a0d0640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f104410c590 0x7f104419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104a0d0640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1044103ae0 0x7f10441a0b40 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104a0d0640 1 --2- 172.21.15.50:0/2721836978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10441031c0 0x7f10441a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.674+0000 7f104a0d0640 1 -- 172.21.15.50:0/2721836978 >> 172.21.15.50:0/2721836978 conn(0x7f1044077fa0 msgr2=0x7f1044108410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.675+0000 7f104a0d0640 1 -- 172.21.15.50:0/2721836978 shutdown_connections 2024-04-23T18:24:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:28.675+0000 7f104a0d0640 1 -- 172.21.15.50:0/2721836978 wait complete. 2024-04-23T18:24:29.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 399900 6.03 MB/sec execute 183 sec latency 22.593 ms 2024-04-23T18:24:30.189 INFO:tasks.quiescer.fs.[cephfs]:Total duration (59.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:24:30.189 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.55 2024-04-23T18:24:30.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:24:30.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 401930 6.07 MB/sec execute 184 sec latency 21.167 ms 2024-04-23T18:24:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.300+0000 7f434cd7c640 1 -- 172.21.15.50:0/3165731831 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43481031e0 msgr2=0x7f43481035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.300+0000 7f434cd7c640 1 --2- 172.21.15.50:0/3165731831 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43481031e0 0x7f43481035c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4338009900 tx=0x7f433802a510 comp rx=0 tx=0).stop 2024-04-23T18:24:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.301+0000 7f434cd7c640 1 -- 172.21.15.50:0/3165731831 shutdown_connections 2024-04-23T18:24:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.301+0000 7f434cd7c640 1 --2- 172.21.15.50:0/3165731831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f434810c520 0x7f434810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.301+0000 7f434cd7c640 1 --2- 172.21.15.50:0/3165731831 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4348103b00 0x7f434810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.301+0000 7f434cd7c640 1 --2- 172.21.15.50:0/3165731831 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43481031e0 0x7f43481035c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.301+0000 7f434cd7c640 1 -- 172.21.15.50:0/3165731831 >> 172.21.15.50:0/3165731831 conn(0x7f4348077f50 msgr2=0x7f4348078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.301+0000 7f434cd7c640 1 -- 172.21.15.50:0/3165731831 shutdown_connections 2024-04-23T18:24:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.301+0000 7f434cd7c640 1 -- 172.21.15.50:0/3165731831 wait complete. 2024-04-23T18:24:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.301+0000 7f434cd7c640 1 Processor -- start 2024-04-23T18:24:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f434cd7c640 1 -- start start 2024-04-23T18:24:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f434cd7c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43481031e0 0x7f434819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f434cd7c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4348103b00 0x7f434819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f434cd7c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f434810c520 0x7f4348198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f434cd7c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f43481121b0 con 0x7f4348103b00 2024-04-23T18:24:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f434cd7c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4348112030 con 0x7f434810c520 2024-04-23T18:24:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f434cd7c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4348112330 con 0x7f43481031e0 2024-04-23T18:24:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f433ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4348103b00 0x7f434819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f433ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4348103b00 0x7f434819e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56914/0 (socket says 172.21.15.50:56914) 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f433ffff640 1 -- 172.21.15.50:0/2412496429 learned_addr learned my addr 172.21.15.50:0/2412496429 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f434659c640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43481031e0 0x7f434819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f4346d9d640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f434810c520 0x7f4348198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f433ffff640 1 -- 172.21.15.50:0/2412496429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43481031e0 msgr2=0x7f434819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f433ffff640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43481031e0 0x7f434819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f433ffff640 1 -- 172.21.15.50:0/2412496429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f434810c520 msgr2=0x7f4348198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f433ffff640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f434810c520 0x7f4348198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f433ffff640 1 -- 172.21.15.50:0/2412496429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4348198d10 con 0x7f4348103b00 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f4346d9d640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f434810c520 0x7f4348198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.302+0000 7f433ffff640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4348103b00 0x7f434819e920 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f433000e9f0 tx=0x7f433000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.303+0000 7f433f7fe640 1 -- 172.21.15.50:0/2412496429 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433000cd80 con 0x7f4348103b00 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.303+0000 7f434cd7c640 1 -- 172.21.15.50:0/2412496429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4348199000 con 0x7f4348103b00 2024-04-23T18:24:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.303+0000 7f433f7fe640 1 -- 172.21.15.50:0/2412496429 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43300054d0 con 0x7f4348103b00 2024-04-23T18:24:30.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.303+0000 7f433f7fe640 1 -- 172.21.15.50:0/2412496429 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4330010690 con 0x7f4348103b00 2024-04-23T18:24:30.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.303+0000 7f434cd7c640 1 -- 172.21.15.50:0/2412496429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43481a52a0 con 0x7f4348103b00 2024-04-23T18:24:30.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.304+0000 7f434cd7c640 1 -- 172.21.15.50:0/2412496429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4348068d70 con 0x7f4348103b00 2024-04-23T18:24:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.305+0000 7f433f7fe640 1 -- 172.21.15.50:0/2412496429 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4330004e30 con 0x7f4348103b00 2024-04-23T18:24:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.306+0000 7f433f7fe640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f43340a3d20 0x7f43340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.306+0000 7f434659c640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f43340a3d20 0x7f43340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.306+0000 7f433f7fe640 1 -- 172.21.15.50:0/2412496429 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f43300d0de0 con 0x7f4348103b00 2024-04-23T18:24:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.306+0000 7f434659c640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f43340a3d20 0x7f43340a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4338031440 tx=0x7f4338005b50 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:30.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.310+0000 7f433f7fe640 1 -- 172.21.15.50:0/2412496429 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4330053ff0 con 0x7f4348103b00 2024-04-23T18:24:30.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:30 smithi050 ceph-mon[29187]: pgmap v184: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 203 op/s 2024-04-23T18:24:30.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:30 smithi177 ceph-mon[32308]: pgmap v184: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 203 op/s 2024-04-23T18:24:30.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.546+0000 7f434cd7c640 1 -- 172.21.15.50:0/2412496429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4348107700 con 0x7f4348103b00 2024-04-23T18:24:30.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.547+0000 7f433f7fe640 1 -- 172.21.15.50:0/2412496429 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11876 (secure 0 0 0) 0x7f433009d490 con 0x7f4348103b00 2024-04-23T18:24:30.541 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:24:30.541 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:23:50.103863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25822":{"gid":25822,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/911343542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":911343542},{"type":"v1","addr":"172.21.15.177:6837","nonce":911343542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:24:30.541 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 30 2024-04-23T18:24:30.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.549+0000 7f434cd7c640 1 -- 172.21.15.50:0/2412496429 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f43340a3d20 msgr2=0x7f43340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.549+0000 7f434cd7c640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f43340a3d20 0x7f43340a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4338031440 tx=0x7f4338005b50 comp rx=0 tx=0).stop 2024-04-23T18:24:30.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.549+0000 7f434cd7c640 1 -- 172.21.15.50:0/2412496429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4348103b00 msgr2=0x7f434819e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.549+0000 7f434cd7c640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4348103b00 0x7f434819e920 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f433000e9f0 tx=0x7f433000edf0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.549+0000 7f434cd7c640 1 -- 172.21.15.50:0/2412496429 shutdown_connections 2024-04-23T18:24:30.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.549+0000 7f434cd7c640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f43340a3d20 0x7f43340a61e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.549+0000 7f434cd7c640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f434810c520 0x7f4348198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.549+0000 7f434cd7c640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4348103b00 0x7f434819e920 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.549+0000 7f434cd7c640 1 --2- 172.21.15.50:0/2412496429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43481031e0 0x7f434819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.549+0000 7f434cd7c640 1 -- 172.21.15.50:0/2412496429 >> 172.21.15.50:0/2412496429 conn(0x7f4348077f50 msgr2=0x7f43480fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:30.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.549+0000 7f434cd7c640 1 -- 172.21.15.50:0/2412496429 shutdown_connections 2024-04-23T18:24:30.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.549+0000 7f434cd7c640 1 -- 172.21.15.50:0/2412496429 wait complete. 2024-04-23T18:24:30.554 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 89.55 --await-for 42.0' 2024-04-23T18:24:30.554 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 89.55 --await-for 42.0 2024-04-23T18:24:30.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:30 smithi188 ceph-mon[32694]: pgmap v184: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 203 op/s 2024-04-23T18:24:30.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.725+0000 7fcdce28a640 1 -- 172.21.15.50:0/2693820212 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdc8103b00 msgr2=0x7fcdc810bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.725+0000 7fcdce28a640 1 --2- 172.21.15.50:0/2693820212 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdc8103b00 0x7fcdc810bfe0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009900 tx=0x7fcdb002a510 comp rx=0 tx=0).stop 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.725+0000 7fcdce28a640 1 -- 172.21.15.50:0/2693820212 shutdown_connections 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.725+0000 7fcdce28a640 1 --2- 172.21.15.50:0/2693820212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc810c520 0x7fcdc810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.725+0000 7fcdce28a640 1 --2- 172.21.15.50:0/2693820212 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdc8103b00 0x7fcdc810bfe0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.725+0000 7fcdce28a640 1 --2- 172.21.15.50:0/2693820212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdc81031e0 0x7fcdc81035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.725+0000 7fcdce28a640 1 -- 172.21.15.50:0/2693820212 >> 172.21.15.50:0/2693820212 conn(0x7fcdc8077f50 msgr2=0x7fcdc8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.725+0000 7fcdce28a640 1 -- 172.21.15.50:0/2693820212 shutdown_connections 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.725+0000 7fcdce28a640 1 -- 172.21.15.50:0/2693820212 wait complete. 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdce28a640 1 Processor -- start 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdce28a640 1 -- start start 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdce28a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc81031e0 0x7fcdc8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdce28a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdc8103b00 0x7fcdc8198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdce28a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdc810c520 0x7fcdc819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdce28a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc81121b0 con 0x7fcdc8103b00 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdce28a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc8112030 con 0x7fcdc81031e0 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdce28a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc8112330 con 0x7fcdc810c520 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdc7fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc81031e0 0x7fcdc8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdc7fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc81031e0 0x7fcdc8198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:58174/0 (socket says 172.21.15.50:58174) 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdc7fff640 1 -- 172.21.15.50:0/1910557972 learned_addr learned my addr 172.21.15.50:0/1910557972 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdbffff640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdc8103b00 0x7fcdc8198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdcc827640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdc810c520 0x7fcdc819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdc7fff640 1 -- 172.21.15.50:0/1910557972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdc810c520 msgr2=0x7fcdc819cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdc7fff640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdc810c520 0x7fcdc819cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdc7fff640 1 -- 172.21.15.50:0/1910557972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdc8103b00 msgr2=0x7fcdc8198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdc7fff640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdc8103b00 0x7fcdc8198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdc7fff640 1 -- 172.21.15.50:0/1910557972 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc819d720 con 0x7fcdc81031e0 2024-04-23T18:24:30.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.726+0000 7fcdbffff640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdc8103b00 0x7fcdc8198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:30.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.727+0000 7fcdc7fff640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc81031e0 0x7fcdc8198730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcdb800e9f0 tx=0x7fcdb800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:30.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.727+0000 7fcdc5ffb640 1 -- 172.21.15.50:0/1910557972 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb800cd80 con 0x7fcdc81031e0 2024-04-23T18:24:30.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.727+0000 7fcdce28a640 1 -- 172.21.15.50:0/1910557972 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc819da10 con 0x7fcdc81031e0 2024-04-23T18:24:30.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.727+0000 7fcdc5ffb640 1 -- 172.21.15.50:0/1910557972 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdb80054d0 con 0x7fcdc81031e0 2024-04-23T18:24:30.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.727+0000 7fcdc5ffb640 1 -- 172.21.15.50:0/1910557972 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb8010690 con 0x7fcdc81031e0 2024-04-23T18:24:30.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.727+0000 7fcdce28a640 1 -- 172.21.15.50:0/1910557972 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc81a52a0 con 0x7fcdc81031e0 2024-04-23T18:24:30.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.729+0000 7fcdc5ffb640 1 -- 172.21.15.50:0/1910557972 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcdb8004e90 con 0x7fcdc81031e0 2024-04-23T18:24:30.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.729+0000 7fcdd0fa2740 1 Processor -- start 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.729+0000 7fcdd0fa2740 1 -- start start 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdd0fa2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a7a0a0 0x55c644a7a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdd0fa2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a9c0 0x55c644a84b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdd0fa2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 0x55c644a87480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdc5ffb640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcdb40a3d20 0x7fcdb40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdd0fa2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c6449df370 con 0x55c644a85090 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdd0fa2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c64497c2e0 con 0x55c644a7a0a0 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdd0fa2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c6449db330 con 0x55c644a7a9c0 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdbffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 0x55c644a87480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdc7fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a9c0 0x55c644a84b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdcc827640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a7a0a0 0x55c644a7a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdbffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 0x55c644a87480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56946/0 (socket says 172.21.15.50:56946) 2024-04-23T18:24:30.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdbffff640 1 -- 172.21.15.50:0/1617961741 learned_addr learned my addr 172.21.15.50:0/1617961741 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdc5ffb640 1 -- 172.21.15.50:0/1910557972 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7fcdb80d0c40 con 0x7fcdc81031e0 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdcc827640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcdb40a3d20 0x7fcdb40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdbffff640 1 -- 172.21.15.50:0/1617961741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a9c0 msgr2=0x55c644a84b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdbffff640 1 --2- 172.21.15.50:0/1617961741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a9c0 0x55c644a84b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdbffff640 1 -- 172.21.15.50:0/1617961741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a7a0a0 msgr2=0x55c644a7a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdbffff640 1 --2- 172.21.15.50:0/1617961741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a7a0a0 0x55c644a7a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdbffff640 1 -- 172.21.15.50:0/1617961741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6447bdd80 con 0x55c644a85090 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdcc827640 1 --2- 172.21.15.50:0/1617961741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a7a0a0 0x55c644a7a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdc7fff640 1 --2- 172.21.15.50:0/1617961741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a9c0 0x55c644a84b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.730+0000 7fcdbffff640 1 --2- 172.21.15.50:0/1617961741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 0x55c644a87480 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009900 tx=0x7fcdb0005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e188655e7d50624b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.731+0000 7fcdcc827640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcdb40a3d20 0x7fcdb40a61e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0005fd0 tx=0x7fcdc0009500 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.731+0000 7fcdbe7fc640 1 -- 172.21.15.50:0/1617961741 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0040070 con 0x55c644a85090 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.731+0000 7fcdbe7fc640 1 -- 172.21.15.50:0/1617961741 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdb0031860 con 0x55c644a85090 2024-04-23T18:24:30.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.731+0000 7fcdbe7fc640 1 -- 172.21.15.50:0/1617961741 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0035040 con 0x55c644a85090 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.731+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/1617961741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 msgr2=0x55c644a87480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.731+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/1617961741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 0x55c644a87480 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009900 tx=0x7fcdb0005d90 comp rx=0 tx=0).stop 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.731+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/1617961741 shutdown_connections 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.731+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/1617961741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 0x55c644a87480 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.731+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/1617961741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a9c0 0x55c644a84b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.731+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/1617961741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a7a0a0 0x55c644a7a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.731+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/1617961741 >> 172.21.15.50:0/1617961741 conn(0x55c644a763c0 msgr2=0x55c644a767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.732+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/1617961741 shutdown_connections 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.732+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/1617961741 wait complete. 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.732+0000 7fcdd0fa2740 1 -- start start 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdd0fa2740 1 Processor -- start 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdd0fa2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a0a0 0x55c644a81fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdd0fa2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a7a9c0 0x55c644a82520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdd0fa2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 0x55c644a82a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdd0fa2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c644a87e20 con 0x55c644a85090 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdd0fa2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c644979fb0 con 0x55c644a7a9c0 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdd0fa2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c644a87fa0 con 0x55c644a7a0a0 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdd0fa2740 10 client.0 _ll_register_callbacks cb 0x55c6449799a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdcc827640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 0x55c644a82a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdbffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a7a9c0 0x55c644a82520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdcc827640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 0x55c644a82a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56948/0 (socket says 172.21.15.50:56948) 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdcc827640 1 -- 172.21.15.50:0/587255326 learned_addr learned my addr 172.21.15.50:0/587255326 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdc7fff640 1 --2- 172.21.15.50:0/587255326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a0a0 0x55c644a81fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdbffff640 1 -- 172.21.15.50:0/587255326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a0a0 msgr2=0x55c644a81fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdbffff640 1 --2- 172.21.15.50:0/587255326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a0a0 0x55c644a81fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdbffff640 1 -- 172.21.15.50:0/587255326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 msgr2=0x55c644a82a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdbffff640 1 --2- 172.21.15.50:0/587255326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 0x55c644a82a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.733+0000 7fcdbffff640 1 -- 172.21.15.50:0/587255326 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c64481e510 con 0x55c644a7a9c0 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcdbffff640 1 --2- 172.21.15.50:0/587255326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a7a9c0 0x55c644a82520 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcdb002aa30 tx=0x7fcdb002a510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=747450f52f0ec72a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcda77fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcda77fe640 1 -- 172.21.15.50:0/587255326 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb004b070 con 0x55c644a7a9c0 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcda77fe640 1 -- 172.21.15.50:0/587255326 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7fcdb003da90 con 0x55c644a7a9c0 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/587255326 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdb0031860 con 0x55c644a7a9c0 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcda77fe640 1 -- 172.21.15.50:0/587255326 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcdb0057050 con 0x55c644a7a9c0 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcdd0fa2740 10 client.25832 fetch_fsmap learned FSMap version 30 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/587255326 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c6446b1670 con 0x55c644a7a9c0 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcda77fe640 1 -- 172.21.15.50:0/587255326 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdb0031860 con 0x55c644a7a9c0 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcda77fe640 1 -- 172.21.15.50:0/587255326 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 30) ==== 8350+0+0 (secure 0 0 0) 0x7fcdb00449b0 con 0x55c644a7a9c0 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcdd0fa2740 10 client.25832 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:24:30.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcdd0fa2740 10 client.25832 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcdd0fa2740 20 client.25832 populate_metadata read hostname 'smithi050' 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/587255326 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c644a95520 0x55c644a97980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcdd0fa2740 4 client.25832 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.734+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/587255326 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6449ecd90 con 0x55c644a95520 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.735+0000 7fcdc7fff640 1 --2- 172.21.15.50:0/587255326 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c644a95520 0x55c644a97980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.735+0000 7fcda77fe640 10 client.25832 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.735+0000 7fcdc7fff640 1 --2- 172.21.15.50:0/587255326 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c644a95520 0x55c644a97980 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae18dc0f8bce4ac0 server_cookie=f735ea8e0ca046c2 in_seq=0 out_seq=0 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcda77fe640 1 -- 172.21.15.50:0/587255326 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c6449ecd90 con 0x55c644a95520 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcda77fe640 10 client.25832 handle_command_reply: tid=0 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcdd0fa2740 1 client.25832 shutdown 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/587255326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a7a9c0 msgr2=0x55c644a82520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/587255326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a7a9c0 0x55c644a82520 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcdb002aa30 tx=0x7fcdb002a510 comp rx=0 tx=0).stop 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/587255326 shutdown_connections 2024-04-23T18:24:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/587255326 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c644a95520 0x55c644a97980 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/587255326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a85090 0x55c644a82a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcda77fe640 0 client.25832 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:30.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/587255326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a7a9c0 0x55c644a82520 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/587255326 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a0a0 0x55c644a81fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/587255326 >> 172.21.15.50:0/587255326 conn(0x55c644a763c0 msgr2=0x55c644a7c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:30.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/587255326 shutdown_connections 2024-04-23T18:24:30.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/587255326 wait complete. 2024-04-23T18:24:30.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.736+0000 7fcdd0fa2740 20 client.25832 trim_cache size 0 max 16384 2024-04-23T18:24:30.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdd0fa2740 1 Processor -- start 2024-04-23T18:24:30.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdd0fa2740 1 -- start start 2024-04-23T18:24:30.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdd0fa2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a0a0 0x55c644a75d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdd0fa2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a9c0 0x55c644a7c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdd0fa2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a85090 0x55c644a9d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdd0fa2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc81b0490 con 0x55c644a7a9c0 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdd0fa2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc81add00 con 0x55c644a85090 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdd0fa2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc81abb50 con 0x55c644a7a0a0 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdbffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a9c0 0x55c644a7c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdbffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a9c0 0x55c644a7c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56964/0 (socket says 172.21.15.50:56964) 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdbffff640 1 -- 172.21.15.50:0/2222543925 learned_addr learned my addr 172.21.15.50:0/2222543925 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdcc827640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a85090 0x55c644a9d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdc7fff640 1 --2- 172.21.15.50:0/2222543925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a0a0 0x55c644a75d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdbffff640 1 -- 172.21.15.50:0/2222543925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a0a0 msgr2=0x55c644a75d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdbffff640 1 --2- 172.21.15.50:0/2222543925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a0a0 0x55c644a75d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdbffff640 1 -- 172.21.15.50:0/2222543925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a85090 msgr2=0x55c644a9d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdbffff640 1 --2- 172.21.15.50:0/2222543925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a85090 0x55c644a9d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdbffff640 1 -- 172.21.15.50:0/2222543925 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c644a80ff0 con 0x55c644a7a9c0 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.755+0000 7fcdcc827640 1 --2- 172.21.15.50:0/2222543925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a85090 0x55c644a9d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.756+0000 7fcdbffff640 1 --2- 172.21.15.50:0/2222543925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a9c0 0x55c644a7c840 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009900 tx=0x7fcdb0031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf996cd6e5276e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.756+0000 7fcdbdffb640 1 -- 172.21.15.50:0/2222543925 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0060070 con 0x55c644a7a9c0 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.756+0000 7fcdbdffb640 1 -- 172.21.15.50:0/2222543925 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdb0005680 con 0x55c644a7a9c0 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.756+0000 7fcdbdffb640 1 -- 172.21.15.50:0/2222543925 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb004b040 con 0x55c644a7a9c0 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.756+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2222543925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a9c0 msgr2=0x55c644a7c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.756+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/2222543925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a9c0 0x55c644a7c840 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009900 tx=0x7fcdb0031400 comp rx=0 tx=0).stop 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2222543925 shutdown_connections 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/2222543925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a85090 0x55c644a9d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/2222543925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a9c0 0x55c644a7c840 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/2222543925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a0a0 0x55c644a75d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2222543925 >> 172.21.15.50:0/2222543925 conn(0x55c644a763c0 msgr2=0x55c644a83810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2222543925 shutdown_connections 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2222543925 wait complete. 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 -- start start 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 Processor -- start 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a0a0 0x55c644a7f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a9c0 0x55c644a7f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a85090 0x55c644afab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc81ac1c0 con 0x55c644a7a0a0 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc81ad630 con 0x55c644a85090 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc81ac890 con 0x55c644a7a9c0 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.757+0000 7fcdd0fa2740 10 client.0 _ll_register_callbacks cb 0x7fcdb0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdc7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a0a0 0x55c644a7f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdcc827640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a85090 0x55c644afab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdc7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a0a0 0x55c644a7f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56974/0 (socket says 172.21.15.50:56974) 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdc7fff640 1 -- 172.21.15.50:0/2690800264 learned_addr learned my addr 172.21.15.50:0/2690800264 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdbffff640 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a9c0 0x55c644a7f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdc7fff640 1 -- 172.21.15.50:0/2690800264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a9c0 msgr2=0x55c644a7f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdc7fff640 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a9c0 0x55c644a7f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdc7fff640 1 -- 172.21.15.50:0/2690800264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a85090 msgr2=0x55c644afab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdc7fff640 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a85090 0x55c644afab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdc7fff640 1 -- 172.21.15.50:0/2690800264 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c644a7a7f0 con 0x55c644a7a0a0 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdcc827640 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a85090 0x55c644afab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdc7fff640 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a0a0 0x55c644a7f140 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fcdb809cb20 tx=0x7fcdb809c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95f7f9f93ece1a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcda77fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcda77fe640 1 -- 172.21.15.50:0/2690800264 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb809d290 con 0x55c644a7a0a0 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcda77fe640 1 -- 172.21.15.50:0/2690800264 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7fcdb80bd0b0 con 0x55c644a7a0a0 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2690800264 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdb0005680 con 0x55c644a7a0a0 2024-04-23T18:24:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcda77fe640 1 -- 172.21.15.50:0/2690800264 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcdb809cb90 con 0x55c644a7a0a0 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdd0fa2740 10 client.16236 fetch_fsmap learned FSMap version 30 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.758+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2690800264 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c644afc070 con 0x55c644a7a0a0 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.759+0000 7fcda77fe640 1 -- 172.21.15.50:0/2690800264 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdb0005680 con 0x55c644a7a0a0 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.759+0000 7fcda77fe640 1 -- 172.21.15.50:0/2690800264 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 30) ==== 8350+0+0 (secure 0 0 0) 0x7fcdb8099200 con 0x55c644a7a0a0 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.759+0000 7fcdd0fa2740 10 client.16236 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.759+0000 7fcdd0fa2740 10 client.16236 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.759+0000 7fcdd0fa2740 20 client.16236 populate_metadata read hostname 'smithi050' 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.759+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c644afc870 0x55c644afec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.759+0000 7fcdd0fa2740 4 client.16236 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 89.55, "await_for": 42.0}] 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.759+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2690800264 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 89.55, "await_for": 42.0}) -- 0x55c6449e6c00 con 0x55c644afc870 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.759+0000 7fcdbffff640 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c644afc870 0x55c644afec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.759+0000 7fcdbffff640 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c644afc870 0x55c644afec90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45ef53bc2f08b6ad server_cookie=808a9ddc16db84fd in_seq=0 out_seq=0 2024-04-23T18:24:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.759+0000 7fcda77fe640 10 client.16236 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcda77fe640 1 -- 172.21.15.50:0/2690800264 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55c6449e6c00 con 0x55c644afc870 2024-04-23T18:24:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcda77fe640 10 client.16236 handle_command_reply: tid=0 2024-04-23T18:24:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcdd0fa2740 1 client.16236 shutdown 2024-04-23T18:24:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2690800264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a0a0 msgr2=0x55c644a7f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a0a0 0x55c644a7f140 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fcdb809cb20 tx=0x7fcdb809c800 comp rx=0 tx=0).stop 2024-04-23T18:24:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2690800264 shutdown_connections 2024-04-23T18:24:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c644afc870 0x55c644afec90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c644a85090 0x55c644afab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c644a7a9c0 0x55c644a7f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcdd0fa2740 1 --2- 172.21.15.50:0/2690800264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c644a7a0a0 0x55c644a7f140 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcda77fe640 0 client.16236 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2690800264 >> 172.21.15.50:0/2690800264 conn(0x55c644a763c0 msgr2=0x55c644a840a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2690800264 shutdown_connections 2024-04-23T18:24:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcdd0fa2740 1 -- 172.21.15.50:0/2690800264 wait complete. 2024-04-23T18:24:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.810+0000 7fcdd0fa2740 20 client.16236 trim_cache size 0 max 16384 2024-04-23T18:24:30.805 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:24:30.805 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 30, 2024-04-23T18:24:30.805 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:24:30.805 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 18, 2024-04-23T18:24:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:24:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: "19aaec48": { 2024-04-23T18:24:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 18, 2024-04-23T18:24:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:24:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:24:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:24:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:24:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:24:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:24:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 89.5, 2024-04-23T18:24:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:24:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:24:30.807 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:24:30.807 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:24:30.807 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:24:30.807 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:24:30.807 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:24:30.807 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:24:30.807 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:24:30.807 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:24:30.807 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:24:30.807 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:24:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.811+0000 7fcdce28a640 1 -- 172.21.15.50:0/1910557972 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcdb40a3d20 msgr2=0x7fcdb40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.811+0000 7fcdce28a640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcdb40a3d20 0x7fcdb40a61e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0005fd0 tx=0x7fcdc0009500 comp rx=0 tx=0).stop 2024-04-23T18:24:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.811+0000 7fcdce28a640 1 -- 172.21.15.50:0/1910557972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc81031e0 msgr2=0x7fcdc8198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.811+0000 7fcdce28a640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc81031e0 0x7fcdc8198730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcdb800e9f0 tx=0x7fcdb800edf0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.811+0000 7fcdce28a640 1 -- 172.21.15.50:0/1910557972 shutdown_connections 2024-04-23T18:24:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.811+0000 7fcdce28a640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcdb40a3d20 0x7fcdb40a61e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.811+0000 7fcdce28a640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdc810c520 0x7fcdc819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.811+0000 7fcdce28a640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdc8103b00 0x7fcdc8198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.811+0000 7fcdce28a640 1 --2- 172.21.15.50:0/1910557972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc81031e0 0x7fcdc8198730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.811+0000 7fcdce28a640 1 -- 172.21.15.50:0/1910557972 >> 172.21.15.50:0/1910557972 conn(0x7fcdc8077f50 msgr2=0x7fcdc80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.811+0000 7fcdce28a640 1 -- 172.21.15.50:0/1910557972 shutdown_connections 2024-04-23T18:24:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:24:30.811+0000 7fcdce28a640 1 -- 172.21.15.50:0/1910557972 wait complete. 2024-04-23T18:24:30.815 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '19aaec48', quiesce took 0.6 seconds. Will release after: 29.55 2024-04-23T18:24:31.208 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 6.06 MB/sec execute 185 sec latency 400.005 ms 2024-04-23T18:24:31.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:31 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2412496429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:31.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:31 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2412496429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:31.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:31 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2412496429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:24:32.209 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 6.02 MB/sec execute 186 sec latency 1400.061 ms 2024-04-23T18:24:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:32 smithi050 ceph-mon[29187]: pgmap v185: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 203 op/s 2024-04-23T18:24:32.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:32 smithi177 ceph-mon[32308]: pgmap v185: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 203 op/s 2024-04-23T18:24:32.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:32 smithi188 ceph-mon[32694]: pgmap v185: 97 pgs: 97 active+clean; 131 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 203 op/s 2024-04-23T18:24:33.209 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.99 MB/sec execute 187 sec latency 2400.225 ms 2024-04-23T18:24:34.209 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.96 MB/sec execute 188 sec latency 3400.335 ms 2024-04-23T18:24:34.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:34 smithi050 ceph-mon[29187]: pgmap v186: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 340 op/s 2024-04-23T18:24:34.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:34 smithi177 ceph-mon[32308]: pgmap v186: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 340 op/s 2024-04-23T18:24:34.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:34 smithi188 ceph-mon[32694]: pgmap v186: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 340 op/s 2024-04-23T18:24:35.209 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.93 MB/sec execute 189 sec latency 4400.453 ms 2024-04-23T18:24:36.209 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.90 MB/sec execute 190 sec latency 5400.505 ms 2024-04-23T18:24:36.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:36 smithi050 ceph-mon[29187]: pgmap v187: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 240 op/s 2024-04-23T18:24:36.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:36 smithi177 ceph-mon[32308]: pgmap v187: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 240 op/s 2024-04-23T18:24:36.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:36 smithi188 ceph-mon[32694]: pgmap v187: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 240 op/s 2024-04-23T18:24:37.209 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.87 MB/sec execute 191 sec latency 6400.618 ms 2024-04-23T18:24:38.209 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.84 MB/sec execute 192 sec latency 7400.734 ms 2024-04-23T18:24:38.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:38 smithi050 ceph-mon[29187]: pgmap v188: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 268 op/s 2024-04-23T18:24:38.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:38 smithi177 ceph-mon[32308]: pgmap v188: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 268 op/s 2024-04-23T18:24:38.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:38 smithi188 ceph-mon[32694]: pgmap v188: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 268 op/s 2024-04-23T18:24:39.209 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.81 MB/sec execute 193 sec latency 8400.851 ms 2024-04-23T18:24:40.209 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.78 MB/sec execute 194 sec latency 9400.974 ms 2024-04-23T18:24:40.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:40 smithi050 ceph-mon[29187]: pgmap v189: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-04-23T18:24:40.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:40 smithi177 ceph-mon[32308]: pgmap v189: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-04-23T18:24:40.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:40 smithi188 ceph-mon[32694]: pgmap v189: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-04-23T18:24:41.209 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.75 MB/sec execute 195 sec latency 10401.027 ms 2024-04-23T18:24:41.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:41 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:24:41.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:41 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:24:41.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:41 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:24:41.758 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.764+0000 7f63dcdf7640 1 -- 172.21.15.50:0/249320439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d8108020 msgr2=0x7f63d8108400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:41.758 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.764+0000 7f63dcdf7640 1 --2- 172.21.15.50:0/249320439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d8108020 0x7f63d8108400 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009900 tx=0x7f63c802a480 comp rx=0 tx=0).stop 2024-04-23T18:24:41.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.765+0000 7f63dcdf7640 1 -- 172.21.15.50:0/249320439 shutdown_connections 2024-04-23T18:24:41.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.765+0000 7f63dcdf7640 1 --2- 172.21.15.50:0/249320439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63d806b160 0x7f63d810edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.765+0000 7f63dcdf7640 1 --2- 172.21.15.50:0/249320439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63d806a7c0 0x7f63d806ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.765+0000 7f63dcdf7640 1 --2- 172.21.15.50:0/249320439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d8108020 0x7f63d8108400 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.765+0000 7f63dcdf7640 1 -- 172.21.15.50:0/249320439 >> 172.21.15.50:0/249320439 conn(0x7f63d8076c50 msgr2=0x7f63d8077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:41.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.765+0000 7f63dcdf7640 1 -- 172.21.15.50:0/249320439 shutdown_connections 2024-04-23T18:24:41.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.765+0000 7f63dcdf7640 1 -- 172.21.15.50:0/249320439 wait complete. 2024-04-23T18:24:41.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.765+0000 7f63dcdf7640 1 Processor -- start 2024-04-23T18:24:41.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.765+0000 7f63dcdf7640 1 -- start start 2024-04-23T18:24:41.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63dcdf7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d806a7c0 0x7f63d8198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63dcdf7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63d806b160 0x7f63d8198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63dcdf7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63d8108020 0x7f63d819cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63dcdf7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f63d8112190 con 0x7f63d806a7c0 2024-04-23T18:24:41.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63dcdf7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f63d8112010 con 0x7f63d806b160 2024-04-23T18:24:41.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63dcdf7640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f63d8112310 con 0x7f63d8108020 2024-04-23T18:24:41.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63d659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d806a7c0 0x7f63d8198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63d659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d806a7c0 0x7f63d8198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36206/0 (socket says 172.21.15.50:36206) 2024-04-23T18:24:41.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63d659c640 1 -- 172.21.15.50:0/1803787971 learned_addr learned my addr 172.21.15.50:0/1803787971 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63cdd9b640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63d806b160 0x7f63d8198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63d8108020 0x7f63d819cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63d659c640 1 -- 172.21.15.50:0/1803787971 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63d8108020 msgr2=0x7f63d819cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63d659c640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63d8108020 0x7f63d819cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63d659c640 1 -- 172.21.15.50:0/1803787971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63d806b160 msgr2=0x7f63d8198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63d659c640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63d806b160 0x7f63d8198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63d659c640 1 -- 172.21.15.50:0/1803787971 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d819d660 con 0x7f63d806a7c0 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63cdd9b640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63d806b160 0x7f63d8198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.766+0000 7f63d659c640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d806a7c0 0x7f63d8198670 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f63c8006010 tx=0x7f63c8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.767+0000 7f63cffff640 1 -- 172.21.15.50:0/1803787971 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c803f030 con 0x7f63d806a7c0 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.767+0000 7f63dcdf7640 1 -- 172.21.15.50:0/1803787971 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63d819d8f0 con 0x7f63d806a7c0 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.767+0000 7f63cffff640 1 -- 172.21.15.50:0/1803787971 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63c8003aa0 con 0x7f63d806a7c0 2024-04-23T18:24:41.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.767+0000 7f63cffff640 1 -- 172.21.15.50:0/1803787971 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c8035070 con 0x7f63d806a7c0 2024-04-23T18:24:41.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.767+0000 7f63dcdf7640 1 -- 172.21.15.50:0/1803787971 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63d81a51e0 con 0x7f63d806a7c0 2024-04-23T18:24:41.769 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.769+0000 7f63dfbe9740 1 Processor -- start 2024-04-23T18:24:41.769 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.769+0000 7f63dfbe9740 1 -- start start 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.769+0000 7f63dfbe9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 0x5582e2600a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.769+0000 7f63dfbe9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600f40 0x5582e260b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.769+0000 7f63dfbe9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e260b800 0x5582e260dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.769+0000 7f63dfbe9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5582e2565970 con 0x5582e2600620 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.769+0000 7f63dfbe9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5582e25024a0 con 0x5582e2600f40 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.769+0000 7f63dfbe9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5582e2561930 con 0x5582e260b800 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63d6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 0x5582e2600a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63d6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 0x5582e2600a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36216/0 (socket says 172.21.15.50:36216) 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63d6d9d640 1 -- 172.21.15.50:0/3684204376 learned_addr learned my addr 172.21.15.50:0/3684204376 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63cffff640 1 -- 172.21.15.50:0/1803787971 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f63c8042030 con 0x7f63d806a7c0 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63d6d9d640 1 -- 172.21.15.50:0/3684204376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e260b800 msgr2=0x5582e260dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/3684204376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e260b800 0x5582e260dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63d6d9d640 1 -- 172.21.15.50:0/3684204376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600f40 msgr2=0x5582e260b2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:24:41.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/3684204376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600f40 0x5582e260b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63d6d9d640 1 -- 172.21.15.50:0/3684204376 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582e2278210 con 0x5582e2600620 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/3684204376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 0x5582e2600a00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009f80 tx=0x7f63d0010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb45414c9276b24 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63ccd99640 1 -- 172.21.15.50:0/3684204376 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0011700 con 0x5582e2600620 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63ccd99640 1 -- 172.21.15.50:0/3684204376 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d0011d50 con 0x5582e2600620 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.770+0000 7f63ccd99640 1 -- 172.21.15.50:0/3684204376 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0019d50 con 0x5582e2600620 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63cffff640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f63c40a3dc0 0x7f63c40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63d659c640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f63c40a3dc0 0x7f63c40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63cffff640 1 -- 172.21.15.50:0/1803787971 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f63c80f0d10 con 0x7f63d806a7c0 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63dfbe9740 1 -- 172.21.15.50:0/3684204376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 msgr2=0x5582e2600a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/3684204376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 0x5582e2600a00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009f80 tx=0x7f63d0010860 comp rx=0 tx=0).stop 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63d659c640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f63c40a3dc0 0x7f63c40a6280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x5582e2601bc0 tx=0x7f63c808e7b0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63dfbe9740 1 -- 172.21.15.50:0/3684204376 shutdown_connections 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/3684204376 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e260b800 0x5582e260dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/3684204376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600f40 0x5582e260b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/3684204376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 0x5582e2600a00 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63dfbe9740 1 -- 172.21.15.50:0/3684204376 >> 172.21.15.50:0/3684204376 conn(0x5582e25fcb30 msgr2=0x5582e25fcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63dfbe9740 1 -- 172.21.15.50:0/3684204376 shutdown_connections 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.771+0000 7f63dfbe9740 1 -- 172.21.15.50:0/3684204376 wait complete. 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63dfbe9740 1 -- start start 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63dfbe9740 1 Processor -- start 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63dfbe9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600620 0x5582e260b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63dfbe9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e2600f40 0x5582e26083f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63dfbe9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e260b800 0x5582e2608930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63dfbe9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5582e260e430 con 0x5582e260b800 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63dfbe9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5582e25000b0 con 0x5582e2600620 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63dfbe9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5582e260e5b0 con 0x5582e2600f40 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63dfbe9740 10 client.0 _ll_register_callbacks cb 0x5582e24edd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63d6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e260b800 0x5582e2608930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63d6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e260b800 0x5582e2608930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36224/0 (socket says 172.21.15.50:36224) 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63d6d9d640 1 -- 172.21.15.50:0/2776729382 learned_addr learned my addr 172.21.15.50:0/2776729382 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63cdd9b640 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e2600f40 0x5582e26083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600620 0x5582e260b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63d6d9d640 1 -- 172.21.15.50:0/2776729382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e2600f40 msgr2=0x5582e26083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e2600f40 0x5582e26083f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63d6d9d640 1 -- 172.21.15.50:0/2776729382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600620 msgr2=0x5582e260b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600620 0x5582e260b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63d6d9d640 1 -- 172.21.15.50:0/2776729382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582e2344620 con 0x5582e260b800 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.772+0000 7f63cdd9b640 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e2600f40 0x5582e26083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e260b800 0x5582e2608930 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f63d0000f80 tx=0x7f63d0013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77dc3f2dda00907d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63bdffb640 1 -- 172.21.15.50:0/2776729382 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0011910 con 0x5582e260b800 2024-04-23T18:24:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63bdffb640 1 -- 172.21.15.50:0/2776729382 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f63d0011ab0 con 0x5582e260b800 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2776729382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63d0011d50 con 0x5582e260b800 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63bdffb640 1 -- 172.21.15.50:0/2776729382 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f63d0003970 con 0x5582e260b800 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63dfbe9740 10 client.16248 fetch_fsmap learned FSMap version 30 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2776729382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5582e261a350 con 0x5582e260b800 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63bdffb640 1 -- 172.21.15.50:0/2776729382 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d0011d50 con 0x5582e260b800 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63bdffb640 1 -- 172.21.15.50:0/2776729382 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 30) ==== 8350+0+0 (secure 0 0 0) 0x7f63d00044b0 con 0x5582e260b800 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63dfbe9740 10 client.16248 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63dfbe9740 10 client.16248 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63dfbe9740 20 client.16248 populate_metadata read hostname 'smithi050' 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582e261b7b0 0x5582e261dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63dfbe9740 4 client.16248 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.773+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2776729382 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582e2573390 con 0x5582e261b7b0 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.774+0000 7f63cdd9b640 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582e261b7b0 0x5582e261dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.774+0000 7f63cdd9b640 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582e261b7b0 0x5582e261dba0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e5a3b1352526bb3 server_cookie=e376e69698616676 in_seq=0 out_seq=0 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.774+0000 7f63bdffb640 10 client.16248 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63bdffb640 1 -- 172.21.15.50:0/2776729382 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582e2573390 con 0x5582e261b7b0 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63bdffb640 10 client.16248 handle_command_reply: tid=0 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63dfbe9740 1 client.16248 shutdown 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2776729382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e260b800 msgr2=0x5582e2608930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e260b800 0x5582e2608930 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f63d0000f80 tx=0x7f63d0013040 comp rx=0 tx=0).stop 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2776729382 shutdown_connections 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582e261b7b0 0x5582e261dba0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e260b800 0x5582e2608930 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63bdffb640 0 client.16248 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e2600f40 0x5582e26083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/2776729382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600620 0x5582e260b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2776729382 >> 172.21.15.50:0/2776729382 conn(0x5582e25fcb30 msgr2=0x5582e26028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2776729382 shutdown_connections 2024-04-23T18:24:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2776729382 wait complete. 2024-04-23T18:24:41.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.775+0000 7f63dfbe9740 20 client.16248 trim_cache size 0 max 16384 2024-04-23T18:24:41.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63dfbe9740 1 Processor -- start 2024-04-23T18:24:41.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63dfbe9740 1 -- start start 2024-04-23T18:24:41.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63dfbe9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e25fcb30 0x5582e2617320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63dfbe9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e2600620 0x5582e25fbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63dfbe9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600f40 0x5582e2623290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63dfbe9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f63d81b0430 con 0x5582e2600f40 2024-04-23T18:24:41.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63dfbe9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f63d81adca0 con 0x5582e25fcb30 2024-04-23T18:24:41.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63dfbe9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f63d81abaf0 con 0x5582e2600620 2024-04-23T18:24:41.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63d6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600f40 0x5582e2623290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63d6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600f40 0x5582e2623290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36234/0 (socket says 172.21.15.50:36234) 2024-04-23T18:24:41.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63d6d9d640 1 -- 172.21.15.50:0/1206171882 learned_addr learned my addr 172.21.15.50:0/1206171882 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:41.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63cdd9b640 1 --2- 172.21.15.50:0/1206171882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e2600620 0x5582e25fbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.790+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/1206171882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e25fcb30 0x5582e2617320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63d6d9d640 1 -- 172.21.15.50:0/1206171882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e2600620 msgr2=0x5582e25fbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/1206171882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e2600620 0x5582e25fbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63d6d9d640 1 -- 172.21.15.50:0/1206171882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e25fcb30 msgr2=0x5582e2617320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/1206171882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e25fcb30 0x5582e2617320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63d6d9d640 1 -- 172.21.15.50:0/1206171882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582e25f8500 con 0x5582e2600f40 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/1206171882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600f40 0x5582e2623290 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f63d0019a40 tx=0x7f63d00059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154243e78de76ee0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63bffff640 1 -- 172.21.15.50:0/1206171882 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0010d60 con 0x5582e2600f40 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63bffff640 1 -- 172.21.15.50:0/1206171882 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d0020040 con 0x5582e2600f40 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63bffff640 1 -- 172.21.15.50:0/1206171882 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d001ab60 con 0x5582e2600f40 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63dfbe9740 1 -- 172.21.15.50:0/1206171882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600f40 msgr2=0x5582e2623290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/1206171882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600f40 0x5582e2623290 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f63d0019a40 tx=0x7f63d00059e0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63dfbe9740 1 -- 172.21.15.50:0/1206171882 shutdown_connections 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/1206171882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600f40 0x5582e2623290 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/1206171882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e2600620 0x5582e25fbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/1206171882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e25fcb30 0x5582e2617320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63dfbe9740 1 -- 172.21.15.50:0/1206171882 >> 172.21.15.50:0/1206171882 conn(0x5582e2608a90 msgr2=0x5582e25fb420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63dfbe9740 1 -- 172.21.15.50:0/1206171882 shutdown_connections 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.791+0000 7f63dfbe9740 1 -- 172.21.15.50:0/1206171882 wait complete. 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63dfbe9740 1 -- start start 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63dfbe9740 1 Processor -- start 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63dfbe9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e25fcb30 0x5582e2613870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63dfbe9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 0x5582e2613db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63dfbe9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600f40 0x5582e2610e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63dfbe9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f63d81ac160 con 0x5582e2600620 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63dfbe9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f63d81ad5d0 con 0x5582e2600f40 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63dfbe9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f63d81ac830 con 0x5582e25fcb30 2024-04-23T18:24:41.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63dfbe9740 10 client.0 _ll_register_callbacks cb 0x7f63d0019e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63cdd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 0x5582e2613db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63cdd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 0x5582e2613db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36238/0 (socket says 172.21.15.50:36238) 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63cdd9b640 1 -- 172.21.15.50:0/2024486716 learned_addr learned my addr 172.21.15.50:0/2024486716 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e25fcb30 0x5582e2613870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600f40 0x5582e2610e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63cdd9b640 1 -- 172.21.15.50:0/2024486716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e25fcb30 msgr2=0x5582e2613870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63cdd9b640 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e25fcb30 0x5582e2613870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63cdd9b640 1 -- 172.21.15.50:0/2024486716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600f40 msgr2=0x5582e2610e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63cdd9b640 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600f40 0x5582e2610e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.792+0000 7f63cdd9b640 1 -- 172.21.15.50:0/2024486716 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582e2611600 con 0x5582e2600620 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63cdd9b640 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 0x5582e2613db0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x5582e25fdcc0 tx=0x7f63c0013c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fec39085b9cc35 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63bdffb640 1 -- 172.21.15.50:0/2024486716 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0004460 con 0x5582e2600620 2024-04-23T18:24:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63bdffb640 1 -- 172.21.15.50:0/2024486716 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f63c0010960 con 0x5582e2600620 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2024486716 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63d0020040 con 0x5582e2600620 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63bdffb640 1 -- 172.21.15.50:0/2024486716 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f63c0010b60 con 0x5582e2600620 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63dfbe9740 10 client.16254 fetch_fsmap learned FSMap version 30 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2024486716 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5582e2681840 con 0x5582e2600620 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63bdffb640 1 -- 172.21.15.50:0/2024486716 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d0020040 con 0x5582e2600620 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63bdffb640 1 -- 172.21.15.50:0/2024486716 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 30) ==== 8350+0+0 (secure 0 0 0) 0x7f63c00039a0 con 0x5582e2600620 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63dfbe9740 10 client.16254 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63dfbe9740 10 client.16254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63dfbe9740 20 client.16254 populate_metadata read hostname 'smithi050' 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582e260b800 0x5582e2684090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63dfbe9740 4 client.16254 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.793+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2024486716 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5582e256d200 con 0x5582e260b800 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.794+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582e260b800 0x5582e2684090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.794+0000 7f63d6d9d640 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582e260b800 0x5582e2684090 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45b03f89d5770853 server_cookie=ce462354845dd800 in_seq=0 out_seq=0 2024-04-23T18:24:41.793 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.794+0000 7f63bdffb640 10 client.16254 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:41.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63bdffb640 1 -- 172.21.15.50:0/2024486716 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5582e256d200 con 0x5582e260b800 2024-04-23T18:24:41.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63bdffb640 10 client.16254 handle_command_reply: tid=0 2024-04-23T18:24:41.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dfbe9740 1 client.16254 shutdown 2024-04-23T18:24:41.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2024486716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 msgr2=0x5582e2613db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:41.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 0x5582e2613db0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x5582e25fdcc0 tx=0x7f63c0013c30 comp rx=0 tx=0).stop 2024-04-23T18:24:41.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2024486716 shutdown_connections 2024-04-23T18:24:41.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582e260b800 0x5582e2684090 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582e2600f40 0x5582e2610e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582e2600620 0x5582e2613db0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63bdffb640 0 client.16254 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dfbe9740 1 --2- 172.21.15.50:0/2024486716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582e25fcb30 0x5582e2613870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2024486716 >> 172.21.15.50:0/2024486716 conn(0x5582e2608a90 msgr2=0x5582e25fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2024486716 shutdown_connections 2024-04-23T18:24:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dfbe9740 1 -- 172.21.15.50:0/2024486716 wait complete. 2024-04-23T18:24:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dfbe9740 20 client.16254 trim_cache size 0 max 16384 2024-04-23T18:24:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:24:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:24:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:24:41.818 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:24:41.818 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dcdf7640 1 -- 172.21.15.50:0/1803787971 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f63c40a3dc0 msgr2=0x7f63c40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:41.818 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dcdf7640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f63c40a3dc0 0x7f63c40a6280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x5582e2601bc0 tx=0x7f63c808e7b0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dcdf7640 1 -- 172.21.15.50:0/1803787971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d806a7c0 msgr2=0x7f63d8198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:41.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dcdf7640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d806a7c0 0x7f63d8198670 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f63c8006010 tx=0x7f63c8004270 comp rx=0 tx=0).stop 2024-04-23T18:24:41.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dcdf7640 1 -- 172.21.15.50:0/1803787971 shutdown_connections 2024-04-23T18:24:41.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dcdf7640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f63c40a3dc0 0x7f63c40a6280 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dcdf7640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63d8108020 0x7f63d819cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dcdf7640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63d806b160 0x7f63d8198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dcdf7640 1 --2- 172.21.15.50:0/1803787971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d806a7c0 0x7f63d8198670 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:41.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dcdf7640 1 -- 172.21.15.50:0/1803787971 >> 172.21.15.50:0/1803787971 conn(0x7f63d8076c50 msgr2=0x7f63d80ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:41.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dcdf7640 1 -- 172.21.15.50:0/1803787971 shutdown_connections 2024-04-23T18:24:41.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:41.823+0000 7f63dcdf7640 1 -- 172.21.15.50:0/1803787971 wait complete. 2024-04-23T18:24:42.210 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.72 MB/sec execute 196 sec latency 11401.120 ms 2024-04-23T18:24:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:42 smithi050 ceph-mon[29187]: pgmap v190: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-04-23T18:24:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:42 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:24:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:42 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:24:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:42 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:24:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:42 smithi177 ceph-mon[32308]: pgmap v190: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-04-23T18:24:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:42 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:24:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:42 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:24:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:42 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:24:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:42 smithi188 ceph-mon[32694]: pgmap v190: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-04-23T18:24:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:42 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:24:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:42 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:24:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:42 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:24:43.209 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.69 MB/sec execute 197 sec latency 12401.225 ms 2024-04-23T18:24:44.210 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.66 MB/sec execute 198 sec latency 13401.344 ms 2024-04-23T18:24:44.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:44 smithi050 ceph-mon[29187]: pgmap v191: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 192 op/s 2024-04-23T18:24:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:44 smithi177 ceph-mon[32308]: pgmap v191: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 192 op/s 2024-04-23T18:24:44.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:44 smithi188 ceph-mon[32694]: pgmap v191: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 192 op/s 2024-04-23T18:24:45.210 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.63 MB/sec execute 199 sec latency 14401.449 ms 2024-04-23T18:24:46.210 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.60 MB/sec execute 200 sec latency 15401.530 ms 2024-04-23T18:24:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:46 smithi050 ceph-mon[29187]: pgmap v192: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 1.1 MiB/s wr, 55 op/s 2024-04-23T18:24:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:46 smithi177 ceph-mon[32308]: pgmap v192: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 1.1 MiB/s wr, 55 op/s 2024-04-23T18:24:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:46 smithi188 ceph-mon[32694]: pgmap v192: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 1.1 MiB/s wr, 55 op/s 2024-04-23T18:24:47.210 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.57 MB/sec execute 201 sec latency 16401.624 ms 2024-04-23T18:24:48.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.55 MB/sec execute 202 sec latency 17401.664 ms 2024-04-23T18:24:48.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:48 smithi050 ceph-mon[29187]: pgmap v193: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-04-23T18:24:48.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:48 smithi177 ceph-mon[32308]: pgmap v193: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-04-23T18:24:48.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:48 smithi188 ceph-mon[32694]: pgmap v193: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-04-23T18:24:49.210 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.52 MB/sec execute 203 sec latency 18401.703 ms 2024-04-23T18:24:50.210 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.49 MB/sec execute 204 sec latency 19401.788 ms 2024-04-23T18:24:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:50 smithi050 ceph-mon[29187]: pgmap v194: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.5 MiB/s wr, 68 op/s 2024-04-23T18:24:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:50 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:50 smithi177 ceph-mon[32308]: pgmap v194: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.5 MiB/s wr, 68 op/s 2024-04-23T18:24:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:50 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:50 smithi188 ceph-mon[32694]: pgmap v194: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.5 MiB/s wr, 68 op/s 2024-04-23T18:24:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:50 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:51.210 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.47 MB/sec execute 205 sec latency 20401.895 ms 2024-04-23T18:24:51.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:51 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:51.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:51 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:51.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:51 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:24:52.210 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.44 MB/sec execute 206 sec latency 21401.968 ms 2024-04-23T18:24:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:52 smithi177 ceph-mon[32308]: pgmap v195: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.5 MiB/s wr, 68 op/s 2024-04-23T18:24:52.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:52 smithi188 ceph-mon[32694]: pgmap v195: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.5 MiB/s wr, 68 op/s 2024-04-23T18:24:52.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:52 smithi050 ceph-mon[29187]: pgmap v195: 97 pgs: 97 active+clean; 122 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 1.5 MiB/s wr, 68 op/s 2024-04-23T18:24:53.210 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.41 MB/sec execute 207 sec latency 22402.050 ms 2024-04-23T18:24:54.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.39 MB/sec execute 208 sec latency 23402.188 ms 2024-04-23T18:24:54.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:54 smithi177 ceph-mon[32308]: pgmap v196: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 2.1 MiB/s wr, 97 op/s 2024-04-23T18:24:54.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:54 smithi188 ceph-mon[32694]: pgmap v196: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 2.1 MiB/s wr, 97 op/s 2024-04-23T18:24:54.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:54 smithi050 ceph-mon[29187]: pgmap v196: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 2.1 MiB/s wr, 97 op/s 2024-04-23T18:24:54.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.922+0000 7f99bccef640 1 -- 172.21.15.50:0/2822811512 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8103220 msgr2=0x7f99b8103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:54.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.922+0000 7f99bccef640 1 --2- 172.21.15.50:0/2822811512 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8103220 0x7f99b8103600 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009900 tx=0x7f99a802a480 comp rx=0 tx=0).stop 2024-04-23T18:24:54.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.922+0000 7f99bccef640 1 -- 172.21.15.50:0/2822811512 shutdown_connections 2024-04-23T18:24:54.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.922+0000 7f99bccef640 1 --2- 172.21.15.50:0/2822811512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99b810c560 0x7f99b810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.922+0000 7f99bccef640 1 --2- 172.21.15.50:0/2822811512 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99b8103b40 0x7f99b810c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.922+0000 7f99bccef640 1 --2- 172.21.15.50:0/2822811512 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8103220 0x7f99b8103600 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.922+0000 7f99bccef640 1 -- 172.21.15.50:0/2822811512 >> 172.21.15.50:0/2822811512 conn(0x7f99b8077fd0 msgr2=0x7f99b80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.922+0000 7f99bccef640 1 -- 172.21.15.50:0/2822811512 shutdown_connections 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.922+0000 7f99bccef640 1 -- 172.21.15.50:0/2822811512 wait complete. 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.923+0000 7f99bccef640 1 Processor -- start 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.923+0000 7f99bccef640 1 -- start start 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.923+0000 7f99bccef640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8103220 0x7f99b8100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.923+0000 7f99bccef640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99b8103b40 0x7f99b81013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.923+0000 7f99bccef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99b810c560 0x7f99b81018f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.923+0000 7f99bccef640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81121b0 con 0x7f99b8103220 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.923+0000 7f99bccef640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f99b8112030 con 0x7f99b810c560 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.923+0000 7f99bccef640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f99b8112330 con 0x7f99b8103b40 2024-04-23T18:24:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.923+0000 7f99b659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8103220 0x7f99b8100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.923+0000 7f99b659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8103220 0x7f99b8100e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33808/0 (socket says 172.21.15.50:33808) 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.923+0000 7f99b659c640 1 -- 172.21.15.50:0/1525534850 learned_addr learned my addr 172.21.15.50:0/1525534850 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99b6d9d640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99b810c560 0x7f99b81018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99b5d9b640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99b8103b40 0x7f99b81013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99b659c640 1 -- 172.21.15.50:0/1525534850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99b8103b40 msgr2=0x7f99b81013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99b659c640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99b8103b40 0x7f99b81013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99b659c640 1 -- 172.21.15.50:0/1525534850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99b810c560 msgr2=0x7f99b81018f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99b659c640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99b810c560 0x7f99b81018f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99b659c640 1 -- 172.21.15.50:0/1525534850 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b81a6bf0 con 0x7f99b8103220 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99b659c640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8103220 0x7f99b8100e70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f99a8005940 tx=0x7f99a80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99a77fe640 1 -- 172.21.15.50:0/1525534850 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a803f030 con 0x7f99b8103220 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99bccef640 1 -- 172.21.15.50:0/1525534850 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b81a6e80 con 0x7f99b8103220 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99a77fe640 1 -- 172.21.15.50:0/1525534850 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99a8003cb0 con 0x7f99b8103220 2024-04-23T18:24:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99a77fe640 1 -- 172.21.15.50:0/1525534850 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8035070 con 0x7f99b8103220 2024-04-23T18:24:54.921 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.924+0000 7f99bccef640 1 -- 172.21.15.50:0/1525534850 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b81a72e0 con 0x7f99b8103220 2024-04-23T18:24:54.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.927+0000 7f99a77fe640 1 -- 172.21.15.50:0/1525534850 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f99a8042030 con 0x7f99b8103220 2024-04-23T18:24:54.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.927+0000 7f99bfa0a740 1 Processor -- start 2024-04-23T18:24:54.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.927+0000 7f99bfa0a740 1 -- start start 2024-04-23T18:24:54.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.927+0000 7f99bfa0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2b620 0x55bdebe2ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.927+0000 7f99bfa0a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 0x55bdebe362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.927+0000 7f99bfa0a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe36800 0x55bdebe38bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.927+0000 7f99bfa0a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bdebd90970 con 0x55bdebe2bf40 2024-04-23T18:24:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.927+0000 7f99bfa0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bdebd2d4a0 con 0x55bdebe2b620 2024-04-23T18:24:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.927+0000 7f99bfa0a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bdebd8c930 con 0x55bdebe36800 2024-04-23T18:24:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.927+0000 7f99b659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 0x55bdebe362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.927+0000 7f99b659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 0x55bdebe362c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33822/0 (socket says 172.21.15.50:33822) 2024-04-23T18:24:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.927+0000 7f99b659c640 1 -- 172.21.15.50:0/3144171604 learned_addr learned my addr 172.21.15.50:0/3144171604 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99a77fe640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99a00a3df0 0x7f99a00a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99b5d9b640 1 --2- 172.21.15.50:0/3144171604 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe36800 0x55bdebe38bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99b6d9d640 1 --2- 172.21.15.50:0/3144171604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2b620 0x55bdebe2ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99b6d9d640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99a00a3df0 0x7f99a00a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99a77fe640 1 -- 172.21.15.50:0/1525534850 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f99a80f0dd0 con 0x7f99b8103220 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99b659c640 1 -- 172.21.15.50:0/3144171604 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe36800 msgr2=0x55bdebe38bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99b659c640 1 --2- 172.21.15.50:0/3144171604 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe36800 0x55bdebe38bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99b659c640 1 -- 172.21.15.50:0/3144171604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2b620 msgr2=0x55bdebe2ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99b659c640 1 --2- 172.21.15.50:0/3144171604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2b620 0x55bdebe2ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99b659c640 1 -- 172.21.15.50:0/3144171604 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdebaa3210 con 0x55bdebe2bf40 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99b659c640 1 --2- 172.21.15.50:0/3144171604 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 0x55bdebe362c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f99a80bca20 tx=0x7f99a802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ccbe885aaef2e85 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99b6d9d640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99a00a3df0 0x7f99a00a62b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f99b000bfd0 tx=0x7f99b0005c20 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99a4ff9640 1 -- 172.21.15.50:0/3144171604 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8049020 con 0x55bdebe2bf40 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99a4ff9640 1 -- 172.21.15.50:0/3144171604 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99a80bd420 con 0x55bdebe2bf40 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.928+0000 7f99a4ff9640 1 -- 172.21.15.50:0/3144171604 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a80b9390 con 0x55bdebe2bf40 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.929+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3144171604 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 msgr2=0x55bdebe362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.929+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/3144171604 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 0x55bdebe362c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f99a80bca20 tx=0x7f99a802a480 comp rx=0 tx=0).stop 2024-04-23T18:24:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.929+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3144171604 shutdown_connections 2024-04-23T18:24:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.929+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/3144171604 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe36800 0x55bdebe38bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.929+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/3144171604 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 0x55bdebe362c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.929+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/3144171604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2b620 0x55bdebe2ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.929+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3144171604 >> 172.21.15.50:0/3144171604 conn(0x55bdebe27b30 msgr2=0x55bdebe27f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.929+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3144171604 shutdown_connections 2024-04-23T18:24:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.929+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3144171604 wait complete. 2024-04-23T18:24:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99bfa0a740 1 -- start start 2024-04-23T18:24:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99bfa0a740 1 Processor -- start 2024-04-23T18:24:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99bfa0a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe36140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99bfa0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2bf40 0x55bdebe33430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99bfa0a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 0x55bdebe33970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99bfa0a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bdebe391f0 con 0x55bdebe36800 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99bfa0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bdebd2b0b0 con 0x55bdebe2bf40 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99bfa0a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bdebe39370 con 0x55bdebe2b620 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99bfa0a740 10 client.0 _ll_register_callbacks cb 0x55bdebd18d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99b6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 0x55bdebe33970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99b6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 0x55bdebe33970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33834/0 (socket says 172.21.15.50:33834) 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99b6d9d640 1 -- 172.21.15.50:0/2788335543 learned_addr learned my addr 172.21.15.50:0/2788335543 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99b659c640 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe36140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.930+0000 7f99b5d9b640 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2bf40 0x55bdebe33430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f99b6d9d640 1 -- 172.21.15.50:0/2788335543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 msgr2=0x55bdebe36140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f99b6d9d640 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe36140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f99b6d9d640 1 -- 172.21.15.50:0/2788335543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2bf40 msgr2=0x55bdebe33430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f99b6d9d640 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2bf40 0x55bdebe33430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f99b6d9d640 1 -- 172.21.15.50:0/2788335543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdebb6f620 con 0x55bdebe36800 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f99b6d9d640 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 0x55bdebe33970 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f99b0011010 tx=0x7f99b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed4b76fe35d104e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f999dffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f999dffb640 1 -- 172.21.15.50:0/2788335543 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b0025070 con 0x55bdebe36800 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f999dffb640 1 -- 172.21.15.50:0/2788335543 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f99b001f860 con 0x55bdebe36800 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f99bfa0a740 1 -- 172.21.15.50:0/2788335543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99a80bd420 con 0x55bdebe36800 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f999dffb640 1 -- 172.21.15.50:0/2788335543 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f99b000ee80 con 0x55bdebe36800 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f99bfa0a740 10 client.16266 fetch_fsmap learned FSMap version 32 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f99bfa0a740 1 -- 172.21.15.50:0/2788335543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bdebe450b0 con 0x55bdebe36800 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f999dffb640 1 -- 172.21.15.50:0/2788335543 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99a80bd420 con 0x55bdebe36800 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.931+0000 7f999dffb640 1 -- 172.21.15.50:0/2788335543 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f99b0012720 con 0x55bdebe36800 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.932+0000 7f99bfa0a740 10 client.16266 fetch_fsmap finished waiting for FSMap version 32 2024-04-23T18:24:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.932+0000 7f99bfa0a740 10 client.16266 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.932+0000 7f99bfa0a740 20 client.16266 populate_metadata read hostname 'smithi050' 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.932+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bdebe465e0 0x55bdebe48a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.932+0000 7f99bfa0a740 4 client.16266 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.932+0000 7f99bfa0a740 1 -- 172.21.15.50:0/2788335543 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdebd9e390 con 0x55bdebe465e0 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.932+0000 7f99b659c640 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bdebe465e0 0x55bdebe48a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.932+0000 7f99b659c640 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bdebe465e0 0x55bdebe48a10 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae248b3309ce158f server_cookie=6a0b3230e1ed4a2f in_seq=0 out_seq=0 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.932+0000 7f999dffb640 10 client.16266 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f999dffb640 1 -- 172.21.15.50:0/2788335543 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bdebd9e390 con 0x55bdebe465e0 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f999dffb640 10 client.16266 handle_command_reply: tid=0 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f99bfa0a740 1 client.16266 shutdown 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f99bfa0a740 1 -- 172.21.15.50:0/2788335543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 msgr2=0x55bdebe33970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 0x55bdebe33970 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f99b0011010 tx=0x7f99b0004a40 comp rx=0 tx=0).stop 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f99bfa0a740 1 -- 172.21.15.50:0/2788335543 shutdown_connections 2024-04-23T18:24:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bdebe465e0 0x55bdebe48a10 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 0x55bdebe33970 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2bf40 0x55bdebe33430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/2788335543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe36140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f999dffb640 0 client.16266 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f99bfa0a740 1 -- 172.21.15.50:0/2788335543 >> 172.21.15.50:0/2788335543 conn(0x55bdebe27b30 msgr2=0x55bdebe2d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f99bfa0a740 1 -- 172.21.15.50:0/2788335543 shutdown_connections 2024-04-23T18:24:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f99bfa0a740 1 -- 172.21.15.50:0/2788335543 wait complete. 2024-04-23T18:24:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.933+0000 7f99bfa0a740 20 client.16266 trim_cache size 0 max 16384 2024-04-23T18:24:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.950+0000 7f99bfa0a740 1 Processor -- start 2024-04-23T18:24:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.950+0000 7f99bfa0a740 1 -- start start 2024-04-23T18:24:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99bfa0a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe42020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99bfa0a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 0x55bdebe26b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99bfa0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe36800 0x55bdebe4e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99bfa0a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81b26d0 con 0x55bdebe2bf40 2024-04-23T18:24:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99bfa0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81aff70 con 0x55bdebe36800 2024-04-23T18:24:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99bfa0a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81addc0 con 0x55bdebe2b620 2024-04-23T18:24:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99b5d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 0x55bdebe26b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99b5d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 0x55bdebe26b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33848/0 (socket says 172.21.15.50:33848) 2024-04-23T18:24:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99b5d9b640 1 -- 172.21.15.50:0/47884795 learned_addr learned my addr 172.21.15.50:0/47884795 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99b6d9d640 1 --2- 172.21.15.50:0/47884795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe36800 0x55bdebe4e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99b659c640 1 --2- 172.21.15.50:0/47884795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe42020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99b5d9b640 1 -- 172.21.15.50:0/47884795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 msgr2=0x55bdebe42020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99b5d9b640 1 --2- 172.21.15.50:0/47884795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe42020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99b5d9b640 1 -- 172.21.15.50:0/47884795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe36800 msgr2=0x55bdebe4e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99b5d9b640 1 --2- 172.21.15.50:0/47884795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe36800 0x55bdebe4e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99b5d9b640 1 -- 172.21.15.50:0/47884795 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdebe23500 con 0x55bdebe2bf40 2024-04-23T18:24:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.951+0000 7f99b5d9b640 1 --2- 172.21.15.50:0/47884795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 0x55bdebe26b30 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00bd30 tx=0x7f99ac012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c80b7f443575f357 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.952+0000 7f999ffff640 1 -- 172.21.15.50:0/47884795 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac004030 con 0x55bdebe2bf40 2024-04-23T18:24:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.952+0000 7f999ffff640 1 -- 172.21.15.50:0/47884795 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99ac0041d0 con 0x55bdebe2bf40 2024-04-23T18:24:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.952+0000 7f999ffff640 1 -- 172.21.15.50:0/47884795 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac010610 con 0x55bdebe2bf40 2024-04-23T18:24:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.952+0000 7f99bfa0a740 1 -- 172.21.15.50:0/47884795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 msgr2=0x55bdebe26b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.952+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/47884795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 0x55bdebe26b30 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00bd30 tx=0x7f99ac012820 comp rx=0 tx=0).stop 2024-04-23T18:24:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.952+0000 7f99bfa0a740 1 -- 172.21.15.50:0/47884795 shutdown_connections 2024-04-23T18:24:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.952+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/47884795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe36800 0x55bdebe4e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.952+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/47884795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe2bf40 0x55bdebe26b30 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.952+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/47884795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe42020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.952+0000 7f99bfa0a740 1 -- 172.21.15.50:0/47884795 >> 172.21.15.50:0/47884795 conn(0x55bdebe27b30 msgr2=0x55bdebe26420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.952+0000 7f99bfa0a740 1 -- 172.21.15.50:0/47884795 shutdown_connections 2024-04-23T18:24:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.952+0000 7f99bfa0a740 1 -- 172.21.15.50:0/47884795 wait complete. 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99bfa0a740 1 -- start start 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99bfa0a740 1 Processor -- start 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99bfa0a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe3e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99bfa0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2bf40 0x55bdebe3bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99bfa0a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 0x55bdebe3c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99bfa0a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81ae430 con 0x55bdebe36800 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99bfa0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81af8a0 con 0x55bdebe2bf40 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99bfa0a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81aeb00 con 0x55bdebe2b620 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99bfa0a740 10 client.0 _ll_register_callbacks cb 0x7f99b0010c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99b6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 0x55bdebe3c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99b6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 0x55bdebe3c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33852/0 (socket says 172.21.15.50:33852) 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99b6d9d640 1 -- 172.21.15.50:0/3050685612 learned_addr learned my addr 172.21.15.50:0/3050685612 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99b5d9b640 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2bf40 0x55bdebe3bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.953+0000 7f99b659c640 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe3e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f99b6d9d640 1 -- 172.21.15.50:0/3050685612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 msgr2=0x55bdebe3e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f99b6d9d640 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe3e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f99b6d9d640 1 -- 172.21.15.50:0/3050685612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2bf40 msgr2=0x55bdebe3bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f99b6d9d640 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2bf40 0x55bdebe3bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f99b6d9d640 1 -- 172.21.15.50:0/3050685612 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdebe3c900 con 0x55bdebe36800 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f99b659c640 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe3e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f99b6d9d640 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 0x55bdebe3c100 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x55bdebe367d0 tx=0x7f99b0012680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73530475d1496aa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f999dffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f999dffb640 1 -- 172.21.15.50:0/3050685612 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b0025040 con 0x55bdebe36800 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f999dffb640 1 -- 172.21.15.50:0/3050685612 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f99b0013850 con 0x55bdebe36800 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3050685612 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99ac0041d0 con 0x55bdebe36800 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f999dffb640 1 -- 172.21.15.50:0/3050685612 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f99b0042050 con 0x55bdebe36800 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f99bfa0a740 10 client.16272 fetch_fsmap learned FSMap version 32 2024-04-23T18:24:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3050685612 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bdebeacaf0 con 0x55bdebe36800 2024-04-23T18:24:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f999dffb640 1 -- 172.21.15.50:0/3050685612 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99ac0041d0 con 0x55bdebe36800 2024-04-23T18:24:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.954+0000 7f999dffb640 1 -- 172.21.15.50:0/3050685612 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f99b001f7f0 con 0x55bdebe36800 2024-04-23T18:24:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.955+0000 7f99bfa0a740 10 client.16272 fetch_fsmap finished waiting for FSMap version 32 2024-04-23T18:24:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.955+0000 7f99bfa0a740 10 client.16272 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:24:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.955+0000 7f99bfa0a740 20 client.16272 populate_metadata read hostname 'smithi050' 2024-04-23T18:24:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.955+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bdebead290 0x55bdebeaf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:24:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.955+0000 7f99bfa0a740 4 client.16272 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:24:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.955+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3050685612 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bdebd98200 con 0x55bdebead290 2024-04-23T18:24:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.955+0000 7f99b659c640 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bdebead290 0x55bdebeaf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:24:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.955+0000 7f99b659c640 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bdebead290 0x55bdebeaf6b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9269416c2ee98ef1 server_cookie=ac29387ddbd58c35 in_seq=0 out_seq=0 2024-04-23T18:24:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.955+0000 7f999dffb640 10 client.16272 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:54.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f999dffb640 1 -- 172.21.15.50:0/3050685612 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bdebd98200 con 0x55bdebead290 2024-04-23T18:24:54.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f999dffb640 10 client.16272 handle_command_reply: tid=0 2024-04-23T18:24:54.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f99bfa0a740 1 client.16272 shutdown 2024-04-23T18:24:54.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3050685612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 msgr2=0x55bdebe3c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:54.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 0x55bdebe3c100 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x55bdebe367d0 tx=0x7f99b0012680 comp rx=0 tx=0).stop 2024-04-23T18:24:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3050685612 shutdown_connections 2024-04-23T18:24:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bdebead290 0x55bdebeaf6b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bdebe36800 0x55bdebe3c100 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdebe2bf40 0x55bdebe3bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f999dffb640 0 client.16272 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:24:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f99bfa0a740 1 --2- 172.21.15.50:0/3050685612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bdebe2b620 0x55bdebe3e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3050685612 >> 172.21.15.50:0/3050685612 conn(0x55bdebe27b30 msgr2=0x55bdebe26cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3050685612 shutdown_connections 2024-04-23T18:24:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f99bfa0a740 1 -- 172.21.15.50:0/3050685612 wait complete. 2024-04-23T18:24:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.983+0000 7f99bfa0a740 20 client.16272 trim_cache size 0 max 16384 2024-04-23T18:24:54.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:24:54.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:24:54.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:24:54.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:24:54.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.984+0000 7f99bccef640 1 -- 172.21.15.50:0/1525534850 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99a00a3df0 msgr2=0x7f99a00a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.984+0000 7f99bccef640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99a00a3df0 0x7f99a00a62b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f99b000bfd0 tx=0x7f99b0005c20 comp rx=0 tx=0).stop 2024-04-23T18:24:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.984+0000 7f99bccef640 1 -- 172.21.15.50:0/1525534850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8103220 msgr2=0x7f99b8100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:24:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.984+0000 7f99bccef640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8103220 0x7f99b8100e70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f99a8005940 tx=0x7f99a80042f0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.984+0000 7f99bccef640 1 -- 172.21.15.50:0/1525534850 shutdown_connections 2024-04-23T18:24:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.984+0000 7f99bccef640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99a00a3df0 0x7f99a00a62b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.984+0000 7f99bccef640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99b810c560 0x7f99b81018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.984+0000 7f99bccef640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99b8103b40 0x7f99b81013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.984+0000 7f99bccef640 1 --2- 172.21.15.50:0/1525534850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b8103220 0x7f99b8100e70 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:24:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.984+0000 7f99bccef640 1 -- 172.21.15.50:0/1525534850 >> 172.21.15.50:0/1525534850 conn(0x7f99b8077fd0 msgr2=0x7f99b80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:24:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.984+0000 7f99bccef640 1 -- 172.21.15.50:0/1525534850 shutdown_connections 2024-04-23T18:24:54.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:24:54.984+0000 7f99bccef640 1 -- 172.21.15.50:0/1525534850 wait complete. 2024-04-23T18:24:55.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.36 MB/sec execute 209 sec latency 24402.304 ms 2024-04-23T18:24:56.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.34 MB/sec execute 210 sec latency 25402.417 ms 2024-04-23T18:24:56.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:56 smithi177 ceph-mon[32308]: pgmap v197: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-04-23T18:24:56.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:56 smithi188 ceph-mon[32694]: pgmap v197: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-04-23T18:24:56.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:56 smithi050 ceph-mon[29187]: pgmap v197: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-04-23T18:24:57.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.31 MB/sec execute 211 sec latency 26402.481 ms 2024-04-23T18:24:58.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.29 MB/sec execute 212 sec latency 27402.534 ms 2024-04-23T18:24:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:24:58 smithi177 ceph-mon[32308]: pgmap v198: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 2.1 MiB/s wr, 103 op/s 2024-04-23T18:24:58.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:24:58 smithi188 ceph-mon[32694]: pgmap v198: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 2.1 MiB/s wr, 103 op/s 2024-04-23T18:24:58.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:24:58 smithi050 ceph-mon[29187]: pgmap v198: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 2.1 MiB/s wr, 103 op/s 2024-04-23T18:24:59.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.26 MB/sec execute 213 sec latency 28402.580 ms 2024-04-23T18:25:00.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 402815 5.24 MB/sec execute 214 sec latency 29402.631 ms 2024-04-23T18:25:00.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:25:00.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.495+0000 7fe029860640 1 -- 172.21.15.50:0/325611075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe024102c20 msgr2=0x7fe024106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.495+0000 7fe029860640 1 --2- 172.21.15.50:0/325611075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe024102c20 0x7fe024106bb0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fe00c009900 tx=0x7fe00c02a440 comp rx=0 tx=0).stop 2024-04-23T18:25:00.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.495+0000 7fe029860640 1 -- 172.21.15.50:0/325611075 shutdown_connections 2024-04-23T18:25:00.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.495+0000 7fe029860640 1 --2- 172.21.15.50:0/325611075 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe024107290 0x7fe02410ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.495+0000 7fe029860640 1 --2- 172.21.15.50:0/325611075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe024102c20 0x7fe024106bb0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.495+0000 7fe029860640 1 --2- 172.21.15.50:0/325611075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe024102270 0x7fe024102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.495+0000 7fe029860640 1 -- 172.21.15.50:0/325611075 >> 172.21.15.50:0/325611075 conn(0x7fe0240790e0 msgr2=0x7fe0240794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:00.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.495+0000 7fe029860640 1 -- 172.21.15.50:0/325611075 shutdown_connections 2024-04-23T18:25:00.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.495+0000 7fe029860640 1 -- 172.21.15.50:0/325611075 wait complete. 2024-04-23T18:25:00.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.496+0000 7fe029860640 1 Processor -- start 2024-04-23T18:25:00.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.496+0000 7fe029860640 1 -- start start 2024-04-23T18:25:00.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.496+0000 7fe029860640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe024102270 0x7fe024074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.496+0000 7fe029860640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe024107290 0x7fe024074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.496+0000 7fe029860640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe024074cd0 0x7fe0241a6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.496+0000 7fe029860640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe0241122f0 con 0x7fe024102270 2024-04-23T18:25:00.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.496+0000 7fe029860640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe024112170 con 0x7fe024107290 2024-04-23T18:25:00.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.496+0000 7fe029860640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe024112470 con 0x7fe024074cd0 2024-04-23T18:25:00.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe022ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe024102270 0x7fe024074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe022ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe024102270 0x7fe024074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33864/0 (socket says 172.21.15.50:33864) 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe022ffd640 1 -- 172.21.15.50:0/944496197 learned_addr learned my addr 172.21.15.50:0/944496197 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe01bfff640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe024107290 0x7fe024074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe0237fe640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe024074cd0 0x7fe0241a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe022ffd640 1 -- 172.21.15.50:0/944496197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe024074cd0 msgr2=0x7fe0241a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe022ffd640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe024074cd0 0x7fe0241a6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe022ffd640 1 -- 172.21.15.50:0/944496197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe024107290 msgr2=0x7fe024074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe022ffd640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe024107290 0x7fe024074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe022ffd640 1 -- 172.21.15.50:0/944496197 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0241a71e0 con 0x7fe024102270 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe022ffd640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe024102270 0x7fe024074250 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fe01400b450 tx=0x7fe01400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe02885e640 1 -- 172.21.15.50:0/944496197 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe014015020 con 0x7fe024102270 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe02885e640 1 -- 172.21.15.50:0/944496197 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe014011d40 con 0x7fe024102270 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe029860640 1 -- 172.21.15.50:0/944496197 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0241a74d0 con 0x7fe024102270 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.497+0000 7fe02885e640 1 -- 172.21.15.50:0/944496197 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01400f340 con 0x7fe024102270 2024-04-23T18:25:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.498+0000 7fe029860640 1 -- 172.21.15.50:0/944496197 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0241a79f0 con 0x7fe024102270 2024-04-23T18:25:00.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.499+0000 7fe029860640 1 -- 172.21.15.50:0/944496197 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe024103690 con 0x7fe024102270 2024-04-23T18:25:00.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.501+0000 7fe02885e640 1 -- 172.21.15.50:0/944496197 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe014018020 con 0x7fe024102270 2024-04-23T18:25:00.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.501+0000 7fe02885e640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe0100a3d50 0x7fe0100a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.501+0000 7fe01bfff640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe0100a3d50 0x7fe0100a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.501+0000 7fe02885e640 1 -- 172.21.15.50:0/944496197 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7fe0140d05e0 con 0x7fe024102270 2024-04-23T18:25:00.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.502+0000 7fe01bfff640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe0100a3d50 0x7fe0100a6210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe00c009230 tx=0x7fe00c005e80 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:00.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.506+0000 7fe02885e640 1 -- 172.21.15.50:0/944496197 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe01401f020 con 0x7fe024102270 2024-04-23T18:25:00.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:00 smithi177 ceph-mon[32308]: pgmap v199: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-04-23T18:25:00.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:00 smithi188 ceph-mon[32694]: pgmap v199: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-04-23T18:25:00.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:00 smithi050 ceph-mon[29187]: pgmap v199: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-04-23T18:25:00.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.742+0000 7fe029860640 1 -- 172.21.15.50:0/944496197 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe024068d70 con 0x7fe024102270 2024-04-23T18:25:00.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.743+0000 7fe02885e640 1 -- 172.21.15.50:0/944496197 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11876 (secure 0 0 0) 0x7fe024068d70 con 0x7fe024102270 2024-04-23T18:25:00.739 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:25:00.739 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:24:49.937177+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25822":{"gid":25822,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/911343542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":911343542},{"type":"v1","addr":"172.21.15.177:6837","nonce":911343542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:25:00.740 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 32 2024-04-23T18:25:00.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.745+0000 7fe029860640 1 -- 172.21.15.50:0/944496197 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe0100a3d50 msgr2=0x7fe0100a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.745+0000 7fe029860640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe0100a3d50 0x7fe0100a6210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe00c009230 tx=0x7fe00c005e80 comp rx=0 tx=0).stop 2024-04-23T18:25:00.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.745+0000 7fe029860640 1 -- 172.21.15.50:0/944496197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe024102270 msgr2=0x7fe024074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.745+0000 7fe029860640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe024102270 0x7fe024074250 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fe01400b450 tx=0x7fe01400b910 comp rx=0 tx=0).stop 2024-04-23T18:25:00.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.745+0000 7fe029860640 1 -- 172.21.15.50:0/944496197 shutdown_connections 2024-04-23T18:25:00.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.745+0000 7fe029860640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe0100a3d50 0x7fe0100a6210 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.745+0000 7fe029860640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe024074cd0 0x7fe0241a6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.745+0000 7fe029860640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe024107290 0x7fe024074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.745+0000 7fe029860640 1 --2- 172.21.15.50:0/944496197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe024102270 0x7fe024074250 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.745+0000 7fe029860640 1 -- 172.21.15.50:0/944496197 >> 172.21.15.50:0/944496197 conn(0x7fe0240790e0 msgr2=0x7fe024100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:00.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.745+0000 7fe029860640 1 -- 172.21.15.50:0/944496197 shutdown_connections 2024-04-23T18:25:00.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.745+0000 7fe029860640 1 -- 172.21.15.50:0/944496197 wait complete. 2024-04-23T18:25:00.749 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id 19aaec48 --release --await' 2024-04-23T18:25:00.750 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id 19aaec48 --release --await 2024-04-23T18:25:00.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.865+0000 7fbbf235d640 1 -- 172.21.15.50:0/1192121086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbec06b170 msgr2=0x7fbbec106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.865+0000 7fbbf235d640 1 --2- 172.21.15.50:0/1192121086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbec06b170 0x7fbbec106ba0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009900 tx=0x7fbbe002a410 comp rx=0 tx=0).stop 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.865+0000 7fbbf235d640 1 -- 172.21.15.50:0/1192121086 shutdown_connections 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.865+0000 7fbbf235d640 1 --2- 172.21.15.50:0/1192121086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbec107280 0x7fbbec10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.865+0000 7fbbf235d640 1 --2- 172.21.15.50:0/1192121086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbec06b170 0x7fbbec106ba0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.865+0000 7fbbf235d640 1 --2- 172.21.15.50:0/1192121086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbec06a7c0 0x7fbbec06aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.865+0000 7fbbf235d640 1 -- 172.21.15.50:0/1192121086 >> 172.21.15.50:0/1192121086 conn(0x7fbbec076c50 msgr2=0x7fbbec077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.865+0000 7fbbf235d640 1 -- 172.21.15.50:0/1192121086 shutdown_connections 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.865+0000 7fbbf235d640 1 -- 172.21.15.50:0/1192121086 wait complete. 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbf235d640 1 Processor -- start 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbf235d640 1 -- start start 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbf235d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbec06a7c0 0x7fbbec198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbf235d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbec107280 0x7fbbec198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbf235d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbec19cf70 0x7fbbec19d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbf235d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec112180 con 0x7fbbec06a7c0 2024-04-23T18:25:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbf235d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec112000 con 0x7fbbec107280 2024-04-23T18:25:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbf235d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec112300 con 0x7fbbec19cf70 2024-04-23T18:25:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbebfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbec06a7c0 0x7fbbec198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbebfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbec06a7c0 0x7fbbec198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33868/0 (socket says 172.21.15.50:33868) 2024-04-23T18:25:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbebfff640 1 -- 172.21.15.50:0/3803864145 learned_addr learned my addr 172.21.15.50:0/3803864145 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbf08fa640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbec19cf70 0x7fbbec19d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbeb7fe640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbec107280 0x7fbbec198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbebfff640 1 -- 172.21.15.50:0/3803864145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbec19cf70 msgr2=0x7fbbec19d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbebfff640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbec19cf70 0x7fbbec19d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbebfff640 1 -- 172.21.15.50:0/3803864145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbec107280 msgr2=0x7fbbec198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbebfff640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbec107280 0x7fbbec198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.866+0000 7fbbebfff640 1 -- 172.21.15.50:0/3803864145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbec19da60 con 0x7fbbec06a7c0 2024-04-23T18:25:00.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.867+0000 7fbbebfff640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbec06a7c0 0x7fbbec198700 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00d950 tx=0x7fbbdc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:00.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.867+0000 7fbbe97fa640 1 -- 172.21.15.50:0/3803864145 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc017070 con 0x7fbbec06a7c0 2024-04-23T18:25:00.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.867+0000 7fbbe97fa640 1 -- 172.21.15.50:0/3803864145 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbdc009c30 con 0x7fbbec06a7c0 2024-04-23T18:25:00.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.867+0000 7fbbf235d640 1 -- 172.21.15.50:0/3803864145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbec19dd50 con 0x7fbbec06a7c0 2024-04-23T18:25:00.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.867+0000 7fbbe97fa640 1 -- 172.21.15.50:0/3803864145 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc00f620 con 0x7fbbec06a7c0 2024-04-23T18:25:00.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.867+0000 7fbbf235d640 1 -- 172.21.15.50:0/3803864145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbec1a5560 con 0x7fbbec06a7c0 2024-04-23T18:25:00.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.869+0000 7fbbe97fa640 1 -- 172.21.15.50:0/3803864145 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbbdc010460 con 0x7fbbec06a7c0 2024-04-23T18:25:00.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.869+0000 7fbbf5074740 1 Processor -- start 2024-04-23T18:25:00.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.869+0000 7fbbf5074740 1 -- start start 2024-04-23T18:25:00.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.869+0000 7fbbf5074740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 0x55ce9ab7f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.869+0000 7fbbf5074740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab89f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.869+0000 7fbbf5074740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab8a480 0x55ce9ab8c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.869+0000 7fbbf5074740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ce9aae46d0 con 0x55ce9ab7f490 2024-04-23T18:25:00.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.869+0000 7fbbf5074740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ce9aa811a0 con 0x55ce9ab8a480 2024-04-23T18:25:00.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.869+0000 7fbbf5074740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ce9aae0690 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbf08fa640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 0x55ce9ab7f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbf08fa640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 0x55ce9ab7f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33882/0 (socket says 172.21.15.50:33882) 2024-04-23T18:25:00.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbf08fa640 1 -- 172.21.15.50:0/3559825019 learned_addr learned my addr 172.21.15.50:0/3559825019 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:00.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbebfff640 1 --2- 172.21.15.50:0/3559825019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab89f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbe97fa640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbbd40a3da0 0x7fbbd40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbeb7fe640 1 --2- 172.21.15.50:0/3559825019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab8a480 0x55ce9ab8c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbf08fa640 1 -- 172.21.15.50:0/3559825019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab7fdb0 msgr2=0x55ce9ab89f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:00.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbf08fa640 1 --2- 172.21.15.50:0/3559825019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab89f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbf08fa640 1 -- 172.21.15.50:0/3559825019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab8a480 msgr2=0x55ce9ab8c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:00.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbf08fa640 1 --2- 172.21.15.50:0/3559825019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab8a480 0x55ce9ab8c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbf08fa640 1 -- 172.21.15.50:0/3559825019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce9aa8b5b0 con 0x55ce9ab7f490 2024-04-23T18:25:00.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbeb7fe640 1 --2- 172.21.15.50:0/3559825019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab8a480 0x55ce9ab8c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:25:00.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbebfff640 1 --2- 172.21.15.50:0/3559825019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab89f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:00.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbe97fa640 1 -- 172.21.15.50:0/3803864145 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7fbbdc0d0410 con 0x7fbbec06a7c0 2024-04-23T18:25:00.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbf08fa640 1 --2- 172.21.15.50:0/3559825019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 0x55ce9ab7f870 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4007f40 tx=0x7fbbe400f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3564a18fee0855 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:00.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbf08fa640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbbd40a3da0 0x7fbbd40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbd27fc640 1 -- 172.21.15.50:0/3559825019 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4010b20 con 0x55ce9ab7f490 2024-04-23T18:25:00.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbd27fc640 1 -- 172.21.15.50:0/3559825019 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbe401add0 con 0x55ce9ab7f490 2024-04-23T18:25:00.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.870+0000 7fbbd27fc640 1 -- 172.21.15.50:0/3559825019 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4019b80 con 0x55ce9ab7f490 2024-04-23T18:25:00.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.871+0000 7fbbf08fa640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbbd40a3da0 0x7fbbd40a6260 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbbe401e000 tx=0x7fbbe401f000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:00.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.871+0000 7fbbf5074740 1 -- 172.21.15.50:0/3559825019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 msgr2=0x55ce9ab7f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.871+0000 7fbbf5074740 1 --2- 172.21.15.50:0/3559825019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 0x55ce9ab7f870 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4007f40 tx=0x7fbbe400f850 comp rx=0 tx=0).stop 2024-04-23T18:25:00.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.871+0000 7fbbf5074740 1 -- 172.21.15.50:0/3559825019 shutdown_connections 2024-04-23T18:25:00.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.871+0000 7fbbf5074740 1 --2- 172.21.15.50:0/3559825019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab8a480 0x55ce9ab8c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.871+0000 7fbbf5074740 1 --2- 172.21.15.50:0/3559825019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab89f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.871+0000 7fbbf5074740 1 --2- 172.21.15.50:0/3559825019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 0x55ce9ab7f870 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.871+0000 7fbbf5074740 1 -- 172.21.15.50:0/3559825019 >> 172.21.15.50:0/3559825019 conn(0x55ce9ab7b7b0 msgr2=0x55ce9ab7bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:00.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.871+0000 7fbbf5074740 1 -- 172.21.15.50:0/3559825019 shutdown_connections 2024-04-23T18:25:00.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.871+0000 7fbbf5074740 1 -- 172.21.15.50:0/3559825019 wait complete. 2024-04-23T18:25:00.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbf5074740 1 -- start start 2024-04-23T18:25:00.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbf5074740 1 Processor -- start 2024-04-23T18:25:00.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbf5074740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 0x55ce9ab89e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbf5074740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab87170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbf5074740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab8a480 0x55ce9ab87780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbf5074740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55ce9ab8d310 con 0x55ce9ab7f490 2024-04-23T18:25:00.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbf5074740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ce9aa7eb20 con 0x55ce9ab8a480 2024-04-23T18:25:00.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbf5074740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55ce9ab8d490 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbf5074740 10 client.0 _ll_register_callbacks cb 0x55ce9aa7e530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:00.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbebfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 0x55ce9ab89e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbebfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 0x55ce9ab89e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33898/0 (socket says 172.21.15.50:33898) 2024-04-23T18:25:00.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbebfff640 1 -- 172.21.15.50:0/4044532177 learned_addr learned my addr 172.21.15.50:0/4044532177 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:00.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbeb7fe640 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab87170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbf08fa640 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab8a480 0x55ce9ab87780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbebfff640 1 -- 172.21.15.50:0/4044532177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab7fdb0 msgr2=0x55ce9ab87170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:00.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbebfff640 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab87170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbebfff640 1 -- 172.21.15.50:0/4044532177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab8a480 msgr2=0x55ce9ab87780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:00.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbebfff640 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab8a480 0x55ce9ab87780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.872+0000 7fbbebfff640 1 -- 172.21.15.50:0/4044532177 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce9a7f6920 con 0x55ce9ab7f490 2024-04-23T18:25:00.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbebfff640 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 0x55ce9ab89e80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0cdb70 tx=0x7fbbdc09bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1817625aee19a89 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:00.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbcb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:25:00.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbcb7fe640 1 -- 172.21.15.50:0/4044532177 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc020070 con 0x55ce9ab7f490 2024-04-23T18:25:00.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbcb7fe640 1 -- 172.21.15.50:0/4044532177 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7fbbdc0a4b40 con 0x55ce9ab7f490 2024-04-23T18:25:00.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbf5074740 1 -- 172.21.15.50:0/4044532177 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbe401add0 con 0x55ce9ab7f490 2024-04-23T18:25:00.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbcb7fe640 1 -- 172.21.15.50:0/4044532177 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbbdc0cdf90 con 0x55ce9ab7f490 2024-04-23T18:25:00.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbf5074740 10 client.16290 fetch_fsmap learned FSMap version 32 2024-04-23T18:25:00.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbf5074740 1 -- 172.21.15.50:0/4044532177 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ce9a8c2d30 con 0x55ce9ab7f490 2024-04-23T18:25:00.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbcb7fe640 1 -- 172.21.15.50:0/4044532177 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbe401add0 con 0x55ce9ab7f490 2024-04-23T18:25:00.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbcb7fe640 1 -- 172.21.15.50:0/4044532177 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fbbdc0b5ae0 con 0x55ce9ab7f490 2024-04-23T18:25:00.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbf5074740 10 client.16290 fetch_fsmap finished waiting for FSMap version 32 2024-04-23T18:25:00.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbf5074740 10 client.16290 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:25:00.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbf5074740 20 client.16290 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:00.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbf5074740 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ce9ab9a500 0x55ce9ab9c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbf5074740 4 client.16290 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:00.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.873+0000 7fbbf5074740 1 -- 172.21.15.50:0/4044532177 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce9aaa2500 con 0x55ce9ab9a500 2024-04-23T18:25:00.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.874+0000 7fbbeb7fe640 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ce9ab9a500 0x55ce9ab9c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.874+0000 7fbbcb7fe640 10 client.16290 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:00.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.874+0000 7fbbeb7fe640 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ce9ab9a500 0x55ce9ab9c8f0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a94712c7cd2a3efe server_cookie=87ec33fa28e24b3b in_seq=0 out_seq=0 2024-04-23T18:25:00.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbcb7fe640 1 -- 172.21.15.50:0/4044532177 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce9aaa2500 con 0x55ce9ab9a500 2024-04-23T18:25:00.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbcb7fe640 10 client.16290 handle_command_reply: tid=0 2024-04-23T18:25:00.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbf5074740 1 client.16290 shutdown 2024-04-23T18:25:00.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbf5074740 1 -- 172.21.15.50:0/4044532177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 msgr2=0x55ce9ab89e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbf5074740 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 0x55ce9ab89e80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0cdb70 tx=0x7fbbdc09bd80 comp rx=0 tx=0).stop 2024-04-23T18:25:00.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbf5074740 1 -- 172.21.15.50:0/4044532177 shutdown_connections 2024-04-23T18:25:00.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbf5074740 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ce9ab9a500 0x55ce9ab9c8f0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbf5074740 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab8a480 0x55ce9ab87780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbf5074740 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab87170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbf5074740 1 --2- 172.21.15.50:0/4044532177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7f490 0x55ce9ab89e80 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbcb7fe640 0 client.16290 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:00.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbf5074740 1 -- 172.21.15.50:0/4044532177 >> 172.21.15.50:0/4044532177 conn(0x55ce9ab7b7b0 msgr2=0x55ce9ab815b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:00.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbf5074740 1 -- 172.21.15.50:0/4044532177 shutdown_connections 2024-04-23T18:25:00.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbf5074740 1 -- 172.21.15.50:0/4044532177 wait complete. 2024-04-23T18:25:00.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.875+0000 7fbbf5074740 20 client.16290 trim_cache size 0 max 16384 2024-04-23T18:25:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbf5074740 1 Processor -- start 2024-04-23T18:25:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbf5074740 1 -- start start 2024-04-23T18:25:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbf5074740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab7f490 0x55ce9ab899b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbf5074740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 0x55ce9aba21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbf5074740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab8a480 0x55ce9aba2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbf5074740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec1b0380 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbf5074740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec1adc20 con 0x55ce9ab7f490 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbf5074740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec1aba70 con 0x55ce9ab8a480 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbeb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 0x55ce9aba21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbeb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 0x55ce9aba21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33912/0 (socket says 172.21.15.50:33912) 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbeb7fe640 1 -- 172.21.15.50:0/947098065 learned_addr learned my addr 172.21.15.50:0/947098065 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbf08fa640 1 --2- 172.21.15.50:0/947098065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab8a480 0x55ce9aba2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbebfff640 1 --2- 172.21.15.50:0/947098065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab7f490 0x55ce9ab899b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbeb7fe640 1 -- 172.21.15.50:0/947098065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab8a480 msgr2=0x55ce9aba2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbeb7fe640 1 --2- 172.21.15.50:0/947098065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab8a480 0x55ce9aba2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbeb7fe640 1 -- 172.21.15.50:0/947098065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab7f490 msgr2=0x55ce9ab899b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbeb7fe640 1 --2- 172.21.15.50:0/947098065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab7f490 0x55ce9ab899b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbeb7fe640 1 -- 172.21.15.50:0/947098065 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce9ab7c7b0 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbf08fa640 1 --2- 172.21.15.50:0/947098065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab8a480 0x55ce9aba2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.893+0000 7fbbebfff640 1 --2- 172.21.15.50:0/947098065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab7f490 0x55ce9ab899b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:25:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbeb7fe640 1 --2- 172.21.15.50:0/947098065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 0x55ce9aba21c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0005940 tx=0x7fbbe0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d421f46231e2760f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbd1ffb640 1 -- 172.21.15.50:0/947098065 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0035040 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbd1ffb640 1 -- 172.21.15.50:0/947098065 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbe0048070 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbd1ffb640 1 -- 172.21.15.50:0/947098065 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0005040 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbf5074740 1 -- 172.21.15.50:0/947098065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 msgr2=0x55ce9aba21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbf5074740 1 --2- 172.21.15.50:0/947098065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 0x55ce9aba21c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0005940 tx=0x7fbbe0000bc0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbf5074740 1 -- 172.21.15.50:0/947098065 shutdown_connections 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbf5074740 1 --2- 172.21.15.50:0/947098065 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab8a480 0x55ce9aba2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbf5074740 1 --2- 172.21.15.50:0/947098065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 0x55ce9aba21c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbf5074740 1 --2- 172.21.15.50:0/947098065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab7f490 0x55ce9ab899b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbf5074740 1 -- 172.21.15.50:0/947098065 >> 172.21.15.50:0/947098065 conn(0x55ce9ab7b7b0 msgr2=0x55ce9ab85a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbf5074740 1 -- 172.21.15.50:0/947098065 shutdown_connections 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.894+0000 7fbbf5074740 1 -- 172.21.15.50:0/947098065 wait complete. 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbf5074740 1 -- start start 2024-04-23T18:25:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbf5074740 1 Processor -- start 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbf5074740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab7f490 0x55ce9ab92ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbf5074740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab8d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbf5074740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab8a480 0x55ce9ab8dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbf5074740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec1ac0e0 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbf5074740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec1ad550 con 0x55ce9ab7f490 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbf5074740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec1ac7b0 con 0x55ce9ab8a480 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbf5074740 10 client.0 _ll_register_callbacks cb 0x7fbbdc09c200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbeb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab8d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbeb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab8d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33914/0 (socket says 172.21.15.50:33914) 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbeb7fe640 1 -- 172.21.15.50:0/3117840581 learned_addr learned my addr 172.21.15.50:0/3117840581 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbf08fa640 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab8a480 0x55ce9ab8dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.895+0000 7fbbebfff640 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab7f490 0x55ce9ab92ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbeb7fe640 1 -- 172.21.15.50:0/3117840581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab8a480 msgr2=0x55ce9ab8dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbeb7fe640 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab8a480 0x55ce9ab8dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbeb7fe640 1 -- 172.21.15.50:0/3117840581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab7f490 msgr2=0x55ce9ab92ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbeb7fe640 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab7f490 0x55ce9ab92ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbeb7fe640 1 -- 172.21.15.50:0/3117840581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce9ab8e430 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbeb7fe640 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab8d6a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0004230 tx=0x7fbbe0004120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e007b324b47e5b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbcb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbcb7fe640 1 -- 172.21.15.50:0/3117840581 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0050020 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbcb7fe640 1 -- 172.21.15.50:0/3117840581 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7fbbe0005040 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbf5074740 1 -- 172.21.15.50:0/3117840581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbe0048070 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbcb7fe640 1 -- 172.21.15.50:0/3117840581 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbbe005b050 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbf5074740 10 client.16296 fetch_fsmap learned FSMap version 32 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbf5074740 1 -- 172.21.15.50:0/3117840581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ce9ac05e80 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbcb7fe640 1 -- 172.21.15.50:0/3117840581 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbe0048070 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbcb7fe640 1 -- 172.21.15.50:0/3117840581 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fbbe0004cd0 con 0x55ce9ab7fdb0 2024-04-23T18:25:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbf5074740 10 client.16296 fetch_fsmap finished waiting for FSMap version 32 2024-04-23T18:25:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbf5074740 10 client.16296 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:25:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbf5074740 20 client.16296 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbf5074740 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ce9ac06540 0x55ce9ac08930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbf5074740 4 client.16296 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "19aaec48", "release": true, "await": true}] 2024-04-23T18:25:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.896+0000 7fbbf5074740 1 -- 172.21.15.50:0/3117840581 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "19aaec48", "release": true, "await": true}) -- 0x55ce9aaf20f0 con 0x55ce9ac06540 2024-04-23T18:25:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.897+0000 7fbbebfff640 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ce9ac06540 0x55ce9ac08930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.897+0000 7fbbebfff640 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ce9ac06540 0x55ce9ac08930 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18f9102bab458bfd server_cookie=4b11161bf1420bb1 in_seq=0 out_seq=0 2024-04-23T18:25:00.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.897+0000 7fbbcb7fe640 10 client.16296 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:00.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbcb7fe640 1 -- 172.21.15.50:0/3117840581 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55ce9aaf20f0 con 0x55ce9ac06540 2024-04-23T18:25:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbcb7fe640 10 client.16296 handle_command_reply: tid=0 2024-04-23T18:25:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbf5074740 1 client.16296 shutdown 2024-04-23T18:25:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbf5074740 1 -- 172.21.15.50:0/3117840581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 msgr2=0x55ce9ab8d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbf5074740 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab8d6a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0004230 tx=0x7fbbe0004120 comp rx=0 tx=0).stop 2024-04-23T18:25:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbf5074740 1 -- 172.21.15.50:0/3117840581 shutdown_connections 2024-04-23T18:25:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbf5074740 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55ce9ac06540 0x55ce9ac08930 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbf5074740 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55ce9ab8a480 0x55ce9ab8dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbcb7fe640 0 client.16296 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbf5074740 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55ce9ab7fdb0 0x55ce9ab8d6a0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbf5074740 1 --2- 172.21.15.50:0/3117840581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce9ab7f490 0x55ce9ab92ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbf5074740 1 -- 172.21.15.50:0/3117840581 >> 172.21.15.50:0/3117840581 conn(0x55ce9ab7b7b0 msgr2=0x55ce9ab80be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:00.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbf5074740 1 -- 172.21.15.50:0/3117840581 shutdown_connections 2024-04-23T18:25:00.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbf5074740 1 -- 172.21.15.50:0/3117840581 wait complete. 2024-04-23T18:25:00.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.903+0000 7fbbf5074740 20 client.16296 trim_cache size 0 max 16384 2024-04-23T18:25:00.899 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:25:00.899 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 32, 2024-04-23T18:25:00.899 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 20, 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "19aaec48": { 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 20, 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 89.5, 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:25:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:25:00.901 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:25:00.901 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:25:00.901 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:25:00.901 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:00.901 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:00.901 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:00.901 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:00.901 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:00.901 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:25:00.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.904+0000 7fbbf235d640 1 -- 172.21.15.50:0/3803864145 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbbd40a3da0 msgr2=0x7fbbd40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.904+0000 7fbbf235d640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbbd40a3da0 0x7fbbd40a6260 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbbe401e000 tx=0x7fbbe401f000 comp rx=0 tx=0).stop 2024-04-23T18:25:00.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.904+0000 7fbbf235d640 1 -- 172.21.15.50:0/3803864145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbec06a7c0 msgr2=0x7fbbec198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:00.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.904+0000 7fbbf235d640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbec06a7c0 0x7fbbec198700 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00d950 tx=0x7fbbdc00de10 comp rx=0 tx=0).stop 2024-04-23T18:25:00.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.904+0000 7fbbf235d640 1 -- 172.21.15.50:0/3803864145 shutdown_connections 2024-04-23T18:25:00.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.904+0000 7fbbf235d640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbbd40a3da0 0x7fbbd40a6260 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.904+0000 7fbbf235d640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbbec19cf70 0x7fbbec19d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.904+0000 7fbbf235d640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbec107280 0x7fbbec198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.904+0000 7fbbf235d640 1 --2- 172.21.15.50:0/3803864145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbec06a7c0 0x7fbbec198700 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:00.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.904+0000 7fbbf235d640 1 -- 172.21.15.50:0/3803864145 >> 172.21.15.50:0/3803864145 conn(0x7fbbec076c50 msgr2=0x7fbbec0ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:00.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.904+0000 7fbbf235d640 1 -- 172.21.15.50:0/3803864145 shutdown_connections 2024-04-23T18:25:00.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:00.904+0000 7fbbf235d640 1 -- 172.21.15.50:0/3803864145 wait complete. 2024-04-23T18:25:00.908 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '19aaec48', seconds elapsed: 30.7 2024-04-23T18:25:00.909 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.55 2024-04-23T18:25:00.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:25:01.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.027+0000 7fca6c243640 1 -- 172.21.15.50:0/1140779007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca64074c20 msgr2=0x7fca6410e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.027+0000 7fca6c243640 1 --2- 172.21.15.50:0/1140779007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca64074c20 0x7fca6410e2a0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fca54009900 tx=0x7fca5402a480 comp rx=0 tx=0).stop 2024-04-23T18:25:01.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.027+0000 7fca6c243640 1 -- 172.21.15.50:0/1140779007 shutdown_connections 2024-04-23T18:25:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.027+0000 7fca6c243640 1 --2- 172.21.15.50:0/1140779007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca6410e7e0 0x7fca64110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.027+0000 7fca6c243640 1 --2- 172.21.15.50:0/1140779007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca64074c20 0x7fca6410e2a0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.027+0000 7fca6c243640 1 --2- 172.21.15.50:0/1140779007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca64074300 0x7fca640746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.027+0000 7fca6c243640 1 -- 172.21.15.50:0/1140779007 >> 172.21.15.50:0/1140779007 conn(0x7fca64079310 msgr2=0x7fca64079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.027+0000 7fca6c243640 1 -- 172.21.15.50:0/1140779007 shutdown_connections 2024-04-23T18:25:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.027+0000 7fca6c243640 1 -- 172.21.15.50:0/1140779007 wait complete. 2024-04-23T18:25:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca6c243640 1 Processor -- start 2024-04-23T18:25:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca6c243640 1 -- start start 2024-04-23T18:25:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca6c243640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca64074300 0x7fca641a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca6c243640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca64074c20 0x7fca641a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca6c243640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca6410e7e0 0x7fca6419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca6c243640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fca641143d0 con 0x7fca64074300 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca6c243640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca64114250 con 0x7fca6410e7e0 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca6c243640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fca64114550 con 0x7fca64074c20 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca69fdf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca64074300 0x7fca641a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca69fdf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca64074300 0x7fca641a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33938/0 (socket says 172.21.15.50:33938) 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca69fdf640 1 -- 172.21.15.50:0/4277042634 learned_addr learned my addr 172.21.15.50:0/4277042634 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca697de640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca64074c20 0x7fca641a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.028+0000 7fca6a7e0640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca6410e7e0 0x7fca6419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca69fdf640 1 -- 172.21.15.50:0/4277042634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca64074c20 msgr2=0x7fca641a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca69fdf640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca64074c20 0x7fca641a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca69fdf640 1 -- 172.21.15.50:0/4277042634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca6410e7e0 msgr2=0x7fca6419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca69fdf640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca6410e7e0 0x7fca6419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca69fdf640 1 -- 172.21.15.50:0/4277042634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca6419af60 con 0x7fca64074300 2024-04-23T18:25:01.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca6a7e0640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca6410e7e0 0x7fca6419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:01.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca697de640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca64074c20 0x7fca641a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:01.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca69fdf640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca64074300 0x7fca641a0620 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fca6000ea60 tx=0x7fca6000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:01.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca5affd640 1 -- 172.21.15.50:0/4277042634 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6000cde0 con 0x7fca64074300 2024-04-23T18:25:01.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca5affd640 1 -- 172.21.15.50:0/4277042634 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca600035f0 con 0x7fca64074300 2024-04-23T18:25:01.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca5affd640 1 -- 172.21.15.50:0/4277042634 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca60010690 con 0x7fca64074300 2024-04-23T18:25:01.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca6c243640 1 -- 172.21.15.50:0/4277042634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca6419b250 con 0x7fca64074300 2024-04-23T18:25:01.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.029+0000 7fca6c243640 1 -- 172.21.15.50:0/4277042634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca641a7470 con 0x7fca64074300 2024-04-23T18:25:01.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.030+0000 7fca6c243640 1 -- 172.21.15.50:0/4277042634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca640759b0 con 0x7fca64074300 2024-04-23T18:25:01.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.031+0000 7fca5affd640 1 -- 172.21.15.50:0/4277042634 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fca600037a0 con 0x7fca64074300 2024-04-23T18:25:01.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.032+0000 7fca5affd640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca500a3d20 0x7fca500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.032+0000 7fca697de640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca500a3d20 0x7fca500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.032+0000 7fca5affd640 1 -- 172.21.15.50:0/4277042634 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7fca600d0af0 con 0x7fca64074300 2024-04-23T18:25:01.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.032+0000 7fca697de640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca500a3d20 0x7fca500a61e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fca54005ab0 tx=0x7fca54005a20 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:01.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.037+0000 7fca5affd640 1 -- 172.21.15.50:0/4277042634 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fca6009d140 con 0x7fca64074300 2024-04-23T18:25:01.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403424 5.22 MB/sec execute 215 sec latency 30095.225 ms 2024-04-23T18:25:01.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.293+0000 7fca6c243640 1 -- 172.21.15.50:0/4277042634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca640691c0 con 0x7fca64074300 2024-04-23T18:25:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.294+0000 7fca5affd640 1 -- 172.21.15.50:0/4277042634 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11876 (secure 0 0 0) 0x7fca6009ca40 con 0x7fca64074300 2024-04-23T18:25:01.287 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:25:01.287 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:24:49.937177+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25822":{"gid":25822,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/911343542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":911343542},{"type":"v1","addr":"172.21.15.177:6837","nonce":911343542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:25:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 32 2024-04-23T18:25:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.296+0000 7fca6c243640 1 -- 172.21.15.50:0/4277042634 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca500a3d20 msgr2=0x7fca500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.296+0000 7fca6c243640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca500a3d20 0x7fca500a61e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fca54005ab0 tx=0x7fca54005a20 comp rx=0 tx=0).stop 2024-04-23T18:25:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.296+0000 7fca6c243640 1 -- 172.21.15.50:0/4277042634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca64074300 msgr2=0x7fca641a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.296+0000 7fca6c243640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca64074300 0x7fca641a0620 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fca6000ea60 tx=0x7fca6000ef20 comp rx=0 tx=0).stop 2024-04-23T18:25:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.296+0000 7fca6c243640 1 -- 172.21.15.50:0/4277042634 shutdown_connections 2024-04-23T18:25:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.296+0000 7fca6c243640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca500a3d20 0x7fca500a61e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.296+0000 7fca6c243640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca6410e7e0 0x7fca6419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.296+0000 7fca6c243640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca64074c20 0x7fca641a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.296+0000 7fca6c243640 1 --2- 172.21.15.50:0/4277042634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca64074300 0x7fca641a0620 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.296+0000 7fca6c243640 1 -- 172.21.15.50:0/4277042634 >> 172.21.15.50:0/4277042634 conn(0x7fca64079310 msgr2=0x7fca640feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.296+0000 7fca6c243640 1 -- 172.21.15.50:0/4277042634 shutdown_connections 2024-04-23T18:25:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.296+0000 7fca6c243640 1 -- 172.21.15.50:0/4277042634 wait complete. 2024-04-23T18:25:01.301 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 89.55 --await-for 42.0' 2024-04-23T18:25:01.301 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 89.55 --await-for 42.0 2024-04-23T18:25:01.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.436+0000 7f40ce6bb640 1 -- 172.21.15.50:0/2899510519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40c8107260 msgr2=0x7f40c810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.436+0000 7f40ce6bb640 1 --2- 172.21.15.50:0/2899510519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40c8107260 0x7f40c810ee30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009ca0 tx=0x7f40c002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.436+0000 7f40ce6bb640 1 -- 172.21.15.50:0/2899510519 shutdown_connections 2024-04-23T18:25:01.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.436+0000 7f40ce6bb640 1 --2- 172.21.15.50:0/2899510519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40c8107260 0x7f40c810ee30 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.436+0000 7f40ce6bb640 1 --2- 172.21.15.50:0/2899510519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c8102b40 0x7f40c8106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.436+0000 7f40ce6bb640 1 --2- 172.21.15.50:0/2899510519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40c8102190 0x7f40c8102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.436+0000 7f40ce6bb640 1 -- 172.21.15.50:0/2899510519 >> 172.21.15.50:0/2899510519 conn(0x7f40c8079310 msgr2=0x7f40c8079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:01.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.436+0000 7f40ce6bb640 1 -- 172.21.15.50:0/2899510519 shutdown_connections 2024-04-23T18:25:01.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.436+0000 7f40ce6bb640 1 -- 172.21.15.50:0/2899510519 wait complete. 2024-04-23T18:25:01.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.437+0000 7f40ce6bb640 1 Processor -- start 2024-04-23T18:25:01.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.437+0000 7f40ce6bb640 1 -- start start 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.437+0000 7f40ce6bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40c8102190 0x7f40c8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.437+0000 7f40ce6bb640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c8107260 0x7f40c8198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.437+0000 7f40ce6bb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40c819cfd0 0x7f40c819d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.437+0000 7f40ce6bb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f40c81121d0 con 0x7f40c8102190 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.437+0000 7f40ce6bb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f40c8112050 con 0x7f40c819cfd0 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.437+0000 7f40ce6bb640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f40c8112350 con 0x7f40c8107260 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c77fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c8107260 0x7f40c8198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40c8102190 0x7f40c8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40ccc58640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40c819cfd0 0x7f40c819d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c77fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c8107260 0x7f40c8198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:43554/0 (socket says 172.21.15.50:43554) 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c77fe640 1 -- 172.21.15.50:0/3018658425 learned_addr learned my addr 172.21.15.50:0/3018658425 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c77fe640 1 -- 172.21.15.50:0/3018658425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40c819cfd0 msgr2=0x7f40c819d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c77fe640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40c819cfd0 0x7f40c819d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c77fe640 1 -- 172.21.15.50:0/3018658425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40c8102190 msgr2=0x7f40c8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c77fe640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40c8102190 0x7f40c8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c77fe640 1 -- 172.21.15.50:0/3018658425 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c819dac0 con 0x7f40c8107260 2024-04-23T18:25:01.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c7fff640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40c8102190 0x7f40c8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:01.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c77fe640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c8107260 0x7f40c8198c50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00e9f0 tx=0x7f40bc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:01.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c57fa640 1 -- 172.21.15.50:0/3018658425 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc00ccc0 con 0x7f40c8107260 2024-04-23T18:25:01.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40ce6bb640 1 -- 172.21.15.50:0/3018658425 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c819ddb0 con 0x7f40c8107260 2024-04-23T18:25:01.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c57fa640 1 -- 172.21.15.50:0/3018658425 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40bc0054d0 con 0x7f40c8107260 2024-04-23T18:25:01.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.438+0000 7f40c57fa640 1 -- 172.21.15.50:0/3018658425 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc010430 con 0x7f40c8107260 2024-04-23T18:25:01.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.439+0000 7f40ce6bb640 1 -- 172.21.15.50:0/3018658425 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c81a55a0 con 0x7f40c8107260 2024-04-23T18:25:01.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.441+0000 7f40c57fa640 1 -- 172.21.15.50:0/3018658425 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f40bc0046c0 con 0x7f40c8107260 2024-04-23T18:25:01.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.441+0000 7f40d13cc740 1 Processor -- start 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.441+0000 7f40d13cc740 1 -- start start 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40d13cc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9e7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40d13cc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9e79c0 0x555b4a9f1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40d13cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f2090 0x555b4a9f4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40d13cc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x555b4a94c370 con 0x555b4a9e70a0 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40d13cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555b4a8e92e0 con 0x555b4a9f2090 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40d13cc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555b4a948330 con 0x555b4a9e79c0 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40ccc58640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40ccc58640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9e7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33960/0 (socket says 172.21.15.50:33960) 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40ccc58640 1 -- 172.21.15.50:0/4151922359 learned_addr learned my addr 172.21.15.50:0/4151922359 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40c7fff640 1 --2- 172.21.15.50:0/4151922359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9e79c0 0x555b4a9f1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40c77fe640 1 --2- 172.21.15.50:0/4151922359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f2090 0x555b4a9f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40c57fa640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f40b00a3d20 0x7f40b00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40ccc58640 1 -- 172.21.15.50:0/4151922359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9e79c0 msgr2=0x555b4a9f1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40ccc58640 1 --2- 172.21.15.50:0/4151922359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9e79c0 0x555b4a9f1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40ccc58640 1 -- 172.21.15.50:0/4151922359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f2090 msgr2=0x555b4a9f4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40ccc58640 1 --2- 172.21.15.50:0/4151922359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f2090 0x555b4a9f4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40ccc58640 1 -- 172.21.15.50:0/4151922359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b4a72ad80 con 0x555b4a9e70a0 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40c7fff640 1 --2- 172.21.15.50:0/4151922359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9e79c0 0x555b4a9f1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40ccc58640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f40b00a3d20 0x7f40b00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40c57fa640 1 -- 172.21.15.50:0/3018658425 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f40bc0d0b20 con 0x7f40c8107260 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.442+0000 7f40ccc58640 1 --2- 172.21.15.50:0/4151922359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9e7480 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f40c00098d0 tx=0x7f40c0009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b85571d1036674 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.443+0000 7f40ae7fc640 1 -- 172.21.15.50:0/4151922359 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0042040 con 0x555b4a9e70a0 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.443+0000 7f40ccc58640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f40b00a3d20 0x7f40b00a61e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f40c004d000 tx=0x7f40c004e000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.443+0000 7f40ae7fc640 1 -- 172.21.15.50:0/4151922359 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40c0036070 con 0x555b4a9e70a0 2024-04-23T18:25:01.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.443+0000 7f40ae7fc640 1 -- 172.21.15.50:0/4151922359 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0004830 con 0x555b4a9e70a0 2024-04-23T18:25:01.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.444+0000 7f40d13cc740 1 -- 172.21.15.50:0/4151922359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 msgr2=0x555b4a9e7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.444+0000 7f40d13cc740 1 --2- 172.21.15.50:0/4151922359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9e7480 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f40c00098d0 tx=0x7f40c0009900 comp rx=0 tx=0).stop 2024-04-23T18:25:01.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.444+0000 7f40d13cc740 1 -- 172.21.15.50:0/4151922359 shutdown_connections 2024-04-23T18:25:01.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.444+0000 7f40d13cc740 1 --2- 172.21.15.50:0/4151922359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f2090 0x555b4a9f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.444+0000 7f40d13cc740 1 --2- 172.21.15.50:0/4151922359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9e79c0 0x555b4a9f1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.444+0000 7f40d13cc740 1 --2- 172.21.15.50:0/4151922359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9e7480 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.444+0000 7f40d13cc740 1 -- 172.21.15.50:0/4151922359 >> 172.21.15.50:0/4151922359 conn(0x555b4a9e33c0 msgr2=0x555b4a9e37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:01.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.444+0000 7f40d13cc740 1 -- 172.21.15.50:0/4151922359 shutdown_connections 2024-04-23T18:25:01.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.444+0000 7f40d13cc740 1 -- 172.21.15.50:0/4151922359 wait complete. 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.444+0000 7f40d13cc740 1 -- start start 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.444+0000 7f40d13cc740 1 Processor -- start 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40d13cc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9ef060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40d13cc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9e79c0 0x555b4a9ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40d13cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f2090 0x555b4a9efae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40d13cc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x555b4a9f4e80 con 0x555b4a9e70a0 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40d13cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555b4a8e6fb0 con 0x555b4a9f2090 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40d13cc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x555b4a9f5000 con 0x555b4a9e79c0 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40d13cc740 10 client.0 _ll_register_callbacks cb 0x555b4a8e69a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9ef060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9ef060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33964/0 (socket says 172.21.15.50:33964) 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40c7fff640 1 -- 172.21.15.50:0/1623744258 learned_addr learned my addr 172.21.15.50:0/1623744258 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40c77fe640 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9e79c0 0x555b4a9ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40ccc58640 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f2090 0x555b4a9efae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40c7fff640 1 -- 172.21.15.50:0/1623744258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9e79c0 msgr2=0x555b4a9ef5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40c7fff640 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9e79c0 0x555b4a9ef5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40c7fff640 1 -- 172.21.15.50:0/1623744258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f2090 msgr2=0x555b4a9efae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40c7fff640 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f2090 0x555b4a9efae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40c7fff640 1 -- 172.21.15.50:0/1623744258 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b4a78b510 con 0x555b4a9e70a0 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40c7fff640 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9ef060 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f40b8011780 tx=0x7f40b8011b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a01f86306add88a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40a77fe640 1 -- 172.21.15.50:0/1623744258 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8010ba0 con 0x555b4a9e70a0 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40a77fe640 1 -- 172.21.15.50:0/1623744258 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f40b8010d40 con 0x555b4a9e70a0 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.445+0000 7f40d13cc740 1 -- 172.21.15.50:0/1623744258 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40c0036070 con 0x555b4a9e70a0 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.446+0000 7f40a77fe640 1 -- 172.21.15.50:0/1623744258 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f40b8005020 con 0x555b4a9e70a0 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.446+0000 7f40d13cc740 10 client.16314 fetch_fsmap learned FSMap version 32 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.446+0000 7f40d13cc740 1 -- 172.21.15.50:0/1623744258 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555b4a61e670 con 0x555b4a9e70a0 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.446+0000 7f40a77fe640 1 -- 172.21.15.50:0/1623744258 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40c0036070 con 0x555b4a9e70a0 2024-04-23T18:25:01.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.446+0000 7f40a77fe640 1 -- 172.21.15.50:0/1623744258 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f40b801a410 con 0x555b4a9e70a0 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.446+0000 7f40d13cc740 10 client.16314 fetch_fsmap finished waiting for FSMap version 32 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.446+0000 7f40d13cc740 10 client.16314 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.446+0000 7f40d13cc740 20 client.16314 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.446+0000 7f40d13cc740 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x555b4aa02720 0x555b4aa04b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.446+0000 7f40d13cc740 4 client.16314 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.446+0000 7f40d13cc740 1 -- 172.21.15.50:0/1623744258 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b4a959d90 con 0x555b4aa02720 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.446+0000 7f40c77fe640 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x555b4aa02720 0x555b4aa04b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.447+0000 7f40a77fe640 10 client.16314 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.447+0000 7f40c77fe640 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x555b4aa02720 0x555b4aa04b10 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c3f9f1618cd932b server_cookie=e4826fc9ccec2946 in_seq=0 out_seq=0 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.447+0000 7f40a77fe640 1 -- 172.21.15.50:0/1623744258 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b4a959d90 con 0x555b4aa02720 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.447+0000 7f40a77fe640 10 client.16314 handle_command_reply: tid=0 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40d13cc740 1 client.16314 shutdown 2024-04-23T18:25:01.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40d13cc740 1 -- 172.21.15.50:0/1623744258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 msgr2=0x555b4a9ef060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40d13cc740 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9ef060 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f40b8011780 tx=0x7f40b8011b80 comp rx=0 tx=0).stop 2024-04-23T18:25:01.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40d13cc740 1 -- 172.21.15.50:0/1623744258 shutdown_connections 2024-04-23T18:25:01.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40d13cc740 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x555b4aa02720 0x555b4aa04b10 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40d13cc740 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f2090 0x555b4a9efae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40d13cc740 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9e79c0 0x555b4a9ef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40d13cc740 1 --2- 172.21.15.50:0/1623744258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9ef060 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40d13cc740 1 -- 172.21.15.50:0/1623744258 >> 172.21.15.50:0/1623744258 conn(0x555b4a9e33c0 msgr2=0x555b4a9e91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:01.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40a77fe640 0 client.16314 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:01.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40d13cc740 1 -- 172.21.15.50:0/1623744258 shutdown_connections 2024-04-23T18:25:01.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40d13cc740 1 -- 172.21.15.50:0/1623744258 wait complete. 2024-04-23T18:25:01.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.448+0000 7f40d13cc740 20 client.16314 trim_cache size 0 max 16384 2024-04-23T18:25:01.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.466+0000 7f40d13cc740 1 Processor -- start 2024-04-23T18:25:01.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.466+0000 7f40d13cc740 1 -- start start 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.466+0000 7f40d13cc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9ed930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.466+0000 7f40d13cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9e79c0 0x555b4a9fdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.466+0000 7f40d13cc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9f2090 0x555b4a9e9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.466+0000 7f40d13cc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f40c81b0450 con 0x555b4a9e70a0 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.466+0000 7f40d13cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f40c81adcf0 con 0x555b4a9e79c0 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.466+0000 7f40d13cc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f40c81abb40 con 0x555b4a9f2090 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9ed930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40c77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9e79c0 0x555b4a9fdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9ed930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33980/0 (socket says 172.21.15.50:33980) 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40c7fff640 1 -- 172.21.15.50:0/3591391389 learned_addr learned my addr 172.21.15.50:0/3591391389 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40ccc58640 1 --2- 172.21.15.50:0/3591391389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9f2090 0x555b4a9e9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40c77fe640 1 -- 172.21.15.50:0/3591391389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9f2090 msgr2=0x555b4a9e9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40c77fe640 1 --2- 172.21.15.50:0/3591391389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9f2090 0x555b4a9e9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40c77fe640 1 -- 172.21.15.50:0/3591391389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 msgr2=0x555b4a9ed930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40c77fe640 1 --2- 172.21.15.50:0/3591391389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9ed930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40c77fe640 1 -- 172.21.15.50:0/3591391389 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b4a9e66b0 con 0x555b4a9e79c0 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40c7fff640 1 --2- 172.21.15.50:0/3591391389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9ed930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40c77fe640 1 --2- 172.21.15.50:0/3591391389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9e79c0 0x555b4a9fdd40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0ce280 tx=0x7f40bc0bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77f2e0777513cc53 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40adffb640 1 -- 172.21.15.50:0/3591391389 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc005680 con 0x555b4a9e79c0 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40adffb640 1 -- 172.21.15.50:0/3591391389 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40bc0d5040 con 0x555b4a9e79c0 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.467+0000 7f40adffb640 1 -- 172.21.15.50:0/3591391389 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc09add0 con 0x555b4a9e79c0 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.468+0000 7f40d13cc740 1 -- 172.21.15.50:0/3591391389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9e79c0 msgr2=0x555b4a9fdd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.468+0000 7f40d13cc740 1 --2- 172.21.15.50:0/3591391389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9e79c0 0x555b4a9fdd40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0ce280 tx=0x7f40bc0bd040 comp rx=0 tx=0).stop 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.468+0000 7f40d13cc740 1 -- 172.21.15.50:0/3591391389 shutdown_connections 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.468+0000 7f40d13cc740 1 --2- 172.21.15.50:0/3591391389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9f2090 0x555b4a9e9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.468+0000 7f40d13cc740 1 --2- 172.21.15.50:0/3591391389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9e79c0 0x555b4a9fdd40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.468+0000 7f40d13cc740 1 --2- 172.21.15.50:0/3591391389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9ed930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.468+0000 7f40d13cc740 1 -- 172.21.15.50:0/3591391389 >> 172.21.15.50:0/3591391389 conn(0x555b4a9e33c0 msgr2=0x555b4a9f1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:01.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.468+0000 7f40d13cc740 1 -- 172.21.15.50:0/3591391389 shutdown_connections 2024-04-23T18:25:01.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.468+0000 7f40d13cc740 1 -- 172.21.15.50:0/3591391389 wait complete. 2024-04-23T18:25:01.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.468+0000 7f40d13cc740 1 -- start start 2024-04-23T18:25:01.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40d13cc740 1 Processor -- start 2024-04-23T18:25:01.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40d13cc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40d13cc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9f2090 0x555b4a9f6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40d13cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f6a20 0x555b4aa6bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40d13cc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f40c81ac1b0 con 0x555b4a9e70a0 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40d13cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f40c81ad620 con 0x555b4a9f6a20 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40d13cc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f40c81ac880 con 0x555b4a9f2090 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40d13cc740 10 client.0 _ll_register_callbacks cb 0x555b4a8e69a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9f5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33986/0 (socket says 172.21.15.50:33986) 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40c77fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9f2090 0x555b4a9f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40c7fff640 1 -- 172.21.15.50:0/745170277 learned_addr learned my addr 172.21.15.50:0/745170277 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.469+0000 7f40ccc58640 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f6a20 0x555b4aa6bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40c7fff640 1 -- 172.21.15.50:0/745170277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9f2090 msgr2=0x555b4a9f6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40c7fff640 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9f2090 0x555b4a9f6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40c7fff640 1 -- 172.21.15.50:0/745170277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f6a20 msgr2=0x555b4aa6bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:01.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40c7fff640 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f6a20 0x555b4aa6bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40c7fff640 1 -- 172.21.15.50:0/745170277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b4aa6c530 con 0x555b4a9e70a0 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40ccc58640 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f6a20 0x555b4aa6bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40c77fe640 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9f2090 0x555b4a9f6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40c7fff640 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9f5f50 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f40b801a860 tx=0x7f40b801a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cda18b191aea91d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40a77fe640 1 -- 172.21.15.50:0/745170277 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8033060 con 0x555b4a9e70a0 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40a77fe640 1 -- 172.21.15.50:0/745170277 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8498+0+0 (secure 0 0 0) 0x7f40b8010e30 con 0x555b4a9e70a0 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40d13cc740 1 -- 172.21.15.50:0/745170277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40bc0d5040 con 0x555b4a9e70a0 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40a77fe640 1 -- 172.21.15.50:0/745170277 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f40b80051c0 con 0x555b4a9e70a0 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40d13cc740 10 client.16320 fetch_fsmap learned FSMap version 32 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40d13cc740 1 -- 172.21.15.50:0/745170277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555b4aa6d7f0 con 0x555b4a9e70a0 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.470+0000 7f40a77fe640 1 -- 172.21.15.50:0/745170277 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40bc0d5040 con 0x555b4a9e70a0 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.471+0000 7f40a77fe640 1 -- 172.21.15.50:0/745170277 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f40b802c460 con 0x555b4a9e70a0 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.471+0000 7f40d13cc740 10 client.16320 fetch_fsmap finished waiting for FSMap version 32 2024-04-23T18:25:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.471+0000 7f40d13cc740 10 client.16320 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:25:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.471+0000 7f40d13cc740 20 client.16320 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.471+0000 7f40d13cc740 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x555b4aa6dfe0 0x555b4aa70400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.471+0000 7f40d13cc740 4 client.16320 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 89.55, "await_for": 42.0}] 2024-04-23T18:25:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.471+0000 7f40d13cc740 1 -- 172.21.15.50:0/745170277 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 89.55, "await_for": 42.0}) -- 0x555b4a953c00 con 0x555b4aa6dfe0 2024-04-23T18:25:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.471+0000 7f40c77fe640 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x555b4aa6dfe0 0x555b4aa70400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.471+0000 7f40a77fe640 10 client.16320 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.471+0000 7f40c77fe640 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x555b4aa6dfe0 0x555b4aa70400 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe06d0caead41262 server_cookie=cf52955bb0506419 in_seq=0 out_seq=0 2024-04-23T18:25:01.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40a77fe640 1 -- 172.21.15.50:0/745170277 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x555b4a953c00 con 0x555b4aa6dfe0 2024-04-23T18:25:01.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40a77fe640 10 client.16320 handle_command_reply: tid=0 2024-04-23T18:25:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40d13cc740 1 client.16320 shutdown 2024-04-23T18:25:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40d13cc740 1 -- 172.21.15.50:0/745170277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 msgr2=0x555b4a9f5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40d13cc740 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9f5f50 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f40b801a860 tx=0x7f40b801a630 comp rx=0 tx=0).stop 2024-04-23T18:25:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40d13cc740 1 -- 172.21.15.50:0/745170277 shutdown_connections 2024-04-23T18:25:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40d13cc740 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x555b4aa6dfe0 0x555b4aa70400 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40d13cc740 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b4a9f6a20 0x555b4aa6bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40d13cc740 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x555b4a9f2090 0x555b4a9f6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40a77fe640 0 client.16320 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40d13cc740 1 --2- 172.21.15.50:0/745170277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x555b4a9e70a0 0x555b4a9f5f50 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40d13cc740 1 -- 172.21.15.50:0/745170277 >> 172.21.15.50:0/745170277 conn(0x555b4a9e33c0 msgr2=0x555b4a9f1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40d13cc740 1 -- 172.21.15.50:0/745170277 shutdown_connections 2024-04-23T18:25:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40d13cc740 1 -- 172.21.15.50:0/745170277 wait complete. 2024-04-23T18:25:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.523+0000 7f40d13cc740 20 client.16320 trim_cache size 0 max 16384 2024-04-23T18:25:01.519 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:25:01.519 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 32, 2024-04-23T18:25:01.519 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:25:01.519 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 22, 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "dea382bb": { 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 22, 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 89.5, 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:25:01.520 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:25:01.521 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:25:01.521 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:25:01.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:01.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:01.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:01.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:01.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:01.521 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:25:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.524+0000 7f40ce6bb640 1 -- 172.21.15.50:0/3018658425 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f40b00a3d20 msgr2=0x7f40b00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.524+0000 7f40ce6bb640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f40b00a3d20 0x7f40b00a61e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f40c004d000 tx=0x7f40c004e000 comp rx=0 tx=0).stop 2024-04-23T18:25:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.524+0000 7f40ce6bb640 1 -- 172.21.15.50:0/3018658425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c8107260 msgr2=0x7f40c8198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.524+0000 7f40ce6bb640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c8107260 0x7f40c8198c50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00e9f0 tx=0x7f40bc00eeb0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.524+0000 7f40ce6bb640 1 -- 172.21.15.50:0/3018658425 shutdown_connections 2024-04-23T18:25:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.524+0000 7f40ce6bb640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f40b00a3d20 0x7f40b00a61e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.524+0000 7f40ce6bb640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40c819cfd0 0x7f40c819d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.524+0000 7f40ce6bb640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40c8107260 0x7f40c8198c50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.524+0000 7f40ce6bb640 1 --2- 172.21.15.50:0/3018658425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40c8102190 0x7f40c8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:01.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.524+0000 7f40ce6bb640 1 -- 172.21.15.50:0/3018658425 >> 172.21.15.50:0/3018658425 conn(0x7f40c8079310 msgr2=0x7f40c81001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:01.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.524+0000 7f40ce6bb640 1 -- 172.21.15.50:0/3018658425 shutdown_connections 2024-04-23T18:25:01.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:01.524+0000 7f40ce6bb640 1 -- 172.21.15.50:0/3018658425 wait complete. 2024-04-23T18:25:01.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:01 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/944496197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:01.528 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'dea382bb', quiesce took 0.6 seconds. Will release after: 29.55 2024-04-23T18:25:01.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:01 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/944496197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:01.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:01 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/944496197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:02.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 5.22 MB/sec execute 216 sec latency 694.906 ms 2024-04-23T18:25:02.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:02 smithi177 ceph-mon[32308]: pgmap v200: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-04-23T18:25:02.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:02 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4277042634' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:02.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:02 smithi188 ceph-mon[32694]: pgmap v200: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-04-23T18:25:02.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:02 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4277042634' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:02.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:02 smithi050 ceph-mon[29187]: pgmap v200: 97 pgs: 97 active+clean; 120 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 1.4 MiB/s wr, 61 op/s 2024-04-23T18:25:02.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:02 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4277042634' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:03.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 5.20 MB/sec execute 217 sec latency 1694.952 ms 2024-04-23T18:25:04.212 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 5.17 MB/sec execute 218 sec latency 2695.094 ms 2024-04-23T18:25:04.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:04 smithi177 ceph-mon[32308]: pgmap v201: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 2.1 MiB/s wr, 119 op/s 2024-04-23T18:25:04.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:04 smithi188 ceph-mon[32694]: pgmap v201: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 2.1 MiB/s wr, 119 op/s 2024-04-23T18:25:04.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:04 smithi050 ceph-mon[29187]: pgmap v201: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 2.1 MiB/s wr, 119 op/s 2024-04-23T18:25:05.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 5.15 MB/sec execute 219 sec latency 3695.196 ms 2024-04-23T18:25:05.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:05 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:05.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:05 smithi177 ceph-mon[32308]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T18:25:05.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:05 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:05.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:05 smithi188 ceph-mon[32694]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T18:25:05.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:05 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:25:05.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:05 smithi050 ceph-mon[29187]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T18:25:06.211 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 5.13 MB/sec execute 220 sec latency 4695.319 ms 2024-04-23T18:25:06.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:06 smithi177 ceph-mon[32308]: pgmap v203: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.8 MiB/s wr, 109 op/s 2024-04-23T18:25:06.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:06 smithi188 ceph-mon[32694]: pgmap v203: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.8 MiB/s wr, 109 op/s 2024-04-23T18:25:06.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:06 smithi050 ceph-mon[29187]: pgmap v203: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.8 MiB/s wr, 109 op/s 2024-04-23T18:25:07.212 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 5.10 MB/sec execute 221 sec latency 5695.459 ms 2024-04-23T18:25:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.083+0000 7fc80b78e640 1 -- 172.21.15.50:0/920676769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc804101aa0 msgr2=0x7fc804105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.083+0000 7fc80b78e640 1 --2- 172.21.15.50:0/920676769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc804101aa0 0x7fc804105a90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4009900 tx=0x7fc7f402a410 comp rx=0 tx=0).stop 2024-04-23T18:25:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.083+0000 7fc80b78e640 1 -- 172.21.15.50:0/920676769 shutdown_connections 2024-04-23T18:25:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.083+0000 7fc80b78e640 1 --2- 172.21.15.50:0/920676769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc804106170 0x7fc80410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.083+0000 7fc80b78e640 1 --2- 172.21.15.50:0/920676769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc804101aa0 0x7fc804105a90 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.083+0000 7fc80b78e640 1 --2- 172.21.15.50:0/920676769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc8041010f0 0x7fc8041014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.083+0000 7fc80b78e640 1 -- 172.21.15.50:0/920676769 >> 172.21.15.50:0/920676769 conn(0x7fc804077fd0 msgr2=0x7fc8040783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.083+0000 7fc80b78e640 1 -- 172.21.15.50:0/920676769 shutdown_connections 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.083+0000 7fc80b78e640 1 -- 172.21.15.50:0/920676769 wait complete. 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc80b78e640 1 Processor -- start 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc80b78e640 1 -- start start 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc80b78e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8041010f0 0x7fc8041986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc80b78e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc804106170 0x7fc804198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc80b78e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc80419cf80 0x7fc80419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc80b78e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc8041110b0 con 0x7fc804106170 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc80b78e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc804110f30 con 0x7fc8041010f0 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc80b78e640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc804111230 con 0x7fc80419cf80 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc808d29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc804106170 0x7fc804198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc808d29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc804106170 0x7fc804198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44258/0 (socket says 172.21.15.50:44258) 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc808d29640 1 -- 172.21.15.50:0/2456925847 learned_addr learned my addr 172.21.15.50:0/2456925847 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc80952a640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8041010f0 0x7fc8041986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc809d2b640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc80419cf80 0x7fc80419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc808d29640 1 -- 172.21.15.50:0/2456925847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc80419cf80 msgr2=0x7fc80419d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc808d29640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc80419cf80 0x7fc80419d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc808d29640 1 -- 172.21.15.50:0/2456925847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8041010f0 msgr2=0x7fc8041986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc808d29640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8041010f0 0x7fc8041986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.084+0000 7fc808d29640 1 -- 172.21.15.50:0/2456925847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80419da70 con 0x7fc804106170 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.085+0000 7fc808d29640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc804106170 0x7fc804198c00 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4032fd0 tx=0x7fc7f4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.085+0000 7fc7fa7fc640 1 -- 172.21.15.50:0/2456925847 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f403f030 con 0x7fc804106170 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.085+0000 7fc80b78e640 1 -- 172.21.15.50:0/2456925847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80419dd00 con 0x7fc804106170 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.085+0000 7fc7fa7fc640 1 -- 172.21.15.50:0/2456925847 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f4004490 con 0x7fc804106170 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.085+0000 7fc7fa7fc640 1 -- 172.21.15.50:0/2456925847 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4035070 con 0x7fc804106170 2024-04-23T18:25:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.085+0000 7fc80b78e640 1 -- 172.21.15.50:0/2456925847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8041a54c0 con 0x7fc804106170 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc7fa7fc640 1 -- 172.21.15.50:0/2456925847 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc7f4042030 con 0x7fc804106170 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc80e580740 1 Processor -- start 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc80e580740 1 -- start start 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc80e580740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2ba620 0x56540e2baa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc80e580740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2c52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc80e580740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2c5800 0x56540e2c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc80e580740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56540e21f970 con 0x56540e2baf40 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc80e580740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56540e1bc4a0 con 0x56540e2c5800 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc80e580740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56540e21b930 con 0x56540e2ba620 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc80952a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2c52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc809d2b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2ba620 0x56540e2baa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc80952a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2c52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44272/0 (socket says 172.21.15.50:44272) 2024-04-23T18:25:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.088+0000 7fc80952a640 1 -- 172.21.15.50:0/1087774162 learned_addr learned my addr 172.21.15.50:0/1087774162 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc808d29640 1 --2- 172.21.15.50:0/1087774162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2c5800 0x56540e2c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc7fa7fc640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc7f00a3dc0 0x7fc7f00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc80952a640 1 -- 172.21.15.50:0/1087774162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2ba620 msgr2=0x56540e2baa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc80952a640 1 --2- 172.21.15.50:0/1087774162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2ba620 0x56540e2baa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc80952a640 1 -- 172.21.15.50:0/1087774162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2c5800 msgr2=0x56540e2c7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc80952a640 1 --2- 172.21.15.50:0/1087774162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2c5800 0x56540e2c7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc80952a640 1 -- 172.21.15.50:0/1087774162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56540df32210 con 0x56540e2baf40 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc808d29640 1 --2- 172.21.15.50:0/1087774162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2c5800 0x56540e2c7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc809d2b640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc7f00a3dc0 0x7fc7f00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc7fa7fc640 1 -- 172.21.15.50:0/2456925847 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8586+0+0 (secure 0 0 0) 0x7fc7f40f0910 con 0x7fc804106170 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc80952a640 1 --2- 172.21.15.50:0/1087774162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2c52c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fc80000b760 tx=0x7fc80000bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1617949df3de27fc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc7ef7fe640 1 -- 172.21.15.50:0/1087774162 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800003dd0 con 0x56540e2baf40 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc7ef7fe640 1 -- 172.21.15.50:0/1087774162 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc800015070 con 0x56540e2baf40 2024-04-23T18:25:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.089+0000 7fc7ef7fe640 1 -- 172.21.15.50:0/1087774162 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800004d90 con 0x56540e2baf40 2024-04-23T18:25:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.090+0000 7fc80e580740 1 -- 172.21.15.50:0/1087774162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 msgr2=0x56540e2c52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.090+0000 7fc809d2b640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc7f00a3dc0 0x7fc7f00a6280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00aed0 tx=0x7fc7fc004a40 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.090+0000 7fc80e580740 1 --2- 172.21.15.50:0/1087774162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2c52c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fc80000b760 tx=0x7fc80000bc20 comp rx=0 tx=0).stop 2024-04-23T18:25:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.090+0000 7fc80e580740 1 -- 172.21.15.50:0/1087774162 shutdown_connections 2024-04-23T18:25:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.090+0000 7fc80e580740 1 --2- 172.21.15.50:0/1087774162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2c5800 0x56540e2c7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.090+0000 7fc80e580740 1 --2- 172.21.15.50:0/1087774162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2c52c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.090+0000 7fc80e580740 1 --2- 172.21.15.50:0/1087774162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2ba620 0x56540e2baa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.090+0000 7fc80e580740 1 -- 172.21.15.50:0/1087774162 >> 172.21.15.50:0/1087774162 conn(0x56540e2b6b30 msgr2=0x56540e2b6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.090+0000 7fc80e580740 1 -- 172.21.15.50:0/1087774162 shutdown_connections 2024-04-23T18:25:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.090+0000 7fc80e580740 1 -- 172.21.15.50:0/1087774162 wait complete. 2024-04-23T18:25:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.090+0000 7fc80e580740 1 -- start start 2024-04-23T18:25:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.090+0000 7fc80e580740 1 Processor -- start 2024-04-23T18:25:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc80e580740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2c5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc80e580740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2baf40 0x56540e2c2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc80e580740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2c5800 0x56540e2c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc80e580740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56540e2c8430 con 0x56540e2c5800 2024-04-23T18:25:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc80e580740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56540e1ba0b0 con 0x56540e2ba620 2024-04-23T18:25:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc80e580740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56540e2c85b0 con 0x56540e2baf40 2024-04-23T18:25:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc80e580740 10 client.0 _ll_register_callbacks cb 0x56540e1a7d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc809d2b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2c5800 0x56540e2c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc809d2b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2c5800 0x56540e2c29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44278/0 (socket says 172.21.15.50:44278) 2024-04-23T18:25:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc809d2b640 1 -- 172.21.15.50:0/1331032576 learned_addr learned my addr 172.21.15.50:0/1331032576 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc80952a640 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc808d29640 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2baf40 0x56540e2c2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc809d2b640 1 -- 172.21.15.50:0/1331032576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2baf40 msgr2=0x56540e2c2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc809d2b640 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2baf40 0x56540e2c2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc809d2b640 1 -- 172.21.15.50:0/1331032576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 msgr2=0x56540e2c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc809d2b640 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc809d2b640 1 -- 172.21.15.50:0/1331032576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56540dffe620 con 0x56540e2c5800 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.091+0000 7fc808d29640 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2baf40 0x56540e2c2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.092+0000 7fc809d2b640 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2c5800 0x56540e2c29a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc012420 tx=0x7fc7fc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3215f95bb9673f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.092+0000 7fc7ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.092+0000 7fc7ecff9640 1 -- 172.21.15.50:0/1331032576 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc022070 con 0x56540e2c5800 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.092+0000 7fc7ecff9640 1 -- 172.21.15.50:0/1331032576 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8586+0+0 (secure 0 0 0) 0x7fc7fc00fbf0 con 0x56540e2c5800 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.092+0000 7fc80e580740 1 -- 172.21.15.50:0/1331032576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc800015070 con 0x56540e2c5800 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.092+0000 7fc7ecff9640 1 -- 172.21.15.50:0/1331032576 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc7fc00fdf0 con 0x56540e2c5800 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.092+0000 7fc80e580740 10 client.16332 fetch_fsmap learned FSMap version 33 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.092+0000 7fc80e580740 1 -- 172.21.15.50:0/1331032576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56540e2d43c0 con 0x56540e2c5800 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.092+0000 7fc7ecff9640 1 -- 172.21.15.50:0/1331032576 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc800015070 con 0x56540e2c5800 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.092+0000 7fc7ecff9640 1 -- 172.21.15.50:0/1331032576 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fc7fc01b040 con 0x56540e2c5800 2024-04-23T18:25:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.092+0000 7fc80e580740 10 client.16332 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.093+0000 7fc80e580740 10 client.16332 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.093+0000 7fc80e580740 20 client.16332 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.093+0000 7fc80e580740 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56540e2d5880 0x56540e2d7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.093+0000 7fc80e580740 4 client.16332 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.093+0000 7fc80e580740 1 -- 172.21.15.50:0/1331032576 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56540e22d390 con 0x56540e2d5880 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.093+0000 7fc80952a640 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56540e2d5880 0x56540e2d7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.093+0000 7fc80952a640 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56540e2d5880 0x56540e2d7cd0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed886c2cf8ce11cb server_cookie=8e0064d9bfbad092 in_seq=0 out_seq=0 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.093+0000 7fc7ecff9640 10 client.16332 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc7ecff9640 1 -- 172.21.15.50:0/1331032576 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56540e22d390 con 0x56540e2d5880 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc7ecff9640 10 client.16332 handle_command_reply: tid=0 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc80e580740 1 client.16332 shutdown 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc80e580740 1 -- 172.21.15.50:0/1331032576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2c5800 msgr2=0x56540e2c29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc80e580740 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2c5800 0x56540e2c29a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc012420 tx=0x7fc7fc004ab0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc80e580740 1 -- 172.21.15.50:0/1331032576 shutdown_connections 2024-04-23T18:25:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc80e580740 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56540e2d5880 0x56540e2d7cd0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc80e580740 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2c5800 0x56540e2c29a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc80e580740 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2baf40 0x56540e2c2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc7ecff9640 0 client.16332 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc80e580740 1 --2- 172.21.15.50:0/1331032576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc80e580740 1 -- 172.21.15.50:0/1331032576 >> 172.21.15.50:0/1331032576 conn(0x56540e2b6b30 msgr2=0x56540e2bc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc80e580740 1 -- 172.21.15.50:0/1331032576 shutdown_connections 2024-04-23T18:25:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc80e580740 1 -- 172.21.15.50:0/1331032576 wait complete. 2024-04-23T18:25:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.094+0000 7fc80e580740 20 client.16332 trim_cache size 0 max 16384 2024-04-23T18:25:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc80e580740 1 Processor -- start 2024-04-23T18:25:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc80e580740 1 -- start start 2024-04-23T18:25:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc80e580740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc80e580740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc80e580740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2c5800 0x56540e2dd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc80e580740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc8041b0480 con 0x56540e2baf40 2024-04-23T18:25:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc80e580740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc8041add50 con 0x56540e2ba620 2024-04-23T18:25:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc80e580740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc8041abbc0 con 0x56540e2c5800 2024-04-23T18:25:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc808d29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc808d29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2b5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44284/0 (socket says 172.21.15.50:44284) 2024-04-23T18:25:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc808d29640 1 -- 172.21.15.50:0/228675270 learned_addr learned my addr 172.21.15.50:0/228675270 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc80952a640 1 --2- 172.21.15.50:0/228675270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc809d2b640 1 --2- 172.21.15.50:0/228675270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2c5800 0x56540e2dd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc808d29640 1 -- 172.21.15.50:0/228675270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2c5800 msgr2=0x56540e2dd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc808d29640 1 --2- 172.21.15.50:0/228675270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2c5800 0x56540e2dd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc808d29640 1 -- 172.21.15.50:0/228675270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 msgr2=0x56540e2d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc808d29640 1 --2- 172.21.15.50:0/228675270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2d12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.110+0000 7fc808d29640 1 -- 172.21.15.50:0/228675270 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56540e2b67f0 con 0x56540e2baf40 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc808d29640 1 --2- 172.21.15.50:0/228675270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2b5b30 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4031440 tx=0x7fc7f402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ce2747e5b131968 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc7eeffd640 1 -- 172.21.15.50:0/228675270 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4049040 con 0x56540e2baf40 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc7eeffd640 1 -- 172.21.15.50:0/228675270 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f40c0c20 con 0x56540e2baf40 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc7eeffd640 1 -- 172.21.15.50:0/228675270 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f40a5e60 con 0x56540e2baf40 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc80e580740 1 -- 172.21.15.50:0/228675270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 msgr2=0x56540e2b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc80e580740 1 --2- 172.21.15.50:0/228675270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2b5b30 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4031440 tx=0x7fc7f402a410 comp rx=0 tx=0).stop 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc80e580740 1 -- 172.21.15.50:0/228675270 shutdown_connections 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc80e580740 1 --2- 172.21.15.50:0/228675270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2c5800 0x56540e2dd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc80e580740 1 --2- 172.21.15.50:0/228675270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2b5b30 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc80e580740 1 --2- 172.21.15.50:0/228675270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc80e580740 1 -- 172.21.15.50:0/228675270 >> 172.21.15.50:0/228675270 conn(0x56540e2b6b30 msgr2=0x56540e2b5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc80e580740 1 -- 172.21.15.50:0/228675270 shutdown_connections 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.111+0000 7fc80e580740 1 -- 172.21.15.50:0/228675270 wait complete. 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.112+0000 7fc80e580740 1 -- start start 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.112+0000 7fc80e580740 1 Processor -- start 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.112+0000 7fc80e580740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.112+0000 7fc80e580740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2cb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.112+0000 7fc80e580740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2c5800 0x56540e2cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.112+0000 7fc80e580740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc8041ac210 con 0x56540e2baf40 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.112+0000 7fc80e580740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc8041ad680 con 0x56540e2ba620 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.112+0000 7fc80e580740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc8041ac8e0 con 0x56540e2c5800 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.112+0000 7fc80e580740 10 client.0 _ll_register_callbacks cb 0x7fc7fc0107a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.112+0000 7fc808d29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.112+0000 7fc808d29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2cb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44296/0 (socket says 172.21.15.50:44296) 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.112+0000 7fc808d29640 1 -- 172.21.15.50:0/3577232039 learned_addr learned my addr 172.21.15.50:0/3577232039 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc809d2b640 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2c5800 0x56540e2cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc80952a640 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc808d29640 1 -- 172.21.15.50:0/3577232039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2c5800 msgr2=0x56540e2cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc808d29640 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2c5800 0x56540e2cb7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc808d29640 1 -- 172.21.15.50:0/3577232039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 msgr2=0x56540e2cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc808d29640 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc808d29640 1 -- 172.21.15.50:0/3577232039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56540e33ae60 con 0x56540e2baf40 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc809d2b640 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2c5800 0x56540e2cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc80952a640 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc808d29640 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2cb100 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4005c40 tx=0x7fc7f40c54e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def0876606d40c4d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc7ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc7ecff9640 1 -- 172.21.15.50:0/3577232039 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4049070 con 0x56540e2baf40 2024-04-23T18:25:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc7ecff9640 1 -- 172.21.15.50:0/3577232039 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8586+0+0 (secure 0 0 0) 0x7fc7f40ee2a0 con 0x56540e2baf40 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc80e580740 1 -- 172.21.15.50:0/3577232039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7f40c0c20 con 0x56540e2baf40 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc7ecff9640 1 -- 172.21.15.50:0/3577232039 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc7f40a5e60 con 0x56540e2baf40 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc80e580740 10 client.16338 fetch_fsmap learned FSMap version 33 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc80e580740 1 -- 172.21.15.50:0/3577232039 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56540e33c010 con 0x56540e2baf40 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc7ecff9640 1 -- 172.21.15.50:0/3577232039 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f40c0c20 con 0x56540e2baf40 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.113+0000 7fc7ecff9640 1 -- 172.21.15.50:0/3577232039 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fc7f40a7140 con 0x56540e2baf40 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.114+0000 7fc80e580740 10 client.16338 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.114+0000 7fc80e580740 10 client.16338 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.114+0000 7fc80e580740 20 client.16338 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.114+0000 7fc80e580740 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56540e33c6e0 0x56540e33eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.114+0000 7fc80e580740 4 client.16338 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.114+0000 7fc80e580740 1 -- 172.21.15.50:0/3577232039 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56540e227200 con 0x56540e33c6e0 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.114+0000 7fc80952a640 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56540e33c6e0 0x56540e33eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.114+0000 7fc80952a640 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56540e33c6e0 0x56540e33eb00 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e4883157c0a977d server_cookie=f546dfa6aaf2e901 in_seq=0 out_seq=0 2024-04-23T18:25:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.114+0000 7fc7ecff9640 10 client.16338 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc7ecff9640 1 -- 172.21.15.50:0/3577232039 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56540e227200 con 0x56540e33c6e0 2024-04-23T18:25:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc7ecff9640 10 client.16338 handle_command_reply: tid=0 2024-04-23T18:25:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc80e580740 1 client.16338 shutdown 2024-04-23T18:25:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc80e580740 1 -- 172.21.15.50:0/3577232039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 msgr2=0x56540e2cb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc80e580740 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2cb100 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4005c40 tx=0x7fc7f40c54e0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc80e580740 1 -- 172.21.15.50:0/3577232039 shutdown_connections 2024-04-23T18:25:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc80e580740 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56540e33c6e0 0x56540e33eb00 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc80e580740 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56540e2c5800 0x56540e2cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc80e580740 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56540e2baf40 0x56540e2cb100 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc7ecff9640 0 client.16338 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc80e580740 1 --2- 172.21.15.50:0/3577232039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56540e2ba620 0x56540e2cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc80e580740 1 -- 172.21.15.50:0/3577232039 >> 172.21.15.50:0/3577232039 conn(0x56540e2b6b30 msgr2=0x56540e2b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc80e580740 1 -- 172.21.15.50:0/3577232039 shutdown_connections 2024-04-23T18:25:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc80e580740 1 -- 172.21.15.50:0/3577232039 wait complete. 2024-04-23T18:25:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.145+0000 7fc80e580740 20 client.16338 trim_cache size 0 max 16384 2024-04-23T18:25:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:25:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:25:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:25:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:25:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.146+0000 7fc80b78e640 1 -- 172.21.15.50:0/2456925847 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc7f00a3dc0 msgr2=0x7fc7f00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.146+0000 7fc80b78e640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc7f00a3dc0 0x7fc7f00a6280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00aed0 tx=0x7fc7fc004a40 comp rx=0 tx=0).stop 2024-04-23T18:25:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.146+0000 7fc80b78e640 1 -- 172.21.15.50:0/2456925847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc804106170 msgr2=0x7fc804198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.146+0000 7fc80b78e640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc804106170 0x7fc804198c00 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4032fd0 tx=0x7fc7f4004270 comp rx=0 tx=0).stop 2024-04-23T18:25:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.146+0000 7fc80b78e640 1 -- 172.21.15.50:0/2456925847 shutdown_connections 2024-04-23T18:25:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.146+0000 7fc80b78e640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc7f00a3dc0 0x7fc7f00a6280 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.146+0000 7fc80b78e640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc80419cf80 0x7fc80419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.146+0000 7fc80b78e640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc804106170 0x7fc804198c00 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.146+0000 7fc80b78e640 1 --2- 172.21.15.50:0/2456925847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8041010f0 0x7fc8041986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.146+0000 7fc80b78e640 1 -- 172.21.15.50:0/2456925847 >> 172.21.15.50:0/2456925847 conn(0x7fc804077fd0 msgr2=0x7fc804069a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.146+0000 7fc80b78e640 1 -- 172.21.15.50:0/2456925847 shutdown_connections 2024-04-23T18:25:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:08.146+0000 7fc80b78e640 1 -- 172.21.15.50:0/2456925847 wait complete. 2024-04-23T18:25:08.212 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 5.08 MB/sec execute 222 sec latency 6695.572 ms 2024-04-23T18:25:08.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:08 smithi177 ceph-mon[32308]: pgmap v204: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-23T18:25:08.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:08 smithi188 ceph-mon[32694]: pgmap v204: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-23T18:25:08.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:08 smithi050 ceph-mon[29187]: pgmap v204: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-23T18:25:09.212 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 5.06 MB/sec execute 223 sec latency 7695.710 ms 2024-04-23T18:25:10.212 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 5.03 MB/sec execute 224 sec latency 8695.773 ms 2024-04-23T18:25:10.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:10 smithi177 ceph-mon[32308]: pgmap v205: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-23T18:25:10.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:10 smithi188 ceph-mon[32694]: pgmap v205: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-23T18:25:10.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:10 smithi050 ceph-mon[29187]: pgmap v205: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-23T18:25:11.212 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 5.01 MB/sec execute 225 sec latency 9695.882 ms 2024-04-23T18:25:12.212 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.99 MB/sec execute 226 sec latency 10696.022 ms 2024-04-23T18:25:12.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:12 smithi188 ceph-mon[32694]: pgmap v206: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-23T18:25:12.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:12 smithi188 ceph-mon[32694]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T18:25:12.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:12 smithi050 ceph-mon[29187]: pgmap v206: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-23T18:25:12.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:12 smithi050 ceph-mon[29187]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T18:25:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:12 smithi177 ceph-mon[32308]: pgmap v206: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-23T18:25:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:12 smithi177 ceph-mon[32308]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T18:25:13.212 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.97 MB/sec execute 227 sec latency 11696.082 ms 2024-04-23T18:25:14.212 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.95 MB/sec execute 228 sec latency 12696.202 ms 2024-04-23T18:25:14.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:14 smithi188 ceph-mon[32694]: pgmap v208: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 685 KiB/s wr, 45 op/s 2024-04-23T18:25:14.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:14 smithi050 ceph-mon[29187]: pgmap v208: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 685 KiB/s wr, 45 op/s 2024-04-23T18:25:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:14 smithi177 ceph-mon[32308]: pgmap v208: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 685 KiB/s wr, 45 op/s 2024-04-23T18:25:15.212 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.92 MB/sec execute 229 sec latency 13696.320 ms 2024-04-23T18:25:16.213 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.90 MB/sec execute 230 sec latency 14696.445 ms 2024-04-23T18:25:16.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:16 smithi188 ceph-mon[32694]: pgmap v209: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 575 KiB/s wr, 38 op/s 2024-04-23T18:25:16.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:16 smithi050 ceph-mon[29187]: pgmap v209: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 575 KiB/s wr, 38 op/s 2024-04-23T18:25:16.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:16 smithi177 ceph-mon[32308]: pgmap v209: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 575 KiB/s wr, 38 op/s 2024-04-23T18:25:17.213 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.88 MB/sec execute 231 sec latency 15696.572 ms 2024-04-23T18:25:18.213 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.86 MB/sec execute 232 sec latency 16696.666 ms 2024-04-23T18:25:18.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:18 smithi188 ceph-mon[32694]: pgmap v210: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 133 KiB/s wr, 19 op/s 2024-04-23T18:25:18.338 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:18 smithi177 ceph-mon[32308]: pgmap v210: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 133 KiB/s wr, 19 op/s 2024-04-23T18:25:18.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:18 smithi050 ceph-mon[29187]: pgmap v210: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 133 KiB/s wr, 19 op/s 2024-04-23T18:25:19.213 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.84 MB/sec execute 233 sec latency 17696.757 ms 2024-04-23T18:25:20.213 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.82 MB/sec execute 234 sec latency 18696.888 ms 2024-04-23T18:25:20.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:20 smithi188 ceph-mon[32694]: pgmap v211: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 133 KiB/s wr, 19 op/s 2024-04-23T18:25:20.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:20 smithi050 ceph-mon[29187]: pgmap v211: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 133 KiB/s wr, 19 op/s 2024-04-23T18:25:20.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:20 smithi177 ceph-mon[32308]: pgmap v211: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 133 KiB/s wr, 19 op/s 2024-04-23T18:25:21.213 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.80 MB/sec execute 235 sec latency 19696.961 ms 2024-04-23T18:25:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.245+0000 7fe90a1cd640 1 -- 172.21.15.50:0/1034084297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe904103b00 msgr2=0x7fe90410bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.245+0000 7fe90a1cd640 1 --2- 172.21.15.50:0/1034084297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe904103b00 0x7fe90410bfe0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8009900 tx=0x7fe8f802a510 comp rx=0 tx=0).stop 2024-04-23T18:25:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.245+0000 7fe90a1cd640 1 -- 172.21.15.50:0/1034084297 shutdown_connections 2024-04-23T18:25:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.245+0000 7fe90a1cd640 1 --2- 172.21.15.50:0/1034084297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe90410c520 0x7fe90410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.245+0000 7fe90a1cd640 1 --2- 172.21.15.50:0/1034084297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe904103b00 0x7fe90410bfe0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.245+0000 7fe90a1cd640 1 --2- 172.21.15.50:0/1034084297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9041031e0 0x7fe9041035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.245+0000 7fe90a1cd640 1 -- 172.21.15.50:0/1034084297 >> 172.21.15.50:0/1034084297 conn(0x7fe904077fd0 msgr2=0x7fe9040783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.245+0000 7fe90a1cd640 1 -- 172.21.15.50:0/1034084297 shutdown_connections 2024-04-23T18:25:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.245+0000 7fe90a1cd640 1 -- 172.21.15.50:0/1034084297 wait complete. 2024-04-23T18:25:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.246+0000 7fe90a1cd640 1 Processor -- start 2024-04-23T18:25:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.246+0000 7fe90a1cd640 1 -- start start 2024-04-23T18:25:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.246+0000 7fe90a1cd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9041031e0 0x7fe904100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.246+0000 7fe90a1cd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe904103b00 0x7fe904101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.246+0000 7fe90a1cd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe90410c520 0x7fe9041018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.246+0000 7fe90a1cd640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe904112190 con 0x7fe9041031e0 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.246+0000 7fe90a1cd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe904112010 con 0x7fe90410c520 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.246+0000 7fe90a1cd640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe904112310 con 0x7fe904103b00 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.246+0000 7fe9037fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9041031e0 0x7fe904100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.246+0000 7fe9037fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9041031e0 0x7fe904100e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60818/0 (socket says 172.21.15.50:60818) 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe9037fe640 1 -- 172.21.15.50:0/2534293656 learned_addr learned my addr 172.21.15.50:0/2534293656 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe903fff640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe90410c520 0x7fe9041018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe902ffd640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe904103b00 0x7fe904101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe9037fe640 1 -- 172.21.15.50:0/2534293656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe904103b00 msgr2=0x7fe904101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe9037fe640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe904103b00 0x7fe904101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe9037fe640 1 -- 172.21.15.50:0/2534293656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe90410c520 msgr2=0x7fe9041018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe9037fe640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe90410c520 0x7fe9041018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe9037fe640 1 -- 172.21.15.50:0/2534293656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9041a6bf0 con 0x7fe9041031e0 2024-04-23T18:25:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe902ffd640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe904103b00 0x7fe904101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe9037fe640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9041031e0 0x7fe904100e30 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400d950 tx=0x7fe8f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe900ff9640 1 -- 172.21.15.50:0/2534293656 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4017070 con 0x7fe9041031e0 2024-04-23T18:25:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe900ff9640 1 -- 172.21.15.50:0/2534293656 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8f4009c30 con 0x7fe9041031e0 2024-04-23T18:25:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe90a1cd640 1 -- 172.21.15.50:0/2534293656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9041a6ee0 con 0x7fe9041031e0 2024-04-23T18:25:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe900ff9640 1 -- 172.21.15.50:0/2534293656 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f400f620 con 0x7fe9041031e0 2024-04-23T18:25:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.247+0000 7fe90a1cd640 1 -- 172.21.15.50:0/2534293656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9041a7350 con 0x7fe9041031e0 2024-04-23T18:25:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.250+0000 7fe900ff9640 1 -- 172.21.15.50:0/2534293656 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe8f4010460 con 0x7fe9041031e0 2024-04-23T18:25:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.250+0000 7fe90cfc0740 1 Processor -- start 2024-04-23T18:25:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.250+0000 7fe90cfc0740 1 -- start start 2024-04-23T18:25:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.250+0000 7fe90cfc0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d5ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.250+0000 7fe90cfc0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 0x560997d662c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.250+0000 7fe90cfc0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d66800 0x560997d68bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.250+0000 7fe90cfc0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560997cc0970 con 0x560997d5bf40 2024-04-23T18:25:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.250+0000 7fe90cfc0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560997c5d4a0 con 0x560997d66800 2024-04-23T18:25:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.250+0000 7fe90cfc0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560997cbc930 con 0x560997d5b620 2024-04-23T18:25:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.250+0000 7fe900ff9640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe8ec0a3dc0 0x7fe8ec0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe9037fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 0x560997d662c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe9037fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 0x560997d662c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60828/0 (socket says 172.21.15.50:60828) 2024-04-23T18:25:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe9037fe640 1 -- 172.21.15.50:0/425763247 learned_addr learned my addr 172.21.15.50:0/425763247 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe903fff640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe8ec0a3dc0 0x7fe8ec0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe903fff640 1 --2- 172.21.15.50:0/425763247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d5ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe902ffd640 1 --2- 172.21.15.50:0/425763247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d66800 0x560997d68bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe900ff9640 1 -- 172.21.15.50:0/2534293656 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fe8f40d0430 con 0x7fe9041031e0 2024-04-23T18:25:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe9037fe640 1 -- 172.21.15.50:0/425763247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 msgr2=0x560997d5ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe9037fe640 1 --2- 172.21.15.50:0/425763247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d5ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe9037fe640 1 -- 172.21.15.50:0/425763247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d66800 msgr2=0x560997d68bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe9037fe640 1 --2- 172.21.15.50:0/425763247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d66800 0x560997d68bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe9037fe640 1 -- 172.21.15.50:0/425763247 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609979d3210 con 0x560997d5bf40 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe903fff640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe8ec0a3dc0 0x7fe8ec0a6280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009fd0 tx=0x7fe8fc009d60 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe9037fe640 1 --2- 172.21.15.50:0/425763247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 0x560997d662c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe8f40cdae0 tx=0x7fe8f409c2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbef35595b759a0c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe8e9ffb640 1 -- 172.21.15.50:0/425763247 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4020030 con 0x560997d5bf40 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe8e9ffb640 1 -- 172.21.15.50:0/425763247 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8f409c3c0 con 0x560997d5bf40 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.251+0000 7fe8e9ffb640 1 -- 172.21.15.50:0/425763247 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f40b0e60 con 0x560997d5bf40 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.252+0000 7fe90cfc0740 1 -- 172.21.15.50:0/425763247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 msgr2=0x560997d662c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.252+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/425763247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 0x560997d662c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe8f40cdae0 tx=0x7fe8f409c2c0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.252+0000 7fe90cfc0740 1 -- 172.21.15.50:0/425763247 shutdown_connections 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.252+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/425763247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d66800 0x560997d68bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.252+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/425763247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 0x560997d662c0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.252+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/425763247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d5ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.252+0000 7fe90cfc0740 1 -- 172.21.15.50:0/425763247 >> 172.21.15.50:0/425763247 conn(0x560997d57b30 msgr2=0x560997d57f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.252+0000 7fe90cfc0740 1 -- 172.21.15.50:0/425763247 shutdown_connections 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.252+0000 7fe90cfc0740 1 -- 172.21.15.50:0/425763247 wait complete. 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe90cfc0740 1 -- start start 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe90cfc0740 1 Processor -- start 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe90cfc0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d661d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe90cfc0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d5bf40 0x560997d634c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe90cfc0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d66800 0x560997d63ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe90cfc0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560997d69430 con 0x560997d66800 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe90cfc0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560997c5b0b0 con 0x560997d5bf40 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe90cfc0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x560997d695b0 con 0x560997d5b620 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe90cfc0740 10 client.0 _ll_register_callbacks cb 0x560997c48d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe903fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d66800 0x560997d63ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe903fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d66800 0x560997d63ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60832/0 (socket says 172.21.15.50:60832) 2024-04-23T18:25:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe903fff640 1 -- 172.21.15.50:0/4116786012 learned_addr learned my addr 172.21.15.50:0/4116786012 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe9037fe640 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d661d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.253+0000 7fe902ffd640 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d5bf40 0x560997d634c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe903fff640 1 -- 172.21.15.50:0/4116786012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 msgr2=0x560997d661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe903fff640 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d661d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe903fff640 1 -- 172.21.15.50:0/4116786012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d5bf40 msgr2=0x560997d634c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe903fff640 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d5bf40 0x560997d634c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe903fff640 1 -- 172.21.15.50:0/4116786012 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560997a9f620 con 0x560997d66800 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe9037fe640 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d661d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe903fff640 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d66800 0x560997d63ad0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc012330 tx=0x7fe8fc0096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c89efd815d6990 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe8e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe8e2ffd640 1 -- 172.21.15.50:0/4116786012 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc003c40 con 0x560997d66800 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe8e2ffd640 1 -- 172.21.15.50:0/4116786012 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fe8fc003de0 con 0x560997d66800 2024-04-23T18:25:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe90cfc0740 1 -- 172.21.15.50:0/4116786012 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8f409c3c0 con 0x560997d66800 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe8e2ffd640 1 -- 172.21.15.50:0/4116786012 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe8fc0118e0 con 0x560997d66800 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe90cfc0740 10 client.16350 fetch_fsmap learned FSMap version 33 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.254+0000 7fe90cfc0740 1 -- 172.21.15.50:0/4116786012 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560997d75410 con 0x560997d66800 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.255+0000 7fe8e2ffd640 1 -- 172.21.15.50:0/4116786012 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8f409c3c0 con 0x560997d66800 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.255+0000 7fe8e2ffd640 1 -- 172.21.15.50:0/4116786012 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fe8fc01f070 con 0x560997d66800 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.255+0000 7fe90cfc0740 10 client.16350 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.255+0000 7fe90cfc0740 10 client.16350 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.255+0000 7fe90cfc0740 20 client.16350 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.255+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560997d769c0 0x560997d78db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.255+0000 7fe90cfc0740 4 client.16350 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.255+0000 7fe90cfc0740 1 -- 172.21.15.50:0/4116786012 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560997cce390 con 0x560997d769c0 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.255+0000 7fe9037fe640 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560997d769c0 0x560997d78db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.256+0000 7fe8e2ffd640 10 client.16350 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:21.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.256+0000 7fe9037fe640 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560997d769c0 0x560997d78db0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9326a48f4f65f30 server_cookie=55001d83a124304c in_seq=0 out_seq=0 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.256+0000 7fe8e2ffd640 1 -- 172.21.15.50:0/4116786012 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560997cce390 con 0x560997d769c0 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe8e2ffd640 10 client.16350 handle_command_reply: tid=0 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe90cfc0740 1 client.16350 shutdown 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe90cfc0740 1 -- 172.21.15.50:0/4116786012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d66800 msgr2=0x560997d63ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d66800 0x560997d63ad0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc012330 tx=0x7fe8fc0096a0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe90cfc0740 1 -- 172.21.15.50:0/4116786012 shutdown_connections 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560997d769c0 0x560997d78db0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d66800 0x560997d63ad0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d5bf40 0x560997d634c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/4116786012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d661d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe8e2ffd640 0 client.16350 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe90cfc0740 1 -- 172.21.15.50:0/4116786012 >> 172.21.15.50:0/4116786012 conn(0x560997d57b30 msgr2=0x560997d5d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe90cfc0740 1 -- 172.21.15.50:0/4116786012 shutdown_connections 2024-04-23T18:25:21.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe90cfc0740 1 -- 172.21.15.50:0/4116786012 wait complete. 2024-04-23T18:25:21.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.257+0000 7fe90cfc0740 20 client.16350 trim_cache size 0 max 16384 2024-04-23T18:25:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.272+0000 7fe90cfc0740 1 Processor -- start 2024-04-23T18:25:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.272+0000 7fe90cfc0740 1 -- start start 2024-04-23T18:25:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.272+0000 7fe90cfc0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d722f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.272+0000 7fe90cfc0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 0x560997d56b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.272+0000 7fe90cfc0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d66800 0x560997d7e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe90cfc0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe9041b26a0 con 0x560997d5bf40 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe90cfc0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe9041aff70 con 0x560997d66800 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe90cfc0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe9041adde0 con 0x560997d5b620 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe902ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 0x560997d56b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe902ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 0x560997d56b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60838/0 (socket says 172.21.15.50:60838) 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe902ffd640 1 -- 172.21.15.50:0/3636994171 learned_addr learned my addr 172.21.15.50:0/3636994171 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe903fff640 1 --2- 172.21.15.50:0/3636994171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d66800 0x560997d7e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe9037fe640 1 --2- 172.21.15.50:0/3636994171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe902ffd640 1 -- 172.21.15.50:0/3636994171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 msgr2=0x560997d722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe902ffd640 1 --2- 172.21.15.50:0/3636994171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d722f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe902ffd640 1 -- 172.21.15.50:0/3636994171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d66800 msgr2=0x560997d7e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe902ffd640 1 --2- 172.21.15.50:0/3636994171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d66800 0x560997d7e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe902ffd640 1 -- 172.21.15.50:0/3636994171 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560997d57810 con 0x560997d5bf40 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe9037fe640 1 --2- 172.21.15.50:0/3636994171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d722f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe902ffd640 1 --2- 172.21.15.50:0/3636994171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 0x560997d56b30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8009900 tx=0x7fe8f8032df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974b97266a34cb5d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe8e97fa640 1 -- 172.21.15.50:0/3636994171 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8035040 con 0x560997d5bf40 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe8e97fa640 1 -- 172.21.15.50:0/3636994171 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8f80316a0 con 0x560997d5bf40 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.273+0000 7fe8e97fa640 1 -- 172.21.15.50:0/3636994171 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f80046c0 con 0x560997d5bf40 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.274+0000 7fe90cfc0740 1 -- 172.21.15.50:0/3636994171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 msgr2=0x560997d56b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.274+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/3636994171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 0x560997d56b30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8009900 tx=0x7fe8f8032df0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.274+0000 7fe90cfc0740 1 -- 172.21.15.50:0/3636994171 shutdown_connections 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.274+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/3636994171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d66800 0x560997d7e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.274+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/3636994171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5bf40 0x560997d56b30 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.274+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/3636994171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d5b620 0x560997d722f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.274+0000 7fe90cfc0740 1 -- 172.21.15.50:0/3636994171 >> 172.21.15.50:0/3636994171 conn(0x560997d57b30 msgr2=0x560997d56420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.274+0000 7fe90cfc0740 1 -- 172.21.15.50:0/3636994171 shutdown_connections 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.274+0000 7fe90cfc0740 1 -- 172.21.15.50:0/3636994171 wait complete. 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.274+0000 7fe90cfc0740 1 -- start start 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe90cfc0740 1 Processor -- start 2024-04-23T18:25:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe90cfc0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5b620 0x560997d6c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe90cfc0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d5bf40 0x560997d6c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe90cfc0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d66800 0x560997d6cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe90cfc0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe9041ae430 con 0x560997d5b620 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe90cfc0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe9041af8a0 con 0x560997d5bf40 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe90cfc0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe9041aeb00 con 0x560997d66800 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe90cfc0740 10 client.0 _ll_register_callbacks cb 0x7fe8fc012030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe9037fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5b620 0x560997d6c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe9037fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5b620 0x560997d6c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60840/0 (socket says 172.21.15.50:60840) 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe9037fe640 1 -- 172.21.15.50:0/2288858819 learned_addr learned my addr 172.21.15.50:0/2288858819 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe902ffd640 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d5bf40 0x560997d6c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe903fff640 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d66800 0x560997d6cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe9037fe640 1 -- 172.21.15.50:0/2288858819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d66800 msgr2=0x560997d6cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe9037fe640 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d66800 0x560997d6cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe9037fe640 1 -- 172.21.15.50:0/2288858819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d5bf40 msgr2=0x560997d6c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe9037fe640 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d5bf40 0x560997d6c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe9037fe640 1 -- 172.21.15.50:0/2288858819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560997ddc0d0 con 0x560997d5b620 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe9037fe640 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5b620 0x560997d6c470 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fe8f409c0c0 tx=0x7fe8f40d0750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a225bdee30d578 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.275+0000 7fe8e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe8e2ffd640 1 -- 172.21.15.50:0/2288858819 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4085070 con 0x560997d5b620 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe8e2ffd640 1 -- 172.21.15.50:0/2288858819 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fe8f4098a30 con 0x560997d5b620 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe90cfc0740 1 -- 172.21.15.50:0/2288858819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8f80316a0 con 0x560997d5b620 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe8e2ffd640 1 -- 172.21.15.50:0/2288858819 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe8f40b0e60 con 0x560997d5b620 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe90cfc0740 10 client.16356 fetch_fsmap learned FSMap version 33 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe90cfc0740 1 -- 172.21.15.50:0/2288858819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560997ddd330 con 0x560997d5b620 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe8e2ffd640 1 -- 172.21.15.50:0/2288858819 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8f80316a0 con 0x560997d5b620 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe8e2ffd640 1 -- 172.21.15.50:0/2288858819 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fe8f4098110 con 0x560997d5b620 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe90cfc0740 10 client.16356 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe90cfc0740 10 client.16356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe90cfc0740 20 client.16356 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560997ddd620 0x560997ddfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe90cfc0740 4 client.16356 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe90cfc0740 1 -- 172.21.15.50:0/2288858819 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560997cc8200 con 0x560997ddd620 2024-04-23T18:25:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.276+0000 7fe902ffd640 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560997ddd620 0x560997ddfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.277+0000 7fe902ffd640 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560997ddd620 0x560997ddfa10 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15e48004c18fe88e server_cookie=8b97473c4330a612 in_seq=0 out_seq=0 2024-04-23T18:25:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.277+0000 7fe8e2ffd640 10 client.16356 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe8e2ffd640 1 -- 172.21.15.50:0/2288858819 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560997cc8200 con 0x560997ddd620 2024-04-23T18:25:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe8e2ffd640 10 client.16356 handle_command_reply: tid=0 2024-04-23T18:25:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe90cfc0740 1 client.16356 shutdown 2024-04-23T18:25:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe90cfc0740 1 -- 172.21.15.50:0/2288858819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5b620 msgr2=0x560997d6c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5b620 0x560997d6c470 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fe8f409c0c0 tx=0x7fe8f40d0750 comp rx=0 tx=0).stop 2024-04-23T18:25:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe90cfc0740 1 -- 172.21.15.50:0/2288858819 shutdown_connections 2024-04-23T18:25:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560997ddd620 0x560997ddfa10 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560997d66800 0x560997d6cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560997d5bf40 0x560997d6c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe8e2ffd640 0 client.16356 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe90cfc0740 1 --2- 172.21.15.50:0/2288858819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560997d5b620 0x560997d6c470 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe90cfc0740 1 -- 172.21.15.50:0/2288858819 >> 172.21.15.50:0/2288858819 conn(0x560997d57b30 msgr2=0x560997d56cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:21.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe90cfc0740 1 -- 172.21.15.50:0/2288858819 shutdown_connections 2024-04-23T18:25:21.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe90cfc0740 1 -- 172.21.15.50:0/2288858819 wait complete. 2024-04-23T18:25:21.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.299+0000 7fe90cfc0740 20 client.16356 trim_cache size 0 max 16384 2024-04-23T18:25:21.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:25:21.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:25:21.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:25:21.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:25:21.295 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.300+0000 7fe90a1cd640 1 -- 172.21.15.50:0/2534293656 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe8ec0a3dc0 msgr2=0x7fe8ec0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:21.295 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.300+0000 7fe90a1cd640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe8ec0a3dc0 0x7fe8ec0a6280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009fd0 tx=0x7fe8fc009d60 comp rx=0 tx=0).stop 2024-04-23T18:25:21.295 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.300+0000 7fe90a1cd640 1 -- 172.21.15.50:0/2534293656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9041031e0 msgr2=0x7fe904100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:21.295 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.300+0000 7fe90a1cd640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9041031e0 0x7fe904100e30 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400d950 tx=0x7fe8f400de10 comp rx=0 tx=0).stop 2024-04-23T18:25:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.300+0000 7fe90a1cd640 1 -- 172.21.15.50:0/2534293656 shutdown_connections 2024-04-23T18:25:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.300+0000 7fe90a1cd640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe8ec0a3dc0 0x7fe8ec0a6280 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.300+0000 7fe90a1cd640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe90410c520 0x7fe9041018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.300+0000 7fe90a1cd640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe904103b00 0x7fe904101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.300+0000 7fe90a1cd640 1 --2- 172.21.15.50:0/2534293656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9041031e0 0x7fe904100e30 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.300+0000 7fe90a1cd640 1 -- 172.21.15.50:0/2534293656 >> 172.21.15.50:0/2534293656 conn(0x7fe904077fd0 msgr2=0x7fe9040fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.300+0000 7fe90a1cd640 1 -- 172.21.15.50:0/2534293656 shutdown_connections 2024-04-23T18:25:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:21.300+0000 7fe90a1cd640 1 -- 172.21.15.50:0/2534293656 wait complete. 2024-04-23T18:25:22.213 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.78 MB/sec execute 236 sec latency 20697.052 ms 2024-04-23T18:25:22.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:22 smithi188 ceph-mon[32694]: pgmap v212: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 133 KiB/s wr, 19 op/s 2024-04-23T18:25:22.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:22 smithi050 ceph-mon[29187]: pgmap v212: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 133 KiB/s wr, 19 op/s 2024-04-23T18:25:22.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:22 smithi177 ceph-mon[32308]: pgmap v212: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 133 KiB/s wr, 19 op/s 2024-04-23T18:25:23.213 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.76 MB/sec execute 237 sec latency 21697.151 ms 2024-04-23T18:25:24.213 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.74 MB/sec execute 238 sec latency 22697.265 ms 2024-04-23T18:25:24.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:24 smithi188 ceph-mon[32694]: pgmap v213: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 146 KiB/s wr, 25 op/s 2024-04-23T18:25:24.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:24 smithi050 ceph-mon[29187]: pgmap v213: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 146 KiB/s wr, 25 op/s 2024-04-23T18:25:24.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:24 smithi177 ceph-mon[32308]: pgmap v213: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 146 KiB/s wr, 25 op/s 2024-04-23T18:25:25.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.72 MB/sec execute 239 sec latency 23697.388 ms 2024-04-23T18:25:26.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.70 MB/sec execute 240 sec latency 24697.514 ms 2024-04-23T18:25:26.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:26 smithi188 ceph-mon[32694]: pgmap v214: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 111 KiB/s wr, 13 op/s 2024-04-23T18:25:26.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:26 smithi050 ceph-mon[29187]: pgmap v214: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 111 KiB/s wr, 13 op/s 2024-04-23T18:25:26.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:26 smithi177 ceph-mon[32308]: pgmap v214: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 111 KiB/s wr, 13 op/s 2024-04-23T18:25:27.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.68 MB/sec execute 241 sec latency 25697.571 ms 2024-04-23T18:25:28.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.66 MB/sec execute 242 sec latency 26697.651 ms 2024-04-23T18:25:28.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:28 smithi188 ceph-mon[32694]: pgmap v215: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 286 KiB/s wr, 20 op/s 2024-04-23T18:25:28.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:28 smithi050 ceph-mon[29187]: pgmap v215: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 286 KiB/s wr, 20 op/s 2024-04-23T18:25:28.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:28 smithi177 ceph-mon[32308]: pgmap v215: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 286 KiB/s wr, 20 op/s 2024-04-23T18:25:29.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.64 MB/sec execute 243 sec latency 27697.775 ms 2024-04-23T18:25:30.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.62 MB/sec execute 244 sec latency 28697.897 ms 2024-04-23T18:25:30.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:30 smithi188 ceph-mon[32694]: pgmap v216: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 205 KiB/s wr, 15 op/s 2024-04-23T18:25:30.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:30 smithi050 ceph-mon[29187]: pgmap v216: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 205 KiB/s wr, 15 op/s 2024-04-23T18:25:30.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:30 smithi177 ceph-mon[32308]: pgmap v216: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 205 KiB/s wr, 15 op/s 2024-04-23T18:25:31.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:25:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.202+0000 7f314dc67640 1 -- 172.21.15.50:0/11743692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3148107280 msgr2=0x7f314810ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.202+0000 7f314dc67640 1 --2- 172.21.15.50:0/11743692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3148107280 0x7f314810ee50 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3140009ca0 tx=0x7f314002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.202+0000 7f314dc67640 1 -- 172.21.15.50:0/11743692 shutdown_connections 2024-04-23T18:25:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.202+0000 7f314dc67640 1 --2- 172.21.15.50:0/11743692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3148107280 0x7f314810ee50 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.202+0000 7f314dc67640 1 --2- 172.21.15.50:0/11743692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148102bf0 0x7f3148106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.202+0000 7f314dc67640 1 --2- 172.21.15.50:0/11743692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3148102240 0x7f3148102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.202+0000 7f314dc67640 1 -- 172.21.15.50:0/11743692 >> 172.21.15.50:0/11743692 conn(0x7f31480790e0 msgr2=0x7f31480794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.202+0000 7f314dc67640 1 -- 172.21.15.50:0/11743692 shutdown_connections 2024-04-23T18:25:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.202+0000 7f314dc67640 1 -- 172.21.15.50:0/11743692 wait complete. 2024-04-23T18:25:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.203+0000 7f314dc67640 1 Processor -- start 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.203+0000 7f314dc67640 1 -- start start 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.203+0000 7f314dc67640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3148102240 0x7f3148074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.203+0000 7f314dc67640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3148107280 0x7f3148074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.203+0000 7f314dc67640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148074cd0 0x7f31481a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.203+0000 7f314dc67640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f31481121e0 con 0x7f3148102240 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.203+0000 7f314dc67640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3148112060 con 0x7f3148107280 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.203+0000 7f314dc67640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3148112360 con 0x7f3148074cd0 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.203+0000 7f31477fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3148102240 0x7f3148074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.203+0000 7f31477fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3148102240 0x7f3148074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51500/0 (socket says 172.21.15.50:51500) 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.203+0000 7f31477fe640 1 -- 172.21.15.50:0/2908874083 learned_addr learned my addr 172.21.15.50:0/2908874083 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f3147fff640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148074cd0 0x7f31481a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f3146ffd640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3148107280 0x7f3148074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f31477fe640 1 -- 172.21.15.50:0/2908874083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148074cd0 msgr2=0x7f31481a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f31477fe640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148074cd0 0x7f31481a68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f31477fe640 1 -- 172.21.15.50:0/2908874083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3148107280 msgr2=0x7f3148074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f31477fe640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3148107280 0x7f3148074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f31477fe640 1 -- 172.21.15.50:0/2908874083 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31481a6fd0 con 0x7f3148102240 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f3146ffd640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3148107280 0x7f3148074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f3147fff640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148074cd0 0x7f31481a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f31477fe640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3148102240 0x7f3148074250 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f313800d950 tx=0x7f313800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f3144ff9640 1 -- 172.21.15.50:0/2908874083 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3138017030 con 0x7f3148102240 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f3144ff9640 1 -- 172.21.15.50:0/2908874083 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3138004020 con 0x7f3148102240 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f314dc67640 1 -- 172.21.15.50:0/2908874083 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31481a72c0 con 0x7f3148102240 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f3144ff9640 1 -- 172.21.15.50:0/2908874083 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31380047a0 con 0x7f3148102240 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.204+0000 7f314dc67640 1 -- 172.21.15.50:0/2908874083 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31481a7810 con 0x7f3148102240 2024-04-23T18:25:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.205+0000 7f314dc67640 1 -- 172.21.15.50:0/2908874083 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3148075780 con 0x7f3148102240 2024-04-23T18:25:31.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.206+0000 7f3144ff9640 1 -- 172.21.15.50:0/2908874083 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3138009a30 con 0x7f3148102240 2024-04-23T18:25:31.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.207+0000 7f3144ff9640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f31300a3d50 0x7f31300a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.207+0000 7f3146ffd640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f31300a3d50 0x7f31300a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.207+0000 7f3144ff9640 1 -- 172.21.15.50:0/2908874083 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f31380d02d0 con 0x7f3148102240 2024-04-23T18:25:31.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.208+0000 7f3146ffd640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f31300a3d50 0x7f31300a6210 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f313c005d90 tx=0x7f313c005d20 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.212+0000 7f3144ff9640 1 -- 172.21.15.50:0/2908874083 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f313809c960 con 0x7f3148102240 2024-04-23T18:25:31.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 403874 4.60 MB/sec execute 245 sec latency 29697.964 ms 2024-04-23T18:25:31.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.438+0000 7f314dc67640 1 -- 172.21.15.50:0/2908874083 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f314810a4a0 con 0x7f3148102240 2024-04-23T18:25:31.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.438+0000 7f3144ff9640 1 -- 172.21.15.50:0/2908874083 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11875 (secure 0 0 0) 0x7f313809c260 con 0x7f3148102240 2024-04-23T18:25:31.432 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:25:31.432 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:25:03.818074+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25822":{"gid":25822,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/911343542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":911343542},{"type":"v1","addr":"172.21.15.177:6837","nonce":911343542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:25:31.432 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 33 2024-04-23T18:25:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.440+0000 7f314dc67640 1 -- 172.21.15.50:0/2908874083 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f31300a3d50 msgr2=0x7f31300a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.440+0000 7f314dc67640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f31300a3d50 0x7f31300a6210 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f313c005d90 tx=0x7f313c005d20 comp rx=0 tx=0).stop 2024-04-23T18:25:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.440+0000 7f314dc67640 1 -- 172.21.15.50:0/2908874083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3148102240 msgr2=0x7f3148074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.440+0000 7f314dc67640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3148102240 0x7f3148074250 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f313800d950 tx=0x7f313800de10 comp rx=0 tx=0).stop 2024-04-23T18:25:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.440+0000 7f314dc67640 1 -- 172.21.15.50:0/2908874083 shutdown_connections 2024-04-23T18:25:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.440+0000 7f314dc67640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f31300a3d50 0x7f31300a6210 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.440+0000 7f314dc67640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148074cd0 0x7f31481a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.440+0000 7f314dc67640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3148107280 0x7f3148074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.440+0000 7f314dc67640 1 --2- 172.21.15.50:0/2908874083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3148102240 0x7f3148074250 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.441+0000 7f314dc67640 1 -- 172.21.15.50:0/2908874083 >> 172.21.15.50:0/2908874083 conn(0x7f31480790e0 msgr2=0x7f31481002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.441+0000 7f314dc67640 1 -- 172.21.15.50:0/2908874083 shutdown_connections 2024-04-23T18:25:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.441+0000 7f314dc67640 1 -- 172.21.15.50:0/2908874083 wait complete. 2024-04-23T18:25:31.445 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id dea382bb --release --await' 2024-04-23T18:25:31.445 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id dea382bb --release --await 2024-04-23T18:25:31.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.560+0000 7fd2b416f640 1 -- 172.21.15.50:0/1093522368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2ac10c510 msgr2=0x7fd2ac10e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.560+0000 7fd2b416f640 1 --2- 172.21.15.50:0/1093522368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2ac10c510 0x7fd2ac10e9b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009ca0 tx=0x7fd2a402a9e0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.560+0000 7fd2b416f640 1 -- 172.21.15.50:0/1093522368 shutdown_connections 2024-04-23T18:25:31.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.560+0000 7fd2b416f640 1 --2- 172.21.15.50:0/1093522368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2ac10c510 0x7fd2ac10e9b0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.560+0000 7fd2b416f640 1 --2- 172.21.15.50:0/1093522368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac103af0 0x7fd2ac10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.560+0000 7fd2b416f640 1 --2- 172.21.15.50:0/1093522368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2ac1031d0 0x7fd2ac1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.560+0000 7fd2b416f640 1 -- 172.21.15.50:0/1093522368 >> 172.21.15.50:0/1093522368 conn(0x7fd2ac077f50 msgr2=0x7fd2ac078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.560+0000 7fd2b416f640 1 -- 172.21.15.50:0/1093522368 shutdown_connections 2024-04-23T18:25:31.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.560+0000 7fd2b416f640 1 -- 172.21.15.50:0/1093522368 wait complete. 2024-04-23T18:25:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b416f640 1 Processor -- start 2024-04-23T18:25:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b416f640 1 -- start start 2024-04-23T18:25:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b416f640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac1031d0 0x7fd2ac1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b416f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2ac103af0 0x7fd2ac198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b416f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2ac10c510 0x7fd2ac19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b416f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac1121a0 con 0x7fd2ac103af0 2024-04-23T18:25:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b416f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac112020 con 0x7fd2ac10c510 2024-04-23T18:25:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b416f640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac112320 con 0x7fd2ac1031d0 2024-04-23T18:25:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b170a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2ac103af0 0x7fd2ac198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b170a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2ac103af0 0x7fd2ac198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51514/0 (socket says 172.21.15.50:51514) 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b170a640 1 -- 172.21.15.50:0/1334964278 learned_addr learned my addr 172.21.15.50:0/1334964278 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b270c640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2ac10c510 0x7fd2ac19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b170a640 1 -- 172.21.15.50:0/1334964278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac1031d0 msgr2=0x7fd2ac1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b1f0b640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac1031d0 0x7fd2ac1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b170a640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac1031d0 0x7fd2ac1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b170a640 1 -- 172.21.15.50:0/1334964278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2ac10c510 msgr2=0x7fd2ac19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b170a640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2ac10c510 0x7fd2ac19cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b170a640 1 -- 172.21.15.50:0/1334964278 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ac19d6d0 con 0x7fd2ac103af0 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.561+0000 7fd2b270c640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2ac10c510 0x7fd2ac19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.562+0000 7fd2b170a640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2ac103af0 0x7fd2ac198c20 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800d770 tx=0x7fd2a800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.562+0000 7fd2a2ffd640 1 -- 172.21.15.50:0/1334964278 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8017070 con 0x7fd2ac103af0 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.562+0000 7fd2a2ffd640 1 -- 172.21.15.50:0/1334964278 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2a80054d0 con 0x7fd2ac103af0 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.562+0000 7fd2b416f640 1 -- 172.21.15.50:0/1334964278 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2ac19d9c0 con 0x7fd2ac103af0 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.562+0000 7fd2a2ffd640 1 -- 172.21.15.50:0/1334964278 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8004c30 con 0x7fd2ac103af0 2024-04-23T18:25:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.562+0000 7fd2b416f640 1 -- 172.21.15.50:0/1334964278 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2ac1a5250 con 0x7fd2ac103af0 2024-04-23T18:25:31.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.564+0000 7fd2a2ffd640 1 -- 172.21.15.50:0/1334964278 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd2a8004e10 con 0x7fd2ac103af0 2024-04-23T18:25:31.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.564+0000 7fd2b6f65740 1 Processor -- start 2024-04-23T18:25:31.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b6f65740 1 -- start start 2024-04-23T18:25:31.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b6f65740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c2f6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b6f65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c300f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b6f65740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c301480 0x55585c303870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b6f65740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55585c25b6d0 con 0x55585c2f6490 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b6f65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55585c1f81a0 con 0x55585c2f6db0 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b6f65740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55585c257690 con 0x55585c301480 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b270c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c2f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b270c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c2f6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51528/0 (socket says 172.21.15.50:51528) 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b270c640 1 -- 172.21.15.50:0/1058066310 learned_addr learned my addr 172.21.15.50:0/1058066310 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2a2ffd640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd2980a3da0 0x7fd2980a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b170a640 1 --2- 172.21.15.50:0/1058066310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c301480 0x55585c303870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b1f0b640 1 --2- 172.21.15.50:0/1058066310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c300f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b270c640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd2980a3da0 0x7fd2980a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b270c640 1 -- 172.21.15.50:0/1058066310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c301480 msgr2=0x55585c303870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b270c640 1 --2- 172.21.15.50:0/1058066310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c301480 0x55585c303870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b270c640 1 -- 172.21.15.50:0/1058066310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 msgr2=0x55585c300f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b270c640 1 --2- 172.21.15.50:0/1058066310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c300f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2a2ffd640 1 -- 172.21.15.50:0/1334964278 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fd2a80d02c0 con 0x7fd2ac103af0 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.565+0000 7fd2b270c640 1 -- 172.21.15.50:0/1058066310 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55585c2025b0 con 0x55585c2f6490 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.566+0000 7fd2b270c640 1 --2- 172.21.15.50:0/1058066310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c2f6870 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4005e30 tx=0x7fd2a4003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a1cab2ed81fbc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.566+0000 7fd297fff640 1 -- 172.21.15.50:0/1058066310 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4041050 con 0x55585c2f6490 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.566+0000 7fd297fff640 1 -- 172.21.15.50:0/1058066310 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2a4042040 con 0x55585c2f6490 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.566+0000 7fd297fff640 1 -- 172.21.15.50:0/1058066310 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a404b020 con 0x55585c2f6490 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.567+0000 7fd2b270c640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd2980a3da0 0x7fd2980a6260 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd2a404c000 tx=0x7fd2a404a000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.567+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058066310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 msgr2=0x55585c2f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.567+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1058066310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c2f6870 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4005e30 tx=0x7fd2a4003a70 comp rx=0 tx=0).stop 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.567+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058066310 shutdown_connections 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.567+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1058066310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c301480 0x55585c303870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.567+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1058066310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c300f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.567+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1058066310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c2f6870 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.567+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058066310 >> 172.21.15.50:0/1058066310 conn(0x55585c2f27b0 msgr2=0x55585c2f2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.567+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058066310 shutdown_connections 2024-04-23T18:25:31.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.567+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058066310 wait complete. 2024-04-23T18:25:31.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.567+0000 7fd2b6f65740 1 -- start start 2024-04-23T18:25:31.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.567+0000 7fd2b6f65740 1 Processor -- start 2024-04-23T18:25:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b6f65740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c300f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b6f65740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c2f6db0 0x55585c2fe200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b6f65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c301480 0x55585c2fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b6f65740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55585c304310 con 0x55585c2f6490 2024-04-23T18:25:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b6f65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55585c1f5b20 con 0x55585c301480 2024-04-23T18:25:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b6f65740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55585c304490 con 0x55585c2f6db0 2024-04-23T18:25:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b6f65740 10 client.0 _ll_register_callbacks cb 0x55585c1f5530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b1f0b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c300f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b1f0b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c300f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51534/0 (socket says 172.21.15.50:51534) 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b1f0b640 1 -- 172.21.15.50:0/1058248589 learned_addr learned my addr 172.21.15.50:0/1058248589 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b1f0b640 1 -- 172.21.15.50:0/1058248589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c2f6db0 msgr2=0x55585c2fe200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b1f0b640 1 --2- 172.21.15.50:0/1058248589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c2f6db0 0x55585c2fe200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b1f0b640 1 -- 172.21.15.50:0/1058248589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c301480 msgr2=0x55585c2fe8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b1f0b640 1 --2- 172.21.15.50:0/1058248589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c301480 0x55585c2fe8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b1f0b640 1 -- 172.21.15.50:0/1058248589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55585bf6d920 con 0x55585c2f6490 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b1f0b640 1 --2- 172.21.15.50:0/1058248589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c300f10 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd29c013710 tx=0x7fd29c013bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3acc6a4b9c1071 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2957fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2957fa640 1 -- 172.21.15.50:0/1058248589 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c011dd0 con 0x55585c2f6490 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2957fa640 1 -- 172.21.15.50:0/1058248589 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fd29c004020 con 0x55585c2f6490 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.568+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058248589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2a4042040 con 0x55585c2f6490 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.569+0000 7fd2957fa640 1 -- 172.21.15.50:0/1058248589 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd29c004220 con 0x55585c2f6490 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.569+0000 7fd2b6f65740 10 client.16374 fetch_fsmap learned FSMap version 33 2024-04-23T18:25:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.569+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058248589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55585c039d30 con 0x55585c2f6490 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.569+0000 7fd2957fa640 1 -- 172.21.15.50:0/1058248589 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2a4042040 con 0x55585c2f6490 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.569+0000 7fd2957fa640 1 -- 172.21.15.50:0/1058248589 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fd29c01a070 con 0x55585c2f6490 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.569+0000 7fd2b6f65740 10 client.16374 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.569+0000 7fd2b6f65740 10 client.16374 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.569+0000 7fd2b6f65740 20 client.16374 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.569+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1058248589 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55585c311740 0x55585c313c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.569+0000 7fd2b6f65740 4 client.16374 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.569+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058248589 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55585c219500 con 0x55585c311740 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.569+0000 7fd2b170a640 1 --2- 172.21.15.50:0/1058248589 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55585c311740 0x55585c313c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.570+0000 7fd2957fa640 10 client.16374 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.570+0000 7fd2b170a640 1 --2- 172.21.15.50:0/1058248589 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55585c311740 0x55585c313c00 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5867e42f37cda583 server_cookie=1ef8530a8647039a in_seq=0 out_seq=0 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.570+0000 7fd2957fa640 1 -- 172.21.15.50:0/1058248589 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55585c219500 con 0x55585c311740 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.570+0000 7fd2957fa640 10 client.16374 handle_command_reply: tid=0 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.570+0000 7fd2b6f65740 1 client.16374 shutdown 2024-04-23T18:25:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.570+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058248589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 msgr2=0x55585c300f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.570+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1058248589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c300f10 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd29c013710 tx=0x7fd29c013bd0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.571+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058248589 shutdown_connections 2024-04-23T18:25:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.571+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1058248589 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55585c311740 0x55585c313c00 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.571+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1058248589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c301480 0x55585c2fe8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.571+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1058248589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c2f6db0 0x55585c2fe200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.571+0000 7fd2957fa640 0 client.16374 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.571+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1058248589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c300f10 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.571+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058248589 >> 172.21.15.50:0/1058248589 conn(0x55585c2f27b0 msgr2=0x55585c2f19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.571+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058248589 shutdown_connections 2024-04-23T18:25:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.571+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1058248589 wait complete. 2024-04-23T18:25:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.571+0000 7fd2b6f65740 20 client.16374 trim_cache size 0 max 16384 2024-04-23T18:25:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.588+0000 7fd2b6f65740 1 Processor -- start 2024-04-23T18:25:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.588+0000 7fd2b6f65740 1 -- start start 2024-04-23T18:25:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.588+0000 7fd2b6f65740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c30d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.588+0000 7fd2b6f65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c313b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.588+0000 7fd2b6f65740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c301480 0x55585c300a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.588+0000 7fd2b6f65740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac1b0450 con 0x55585c2f6490 2024-04-23T18:25:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.588+0000 7fd2b6f65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac1add20 con 0x55585c2f6db0 2024-04-23T18:25:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.588+0000 7fd2b6f65740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac1abb90 con 0x55585c301480 2024-04-23T18:25:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.588+0000 7fd2b1f0b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c30d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.588+0000 7fd2b1f0b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c30d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51542/0 (socket says 172.21.15.50:51542) 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.588+0000 7fd2b1f0b640 1 -- 172.21.15.50:0/4028031959 learned_addr learned my addr 172.21.15.50:0/4028031959 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.589+0000 7fd2b170a640 1 --2- 172.21.15.50:0/4028031959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c313b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.589+0000 7fd2b270c640 1 --2- 172.21.15.50:0/4028031959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c301480 0x55585c300a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.589+0000 7fd2b1f0b640 1 -- 172.21.15.50:0/4028031959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c301480 msgr2=0x55585c300a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.589+0000 7fd2b1f0b640 1 --2- 172.21.15.50:0/4028031959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c301480 0x55585c300a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.589+0000 7fd2b1f0b640 1 -- 172.21.15.50:0/4028031959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 msgr2=0x55585c313b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.589+0000 7fd2b1f0b640 1 --2- 172.21.15.50:0/4028031959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c313b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.589+0000 7fd2b1f0b640 1 -- 172.21.15.50:0/4028031959 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55585c2fd1b0 con 0x55585c2f6490 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.589+0000 7fd2b270c640 1 --2- 172.21.15.50:0/4028031959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c301480 0x55585c300a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.589+0000 7fd2b1f0b640 1 --2- 172.21.15.50:0/4028031959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c30d1d0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd29c000c00 tx=0x7fd29c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7df5c5992d5dbe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.589+0000 7fd2977fe640 1 -- 172.21.15.50:0/4028031959 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c030070 con 0x55585c2f6490 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.589+0000 7fd2977fe640 1 -- 172.21.15.50:0/4028031959 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd29c023a20 con 0x55585c2f6490 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.589+0000 7fd2977fe640 1 -- 172.21.15.50:0/4028031959 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c01b3f0 con 0x55585c2f6490 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 -- 172.21.15.50:0/4028031959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 msgr2=0x55585c30d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/4028031959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c30d1d0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd29c000c00 tx=0x7fd29c009310 comp rx=0 tx=0).stop 2024-04-23T18:25:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 -- 172.21.15.50:0/4028031959 shutdown_connections 2024-04-23T18:25:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/4028031959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c301480 0x55585c300a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/4028031959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c313b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/4028031959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c2f6490 0x55585c30d1d0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 -- 172.21.15.50:0/4028031959 >> 172.21.15.50:0/4028031959 conn(0x55585c2f27b0 msgr2=0x55585c2f7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 -- 172.21.15.50:0/4028031959 shutdown_connections 2024-04-23T18:25:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 -- 172.21.15.50:0/4028031959 wait complete. 2024-04-23T18:25:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 -- start start 2024-04-23T18:25:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 Processor -- start 2024-04-23T18:25:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c2f6490 0x55585c3096f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c309c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c301480 0x55585c3046a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac1ac1e0 con 0x55585c301480 2024-04-23T18:25:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac1ad650 con 0x55585c2f6db0 2024-04-23T18:25:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac1ac8b0 con 0x55585c2f6490 2024-04-23T18:25:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.590+0000 7fd2b6f65740 10 client.0 _ll_register_callbacks cb 0x7fd29c009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b270c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c301480 0x55585c3046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b270c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c301480 0x55585c3046a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51546/0 (socket says 172.21.15.50:51546) 2024-04-23T18:25:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b270c640 1 -- 172.21.15.50:0/1875374128 learned_addr learned my addr 172.21.15.50:0/1875374128 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b1f0b640 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c2f6490 0x55585c3096f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b170a640 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c309c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b270c640 1 -- 172.21.15.50:0/1875374128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c2f6490 msgr2=0x55585c3096f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b270c640 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c2f6490 0x55585c3096f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b270c640 1 -- 172.21.15.50:0/1875374128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 msgr2=0x55585c309c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b270c640 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c309c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b270c640 1 -- 172.21.15.50:0/1875374128 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55585c2fec60 con 0x55585c301480 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b1f0b640 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c2f6490 0x55585c3096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b170a640 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c309c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b270c640 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c301480 0x55585c3046a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4005db0 tx=0x7fd2a40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87adfc134088373 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2957fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2957fa640 1 -- 172.21.15.50:0/1875374128 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4041040 con 0x55585c301480 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2957fa640 1 -- 172.21.15.50:0/1875374128 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fd2a4003d20 con 0x55585c301480 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1875374128 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd29c023a20 con 0x55585c301480 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2957fa640 1 -- 172.21.15.50:0/1875374128 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd2a404b040 con 0x55585c301480 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b6f65740 10 client.16380 fetch_fsmap learned FSMap version 33 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.591+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1875374128 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55585c3784a0 con 0x55585c301480 2024-04-23T18:25:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.592+0000 7fd2957fa640 1 -- 172.21.15.50:0/1875374128 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd29c023a20 con 0x55585c301480 2024-04-23T18:25:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.592+0000 7fd2957fa640 1 -- 172.21.15.50:0/1875374128 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fd2a4047570 con 0x55585c301480 2024-04-23T18:25:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.592+0000 7fd2b6f65740 10 client.16380 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:25:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.592+0000 7fd2b6f65740 10 client.16380 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:25:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.592+0000 7fd2b6f65740 20 client.16380 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.592+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55585c378820 0x55585c37ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.592+0000 7fd2b6f65740 4 client.16380 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "dea382bb", "release": true, "await": true}] 2024-04-23T18:25:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.592+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1875374128 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "dea382bb", "release": true, "await": true}) -- 0x55585c2690f0 con 0x55585c378820 2024-04-23T18:25:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.592+0000 7fd2b1f0b640 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55585c378820 0x55585c37ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.592+0000 7fd2957fa640 10 client.16380 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.592+0000 7fd2b1f0b640 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55585c378820 0x55585c37ac10 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d07409a9a2963d4 server_cookie=5ce94dde3e4ac3b8 in_seq=0 out_seq=0 2024-04-23T18:25:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.597+0000 7fd2957fa640 1 -- 172.21.15.50:0/1875374128 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55585c2690f0 con 0x55585c378820 2024-04-23T18:25:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.597+0000 7fd2957fa640 10 client.16380 handle_command_reply: tid=0 2024-04-23T18:25:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.597+0000 7fd2b6f65740 1 client.16380 shutdown 2024-04-23T18:25:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.597+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1875374128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c301480 msgr2=0x55585c3046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.597+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c301480 0x55585c3046a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4005db0 tx=0x7fd2a40095c0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.597+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1875374128 shutdown_connections 2024-04-23T18:25:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.597+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55585c378820 0x55585c37ac10 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.597+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55585c301480 0x55585c3046a0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.597+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55585c2f6db0 0x55585c309c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.597+0000 7fd2b6f65740 1 --2- 172.21.15.50:0/1875374128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55585c2f6490 0x55585c3096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.597+0000 7fd2957fa640 0 client.16380 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.597+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1875374128 >> 172.21.15.50:0/1875374128 conn(0x55585c2f27b0 msgr2=0x55585c2f8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1875374128 shutdown_connections 2024-04-23T18:25:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b6f65740 1 -- 172.21.15.50:0/1875374128 wait complete. 2024-04-23T18:25:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b6f65740 20 client.16380 trim_cache size 0 max 16384 2024-04-23T18:25:31.592 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 33, 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 24, 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "dea382bb": { 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 24, 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 89.5, 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:25:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:25:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:25:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:25:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:25:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:25:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:25:31.594 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:25:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b416f640 1 -- 172.21.15.50:0/1334964278 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd2980a3da0 msgr2=0x7fd2980a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b416f640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd2980a3da0 0x7fd2980a6260 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd2a404c000 tx=0x7fd2a404a000 comp rx=0 tx=0).stop 2024-04-23T18:25:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b416f640 1 -- 172.21.15.50:0/1334964278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2ac103af0 msgr2=0x7fd2ac198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b416f640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2ac103af0 0x7fd2ac198c20 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800d770 tx=0x7fd2a800dc30 comp rx=0 tx=0).stop 2024-04-23T18:25:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b416f640 1 -- 172.21.15.50:0/1334964278 shutdown_connections 2024-04-23T18:25:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b416f640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd2980a3da0 0x7fd2980a6260 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b416f640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2ac10c510 0x7fd2ac19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b416f640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2ac103af0 0x7fd2ac198c20 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b416f640 1 --2- 172.21.15.50:0/1334964278 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2ac1031d0 0x7fd2ac1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b416f640 1 -- 172.21.15.50:0/1334964278 >> 172.21.15.50:0/1334964278 conn(0x7fd2ac077f50 msgr2=0x7fd2ac0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b416f640 1 -- 172.21.15.50:0/1334964278 shutdown_connections 2024-04-23T18:25:31.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:25:31.598+0000 7fd2b416f640 1 -- 172.21.15.50:0/1334964278 wait complete. 2024-04-23T18:25:31.602 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'dea382bb', seconds elapsed: 30.7 2024-04-23T18:25:32.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 404950 4.61 MB/sec execute 246 sec latency 30077.826 ms 2024-04-23T18:25:32.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:32 smithi188 ceph-mon[32694]: pgmap v217: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 205 KiB/s wr, 15 op/s 2024-04-23T18:25:32.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:32 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2908874083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:32 smithi050 ceph-mon[29187]: pgmap v217: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 205 KiB/s wr, 15 op/s 2024-04-23T18:25:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:32 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2908874083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:32.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:32 smithi177 ceph-mon[32308]: pgmap v217: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 205 KiB/s wr, 15 op/s 2024-04-23T18:25:32.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:32 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2908874083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:25:33.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 406892 4.62 MB/sec execute 247 sec latency 18.094 ms 2024-04-23T18:25:34.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 408851 4.65 MB/sec execute 248 sec latency 27.259 ms 2024-04-23T18:25:34.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:34 smithi188 ceph-mon[32694]: pgmap v218: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 208 KiB/s wr, 18 op/s 2024-04-23T18:25:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.396+0000 7f0f06b5d640 1 -- 172.21.15.50:0/2715745817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f00101aa0 msgr2=0x7f0f00105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.396+0000 7f0f06b5d640 1 --2- 172.21.15.50:0/2715745817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f00101aa0 0x7f0f00105a90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009900 tx=0x7f0eec02a410 comp rx=0 tx=0).stop 2024-04-23T18:25:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.396+0000 7f0f06b5d640 1 -- 172.21.15.50:0/2715745817 shutdown_connections 2024-04-23T18:25:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.396+0000 7f0f06b5d640 1 --2- 172.21.15.50:0/2715745817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f00106170 0x7f0f0010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.396+0000 7f0f06b5d640 1 --2- 172.21.15.50:0/2715745817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f00101aa0 0x7f0f00105a90 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.396+0000 7f0f06b5d640 1 --2- 172.21.15.50:0/2715745817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f001010f0 0x7f0f001014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.396+0000 7f0f06b5d640 1 -- 172.21.15.50:0/2715745817 >> 172.21.15.50:0/2715745817 conn(0x7f0f00077fd0 msgr2=0x7f0f000783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.396+0000 7f0f06b5d640 1 -- 172.21.15.50:0/2715745817 shutdown_connections 2024-04-23T18:25:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.397+0000 7f0f06b5d640 1 -- 172.21.15.50:0/2715745817 wait complete. 2024-04-23T18:25:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.397+0000 7f0f06b5d640 1 Processor -- start 2024-04-23T18:25:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.397+0000 7f0f06b5d640 1 -- start start 2024-04-23T18:25:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.397+0000 7f0f06b5d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f001010f0 0x7f0f001994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.397+0000 7f0f06b5d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f00106170 0x7f0f00199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.397+0000 7f0f06b5d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f00199ff0 0x7f0f001a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.397+0000 7f0f06b5d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0f00111090 con 0x7f0f001010f0 2024-04-23T18:25:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.397+0000 7f0f06b5d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0f00110f10 con 0x7f0f00106170 2024-04-23T18:25:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.397+0000 7f0f06b5d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0f00111210 con 0x7f0f00199ff0 2024-04-23T18:25:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0f048f9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f001010f0 0x7f0f001994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0efffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f00106170 0x7f0f00199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0f050fa640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f00199ff0 0x7f0f001a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0f048f9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f001010f0 0x7f0f001994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42778/0 (socket says 172.21.15.50:42778) 2024-04-23T18:25:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0f048f9640 1 -- 172.21.15.50:0/3838160545 learned_addr learned my addr 172.21.15.50:0/3838160545 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0efffff640 1 -- 172.21.15.50:0/3838160545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f00199ff0 msgr2=0x7f0f001a4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0efffff640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f00199ff0 0x7f0f001a4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0efffff640 1 -- 172.21.15.50:0/3838160545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f001010f0 msgr2=0x7f0f001994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0efffff640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f001010f0 0x7f0f001994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0efffff640 1 -- 172.21.15.50:0/3838160545 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f00197340 con 0x7f0f00106170 2024-04-23T18:25:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0f050fa640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f00199ff0 0x7f0f001a4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0f048f9640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f001010f0 0x7f0f001994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.398+0000 7f0efffff640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f00106170 0x7f0f00199a10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0eec02a930 tx=0x7f0eec031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.399+0000 7f0efdffb640 1 -- 172.21.15.50:0/3838160545 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec03f070 con 0x7f0f00106170 2024-04-23T18:25:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.399+0000 7f0efdffb640 1 -- 172.21.15.50:0/3838160545 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eec004490 con 0x7f0f00106170 2024-04-23T18:25:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.399+0000 7f0f06b5d640 1 -- 172.21.15.50:0/3838160545 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f001975d0 con 0x7f0f00106170 2024-04-23T18:25:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.399+0000 7f0efdffb640 1 -- 172.21.15.50:0/3838160545 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec035070 con 0x7f0f00106170 2024-04-23T18:25:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.399+0000 7f0f06b5d640 1 -- 172.21.15.50:0/3838160545 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f00197b10 con 0x7f0f00106170 2024-04-23T18:25:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.401+0000 7f0efdffb640 1 -- 172.21.15.50:0/3838160545 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0eec03b460 con 0x7f0f00106170 2024-04-23T18:25:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.401+0000 7f0f0987a740 1 Processor -- start 2024-04-23T18:25:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.401+0000 7f0f0987a740 1 -- start start 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0f0987a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53620 0x55f41ab53a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0f0987a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53f40 0x55f41ab5e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0f0987a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab60bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0f0987a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f41aab8970 con 0x55f41ab53620 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0f0987a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f41aa554a0 con 0x55f41ab53f40 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0f0987a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f41aab4930 con 0x55f41ab5e800 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0efdffb640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0ef00a3e40 0x7f0ef00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0f050fa640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53620 0x55f41ab53a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0efffff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab60bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0f050fa640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53620 0x55f41ab53a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42790/0 (socket says 172.21.15.50:42790) 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0f050fa640 1 -- 172.21.15.50:0/3881905720 learned_addr learned my addr 172.21.15.50:0/3881905720 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0f048f9640 1 --2- 172.21.15.50:0/3881905720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53f40 0x55f41ab5e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0efdffb640 1 -- 172.21.15.50:0/3838160545 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f0eec0f0bc0 con 0x7f0f00106170 2024-04-23T18:25:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0f050fa640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0ef00a3e40 0x7f0ef00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0efffff640 1 -- 172.21.15.50:0/3881905720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53f40 msgr2=0x55f41ab5e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0efffff640 1 --2- 172.21.15.50:0/3881905720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53f40 0x55f41ab5e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0efffff640 1 -- 172.21.15.50:0/3881905720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53620 msgr2=0x55f41ab53a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0efffff640 1 --2- 172.21.15.50:0/3881905720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53620 0x55f41ab53a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0efffff640 1 -- 172.21.15.50:0/3881905720 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f41a7cb210 con 0x55f41ab5e800 2024-04-23T18:25:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.402+0000 7f0efffff640 1 --2- 172.21.15.50:0/3881905720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab60bf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0ee300 tx=0x7f0eec03b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8ffe8c8ee1bf6a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.403+0000 7f0eeaffd640 1 -- 172.21.15.50:0/3881905720 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec048020 con 0x55f41ab5e800 2024-04-23T18:25:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.403+0000 7f0f050fa640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0ef00a3e40 0x7f0ef00a6300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8005c40 tx=0x7f0ef8004ab0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.403+0000 7f0eeaffd640 1 -- 172.21.15.50:0/3881905720 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eec0c0f00 con 0x55f41ab5e800 2024-04-23T18:25:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.403+0000 7f0eeaffd640 1 -- 172.21.15.50:0/3881905720 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec0bce60 con 0x55f41ab5e800 2024-04-23T18:25:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.404+0000 7f0f0987a740 1 -- 172.21.15.50:0/3881905720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 msgr2=0x55f41ab60bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.404+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3881905720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab60bf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0ee300 tx=0x7f0eec03b790 comp rx=0 tx=0).stop 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.404+0000 7f0f0987a740 1 -- 172.21.15.50:0/3881905720 shutdown_connections 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.404+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3881905720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab60bf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.404+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3881905720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53f40 0x55f41ab5e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.404+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3881905720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53620 0x55f41ab53a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.404+0000 7f0f0987a740 1 -- 172.21.15.50:0/3881905720 >> 172.21.15.50:0/3881905720 conn(0x55f41ab4fb30 msgr2=0x55f41ab4ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.404+0000 7f0f0987a740 1 -- 172.21.15.50:0/3881905720 shutdown_connections 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.404+0000 7f0f0987a740 1 -- 172.21.15.50:0/3881905720 wait complete. 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.404+0000 7f0f0987a740 1 -- start start 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.404+0000 7f0f0987a740 1 Processor -- start 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f0987a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53620 0x55f41ab5e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f0987a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab5b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f0987a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab5b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f0987a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f41ab611f0 con 0x55f41ab53f40 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f0987a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f41aa530b0 con 0x55f41ab53620 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f0987a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f41ab61370 con 0x55f41ab5e800 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f0987a740 10 client.0 _ll_register_callbacks cb 0x55f41aa40d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0efffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab5b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f050fa640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab5b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f048f9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53620 0x55f41ab5e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f050fa640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab5b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:58324/0 (socket says 172.21.15.50:58324) 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0efffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab5b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42806/0 (socket says 172.21.15.50:42806) 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f050fa640 1 -- 172.21.15.50:0/3726505222 learned_addr learned my addr 172.21.15.50:0/3726505222 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f050fa640 1 -- 172.21.15.50:0/3726505222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53620 msgr2=0x55f41ab5e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f050fa640 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53620 0x55f41ab5e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f050fa640 1 -- 172.21.15.50:0/3726505222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 msgr2=0x55f41ab5b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f050fa640 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab5b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f050fa640 1 -- 172.21.15.50:0/3726505222 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f41a897620 con 0x55f41ab5e800 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f048f9640 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53620 0x55f41ab5e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0efffff640 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab5b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.405+0000 7f0f050fa640 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab5b9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800ff90 tx=0x7f0ef8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3caeb027eea4251b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0edffff640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0edffff640 1 -- 172.21.15.50:0/3726505222 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8004d20 con 0x55f41ab5e800 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0edffff640 1 -- 172.21.15.50:0/3726505222 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f0ef801e620 con 0x55f41ab5e800 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0f0987a740 1 -- 172.21.15.50:0/3726505222 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0eec0c0f00 con 0x55f41ab5e800 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0edffff640 1 -- 172.21.15.50:0/3726505222 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0ef801e820 con 0x55f41ab5e800 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0f0987a740 10 client.26026 fetch_fsmap learned FSMap version 33 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0f0987a740 1 -- 172.21.15.50:0/3726505222 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f41ab6d090 con 0x55f41ab5e800 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0edffff640 1 -- 172.21.15.50:0/3726505222 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eec0c0f00 con 0x55f41ab5e800 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0edffff640 1 -- 172.21.15.50:0/3726505222 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f0ef8027030 con 0x55f41ab5e800 2024-04-23T18:25:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0f0987a740 10 client.26026 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:25:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0f0987a740 10 client.26026 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0f0987a740 20 client.26026 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f41ab6e5a0 0x55f41ab70990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0f0987a740 4 client.26026 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.406+0000 7f0f0987a740 1 -- 172.21.15.50:0/3726505222 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f41aac6390 con 0x55f41ab6e5a0 2024-04-23T18:25:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.407+0000 7f0f048f9640 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f41ab6e5a0 0x55f41ab70990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.407+0000 7f0f048f9640 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f41ab6e5a0 0x55f41ab70990 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32a1feef8848f6f7 server_cookie=23c5a6b87b7342d3 in_seq=0 out_seq=0 2024-04-23T18:25:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.407+0000 7f0edffff640 10 client.26026 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0edffff640 1 -- 172.21.15.50:0/3726505222 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f41aac6390 con 0x55f41ab6e5a0 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0edffff640 10 client.26026 handle_command_reply: tid=0 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0f0987a740 1 client.26026 shutdown 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0f0987a740 1 -- 172.21.15.50:0/3726505222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 msgr2=0x55f41ab5b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab5b9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800ff90 tx=0x7f0ef8004a40 comp rx=0 tx=0).stop 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0f0987a740 1 -- 172.21.15.50:0/3726505222 shutdown_connections 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f41ab6e5a0 0x55f41ab70990 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab5b9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0edffff640 0 client.26026 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab5b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3726505222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53620 0x55f41ab5e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0f0987a740 1 -- 172.21.15.50:0/3726505222 >> 172.21.15.50:0/3726505222 conn(0x55f41ab4fb30 msgr2=0x55f41ab558b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0f0987a740 1 -- 172.21.15.50:0/3726505222 shutdown_connections 2024-04-23T18:25:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0f0987a740 1 -- 172.21.15.50:0/3726505222 wait complete. 2024-04-23T18:25:34.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.408+0000 7f0f0987a740 20 client.26026 trim_cache size 0 max 16384 2024-04-23T18:25:34.427 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f0987a740 1 Processor -- start 2024-04-23T18:25:34.427 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f0987a740 1 -- start start 2024-04-23T18:25:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f0987a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53620 0x55f41ab6a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f0987a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab4eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f0987a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab76260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f0987a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001b2640 con 0x55f41ab53f40 2024-04-23T18:25:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f0987a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001aff10 con 0x55f41ab53620 2024-04-23T18:25:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f0987a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001add80 con 0x55f41ab5e800 2024-04-23T18:25:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f050fa640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab76260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f048f9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53620 0x55f41ab6a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f050fa640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab76260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:58336/0 (socket says 172.21.15.50:58336) 2024-04-23T18:25:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f048f9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53620 0x55f41ab6a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:36224/0 (socket says 172.21.15.50:36224) 2024-04-23T18:25:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.434+0000 7f0f050fa640 1 -- 172.21.15.50:0/1425860126 learned_addr learned my addr 172.21.15.50:0/1425860126 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.435+0000 7f0efffff640 1 --2- 172.21.15.50:0/1425860126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab4eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.435+0000 7f0f050fa640 1 -- 172.21.15.50:0/1425860126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53620 msgr2=0x55f41ab6a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.435+0000 7f0f050fa640 1 --2- 172.21.15.50:0/1425860126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53620 0x55f41ab6a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.435+0000 7f0f050fa640 1 -- 172.21.15.50:0/1425860126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 msgr2=0x55f41ab4eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.435+0000 7f0f050fa640 1 --2- 172.21.15.50:0/1425860126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab4eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.435+0000 7f0f050fa640 1 -- 172.21.15.50:0/1425860126 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f41ab5bd10 con 0x55f41ab5e800 2024-04-23T18:25:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.435+0000 7f0f050fa640 1 --2- 172.21.15.50:0/1425860126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab76260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0ef801e3e0 tx=0x7f0ef80123e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4bc0072b29664f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.435+0000 7f0eea7fc640 1 -- 172.21.15.50:0/1425860126 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8005a10 con 0x55f41ab5e800 2024-04-23T18:25:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.435+0000 7f0eea7fc640 1 -- 172.21.15.50:0/1425860126 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ef8004b20 con 0x55f41ab5e800 2024-04-23T18:25:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.435+0000 7f0eea7fc640 1 -- 172.21.15.50:0/1425860126 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8030680 con 0x55f41ab5e800 2024-04-23T18:25:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.436+0000 7f0f0987a740 1 -- 172.21.15.50:0/1425860126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 msgr2=0x55f41ab76260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.436+0000 7f0f0987a740 1 --2- 172.21.15.50:0/1425860126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab76260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0ef801e3e0 tx=0x7f0ef80123e0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.436+0000 7f0f0987a740 1 -- 172.21.15.50:0/1425860126 shutdown_connections 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.436+0000 7f0f0987a740 1 --2- 172.21.15.50:0/1425860126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab5e800 0x55f41ab76260 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.436+0000 7f0f0987a740 1 --2- 172.21.15.50:0/1425860126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab4eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.436+0000 7f0f0987a740 1 --2- 172.21.15.50:0/1425860126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab53620 0x55f41ab6a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.436+0000 7f0f0987a740 1 -- 172.21.15.50:0/1425860126 >> 172.21.15.50:0/1425860126 conn(0x55f41ab4fb30 msgr2=0x55f41ab4e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.436+0000 7f0f0987a740 1 -- 172.21.15.50:0/1425860126 shutdown_connections 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.436+0000 7f0f0987a740 1 -- 172.21.15.50:0/1425860126 wait complete. 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.437+0000 7f0f0987a740 1 -- start start 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.437+0000 7f0f0987a740 1 Processor -- start 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.437+0000 7f0f0987a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab53620 0x55f41ab636f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.437+0000 7f0f0987a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab63c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.437+0000 7f0f0987a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab5e800 0x55f41abdb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f0987a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001ae3d0 con 0x55f41ab53f40 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f0987a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001af840 con 0x55f41ab5e800 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f0987a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001aeaa0 con 0x55f41ab53620 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f0987a740 10 client.0 _ll_register_callbacks cb 0x7f0ef8013f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f048f9640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab53620 0x55f41ab636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0efffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab63c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f048f9640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab53620 0x55f41ab636f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:58352/0 (socket says 172.21.15.50:58352) 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f048f9640 1 -- 172.21.15.50:0/3435162369 learned_addr learned my addr 172.21.15.50:0/3435162369 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f050fa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab5e800 0x55f41abdb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f050fa640 1 -- 172.21.15.50:0/3435162369 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab53620 msgr2=0x55f41ab636f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f050fa640 1 --2- 172.21.15.50:0/3435162369 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab53620 0x55f41ab636f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f050fa640 1 -- 172.21.15.50:0/3435162369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 msgr2=0x55f41ab63c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f050fa640 1 --2- 172.21.15.50:0/3435162369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab63c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f050fa640 1 -- 172.21.15.50:0/3435162369 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f41ab60f60 con 0x55f41ab5e800 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0efffff640 1 --2- 172.21.15.50:0/3435162369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab63c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0f050fa640 1 --2- 172.21.15.50:0/3435162369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab5e800 0x55f41abdb2e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8005620 tx=0x7f0ef8005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cab124453cdbaf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.438+0000 7f0edffff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0edffff640 1 -- 172.21.15.50:0/3435162369 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8010c90 con 0x55f41ab5e800 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0edffff640 1 -- 172.21.15.50:0/3435162369 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f0ef801ebd0 con 0x55f41ab5e800 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0f0987a740 1 -- 172.21.15.50:0/3435162369 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ef8004b20 con 0x55f41ab5e800 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0edffff640 1 -- 172.21.15.50:0/3435162369 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0ef801edd0 con 0x55f41ab5e800 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0f0987a740 10 client.25973 fetch_fsmap learned FSMap version 33 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0f0987a740 1 -- 172.21.15.50:0/3435162369 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f41abdc930 con 0x55f41ab5e800 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0edffff640 1 -- 172.21.15.50:0/3435162369 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ef8004b20 con 0x55f41ab5e800 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0edffff640 1 -- 172.21.15.50:0/3435162369 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f0ef803f080 con 0x55f41ab5e800 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0f0987a740 10 client.25973 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0f0987a740 10 client.25973 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0f0987a740 20 client.25973 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3435162369 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f41ab6e5a0 0x55f41abdef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0f0987a740 4 client.25973 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0f0987a740 1 -- 172.21.15.50:0/3435162369 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f41aac0200 con 0x55f41ab6e5a0 2024-04-23T18:25:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.439+0000 7f0f048f9640 1 --2- 172.21.15.50:0/3435162369 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f41ab6e5a0 0x55f41abdef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.440+0000 7f0edffff640 10 client.25973 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:34.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.440+0000 7f0f048f9640 1 --2- 172.21.15.50:0/3435162369 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f41ab6e5a0 0x55f41abdef40 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9e269eff7ac316b server_cookie=f8310723d4d74d20 in_seq=0 out_seq=0 2024-04-23T18:25:34.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:34 smithi050 ceph-mon[29187]: pgmap v218: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 208 KiB/s wr, 18 op/s 2024-04-23T18:25:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0edffff640 1 -- 172.21.15.50:0/3435162369 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f41aac0200 con 0x55f41ab6e5a0 2024-04-23T18:25:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0edffff640 10 client.25973 handle_command_reply: tid=0 2024-04-23T18:25:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0f0987a740 1 client.25973 shutdown 2024-04-23T18:25:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0f0987a740 1 -- 172.21.15.50:0/3435162369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab5e800 msgr2=0x55f41abdb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3435162369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab5e800 0x55f41abdb2e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8005620 tx=0x7f0ef8005a20 comp rx=0 tx=0).stop 2024-04-23T18:25:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0f0987a740 1 -- 172.21.15.50:0/3435162369 shutdown_connections 2024-04-23T18:25:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3435162369 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f41ab6e5a0 0x55f41abdef40 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3435162369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41ab5e800 0x55f41abdb2e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0edffff640 0 client.25973 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3435162369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f41ab53f40 0x55f41ab63c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0f0987a740 1 --2- 172.21.15.50:0/3435162369 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f41ab53620 0x55f41ab636f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0f0987a740 1 -- 172.21.15.50:0/3435162369 >> 172.21.15.50:0/3435162369 conn(0x55f41ab4fb30 msgr2=0x55f41ab4ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0f0987a740 1 -- 172.21.15.50:0/3435162369 shutdown_connections 2024-04-23T18:25:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0f0987a740 1 -- 172.21.15.50:0/3435162369 wait complete. 2024-04-23T18:25:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.467+0000 7f0f0987a740 20 client.25973 trim_cache size 0 max 16384 2024-04-23T18:25:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:25:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:25:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:25:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:25:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.468+0000 7f0f06b5d640 1 -- 172.21.15.50:0/3838160545 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0ef00a3e40 msgr2=0x7f0ef00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.468+0000 7f0f06b5d640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0ef00a3e40 0x7f0ef00a6300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8005c40 tx=0x7f0ef8004ab0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.468+0000 7f0f06b5d640 1 -- 172.21.15.50:0/3838160545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f00106170 msgr2=0x7f0f00199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.468+0000 7f0f06b5d640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f00106170 0x7f0f00199a10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0eec02a930 tx=0x7f0eec031a20 comp rx=0 tx=0).stop 2024-04-23T18:25:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.468+0000 7f0f06b5d640 1 -- 172.21.15.50:0/3838160545 shutdown_connections 2024-04-23T18:25:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.468+0000 7f0f06b5d640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0ef00a3e40 0x7f0ef00a6300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.468+0000 7f0f06b5d640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f00199ff0 0x7f0f001a4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.468+0000 7f0f06b5d640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f00106170 0x7f0f00199a10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.468+0000 7f0f06b5d640 1 --2- 172.21.15.50:0/3838160545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f001010f0 0x7f0f001994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.469+0000 7f0f06b5d640 1 -- 172.21.15.50:0/3838160545 >> 172.21.15.50:0/3838160545 conn(0x7f0f00077fd0 msgr2=0x7f0f000ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.469+0000 7f0f06b5d640 1 -- 172.21.15.50:0/3838160545 shutdown_connections 2024-04-23T18:25:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:34.469+0000 7f0f06b5d640 1 -- 172.21.15.50:0/3838160545 wait complete. 2024-04-23T18:25:34.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:34 smithi177 ceph-mon[32308]: pgmap v218: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 208 KiB/s wr, 18 op/s 2024-04-23T18:25:35.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 410958 4.68 MB/sec execute 249 sec latency 24.472 ms 2024-04-23T18:25:36.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 412655 4.70 MB/sec execute 250 sec latency 50.269 ms 2024-04-23T18:25:36.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:36 smithi188 ceph-mon[32694]: pgmap v219: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 178 KiB/s wr, 10 op/s 2024-04-23T18:25:36.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:36 smithi050 ceph-mon[29187]: pgmap v219: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 178 KiB/s wr, 10 op/s 2024-04-23T18:25:36.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:36 smithi177 ceph-mon[32308]: pgmap v219: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 178 KiB/s wr, 10 op/s 2024-04-23T18:25:37.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 414465 4.72 MB/sec execute 251 sec latency 16.720 ms 2024-04-23T18:25:38.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 416166 4.74 MB/sec execute 252 sec latency 23.514 ms 2024-04-23T18:25:38.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:38 smithi188 ceph-mon[32694]: pgmap v220: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 997 KiB/s wr, 116 op/s 2024-04-23T18:25:38.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:38 smithi050 ceph-mon[29187]: pgmap v220: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 997 KiB/s wr, 116 op/s 2024-04-23T18:25:38.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:38 smithi177 ceph-mon[32308]: pgmap v220: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 997 KiB/s wr, 116 op/s 2024-04-23T18:25:39.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 418011 4.76 MB/sec execute 253 sec latency 16.818 ms 2024-04-23T18:25:40.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 419656 4.77 MB/sec execute 254 sec latency 22.160 ms 2024-04-23T18:25:40.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:40 smithi188 ceph-mon[32694]: pgmap v221: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 823 KiB/s wr, 109 op/s 2024-04-23T18:25:40.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:40 smithi050 ceph-mon[29187]: pgmap v221: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 823 KiB/s wr, 109 op/s 2024-04-23T18:25:40.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:40 smithi177 ceph-mon[32308]: pgmap v221: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 823 KiB/s wr, 109 op/s 2024-04-23T18:25:41.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 422073 4.82 MB/sec execute 255 sec latency 13.794 ms 2024-04-23T18:25:42.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 423567 4.82 MB/sec execute 256 sec latency 64.081 ms 2024-04-23T18:25:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:42 smithi050 ceph-mon[29187]: pgmap v222: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 823 KiB/s wr, 109 op/s 2024-04-23T18:25:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:42 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:25:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:42 smithi177 ceph-mon[32308]: pgmap v222: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 823 KiB/s wr, 109 op/s 2024-04-23T18:25:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:42 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:25:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:42 smithi188 ceph-mon[32694]: pgmap v222: 97 pgs: 97 active+clean; 126 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 823 KiB/s wr, 109 op/s 2024-04-23T18:25:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:42 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:25:43.214 INFO:tasks.workunit.client.0.smithi050.stdout: 1 425737 4.85 MB/sec execute 257 sec latency 17.499 ms 2024-04-23T18:25:43.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:43 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:25:43.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:43 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:25:43.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:43 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:25:43.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:43 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:25:43.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:43 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:25:43.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:43 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:25:43.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:43 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:25:43.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:43 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:25:43.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:43 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:25:44.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 427168 4.86 MB/sec execute 258 sec latency 24.625 ms 2024-04-23T18:25:44.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:44 smithi050 ceph-mon[29187]: pgmap v223: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 200 op/s 2024-04-23T18:25:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:44 smithi177 ceph-mon[32308]: pgmap v223: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 200 op/s 2024-04-23T18:25:44.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:44 smithi188 ceph-mon[32694]: pgmap v223: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 200 op/s 2024-04-23T18:25:45.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 429335 4.89 MB/sec execute 259 sec latency 28.707 ms 2024-04-23T18:25:46.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 430914 4.90 MB/sec execute 260 sec latency 20.945 ms 2024-04-23T18:25:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:46 smithi050 ceph-mon[29187]: pgmap v224: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 196 op/s 2024-04-23T18:25:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:46 smithi177 ceph-mon[32308]: pgmap v224: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 196 op/s 2024-04-23T18:25:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:46 smithi188 ceph-mon[32694]: pgmap v224: 97 pgs: 97 active+clean; 133 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 196 op/s 2024-04-23T18:25:47.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 433044 4.93 MB/sec execute 261 sec latency 127.548 ms 2024-04-23T18:25:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.563+0000 7f02ebe7c640 1 -- 172.21.15.50:0/1778624023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02e410c520 msgr2=0x7f02e410e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.563+0000 7f02ebe7c640 1 --2- 172.21.15.50:0/1778624023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02e410c520 0x7f02e410e9c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009900 tx=0x7f02dc02a510 comp rx=0 tx=0).stop 2024-04-23T18:25:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.563+0000 7f02ebe7c640 1 -- 172.21.15.50:0/1778624023 shutdown_connections 2024-04-23T18:25:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.563+0000 7f02ebe7c640 1 --2- 172.21.15.50:0/1778624023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02e410c520 0x7f02e410e9c0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.563+0000 7f02ebe7c640 1 --2- 172.21.15.50:0/1778624023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e4103b00 0x7f02e410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.563+0000 7f02ebe7c640 1 --2- 172.21.15.50:0/1778624023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02e41031e0 0x7f02e41035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.563+0000 7f02ebe7c640 1 -- 172.21.15.50:0/1778624023 >> 172.21.15.50:0/1778624023 conn(0x7f02e4077fd0 msgr2=0x7f02e40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.563+0000 7f02ebe7c640 1 -- 172.21.15.50:0/1778624023 shutdown_connections 2024-04-23T18:25:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.563+0000 7f02ebe7c640 1 -- 172.21.15.50:0/1778624023 wait complete. 2024-04-23T18:25:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02ebe7c640 1 Processor -- start 2024-04-23T18:25:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02ebe7c640 1 -- start start 2024-04-23T18:25:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02ebe7c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02e41031e0 0x7f02e41986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02ebe7c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02e4103b00 0x7f02e4198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02ebe7c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e410c520 0x7f02e419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02ebe7c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f02e4112190 con 0x7f02e4103b00 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02ebe7c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02e4112010 con 0x7f02e410c520 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02ebe7c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f02e4112310 con 0x7f02e41031e0 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02e9417640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02e4103b00 0x7f02e4198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02e9417640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02e4103b00 0x7f02e4198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35884/0 (socket says 172.21.15.50:35884) 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02e9417640 1 -- 172.21.15.50:0/825206422 learned_addr learned my addr 172.21.15.50:0/825206422 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02ea419640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e410c520 0x7f02e419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.564+0000 7f02e9c18640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02e41031e0 0x7f02e41986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.565+0000 7f02e9417640 1 -- 172.21.15.50:0/825206422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02e41031e0 msgr2=0x7f02e41986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.565+0000 7f02e9417640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02e41031e0 0x7f02e41986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.565+0000 7f02e9417640 1 -- 172.21.15.50:0/825206422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e410c520 msgr2=0x7f02e419cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.565+0000 7f02e9417640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e410c520 0x7f02e419cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.565+0000 7f02e9417640 1 -- 172.21.15.50:0/825206422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e419d670 con 0x7f02e4103b00 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.565+0000 7f02e9417640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02e4103b00 0x7f02e4198c10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f02d800ea60 tx=0x7f02d800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.565+0000 7f02e2ffd640 1 -- 172.21.15.50:0/825206422 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d800cde0 con 0x7f02e4103b00 2024-04-23T18:25:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.565+0000 7f02ebe7c640 1 -- 172.21.15.50:0/825206422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e419d960 con 0x7f02e4103b00 2024-04-23T18:25:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.565+0000 7f02e2ffd640 1 -- 172.21.15.50:0/825206422 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02d80035f0 con 0x7f02e4103b00 2024-04-23T18:25:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.565+0000 7f02e2ffd640 1 -- 172.21.15.50:0/825206422 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d8010690 con 0x7f02e4103b00 2024-04-23T18:25:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.565+0000 7f02ebe7c640 1 -- 172.21.15.50:0/825206422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e41a51f0 con 0x7f02e4103b00 2024-04-23T18:25:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.567+0000 7f02e2ffd640 1 -- 172.21.15.50:0/825206422 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f02d80037a0 con 0x7f02e4103b00 2024-04-23T18:25:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02eeb98740 1 Processor -- start 2024-04-23T18:25:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02eeb98740 1 -- start start 2024-04-23T18:25:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02eeb98740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2620 0x55b4f5de2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02eeb98740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 0x55b4f5ded2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02eeb98740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5ded800 0x55b4f5defbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02eeb98740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b4f5d47970 con 0x55b4f5de2f40 2024-04-23T18:25:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02eeb98740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b4f5ce44a0 con 0x55b4f5de2620 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02eeb98740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b4f5d43930 con 0x55b4f5ded800 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02e9c18640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 0x55b4f5ded2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02ea419640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2620 0x55b4f5de2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02ea419640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2620 0x55b4f5de2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:48490/0 (socket says 172.21.15.50:48490) 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02e2ffd640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f02cc0a3dc0 0x7f02cc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02ea419640 1 -- 172.21.15.50:0/3612409747 learned_addr learned my addr 172.21.15.50:0/3612409747 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02e9c18640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 0x55b4f5ded2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35892/0 (socket says 172.21.15.50:35892) 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02e9417640 1 --2- 172.21.15.50:0/3612409747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5ded800 0x55b4f5defbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.568+0000 7f02ea419640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f02cc0a3dc0 0x7f02cc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.569+0000 7f02e9c18640 1 -- 172.21.15.50:0/3612409747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5ded800 msgr2=0x55b4f5defbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.569+0000 7f02e9c18640 1 --2- 172.21.15.50:0/3612409747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5ded800 0x55b4f5defbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.569+0000 7f02e9c18640 1 -- 172.21.15.50:0/3612409747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2620 msgr2=0x55b4f5de2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.569+0000 7f02e9c18640 1 --2- 172.21.15.50:0/3612409747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2620 0x55b4f5de2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.569+0000 7f02e9c18640 1 -- 172.21.15.50:0/3612409747 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4f5a5a210 con 0x55b4f5de2f40 2024-04-23T18:25:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.569+0000 7f02ea419640 1 --2- 172.21.15.50:0/3612409747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2620 0x55b4f5de2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.569+0000 7f02e2ffd640 1 -- 172.21.15.50:0/825206422 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f02d80d0a70 con 0x7f02e4103b00 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.569+0000 7f02e9c18640 1 --2- 172.21.15.50:0/3612409747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 0x55b4f5ded2c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f02d4009fa0 tx=0x7f02d40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afd7b80e37a09cf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.569+0000 7f02ea419640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f02cc0a3dc0 0x7f02cc0a6280 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f02dc005c40 tx=0x7f02dc02a510 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.569+0000 7f02cbfff640 1 -- 172.21.15.50:0/3612409747 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d4003c30 con 0x55b4f5de2f40 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.569+0000 7f02cbfff640 1 -- 172.21.15.50:0/3612409747 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02d4003dd0 con 0x55b4f5de2f40 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.569+0000 7f02cbfff640 1 -- 172.21.15.50:0/3612409747 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d400fa30 con 0x55b4f5de2f40 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.570+0000 7f02eeb98740 1 -- 172.21.15.50:0/3612409747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 msgr2=0x55b4f5ded2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.570+0000 7f02eeb98740 1 --2- 172.21.15.50:0/3612409747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 0x55b4f5ded2c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f02d4009fa0 tx=0x7f02d40118d0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.570+0000 7f02eeb98740 1 -- 172.21.15.50:0/3612409747 shutdown_connections 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.570+0000 7f02eeb98740 1 --2- 172.21.15.50:0/3612409747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5ded800 0x55b4f5defbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.570+0000 7f02eeb98740 1 --2- 172.21.15.50:0/3612409747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 0x55b4f5ded2c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.570+0000 7f02eeb98740 1 --2- 172.21.15.50:0/3612409747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2620 0x55b4f5de2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.570+0000 7f02eeb98740 1 -- 172.21.15.50:0/3612409747 >> 172.21.15.50:0/3612409747 conn(0x55b4f5ddeb30 msgr2=0x55b4f5ddef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.570+0000 7f02eeb98740 1 -- 172.21.15.50:0/3612409747 shutdown_connections 2024-04-23T18:25:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.570+0000 7f02eeb98740 1 -- 172.21.15.50:0/3612409747 wait complete. 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.570+0000 7f02eeb98740 1 -- start start 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.570+0000 7f02eeb98740 1 Processor -- start 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02eeb98740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2620 0x55b4f5ded1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02eeb98740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2f40 0x55b4f5dea4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02eeb98740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5ded800 0x55b4f5deaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02eeb98740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b4f5df0430 con 0x55b4f5de2620 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02eeb98740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b4f5ce20b0 con 0x55b4f5ded800 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02eeb98740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b4f5df05b0 con 0x55b4f5de2f40 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02eeb98740 10 client.0 _ll_register_callbacks cb 0x55b4f5ccfd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02e9417640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2f40 0x55b4f5dea4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02ea419640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5ded800 0x55b4f5deaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02e9417640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2f40 0x55b4f5dea4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:48070/0 (socket says 172.21.15.50:48070) 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02e9c18640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2620 0x55b4f5ded1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02e9417640 1 -- 172.21.15.50:0/1107237209 learned_addr learned my addr 172.21.15.50:0/1107237209 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02e9417640 1 -- 172.21.15.50:0/1107237209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5ded800 msgr2=0x55b4f5deaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02e9417640 1 --2- 172.21.15.50:0/1107237209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5ded800 0x55b4f5deaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02e9417640 1 -- 172.21.15.50:0/1107237209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2620 msgr2=0x55b4f5ded1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02e9417640 1 --2- 172.21.15.50:0/1107237209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2620 0x55b4f5ded1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02e9417640 1 -- 172.21.15.50:0/1107237209 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4f5b26620 con 0x55b4f5de2f40 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02e9417640 1 --2- 172.21.15.50:0/1107237209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2f40 0x55b4f5dea4c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f02d8098e60 tx=0x7f02d80bd050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd20789f891fefea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.571+0000 7f02c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02c97fa640 1 -- 172.21.15.50:0/1107237209 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d809c360 con 0x55b4f5de2f40 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02c97fa640 1 -- 172.21.15.50:0/1107237209 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f02d80a9cd0 con 0x55b4f5de2f40 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02eeb98740 1 -- 172.21.15.50:0/1107237209 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02d4003dd0 con 0x55b4f5de2f40 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02c97fa640 1 -- 172.21.15.50:0/1107237209 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f02d809d100 con 0x55b4f5de2f40 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02eeb98740 10 client.26044 fetch_fsmap learned FSMap version 33 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02eeb98740 1 -- 172.21.15.50:0/1107237209 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b4f5dfc420 con 0x55b4f5de2f40 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02c97fa640 1 -- 172.21.15.50:0/1107237209 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02d80b7030 con 0x55b4f5de2f40 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02c97fa640 1 -- 172.21.15.50:0/1107237209 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f02d80b9070 con 0x55b4f5de2f40 2024-04-23T18:25:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02eeb98740 10 client.26044 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:25:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02eeb98740 10 client.26044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02eeb98740 20 client.26044 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02eeb98740 1 --2- 172.21.15.50:0/1107237209 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b4f5dfd890 0x55b4f5dffd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02eeb98740 4 client.26044 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:25:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.572+0000 7f02eeb98740 1 -- 172.21.15.50:0/1107237209 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4f5d55390 con 0x55b4f5dfd890 2024-04-23T18:25:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.573+0000 7f02e9c18640 1 --2- 172.21.15.50:0/1107237209 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b4f5dfd890 0x55b4f5dffd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.573+0000 7f02e9c18640 1 --2- 172.21.15.50:0/1107237209 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b4f5dfd890 0x55b4f5dffd00 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81efa54bb0ddbb26 server_cookie=fe303ff12aaee72e in_seq=0 out_seq=0 2024-04-23T18:25:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.573+0000 7f02c97fa640 10 client.26044 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:47.572 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.574+0000 7f02c97fa640 1 -- 172.21.15.50:0/1107237209 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b4f5d55390 con 0x55b4f5dfd890 2024-04-23T18:25:47.572 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.574+0000 7f02c97fa640 10 client.26044 handle_command_reply: tid=0 2024-04-23T18:25:47.572 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.574+0000 7f02eeb98740 1 client.26044 shutdown 2024-04-23T18:25:47.572 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.574+0000 7f02eeb98740 1 -- 172.21.15.50:0/1107237209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2f40 msgr2=0x55b4f5dea4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:47.572 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.574+0000 7f02eeb98740 1 --2- 172.21.15.50:0/1107237209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2f40 0x55b4f5dea4c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f02d8098e60 tx=0x7f02d80bd050 comp rx=0 tx=0).stop 2024-04-23T18:25:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.575+0000 7f02eeb98740 1 -- 172.21.15.50:0/1107237209 shutdown_connections 2024-04-23T18:25:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.575+0000 7f02eeb98740 1 --2- 172.21.15.50:0/1107237209 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b4f5dfd890 0x55b4f5dffd00 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.575+0000 7f02eeb98740 1 --2- 172.21.15.50:0/1107237209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5ded800 0x55b4f5deaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.575+0000 7f02eeb98740 1 --2- 172.21.15.50:0/1107237209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2f40 0x55b4f5dea4c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.575+0000 7f02c97fa640 0 client.26044 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.575+0000 7f02eeb98740 1 --2- 172.21.15.50:0/1107237209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2620 0x55b4f5ded1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.575+0000 7f02eeb98740 1 -- 172.21.15.50:0/1107237209 >> 172.21.15.50:0/1107237209 conn(0x55b4f5ddeb30 msgr2=0x55b4f5de48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.575+0000 7f02eeb98740 1 -- 172.21.15.50:0/1107237209 shutdown_connections 2024-04-23T18:25:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.575+0000 7f02eeb98740 1 -- 172.21.15.50:0/1107237209 wait complete. 2024-04-23T18:25:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.575+0000 7f02eeb98740 20 client.26044 trim_cache size 0 max 16384 2024-04-23T18:25:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02eeb98740 1 Processor -- start 2024-04-23T18:25:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02eeb98740 1 -- start start 2024-04-23T18:25:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02eeb98740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2620 0x55b4f5df92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02eeb98740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 0x55b4f5e05610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02eeb98740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5ded800 0x55b4f5e05b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02eeb98740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f02e41ae2d0 con 0x55b4f5de2f40 2024-04-23T18:25:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02eeb98740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02e41abba0 con 0x55b4f5de2620 2024-04-23T18:25:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02eeb98740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f02e41a9a10 con 0x55b4f5ded800 2024-04-23T18:25:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02e9417640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 0x55b4f5e05610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02e9c18640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2620 0x55b4f5df92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02e9417640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 0x55b4f5e05610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35910/0 (socket says 172.21.15.50:35910) 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02e9417640 1 -- 172.21.15.50:0/2641201212 learned_addr learned my addr 172.21.15.50:0/2641201212 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02ea419640 1 --2- 172.21.15.50:0/2641201212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5ded800 0x55b4f5e05b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02e9417640 1 -- 172.21.15.50:0/2641201212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5ded800 msgr2=0x55b4f5e05b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02e9417640 1 --2- 172.21.15.50:0/2641201212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5ded800 0x55b4f5e05b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02e9417640 1 -- 172.21.15.50:0/2641201212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2620 msgr2=0x55b4f5df92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02e9417640 1 --2- 172.21.15.50:0/2641201212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2620 0x55b4f5df92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.592+0000 7f02e9417640 1 -- 172.21.15.50:0/2641201212 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4f5de35b0 con 0x55b4f5de2f40 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.593+0000 7f02e9417640 1 --2- 172.21.15.50:0/2641201212 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 0x55b4f5e05610 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f02d800ea30 tx=0x7f02d8003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a604761845c3573 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.593+0000 7f02cb7fe640 1 -- 172.21.15.50:0/2641201212 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d809c360 con 0x55b4f5de2f40 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.593+0000 7f02cb7fe640 1 -- 172.21.15.50:0/2641201212 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02d80b7070 con 0x55b4f5de2f40 2024-04-23T18:25:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.593+0000 7f02cb7fe640 1 -- 172.21.15.50:0/2641201212 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d8098ca0 con 0x55b4f5de2f40 2024-04-23T18:25:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.593+0000 7f02eeb98740 1 -- 172.21.15.50:0/2641201212 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 msgr2=0x55b4f5e05610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.593+0000 7f02eeb98740 1 --2- 172.21.15.50:0/2641201212 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 0x55b4f5e05610 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f02d800ea30 tx=0x7f02d8003e90 comp rx=0 tx=0).stop 2024-04-23T18:25:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.593+0000 7f02eeb98740 1 -- 172.21.15.50:0/2641201212 shutdown_connections 2024-04-23T18:25:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.593+0000 7f02eeb98740 1 --2- 172.21.15.50:0/2641201212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5ded800 0x55b4f5e05b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.593+0000 7f02eeb98740 1 --2- 172.21.15.50:0/2641201212 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5de2f40 0x55b4f5e05610 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.593+0000 7f02eeb98740 1 --2- 172.21.15.50:0/2641201212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2620 0x55b4f5df92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.593+0000 7f02eeb98740 1 -- 172.21.15.50:0/2641201212 >> 172.21.15.50:0/2641201212 conn(0x55b4f5ddeb30 msgr2=0x55b4f5ddef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.594+0000 7f02eeb98740 1 -- 172.21.15.50:0/2641201212 shutdown_connections 2024-04-23T18:25:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.594+0000 7f02eeb98740 1 -- 172.21.15.50:0/2641201212 wait complete. 2024-04-23T18:25:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.594+0000 7f02eeb98740 1 -- start start 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.594+0000 7f02eeb98740 1 Processor -- start 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.594+0000 7f02eeb98740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2620 0x55b4f5df3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.594+0000 7f02eeb98740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2f40 0x55b4f5df3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.594+0000 7f02eeb98740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5ded800 0x55b4f5df3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.594+0000 7f02eeb98740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f02e41aa060 con 0x55b4f5ded800 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.594+0000 7f02eeb98740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02e41ab4d0 con 0x55b4f5de2f40 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.594+0000 7f02eeb98740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f02e41aa730 con 0x55b4f5de2620 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02eeb98740 10 client.0 _ll_register_callbacks cb 0x7f02d809c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02ea419640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5ded800 0x55b4f5df3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02e9417640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2f40 0x55b4f5df3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02e9c18640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2620 0x55b4f5df3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02e9417640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2f40 0x55b4f5df3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:48536/0 (socket says 172.21.15.50:48536) 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02e9c18640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2620 0x55b4f5df3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:48086/0 (socket says 172.21.15.50:48086) 2024-04-23T18:25:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02e9417640 1 -- 172.21.15.50:0/3368567287 learned_addr learned my addr 172.21.15.50:0/3368567287 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02ea419640 1 -- 172.21.15.50:0/3368567287 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2620 msgr2=0x55b4f5df3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02ea419640 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2620 0x55b4f5df3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02ea419640 1 -- 172.21.15.50:0/3368567287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2f40 msgr2=0x55b4f5df3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02ea419640 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2f40 0x55b4f5df3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02ea419640 1 -- 172.21.15.50:0/3368567287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4f5e62f60 con 0x55b4f5ded800 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02e9c18640 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2620 0x55b4f5df3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02e9417640 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2f40 0x55b4f5df3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02ea419640 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5ded800 0x55b4f5df3d80 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f02dc031440 tx=0x7f02dc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a57f3f29f655bd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02c97fa640 1 -- 172.21.15.50:0/3368567287 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc035040 con 0x55b4f5ded800 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02c97fa640 1 -- 172.21.15.50:0/3368567287 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f02dc0043e0 con 0x55b4f5ded800 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02eeb98740 1 -- 172.21.15.50:0/3368567287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02d80b7070 con 0x55b4f5ded800 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.595+0000 7f02c97fa640 1 -- 172.21.15.50:0/3368567287 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f02dc05c050 con 0x55b4f5ded800 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.596+0000 7f02eeb98740 10 client.16416 fetch_fsmap learned FSMap version 33 2024-04-23T18:25:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.596+0000 7f02eeb98740 1 -- 172.21.15.50:0/3368567287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b4f5e64250 con 0x55b4f5ded800 2024-04-23T18:25:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.596+0000 7f02c97fa640 1 -- 172.21.15.50:0/3368567287 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02d80b7070 con 0x55b4f5ded800 2024-04-23T18:25:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.596+0000 7f02c97fa640 1 -- 172.21.15.50:0/3368567287 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f02dc004c60 con 0x55b4f5ded800 2024-04-23T18:25:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.596+0000 7f02eeb98740 10 client.16416 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:25:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.596+0000 7f02eeb98740 10 client.16416 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:25:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.596+0000 7f02eeb98740 20 client.16416 populate_metadata read hostname 'smithi050' 2024-04-23T18:25:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.596+0000 7f02eeb98740 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b4f5e649f0 0x55b4f5e66e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:25:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.596+0000 7f02eeb98740 4 client.16416 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:25:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.596+0000 7f02eeb98740 1 -- 172.21.15.50:0/3368567287 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b4f5d4f200 con 0x55b4f5e649f0 2024-04-23T18:25:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.596+0000 7f02e9c18640 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b4f5e649f0 0x55b4f5e66e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:25:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.597+0000 7f02e9c18640 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b4f5e649f0 0x55b4f5e66e10 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d52a4a82427ed11 server_cookie=3b821c93483c7d7a in_seq=0 out_seq=0 2024-04-23T18:25:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.597+0000 7f02c97fa640 10 client.16416 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02c97fa640 1 -- 172.21.15.50:0/3368567287 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b4f5d4f200 con 0x55b4f5e649f0 2024-04-23T18:25:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02c97fa640 10 client.16416 handle_command_reply: tid=0 2024-04-23T18:25:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02eeb98740 1 client.16416 shutdown 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02eeb98740 1 -- 172.21.15.50:0/3368567287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5ded800 msgr2=0x55b4f5df3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02eeb98740 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5ded800 0x55b4f5df3d80 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f02dc031440 tx=0x7f02dc009260 comp rx=0 tx=0).stop 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02eeb98740 1 -- 172.21.15.50:0/3368567287 shutdown_connections 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02eeb98740 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b4f5e649f0 0x55b4f5e66e10 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02eeb98740 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4f5ded800 0x55b4f5df3d80 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02eeb98740 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4f5de2f40 0x55b4f5df3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02eeb98740 1 --2- 172.21.15.50:0/3368567287 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4f5de2620 0x55b4f5df3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02eeb98740 1 -- 172.21.15.50:0/3368567287 >> 172.21.15.50:0/3368567287 conn(0x55b4f5ddeb30 msgr2=0x55b4f5de8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02c97fa640 0 client.16416 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02eeb98740 1 -- 172.21.15.50:0/3368567287 shutdown_connections 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02eeb98740 1 -- 172.21.15.50:0/3368567287 wait complete. 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.658+0000 7f02eeb98740 20 client.16416 trim_cache size 0 max 16384 2024-04-23T18:25:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:25:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:25:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:25:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:25:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.659+0000 7f02ebe7c640 1 -- 172.21.15.50:0/825206422 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f02cc0a3dc0 msgr2=0x7f02cc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.659+0000 7f02ebe7c640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f02cc0a3dc0 0x7f02cc0a6280 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f02dc005c40 tx=0x7f02dc02a510 comp rx=0 tx=0).stop 2024-04-23T18:25:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.659+0000 7f02ebe7c640 1 -- 172.21.15.50:0/825206422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02e4103b00 msgr2=0x7f02e4198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:25:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.659+0000 7f02ebe7c640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02e4103b00 0x7f02e4198c10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f02d800ea60 tx=0x7f02d800ef20 comp rx=0 tx=0).stop 2024-04-23T18:25:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.659+0000 7f02ebe7c640 1 -- 172.21.15.50:0/825206422 shutdown_connections 2024-04-23T18:25:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.659+0000 7f02ebe7c640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f02cc0a3dc0 0x7f02cc0a6280 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.659+0000 7f02ebe7c640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e410c520 0x7f02e419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.659+0000 7f02ebe7c640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02e4103b00 0x7f02e4198c10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.659+0000 7f02ebe7c640 1 --2- 172.21.15.50:0/825206422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02e41031e0 0x7f02e41986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:25:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.659+0000 7f02ebe7c640 1 -- 172.21.15.50:0/825206422 >> 172.21.15.50:0/825206422 conn(0x7f02e4077fd0 msgr2=0x7f02e40fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:25:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.659+0000 7f02ebe7c640 1 -- 172.21.15.50:0/825206422 shutdown_connections 2024-04-23T18:25:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:25:47.659+0000 7f02ebe7c640 1 -- 172.21.15.50:0/825206422 wait complete. 2024-04-23T18:25:48.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 434275 4.93 MB/sec execute 262 sec latency 85.680 ms 2024-04-23T18:25:48.338 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:48 smithi177 ceph-mon[32308]: pgmap v225: 97 pgs: 97 active+clean; 141 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 2.3 MiB/s wr, 287 op/s 2024-04-23T18:25:48.380 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:48 smithi188 ceph-mon[32694]: pgmap v225: 97 pgs: 97 active+clean; 141 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 2.3 MiB/s wr, 287 op/s 2024-04-23T18:25:48.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:48 smithi050 ceph-mon[29187]: pgmap v225: 97 pgs: 97 active+clean; 141 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 2.3 MiB/s wr, 287 op/s 2024-04-23T18:25:49.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 436580 4.97 MB/sec execute 263 sec latency 15.474 ms 2024-04-23T18:25:50.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 437892 4.97 MB/sec execute 264 sec latency 23.358 ms 2024-04-23T18:25:50.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:50 smithi050 ceph-mon[29187]: pgmap v226: 97 pgs: 97 active+clean; 141 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-04-23T18:25:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:50 smithi177 ceph-mon[32308]: pgmap v226: 97 pgs: 97 active+clean; 141 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-04-23T18:25:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:50 smithi188 ceph-mon[32694]: pgmap v226: 97 pgs: 97 active+clean; 141 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-04-23T18:25:51.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 440127 5.00 MB/sec execute 265 sec latency 16.242 ms 2024-04-23T18:25:52.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 441630 5.00 MB/sec execute 266 sec latency 24.898 ms 2024-04-23T18:25:52.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:52 smithi050 ceph-mon[29187]: pgmap v227: 97 pgs: 97 active+clean; 141 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-04-23T18:25:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:52 smithi177 ceph-mon[32308]: pgmap v227: 97 pgs: 97 active+clean; 141 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-04-23T18:25:52.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:52 smithi188 ceph-mon[32694]: pgmap v227: 97 pgs: 97 active+clean; 141 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.5 MiB/s wr, 181 op/s 2024-04-23T18:25:53.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 443788 5.04 MB/sec execute 267 sec latency 17.531 ms 2024-04-23T18:25:54.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 445099 5.04 MB/sec execute 268 sec latency 26.241 ms 2024-04-23T18:25:54.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:54 smithi050 ceph-mon[29187]: pgmap v228: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.2 MiB/s wr, 310 op/s 2024-04-23T18:25:54.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:54 smithi177 ceph-mon[32308]: pgmap v228: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.2 MiB/s wr, 310 op/s 2024-04-23T18:25:54.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:54 smithi188 ceph-mon[32694]: pgmap v228: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.2 MiB/s wr, 310 op/s 2024-04-23T18:25:55.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 447349 5.07 MB/sec execute 269 sec latency 14.882 ms 2024-04-23T18:25:56.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 449087 5.07 MB/sec execute 270 sec latency 20.478 ms 2024-04-23T18:25:56.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:56 smithi050 ceph-mon[29187]: pgmap v229: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 219 op/s 2024-04-23T18:25:56.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:56 smithi177 ceph-mon[32308]: pgmap v229: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 219 op/s 2024-04-23T18:25:56.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:56 smithi188 ceph-mon[32694]: pgmap v229: 97 pgs: 97 active+clean; 136 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 219 op/s 2024-04-23T18:25:57.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 451000 5.10 MB/sec execute 271 sec latency 12.712 ms 2024-04-23T18:25:58.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 452791 5.11 MB/sec execute 272 sec latency 19.191 ms 2024-04-23T18:25:58.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:25:58 smithi050 ceph-mon[29187]: pgmap v230: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.3 MiB/s wr, 312 op/s 2024-04-23T18:25:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:25:58 smithi177 ceph-mon[32308]: pgmap v230: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.3 MiB/s wr, 312 op/s 2024-04-23T18:25:58.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:25:58 smithi188 ceph-mon[32694]: pgmap v230: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.3 MiB/s wr, 312 op/s 2024-04-23T18:25:59.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 454702 5.14 MB/sec execute 273 sec latency 22.026 ms 2024-04-23T18:26:00.215 INFO:tasks.workunit.client.0.smithi050.stdout: 1 456536 5.14 MB/sec execute 274 sec latency 14.433 ms 2024-04-23T18:26:00.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:00 smithi050 ceph-mon[29187]: pgmap v231: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 221 op/s 2024-04-23T18:26:00.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:00 smithi177 ceph-mon[32308]: pgmap v231: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 221 op/s 2024-04-23T18:26:00.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:00 smithi188 ceph-mon[32694]: pgmap v231: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 221 op/s 2024-04-23T18:26:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.757+0000 7fe556508640 1 -- 172.21.15.50:0/3443879724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550107fb0 msgr2=0x7fe550108390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.757+0000 7fe556508640 1 --2- 172.21.15.50:0/3443879724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550107fb0 0x7fe550108390 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe540009900 tx=0x7fe54002a410 comp rx=0 tx=0).stop 2024-04-23T18:26:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.757+0000 7fe556508640 1 -- 172.21.15.50:0/3443879724 shutdown_connections 2024-04-23T18:26:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.757+0000 7fe556508640 1 --2- 172.21.15.50:0/3443879724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55006b140 0x7fe55010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.757+0000 7fe556508640 1 --2- 172.21.15.50:0/3443879724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe55006a7c0 0x7fe55006ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.757+0000 7fe556508640 1 --2- 172.21.15.50:0/3443879724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550107fb0 0x7fe550108390 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.757+0000 7fe556508640 1 -- 172.21.15.50:0/3443879724 >> 172.21.15.50:0/3443879724 conn(0x7fe550076c50 msgr2=0x7fe550077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:00.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.758+0000 7fe556508640 1 -- 172.21.15.50:0/3443879724 shutdown_connections 2024-04-23T18:26:00.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.758+0000 7fe556508640 1 -- 172.21.15.50:0/3443879724 wait complete. 2024-04-23T18:26:00.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.758+0000 7fe556508640 1 Processor -- start 2024-04-23T18:26:00.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.758+0000 7fe556508640 1 -- start start 2024-04-23T18:26:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe556508640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe55006a7c0 0x7fe550198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe556508640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55006b140 0x7fe550198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe556508640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550107fb0 0x7fe55019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe556508640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe550112130 con 0x7fe55006a7c0 2024-04-23T18:26:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe556508640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe550111fb0 con 0x7fe550107fb0 2024-04-23T18:26:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe556508640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501122b0 con 0x7fe55006b140 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe55006a7c0 0x7fe550198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe55006a7c0 0x7fe550198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51726/0 (socket says 172.21.15.50:51726) 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe54ffff640 1 -- 172.21.15.50:0/2453078219 learned_addr learned my addr 172.21.15.50:0/2453078219 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe554aa5640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550107fb0 0x7fe55019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55006b140 0x7fe550198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe554aa5640 1 -- 172.21.15.50:0/2453078219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55006b140 msgr2=0x7fe550198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe554aa5640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55006b140 0x7fe550198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe554aa5640 1 -- 172.21.15.50:0/2453078219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe55006a7c0 msgr2=0x7fe550198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe554aa5640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe55006a7c0 0x7fe550198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe554aa5640 1 -- 172.21.15.50:0/2453078219 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55019d620 con 0x7fe550107fb0 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe54ffff640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe55006a7c0 0x7fe550198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.759+0000 7fe554aa5640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550107fb0 0x7fe55019cef0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe548005fd0 tx=0x7fe54800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.760+0000 7fe54d7fa640 1 -- 172.21.15.50:0/2453078219 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54800cd80 con 0x7fe550107fb0 2024-04-23T18:26:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.760+0000 7fe556508640 1 -- 172.21.15.50:0/2453078219 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe55019d910 con 0x7fe550107fb0 2024-04-23T18:26:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.760+0000 7fe54d7fa640 1 -- 172.21.15.50:0/2453078219 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe548005020 con 0x7fe550107fb0 2024-04-23T18:26:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.760+0000 7fe556508640 1 -- 172.21.15.50:0/2453078219 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5501a51a0 con 0x7fe550107fb0 2024-04-23T18:26:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.760+0000 7fe54d7fa640 1 -- 172.21.15.50:0/2453078219 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe548010690 con 0x7fe550107fb0 2024-04-23T18:26:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.762+0000 7fe54d7fa640 1 -- 172.21.15.50:0/2453078219 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe5480054d0 con 0x7fe550107fb0 2024-04-23T18:26:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe559225740 1 Processor -- start 2024-04-23T18:26:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe559225740 1 -- start start 2024-04-23T18:26:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe559225740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 0x55bf86c06a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe559225740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c112c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe559225740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 0x55bf86c13bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe54d7fa640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe5380a3e40 0x7fe5380a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe559225740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bf86b6b970 con 0x55bf86c06620 2024-04-23T18:26:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe559225740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bf86b084a0 con 0x55bf86c11800 2024-04-23T18:26:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe559225740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bf86b67930 con 0x55bf86c06f40 2024-04-23T18:26:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe54f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 0x55bf86c06a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe554aa5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 0x55bf86c13bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe54f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 0x55bf86c06a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51734/0 (socket says 172.21.15.50:51734) 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe54f7fe640 1 -- 172.21.15.50:0/3008000237 learned_addr learned my addr 172.21.15.50:0/3008000237 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe54ffff640 1 --2- 172.21.15.50:0/3008000237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c112c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe5380a3e40 0x7fe5380a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.763+0000 7fe54d7fa640 1 -- 172.21.15.50:0/2453078219 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fe5480d0d40 con 0x7fe550107fb0 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.764+0000 7fe54ffff640 1 -- 172.21.15.50:0/3008000237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 msgr2=0x55bf86c13bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.764+0000 7fe54ffff640 1 --2- 172.21.15.50:0/3008000237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 0x55bf86c13bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.764+0000 7fe54ffff640 1 -- 172.21.15.50:0/3008000237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 msgr2=0x55bf86c06a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.764+0000 7fe54ffff640 1 --2- 172.21.15.50:0/3008000237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 0x55bf86c06a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.764+0000 7fe54ffff640 1 -- 172.21.15.50:0/3008000237 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf8687e210 con 0x55bf86c06f40 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.764+0000 7fe54ffff640 1 --2- 172.21.15.50:0/3008000237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c112c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe540009230 tx=0x7fe540032ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57e300aa90f6d188 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.764+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe5380a3e40 0x7fe5380a6300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe544009fd0 tx=0x7fe544009890 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.764+0000 7fe5367fc640 1 -- 172.21.15.50:0/3008000237 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540040070 con 0x55bf86c06f40 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.764+0000 7fe5367fc640 1 -- 172.21.15.50:0/3008000237 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5400317f0 con 0x55bf86c06f40 2024-04-23T18:26:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.764+0000 7fe5367fc640 1 -- 172.21.15.50:0/3008000237 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540035040 con 0x55bf86c06f40 2024-04-23T18:26:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.765+0000 7fe559225740 1 -- 172.21.15.50:0/3008000237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 msgr2=0x55bf86c112c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.765+0000 7fe559225740 1 --2- 172.21.15.50:0/3008000237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c112c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe540009230 tx=0x7fe540032ee0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.765+0000 7fe559225740 1 -- 172.21.15.50:0/3008000237 shutdown_connections 2024-04-23T18:26:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.765+0000 7fe559225740 1 --2- 172.21.15.50:0/3008000237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 0x55bf86c13bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.765+0000 7fe559225740 1 --2- 172.21.15.50:0/3008000237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c112c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.765+0000 7fe559225740 1 --2- 172.21.15.50:0/3008000237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 0x55bf86c06a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.765+0000 7fe559225740 1 -- 172.21.15.50:0/3008000237 >> 172.21.15.50:0/3008000237 conn(0x55bf86c02b30 msgr2=0x55bf86c02f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.765+0000 7fe559225740 1 -- 172.21.15.50:0/3008000237 shutdown_connections 2024-04-23T18:26:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.765+0000 7fe559225740 1 -- 172.21.15.50:0/3008000237 wait complete. 2024-04-23T18:26:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.765+0000 7fe559225740 1 -- start start 2024-04-23T18:26:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe559225740 1 Processor -- start 2024-04-23T18:26:00.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe559225740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c06620 0x55bf86c11130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe559225740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c0e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe559225740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c11800 0x55bf86c0e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe559225740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bf86c14430 con 0x55bf86c11800 2024-04-23T18:26:00.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe559225740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bf86b060b0 con 0x55bf86c06620 2024-04-23T18:26:00.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe559225740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bf86c145b0 con 0x55bf86c06f40 2024-04-23T18:26:00.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe559225740 10 client.0 _ll_register_callbacks cb 0x55bf86af3d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:00.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c06620 0x55bf86c11130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe554aa5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c11800 0x55bf86c0e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c06620 0x55bf86c11130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:43432/0 (socket says 172.21.15.50:43432) 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe54ffff640 1 -- 172.21.15.50:0/2044155443 learned_addr learned my addr 172.21.15.50:0/2044155443 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe554aa5640 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c11800 0x55bf86c0e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51742/0 (socket says 172.21.15.50:51742) 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c0e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe54f7fe640 1 -- 172.21.15.50:0/2044155443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c06620 msgr2=0x55bf86c11130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c06620 0x55bf86c11130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.766+0000 7fe54f7fe640 1 -- 172.21.15.50:0/2044155443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c11800 msgr2=0x55bf86c0e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.767+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c11800 0x55bf86c0e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.767+0000 7fe54f7fe640 1 -- 172.21.15.50:0/2044155443 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf8694a620 con 0x55bf86c06f40 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.767+0000 7fe54ffff640 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c06620 0x55bf86c11130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.767+0000 7fe554aa5640 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c11800 0x55bf86c0e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.767+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c0e420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe54400f790 tx=0x7fe5440097a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e6b79474a210606 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.767+0000 7fe52f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.767+0000 7fe52f7fe640 1 -- 172.21.15.50:0/2044155443 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544026070 con 0x55bf86c06f40 2024-04-23T18:26:00.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.767+0000 7fe559225740 1 -- 172.21.15.50:0/2044155443 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5400317f0 con 0x55bf86c06f40 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.767+0000 7fe52f7fe640 1 -- 172.21.15.50:0/2044155443 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fe544005220 con 0x55bf86c06f40 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.767+0000 7fe52f7fe640 1 -- 172.21.15.50:0/2044155443 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe544005420 con 0x55bf86c06f40 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.767+0000 7fe559225740 10 client.26062 fetch_fsmap learned FSMap version 33 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.767+0000 7fe559225740 1 -- 172.21.15.50:0/2044155443 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55bf86c20390 con 0x55bf86c06f40 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.768+0000 7fe52f7fe640 1 -- 172.21.15.50:0/2044155443 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5400317f0 con 0x55bf86c06f40 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.768+0000 7fe52f7fe640 1 -- 172.21.15.50:0/2044155443 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fe544020450 con 0x55bf86c06f40 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.768+0000 7fe559225740 10 client.26062 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.768+0000 7fe559225740 10 client.26062 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.768+0000 7fe559225740 20 client.26062 populate_metadata read hostname 'smithi050' 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.768+0000 7fe559225740 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf86c21800 0x55bf86c23c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.768+0000 7fe559225740 4 client.26062 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.768+0000 7fe559225740 1 -- 172.21.15.50:0/2044155443 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf86b79390 con 0x55bf86c21800 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.768+0000 7fe54ffff640 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf86c21800 0x55bf86c23c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.769+0000 7fe54ffff640 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf86c21800 0x55bf86c23c50 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d34c8c493ff47f8 server_cookie=44a16734ee794151 in_seq=0 out_seq=0 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.769+0000 7fe52f7fe640 10 client.26062 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe52f7fe640 1 -- 172.21.15.50:0/2044155443 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf86b79390 con 0x55bf86c21800 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe52f7fe640 10 client.26062 handle_command_reply: tid=0 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe559225740 1 client.26062 shutdown 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe559225740 1 -- 172.21.15.50:0/2044155443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 msgr2=0x55bf86c0e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe559225740 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c0e420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe54400f790 tx=0x7fe5440097a0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe559225740 1 -- 172.21.15.50:0/2044155443 shutdown_connections 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe559225740 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf86c21800 0x55bf86c23c50 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe559225740 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c11800 0x55bf86c0e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe559225740 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c0e420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe559225740 1 --2- 172.21.15.50:0/2044155443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c06620 0x55bf86c11130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe52f7fe640 0 client.26062 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe559225740 1 -- 172.21.15.50:0/2044155443 >> 172.21.15.50:0/2044155443 conn(0x55bf86c02b30 msgr2=0x55bf86c088b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe559225740 1 -- 172.21.15.50:0/2044155443 shutdown_connections 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe559225740 1 -- 172.21.15.50:0/2044155443 wait complete. 2024-04-23T18:26:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.770+0000 7fe559225740 20 client.26062 trim_cache size 0 max 16384 2024-04-23T18:26:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.787+0000 7fe559225740 1 Processor -- start 2024-04-23T18:26:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe559225740 1 -- start start 2024-04-23T18:26:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe559225740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 0x55bf86c10360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe559225740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c295b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe559225740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 0x55bf86c29af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe559225740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501b0360 con 0x55bf86c06620 2024-04-23T18:26:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe559225740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501adc30 con 0x55bf86c11800 2024-04-23T18:26:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe559225740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501abaa0 con 0x55bf86c06f40 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 0x55bf86c10360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 0x55bf86c10360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51746/0 (socket says 172.21.15.50:51746) 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe54ffff640 1 -- 172.21.15.50:0/1794241117 learned_addr learned my addr 172.21.15.50:0/1794241117 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/1794241117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c295b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe554aa5640 1 --2- 172.21.15.50:0/1794241117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 0x55bf86c29af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe54ffff640 1 -- 172.21.15.50:0/1794241117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 msgr2=0x55bf86c295b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe54ffff640 1 --2- 172.21.15.50:0/1794241117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c295b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe54ffff640 1 -- 172.21.15.50:0/1794241117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 msgr2=0x55bf86c29af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe54ffff640 1 --2- 172.21.15.50:0/1794241117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 0x55bf86c29af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.788+0000 7fe54ffff640 1 -- 172.21.15.50:0/1794241117 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf86bfe2e0 con 0x55bf86c06620 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.789+0000 7fe54ffff640 1 --2- 172.21.15.50:0/1794241117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 0x55bf86c10360 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe540000c00 tx=0x7fe5400053a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434a79330f8d6d5d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.789+0000 7fe535ffb640 1 -- 172.21.15.50:0/1794241117 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540053020 con 0x55bf86c06620 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.789+0000 7fe535ffb640 1 -- 172.21.15.50:0/1794241117 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5400035f0 con 0x55bf86c06620 2024-04-23T18:26:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.789+0000 7fe535ffb640 1 -- 172.21.15.50:0/1794241117 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540035040 con 0x55bf86c06620 2024-04-23T18:26:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.790+0000 7fe559225740 1 -- 172.21.15.50:0/1794241117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 msgr2=0x55bf86c10360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.790+0000 7fe559225740 1 --2- 172.21.15.50:0/1794241117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 0x55bf86c10360 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe540000c00 tx=0x7fe5400053a0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.790+0000 7fe559225740 1 -- 172.21.15.50:0/1794241117 shutdown_connections 2024-04-23T18:26:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.790+0000 7fe559225740 1 --2- 172.21.15.50:0/1794241117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 0x55bf86c29af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.790+0000 7fe559225740 1 --2- 172.21.15.50:0/1794241117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06f40 0x55bf86c295b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.790+0000 7fe559225740 1 --2- 172.21.15.50:0/1794241117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06620 0x55bf86c10360 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.790+0000 7fe559225740 1 -- 172.21.15.50:0/1794241117 >> 172.21.15.50:0/1794241117 conn(0x55bf86c02b30 msgr2=0x55bf86c0fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.790+0000 7fe559225740 1 -- 172.21.15.50:0/1794241117 shutdown_connections 2024-04-23T18:26:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.790+0000 7fe559225740 1 -- 172.21.15.50:0/1794241117 wait complete. 2024-04-23T18:26:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.790+0000 7fe559225740 1 -- start start 2024-04-23T18:26:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.790+0000 7fe559225740 1 Processor -- start 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe559225740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06620 0x55bf86c19980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe559225740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06f40 0x55bf86c16d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe559225740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 0x55bf86c17250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe559225740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501ac0f0 con 0x55bf86c06f40 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe559225740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501ad560 con 0x55bf86c11800 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe559225740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501ac7c0 con 0x55bf86c06620 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe559225740 10 client.0 _ll_register_callbacks cb 0x7fe54400fe20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe54f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06f40 0x55bf86c16d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe54f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06f40 0x55bf86c16d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51758/0 (socket says 172.21.15.50:51758) 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe54f7fe640 1 -- 172.21.15.50:0/2136107937 learned_addr learned my addr 172.21.15.50:0/2136107937 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe554aa5640 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 0x55bf86c17250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe54ffff640 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06620 0x55bf86c19980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe54f7fe640 1 -- 172.21.15.50:0/2136107937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06620 msgr2=0x55bf86c19980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06620 0x55bf86c19980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe54f7fe640 1 -- 172.21.15.50:0/2136107937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 msgr2=0x55bf86c17250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 0x55bf86c17250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe54f7fe640 1 -- 172.21.15.50:0/2136107937 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf86c17810 con 0x55bf86c06f40 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe54ffff640 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06620 0x55bf86c19980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06f40 0x55bf86c16d10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe544009f10 tx=0x7fe54401ff80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fad45e552dd142c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.791+0000 7fe52f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe52f7fe640 1 -- 172.21.15.50:0/2136107937 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544036020 con 0x55bf86c06f40 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe52f7fe640 1 -- 172.21.15.50:0/2136107937 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fe544004440 con 0x55bf86c06f40 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe559225740 1 -- 172.21.15.50:0/2136107937 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5400035f0 con 0x55bf86c06f40 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe52f7fe640 1 -- 172.21.15.50:0/2136107937 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe544041050 con 0x55bf86c06f40 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe559225740 10 client.16434 fetch_fsmap learned FSMap version 33 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe559225740 1 -- 172.21.15.50:0/2136107937 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55bf86c87bf0 con 0x55bf86c06f40 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe52f7fe640 1 -- 172.21.15.50:0/2136107937 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5400035f0 con 0x55bf86c06f40 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe52f7fe640 1 -- 172.21.15.50:0/2136107937 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fe544003960 con 0x55bf86c06f40 2024-04-23T18:26:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe559225740 10 client.16434 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:26:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe559225740 10 client.16434 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe559225740 20 client.16434 populate_metadata read hostname 'smithi050' 2024-04-23T18:26:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe559225740 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf86c88330 0x55bf86c8a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe559225740 4 client.16434 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe559225740 1 -- 172.21.15.50:0/2136107937 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bf86b73200 con 0x55bf86c88330 2024-04-23T18:26:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.792+0000 7fe54ffff640 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf86c88330 0x55bf86c8a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.793+0000 7fe52f7fe640 10 client.16434 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.793+0000 7fe54ffff640 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf86c88330 0x55bf86c8a750 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b4bc428094985fd server_cookie=fa55fa6fdc05874b in_seq=0 out_seq=0 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe52f7fe640 1 -- 172.21.15.50:0/2136107937 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bf86b73200 con 0x55bf86c88330 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe52f7fe640 10 client.16434 handle_command_reply: tid=0 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe559225740 1 client.16434 shutdown 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe559225740 1 -- 172.21.15.50:0/2136107937 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06f40 msgr2=0x55bf86c16d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe559225740 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06f40 0x55bf86c16d10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe544009f10 tx=0x7fe54401ff80 comp rx=0 tx=0).stop 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe559225740 1 -- 172.21.15.50:0/2136107937 shutdown_connections 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe559225740 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf86c88330 0x55bf86c8a750 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe559225740 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf86c11800 0x55bf86c17250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe559225740 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf86c06f40 0x55bf86c16d10 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe559225740 1 --2- 172.21.15.50:0/2136107937 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf86c06620 0x55bf86c19980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe52f7fe640 0 client.16434 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe559225740 1 -- 172.21.15.50:0/2136107937 >> 172.21.15.50:0/2136107937 conn(0x55bf86c02b30 msgr2=0x55bf86c104e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.855+0000 7fe559225740 1 -- 172.21.15.50:0/2136107937 shutdown_connections 2024-04-23T18:26:00.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.856+0000 7fe559225740 1 -- 172.21.15.50:0/2136107937 wait complete. 2024-04-23T18:26:00.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.856+0000 7fe559225740 20 client.16434 trim_cache size 0 max 16384 2024-04-23T18:26:00.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:26:00.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:26:00.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:26:00.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:26:00.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.856+0000 7fe556508640 1 -- 172.21.15.50:0/2453078219 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe5380a3e40 msgr2=0x7fe5380a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.856+0000 7fe556508640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe5380a3e40 0x7fe5380a6300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe544009fd0 tx=0x7fe544009890 comp rx=0 tx=0).stop 2024-04-23T18:26:00.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.856+0000 7fe556508640 1 -- 172.21.15.50:0/2453078219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550107fb0 msgr2=0x7fe55019cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:00.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.856+0000 7fe556508640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550107fb0 0x7fe55019cef0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe548005fd0 tx=0x7fe54800ee60 comp rx=0 tx=0).stop 2024-04-23T18:26:00.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.856+0000 7fe556508640 1 -- 172.21.15.50:0/2453078219 shutdown_connections 2024-04-23T18:26:00.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.856+0000 7fe556508640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe5380a3e40 0x7fe5380a6300 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.856+0000 7fe556508640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550107fb0 0x7fe55019cef0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.856+0000 7fe556508640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55006b140 0x7fe550198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.856+0000 7fe556508640 1 --2- 172.21.15.50:0/2453078219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe55006a7c0 0x7fe550198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:00.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.856+0000 7fe556508640 1 -- 172.21.15.50:0/2453078219 >> 172.21.15.50:0/2453078219 conn(0x7fe550076c50 msgr2=0x7fe5500ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:00.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.857+0000 7fe556508640 1 -- 172.21.15.50:0/2453078219 shutdown_connections 2024-04-23T18:26:00.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:00.857+0000 7fe556508640 1 -- 172.21.15.50:0/2453078219 wait complete. 2024-04-23T18:26:01.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 457855 5.17 MB/sec execute 275 sec latency 59.750 ms 2024-04-23T18:26:02.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 458781 5.20 MB/sec execute 276 sec latency 46.840 ms 2024-04-23T18:26:02.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:02 smithi050 ceph-mon[29187]: pgmap v232: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 221 op/s 2024-04-23T18:26:02.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:02 smithi177 ceph-mon[32308]: pgmap v232: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 221 op/s 2024-04-23T18:26:02.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:02 smithi188 ceph-mon[32694]: pgmap v232: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 221 op/s 2024-04-23T18:26:03.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 460796 5.23 MB/sec execute 277 sec latency 24.466 ms 2024-04-23T18:26:04.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 462499 5.26 MB/sec execute 278 sec latency 51.018 ms 2024-04-23T18:26:04.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:04 smithi050 ceph-mon[29187]: pgmap v233: 97 pgs: 97 active+clean; 139 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 386 op/s 2024-04-23T18:26:04.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:04 smithi177 ceph-mon[32308]: pgmap v233: 97 pgs: 97 active+clean; 139 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 386 op/s 2024-04-23T18:26:04.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:04 smithi188 ceph-mon[32694]: pgmap v233: 97 pgs: 97 active+clean; 139 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 386 op/s 2024-04-23T18:26:05.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 464388 5.27 MB/sec execute 279 sec latency 35.931 ms 2024-04-23T18:26:05.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:05 smithi050 ceph-mon[29187]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T18:26:05.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:05 smithi177 ceph-mon[32308]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T18:26:05.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:05 smithi188 ceph-mon[32694]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T18:26:06.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 466313 5.29 MB/sec execute 280 sec latency 23.802 ms 2024-04-23T18:26:06.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:06 smithi050 ceph-mon[29187]: pgmap v235: 97 pgs: 97 active+clean; 139 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 308 op/s 2024-04-23T18:26:06.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:06 smithi177 ceph-mon[32308]: pgmap v235: 97 pgs: 97 active+clean; 139 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 308 op/s 2024-04-23T18:26:06.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:06 smithi188 ceph-mon[32694]: pgmap v235: 97 pgs: 97 active+clean; 139 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 308 op/s 2024-04-23T18:26:07.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 468456 5.31 MB/sec execute 281 sec latency 13.178 ms 2024-04-23T18:26:08.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 470061 5.32 MB/sec execute 282 sec latency 37.345 ms 2024-04-23T18:26:08.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:08 smithi050 ceph-mon[29187]: pgmap v236: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 306 op/s 2024-04-23T18:26:08.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:08 smithi177 ceph-mon[32308]: pgmap v236: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 306 op/s 2024-04-23T18:26:08.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:08 smithi188 ceph-mon[32694]: pgmap v236: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 306 op/s 2024-04-23T18:26:09.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 472264 5.35 MB/sec execute 283 sec latency 33.252 ms 2024-04-23T18:26:10.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 474011 5.37 MB/sec execute 284 sec latency 19.177 ms 2024-04-23T18:26:10.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:10 smithi050 ceph-mon[29187]: pgmap v237: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 306 op/s 2024-04-23T18:26:10.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:10 smithi177 ceph-mon[32308]: pgmap v237: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 306 op/s 2024-04-23T18:26:10.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:10 smithi188 ceph-mon[32694]: pgmap v237: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 306 op/s 2024-04-23T18:26:11.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 476324 5.39 MB/sec execute 285 sec latency 17.318 ms 2024-04-23T18:26:12.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 477950 5.40 MB/sec execute 286 sec latency 18.625 ms 2024-04-23T18:26:12.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:12 smithi188 ceph-mon[32694]: pgmap v238: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 306 op/s 2024-04-23T18:26:12.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:12 smithi188 ceph-mon[32694]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T18:26:12.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:12 smithi050 ceph-mon[29187]: pgmap v238: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 306 op/s 2024-04-23T18:26:12.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:12 smithi050 ceph-mon[29187]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T18:26:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:12 smithi177 ceph-mon[32308]: pgmap v238: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 306 op/s 2024-04-23T18:26:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:12 smithi177 ceph-mon[32308]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T18:26:13.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 479981 5.42 MB/sec execute 287 sec latency 50.159 ms 2024-04-23T18:26:13.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.951+0000 7fdde3cd0640 1 -- 172.21.15.50:0/537935128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdddc1021a0 msgr2=0x7fdddc102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:13.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.951+0000 7fdde3cd0640 1 --2- 172.21.15.50:0/537935128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdddc1021a0 0x7fdddc102580 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fddd8009860 tx=0x7fddd802a340 comp rx=0 tx=0).stop 2024-04-23T18:26:13.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.951+0000 7fdde3cd0640 1 -- 172.21.15.50:0/537935128 shutdown_connections 2024-04-23T18:26:13.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.951+0000 7fdde3cd0640 1 --2- 172.21.15.50:0/537935128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdddc107270 0x7fdddc10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.951+0000 7fdde3cd0640 1 --2- 172.21.15.50:0/537935128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdddc102b50 0x7fdddc106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.951+0000 7fdde3cd0640 1 --2- 172.21.15.50:0/537935128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdddc1021a0 0x7fdddc102580 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.951+0000 7fdde3cd0640 1 -- 172.21.15.50:0/537935128 >> 172.21.15.50:0/537935128 conn(0x7fdddc0793e0 msgr2=0x7fdddc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:13.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.951+0000 7fdde3cd0640 1 -- 172.21.15.50:0/537935128 shutdown_connections 2024-04-23T18:26:13.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.951+0000 7fdde3cd0640 1 -- 172.21.15.50:0/537935128 wait complete. 2024-04-23T18:26:13.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.952+0000 7fdde3cd0640 1 Processor -- start 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.952+0000 7fdde3cd0640 1 -- start start 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.952+0000 7fdde3cd0640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdddc1021a0 0x7fdddc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.952+0000 7fdde3cd0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdddc107270 0x7fdddc198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.952+0000 7fdde3cd0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdddc19cf90 0x7fdddc19d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.952+0000 7fdde3cd0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1121d0 con 0x7fdddc107270 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.952+0000 7fdde3cd0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc112050 con 0x7fdddc19cf90 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.952+0000 7fdde3cd0640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc112350 con 0x7fdddc1021a0 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.952+0000 7fdde126b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdddc107270 0x7fdddc198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.952+0000 7fdde126b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdddc107270 0x7fdddc198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38884/0 (socket says 172.21.15.50:38884) 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.952+0000 7fdde126b640 1 -- 172.21.15.50:0/1910395559 learned_addr learned my addr 172.21.15.50:0/1910395559 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.952+0000 7fdde1a6c640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdddc1021a0 0x7fdddc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.953+0000 7fdde226d640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdddc19cf90 0x7fdddc19d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.953+0000 7fdde126b640 1 -- 172.21.15.50:0/1910395559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdddc1021a0 msgr2=0x7fdddc1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:13.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.953+0000 7fdde126b640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdddc1021a0 0x7fdddc1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.953+0000 7fdde126b640 1 -- 172.21.15.50:0/1910395559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdddc19cf90 msgr2=0x7fdddc19d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:13.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.953+0000 7fdde126b640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdddc19cf90 0x7fdddc19d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.953+0000 7fdde126b640 1 -- 172.21.15.50:0/1910395559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdddc19da80 con 0x7fdddc107270 2024-04-23T18:26:13.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.953+0000 7fdde126b640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdddc107270 0x7fdddc198c10 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00e9f0 tx=0x7fddcc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:13.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.953+0000 7fddd2ffd640 1 -- 172.21.15.50:0/1910395559 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc00ccc0 con 0x7fdddc107270 2024-04-23T18:26:13.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.953+0000 7fdde3cd0640 1 -- 172.21.15.50:0/1910395559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdddc19dd70 con 0x7fdddc107270 2024-04-23T18:26:13.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.953+0000 7fdde3cd0640 1 -- 172.21.15.50:0/1910395559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdddc1a5560 con 0x7fdddc107270 2024-04-23T18:26:13.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.953+0000 7fddd2ffd640 1 -- 172.21.15.50:0/1910395559 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddcc0054d0 con 0x7fdddc107270 2024-04-23T18:26:13.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.953+0000 7fddd2ffd640 1 -- 172.21.15.50:0/1910395559 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc010430 con 0x7fdddc107270 2024-04-23T18:26:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.955+0000 7fddd2ffd640 1 -- 172.21.15.50:0/1910395559 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fddcc0046c0 con 0x7fdddc107270 2024-04-23T18:26:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde69e2740 1 Processor -- start 2024-04-23T18:26:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde69e2740 1 -- start start 2024-04-23T18:26:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde69e2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afcb60 0x563b72afb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde69e2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 0x563b72afbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fddd2ffd640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fddc80a3dc0 0x7fddc80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde69e2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72b0eea0 0x563b72b11290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde69e2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563b72a60970 con 0x563b72afba20 2024-04-23T18:26:13.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde69e2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563b729fd4a0 con 0x563b72b0eea0 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde69e2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563b72a5c930 con 0x563b72afcb60 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde1a6c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 0x563b72afbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde126b640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fddc80a3dc0 0x7fddc80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde1a6c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 0x563b72afbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38888/0 (socket says 172.21.15.50:38888) 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde1a6c640 1 -- 172.21.15.50:0/1005446758 learned_addr learned my addr 172.21.15.50:0/1005446758 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.956+0000 7fdde226d640 1 --2- 172.21.15.50:0/1005446758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afcb60 0x563b72afb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fdde226d640 1 --2- 172.21.15.50:0/1005446758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72b0eea0 0x563b72b11290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fddd2ffd640 1 -- 172.21.15.50:0/1910395559 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7fddcc0d0c00 con 0x7fdddc107270 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fdde1a6c640 1 -- 172.21.15.50:0/1005446758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afcb60 msgr2=0x563b72afb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fdde1a6c640 1 --2- 172.21.15.50:0/1005446758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afcb60 0x563b72afb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fdde1a6c640 1 -- 172.21.15.50:0/1005446758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72b0eea0 msgr2=0x563b72b11290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fdde1a6c640 1 --2- 172.21.15.50:0/1005446758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72b0eea0 0x563b72b11290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fdde1a6c640 1 -- 172.21.15.50:0/1005446758 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b72773210 con 0x563b72afba20 2024-04-23T18:26:13.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fdde226d640 1 --2- 172.21.15.50:0/1005446758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afcb60 0x563b72afb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fdde126b640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fddc80a3dc0 0x7fddc80a6280 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fddcc0ce3f0 tx=0x7fddcc00ce60 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fdde1a6c640 1 --2- 172.21.15.50:0/1005446758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 0x563b72afbea0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fddd8009860 tx=0x7fddd8005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1e716e60db218b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fddc7fff640 1 -- 172.21.15.50:0/1005446758 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8037040 con 0x563b72afba20 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fddc7fff640 1 -- 172.21.15.50:0/1005446758 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddd8030450 con 0x563b72afba20 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.957+0000 7fddc7fff640 1 -- 172.21.15.50:0/1005446758 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8003970 con 0x563b72afba20 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.958+0000 7fdde69e2740 1 -- 172.21.15.50:0/1005446758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 msgr2=0x563b72afbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.958+0000 7fdde69e2740 1 --2- 172.21.15.50:0/1005446758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 0x563b72afbea0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fddd8009860 tx=0x7fddd8005b10 comp rx=0 tx=0).stop 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.958+0000 7fdde69e2740 1 -- 172.21.15.50:0/1005446758 shutdown_connections 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.958+0000 7fdde69e2740 1 --2- 172.21.15.50:0/1005446758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72b0eea0 0x563b72b11290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.958+0000 7fdde69e2740 1 --2- 172.21.15.50:0/1005446758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 0x563b72afbea0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.958+0000 7fdde69e2740 1 --2- 172.21.15.50:0/1005446758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afcb60 0x563b72afb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.958+0000 7fdde69e2740 1 -- 172.21.15.50:0/1005446758 >> 172.21.15.50:0/1005446758 conn(0x563b72af7b30 msgr2=0x563b72af7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:13.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.958+0000 7fdde69e2740 1 -- 172.21.15.50:0/1005446758 shutdown_connections 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.958+0000 7fdde69e2740 1 -- 172.21.15.50:0/1005446758 wait complete. 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.958+0000 7fdde69e2740 1 -- start start 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde69e2740 1 Processor -- start 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde69e2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72afba20 0x563b72b01cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde69e2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afcb60 0x563b72afefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde69e2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72b0eea0 0x563b72aff540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde69e2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563b72b11ba0 con 0x563b72b0eea0 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde69e2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563b729fb0b0 con 0x563b72afba20 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde69e2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563b72b11d20 con 0x563b72afcb60 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde69e2740 10 client.0 _ll_register_callbacks cb 0x563b729e8d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde226d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72b0eea0 0x563b72aff540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde1a6c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afcb60 0x563b72afefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde226d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72b0eea0 0x563b72aff540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38900/0 (socket says 172.21.15.50:38900) 2024-04-23T18:26:13.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde1a6c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afcb60 0x563b72afefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:55090/0 (socket says 172.21.15.50:55090) 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde226d640 1 -- 172.21.15.50:0/2613236469 learned_addr learned my addr 172.21.15.50:0/2613236469 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde226d640 1 --2- 172.21.15.50:0/2613236469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72afba20 0x563b72b01cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde226d640 1 -- 172.21.15.50:0/2613236469 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afcb60 msgr2=0x563b72afefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde226d640 1 --2- 172.21.15.50:0/2613236469 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afcb60 0x563b72afefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde226d640 1 -- 172.21.15.50:0/2613236469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72afba20 msgr2=0x563b72b01cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde226d640 1 --2- 172.21.15.50:0/2613236469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72afba20 0x563b72b01cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.959+0000 7fdde226d640 1 -- 172.21.15.50:0/2613236469 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b7283f620 con 0x563b72b0eea0 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fdde226d640 1 --2- 172.21.15.50:0/2613236469 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72b0eea0 0x563b72aff540 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fddd4010bc0 tx=0x7fddd4019040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10dc25fb87130025 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fddc57fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fddc57fa640 1 -- 172.21.15.50:0/2613236469 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd4004d20 con 0x563b72b0eea0 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fddc57fa640 1 -- 172.21.15.50:0/2613236469 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7fddd4005020 con 0x563b72b0eea0 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fdde69e2740 1 -- 172.21.15.50:0/2613236469 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddd8030450 con 0x563b72b0eea0 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fddc57fa640 1 -- 172.21.15.50:0/2613236469 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fddd4005220 con 0x563b72b0eea0 2024-04-23T18:26:13.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fdde69e2740 10 client.16446 fetch_fsmap learned FSMap version 33 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fdde69e2740 1 -- 172.21.15.50:0/2613236469 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563b72b1dc20 con 0x563b72b0eea0 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fddc57fa640 1 -- 172.21.15.50:0/2613236469 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddd8030450 con 0x563b72b0eea0 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fddc57fa640 1 -- 172.21.15.50:0/2613236469 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fddd400b040 con 0x563b72b0eea0 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fdde69e2740 10 client.16446 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fdde69e2740 10 client.16446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fdde69e2740 20 client.16446 populate_metadata read hostname 'smithi050' 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fdde69e2740 1 --2- 172.21.15.50:0/2613236469 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563b72b1f0e0 0x563b72b214d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fdde69e2740 4 client.16446 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.960+0000 7fdde69e2740 1 -- 172.21.15.50:0/2613236469 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b72a6e390 con 0x563b72b1f0e0 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.961+0000 7fdde1a6c640 1 --2- 172.21.15.50:0/2613236469 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563b72b1f0e0 0x563b72b214d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.961+0000 7fddc57fa640 10 client.16446 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.961+0000 7fdde1a6c640 1 --2- 172.21.15.50:0/2613236469 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563b72b1f0e0 0x563b72b214d0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bd3c2d007e77061 server_cookie=18f97096dcb86da in_seq=0 out_seq=0 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fddc57fa640 1 -- 172.21.15.50:0/2613236469 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563b72a6e390 con 0x563b72b1f0e0 2024-04-23T18:26:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fddc57fa640 10 client.16446 handle_command_reply: tid=0 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fdde69e2740 1 client.16446 shutdown 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fdde69e2740 1 -- 172.21.15.50:0/2613236469 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72b0eea0 msgr2=0x563b72aff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fdde69e2740 1 --2- 172.21.15.50:0/2613236469 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72b0eea0 0x563b72aff540 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fddd4010bc0 tx=0x7fddd4019040 comp rx=0 tx=0).stop 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fdde69e2740 1 -- 172.21.15.50:0/2613236469 shutdown_connections 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fdde69e2740 1 --2- 172.21.15.50:0/2613236469 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563b72b1f0e0 0x563b72b214d0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fdde69e2740 1 --2- 172.21.15.50:0/2613236469 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72b0eea0 0x563b72aff540 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fdde69e2740 1 --2- 172.21.15.50:0/2613236469 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afcb60 0x563b72afefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fdde69e2740 1 --2- 172.21.15.50:0/2613236469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72afba20 0x563b72b01cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fddc57fa640 0 client.16446 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fdde69e2740 1 -- 172.21.15.50:0/2613236469 >> 172.21.15.50:0/2613236469 conn(0x563b72af7b30 msgr2=0x563b72afd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fdde69e2740 1 -- 172.21.15.50:0/2613236469 shutdown_connections 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fdde69e2740 1 -- 172.21.15.50:0/2613236469 wait complete. 2024-04-23T18:26:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.962+0000 7fdde69e2740 20 client.16446 trim_cache size 0 max 16384 2024-04-23T18:26:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.978+0000 7fdde69e2740 1 Processor -- start 2024-04-23T18:26:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.978+0000 7fdde69e2740 1 -- start start 2024-04-23T18:26:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.978+0000 7fdde69e2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 0x563b72b1aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.978+0000 7fdde69e2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72afcb60 0x563b72b017e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.978+0000 7fdde69e2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72b0eea0 0x563b72b26da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.978+0000 7fdde69e2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1ae270 con 0x563b72afba20 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde69e2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1abb40 con 0x563b72afcb60 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde69e2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1a99b0 con 0x563b72b0eea0 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde226d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 0x563b72b1aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde226d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 0x563b72b1aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38912/0 (socket says 172.21.15.50:38912) 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde226d640 1 -- 172.21.15.50:0/1330033940 learned_addr learned my addr 172.21.15.50:0/1330033940 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde226d640 1 --2- 172.21.15.50:0/1330033940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72b0eea0 0x563b72b26da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde1a6c640 1 --2- 172.21.15.50:0/1330033940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72afcb60 0x563b72b017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde226d640 1 -- 172.21.15.50:0/1330033940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72b0eea0 msgr2=0x563b72b26da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde226d640 1 --2- 172.21.15.50:0/1330033940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72b0eea0 0x563b72b26da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde226d640 1 -- 172.21.15.50:0/1330033940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72afcb60 msgr2=0x563b72b017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde226d640 1 --2- 172.21.15.50:0/1330033940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72afcb60 0x563b72b017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde226d640 1 -- 172.21.15.50:0/1330033940 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b72af32e0 con 0x563b72afba20 2024-04-23T18:26:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fdde226d640 1 --2- 172.21.15.50:0/1330033940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 0x563b72b1aad0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fddd400a010 tx=0x7fddd4012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbef544b8b547269 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fddc77fe640 1 -- 172.21.15.50:0/1330033940 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd402d040 con 0x563b72afba20 2024-04-23T18:26:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fddc77fe640 1 -- 172.21.15.50:0/1330033940 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddd4009360 con 0x563b72afba20 2024-04-23T18:26:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.979+0000 7fddc77fe640 1 -- 172.21.15.50:0/1330033940 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd40059d0 con 0x563b72afba20 2024-04-23T18:26:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.980+0000 7fdde69e2740 1 -- 172.21.15.50:0/1330033940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 msgr2=0x563b72b1aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.980+0000 7fdde69e2740 1 --2- 172.21.15.50:0/1330033940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 0x563b72b1aad0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fddd400a010 tx=0x7fddd4012040 comp rx=0 tx=0).stop 2024-04-23T18:26:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.980+0000 7fdde69e2740 1 -- 172.21.15.50:0/1330033940 shutdown_connections 2024-04-23T18:26:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.980+0000 7fdde69e2740 1 --2- 172.21.15.50:0/1330033940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72b0eea0 0x563b72b26da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.980+0000 7fdde69e2740 1 --2- 172.21.15.50:0/1330033940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72afcb60 0x563b72b017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.980+0000 7fdde69e2740 1 --2- 172.21.15.50:0/1330033940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afba20 0x563b72b1aad0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.980+0000 7fdde69e2740 1 -- 172.21.15.50:0/1330033940 >> 172.21.15.50:0/1330033940 conn(0x563b72af7b30 msgr2=0x563b72b0ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.980+0000 7fdde69e2740 1 -- 172.21.15.50:0/1330033940 shutdown_connections 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.980+0000 7fdde69e2740 1 -- 172.21.15.50:0/1330033940 wait complete. 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.980+0000 7fdde69e2740 1 -- start start 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.980+0000 7fdde69e2740 1 Processor -- start 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde69e2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afba20 0x563b72b0c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde69e2740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afcb60 0x563b72b0cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde69e2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72b0eea0 0x563b72b0d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde69e2740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1aa000 con 0x563b72afcb60 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde69e2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1ab470 con 0x563b72b0eea0 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde69e2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1aa6d0 con 0x563b72afba20 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde69e2740 10 client.0 _ll_register_callbacks cb 0x7fddd4010750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde1a6c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afcb60 0x563b72b0cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde1a6c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afcb60 0x563b72b0cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38918/0 (socket says 172.21.15.50:38918) 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde1a6c640 1 -- 172.21.15.50:0/254277137 learned_addr learned my addr 172.21.15.50:0/254277137 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde226d640 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afba20 0x563b72b0c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde226d640 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72b0eea0 0x563b72b0d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde1a6c640 1 -- 172.21.15.50:0/254277137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afba20 msgr2=0x563b72b0c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde1a6c640 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afba20 0x563b72b0c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde1a6c640 1 -- 172.21.15.50:0/254277137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72b0eea0 msgr2=0x563b72b0d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde1a6c640 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72b0eea0 0x563b72b0d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde1a6c640 1 -- 172.21.15.50:0/254277137 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b72b84610 con 0x563b72afcb60 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde1a6c640 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afcb60 0x563b72b0cdb0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fddd802a860 tx=0x7fddd8005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4d8bbae6ee262 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fddc57fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fddc57fa640 1 -- 172.21.15.50:0/254277137 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8037040 con 0x563b72afcb60 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fddc57fa640 1 -- 172.21.15.50:0/254277137 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7fddd8030680 con 0x563b72afcb60 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.981+0000 7fdde69e2740 1 -- 172.21.15.50:0/254277137 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddd4009360 con 0x563b72afcb60 2024-04-23T18:26:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fddc57fa640 1 -- 172.21.15.50:0/254277137 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fddd8030450 con 0x563b72afcb60 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fdde69e2740 10 client.16452 fetch_fsmap learned FSMap version 33 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fdde69e2740 1 -- 172.21.15.50:0/254277137 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563b72b857e0 con 0x563b72afcb60 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fddc57fa640 1 -- 172.21.15.50:0/254277137 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddd4009360 con 0x563b72afcb60 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fddc57fa640 1 -- 172.21.15.50:0/254277137 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fddd804d310 con 0x563b72afcb60 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fdde69e2740 10 client.16452 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fdde69e2740 10 client.16452 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fdde69e2740 20 client.16452 populate_metadata read hostname 'smithi050' 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fdde69e2740 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563b72b85be0 0x563b72b87fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fdde69e2740 4 client.16452 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fdde69e2740 1 -- 172.21.15.50:0/254277137 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563b72a68200 con 0x563b72b85be0 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fdde226d640 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563b72b85be0 0x563b72b87fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fdde226d640 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563b72b85be0 0x563b72b87fd0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abed4d62213441df server_cookie=9529d7b5877c9f83 in_seq=0 out_seq=0 2024-04-23T18:26:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:13.982+0000 7fddc57fa640 10 client.16452 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.049+0000 7fddc57fa640 1 -- 172.21.15.50:0/254277137 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563b72a68200 con 0x563b72b85be0 2024-04-23T18:26:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.049+0000 7fddc57fa640 10 client.16452 handle_command_reply: tid=0 2024-04-23T18:26:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.049+0000 7fdde69e2740 1 client.16452 shutdown 2024-04-23T18:26:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.050+0000 7fdde69e2740 1 -- 172.21.15.50:0/254277137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afcb60 msgr2=0x563b72b0cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.050+0000 7fdde69e2740 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afcb60 0x563b72b0cdb0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fddd802a860 tx=0x7fddd8005fb0 comp rx=0 tx=0).stop 2024-04-23T18:26:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.050+0000 7fdde69e2740 1 -- 172.21.15.50:0/254277137 shutdown_connections 2024-04-23T18:26:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.050+0000 7fdde69e2740 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563b72b85be0 0x563b72b87fd0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.050+0000 7fddc57fa640 0 client.16452 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.050+0000 7fdde69e2740 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563b72b0eea0 0x563b72b0d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.050+0000 7fdde69e2740 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563b72afcb60 0x563b72b0cdb0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.050+0000 7fdde69e2740 1 --2- 172.21.15.50:0/254277137 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563b72afba20 0x563b72b0c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.050+0000 7fdde69e2740 1 -- 172.21.15.50:0/254277137 >> 172.21.15.50:0/254277137 conn(0x563b72af7b30 msgr2=0x563b72af6420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.050+0000 7fdde69e2740 1 -- 172.21.15.50:0/254277137 shutdown_connections 2024-04-23T18:26:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.050+0000 7fdde69e2740 1 -- 172.21.15.50:0/254277137 wait complete. 2024-04-23T18:26:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.050+0000 7fdde69e2740 20 client.16452 trim_cache size 0 max 16384 2024-04-23T18:26:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:26:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:26:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:26:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:26:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.051+0000 7fdde3cd0640 1 -- 172.21.15.50:0/1910395559 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fddc80a3dc0 msgr2=0x7fddc80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.051+0000 7fdde3cd0640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fddc80a3dc0 0x7fddc80a6280 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fddcc0ce3f0 tx=0x7fddcc00ce60 comp rx=0 tx=0).stop 2024-04-23T18:26:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.051+0000 7fdde3cd0640 1 -- 172.21.15.50:0/1910395559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdddc107270 msgr2=0x7fdddc198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.051+0000 7fdde3cd0640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdddc107270 0x7fdddc198c10 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00e9f0 tx=0x7fddcc00eeb0 comp rx=0 tx=0).stop 2024-04-23T18:26:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.051+0000 7fdde3cd0640 1 -- 172.21.15.50:0/1910395559 shutdown_connections 2024-04-23T18:26:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.051+0000 7fdde3cd0640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fddc80a3dc0 0x7fddc80a6280 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.051+0000 7fdde3cd0640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdddc19cf90 0x7fdddc19d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.051+0000 7fdde3cd0640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdddc107270 0x7fdddc198c10 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.051+0000 7fdde3cd0640 1 --2- 172.21.15.50:0/1910395559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdddc1021a0 0x7fdddc1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.051+0000 7fdde3cd0640 1 -- 172.21.15.50:0/1910395559 >> 172.21.15.50:0/1910395559 conn(0x7fdddc0793e0 msgr2=0x7fdddc0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.051+0000 7fdde3cd0640 1 -- 172.21.15.50:0/1910395559 shutdown_connections 2024-04-23T18:26:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:14.051+0000 7fdde3cd0640 1 -- 172.21.15.50:0/1910395559 wait complete. 2024-04-23T18:26:14.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 481363 5.42 MB/sec execute 288 sec latency 40.561 ms 2024-04-23T18:26:14.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:14 smithi188 ceph-mon[32694]: pgmap v240: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 287 op/s 2024-04-23T18:26:14.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:14 smithi050 ceph-mon[29187]: pgmap v240: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 287 op/s 2024-04-23T18:26:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:14 smithi177 ceph-mon[32308]: pgmap v240: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 287 op/s 2024-04-23T18:26:15.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 483603 5.45 MB/sec execute 289 sec latency 23.256 ms 2024-04-23T18:26:16.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 485619 5.46 MB/sec execute 290 sec latency 21.585 ms 2024-04-23T18:26:16.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:16 smithi188 ceph-mon[32694]: pgmap v241: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 245 op/s 2024-04-23T18:26:16.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:16 smithi050 ceph-mon[29187]: pgmap v241: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 245 op/s 2024-04-23T18:26:16.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:16 smithi177 ceph-mon[32308]: pgmap v241: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 245 op/s 2024-04-23T18:26:17.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 487551 5.48 MB/sec execute 291 sec latency 21.283 ms 2024-04-23T18:26:18.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 489689 5.51 MB/sec execute 292 sec latency 57.831 ms 2024-04-23T18:26:18.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:18 smithi188 ceph-mon[32694]: pgmap v242: 97 pgs: 97 active+clean; 151 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 322 op/s 2024-04-23T18:26:18.338 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:18 smithi177 ceph-mon[32308]: pgmap v242: 97 pgs: 97 active+clean; 151 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 322 op/s 2024-04-23T18:26:18.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:18 smithi050 ceph-mon[29187]: pgmap v242: 97 pgs: 97 active+clean; 151 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 322 op/s 2024-04-23T18:26:19.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 491540 5.52 MB/sec execute 293 sec latency 24.440 ms 2024-04-23T18:26:20.216 INFO:tasks.workunit.client.0.smithi050.stdout: 1 494051 5.55 MB/sec execute 294 sec latency 13.549 ms 2024-04-23T18:26:20.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:20 smithi188 ceph-mon[32694]: pgmap v243: 97 pgs: 97 active+clean; 151 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 322 op/s 2024-04-23T18:26:20.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:20 smithi050 ceph-mon[29187]: pgmap v243: 97 pgs: 97 active+clean; 151 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 322 op/s 2024-04-23T18:26:20.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:20 smithi177 ceph-mon[32308]: pgmap v243: 97 pgs: 97 active+clean; 151 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 322 op/s 2024-04-23T18:26:21.217 INFO:tasks.workunit.client.0.smithi050.stdout: 1 495695 5.56 MB/sec execute 295 sec latency 19.240 ms 2024-04-23T18:26:22.217 INFO:tasks.workunit.client.0.smithi050.stdout: 1 497947 5.58 MB/sec execute 296 sec latency 13.286 ms 2024-04-23T18:26:22.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:22 smithi188 ceph-mon[32694]: pgmap v244: 97 pgs: 97 active+clean; 151 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 322 op/s 2024-04-23T18:26:22.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:22 smithi050 ceph-mon[29187]: pgmap v244: 97 pgs: 97 active+clean; 151 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 322 op/s 2024-04-23T18:26:22.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:22 smithi177 ceph-mon[32308]: pgmap v244: 97 pgs: 97 active+clean; 151 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 322 op/s 2024-04-23T18:26:23.217 INFO:tasks.workunit.client.0.smithi050.stdout: 1 499842 5.59 MB/sec execute 297 sec latency 68.199 ms 2024-04-23T18:26:24.217 INFO:tasks.workunit.client.0.smithi050.stdout: 1 501561 5.61 MB/sec execute 298 sec latency 23.048 ms 2024-04-23T18:26:24.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:24 smithi188 ceph-mon[32694]: pgmap v245: 97 pgs: 97 active+clean; 159 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 394 op/s 2024-04-23T18:26:24.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:24 smithi050 ceph-mon[29187]: pgmap v245: 97 pgs: 97 active+clean; 159 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 394 op/s 2024-04-23T18:26:24.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:24 smithi177 ceph-mon[32308]: pgmap v245: 97 pgs: 97 active+clean; 159 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 394 op/s 2024-04-23T18:26:25.217 INFO:tasks.workunit.client.0.smithi050.stdout: 1 503719 5.63 MB/sec execute 299 sec latency 18.947 ms 2024-04-23T18:26:26.217 INFO:tasks.workunit.client.0.smithi050.stdout: 1 505646 5.64 MB/sec execute 300 sec latency 19.288 ms 2024-04-23T18:26:26.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:26 smithi188 ceph-mon[32694]: pgmap v246: 97 pgs: 97 active+clean; 159 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 265 op/s 2024-04-23T18:26:26.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:26 smithi050 ceph-mon[29187]: pgmap v246: 97 pgs: 97 active+clean; 159 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 265 op/s 2024-04-23T18:26:26.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:26 smithi177 ceph-mon[32308]: pgmap v246: 97 pgs: 97 active+clean; 159 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 265 op/s 2024-04-23T18:26:27.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.145+0000 7ff5faa7c640 1 -- 172.21.15.50:0/3129568956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f410e910 msgr2=0x7ff5f4110d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:27.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.145+0000 7ff5faa7c640 1 --2- 172.21.15.50:0/3129568956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f410e910 0x7ff5f4110d00 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009ca0 tx=0x7ff5ec02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.145+0000 7ff5faa7c640 1 -- 172.21.15.50:0/3129568956 shutdown_connections 2024-04-23T18:26:27.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.145+0000 7ff5faa7c640 1 --2- 172.21.15.50:0/3129568956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f410e910 0x7ff5f4110d00 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.145+0000 7ff5faa7c640 1 --2- 172.21.15.50:0/3129568956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5f4101680 0x7ff5f410e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.145+0000 7ff5faa7c640 1 --2- 172.21.15.50:0/3129568956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5f4100d60 0x7ff5f4101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.145+0000 7ff5faa7c640 1 -- 172.21.15.50:0/3129568956 >> 172.21.15.50:0/3129568956 conn(0x7ff5f4077fd0 msgr2=0x7ff5f40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:27.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.145+0000 7ff5faa7c640 1 -- 172.21.15.50:0/3129568956 shutdown_connections 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.145+0000 7ff5faa7c640 1 -- 172.21.15.50:0/3129568956 wait complete. 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.145+0000 7ff5faa7c640 1 Processor -- start 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5faa7c640 1 -- start start 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5faa7c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5f4100d60 0x7ff5f41a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5faa7c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5f4101680 0x7ff5f41a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5faa7c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f410e910 0x7ff5f419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5faa7c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41143d0 con 0x7ff5f410e910 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5faa7c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f4114250 con 0x7ff5f4100d60 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5faa7c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f4114550 con 0x7ff5f4101680 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f9019640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f410e910 0x7ff5f419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f8818640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5f4100d60 0x7ff5f41a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f9019640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f410e910 0x7ff5f419a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37508/0 (socket says 172.21.15.50:37508) 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f9019640 1 -- 172.21.15.50:0/2120635827 learned_addr learned my addr 172.21.15.50:0/2120635827 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:27.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f8818640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5f4100d60 0x7ff5f41a05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:35042/0 (socket says 172.21.15.50:35042) 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f3fff640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5f4101680 0x7ff5f41a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f9019640 1 -- 172.21.15.50:0/2120635827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5f4101680 msgr2=0x7ff5f41a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f9019640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5f4101680 0x7ff5f41a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f9019640 1 -- 172.21.15.50:0/2120635827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5f4100d60 msgr2=0x7ff5f41a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f9019640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5f4100d60 0x7ff5f41a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f9019640 1 -- 172.21.15.50:0/2120635827 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f419af50 con 0x7ff5f410e910 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f8818640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5f4100d60 0x7ff5f41a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.146+0000 7ff5f9019640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f410e910 0x7ff5f419a690 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0319f0 tx=0x7ff5ec031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.147+0000 7ff5f1ffb640 1 -- 172.21.15.50:0/2120635827 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec003b70 con 0x7ff5f410e910 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.147+0000 7ff5faa7c640 1 -- 172.21.15.50:0/2120635827 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f419b1e0 con 0x7ff5f410e910 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.147+0000 7ff5f1ffb640 1 -- 172.21.15.50:0/2120635827 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5ec003d10 con 0x7ff5f410e910 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.147+0000 7ff5f1ffb640 1 -- 172.21.15.50:0/2120635827 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec02fa50 con 0x7ff5f410e910 2024-04-23T18:26:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.147+0000 7ff5faa7c640 1 -- 172.21.15.50:0/2120635827 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f41a7550 con 0x7ff5f410e910 2024-04-23T18:26:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.149+0000 7ff5f1ffb640 1 -- 172.21.15.50:0/2120635827 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff5ec02fbf0 con 0x7ff5f410e910 2024-04-23T18:26:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.149+0000 7ff5fd790740 1 Processor -- start 2024-04-23T18:26:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.149+0000 7ff5fd790740 1 -- start start 2024-04-23T18:26:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5fd790740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614620 0x55cf18614a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5fd790740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf18614f40 0x55cf1861f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5fd790740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf1861f800 0x55cf18621bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5fd790740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cf18579970 con 0x55cf1861f800 2024-04-23T18:26:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5fd790740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cf185164a0 con 0x55cf18614620 2024-04-23T18:26:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5fd790740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55cf18575930 con 0x55cf18614f40 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f8818640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf18614f40 0x55cf1861f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf1861f800 0x55cf18621bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f8818640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf18614f40 0x55cf1861f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:57854/0 (socket says 172.21.15.50:57854) 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f8818640 1 -- 172.21.15.50:0/2511784362 learned_addr learned my addr 172.21.15.50:0/2511784362 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f3fff640 1 --2- 172.21.15.50:0/2511784362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614620 0x55cf18614a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f1ffb640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5e00a3dc0 0x7ff5e00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f8818640 1 -- 172.21.15.50:0/2511784362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614620 msgr2=0x55cf18614a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f8818640 1 --2- 172.21.15.50:0/2511784362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614620 0x55cf18614a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f8818640 1 -- 172.21.15.50:0/2511784362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf1861f800 msgr2=0x55cf18621bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f8818640 1 --2- 172.21.15.50:0/2511784362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf1861f800 0x55cf18621bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f8818640 1 -- 172.21.15.50:0/2511784362 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf1828c210 con 0x55cf18614f40 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f9019640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5e00a3dc0 0x7ff5e00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f1ffb640 1 -- 172.21.15.50:0/2120635827 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7ff5ec0f1960 con 0x7ff5f410e910 2024-04-23T18:26:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.150+0000 7ff5f8818640 1 --2- 172.21.15.50:0/2511784362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf18614f40 0x55cf1861f2c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40115f0 tx=0x7ff5e4011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b880ce953f0fef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.151+0000 7ff5eaffd640 1 -- 172.21.15.50:0/2511784362 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e401e070 con 0x55cf18614f40 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.151+0000 7ff5eaffd640 1 -- 172.21.15.50:0/2511784362 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5e400fa70 con 0x55cf18614f40 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.151+0000 7ff5eaffd640 1 -- 172.21.15.50:0/2511784362 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4019b50 con 0x55cf18614f40 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.152+0000 7ff5f9019640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5e00a3dc0 0x7ff5e00a6280 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0c2fd0 tx=0x7ff5ec08f4d0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.152+0000 7ff5fd790740 1 -- 172.21.15.50:0/2511784362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf18614f40 msgr2=0x55cf1861f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.152+0000 7ff5fd790740 1 --2- 172.21.15.50:0/2511784362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf18614f40 0x55cf1861f2c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40115f0 tx=0x7ff5e4011ab0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.152+0000 7ff5fd790740 1 -- 172.21.15.50:0/2511784362 shutdown_connections 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.152+0000 7ff5fd790740 1 --2- 172.21.15.50:0/2511784362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf1861f800 0x55cf18621bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.152+0000 7ff5fd790740 1 --2- 172.21.15.50:0/2511784362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf18614f40 0x55cf1861f2c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.152+0000 7ff5fd790740 1 --2- 172.21.15.50:0/2511784362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614620 0x55cf18614a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.152+0000 7ff5fd790740 1 -- 172.21.15.50:0/2511784362 >> 172.21.15.50:0/2511784362 conn(0x55cf18610b30 msgr2=0x55cf18610f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.152+0000 7ff5fd790740 1 -- 172.21.15.50:0/2511784362 shutdown_connections 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.152+0000 7ff5fd790740 1 -- 172.21.15.50:0/2511784362 wait complete. 2024-04-23T18:26:27.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.152+0000 7ff5fd790740 1 -- start start 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.152+0000 7ff5fd790740 1 Processor -- start 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5fd790740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 0x55cf1861f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5fd790740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614f40 0x55cf1861c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5fd790740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf1861f800 0x55cf1861c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5fd790740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cf18622430 con 0x55cf18614620 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5fd790740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cf185140b0 con 0x55cf18614f40 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5fd790740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55cf186225b0 con 0x55cf1861f800 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5fd790740 10 client.0 _ll_register_callbacks cb 0x55cf18501d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5f3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 0x55cf1861f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5f3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 0x55cf1861f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37528/0 (socket says 172.21.15.50:37528) 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5f3fff640 1 -- 172.21.15.50:0/3912280196 learned_addr learned my addr 172.21.15.50:0/3912280196 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5f8818640 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614f40 0x55cf1861c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5f3fff640 1 -- 172.21.15.50:0/3912280196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf1861f800 msgr2=0x55cf1861c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:27.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5f3fff640 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf1861f800 0x55cf1861c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5f3fff640 1 -- 172.21.15.50:0/3912280196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614f40 msgr2=0x55cf1861c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5f3fff640 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614f40 0x55cf1861c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5f3fff640 1 -- 172.21.15.50:0/3912280196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf18358620 con 0x55cf18614620 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5f8818640 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614f40 0x55cf1861c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5f3fff640 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 0x55cf1861f130 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc009920 tx=0x7ff5dc003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693b458bbfba3102 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.153+0000 7ff5d3fff640 1 -- 172.21.15.50:0/3912280196 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc023060 con 0x55cf18614620 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5fd790740 1 -- 172.21.15.50:0/3912280196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5e400fa70 con 0x55cf18614620 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5d3fff640 1 -- 172.21.15.50:0/3912280196 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7ff5dc0128c0 con 0x55cf18614620 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5d3fff640 1 -- 172.21.15.50:0/3912280196 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff5dc012ac0 con 0x55cf18614620 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5fd790740 10 client.16464 fetch_fsmap learned FSMap version 33 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5fd790740 1 -- 172.21.15.50:0/3912280196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55cf1862e3e0 con 0x55cf18614620 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5d3fff640 1 -- 172.21.15.50:0/3912280196 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5e400fa70 con 0x55cf18614620 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5d3fff640 1 -- 172.21.15.50:0/3912280196 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7ff5dc01d030 con 0x55cf18614620 2024-04-23T18:26:27.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5fd790740 10 client.16464 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:26:27.153 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5fd790740 10 client.16464 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:27.153 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5fd790740 20 client.16464 populate_metadata read hostname 'smithi050' 2024-04-23T18:26:27.153 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5fd790740 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cf1862f8a0 0x55cf18631c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.153 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5fd790740 4 client.16464 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:27.153 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5fd790740 1 -- 172.21.15.50:0/3912280196 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf18587390 con 0x55cf1862f8a0 2024-04-23T18:26:27.153 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.154+0000 7ff5f8818640 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cf1862f8a0 0x55cf18631c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.153 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.155+0000 7ff5d3fff640 10 client.16464 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:27.153 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.155+0000 7ff5f8818640 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cf1862f8a0 0x55cf18631c90 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6c6596fd51df9a4 server_cookie=935c94d778efaf7e in_seq=0 out_seq=0 2024-04-23T18:26:27.154 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5d3fff640 1 -- 172.21.15.50:0/3912280196 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf18587390 con 0x55cf1862f8a0 2024-04-23T18:26:27.154 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5d3fff640 10 client.16464 handle_command_reply: tid=0 2024-04-23T18:26:27.154 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5fd790740 1 client.16464 shutdown 2024-04-23T18:26:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5fd790740 1 -- 172.21.15.50:0/3912280196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 msgr2=0x55cf1861f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5fd790740 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 0x55cf1861f130 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc009920 tx=0x7ff5dc003a80 comp rx=0 tx=0).stop 2024-04-23T18:26:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5fd790740 1 -- 172.21.15.50:0/3912280196 shutdown_connections 2024-04-23T18:26:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5fd790740 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cf1862f8a0 0x55cf18631c90 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5fd790740 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf1861f800 0x55cf1861c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5fd790740 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614f40 0x55cf1861c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5d3fff640 0 client.16464 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5fd790740 1 --2- 172.21.15.50:0/3912280196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 0x55cf1861f130 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5fd790740 1 -- 172.21.15.50:0/3912280196 >> 172.21.15.50:0/3912280196 conn(0x55cf18610b30 msgr2=0x55cf18616860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5fd790740 1 -- 172.21.15.50:0/3912280196 shutdown_connections 2024-04-23T18:26:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5fd790740 1 -- 172.21.15.50:0/3912280196 wait complete. 2024-04-23T18:26:27.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.156+0000 7ff5fd790740 20 client.16464 trim_cache size 0 max 16384 2024-04-23T18:26:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.173+0000 7ff5fd790740 1 Processor -- start 2024-04-23T18:26:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.173+0000 7ff5fd790740 1 -- start start 2024-04-23T18:26:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5fd790740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614620 0x55cf1862b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5fd790740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614f40 0x55cf18617af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5fd790740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf1861f800 0x55cf186166e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5fd790740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41b26d0 con 0x55cf18614f40 2024-04-23T18:26:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5fd790740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41affa0 con 0x55cf18614620 2024-04-23T18:26:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5fd790740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41ade10 con 0x55cf1861f800 2024-04-23T18:26:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5f3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614620 0x55cf1862b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5f3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614620 0x55cf1862b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:35072/0 (socket says 172.21.15.50:35072) 2024-04-23T18:26:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5f3fff640 1 -- 172.21.15.50:0/1560341206 learned_addr learned my addr 172.21.15.50:0/1560341206 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5f8818640 1 --2- 172.21.15.50:0/1560341206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614f40 0x55cf18617af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5f3fff640 1 --2- 172.21.15.50:0/1560341206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf1861f800 0x55cf186166e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5f8818640 1 -- 172.21.15.50:0/1560341206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf1861f800 msgr2=0x55cf186166e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5f8818640 1 --2- 172.21.15.50:0/1560341206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf1861f800 0x55cf186166e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5f8818640 1 -- 172.21.15.50:0/1560341206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614620 msgr2=0x55cf1862b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5f8818640 1 --2- 172.21.15.50:0/1560341206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614620 0x55cf1862b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5f8818640 1 -- 172.21.15.50:0/1560341206 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf18632000 con 0x55cf18614f40 2024-04-23T18:26:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.174+0000 7ff5f3fff640 1 --2- 172.21.15.50:0/1560341206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614620 0x55cf1862b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.175+0000 7ff5f8818640 1 --2- 172.21.15.50:0/1560341206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614f40 0x55cf18617af0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4005890 tx=0x7ff5e4019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38943a4f34b046d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.175+0000 7ff5ea7fc640 1 -- 172.21.15.50:0/1560341206 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e401e040 con 0x55cf18614f40 2024-04-23T18:26:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.175+0000 7ff5ea7fc640 1 -- 172.21.15.50:0/1560341206 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5e40054f0 con 0x55cf18614f40 2024-04-23T18:26:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.175+0000 7ff5ea7fc640 1 -- 172.21.15.50:0/1560341206 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4019a00 con 0x55cf18614f40 2024-04-23T18:26:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.176+0000 7ff5fd790740 1 -- 172.21.15.50:0/1560341206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614f40 msgr2=0x55cf18617af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.176+0000 7ff5fd790740 1 --2- 172.21.15.50:0/1560341206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614f40 0x55cf18617af0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4005890 tx=0x7ff5e4019730 comp rx=0 tx=0).stop 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.176+0000 7ff5fd790740 1 -- 172.21.15.50:0/1560341206 shutdown_connections 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.176+0000 7ff5fd790740 1 --2- 172.21.15.50:0/1560341206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf1861f800 0x55cf186166e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.176+0000 7ff5fd790740 1 --2- 172.21.15.50:0/1560341206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614f40 0x55cf18617af0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.176+0000 7ff5fd790740 1 --2- 172.21.15.50:0/1560341206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf18614620 0x55cf1862b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.176+0000 7ff5fd790740 1 -- 172.21.15.50:0/1560341206 >> 172.21.15.50:0/1560341206 conn(0x55cf18610b30 msgr2=0x55cf18615fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.176+0000 7ff5fd790740 1 -- 172.21.15.50:0/1560341206 shutdown_connections 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.176+0000 7ff5fd790740 1 -- 172.21.15.50:0/1560341206 wait complete. 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.176+0000 7ff5fd790740 1 -- start start 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.176+0000 7ff5fd790740 1 Processor -- start 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5fd790740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 0x55cf18622de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5fd790740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf18614f40 0x55cf18623320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5fd790740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf1861f800 0x55cf18627250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5fd790740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41ae460 con 0x55cf18614620 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5fd790740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41af8d0 con 0x55cf1861f800 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5fd790740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41aeb30 con 0x55cf18614f40 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5fd790740 10 client.0 _ll_register_callbacks cb 0x7ff5dc0044d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5f3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 0x55cf18622de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5f3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 0x55cf18622de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37542/0 (socket says 172.21.15.50:37542) 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5f8818640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf18614f40 0x55cf18623320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5f3fff640 1 -- 172.21.15.50:0/1870345581 learned_addr learned my addr 172.21.15.50:0/1870345581 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5f3fff640 1 --2- 172.21.15.50:0/1870345581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf1861f800 0x55cf18627250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5f3fff640 1 -- 172.21.15.50:0/1870345581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf18614f40 msgr2=0x55cf18623320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5f3fff640 1 --2- 172.21.15.50:0/1870345581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf18614f40 0x55cf18623320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5f3fff640 1 -- 172.21.15.50:0/1870345581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf1861f800 msgr2=0x55cf18627250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5f3fff640 1 --2- 172.21.15.50:0/1870345581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf1861f800 0x55cf18627250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.177+0000 7ff5f3fff640 1 -- 172.21.15.50:0/1870345581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf186278c0 con 0x55cf18614620 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5f3fff640 1 --2- 172.21.15.50:0/1870345581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 0x55cf18622de0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc02a040 tx=0x7ff5dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5f3a4b225dfabb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5d3fff640 1 -- 172.21.15.50:0/1870345581 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc00ea50 con 0x55cf18614620 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5d3fff640 1 -- 172.21.15.50:0/1870345581 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7ff5dc00ebf0 con 0x55cf18614620 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5fd790740 1 -- 172.21.15.50:0/1870345581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5e40054f0 con 0x55cf18614620 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5d3fff640 1 -- 172.21.15.50:0/1870345581 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff5dc00edf0 con 0x55cf18614620 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5fd790740 10 client.16470 fetch_fsmap learned FSMap version 33 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5fd790740 1 -- 172.21.15.50:0/1870345581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55cf18695950 con 0x55cf18614620 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5d3fff640 1 -- 172.21.15.50:0/1870345581 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5e40054f0 con 0x55cf18614620 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5d3fff640 1 -- 172.21.15.50:0/1870345581 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7ff5dc01d040 con 0x55cf18614620 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5fd790740 10 client.16470 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5fd790740 10 client.16470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5fd790740 20 client.16470 populate_metadata read hostname 'smithi050' 2024-04-23T18:26:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5fd790740 1 --2- 172.21.15.50:0/1870345581 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cf18695fc0 0x55cf186983e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5fd790740 4 client.16470 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.178+0000 7ff5fd790740 1 -- 172.21.15.50:0/1870345581 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cf18581200 con 0x55cf18695fc0 2024-04-23T18:26:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.179+0000 7ff5f8818640 1 --2- 172.21.15.50:0/1870345581 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cf18695fc0 0x55cf186983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.179+0000 7ff5d3fff640 10 client.16470 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.179+0000 7ff5f8818640 1 --2- 172.21.15.50:0/1870345581 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cf18695fc0 0x55cf186983e0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5bc904866594401 server_cookie=9563369e73a34a18 in_seq=0 out_seq=0 2024-04-23T18:26:27.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 507845 5.67 MB/sec execute 301 sec latency 18.873 ms 2024-04-23T18:26:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.241+0000 7ff5d3fff640 1 -- 172.21.15.50:0/1870345581 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cf18581200 con 0x55cf18695fc0 2024-04-23T18:26:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.241+0000 7ff5d3fff640 10 client.16470 handle_command_reply: tid=0 2024-04-23T18:26:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.241+0000 7ff5fd790740 1 client.16470 shutdown 2024-04-23T18:26:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.241+0000 7ff5fd790740 1 -- 172.21.15.50:0/1870345581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 msgr2=0x55cf18622de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.241+0000 7ff5fd790740 1 --2- 172.21.15.50:0/1870345581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 0x55cf18622de0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc02a040 tx=0x7ff5dc004730 comp rx=0 tx=0).stop 2024-04-23T18:26:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.241+0000 7ff5fd790740 1 -- 172.21.15.50:0/1870345581 shutdown_connections 2024-04-23T18:26:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.241+0000 7ff5fd790740 1 --2- 172.21.15.50:0/1870345581 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cf18695fc0 0x55cf186983e0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.241+0000 7ff5fd790740 1 --2- 172.21.15.50:0/1870345581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf1861f800 0x55cf18627250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.241+0000 7ff5fd790740 1 --2- 172.21.15.50:0/1870345581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cf18614f40 0x55cf18623320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.241+0000 7ff5d3fff640 0 client.16470 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.241+0000 7ff5fd790740 1 --2- 172.21.15.50:0/1870345581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cf18614620 0x55cf18622de0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.242+0000 7ff5fd790740 1 -- 172.21.15.50:0/1870345581 >> 172.21.15.50:0/1870345581 conn(0x55cf18610b30 msgr2=0x55cf18616860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.242+0000 7ff5fd790740 1 -- 172.21.15.50:0/1870345581 shutdown_connections 2024-04-23T18:26:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.242+0000 7ff5fd790740 1 -- 172.21.15.50:0/1870345581 wait complete. 2024-04-23T18:26:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.242+0000 7ff5fd790740 20 client.16470 trim_cache size 0 max 16384 2024-04-23T18:26:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:26:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:26:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:26:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:26:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.242+0000 7ff5faa7c640 1 -- 172.21.15.50:0/2120635827 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5e00a3dc0 msgr2=0x7ff5e00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.242+0000 7ff5faa7c640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5e00a3dc0 0x7ff5e00a6280 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0c2fd0 tx=0x7ff5ec08f4d0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.242+0000 7ff5faa7c640 1 -- 172.21.15.50:0/2120635827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f410e910 msgr2=0x7ff5f419a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.242+0000 7ff5faa7c640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f410e910 0x7ff5f419a690 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0319f0 tx=0x7ff5ec031a20 comp rx=0 tx=0).stop 2024-04-23T18:26:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.243+0000 7ff5faa7c640 1 -- 172.21.15.50:0/2120635827 shutdown_connections 2024-04-23T18:26:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.243+0000 7ff5faa7c640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5e00a3dc0 0x7ff5e00a6280 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.243+0000 7ff5faa7c640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f410e910 0x7ff5f419a690 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.243+0000 7ff5faa7c640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5f4101680 0x7ff5f41a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.243+0000 7ff5faa7c640 1 --2- 172.21.15.50:0/2120635827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5f4100d60 0x7ff5f41a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.243+0000 7ff5faa7c640 1 -- 172.21.15.50:0/2120635827 >> 172.21.15.50:0/2120635827 conn(0x7ff5f4077fd0 msgr2=0x7ff5f410fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.243+0000 7ff5faa7c640 1 -- 172.21.15.50:0/2120635827 shutdown_connections 2024-04-23T18:26:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:27.243+0000 7ff5faa7c640 1 -- 172.21.15.50:0/2120635827 wait complete. 2024-04-23T18:26:28.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 509176 5.67 MB/sec execute 302 sec latency 105.086 ms 2024-04-23T18:26:28.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:28 smithi188 ceph-mon[32694]: pgmap v247: 97 pgs: 97 active+clean; 167 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 379 op/s 2024-04-23T18:26:28.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:28 smithi050 ceph-mon[29187]: pgmap v247: 97 pgs: 97 active+clean; 167 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 379 op/s 2024-04-23T18:26:28.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:28 smithi177 ceph-mon[32308]: pgmap v247: 97 pgs: 97 active+clean; 167 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 379 op/s 2024-04-23T18:26:29.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 511220 5.69 MB/sec execute 303 sec latency 33.048 ms 2024-04-23T18:26:30.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 512965 5.70 MB/sec execute 304 sec latency 26.933 ms 2024-04-23T18:26:30.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:30 smithi188 ceph-mon[32694]: pgmap v248: 97 pgs: 97 active+clean; 167 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-04-23T18:26:30.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:30 smithi050 ceph-mon[29187]: pgmap v248: 97 pgs: 97 active+clean; 167 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-04-23T18:26:30.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:30 smithi177 ceph-mon[32308]: pgmap v248: 97 pgs: 97 active+clean; 167 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-04-23T18:26:31.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 515335 5.73 MB/sec execute 305 sec latency 13.749 ms 2024-04-23T18:26:32.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 516950 5.73 MB/sec execute 306 sec latency 19.647 ms 2024-04-23T18:26:32.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:32 smithi188 ceph-mon[32694]: pgmap v249: 97 pgs: 97 active+clean; 167 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-04-23T18:26:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:32 smithi050 ceph-mon[29187]: pgmap v249: 97 pgs: 97 active+clean; 167 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-04-23T18:26:32.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:32 smithi177 ceph-mon[32308]: pgmap v249: 97 pgs: 97 active+clean; 167 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-04-23T18:26:33.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 519079 5.76 MB/sec execute 307 sec latency 15.134 ms 2024-04-23T18:26:34.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 520437 5.75 MB/sec execute 308 sec latency 78.050 ms 2024-04-23T18:26:34.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:34 smithi188 ceph-mon[32694]: pgmap v250: 97 pgs: 97 active+clean; 161 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 377 op/s 2024-04-23T18:26:34.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:34 smithi050 ceph-mon[29187]: pgmap v250: 97 pgs: 97 active+clean; 161 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 377 op/s 2024-04-23T18:26:34.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:34 smithi177 ceph-mon[32308]: pgmap v250: 97 pgs: 97 active+clean; 161 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 377 op/s 2024-04-23T18:26:35.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 522538 5.78 MB/sec execute 309 sec latency 23.006 ms 2024-04-23T18:26:36.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 524108 5.78 MB/sec execute 310 sec latency 15.186 ms 2024-04-23T18:26:36.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:36 smithi188 ceph-mon[32694]: pgmap v251: 97 pgs: 97 active+clean; 161 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-04-23T18:26:36.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:36 smithi050 ceph-mon[29187]: pgmap v251: 97 pgs: 97 active+clean; 161 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-04-23T18:26:36.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:36 smithi177 ceph-mon[32308]: pgmap v251: 97 pgs: 97 active+clean; 161 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 265 op/s 2024-04-23T18:26:37.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 526299 5.81 MB/sec execute 311 sec latency 12.603 ms 2024-04-23T18:26:38.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 528219 5.81 MB/sec execute 312 sec latency 22.989 ms 2024-04-23T18:26:38.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:38 smithi050 ceph-mon[29187]: pgmap v252: 97 pgs: 97 active+clean; 188 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 373 op/s 2024-04-23T18:26:38.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:38 smithi177 ceph-mon[32308]: pgmap v252: 97 pgs: 97 active+clean; 188 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 373 op/s 2024-04-23T18:26:38.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:38 smithi188 ceph-mon[32694]: pgmap v252: 97 pgs: 97 active+clean; 188 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 373 op/s 2024-04-23T18:26:39.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 529895 5.83 MB/sec execute 313 sec latency 46.909 ms 2024-04-23T18:26:40.218 INFO:tasks.workunit.client.0.smithi050.stdout: 1 531838 5.84 MB/sec execute 314 sec latency 19.919 ms 2024-04-23T18:26:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.333+0000 7f11604cb640 1 -- 172.21.15.50:0/641159776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11581010f0 msgr2=0x7f11581014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.333+0000 7f11604cb640 1 --2- 172.21.15.50:0/641159776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11581010f0 0x7f11581014d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f114c009900 tx=0x7f114c02a410 comp rx=0 tx=0).stop 2024-04-23T18:26:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.333+0000 7f11604cb640 1 -- 172.21.15.50:0/641159776 shutdown_connections 2024-04-23T18:26:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.333+0000 7f11604cb640 1 --2- 172.21.15.50:0/641159776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1158106170 0x7f115810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.333+0000 7f11604cb640 1 --2- 172.21.15.50:0/641159776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1158101aa0 0x7f1158105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.333+0000 7f11604cb640 1 --2- 172.21.15.50:0/641159776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11581010f0 0x7f11581014d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.333+0000 7f11604cb640 1 -- 172.21.15.50:0/641159776 >> 172.21.15.50:0/641159776 conn(0x7f1158077fd0 msgr2=0x7f11580783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.333+0000 7f11604cb640 1 -- 172.21.15.50:0/641159776 shutdown_connections 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.333+0000 7f11604cb640 1 -- 172.21.15.50:0/641159776 wait complete. 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.333+0000 7f11604cb640 1 Processor -- start 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.333+0000 7f11604cb640 1 -- start start 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f11604cb640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11581010f0 0x7f11581986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f11604cb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1158106170 0x7f1158198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f11604cb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f115819cf90 0x7f115819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f11604cb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f11581121f0 con 0x7f1158106170 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f11604cb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1158112070 con 0x7f115819cf90 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f11604cb640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1158112370 con 0x7f11581010f0 2024-04-23T18:26:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f115da66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1158106170 0x7f1158198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f115da66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1158106170 0x7f1158198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46364/0 (socket says 172.21.15.50:46364) 2024-04-23T18:26:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f115da66640 1 -- 172.21.15.50:0/740771752 learned_addr learned my addr 172.21.15.50:0/740771752 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f115e267640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11581010f0 0x7f11581986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f115ea68640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f115819cf90 0x7f115819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f115da66640 1 -- 172.21.15.50:0/740771752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11581010f0 msgr2=0x7f11581986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f115da66640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11581010f0 0x7f11581986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f115da66640 1 -- 172.21.15.50:0/740771752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f115819cf90 msgr2=0x7f115819d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f115da66640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f115819cf90 0x7f115819d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f115da66640 1 -- 172.21.15.50:0/740771752 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115819da80 con 0x7f1158106170 2024-04-23T18:26:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f115ea68640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f115819cf90 0x7f115819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:26:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f115da66640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1158106170 0x7f1158198c10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f115400e9f0 tx=0x7f115400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f114b7fe640 1 -- 172.21.15.50:0/740771752 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115400ccc0 con 0x7f1158106170 2024-04-23T18:26:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f114b7fe640 1 -- 172.21.15.50:0/740771752 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11540054d0 con 0x7f1158106170 2024-04-23T18:26:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f114b7fe640 1 -- 172.21.15.50:0/740771752 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1154010430 con 0x7f1158106170 2024-04-23T18:26:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f11604cb640 1 -- 172.21.15.50:0/740771752 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f115819dd70 con 0x7f1158106170 2024-04-23T18:26:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.334+0000 7f11604cb640 1 -- 172.21.15.50:0/740771752 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11581a5510 con 0x7f1158106170 2024-04-23T18:26:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.336+0000 7f114b7fe640 1 -- 172.21.15.50:0/740771752 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f11540046c0 con 0x7f1158106170 2024-04-23T18:26:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.337+0000 7f11631da740 1 Processor -- start 2024-04-23T18:26:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.337+0000 7f11631da740 1 -- start start 2024-04-23T18:26:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.337+0000 7f11631da740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 0x5608b43b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.337+0000 7f11631da740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43b4f40 0x5608b43bf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.337+0000 7f11631da740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43bf800 0x5608b43c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.337+0000 7f11631da740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5608b4319970 con 0x5608b43bf800 2024-04-23T18:26:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.337+0000 7f11631da740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5608b42b64a0 con 0x5608b43b4f40 2024-04-23T18:26:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.337+0000 7f11631da740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5608b4315930 con 0x5608b43b4620 2024-04-23T18:26:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115e267640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43b4f40 0x5608b43bf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115ea68640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 0x5608b43b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115da66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43bf800 0x5608b43c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f114b7fe640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f11440a3dc0 0x7f11440a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115ea68640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 0x5608b43b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:54062/0 (socket says 172.21.15.50:54062) 2024-04-23T18:26:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115e267640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43b4f40 0x5608b43bf2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:56558/0 (socket says 172.21.15.50:56558) 2024-04-23T18:26:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115ea68640 1 -- 172.21.15.50:0/964831667 learned_addr learned my addr 172.21.15.50:0/964831667 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115e267640 1 -- 172.21.15.50:0/964831667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 msgr2=0x5608b43b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115ea68640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f11440a3dc0 0x7f11440a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115e267640 1 --2- 172.21.15.50:0/964831667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 0x5608b43b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115e267640 1 -- 172.21.15.50:0/964831667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43bf800 msgr2=0x5608b43c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115e267640 1 --2- 172.21.15.50:0/964831667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43bf800 0x5608b43c1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f114b7fe640 1 -- 172.21.15.50:0/740771752 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f11540d0c10 con 0x7f1158106170 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115e267640 1 -- 172.21.15.50:0/964831667 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608b402c210 con 0x5608b43b4f40 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115e267640 1 --2- 172.21.15.50:0/964831667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43b4f40 0x5608b43bf2c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f114c009230 tx=0x7f114c032b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa2e84f87f805516 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f115ea68640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f11440a3dc0 0x7f11440a6280 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1150009fd0 tx=0x7f1150004ab0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f1148bfe640 1 -- 172.21.15.50:0/964831667 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c040070 con 0x5608b43b4f40 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f1148bfe640 1 -- 172.21.15.50:0/964831667 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f114c004030 con 0x5608b43b4f40 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.338+0000 7f1148bfe640 1 -- 172.21.15.50:0/964831667 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c035040 con 0x5608b43b4f40 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.339+0000 7f11631da740 1 -- 172.21.15.50:0/964831667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43b4f40 msgr2=0x5608b43bf2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.339+0000 7f11631da740 1 --2- 172.21.15.50:0/964831667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43b4f40 0x5608b43bf2c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f114c009230 tx=0x7f114c032b20 comp rx=0 tx=0).stop 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.339+0000 7f11631da740 1 -- 172.21.15.50:0/964831667 shutdown_connections 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.339+0000 7f11631da740 1 --2- 172.21.15.50:0/964831667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43bf800 0x5608b43c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.339+0000 7f11631da740 1 --2- 172.21.15.50:0/964831667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43b4f40 0x5608b43bf2c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.339+0000 7f11631da740 1 --2- 172.21.15.50:0/964831667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 0x5608b43b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.339+0000 7f11631da740 1 -- 172.21.15.50:0/964831667 >> 172.21.15.50:0/964831667 conn(0x5608b43b0b30 msgr2=0x5608b43b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.339+0000 7f11631da740 1 -- 172.21.15.50:0/964831667 shutdown_connections 2024-04-23T18:26:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.339+0000 7f11631da740 1 -- 172.21.15.50:0/964831667 wait complete. 2024-04-23T18:26:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.339+0000 7f11631da740 1 -- start start 2024-04-23T18:26:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f11631da740 1 Processor -- start 2024-04-23T18:26:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f11631da740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 0x5608b43bf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f11631da740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4f40 0x5608b43bc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f11631da740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43bf800 0x5608b43bc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f11631da740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5608b43c23d0 con 0x5608b43b4f40 2024-04-23T18:26:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f11631da740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5608b42b40b0 con 0x5608b43bf800 2024-04-23T18:26:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f11631da740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5608b43c2550 con 0x5608b43b4620 2024-04-23T18:26:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f11631da740 10 client.0 _ll_register_callbacks cb 0x5608b42a1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115e267640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 0x5608b43bf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115da66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4f40 0x5608b43bc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115ea68640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43bf800 0x5608b43bc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115e267640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 0x5608b43bf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:54070/0 (socket says 172.21.15.50:54070) 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115da66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4f40 0x5608b43bc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46390/0 (socket says 172.21.15.50:46390) 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115e267640 1 -- 172.21.15.50:0/1004368699 learned_addr learned my addr 172.21.15.50:0/1004368699 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115ea68640 1 -- 172.21.15.50:0/1004368699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 msgr2=0x5608b43bf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115ea68640 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 0x5608b43bf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115ea68640 1 -- 172.21.15.50:0/1004368699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4f40 msgr2=0x5608b43bc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115ea68640 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4f40 0x5608b43bc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115ea68640 1 -- 172.21.15.50:0/1004368699 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608b40f8620 con 0x5608b43bf800 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115e267640 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 0x5608b43bf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.340+0000 7f115da66640 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4f40 0x5608b43bc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f115ea68640 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43bf800 0x5608b43bc960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1150011fe0 tx=0x7f1150004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfc63269db19a28b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f1141ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f1141ffb640 1 -- 172.21.15.50:0/1004368699 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150004b20 con 0x5608b43bf800 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f1141ffb640 1 -- 172.21.15.50:0/1004368699 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f1150028410 con 0x5608b43bf800 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f11631da740 1 -- 172.21.15.50:0/1004368699 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f114c004030 con 0x5608b43bf800 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f1141ffb640 1 -- 172.21.15.50:0/1004368699 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1150028610 con 0x5608b43bf800 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f11631da740 10 client.26057 fetch_fsmap learned FSMap version 33 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f11631da740 1 -- 172.21.15.50:0/1004368699 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5608b43ce340 con 0x5608b43bf800 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f1141ffb640 1 -- 172.21.15.50:0/1004368699 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f114c004030 con 0x5608b43bf800 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f1141ffb640 1 -- 172.21.15.50:0/1004368699 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f115001dad0 con 0x5608b43bf800 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f11631da740 10 client.26057 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f11631da740 10 client.26057 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f11631da740 20 client.26057 populate_metadata read hostname 'smithi050' 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f11631da740 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5608b43cf850 0x5608b43d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f11631da740 4 client.26057 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f11631da740 1 -- 172.21.15.50:0/1004368699 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608b4327390 con 0x5608b43cf850 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.341+0000 7f115e267640 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5608b43cf850 0x5608b43d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.342+0000 7f1141ffb640 10 client.26057 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.342+0000 7f115e267640 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5608b43cf850 0x5608b43d1c40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72c5346796d3da63 server_cookie=1508477050f38f3f in_seq=0 out_seq=0 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.342+0000 7f1141ffb640 1 -- 172.21.15.50:0/1004368699 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5608b4327390 con 0x5608b43cf850 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.342+0000 7f1141ffb640 10 client.26057 handle_command_reply: tid=0 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.342+0000 7f11631da740 1 client.26057 shutdown 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.343+0000 7f11631da740 1 -- 172.21.15.50:0/1004368699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43bf800 msgr2=0x5608b43bc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.343+0000 7f11631da740 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43bf800 0x5608b43bc960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1150011fe0 tx=0x7f1150004a40 comp rx=0 tx=0).stop 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.343+0000 7f11631da740 1 -- 172.21.15.50:0/1004368699 shutdown_connections 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.343+0000 7f11631da740 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5608b43cf850 0x5608b43d1c40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.343+0000 7f11631da740 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43bf800 0x5608b43bc960 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.343+0000 7f11631da740 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4f40 0x5608b43bc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.343+0000 7f11631da740 1 --2- 172.21.15.50:0/1004368699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4620 0x5608b43bf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.343+0000 7f1141ffb640 0 client.26057 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.343+0000 7f11631da740 1 -- 172.21.15.50:0/1004368699 >> 172.21.15.50:0/1004368699 conn(0x5608b43b0b30 msgr2=0x5608b43b6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.343+0000 7f11631da740 1 -- 172.21.15.50:0/1004368699 shutdown_connections 2024-04-23T18:26:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.343+0000 7f11631da740 1 -- 172.21.15.50:0/1004368699 wait complete. 2024-04-23T18:26:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.343+0000 7f11631da740 20 client.26057 trim_cache size 0 max 16384 2024-04-23T18:26:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f11631da740 1 Processor -- start 2024-04-23T18:26:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f11631da740 1 -- start start 2024-04-23T18:26:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f11631da740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 0x5608b43cb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f11631da740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4f40 0x5608b43afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f11631da740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43bf800 0x5608b43d7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f11631da740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f11581b04a0 con 0x5608b43b4620 2024-04-23T18:26:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f11631da740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f11581add70 con 0x5608b43bf800 2024-04-23T18:26:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f11631da740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f11581abbe0 con 0x5608b43b4f40 2024-04-23T18:26:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f115e267640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 0x5608b43cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f115e267640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 0x5608b43cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46404/0 (socket says 172.21.15.50:46404) 2024-04-23T18:26:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f115e267640 1 -- 172.21.15.50:0/1238490958 learned_addr learned my addr 172.21.15.50:0/1238490958 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f115da66640 1 --2- 172.21.15.50:0/1238490958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4f40 0x5608b43afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f115ea68640 1 --2- 172.21.15.50:0/1238490958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43bf800 0x5608b43d7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f115e267640 1 -- 172.21.15.50:0/1238490958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4f40 msgr2=0x5608b43afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f115e267640 1 --2- 172.21.15.50:0/1238490958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4f40 0x5608b43afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f115e267640 1 -- 172.21.15.50:0/1238490958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43bf800 msgr2=0x5608b43d7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f115e267640 1 --2- 172.21.15.50:0/1238490958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43bf800 0x5608b43d7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.358+0000 7f115e267640 1 -- 172.21.15.50:0/1238490958 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608b43c1f60 con 0x5608b43b4620 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f115e267640 1 --2- 172.21.15.50:0/1238490958 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 0x5608b43cb290 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f114c02a910 tx=0x7f114c031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b3d982864ab12cd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f1143fff640 1 -- 172.21.15.50:0/1238490958 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c055070 con 0x5608b43b4620 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f1143fff640 1 -- 172.21.15.50:0/1238490958 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f114c004460 con 0x5608b43b4620 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f1143fff640 1 -- 172.21.15.50:0/1238490958 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c040040 con 0x5608b43b4620 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f11631da740 1 -- 172.21.15.50:0/1238490958 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 msgr2=0x5608b43cb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f11631da740 1 --2- 172.21.15.50:0/1238490958 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 0x5608b43cb290 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f114c02a910 tx=0x7f114c031400 comp rx=0 tx=0).stop 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f11631da740 1 -- 172.21.15.50:0/1238490958 shutdown_connections 2024-04-23T18:26:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f11631da740 1 --2- 172.21.15.50:0/1238490958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43bf800 0x5608b43d7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f11631da740 1 --2- 172.21.15.50:0/1238490958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43b4f40 0x5608b43afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f11631da740 1 --2- 172.21.15.50:0/1238490958 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 0x5608b43cb290 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f11631da740 1 -- 172.21.15.50:0/1238490958 >> 172.21.15.50:0/1238490958 conn(0x5608b43b0b30 msgr2=0x5608b43af420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f11631da740 1 -- 172.21.15.50:0/1238490958 shutdown_connections 2024-04-23T18:26:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.359+0000 7f11631da740 1 -- 172.21.15.50:0/1238490958 wait complete. 2024-04-23T18:26:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f11631da740 1 -- start start 2024-04-23T18:26:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f11631da740 1 Processor -- start 2024-04-23T18:26:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f11631da740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 0x5608b43c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f11631da740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43b4f40 0x5608b43c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f11631da740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43bf800 0x5608b4433dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f11631da740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f11581ac230 con 0x5608b43b4620 2024-04-23T18:26:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f11631da740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f11581ad6a0 con 0x5608b43b4f40 2024-04-23T18:26:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f11631da740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f11581ac900 con 0x5608b43bf800 2024-04-23T18:26:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f11631da740 10 client.0 _ll_register_callbacks cb 0x7f1150011be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f115e267640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 0x5608b43c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f115e267640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 0x5608b43c48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46418/0 (socket says 172.21.15.50:46418) 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f115e267640 1 -- 172.21.15.50:0/3863692769 learned_addr learned my addr 172.21.15.50:0/3863692769 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f115da66640 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43b4f40 0x5608b43c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f115ea68640 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43bf800 0x5608b4433dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f115e267640 1 -- 172.21.15.50:0/3863692769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43bf800 msgr2=0x5608b4433dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f115e267640 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43bf800 0x5608b4433dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f115e267640 1 -- 172.21.15.50:0/3863692769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43b4f40 msgr2=0x5608b43c4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f115e267640 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43b4f40 0x5608b43c4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.360+0000 7f115e267640 1 -- 172.21.15.50:0/3863692769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608b43d1fb0 con 0x5608b43b4620 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f115e267640 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 0x5608b43c48a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f114c0095a0 tx=0x7f114c005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9eb8bcf26d8210 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f1141ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f1141ffb640 1 -- 172.21.15.50:0/3863692769 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c05e060 con 0x5608b43b4620 2024-04-23T18:26:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f1141ffb640 1 -- 172.21.15.50:0/3863692769 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f114c031d10 con 0x5608b43b4620 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f11631da740 1 -- 172.21.15.50:0/3863692769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f114c004460 con 0x5608b43b4620 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f1141ffb640 1 -- 172.21.15.50:0/3863692769 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f114c040070 con 0x5608b43b4620 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f11631da740 10 client.16488 fetch_fsmap learned FSMap version 33 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f11631da740 1 -- 172.21.15.50:0/3863692769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5608b44353e0 con 0x5608b43b4620 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f1141ffb640 1 -- 172.21.15.50:0/3863692769 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f114c004460 con 0x5608b43b4620 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f1141ffb640 1 -- 172.21.15.50:0/3863692769 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f114c04f030 con 0x5608b43b4620 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f11631da740 10 client.16488 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f11631da740 10 client.16488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f11631da740 20 client.16488 populate_metadata read hostname 'smithi050' 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f11631da740 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5608b4435660 0x5608b4437a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f11631da740 4 client.16488 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.361+0000 7f11631da740 1 -- 172.21.15.50:0/3863692769 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5608b4321200 con 0x5608b4435660 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.362+0000 7f115da66640 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5608b4435660 0x5608b4437a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.362+0000 7f1141ffb640 10 client.16488 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.362+0000 7f115da66640 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5608b4435660 0x5608b4437a50 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd3da887c1131e9e server_cookie=22cf427b9eef1134 in_seq=0 out_seq=0 2024-04-23T18:26:40.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.418+0000 7f1141ffb640 1 -- 172.21.15.50:0/3863692769 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5608b4321200 con 0x5608b4435660 2024-04-23T18:26:40.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.418+0000 7f1141ffb640 10 client.16488 handle_command_reply: tid=0 2024-04-23T18:26:40.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.418+0000 7f11631da740 1 client.16488 shutdown 2024-04-23T18:26:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.418+0000 7f11631da740 1 -- 172.21.15.50:0/3863692769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 msgr2=0x5608b43c48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.418+0000 7f11631da740 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 0x5608b43c48a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f114c0095a0 tx=0x7f114c005900 comp rx=0 tx=0).stop 2024-04-23T18:26:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11631da740 1 -- 172.21.15.50:0/3863692769 shutdown_connections 2024-04-23T18:26:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11631da740 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5608b4435660 0x5608b4437a50 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f1141ffb640 0 client.16488 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11631da740 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5608b43bf800 0x5608b4433dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11631da740 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5608b43b4f40 0x5608b43c4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11631da740 1 --2- 172.21.15.50:0/3863692769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5608b43b4620 0x5608b43c48a0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11631da740 1 -- 172.21.15.50:0/3863692769 >> 172.21.15.50:0/3863692769 conn(0x5608b43b0b30 msgr2=0x5608b43ba8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11631da740 1 -- 172.21.15.50:0/3863692769 shutdown_connections 2024-04-23T18:26:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11631da740 1 -- 172.21.15.50:0/3863692769 wait complete. 2024-04-23T18:26:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11631da740 20 client.16488 trim_cache size 0 max 16384 2024-04-23T18:26:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:26:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:26:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:26:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:26:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11604cb640 1 -- 172.21.15.50:0/740771752 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f11440a3dc0 msgr2=0x7f11440a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11604cb640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f11440a3dc0 0x7f11440a6280 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1150009fd0 tx=0x7f1150004ab0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11604cb640 1 -- 172.21.15.50:0/740771752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1158106170 msgr2=0x7f1158198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.419+0000 7f11604cb640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1158106170 0x7f1158198c10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f115400e9f0 tx=0x7f115400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.420+0000 7f11604cb640 1 -- 172.21.15.50:0/740771752 shutdown_connections 2024-04-23T18:26:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.420+0000 7f11604cb640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f11440a3dc0 0x7f11440a6280 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.420+0000 7f11604cb640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f115819cf90 0x7f115819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.420+0000 7f11604cb640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1158106170 0x7f1158198c10 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.420+0000 7f11604cb640 1 --2- 172.21.15.50:0/740771752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11581010f0 0x7f11581986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.420+0000 7f11604cb640 1 -- 172.21.15.50:0/740771752 >> 172.21.15.50:0/740771752 conn(0x7f1158077fd0 msgr2=0x7f11580ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.420+0000 7f11604cb640 1 -- 172.21.15.50:0/740771752 shutdown_connections 2024-04-23T18:26:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:40.420+0000 7f11604cb640 1 -- 172.21.15.50:0/740771752 wait complete. 2024-04-23T18:26:40.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:40 smithi050 ceph-mon[29187]: pgmap v253: 97 pgs: 97 active+clean; 188 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 259 op/s 2024-04-23T18:26:40.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:40 smithi177 ceph-mon[32308]: pgmap v253: 97 pgs: 97 active+clean; 188 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 259 op/s 2024-04-23T18:26:40.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:40 smithi188 ceph-mon[32694]: pgmap v253: 97 pgs: 97 active+clean; 188 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 259 op/s 2024-04-23T18:26:41.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 533451 5.85 MB/sec execute 315 sec latency 101.007 ms 2024-04-23T18:26:42.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 535305 5.86 MB/sec execute 316 sec latency 20.593 ms 2024-04-23T18:26:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:42 smithi050 ceph-mon[29187]: pgmap v254: 97 pgs: 97 active+clean; 188 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 259 op/s 2024-04-23T18:26:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:42 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:42 smithi177 ceph-mon[32308]: pgmap v254: 97 pgs: 97 active+clean; 188 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 259 op/s 2024-04-23T18:26:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:42 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:42 smithi188 ceph-mon[32694]: pgmap v254: 97 pgs: 97 active+clean; 188 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 259 op/s 2024-04-23T18:26:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:42 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:43.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 537028 5.88 MB/sec execute 317 sec latency 15.166 ms 2024-04-23T18:26:43.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:43 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:26:43.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:43 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:26:43.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:43 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:26:44.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 538838 5.88 MB/sec execute 318 sec latency 66.409 ms 2024-04-23T18:26:44.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:44 smithi050 ceph-mon[29187]: pgmap v255: 97 pgs: 97 active+clean; 183 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 402 op/s 2024-04-23T18:26:44.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:44 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:26:44.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:44 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:26:44.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:44 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:26:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:44 smithi177 ceph-mon[32308]: pgmap v255: 97 pgs: 97 active+clean; 183 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 402 op/s 2024-04-23T18:26:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:44 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:26:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:44 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:26:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:44 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:26:44.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:44 smithi188 ceph-mon[32694]: pgmap v255: 97 pgs: 97 active+clean; 183 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 402 op/s 2024-04-23T18:26:44.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:44 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:26:44.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:44 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:26:44.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:44 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:26:45.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 540658 5.90 MB/sec execute 319 sec latency 21.100 ms 2024-04-23T18:26:46.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 542756 5.92 MB/sec execute 320 sec latency 14.788 ms 2024-04-23T18:26:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:46 smithi050 ceph-mon[29187]: pgmap v256: 97 pgs: 97 active+clean; 183 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.3 MiB/s wr, 250 op/s 2024-04-23T18:26:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:46 smithi177 ceph-mon[32308]: pgmap v256: 97 pgs: 97 active+clean; 183 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.3 MiB/s wr, 250 op/s 2024-04-23T18:26:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:46 smithi188 ceph-mon[32694]: pgmap v256: 97 pgs: 97 active+clean; 183 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.3 MiB/s wr, 250 op/s 2024-04-23T18:26:47.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 544708 5.93 MB/sec execute 321 sec latency 19.166 ms 2024-04-23T18:26:47.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:47 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:47.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:47 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:47.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:47 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:48.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 547199 5.96 MB/sec execute 322 sec latency 13.843 ms 2024-04-23T18:26:48.438 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:48 smithi177 ceph-mon[32308]: pgmap v257: 97 pgs: 97 active+clean; 190 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.7 MiB/s wr, 358 op/s 2024-04-23T18:26:48.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:48 smithi050 ceph-mon[29187]: pgmap v257: 97 pgs: 97 active+clean; 190 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.7 MiB/s wr, 358 op/s 2024-04-23T18:26:48.479 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:48 smithi188 ceph-mon[32694]: pgmap v257: 97 pgs: 97 active+clean; 190 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.7 MiB/s wr, 358 op/s 2024-04-23T18:26:49.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 548710 5.96 MB/sec execute 323 sec latency 42.682 ms 2024-04-23T18:26:50.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 550868 5.98 MB/sec execute 324 sec latency 31.372 ms 2024-04-23T18:26:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:50 smithi050 ceph-mon[29187]: pgmap v258: 97 pgs: 97 active+clean; 190 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 251 op/s 2024-04-23T18:26:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:50 smithi177 ceph-mon[32308]: pgmap v258: 97 pgs: 97 active+clean; 190 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 251 op/s 2024-04-23T18:26:50.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:50 smithi188 ceph-mon[32694]: pgmap v258: 97 pgs: 97 active+clean; 190 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 251 op/s 2024-04-23T18:26:51.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 552591 5.98 MB/sec execute 325 sec latency 17.053 ms 2024-04-23T18:26:52.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 554462 6.01 MB/sec execute 326 sec latency 155.965 ms 2024-04-23T18:26:52.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:52 smithi050 ceph-mon[29187]: pgmap v259: 97 pgs: 97 active+clean; 190 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 251 op/s 2024-04-23T18:26:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:52 smithi177 ceph-mon[32308]: pgmap v259: 97 pgs: 97 active+clean; 190 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 251 op/s 2024-04-23T18:26:52.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:52 smithi188 ceph-mon[32694]: pgmap v259: 97 pgs: 97 active+clean; 190 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 251 op/s 2024-04-23T18:26:53.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 556006 6.01 MB/sec execute 327 sec latency 214.102 ms 2024-04-23T18:26:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.512+0000 7feef2776640 1 -- 172.21.15.50:0/348064521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feeec1080e0 msgr2=0x7feeec1084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.512+0000 7feef2776640 1 --2- 172.21.15.50:0/348064521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feeec1080e0 0x7feeec1084c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7feedc009900 tx=0x7feedc02a480 comp rx=0 tx=0).stop 2024-04-23T18:26:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.512+0000 7feef2776640 1 -- 172.21.15.50:0/348064521 shutdown_connections 2024-04-23T18:26:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.512+0000 7feef2776640 1 --2- 172.21.15.50:0/348064521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeec06b120 0x7feeec10edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.512+0000 7feef2776640 1 --2- 172.21.15.50:0/348064521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec06a780 0x7feeec06abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.512+0000 7feef2776640 1 --2- 172.21.15.50:0/348064521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feeec1080e0 0x7feeec1084c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.512+0000 7feef2776640 1 -- 172.21.15.50:0/348064521 >> 172.21.15.50:0/348064521 conn(0x7feeec076c50 msgr2=0x7feeec077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.512+0000 7feef2776640 1 -- 172.21.15.50:0/348064521 shutdown_connections 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.512+0000 7feef2776640 1 -- 172.21.15.50:0/348064521 wait complete. 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feef2776640 1 Processor -- start 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feef2776640 1 -- start start 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feef2776640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feeec06a780 0x7feeec1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feef2776640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec06b120 0x7feeec198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feef2776640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeec1080e0 0x7feeec19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feef2776640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7feeec1121b0 con 0x7feeec06a780 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feef2776640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feeec112030 con 0x7feeec1080e0 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feef2776640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7feeec112330 con 0x7feeec06b120 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feeebfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feeec06a780 0x7feeec1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feeebfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feeec06a780 0x7feeec1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46278/0 (socket says 172.21.15.50:46278) 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feeebfff640 1 -- 172.21.15.50:0/2198280795 learned_addr learned my addr 172.21.15.50:0/2198280795 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feeeb7fe640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec06b120 0x7feeec198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.513+0000 7feef0d13640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeec1080e0 0x7feeec19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.514+0000 7feef0d13640 1 -- 172.21.15.50:0/2198280795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec06b120 msgr2=0x7feeec198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.514+0000 7feef0d13640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec06b120 0x7feeec198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.514+0000 7feef0d13640 1 -- 172.21.15.50:0/2198280795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feeec06a780 msgr2=0x7feeec1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.514+0000 7feef0d13640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feeec06a780 0x7feeec1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.514+0000 7feef0d13640 1 -- 172.21.15.50:0/2198280795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeec19d5e0 con 0x7feeec1080e0 2024-04-23T18:26:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.514+0000 7feef0d13640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeec1080e0 0x7feeec19ceb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feee4009fd0 tx=0x7feee400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.514+0000 7feee97fa640 1 -- 172.21.15.50:0/2198280795 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee400ce20 con 0x7feeec1080e0 2024-04-23T18:26:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.514+0000 7feee97fa640 1 -- 172.21.15.50:0/2198280795 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feee4005020 con 0x7feeec1080e0 2024-04-23T18:26:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.514+0000 7feef2776640 1 -- 172.21.15.50:0/2198280795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeec19d8d0 con 0x7feeec1080e0 2024-04-23T18:26:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.514+0000 7feee97fa640 1 -- 172.21.15.50:0/2198280795 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4010690 con 0x7feeec1080e0 2024-04-23T18:26:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.514+0000 7feef2776640 1 -- 172.21.15.50:0/2198280795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeec1a5160 con 0x7feeec1080e0 2024-04-23T18:26:53.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.516+0000 7feee97fa640 1 -- 172.21.15.50:0/2198280795 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7feee40054d0 con 0x7feeec1080e0 2024-04-23T18:26:53.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feef556f740 1 Processor -- start 2024-04-23T18:26:53.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feef556f740 1 -- start start 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feee97fa640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feed40a3dc0 0x7feed40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feef556f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488c540 0x55702488c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feef556f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x55702489b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feef556f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702489bc10 0x55702489e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feef556f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5570247f1970 con 0x55702488ce60 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feef556f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55702478e4a0 con 0x55702488c540 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feef556f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5570247ed930 con 0x55702489bc10 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feeeb7fe640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feed40a3dc0 0x7feed40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feef0d13640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702489bc10 0x55702489e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feeebfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488c540 0x55702488c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feeebfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488c540 0x55702488c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:46632/0 (socket says 172.21.15.50:46632) 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feeebfff640 1 -- 172.21.15.50:0/4208352412 learned_addr learned my addr 172.21.15.50:0/4208352412 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feeeb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x55702489b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feee97fa640 1 -- 172.21.15.50:0/2198280795 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7feee40d0f40 con 0x7feeec1080e0 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feeebfff640 1 -- 172.21.15.50:0/4208352412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702489bc10 msgr2=0x55702489e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feeebfff640 1 --2- 172.21.15.50:0/4208352412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702489bc10 0x55702489e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feeebfff640 1 -- 172.21.15.50:0/4208352412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 msgr2=0x55702489b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feeebfff640 1 --2- 172.21.15.50:0/4208352412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x55702489b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feeebfff640 1 -- 172.21.15.50:0/4208352412 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557024504210 con 0x55702488c540 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.517+0000 7feef0d13640 1 --2- 172.21.15.50:0/4208352412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702489bc10 0x55702489e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.518+0000 7feeebfff640 1 --2- 172.21.15.50:0/4208352412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488c540 0x55702488c920 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feedc000c00 tx=0x7feedc031b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9fe04249f8fa234 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.518+0000 7feeeb7fe640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feed40a3dc0 0x7feed40a6280 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7feee0009fd0 tx=0x7feee0009b80 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.518+0000 7feed27fc640 1 -- 172.21.15.50:0/4208352412 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feedc040070 con 0x55702488c540 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.518+0000 7feed27fc640 1 -- 172.21.15.50:0/4208352412 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feedc004030 con 0x55702488c540 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.518+0000 7feed27fc640 1 -- 172.21.15.50:0/4208352412 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feedc035040 con 0x55702488c540 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.518+0000 7feef556f740 1 -- 172.21.15.50:0/4208352412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488c540 msgr2=0x55702488c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.518+0000 7feef556f740 1 --2- 172.21.15.50:0/4208352412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488c540 0x55702488c920 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feedc000c00 tx=0x7feedc031b10 comp rx=0 tx=0).stop 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.518+0000 7feef556f740 1 -- 172.21.15.50:0/4208352412 shutdown_connections 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.518+0000 7feef556f740 1 --2- 172.21.15.50:0/4208352412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702489bc10 0x55702489e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.518+0000 7feef556f740 1 --2- 172.21.15.50:0/4208352412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x55702489b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.518+0000 7feef556f740 1 --2- 172.21.15.50:0/4208352412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488c540 0x55702488c920 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef556f740 1 -- 172.21.15.50:0/4208352412 >> 172.21.15.50:0/4208352412 conn(0x557024888b30 msgr2=0x557024888f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef556f740 1 -- 172.21.15.50:0/4208352412 shutdown_connections 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef556f740 1 -- 172.21.15.50:0/4208352412 wait complete. 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef556f740 1 -- start start 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef556f740 1 Processor -- start 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef556f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702488c540 0x55702489b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef556f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488ce60 0x5570248987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef556f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702489bc10 0x557024898cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef556f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55702489e7e0 con 0x55702489bc10 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef556f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55702478c0b0 con 0x55702488ce60 2024-04-23T18:26:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef556f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55702489e960 con 0x55702488c540 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef556f740 10 client.0 _ll_register_callbacks cb 0x557024779d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.519+0000 7feef0d13640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702489bc10 0x557024898cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feeebfff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702488c540 0x55702489b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feeeb7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488ce60 0x5570248987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feef0d13640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702489bc10 0x557024898cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46294/0 (socket says 172.21.15.50:46294) 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feef0d13640 1 -- 172.21.15.50:0/2587443199 learned_addr learned my addr 172.21.15.50:0/2587443199 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feeebfff640 1 -- 172.21.15.50:0/2587443199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488ce60 msgr2=0x5570248987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feeebfff640 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488ce60 0x5570248987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feeebfff640 1 -- 172.21.15.50:0/2587443199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702489bc10 msgr2=0x557024898cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feeebfff640 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702489bc10 0x557024898cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feeebfff640 1 -- 172.21.15.50:0/2587443199 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570245d0620 con 0x55702488c540 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feef0d13640 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702489bc10 0x557024898cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feeeb7fe640 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488ce60 0x5570248987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feeebfff640 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702488c540 0x55702489b4c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feedc005940 tx=0x7feedc005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42371d91d8114b13 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:53.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feecb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feecb7fe640 1 -- 172.21.15.50:0/2587443199 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feedc040070 con 0x55702488c540 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feef556f740 1 -- 172.21.15.50:0/2587443199 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feedc004030 con 0x55702488c540 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feecb7fe640 1 -- 172.21.15.50:0/2587443199 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7feedc005100 con 0x55702488c540 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feecb7fe640 1 -- 172.21.15.50:0/2587443199 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7feedc056050 con 0x55702488c540 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feef556f740 10 client.26131 fetch_fsmap learned FSMap version 35 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.520+0000 7feef556f740 1 -- 172.21.15.50:0/2587443199 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55702489e7e0 con 0x55702488c540 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.521+0000 7feecb7fe640 1 -- 172.21.15.50:0/2587443199 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feedc004030 con 0x55702488c540 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.521+0000 7feecb7fe640 1 -- 172.21.15.50:0/2587443199 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7feedc0035f0 con 0x55702488c540 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.521+0000 7feef556f740 10 client.26131 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.521+0000 7feef556f740 10 client.26131 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.521+0000 7feef556f740 20 client.26131 populate_metadata read hostname 'smithi050' 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.521+0000 7feef556f740 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5570248ab7d0 0x5570248adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.521+0000 7feef556f740 4 client.26131 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:26:53.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.521+0000 7feef556f740 1 -- 172.21.15.50:0/2587443199 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570247ff390 con 0x5570248ab7d0 2024-04-23T18:26:53.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.521+0000 7feeeb7fe640 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5570248ab7d0 0x5570248adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.521+0000 7feeeb7fe640 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5570248ab7d0 0x5570248adbc0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c0fc6b86f409bbd server_cookie=93a7e6b419992f08 in_seq=0 out_seq=0 2024-04-23T18:26:53.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.521+0000 7feecb7fe640 10 client.26131 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feecb7fe640 1 -- 172.21.15.50:0/2587443199 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5570247ff390 con 0x5570248ab7d0 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feecb7fe640 10 client.26131 handle_command_reply: tid=0 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feef556f740 1 client.26131 shutdown 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feef556f740 1 -- 172.21.15.50:0/2587443199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702488c540 msgr2=0x55702489b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feef556f740 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702488c540 0x55702489b4c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feedc005940 tx=0x7feedc005e30 comp rx=0 tx=0).stop 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feef556f740 1 -- 172.21.15.50:0/2587443199 shutdown_connections 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feef556f740 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5570248ab7d0 0x5570248adbc0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feef556f740 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702489bc10 0x557024898cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feecb7fe640 0 client.26131 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feef556f740 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488ce60 0x5570248987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feef556f740 1 --2- 172.21.15.50:0/2587443199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702488c540 0x55702489b4c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feef556f740 1 -- 172.21.15.50:0/2587443199 >> 172.21.15.50:0/2587443199 conn(0x557024888b30 msgr2=0x55702488e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feef556f740 1 -- 172.21.15.50:0/2587443199 shutdown_connections 2024-04-23T18:26:53.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.522+0000 7feef556f740 1 -- 172.21.15.50:0/2587443199 wait complete. 2024-04-23T18:26:53.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.523+0000 7feef556f740 20 client.26131 trim_cache size 0 max 16384 2024-04-23T18:26:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.538+0000 7feef556f740 1 Processor -- start 2024-04-23T18:26:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.538+0000 7feef556f740 1 -- start start 2024-04-23T18:26:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.538+0000 7feef556f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557024888b30 0x5570248a76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.538+0000 7feef556f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702488c540 0x557024893f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.538+0000 7feef556f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x557024883b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.538+0000 7feef556f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7feeec1ae180 con 0x55702488ce60 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.538+0000 7feef556f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feeec1ab9f0 con 0x557024888b30 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.538+0000 7feef556f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7feeec1a9860 con 0x55702488c540 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feef0d13640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x557024883b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feeebfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557024888b30 0x5570248a76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feeebfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557024888b30 0x5570248a76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:46644/0 (socket says 172.21.15.50:46644) 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feef0d13640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x557024883b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46296/0 (socket says 172.21.15.50:46296) 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feeebfff640 1 -- 172.21.15.50:0/3238642743 learned_addr learned my addr 172.21.15.50:0/3238642743 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feeeb7fe640 1 --2- 172.21.15.50:0/3238642743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702488c540 0x557024893f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feef0d13640 1 -- 172.21.15.50:0/3238642743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702488c540 msgr2=0x557024893f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feef0d13640 1 --2- 172.21.15.50:0/3238642743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702488c540 0x557024893f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feef0d13640 1 -- 172.21.15.50:0/3238642743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557024888b30 msgr2=0x5570248a76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feef0d13640 1 --2- 172.21.15.50:0/3238642743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557024888b30 0x5570248a76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feef0d13640 1 -- 172.21.15.50:0/3238642743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570248842e0 con 0x55702488ce60 2024-04-23T18:26:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feef0d13640 1 --2- 172.21.15.50:0/3238642743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x557024883b10 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7feee409cb70 tx=0x7feee400af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b1d53faf87cdd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feed1ffb640 1 -- 172.21.15.50:0/3238642743 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee401c1c0 con 0x55702488ce60 2024-04-23T18:26:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feed1ffb640 1 -- 172.21.15.50:0/3238642743 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feee409b170 con 0x55702488ce60 2024-04-23T18:26:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.539+0000 7feed1ffb640 1 -- 172.21.15.50:0/3238642743 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee40a5650 con 0x55702488ce60 2024-04-23T18:26:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.540+0000 7feef556f740 1 -- 172.21.15.50:0/3238642743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 msgr2=0x557024883b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.540+0000 7feef556f740 1 --2- 172.21.15.50:0/3238642743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x557024883b10 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7feee409cb70 tx=0x7feee400af00 comp rx=0 tx=0).stop 2024-04-23T18:26:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.540+0000 7feef556f740 1 -- 172.21.15.50:0/3238642743 shutdown_connections 2024-04-23T18:26:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.540+0000 7feef556f740 1 --2- 172.21.15.50:0/3238642743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x557024883b10 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.540+0000 7feef556f740 1 --2- 172.21.15.50:0/3238642743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55702488c540 0x557024893f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.540+0000 7feef556f740 1 --2- 172.21.15.50:0/3238642743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557024888b30 0x5570248a76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.540+0000 7feef556f740 1 -- 172.21.15.50:0/3238642743 >> 172.21.15.50:0/3238642743 conn(0x557024898e50 msgr2=0x557024899fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.540+0000 7feef556f740 1 -- 172.21.15.50:0/3238642743 shutdown_connections 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.540+0000 7feef556f740 1 -- 172.21.15.50:0/3238642743 wait complete. 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.540+0000 7feef556f740 1 -- start start 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.540+0000 7feef556f740 1 Processor -- start 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feef556f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557024888b30 0x557024895550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feef556f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488c540 0x557024895a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feef556f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x557024895fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feef556f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7feeec1a9eb0 con 0x55702488ce60 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feef556f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feeec1ab320 con 0x55702488c540 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feef556f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7feeec1aa580 con 0x557024888b30 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feef556f740 10 client.0 _ll_register_callbacks cb 0x7feedc009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feeebfff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557024888b30 0x557024895550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feef0d13640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x557024895fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feeebfff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557024888b30 0x557024895550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:41642/0 (socket says 172.21.15.50:41642) 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feeebfff640 1 -- 172.21.15.50:0/2597244117 learned_addr learned my addr 172.21.15.50:0/2597244117 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feeeb7fe640 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488c540 0x557024895a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feeebfff640 1 -- 172.21.15.50:0/2597244117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488c540 msgr2=0x557024895a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feeebfff640 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488c540 0x557024895a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feeebfff640 1 -- 172.21.15.50:0/2597244117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 msgr2=0x557024895fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feeebfff640 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x557024895fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feeebfff640 1 -- 172.21.15.50:0/2597244117 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570248adf30 con 0x557024888b30 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feef0d13640 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x557024895fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feeebfff640 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557024888b30 0x557024895550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feedc003a10 tx=0x7feedc005020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fd8270b1077f735 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.541+0000 7feecb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feecb7fe640 1 -- 172.21.15.50:0/2597244117 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feedc040040 con 0x557024888b30 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feecb7fe640 1 -- 172.21.15.50:0/2597244117 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7feedc003e10 con 0x557024888b30 2024-04-23T18:26:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feef556f740 1 -- 172.21.15.50:0/2597244117 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feee409b170 con 0x557024888b30 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feecb7fe640 1 -- 172.21.15.50:0/2597244117 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7feedc068050 con 0x557024888b30 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feef556f740 10 client.26137 fetch_fsmap learned FSMap version 35 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feef556f740 1 -- 172.21.15.50:0/2597244117 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557024911e20 con 0x557024888b30 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feecb7fe640 1 -- 172.21.15.50:0/2597244117 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feee409b170 con 0x557024888b30 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feecb7fe640 1 -- 172.21.15.50:0/2597244117 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7feedc004770 con 0x557024888b30 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feef556f740 10 client.26137 fetch_fsmap finished waiting for FSMap version 35 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feef556f740 10 client.26137 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feef556f740 20 client.26137 populate_metadata read hostname 'smithi050' 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feef556f740 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55702489bc10 0x557024914480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feef556f740 4 client.26137 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feef556f740 1 -- 172.21.15.50:0/2597244117 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5570247f9200 con 0x55702489bc10 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.542+0000 7feeeb7fe640 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55702489bc10 0x557024914480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.543+0000 7feeeb7fe640 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55702489bc10 0x557024914480 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73d5c683e062ad38 server_cookie=76fa623404a2c151 in_seq=0 out_seq=0 2024-04-23T18:26:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.543+0000 7feecb7fe640 10 client.26137 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:53.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feecb7fe640 1 -- 172.21.15.50:0/2597244117 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5570247f9200 con 0x55702489bc10 2024-04-23T18:26:53.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feecb7fe640 10 client.26137 handle_command_reply: tid=0 2024-04-23T18:26:53.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feef556f740 1 client.26137 shutdown 2024-04-23T18:26:53.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feef556f740 1 -- 172.21.15.50:0/2597244117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557024888b30 msgr2=0x557024895550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:53.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feef556f740 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557024888b30 0x557024895550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feedc003a10 tx=0x7feedc005020 comp rx=0 tx=0).stop 2024-04-23T18:26:53.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feef556f740 1 -- 172.21.15.50:0/2597244117 shutdown_connections 2024-04-23T18:26:53.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feef556f740 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55702489bc10 0x557024914480 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feef556f740 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55702488ce60 0x557024895fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feef556f740 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55702488c540 0x557024895a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feef556f740 1 --2- 172.21.15.50:0/2597244117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557024888b30 0x557024895550 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feecb7fe640 0 client.26137 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:26:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feef556f740 1 -- 172.21.15.50:0/2597244117 >> 172.21.15.50:0/2597244117 conn(0x557024898e50 msgr2=0x5570248977e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feef556f740 1 -- 172.21.15.50:0/2597244117 shutdown_connections 2024-04-23T18:26:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feef556f740 1 -- 172.21.15.50:0/2597244117 wait complete. 2024-04-23T18:26:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.599+0000 7feef556f740 20 client.26137 trim_cache size 0 max 16384 2024-04-23T18:26:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:26:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:26:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:26:53.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:26:53.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.600+0000 7feef2776640 1 -- 172.21.15.50:0/2198280795 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feed40a3dc0 msgr2=0x7feed40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.600+0000 7feef2776640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feed40a3dc0 0x7feed40a6280 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7feee0009fd0 tx=0x7feee0009b80 comp rx=0 tx=0).stop 2024-04-23T18:26:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.600+0000 7feef2776640 1 -- 172.21.15.50:0/2198280795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeec1080e0 msgr2=0x7feeec19ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:26:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.600+0000 7feef2776640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeec1080e0 0x7feeec19ceb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feee4009fd0 tx=0x7feee400ef10 comp rx=0 tx=0).stop 2024-04-23T18:26:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.600+0000 7feef2776640 1 -- 172.21.15.50:0/2198280795 shutdown_connections 2024-04-23T18:26:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.600+0000 7feef2776640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feed40a3dc0 0x7feed40a6280 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.600+0000 7feef2776640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeec1080e0 0x7feeec19ceb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.600+0000 7feef2776640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec06b120 0x7feeec198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.600+0000 7feef2776640 1 --2- 172.21.15.50:0/2198280795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feeec06a780 0x7feeec1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:26:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.600+0000 7feef2776640 1 -- 172.21.15.50:0/2198280795 >> 172.21.15.50:0/2198280795 conn(0x7feeec076c50 msgr2=0x7feeec0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:26:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.600+0000 7feef2776640 1 -- 172.21.15.50:0/2198280795 shutdown_connections 2024-04-23T18:26:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:26:53.600+0000 7feef2776640 1 -- 172.21.15.50:0/2198280795 wait complete. 2024-04-23T18:26:54.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 557865 6.03 MB/sec execute 328 sec latency 176.816 ms 2024-04-23T18:26:54.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:54 smithi050 ceph-mon[29187]: pgmap v260: 97 pgs: 97 active+clean; 197 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 367 op/s 2024-04-23T18:26:54.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:54 smithi177 ceph-mon[32308]: pgmap v260: 97 pgs: 97 active+clean; 197 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 367 op/s 2024-04-23T18:26:54.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:54 smithi188 ceph-mon[32694]: pgmap v260: 97 pgs: 97 active+clean; 197 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 367 op/s 2024-04-23T18:26:55.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 559299 6.03 MB/sec execute 329 sec latency 25.365 ms 2024-04-23T18:26:56.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 561575 6.05 MB/sec execute 330 sec latency 14.740 ms 2024-04-23T18:26:56.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:56 smithi050 ceph-mon[29187]: pgmap v261: 97 pgs: 97 active+clean; 197 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 223 op/s 2024-04-23T18:26:56.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:56 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:56.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:56 smithi177 ceph-mon[32308]: pgmap v261: 97 pgs: 97 active+clean; 197 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 223 op/s 2024-04-23T18:26:56.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:56 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:56.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:56 smithi188 ceph-mon[32694]: pgmap v261: 97 pgs: 97 active+clean; 197 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 223 op/s 2024-04-23T18:26:56.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:56 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:26:57.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 563455 6.05 MB/sec execute 331 sec latency 22.621 ms 2024-04-23T18:26:58.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 565277 6.07 MB/sec execute 332 sec latency 23.105 ms 2024-04-23T18:26:58.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:26:58 smithi050 ceph-mon[29187]: pgmap v262: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 362 op/s 2024-04-23T18:26:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:26:58 smithi177 ceph-mon[32308]: pgmap v262: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 362 op/s 2024-04-23T18:26:58.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:26:58 smithi188 ceph-mon[32694]: pgmap v262: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 362 op/s 2024-04-23T18:26:59.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 567203 6.07 MB/sec execute 333 sec latency 47.116 ms 2024-04-23T18:27:00.219 INFO:tasks.workunit.client.0.smithi050.stdout: 1 568911 6.09 MB/sec execute 334 sec latency 24.617 ms 2024-04-23T18:27:00.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:00 smithi050 ceph-mon[29187]: pgmap v263: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 253 op/s 2024-04-23T18:27:00.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:00 smithi177 ceph-mon[32308]: pgmap v263: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 253 op/s 2024-04-23T18:27:00.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:00 smithi188 ceph-mon[32694]: pgmap v263: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 253 op/s 2024-04-23T18:27:01.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 570629 6.09 MB/sec execute 335 sec latency 28.732 ms 2024-04-23T18:27:02.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 572295 6.11 MB/sec execute 336 sec latency 21.056 ms 2024-04-23T18:27:02.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:02 smithi050 ceph-mon[29187]: pgmap v264: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 253 op/s 2024-04-23T18:27:02.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:02 smithi177 ceph-mon[32308]: pgmap v264: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 253 op/s 2024-04-23T18:27:02.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:02 smithi188 ceph-mon[32694]: pgmap v264: 97 pgs: 97 active+clean; 191 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 253 op/s 2024-04-23T18:27:03.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 574088 6.11 MB/sec execute 337 sec latency 44.189 ms 2024-04-23T18:27:04.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 575827 6.13 MB/sec execute 338 sec latency 33.336 ms 2024-04-23T18:27:04.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:04 smithi050 ceph-mon[29187]: pgmap v265: 97 pgs: 97 active+clean; 198 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 396 op/s 2024-04-23T18:27:04.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:04 smithi177 ceph-mon[32308]: pgmap v265: 97 pgs: 97 active+clean; 198 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 396 op/s 2024-04-23T18:27:04.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:04 smithi188 ceph-mon[32694]: pgmap v265: 97 pgs: 97 active+clean; 198 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 396 op/s 2024-04-23T18:27:05.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 577210 6.13 MB/sec execute 339 sec latency 32.757 ms 2024-04-23T18:27:05.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:05 smithi050 ceph-mon[29187]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T18:27:05.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:05 smithi177 ceph-mon[32308]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T18:27:05.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:05 smithi188 ceph-mon[32694]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T18:27:06.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 579360 6.15 MB/sec execute 340 sec latency 28.043 ms 2024-04-23T18:27:06.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:06 smithi050 ceph-mon[29187]: pgmap v267: 97 pgs: 97 active+clean; 198 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-04-23T18:27:06.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:06 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:06.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:06 smithi177 ceph-mon[32308]: pgmap v267: 97 pgs: 97 active+clean; 198 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-04-23T18:27:06.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:06 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:06.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:06 smithi188 ceph-mon[32694]: pgmap v267: 97 pgs: 97 active+clean; 198 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 337 op/s 2024-04-23T18:27:06.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:06 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.693+0000 7f1e6919b640 1 -- 172.21.15.50:0/638137351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e64106170 msgr2=0x7f1e6410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.693+0000 7f1e6919b640 1 --2- 172.21.15.50:0/638137351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e64106170 0x7f1e6410dd40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009ca0 tx=0x7f1e5c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.693+0000 7f1e6919b640 1 -- 172.21.15.50:0/638137351 shutdown_connections 2024-04-23T18:27:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.693+0000 7f1e6919b640 1 --2- 172.21.15.50:0/638137351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e64106170 0x7f1e6410dd40 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.693+0000 7f1e6919b640 1 --2- 172.21.15.50:0/638137351 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e64101aa0 0x7f1e64105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.693+0000 7f1e6919b640 1 --2- 172.21.15.50:0/638137351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e641010f0 0x7f1e641014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.693+0000 7f1e6919b640 1 -- 172.21.15.50:0/638137351 >> 172.21.15.50:0/638137351 conn(0x7f1e64077fd0 msgr2=0x7f1e640783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.693+0000 7f1e6919b640 1 -- 172.21.15.50:0/638137351 shutdown_connections 2024-04-23T18:27:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.693+0000 7f1e6919b640 1 -- 172.21.15.50:0/638137351 wait complete. 2024-04-23T18:27:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.693+0000 7f1e6919b640 1 Processor -- start 2024-04-23T18:27:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e6919b640 1 -- start start 2024-04-23T18:27:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e6919b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e641010f0 0x7f1e64198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e6919b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e64106170 0x7f1e64198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e6919b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e6419cf20 0x7f1e6419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e6919b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1e64112220 con 0x7f1e6419cf20 2024-04-23T18:27:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e6919b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1e641120a0 con 0x7f1e641010f0 2024-04-23T18:27:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e6919b640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1e641123a0 con 0x7f1e64106170 2024-04-23T18:27:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e6259c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e64106170 0x7f1e64198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e62d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e641010f0 0x7f1e64198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e6359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e6419cf20 0x7f1e6419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e6259c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e64106170 0x7f1e64198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:40386/0 (socket says 172.21.15.50:40386) 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e62d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e641010f0 0x7f1e64198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:57174/0 (socket says 172.21.15.50:57174) 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e6259c640 1 -- 172.21.15.50:0/1946891740 learned_addr learned my addr 172.21.15.50:0/1946891740 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e62d9d640 1 -- 172.21.15.50:0/1946891740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e64106170 msgr2=0x7f1e64198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e62d9d640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e64106170 0x7f1e64198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e62d9d640 1 -- 172.21.15.50:0/1946891740 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e6419cf20 msgr2=0x7f1e6419d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e62d9d640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e6419cf20 0x7f1e6419d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e62d9d640 1 -- 172.21.15.50:0/1946891740 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6419da10 con 0x7f1e641010f0 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.694+0000 7f1e6259c640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e64106170 0x7f1e64198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.695+0000 7f1e62d9d640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e641010f0 0x7f1e64198660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400e9f0 tx=0x7f1e5400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.695+0000 7f1e4bfff640 1 -- 172.21.15.50:0/1946891740 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5400ccc0 con 0x7f1e641010f0 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.695+0000 7f1e4bfff640 1 -- 172.21.15.50:0/1946891740 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e540108e0 con 0x7f1e641010f0 2024-04-23T18:27:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.695+0000 7f1e6919b640 1 -- 172.21.15.50:0/1946891740 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e6419dd00 con 0x7f1e641010f0 2024-04-23T18:27:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.695+0000 7f1e4bfff640 1 -- 172.21.15.50:0/1946891740 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54018880 con 0x7f1e641010f0 2024-04-23T18:27:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.695+0000 7f1e6919b640 1 -- 172.21.15.50:0/1946891740 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e641a5540 con 0x7f1e641010f0 2024-04-23T18:27:06.697 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.697+0000 7f1e6bf8e740 1 Processor -- start 2024-04-23T18:27:06.697 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6bf8e740 1 -- start start 2024-04-23T18:27:06.697 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6bf8e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 0x5600455fba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6bf8e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600455fbf40 0x5600456062c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6bf8e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560045606800 0x560045608bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6bf8e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560045560970 con 0x5600455fbf40 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6bf8e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5600454fd4a0 con 0x5600455fb620 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6bf8e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56004555c930 con 0x560045606800 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e4bfff640 1 -- 172.21.15.50:0/1946891740 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1e54010430 con 0x7f1e641010f0 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6359e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 0x5600455fba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6259c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560045606800 0x560045608bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6359e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 0x5600455fba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:57178/0 (socket says 172.21.15.50:57178) 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6359e640 1 -- 172.21.15.50:0/2335086377 learned_addr learned my addr 172.21.15.50:0/2335086377 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e62d9d640 1 --2- 172.21.15.50:0/2335086377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600455fbf40 0x5600456062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6359e640 1 -- 172.21.15.50:0/2335086377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560045606800 msgr2=0x560045608bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6359e640 1 --2- 172.21.15.50:0/2335086377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560045606800 0x560045608bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6359e640 1 -- 172.21.15.50:0/2335086377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600455fbf40 msgr2=0x5600456062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6359e640 1 --2- 172.21.15.50:0/2335086377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600455fbf40 0x5600456062c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6359e640 1 -- 172.21.15.50:0/2335086377 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560045273210 con 0x5600455fb620 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.698+0000 7f1e6359e640 1 --2- 172.21.15.50:0/2335086377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 0x5600455fba00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009ca0 tx=0x7f1e5c0093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4d1bec8794475b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e497fa640 1 -- 172.21.15.50:0/2335086377 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c031cf0 con 0x5600455fb620 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e4bfff640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1e4c0a3e40 0x7f1e4c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e497fa640 1 -- 172.21.15.50:0/2335086377 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e5c036030 con 0x5600455fb620 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e497fa640 1 -- 172.21.15.50:0/2335086377 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c0042a0 con 0x5600455fb620 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e62d9d640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1e4c0a3e40 0x7f1e4c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e4bfff640 1 -- 172.21.15.50:0/1946891740 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f1e540d0ac0 con 0x7f1e641010f0 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2335086377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 msgr2=0x5600455fba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e62d9d640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1e4c0a3e40 0x7f1e4c0a6300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e540ce280 tx=0x7f1e54009700 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/2335086377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 0x5600455fba00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009ca0 tx=0x7f1e5c0093e0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2335086377 shutdown_connections 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/2335086377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x560045606800 0x560045608bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/2335086377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600455fbf40 0x5600456062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/2335086377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 0x5600455fba00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2335086377 >> 172.21.15.50:0/2335086377 conn(0x5600455f7b30 msgr2=0x5600455f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.699+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2335086377 shutdown_connections 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2335086377 wait complete. 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6bf8e740 1 -- start start 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6bf8e740 1 Processor -- start 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6bf8e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 0x560045606090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6bf8e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fbf40 0x5600456033f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6bf8e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045606800 0x560045603930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6bf8e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x560045609250 con 0x560045606800 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6bf8e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5600454fb0b0 con 0x5600455fb620 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6bf8e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5600456093d0 con 0x5600455fbf40 2024-04-23T18:27:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6bf8e740 10 client.0 _ll_register_callbacks cb 0x5600454e8d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6359e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 0x560045606090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045606800 0x560045603930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6259c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fbf40 0x5600456033f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045606800 0x560045603930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32796/0 (socket says 172.21.15.50:32796) 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6359e640 1 -- 172.21.15.50:0/2103464226 learned_addr learned my addr 172.21.15.50:0/2103464226 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.700+0000 7f1e6259c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fbf40 0x5600456033f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:40410/0 (socket says 172.21.15.50:40410) 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e6259c640 1 -- 172.21.15.50:0/2103464226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 msgr2=0x560045606090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e6259c640 1 --2- 172.21.15.50:0/2103464226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 0x560045606090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e6259c640 1 -- 172.21.15.50:0/2103464226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045606800 msgr2=0x560045603930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e6259c640 1 --2- 172.21.15.50:0/2103464226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045606800 0x560045603930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e6259c640 1 -- 172.21.15.50:0/2103464226 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56004533f620 con 0x5600455fbf40 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e6359e640 1 --2- 172.21.15.50:0/2103464226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 0x560045606090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e6259c640 1 --2- 172.21.15.50:0/2103464226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fbf40 0x5600456033f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x5600455f8cc0 tx=0x7f1e5800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb9888b4c80ff788 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e46ff5640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e46ff5640 1 -- 172.21.15.50:0/2103464226 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e58012110 con 0x5600455fbf40 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2103464226 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e5c036030 con 0x5600455fbf40 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e46ff5640 1 -- 172.21.15.50:0/2103464226 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f1e58004050 con 0x5600455fbf40 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e46ff5640 1 -- 172.21.15.50:0/2103464226 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1e580122b0 con 0x5600455fbf40 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e6bf8e740 10 client.26149 fetch_fsmap learned FSMap version 37 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.701+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2103464226 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560045615160 con 0x5600455fbf40 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.702+0000 7f1e46ff5640 1 -- 172.21.15.50:0/2103464226 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e5c036030 con 0x5600455fbf40 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.702+0000 7f1e46ff5640 1 -- 172.21.15.50:0/2103464226 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f1e580105f0 con 0x5600455fbf40 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.702+0000 7f1e6bf8e740 10 client.26149 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.702+0000 7f1e6bf8e740 10 client.26149 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.702+0000 7f1e6bf8e740 20 client.26149 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.702+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/2103464226 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560045616620 0x560045618a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.702+0000 7f1e6bf8e740 4 client.26149 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.702+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2103464226 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56004556e390 con 0x560045616620 2024-04-23T18:27:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.702+0000 7f1e6359e640 1 --2- 172.21.15.50:0/2103464226 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560045616620 0x560045618a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.703+0000 7f1e46ff5640 10 client.26149 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.703+0000 7f1e6359e640 1 --2- 172.21.15.50:0/2103464226 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560045616620 0x560045618a10 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a85ec319a93d2bc4 server_cookie=5315e27bb73a47ab in_seq=0 out_seq=0 2024-04-23T18:27:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.703+0000 7f1e46ff5640 1 -- 172.21.15.50:0/2103464226 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56004556e390 con 0x560045616620 2024-04-23T18:27:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.703+0000 7f1e46ff5640 10 client.26149 handle_command_reply: tid=0 2024-04-23T18:27:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.703+0000 7f1e6bf8e740 1 client.26149 shutdown 2024-04-23T18:27:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.704+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2103464226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fbf40 msgr2=0x5600456033f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.704+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/2103464226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fbf40 0x5600456033f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x5600455f8cc0 tx=0x7f1e5800f790 comp rx=0 tx=0).stop 2024-04-23T18:27:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.704+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2103464226 shutdown_connections 2024-04-23T18:27:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.704+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/2103464226 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x560045616620 0x560045618a10 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.704+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/2103464226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045606800 0x560045603930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.704+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/2103464226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fbf40 0x5600456033f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.704+0000 7f1e46ff5640 0 client.26149 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.704+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/2103464226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5600455fb620 0x560045606090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.704+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2103464226 >> 172.21.15.50:0/2103464226 conn(0x5600455f7b30 msgr2=0x5600455fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.704+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2103464226 shutdown_connections 2024-04-23T18:27:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.704+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/2103464226 wait complete. 2024-04-23T18:27:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.704+0000 7f1e6bf8e740 20 client.26149 trim_cache size 0 max 16384 2024-04-23T18:27:06.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6bf8e740 1 Processor -- start 2024-04-23T18:27:06.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6bf8e740 1 -- start start 2024-04-23T18:27:06.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6bf8e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fb620 0x560045612110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6bf8e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560045606800 0x5600455fe970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6bf8e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 0x560045605c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6bf8e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1e641b0530 con 0x560045603a20 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6bf8e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1e641adda0 con 0x560045606800 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6bf8e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1e641abc10 con 0x5600455fb620 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 0x560045605c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 0x560045605c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32812/0 (socket says 172.21.15.50:32812) 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6359e640 1 -- 172.21.15.50:0/3128563422 learned_addr learned my addr 172.21.15.50:0/3128563422 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6259c640 1 --2- 172.21.15.50:0/3128563422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560045606800 0x5600455fe970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6359e640 1 --2- 172.21.15.50:0/3128563422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fb620 0x560045612110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6359e640 1 -- 172.21.15.50:0/3128563422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fb620 msgr2=0x560045612110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6359e640 1 --2- 172.21.15.50:0/3128563422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fb620 0x560045612110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6359e640 1 -- 172.21.15.50:0/3128563422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560045606800 msgr2=0x5600455fe970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6359e640 1 --2- 172.21.15.50:0/3128563422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560045606800 0x5600455fe970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.720+0000 7f1e6359e640 1 -- 172.21.15.50:0/3128563422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560045618d80 con 0x560045603a20 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.721+0000 7f1e6359e640 1 --2- 172.21.15.50:0/3128563422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 0x560045605c30 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c02ade0 tx=0x7f1e5c005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff886248a7db28 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.721+0000 7f1e48ff9640 1 -- 172.21.15.50:0/3128563422 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c04a040 con 0x560045603a20 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.721+0000 7f1e48ff9640 1 -- 172.21.15.50:0/3128563422 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e5c036050 con 0x560045603a20 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.721+0000 7f1e48ff9640 1 -- 172.21.15.50:0/3128563422 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c004640 con 0x560045603a20 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.721+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/3128563422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 msgr2=0x560045605c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.721+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/3128563422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 0x560045605c30 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c02ade0 tx=0x7f1e5c005df0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.721+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/3128563422 shutdown_connections 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.721+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/3128563422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 0x560045605c30 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.721+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/3128563422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560045606800 0x5600455fe970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.721+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/3128563422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fb620 0x560045612110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.721+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/3128563422 >> 172.21.15.50:0/3128563422 conn(0x5600455f7b30 msgr2=0x5600455fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.721+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/3128563422 shutdown_connections 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/3128563422 wait complete. 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6bf8e740 1 -- start start 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6bf8e740 1 Processor -- start 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6bf8e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fb620 0x56004560e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6bf8e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 0x5600455ffcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6bf8e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560045606800 0x560045600200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6bf8e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1e641ac260 con 0x560045603a20 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6bf8e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1e641ad6d0 con 0x560045606800 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6bf8e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1e641ac930 con 0x5600455fb620 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6bf8e740 10 client.0 _ll_register_callbacks cb 0x7f1e58012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 0x5600455ffcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 0x5600455ffcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32826/0 (socket says 172.21.15.50:32826) 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6259c640 1 -- 172.21.15.50:0/378487904 learned_addr learned my addr 172.21.15.50:0/378487904 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6359e640 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fb620 0x56004560e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.722+0000 7f1e6359e640 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560045606800 0x560045600200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e6259c640 1 -- 172.21.15.50:0/378487904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fb620 msgr2=0x56004560e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e6259c640 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fb620 0x56004560e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e6259c640 1 -- 172.21.15.50:0/378487904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560045606800 msgr2=0x560045600200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e6259c640 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560045606800 0x560045600200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e6259c640 1 -- 172.21.15.50:0/378487904 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600456008c0 con 0x560045603a20 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e6359e640 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560045606800 0x560045600200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e6259c640 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 0x5600455ffcc0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1e58009600 tx=0x7f1e58009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f858eb8d4a64b3d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e46ff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e46ff5640 1 -- 172.21.15.50:0/378487904 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e58010640 con 0x560045603a20 2024-04-23T18:27:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e46ff5640 1 -- 172.21.15.50:0/378487904 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f1e5801c420 con 0x560045603a20 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/378487904 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e5c036050 con 0x560045603a20 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e46ff5640 1 -- 172.21.15.50:0/378487904 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1e5801c620 con 0x560045603a20 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e6bf8e740 10 client.16524 fetch_fsmap learned FSMap version 37 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/378487904 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56004567ca20 con 0x560045603a20 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e46ff5640 1 -- 172.21.15.50:0/378487904 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e5c036050 con 0x560045603a20 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e46ff5640 1 -- 172.21.15.50:0/378487904 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f1e580356c0 con 0x560045603a20 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.723+0000 7f1e6bf8e740 10 client.16524 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.724+0000 7f1e6bf8e740 10 client.16524 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.724+0000 7f1e6bf8e740 20 client.16524 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.724+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56004567d190 0x56004567f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.724+0000 7f1e6bf8e740 4 client.16524 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.724+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/378487904 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560045568200 con 0x56004567d190 2024-04-23T18:27:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.724+0000 7f1e6359e640 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56004567d190 0x56004567f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.724+0000 7f1e6359e640 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56004567d190 0x56004567f5b0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9db3b98e8e2db0 server_cookie=c8934822d87fdfb3 in_seq=0 out_seq=0 2024-04-23T18:27:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.724+0000 7f1e46ff5640 10 client.16524 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e46ff5640 1 -- 172.21.15.50:0/378487904 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560045568200 con 0x56004567d190 2024-04-23T18:27:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e46ff5640 10 client.16524 handle_command_reply: tid=0 2024-04-23T18:27:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e6bf8e740 1 client.16524 shutdown 2024-04-23T18:27:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/378487904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 msgr2=0x5600455ffcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 0x5600455ffcc0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1e58009600 tx=0x7f1e58009c80 comp rx=0 tx=0).stop 2024-04-23T18:27:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/378487904 shutdown_connections 2024-04-23T18:27:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56004567d190 0x56004567f5b0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560045606800 0x560045600200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x560045603a20 0x5600455ffcc0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e46ff5640 0 client.16524 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e6bf8e740 1 --2- 172.21.15.50:0/378487904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5600455fb620 0x56004560e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/378487904 >> 172.21.15.50:0/378487904 conn(0x5600455f7b30 msgr2=0x5600455fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/378487904 shutdown_connections 2024-04-23T18:27:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.784+0000 7f1e6bf8e740 1 -- 172.21.15.50:0/378487904 wait complete. 2024-04-23T18:27:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6bf8e740 20 client.16524 trim_cache size 0 max 16384 2024-04-23T18:27:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:27:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:27:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:27:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:27:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6919b640 1 -- 172.21.15.50:0/1946891740 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1e4c0a3e40 msgr2=0x7f1e4c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6919b640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1e4c0a3e40 0x7f1e4c0a6300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e540ce280 tx=0x7f1e54009700 comp rx=0 tx=0).stop 2024-04-23T18:27:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6919b640 1 -- 172.21.15.50:0/1946891740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e641010f0 msgr2=0x7f1e64198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6919b640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e641010f0 0x7f1e64198660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400e9f0 tx=0x7f1e5400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6919b640 1 -- 172.21.15.50:0/1946891740 shutdown_connections 2024-04-23T18:27:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6919b640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1e4c0a3e40 0x7f1e4c0a6300 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6919b640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e6419cf20 0x7f1e6419d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6919b640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e64106170 0x7f1e64198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6919b640 1 --2- 172.21.15.50:0/1946891740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e641010f0 0x7f1e64198660 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6919b640 1 -- 172.21.15.50:0/1946891740 >> 172.21.15.50:0/1946891740 conn(0x7f1e64077fd0 msgr2=0x7f1e640fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6919b640 1 -- 172.21.15.50:0/1946891740 shutdown_connections 2024-04-23T18:27:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:06.785+0000 7f1e6919b640 1 -- 172.21.15.50:0/1946891740 wait complete. 2024-04-23T18:27:07.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 580672 6.15 MB/sec execute 341 sec latency 97.789 ms 2024-04-23T18:27:07.787 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:27:07 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: ignoring --setuser ceph since I am not root 2024-04-23T18:27:07.787 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:27:07 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: ignoring --setgroup ceph since I am not root 2024-04-23T18:27:07.787 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:27:07 smithi177 ceph-mds[66745]: -- 172.21.15.177:0/193027115 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5608b7023a40 con 0x5608b6ff7000 2024-04-23T18:27:07.787 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:27:07 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: starting mds.h at 2024-04-23T18:27:08.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 582862 6.17 MB/sec execute 342 sec latency 14.170 ms 2024-04-23T18:27:08.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:08 smithi050 ceph-mon[29187]: pgmap v268: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 296 op/s 2024-04-23T18:27:08.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:08 smithi050 ceph-mon[29187]: daemon mds.h restarted 2024-04-23T18:27:08.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:08 smithi177 ceph-mon[32308]: pgmap v268: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 296 op/s 2024-04-23T18:27:08.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:08 smithi177 ceph-mon[32308]: daemon mds.h restarted 2024-04-23T18:27:08.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:08 smithi188 ceph-mon[32694]: pgmap v268: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 296 op/s 2024-04-23T18:27:08.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:08 smithi188 ceph-mon[32694]: daemon mds.h restarted 2024-04-23T18:27:09.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 584060 6.17 MB/sec execute 343 sec latency 41.698 ms 2024-04-23T18:27:09.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:09 smithi050 ceph-mon[29187]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T18:27:09.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:09 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6836/4033440314,v1:172.21.15.177:6837/4033440314] up:boot 2024-04-23T18:27:09.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:09 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:27:09.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:09 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:27:09.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:09 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:09.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:09 smithi177 ceph-mon[32308]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T18:27:09.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:09 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6836/4033440314,v1:172.21.15.177:6837/4033440314] up:boot 2024-04-23T18:27:09.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:09 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:27:09.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:09 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:27:09.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:09 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:09.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:09 smithi188 ceph-mon[32694]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T18:27:09.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:09 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6836/4033440314,v1:172.21.15.177:6837/4033440314] up:boot 2024-04-23T18:27:09.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:09 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:27:09.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:09 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:27:09.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:09 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:10.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 586213 6.19 MB/sec execute 344 sec latency 23.517 ms 2024-04-23T18:27:10.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:10 smithi050 ceph-mon[29187]: pgmap v270: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 371 op/s 2024-04-23T18:27:10.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:10 smithi177 ceph-mon[32308]: pgmap v270: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 371 op/s 2024-04-23T18:27:10.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:10 smithi188 ceph-mon[32694]: pgmap v270: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 371 op/s 2024-04-23T18:27:11.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 587567 6.19 MB/sec execute 345 sec latency 21.762 ms 2024-04-23T18:27:12.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 589846 6.21 MB/sec execute 346 sec latency 18.000 ms 2024-04-23T18:27:12.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:12 smithi188 ceph-mon[32694]: pgmap v271: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-04-23T18:27:12.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:12 smithi188 ceph-mon[32694]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T18:27:12.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:12 smithi050 ceph-mon[29187]: pgmap v271: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-04-23T18:27:12.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:12 smithi050 ceph-mon[29187]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T18:27:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:12 smithi177 ceph-mon[32308]: pgmap v271: 97 pgs: 97 active+clean; 205 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-04-23T18:27:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:12 smithi177 ceph-mon[32308]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T18:27:13.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 591126 6.21 MB/sec execute 347 sec latency 28.555 ms 2024-04-23T18:27:14.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 593382 6.23 MB/sec execute 348 sec latency 32.681 ms 2024-04-23T18:27:14.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:14 smithi188 ceph-mon[32694]: pgmap v273: 97 pgs: 97 active+clean; 196 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 377 op/s 2024-04-23T18:27:14.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:14 smithi050 ceph-mon[29187]: pgmap v273: 97 pgs: 97 active+clean; 196 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 377 op/s 2024-04-23T18:27:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:14 smithi177 ceph-mon[32308]: pgmap v273: 97 pgs: 97 active+clean; 196 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 377 op/s 2024-04-23T18:27:15.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 594612 6.23 MB/sec execute 349 sec latency 34.379 ms 2024-04-23T18:27:16.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 596772 6.25 MB/sec execute 350 sec latency 33.175 ms 2024-04-23T18:27:16.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:16 smithi188 ceph-mon[32694]: pgmap v274: 97 pgs: 97 active+clean; 196 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-04-23T18:27:16.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:16 smithi050 ceph-mon[29187]: pgmap v274: 97 pgs: 97 active+clean; 196 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-04-23T18:27:16.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:16 smithi177 ceph-mon[32308]: pgmap v274: 97 pgs: 97 active+clean; 196 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-04-23T18:27:17.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 598121 6.25 MB/sec execute 351 sec latency 21.957 ms 2024-04-23T18:27:18.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 600318 6.27 MB/sec execute 352 sec latency 16.380 ms 2024-04-23T18:27:18.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:18 smithi188 ceph-mon[32694]: pgmap v275: 97 pgs: 97 active+clean; 202 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 353 op/s 2024-04-23T18:27:18.338 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:18 smithi177 ceph-mon[32308]: pgmap v275: 97 pgs: 97 active+clean; 202 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 353 op/s 2024-04-23T18:27:18.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:18 smithi050 ceph-mon[29187]: pgmap v275: 97 pgs: 97 active+clean; 202 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 353 op/s 2024-04-23T18:27:19.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 601665 6.27 MB/sec execute 353 sec latency 29.678 ms 2024-04-23T18:27:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.878+0000 7f09219bb640 1 -- 172.21.15.50:0/4230811483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c106170 msgr2=0x7f091c10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.878+0000 7f09219bb640 1 --2- 172.21.15.50:0/4230811483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c106170 0x7f091c10dd40 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f0914009ca0 tx=0x7f091402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.878+0000 7f09219bb640 1 -- 172.21.15.50:0/4230811483 shutdown_connections 2024-04-23T18:27:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.878+0000 7f09219bb640 1 --2- 172.21.15.50:0/4230811483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c106170 0x7f091c10dd40 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.878+0000 7f09219bb640 1 --2- 172.21.15.50:0/4230811483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f091c101aa0 0x7f091c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.878+0000 7f09219bb640 1 --2- 172.21.15.50:0/4230811483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f091c1010f0 0x7f091c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.878+0000 7f09219bb640 1 -- 172.21.15.50:0/4230811483 >> 172.21.15.50:0/4230811483 conn(0x7f091c077fd0 msgr2=0x7f091c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.878+0000 7f09219bb640 1 -- 172.21.15.50:0/4230811483 shutdown_connections 2024-04-23T18:27:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.878+0000 7f09219bb640 1 -- 172.21.15.50:0/4230811483 wait complete. 2024-04-23T18:27:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f09219bb640 1 Processor -- start 2024-04-23T18:27:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f09219bb640 1 -- start start 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f09219bb640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f091c1010f0 0x7f091c197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f09219bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c106170 0x7f091c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f09219bb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f091c19bde0 0x7f091c19c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f09219bb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1110c0 con 0x7f091c106170 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f09219bb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f091c110f40 con 0x7f091c19bde0 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f09219bb640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f091c111240 con 0x7f091c1010f0 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f0913fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c106170 0x7f091c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f0913fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c106170 0x7f091c197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51348/0 (socket says 172.21.15.50:51348) 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f091affd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f091c1010f0 0x7f091c197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f0913fff640 1 -- 172.21.15.50:0/3584099087 learned_addr learned my addr 172.21.15.50:0/3584099087 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.879+0000 7f091b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f091c19bde0 0x7f091c19c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f091affd640 1 -- 172.21.15.50:0/3584099087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f091c19bde0 msgr2=0x7f091c19c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f091affd640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f091c19bde0 0x7f091c19c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f091affd640 1 -- 172.21.15.50:0/3584099087 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c106170 msgr2=0x7f091c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f091affd640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c106170 0x7f091c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f091affd640 1 -- 172.21.15.50:0/3584099087 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f091c19c8d0 con 0x7f091c1010f0 2024-04-23T18:27:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f091b7fe640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f091c19bde0 0x7f091c19c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f0913fff640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c106170 0x7f091c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f091affd640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f091c1010f0 0x7f091c197520 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f090c00e9f0 tx=0x7f090c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f09209b9640 1 -- 172.21.15.50:0/3584099087 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c00ccc0 con 0x7f091c1010f0 2024-04-23T18:27:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f09219bb640 1 -- 172.21.15.50:0/3584099087 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f091c19cbc0 con 0x7f091c1010f0 2024-04-23T18:27:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f09209b9640 1 -- 172.21.15.50:0/3584099087 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f090c0108e0 con 0x7f091c1010f0 2024-04-23T18:27:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f09219bb640 1 -- 172.21.15.50:0/3584099087 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f091c1a4400 con 0x7f091c1010f0 2024-04-23T18:27:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.880+0000 7f09209b9640 1 -- 172.21.15.50:0/3584099087 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c018820 con 0x7f091c1010f0 2024-04-23T18:27:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.882+0000 7f09247af740 1 Processor -- start 2024-04-23T18:27:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f09247af740 1 -- start start 2024-04-23T18:27:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f09247af740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1620 0x5588609d1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f09247af740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609d1f40 0x5588609dc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f09247af740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609dc800 0x5588609debf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f09247af740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558860936970 con 0x5588609dc800 2024-04-23T18:27:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f09247af740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5588608d34a0 con 0x5588609d1f40 2024-04-23T18:27:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f09247af740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558860932930 con 0x5588609d1620 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f0913fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609dc800 0x5588609debf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f091affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609d1f40 0x5588609dc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f091affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609d1f40 0x5588609dc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:36526/0 (socket says 172.21.15.50:36526) 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f0913fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609dc800 0x5588609debf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51364/0 (socket says 172.21.15.50:51364) 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f091affd640 1 -- 172.21.15.50:0/2823876536 learned_addr learned my addr 172.21.15.50:0/2823876536 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f091affd640 1 -- 172.21.15.50:0/2823876536 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1620 msgr2=0x5588609d1a00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f091affd640 1 --2- 172.21.15.50:0/2823876536 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1620 0x5588609d1a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f091affd640 1 -- 172.21.15.50:0/2823876536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609dc800 msgr2=0x5588609debf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f091affd640 1 --2- 172.21.15.50:0/2823876536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609dc800 0x5588609debf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f091affd640 1 -- 172.21.15.50:0/2823876536 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558860649210 con 0x5588609d1f40 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f0913fff640 1 --2- 172.21.15.50:0/2823876536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609dc800 0x5588609debf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f091affd640 1 --2- 172.21.15.50:0/2823876536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609d1f40 0x5588609dc2c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f090c00cea0 tx=0x7f090c010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a046c2e68a4c5ef server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:19.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.883+0000 7f0911ffb640 1 -- 172.21.15.50:0/2823876536 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c055030 con 0x5588609d1f40 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.884+0000 7f0911ffb640 1 -- 172.21.15.50:0/2823876536 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f090c0046c0 con 0x5588609d1f40 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.884+0000 7f0911ffb640 1 -- 172.21.15.50:0/2823876536 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c026060 con 0x5588609d1f40 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.884+0000 7f09247af740 1 -- 172.21.15.50:0/2823876536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609d1f40 msgr2=0x5588609dc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.884+0000 7f09247af740 1 --2- 172.21.15.50:0/2823876536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609d1f40 0x5588609dc2c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f090c00cea0 tx=0x7f090c010800 comp rx=0 tx=0).stop 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.884+0000 7f09247af740 1 -- 172.21.15.50:0/2823876536 shutdown_connections 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.884+0000 7f09247af740 1 --2- 172.21.15.50:0/2823876536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609dc800 0x5588609debf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.884+0000 7f09247af740 1 --2- 172.21.15.50:0/2823876536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609d1f40 0x5588609dc2c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.884+0000 7f09247af740 1 --2- 172.21.15.50:0/2823876536 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1620 0x5588609d1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.884+0000 7f09247af740 1 -- 172.21.15.50:0/2823876536 >> 172.21.15.50:0/2823876536 conn(0x5588609cdb30 msgr2=0x5588609cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.884+0000 7f09247af740 1 -- 172.21.15.50:0/2823876536 shutdown_connections 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.884+0000 7f09247af740 1 -- 172.21.15.50:0/2823876536 wait complete. 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.884+0000 7f09247af740 1 -- start start 2024-04-23T18:27:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f09247af740 1 Processor -- start 2024-04-23T18:27:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f09247af740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 0x5588609dc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f09247af740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1f40 0x5588609d94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f09247af740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609d9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f09247af740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5588609df430 con 0x5588609d1620 2024-04-23T18:27:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f09247af740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5588608d10b0 con 0x5588609dc800 2024-04-23T18:27:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f09247af740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5588609df5b0 con 0x5588609d1f40 2024-04-23T18:27:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f091b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 0x5588609dc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f091b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 0x5588609dc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51376/0 (socket says 172.21.15.50:51376) 2024-04-23T18:27:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f091b7fe640 1 -- 172.21.15.50:0/2721336643 learned_addr learned my addr 172.21.15.50:0/2721336643 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f09247af740 10 client.0 _ll_register_callbacks cb 0x5588608bed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f0913fff640 1 --2- 172.21.15.50:0/2721336643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609d9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f091b7fe640 1 -- 172.21.15.50:0/2721336643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1f40 msgr2=0x5588609d94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f091b7fe640 1 --2- 172.21.15.50:0/2721336643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1f40 0x5588609d94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f091b7fe640 1 -- 172.21.15.50:0/2721336643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 msgr2=0x5588609d9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f091b7fe640 1 --2- 172.21.15.50:0/2721336643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609d9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.885+0000 7f091b7fe640 1 -- 172.21.15.50:0/2721336643 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558860715620 con 0x5588609d1620 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f091b7fe640 1 --2- 172.21.15.50:0/2721336643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 0x5588609dc1c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0914009400 tx=0x7f0914031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512f258597d582f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f08faffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f08faffd640 1 -- 172.21.15.50:0/2721336643 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914003b70 con 0x5588609d1620 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f08faffd640 1 -- 172.21.15.50:0/2721336643 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f0914003d10 con 0x5588609d1620 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f09247af740 1 -- 172.21.15.50:0/2721336643 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f090c0046c0 con 0x5588609d1620 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f08faffd640 1 -- 172.21.15.50:0/2721336643 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f0914004040 con 0x5588609d1620 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f09247af740 10 client.16542 fetch_fsmap learned FSMap version 39 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f09247af740 1 -- 172.21.15.50:0/2721336643 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5588609eb3a0 con 0x5588609d1620 2024-04-23T18:27:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f08faffd640 1 -- 172.21.15.50:0/2721336643 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f090c0046c0 con 0x5588609d1620 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f08faffd640 1 -- 172.21.15.50:0/2721336643 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f0914044070 con 0x5588609d1620 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f09247af740 10 client.16542 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f09209b9640 1 -- 172.21.15.50:0/3584099087 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f090c058070 con 0x7f091c1010f0 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f09247af740 10 client.16542 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f09247af740 20 client.16542 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.886+0000 7f09247af740 1 --2- 172.21.15.50:0/2721336643 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5588609ec900 0x5588609eed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.887+0000 7f09247af740 4 client.16542 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.887+0000 7f09247af740 1 -- 172.21.15.50:0/2721336643 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558860944390 con 0x5588609ec900 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.887+0000 7f091affd640 1 --2- 172.21.15.50:0/2721336643 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5588609ec900 0x5588609eed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.887+0000 7f09209b9640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f09080a3cd0 0x7f09080a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.887+0000 7f08faffd640 10 client.16542 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.887+0000 7f091affd640 1 --2- 172.21.15.50:0/2721336643 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5588609ec900 0x5588609eed60 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d931425cfd5b229 server_cookie=751d7b01f07954f4 in_seq=0 out_seq=0 2024-04-23T18:27:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.887+0000 7f0913fff640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f09080a3cd0 0x7f09080a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.887+0000 7f09209b9640 1 -- 172.21.15.50:0/3584099087 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f090c0d73e0 con 0x7f091c1010f0 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f0913fff640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f09080a3cd0 0x7f09080a6190 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x5588609cef50 tx=0x7f0904014040 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f08faffd640 1 -- 172.21.15.50:0/2721336643 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558860944390 con 0x5588609ec900 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f08faffd640 10 client.16542 handle_command_reply: tid=0 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f09247af740 1 client.16542 shutdown 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f09247af740 1 -- 172.21.15.50:0/2721336643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 msgr2=0x5588609dc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f09247af740 1 --2- 172.21.15.50:0/2721336643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 0x5588609dc1c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0914009400 tx=0x7f0914031a30 comp rx=0 tx=0).stop 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f09247af740 1 -- 172.21.15.50:0/2721336643 shutdown_connections 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f09247af740 1 --2- 172.21.15.50:0/2721336643 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5588609ec900 0x5588609eed60 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f09247af740 1 --2- 172.21.15.50:0/2721336643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609d9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f09247af740 1 --2- 172.21.15.50:0/2721336643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1f40 0x5588609d94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f09247af740 1 --2- 172.21.15.50:0/2721336643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 0x5588609dc1c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f09247af740 1 -- 172.21.15.50:0/2721336643 >> 172.21.15.50:0/2721336643 conn(0x5588609cdb30 msgr2=0x5588609d3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f08faffd640 0 client.16542 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f09247af740 1 -- 172.21.15.50:0/2721336643 shutdown_connections 2024-04-23T18:27:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f09247af740 1 -- 172.21.15.50:0/2721336643 wait complete. 2024-04-23T18:27:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.888+0000 7f09247af740 20 client.16542 trim_cache size 0 max 16384 2024-04-23T18:27:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f09247af740 1 Processor -- start 2024-04-23T18:27:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f09247af740 1 -- start start 2024-04-23T18:27:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f09247af740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 0x5588609dbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f09247af740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1f40 0x5588609f4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f09247af740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f09247af740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1ae360 con 0x5588609d1620 2024-04-23T18:27:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f09247af740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1abbd0 con 0x5588609dc800 2024-04-23T18:27:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f09247af740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1a9a40 con 0x5588609d1f40 2024-04-23T18:27:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f091affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 0x5588609dbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f091affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 0x5588609dbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51382/0 (socket says 172.21.15.50:51382) 2024-04-23T18:27:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f091affd640 1 -- 172.21.15.50:0/3057918953 learned_addr learned my addr 172.21.15.50:0/3057918953 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f091b7fe640 1 --2- 172.21.15.50:0/3057918953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f0913fff640 1 --2- 172.21.15.50:0/3057918953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1f40 0x5588609f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f091affd640 1 -- 172.21.15.50:0/3057918953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1f40 msgr2=0x5588609f4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f091affd640 1 --2- 172.21.15.50:0/3057918953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1f40 0x5588609f4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f091affd640 1 -- 172.21.15.50:0/3057918953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 msgr2=0x5588609f4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f091affd640 1 --2- 172.21.15.50:0/3057918953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609f4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f091affd640 1 -- 172.21.15.50:0/3057918953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588609c92e0 con 0x5588609d1620 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.904+0000 7f091affd640 1 --2- 172.21.15.50:0/3057918953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 0x5588609dbcf0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f090c09f9d0 tx=0x7f090c0d8a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8cb4095ecfb8eb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.905+0000 7f09117fa640 1 -- 172.21.15.50:0/3057918953 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c026040 con 0x5588609d1620 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.905+0000 7f09117fa640 1 -- 172.21.15.50:0/3057918953 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f090c010a90 con 0x5588609d1620 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.905+0000 7f09117fa640 1 -- 172.21.15.50:0/3057918953 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c0a2ed0 con 0x5588609d1620 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.905+0000 7f09247af740 1 -- 172.21.15.50:0/3057918953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 msgr2=0x5588609dbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.905+0000 7f09247af740 1 --2- 172.21.15.50:0/3057918953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 0x5588609dbcf0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f090c09f9d0 tx=0x7f090c0d8a70 comp rx=0 tx=0).stop 2024-04-23T18:27:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.905+0000 7f09247af740 1 -- 172.21.15.50:0/3057918953 shutdown_connections 2024-04-23T18:27:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.905+0000 7f09247af740 1 --2- 172.21.15.50:0/3057918953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.905+0000 7f09247af740 1 --2- 172.21.15.50:0/3057918953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1f40 0x5588609f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.905+0000 7f09247af740 1 --2- 172.21.15.50:0/3057918953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1620 0x5588609dbcf0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.905+0000 7f09247af740 1 -- 172.21.15.50:0/3057918953 >> 172.21.15.50:0/3057918953 conn(0x5588609cdb30 msgr2=0x5588609d3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f09247af740 1 -- 172.21.15.50:0/3057918953 shutdown_connections 2024-04-23T18:27:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f09247af740 1 -- 172.21.15.50:0/3057918953 wait complete. 2024-04-23T18:27:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f09247af740 1 -- start start 2024-04-23T18:27:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f09247af740 1 Processor -- start 2024-04-23T18:27:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f09247af740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1620 0x5588609e4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f09247af740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1f40 0x5588609e4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f09247af740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609dfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f09247af740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1aa090 con 0x5588609d1f40 2024-04-23T18:27:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f09247af740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1ab500 con 0x5588609dc800 2024-04-23T18:27:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f09247af740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1aa760 con 0x5588609d1620 2024-04-23T18:27:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f09247af740 10 client.0 _ll_register_callbacks cb 0x7f09140318f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f0913fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1f40 0x5588609e4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f091affd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1620 0x5588609e4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f0913fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1f40 0x5588609e4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51394/0 (socket says 172.21.15.50:51394) 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f0913fff640 1 -- 172.21.15.50:0/2782810535 learned_addr learned my addr 172.21.15.50:0/2782810535 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.906+0000 7f091b7fe640 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609dfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f0913fff640 1 -- 172.21.15.50:0/2782810535 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1620 msgr2=0x5588609e4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f0913fff640 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1620 0x5588609e4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f0913fff640 1 -- 172.21.15.50:0/2782810535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 msgr2=0x5588609dfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f0913fff640 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609dfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f0913fff640 1 -- 172.21.15.50:0/2782810535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588609e0400 con 0x5588609d1f40 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f091affd640 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1620 0x5588609e4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f091b7fe640 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609dfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f0913fff640 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1f40 0x5588609e4c50 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f091c100990 tx=0x7f09040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d93e7815c38e1fc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f08faffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f08faffd640 1 -- 172.21.15.50:0/2782810535 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0904003ac0 con 0x5588609d1f40 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f08faffd640 1 -- 172.21.15.50:0/2782810535 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f0904003c60 con 0x5588609d1f40 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f09247af740 1 -- 172.21.15.50:0/2782810535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f090c010a90 con 0x5588609d1f40 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f08faffd640 1 -- 172.21.15.50:0/2782810535 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f0904037050 con 0x5588609d1f40 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f09247af740 10 client.16548 fetch_fsmap learned FSMap version 39 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f09247af740 1 -- 172.21.15.50:0/2782810535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x558860a52a70 con 0x5588609d1f40 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.907+0000 7f08faffd640 1 -- 172.21.15.50:0/2782810535 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f090c010a90 con 0x5588609d1f40 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.908+0000 7f08faffd640 1 -- 172.21.15.50:0/2782810535 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f090402a450 con 0x5588609d1f40 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.908+0000 7f09247af740 10 client.16548 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.908+0000 7f09247af740 10 client.16548 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.908+0000 7f09247af740 20 client.16548 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.908+0000 7f09247af740 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x558860a53270 0x558860a55690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.908+0000 7f09247af740 4 client.16548 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:27:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.908+0000 7f09247af740 1 -- 172.21.15.50:0/2782810535 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55886093e200 con 0x558860a53270 2024-04-23T18:27:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.908+0000 7f091affd640 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x558860a53270 0x558860a55690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.908+0000 7f091affd640 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x558860a53270 0x558860a55690 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=138232edc3b2d876 server_cookie=8d59f75be597f121 in_seq=0 out_seq=0 2024-04-23T18:27:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.908+0000 7f08faffd640 10 client.16548 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f08faffd640 1 -- 172.21.15.50:0/2782810535 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55886093e200 con 0x558860a53270 2024-04-23T18:27:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f08faffd640 10 client.16548 handle_command_reply: tid=0 2024-04-23T18:27:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f09247af740 1 client.16548 shutdown 2024-04-23T18:27:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f09247af740 1 -- 172.21.15.50:0/2782810535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1f40 msgr2=0x5588609e4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f09247af740 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1f40 0x5588609e4c50 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f091c100990 tx=0x7f09040046c0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f09247af740 1 -- 172.21.15.50:0/2782810535 shutdown_connections 2024-04-23T18:27:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f09247af740 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x558860a53270 0x558860a55690 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f09247af740 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588609dc800 0x5588609dfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f09247af740 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5588609d1f40 0x5588609e4c50 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f09247af740 1 --2- 172.21.15.50:0/2782810535 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5588609d1620 0x5588609e4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f09247af740 1 -- 172.21.15.50:0/2782810535 >> 172.21.15.50:0/2782810535 conn(0x5588609cdb30 msgr2=0x5588609cc420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f08faffd640 0 client.16548 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f09247af740 1 -- 172.21.15.50:0/2782810535 shutdown_connections 2024-04-23T18:27:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.957+0000 7f09247af740 1 -- 172.21.15.50:0/2782810535 wait complete. 2024-04-23T18:27:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.958+0000 7f09247af740 20 client.16548 trim_cache size 0 max 16384 2024-04-23T18:27:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:27:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:27:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:27:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:27:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.958+0000 7f09219bb640 1 -- 172.21.15.50:0/3584099087 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f09080a3cd0 msgr2=0x7f09080a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.958+0000 7f09219bb640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f09080a3cd0 0x7f09080a6190 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x5588609cef50 tx=0x7f0904014040 comp rx=0 tx=0).stop 2024-04-23T18:27:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.958+0000 7f09219bb640 1 -- 172.21.15.50:0/3584099087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f091c1010f0 msgr2=0x7f091c197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.958+0000 7f09219bb640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f091c1010f0 0x7f091c197520 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f090c00e9f0 tx=0x7f090c00eeb0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.958+0000 7f09219bb640 1 -- 172.21.15.50:0/3584099087 shutdown_connections 2024-04-23T18:27:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.958+0000 7f09219bb640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f09080a3cd0 0x7f09080a6190 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.958+0000 7f09219bb640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f091c19bde0 0x7f091c19c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.958+0000 7f09219bb640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c106170 0x7f091c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.958+0000 7f09219bb640 1 --2- 172.21.15.50:0/3584099087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f091c1010f0 0x7f091c197520 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.958+0000 7f09219bb640 1 -- 172.21.15.50:0/3584099087 >> 172.21.15.50:0/3584099087 conn(0x7f091c077fd0 msgr2=0x7f091c0fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.958+0000 7f09219bb640 1 -- 172.21.15.50:0/3584099087 shutdown_connections 2024-04-23T18:27:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:19.959+0000 7f09219bb640 1 -- 172.21.15.50:0/3584099087 wait complete. 2024-04-23T18:27:20.220 INFO:tasks.workunit.client.0.smithi050.stdout: 1 603258 6.27 MB/sec execute 354 sec latency 59.066 ms 2024-04-23T18:27:20.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:20 smithi188 ceph-mon[32694]: pgmap v276: 97 pgs: 97 active+clean; 202 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 300 op/s 2024-04-23T18:27:20.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:20 smithi050 ceph-mon[29187]: pgmap v276: 97 pgs: 97 active+clean; 202 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 300 op/s 2024-04-23T18:27:20.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:20 smithi177 ceph-mon[32308]: pgmap v276: 97 pgs: 97 active+clean; 202 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 300 op/s 2024-04-23T18:27:21.221 INFO:tasks.workunit.client.0.smithi050.stdout: 1 604872 6.28 MB/sec execute 355 sec latency 34.387 ms 2024-04-23T18:27:21.360 INFO:tasks.quiescer.fs.[cephfs]:Total duration (44.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:27:21.361 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.05 2024-04-23T18:27:21.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:27:21.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.477+0000 7f605aefa640 1 -- 172.21.15.50:0/2701176002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6054102bf0 msgr2=0x7f6054106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.477+0000 7f605aefa640 1 --2- 172.21.15.50:0/2701176002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6054102bf0 0x7f6054106ba0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6040009900 tx=0x7f604002a410 comp rx=0 tx=0).stop 2024-04-23T18:27:21.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.477+0000 7f605aefa640 1 -- 172.21.15.50:0/2701176002 shutdown_connections 2024-04-23T18:27:21.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.477+0000 7f605aefa640 1 --2- 172.21.15.50:0/2701176002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6054107280 0x7f605410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.477+0000 7f605aefa640 1 --2- 172.21.15.50:0/2701176002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6054102bf0 0x7f6054106ba0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.477+0000 7f605aefa640 1 --2- 172.21.15.50:0/2701176002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6054102240 0x7f6054102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.477+0000 7f605aefa640 1 -- 172.21.15.50:0/2701176002 >> 172.21.15.50:0/2701176002 conn(0x7f60540790e0 msgr2=0x7f60540794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:21.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.477+0000 7f605aefa640 1 -- 172.21.15.50:0/2701176002 shutdown_connections 2024-04-23T18:27:21.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.477+0000 7f605aefa640 1 -- 172.21.15.50:0/2701176002 wait complete. 2024-04-23T18:27:21.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f605aefa640 1 Processor -- start 2024-04-23T18:27:21.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f605aefa640 1 -- start start 2024-04-23T18:27:21.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f605aefa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6054102240 0x7f6054198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f605aefa640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6054107280 0x7f6054198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f605aefa640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f605419d010 0x7f605419d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f605aefa640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6054112210 con 0x7f6054107280 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f605aefa640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6054112090 con 0x7f6054102240 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f605aefa640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6054112390 con 0x7f605419d010 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f6053fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6054107280 0x7f6054198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f6058c96640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6054102240 0x7f6054198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f6053fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6054107280 0x7f6054198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51426/0 (socket says 172.21.15.50:51426) 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f6053fff640 1 -- 172.21.15.50:0/4262847763 learned_addr learned my addr 172.21.15.50:0/4262847763 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.478+0000 7f6059497640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f605419d010 0x7f605419d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.479+0000 7f6053fff640 1 -- 172.21.15.50:0/4262847763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f605419d010 msgr2=0x7f605419d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.479+0000 7f6053fff640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f605419d010 0x7f605419d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.479+0000 7f6053fff640 1 -- 172.21.15.50:0/4262847763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6054102240 msgr2=0x7f6054198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.479+0000 7f6053fff640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6054102240 0x7f6054198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.479+0000 7f6053fff640 1 -- 172.21.15.50:0/4262847763 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605419db00 con 0x7f6054107280 2024-04-23T18:27:21.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.479+0000 7f6058c96640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6054102240 0x7f6054198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:21.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.479+0000 7f6053fff640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6054107280 0x7f6054198c90 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6040005ab0 tx=0x7f6040003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:21.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.479+0000 7f6051ffb640 1 -- 172.21.15.50:0/4262847763 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604003f070 con 0x7f6054107280 2024-04-23T18:27:21.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.479+0000 7f6051ffb640 1 -- 172.21.15.50:0/4262847763 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6040031b50 con 0x7f6054107280 2024-04-23T18:27:21.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.479+0000 7f605aefa640 1 -- 172.21.15.50:0/4262847763 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605419dd90 con 0x7f6054107280 2024-04-23T18:27:21.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.479+0000 7f6051ffb640 1 -- 172.21.15.50:0/4262847763 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040035030 con 0x7f6054107280 2024-04-23T18:27:21.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.479+0000 7f605aefa640 1 -- 172.21.15.50:0/4262847763 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60541a55b0 con 0x7f6054107280 2024-04-23T18:27:21.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.480+0000 7f605aefa640 1 -- 172.21.15.50:0/4262847763 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60540691c0 con 0x7f6054107280 2024-04-23T18:27:21.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.482+0000 7f6051ffb640 1 -- 172.21.15.50:0/4262847763 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6040042070 con 0x7f6054107280 2024-04-23T18:27:21.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.482+0000 7f6051ffb640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f60440a3da0 0x7f60440a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.482+0000 7f6058c96640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f60440a3da0 0x7f60440a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.482+0000 7f6051ffb640 1 -- 172.21.15.50:0/4262847763 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f60400f0c90 con 0x7f6054107280 2024-04-23T18:27:21.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.483+0000 7f6058c96640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f60440a3da0 0x7f60440a6260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6048009fc0 tx=0x7f6048005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:21.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.487+0000 7f6051ffb640 1 -- 172.21.15.50:0/4262847763 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f60400bccd0 con 0x7f6054107280 2024-04-23T18:27:21.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.739+0000 7f605aefa640 1 -- 172.21.15.50:0/4262847763 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6054102bf0 con 0x7f6054107280 2024-04-23T18:27:21.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.739+0000 7f6051ffb640 1 -- 172.21.15.50:0/4262847763 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11878 (secure 0 0 0) 0x7f60400c0fd0 con 0x7f6054107280 2024-04-23T18:27:21.733 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:27:21.733 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":39,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:27:08.151749+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/4033440314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4033440314},{"type":"v1","addr":"172.21.15.177:6837","nonce":4033440314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:27:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 39 2024-04-23T18:27:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.741+0000 7f605aefa640 1 -- 172.21.15.50:0/4262847763 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f60440a3da0 msgr2=0x7f60440a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.741+0000 7f605aefa640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f60440a3da0 0x7f60440a6260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6048009fc0 tx=0x7f6048005f90 comp rx=0 tx=0).stop 2024-04-23T18:27:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.741+0000 7f605aefa640 1 -- 172.21.15.50:0/4262847763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6054107280 msgr2=0x7f6054198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.741+0000 7f605aefa640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6054107280 0x7f6054198c90 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6040005ab0 tx=0x7f6040003b90 comp rx=0 tx=0).stop 2024-04-23T18:27:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.742+0000 7f605aefa640 1 -- 172.21.15.50:0/4262847763 shutdown_connections 2024-04-23T18:27:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.742+0000 7f605aefa640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f60440a3da0 0x7f60440a6260 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.742+0000 7f605aefa640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f605419d010 0x7f605419d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.742+0000 7f605aefa640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6054107280 0x7f6054198c90 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.742+0000 7f605aefa640 1 --2- 172.21.15.50:0/4262847763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6054102240 0x7f6054198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.742+0000 7f605aefa640 1 -- 172.21.15.50:0/4262847763 >> 172.21.15.50:0/4262847763 conn(0x7f60540790e0 msgr2=0x7f6054100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.742+0000 7f605aefa640 1 -- 172.21.15.50:0/4262847763 shutdown_connections 2024-04-23T18:27:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.742+0000 7f605aefa640 1 -- 172.21.15.50:0/4262847763 wait complete. 2024-04-23T18:27:21.746 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 82.05 --await-for 42.0' 2024-04-23T18:27:21.746 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 82.05 --await-for 42.0 2024-04-23T18:27:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.866+0000 7f0b8700b640 1 -- 172.21.15.50:0/4166274378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b80101090 msgr2=0x7f0b80101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.866+0000 7f0b8700b640 1 --2- 172.21.15.50:0/4166274378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b80101090 0x7f0b80101470 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0b74009900 tx=0x7f0b7402a410 comp rx=0 tx=0).stop 2024-04-23T18:27:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 -- 172.21.15.50:0/4166274378 shutdown_connections 2024-04-23T18:27:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 --2- 172.21.15.50:0/4166274378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b80106160 0x7f0b8010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 --2- 172.21.15.50:0/4166274378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b80101a40 0x7f0b80105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 --2- 172.21.15.50:0/4166274378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b80101090 0x7f0b80101470 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 -- 172.21.15.50:0/4166274378 >> 172.21.15.50:0/4166274378 conn(0x7f0b800771d0 msgr2=0x7f0b800775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 -- 172.21.15.50:0/4166274378 shutdown_connections 2024-04-23T18:27:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 -- 172.21.15.50:0/4166274378 wait complete. 2024-04-23T18:27:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 Processor -- start 2024-04-23T18:27:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 -- start start 2024-04-23T18:27:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b80101090 0x7f0b8019e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b80106160 0x7f0b8019e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b80198620 0x7f0b80198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0b80111140 con 0x7f0b80198620 2024-04-23T18:27:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0b80110fc0 con 0x7f0b80106160 2024-04-23T18:27:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.867+0000 7f0b8700b640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0b801112c0 con 0x7f0b80101090 2024-04-23T18:27:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b855a8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b80198620 0x7f0b80198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b7ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b80106160 0x7f0b8019e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b84da7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b80101090 0x7f0b8019e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b84da7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b80101090 0x7f0b8019e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:55880/0 (socket says 172.21.15.50:55880) 2024-04-23T18:27:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b7ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b80106160 0x7f0b8019e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:39730/0 (socket says 172.21.15.50:39730) 2024-04-23T18:27:21.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b855a8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b80198620 0x7f0b80198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45380/0 (socket says 172.21.15.50:45380) 2024-04-23T18:27:21.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b84da7640 1 -- 172.21.15.50:0/3691911730 learned_addr learned my addr 172.21.15.50:0/3691911730 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:21.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b855a8640 1 -- 172.21.15.50:0/3691911730 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b80101090 msgr2=0x7f0b8019e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b855a8640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b80101090 0x7f0b8019e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b855a8640 1 -- 172.21.15.50:0/3691911730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b80106160 msgr2=0x7f0b8019e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b855a8640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b80106160 0x7f0b8019e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b855a8640 1 -- 172.21.15.50:0/3691911730 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b80199140 con 0x7f0b80198620 2024-04-23T18:27:21.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b84da7640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b80101090 0x7f0b8019e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:21.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b7ffff640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b80106160 0x7f0b8019e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:21.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b855a8640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b80198620 0x7f0b80198ad0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0b7800be20 tx=0x7f0b7800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:21.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b7dffb640 1 -- 172.21.15.50:0/3691911730 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b78019070 con 0x7f0b80198620 2024-04-23T18:27:21.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b7dffb640 1 -- 172.21.15.50:0/3691911730 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b780148f0 con 0x7f0b80198620 2024-04-23T18:27:21.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b8700b640 1 -- 172.21.15.50:0/3691911730 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b80199430 con 0x7f0b80198620 2024-04-23T18:27:21.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.868+0000 7f0b7dffb640 1 -- 172.21.15.50:0/3691911730 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b78012840 con 0x7f0b80198620 2024-04-23T18:27:21.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.869+0000 7f0b8700b640 1 -- 172.21.15.50:0/3691911730 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b801a5630 con 0x7f0b80198620 2024-04-23T18:27:21.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b7dffb640 1 -- 172.21.15.50:0/3691911730 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0b7801c070 con 0x7f0b80198620 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b89df5740 1 Processor -- start 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b89df5740 1 -- start start 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b7dffb640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0b700a3d20 0x7f0b700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b89df5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de1f7f80 0x5560de1f8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b89df5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de206f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b89df5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de2074a0 0x5560de209890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b89df5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5560de15d370 con 0x5560de1f88a0 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b89df5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5560de0fa2e0 con 0x5560de2074a0 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b89df5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5560de159330 con 0x5560de1f7f80 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b7ffff640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0b700a3d20 0x7f0b700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b7dffb640 1 -- 172.21.15.50:0/3691911730 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f0b780cfef0 con 0x7f0b80198620 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b7ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de206f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b7ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de206f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45382/0 (socket says 172.21.15.50:45382) 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b7ffff640 1 -- 172.21.15.50:0/3010117486 learned_addr learned my addr 172.21.15.50:0/3010117486 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.871+0000 7f0b855a8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de2074a0 0x5560de209890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.872+0000 7f0b84da7640 1 --2- 172.21.15.50:0/3010117486 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de1f7f80 0x5560de1f8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.872+0000 7f0b7ffff640 1 -- 172.21.15.50:0/3010117486 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de1f7f80 msgr2=0x5560de1f8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.872+0000 7f0b7ffff640 1 --2- 172.21.15.50:0/3010117486 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de1f7f80 0x5560de1f8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.872+0000 7f0b7ffff640 1 -- 172.21.15.50:0/3010117486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de2074a0 msgr2=0x5560de209890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.872+0000 7f0b7ffff640 1 --2- 172.21.15.50:0/3010117486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de2074a0 0x5560de209890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.872+0000 7f0b7ffff640 1 -- 172.21.15.50:0/3010117486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560ddf3bd80 con 0x5560de1f88a0 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.872+0000 7f0b855a8640 1 --2- 172.21.15.50:0/3010117486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de2074a0 0x5560de209890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.872+0000 7f0b7ffff640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0b700a3d20 0x7f0b700a61e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c011d10 tx=0x7f0b6c009f50 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.872+0000 7f0b7ffff640 1 --2- 172.21.15.50:0/3010117486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de206f60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c005b90 tx=0x7f0b6c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5467c79f81636fe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.872+0000 7f0b6affd640 1 -- 172.21.15.50:0/3010117486 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c024070 con 0x5560de1f88a0 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.872+0000 7f0b6affd640 1 -- 172.21.15.50:0/3010117486 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b6c011d40 con 0x5560de1f88a0 2024-04-23T18:27:21.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.872+0000 7f0b6affd640 1 -- 172.21.15.50:0/3010117486 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c01fce0 con 0x5560de1f88a0 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.873+0000 7f0b89df5740 1 -- 172.21.15.50:0/3010117486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 msgr2=0x5560de206f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.873+0000 7f0b89df5740 1 --2- 172.21.15.50:0/3010117486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de206f60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c005b90 tx=0x7f0b6c01d000 comp rx=0 tx=0).stop 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.873+0000 7f0b89df5740 1 -- 172.21.15.50:0/3010117486 shutdown_connections 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.873+0000 7f0b89df5740 1 --2- 172.21.15.50:0/3010117486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de2074a0 0x5560de209890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.873+0000 7f0b89df5740 1 --2- 172.21.15.50:0/3010117486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de206f60 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.873+0000 7f0b89df5740 1 --2- 172.21.15.50:0/3010117486 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de1f7f80 0x5560de1f8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.873+0000 7f0b89df5740 1 -- 172.21.15.50:0/3010117486 >> 172.21.15.50:0/3010117486 conn(0x5560de1f43c0 msgr2=0x5560de1f47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.873+0000 7f0b89df5740 1 -- 172.21.15.50:0/3010117486 shutdown_connections 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.873+0000 7f0b89df5740 1 -- 172.21.15.50:0/3010117486 wait complete. 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.873+0000 7f0b89df5740 1 -- start start 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.873+0000 7f0b89df5740 1 Processor -- start 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b89df5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de1f7f80 0x5560de204410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b89df5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de204950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b89df5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de2074a0 0x5560de204e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b89df5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5560de20a280 con 0x5560de1f88a0 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b89df5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5560de0f7fb0 con 0x5560de1f7f80 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b89df5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5560de20a400 con 0x5560de2074a0 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b89df5740 10 client.0 _ll_register_callbacks cb 0x5560de0f79a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b84da7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de1f7f80 0x5560de204410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b7ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de204950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b855a8640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de2074a0 0x5560de204e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b7ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de204950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45396/0 (socket says 172.21.15.50:45396) 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b84da7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de1f7f80 0x5560de204410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:39750/0 (socket says 172.21.15.50:39750) 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b7ffff640 1 -- 172.21.15.50:0/899661771 learned_addr learned my addr 172.21.15.50:0/899661771 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b7ffff640 1 -- 172.21.15.50:0/899661771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de2074a0 msgr2=0x5560de204e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b7ffff640 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de2074a0 0x5560de204e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b7ffff640 1 -- 172.21.15.50:0/899661771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de1f7f80 msgr2=0x5560de204410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b7ffff640 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de1f7f80 0x5560de204410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b7ffff640 1 -- 172.21.15.50:0/899661771 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560ddf9c510 con 0x5560de1f88a0 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b84da7640 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de1f7f80 0x5560de204410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.874+0000 7f0b855a8640 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de2074a0 0x5560de204e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b7ffff640 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de204950 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c020f20 tx=0x7f0b6c01f8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c7645ba800e04c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b5ffff640 1 -- 172.21.15.50:0/899661771 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c024070 con 0x5560de1f88a0 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b5ffff640 1 -- 172.21.15.50:0/899661771 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f0b6c01fa00 con 0x5560de1f88a0 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b89df5740 1 -- 172.21.15.50:0/899661771 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b6c011d40 con 0x5560de1f88a0 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b5ffff640 1 -- 172.21.15.50:0/899661771 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f0b6c01fc00 con 0x5560de1f88a0 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b89df5740 10 client.16566 fetch_fsmap learned FSMap version 39 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b89df5740 1 -- 172.21.15.50:0/899661771 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5560dde2f670 con 0x5560de1f88a0 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b5ffff640 1 -- 172.21.15.50:0/899661771 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b6c011d40 con 0x5560de1f88a0 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b5ffff640 1 -- 172.21.15.50:0/899661771 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f0b6c003790 con 0x5560de1f88a0 2024-04-23T18:27:21.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b89df5740 10 client.16566 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b89df5740 10 client.16566 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.875+0000 7f0b89df5740 20 client.16566 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.876+0000 7f0b89df5740 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5560de217720 0x5560de219b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.876+0000 7f0b89df5740 4 client.16566 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.876+0000 7f0b89df5740 1 -- 172.21.15.50:0/899661771 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560de16ad90 con 0x5560de217720 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.876+0000 7f0b84da7640 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5560de217720 0x5560de219b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.876+0000 7f0b84da7640 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5560de217720 0x5560de219b10 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2827e13d03c9ff1f server_cookie=1389a8fb9743e0e4 in_seq=0 out_seq=0 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.876+0000 7f0b5ffff640 10 client.16566 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b5ffff640 1 -- 172.21.15.50:0/899661771 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5560de16ad90 con 0x5560de217720 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b5ffff640 10 client.16566 handle_command_reply: tid=0 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b89df5740 1 client.16566 shutdown 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b89df5740 1 -- 172.21.15.50:0/899661771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 msgr2=0x5560de204950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b89df5740 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de204950 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c020f20 tx=0x7f0b6c01f8e0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b89df5740 1 -- 172.21.15.50:0/899661771 shutdown_connections 2024-04-23T18:27:21.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b89df5740 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5560de217720 0x5560de219b10 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b89df5740 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de2074a0 0x5560de204e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b89df5740 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de204950 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b89df5740 1 --2- 172.21.15.50:0/899661771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de1f7f80 0x5560de204410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b5ffff640 0 client.16566 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:21.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b89df5740 1 -- 172.21.15.50:0/899661771 >> 172.21.15.50:0/899661771 conn(0x5560de1f43c0 msgr2=0x5560de1fa170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:21.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b89df5740 1 -- 172.21.15.50:0/899661771 shutdown_connections 2024-04-23T18:27:21.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b89df5740 1 -- 172.21.15.50:0/899661771 wait complete. 2024-04-23T18:27:21.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.877+0000 7f0b89df5740 20 client.16566 trim_cache size 0 max 16384 2024-04-23T18:27:21.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.895+0000 7f0b89df5740 1 Processor -- start 2024-04-23T18:27:21.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.895+0000 7f0b89df5740 1 -- start start 2024-04-23T18:27:21.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.895+0000 7f0b89df5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de1f7f80 0x5560de2062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.895+0000 7f0b89df5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de213180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.895+0000 7f0b89df5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de2074a0 0x5560de202ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.895+0000 7f0b89df5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0b801ae3a0 con 0x5560de1f88a0 2024-04-23T18:27:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.895+0000 7f0b89df5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0b801abc10 con 0x5560de1f7f80 2024-04-23T18:27:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.895+0000 7f0b89df5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0b801a9a80 con 0x5560de2074a0 2024-04-23T18:27:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b7ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de213180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b7ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de213180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45412/0 (socket says 172.21.15.50:45412) 2024-04-23T18:27:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b7ffff640 1 -- 172.21.15.50:0/3789309395 learned_addr learned my addr 172.21.15.50:0/3789309395 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b84da7640 1 --2- 172.21.15.50:0/3789309395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de1f7f80 0x5560de2062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b855a8640 1 --2- 172.21.15.50:0/3789309395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de2074a0 0x5560de202ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b7ffff640 1 -- 172.21.15.50:0/3789309395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de2074a0 msgr2=0x5560de202ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b7ffff640 1 --2- 172.21.15.50:0/3789309395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de2074a0 0x5560de202ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b7ffff640 1 -- 172.21.15.50:0/3789309395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de1f7f80 msgr2=0x5560de2062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b7ffff640 1 --2- 172.21.15.50:0/3789309395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de1f7f80 0x5560de2062b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b7ffff640 1 -- 172.21.15.50:0/3789309395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560de1f86d0 con 0x5560de1f88a0 2024-04-23T18:27:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b7ffff640 1 --2- 172.21.15.50:0/3789309395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de213180 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c023040 tx=0x7f0b6c009cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ffc7bced28eaea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b6a7fc640 1 -- 172.21.15.50:0/3789309395 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c024040 con 0x5560de1f88a0 2024-04-23T18:27:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b6a7fc640 1 -- 172.21.15.50:0/3789309395 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b6c00fc20 con 0x5560de1f88a0 2024-04-23T18:27:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.896+0000 7f0b6a7fc640 1 -- 172.21.15.50:0/3789309395 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c011750 con 0x5560de1f88a0 2024-04-23T18:27:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.897+0000 7f0b89df5740 1 -- 172.21.15.50:0/3789309395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 msgr2=0x5560de213180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.897+0000 7f0b89df5740 1 --2- 172.21.15.50:0/3789309395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de213180 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c023040 tx=0x7f0b6c009cf0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.897+0000 7f0b89df5740 1 -- 172.21.15.50:0/3789309395 shutdown_connections 2024-04-23T18:27:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.897+0000 7f0b89df5740 1 --2- 172.21.15.50:0/3789309395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de2074a0 0x5560de202ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.897+0000 7f0b89df5740 1 --2- 172.21.15.50:0/3789309395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de213180 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.897+0000 7f0b89df5740 1 --2- 172.21.15.50:0/3789309395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de1f7f80 0x5560de2062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.897+0000 7f0b89df5740 1 -- 172.21.15.50:0/3789309395 >> 172.21.15.50:0/3789309395 conn(0x5560de1f43c0 msgr2=0x5560de1fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.897+0000 7f0b89df5740 1 -- 172.21.15.50:0/3789309395 shutdown_connections 2024-04-23T18:27:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.897+0000 7f0b89df5740 1 -- 172.21.15.50:0/3789309395 wait complete. 2024-04-23T18:27:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b89df5740 1 -- start start 2024-04-23T18:27:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b89df5740 1 Processor -- start 2024-04-23T18:27:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b89df5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de1f7f80 0x5560de20f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b89df5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de200ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b89df5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de2074a0 0x5560de201220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b89df5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0b801aa0d0 con 0x5560de1f88a0 2024-04-23T18:27:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b89df5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0b801ab540 con 0x5560de2074a0 2024-04-23T18:27:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b89df5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0b801aa7a0 con 0x5560de1f7f80 2024-04-23T18:27:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b89df5740 10 client.0 _ll_register_callbacks cb 0x7f0b6c0116a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b7ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de200ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b84da7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de1f7f80 0x5560de20f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b84da7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de1f7f80 0x5560de20f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:55914/0 (socket says 172.21.15.50:55914) 2024-04-23T18:27:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b7ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de200ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45426/0 (socket says 172.21.15.50:45426) 2024-04-23T18:27:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b84da7640 1 -- 172.21.15.50:0/4172688223 learned_addr learned my addr 172.21.15.50:0/4172688223 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.898+0000 7f0b855a8640 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de2074a0 0x5560de201220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b855a8640 1 -- 172.21.15.50:0/4172688223 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de1f7f80 msgr2=0x5560de20f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b855a8640 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de1f7f80 0x5560de20f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b855a8640 1 -- 172.21.15.50:0/4172688223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 msgr2=0x5560de200ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b855a8640 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de200ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b855a8640 1 -- 172.21.15.50:0/4172688223 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560de2019d0 con 0x5560de2074a0 2024-04-23T18:27:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b84da7640 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de1f7f80 0x5560de20f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b855a8640 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de2074a0 0x5560de201220 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b780cd740 tx=0x7f0b78004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3146260323e54a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:27:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b5ffff640 1 -- 172.21.15.50:0/4172688223 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b7804a640 con 0x5560de2074a0 2024-04-23T18:27:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b89df5740 1 -- 172.21.15.50:0/4172688223 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b6c00fc20 con 0x5560de2074a0 2024-04-23T18:27:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b5ffff640 1 -- 172.21.15.50:0/4172688223 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f0b7804a0b0 con 0x5560de2074a0 2024-04-23T18:27:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b5ffff640 1 -- 172.21.15.50:0/4172688223 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f0b7804a7e0 con 0x5560de2074a0 2024-04-23T18:27:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b89df5740 10 client.26144 fetch_fsmap learned FSMap version 39 2024-04-23T18:27:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.899+0000 7f0b89df5740 1 -- 172.21.15.50:0/4172688223 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5560de27dba0 con 0x5560de2074a0 2024-04-23T18:27:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.900+0000 7f0b5ffff640 1 -- 172.21.15.50:0/4172688223 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b6c00fc20 con 0x5560de2074a0 2024-04-23T18:27:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.900+0000 7f0b5ffff640 1 -- 172.21.15.50:0/4172688223 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f0b78096bb0 con 0x5560de2074a0 2024-04-23T18:27:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.900+0000 7f0b89df5740 10 client.26144 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:27:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.900+0000 7f0b89df5740 10 client.26144 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:27:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.900+0000 7f0b89df5740 20 client.26144 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.900+0000 7f0b89df5740 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5560de27e2e0 0x5560de280700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.900+0000 7f0b89df5740 4 client.26144 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.05, "await_for": 42.0}] 2024-04-23T18:27:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.900+0000 7f0b89df5740 1 -- 172.21.15.50:0/4172688223 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.05, "await_for": 42.0}) -- 0x5560de164c00 con 0x5560de27e2e0 2024-04-23T18:27:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.900+0000 7f0b84da7640 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5560de27e2e0 0x5560de280700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.900+0000 7f0b84da7640 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5560de27e2e0 0x5560de280700 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1eb6b12a5dad2a9 server_cookie=a820d58566c3c2bd in_seq=0 out_seq=0 2024-04-23T18:27:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.900+0000 7f0b5ffff640 10 client.26144 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:21.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b5ffff640 1 -- 172.21.15.50:0/4172688223 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5560de164c00 con 0x5560de27e2e0 2024-04-23T18:27:21.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b5ffff640 10 client.26144 handle_command_reply: tid=0 2024-04-23T18:27:21.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b89df5740 1 client.26144 shutdown 2024-04-23T18:27:21.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b89df5740 1 -- 172.21.15.50:0/4172688223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de2074a0 msgr2=0x5560de201220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b89df5740 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de2074a0 0x5560de201220 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b780cd740 tx=0x7f0b78004b50 comp rx=0 tx=0).stop 2024-04-23T18:27:21.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b89df5740 1 -- 172.21.15.50:0/4172688223 shutdown_connections 2024-04-23T18:27:21.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b89df5740 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5560de27e2e0 0x5560de280700 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b89df5740 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560de2074a0 0x5560de201220 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b89df5740 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5560de1f88a0 0x5560de200ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b89df5740 1 --2- 172.21.15.50:0/4172688223 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5560de1f7f80 0x5560de20f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b89df5740 1 -- 172.21.15.50:0/4172688223 >> 172.21.15.50:0/4172688223 conn(0x5560de1f43c0 msgr2=0x5560de212170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:21.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b5ffff640 0 client.26144 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:21.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.960+0000 7f0b89df5740 1 -- 172.21.15.50:0/4172688223 shutdown_connections 2024-04-23T18:27:21.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b89df5740 1 -- 172.21.15.50:0/4172688223 wait complete. 2024-04-23T18:27:21.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b89df5740 20 client.26144 trim_cache size 0 max 16384 2024-04-23T18:27:21.955 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 39, 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 26, 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "c31bc802": { 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 26, 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 82.0, 2024-04-23T18:27:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:27:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:27:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:27:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:27:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:27:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:27:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:21.957 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:27:21.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b8700b640 1 -- 172.21.15.50:0/3691911730 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0b700a3d20 msgr2=0x7f0b700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b8700b640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0b700a3d20 0x7f0b700a61e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c011d10 tx=0x7f0b6c009f50 comp rx=0 tx=0).stop 2024-04-23T18:27:21.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b8700b640 1 -- 172.21.15.50:0/3691911730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b80198620 msgr2=0x7f0b80198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:21.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b8700b640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b80198620 0x7f0b80198ad0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0b7800be20 tx=0x7f0b7800bf10 comp rx=0 tx=0).stop 2024-04-23T18:27:21.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b8700b640 1 -- 172.21.15.50:0/3691911730 shutdown_connections 2024-04-23T18:27:21.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b8700b640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f0b700a3d20 0x7f0b700a61e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b8700b640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b80198620 0x7f0b80198ad0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b8700b640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b80106160 0x7f0b8019e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b8700b640 1 --2- 172.21.15.50:0/3691911730 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b80101090 0x7f0b8019e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:21.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b8700b640 1 -- 172.21.15.50:0/3691911730 >> 172.21.15.50:0/3691911730 conn(0x7f0b800771d0 msgr2=0x7f0b8006aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:21.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b8700b640 1 -- 172.21.15.50:0/3691911730 shutdown_connections 2024-04-23T18:27:21.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:21.961+0000 7f0b8700b640 1 -- 172.21.15.50:0/3691911730 wait complete. 2024-04-23T18:27:21.965 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c31bc802', quiesce took 0.6 seconds. Will release after: 22.05 2024-04-23T18:27:22.221 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.28 MB/sec execute 356 sec latency 277.820 ms 2024-04-23T18:27:22.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:22 smithi188 ceph-mon[32694]: pgmap v277: 97 pgs: 97 active+clean; 202 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 300 op/s 2024-04-23T18:27:22.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:22 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4262847763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:22.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:22 smithi050 ceph-mon[29187]: pgmap v277: 97 pgs: 97 active+clean; 202 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 300 op/s 2024-04-23T18:27:22.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:22 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4262847763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:22.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:22 smithi177 ceph-mon[32308]: pgmap v277: 97 pgs: 97 active+clean; 202 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 300 op/s 2024-04-23T18:27:22.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:22 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4262847763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:23.221 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.26 MB/sec execute 357 sec latency 1277.878 ms 2024-04-23T18:27:24.221 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.24 MB/sec execute 358 sec latency 2277.943 ms 2024-04-23T18:27:24.329 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:24 smithi188 ceph-mon[32694]: pgmap v278: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.5 MiB/s wr, 393 op/s 2024-04-23T18:27:24.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:24 smithi050 ceph-mon[29187]: pgmap v278: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.5 MiB/s wr, 393 op/s 2024-04-23T18:27:24.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:24 smithi177 ceph-mon[32308]: pgmap v278: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.5 MiB/s wr, 393 op/s 2024-04-23T18:27:25.221 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.22 MB/sec execute 359 sec latency 3278.071 ms 2024-04-23T18:27:26.221 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.21 MB/sec execute 360 sec latency 4278.112 ms 2024-04-23T18:27:26.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:26 smithi050 ceph-mon[29187]: pgmap v279: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 232 op/s 2024-04-23T18:27:26.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:26 smithi177 ceph-mon[32308]: pgmap v279: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 232 op/s 2024-04-23T18:27:26.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:26 smithi188 ceph-mon[32694]: pgmap v279: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 232 op/s 2024-04-23T18:27:27.221 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.19 MB/sec execute 361 sec latency 5278.211 ms 2024-04-23T18:27:28.221 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.17 MB/sec execute 362 sec latency 6278.283 ms 2024-04-23T18:27:28.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:28 smithi050 ceph-mon[29187]: pgmap v280: 97 pgs: 97 active+clean; 202 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 266 op/s 2024-04-23T18:27:28.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:28 smithi177 ceph-mon[32308]: pgmap v280: 97 pgs: 97 active+clean; 202 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 266 op/s 2024-04-23T18:27:28.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:28 smithi188 ceph-mon[32694]: pgmap v280: 97 pgs: 97 active+clean; 202 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 266 op/s 2024-04-23T18:27:29.221 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.15 MB/sec execute 363 sec latency 7278.352 ms 2024-04-23T18:27:30.221 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.14 MB/sec execute 364 sec latency 8278.513 ms 2024-04-23T18:27:30.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:30 smithi050 ceph-mon[29187]: pgmap v281: 97 pgs: 97 active+clean; 202 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.4 MiB/s wr, 163 op/s 2024-04-23T18:27:30.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:30 smithi177 ceph-mon[32308]: pgmap v281: 97 pgs: 97 active+clean; 202 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.4 MiB/s wr, 163 op/s 2024-04-23T18:27:30.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:30 smithi188 ceph-mon[32694]: pgmap v281: 97 pgs: 97 active+clean; 202 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.4 MiB/s wr, 163 op/s 2024-04-23T18:27:31.221 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.12 MB/sec execute 365 sec latency 9278.607 ms 2024-04-23T18:27:32.222 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.10 MB/sec execute 366 sec latency 10278.735 ms 2024-04-23T18:27:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:32 smithi050 ceph-mon[29187]: pgmap v282: 97 pgs: 97 active+clean; 202 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.4 MiB/s wr, 163 op/s 2024-04-23T18:27:32.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:32 smithi177 ceph-mon[32308]: pgmap v282: 97 pgs: 97 active+clean; 202 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.4 MiB/s wr, 163 op/s 2024-04-23T18:27:32.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:32 smithi188 ceph-mon[32694]: pgmap v282: 97 pgs: 97 active+clean; 202 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.4 MiB/s wr, 163 op/s 2024-04-23T18:27:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.059+0000 7f608779b640 1 -- 172.21.15.50:0/2875398123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608006b3e0 msgr2=0x7f6080106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.059+0000 7f608779b640 1 --2- 172.21.15.50:0/2875398123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608006b3e0 0x7f6080106b80 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6070009900 tx=0x7f607002a410 comp rx=0 tx=0).stop 2024-04-23T18:27:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.059+0000 7f608779b640 1 -- 172.21.15.50:0/2875398123 shutdown_connections 2024-04-23T18:27:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.059+0000 7f608779b640 1 --2- 172.21.15.50:0/2875398123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6080107260 0x7f608010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.059+0000 7f608779b640 1 --2- 172.21.15.50:0/2875398123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f608006b3e0 0x7f6080106b80 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.059+0000 7f608779b640 1 --2- 172.21.15.50:0/2875398123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f608006aa30 0x7f608006ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.059+0000 7f608779b640 1 -- 172.21.15.50:0/2875398123 >> 172.21.15.50:0/2875398123 conn(0x7f6080076c50 msgr2=0x7f6080077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.059+0000 7f608779b640 1 -- 172.21.15.50:0/2875398123 shutdown_connections 2024-04-23T18:27:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.059+0000 7f608779b640 1 -- 172.21.15.50:0/2875398123 wait complete. 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f608779b640 1 Processor -- start 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f608779b640 1 -- start start 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f608779b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f608006aa30 0x7f6080101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f608779b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6080107260 0x7f60801017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f608779b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6080101cf0 0x7f60801a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f608779b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f60801121a0 con 0x7f6080107260 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f608779b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6080112020 con 0x7f6080101cf0 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f608779b640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6080112320 con 0x7f608006aa30 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f6084d36640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6080107260 0x7f60801017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f6084d36640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6080107260 0x7f60801017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36382/0 (socket says 172.21.15.50:36382) 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f6084d36640 1 -- 172.21.15.50:0/3558805901 learned_addr learned my addr 172.21.15.50:0/3558805901 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f6085537640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f608006aa30 0x7f6080101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.060+0000 7f6085d38640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6080101cf0 0x7f60801a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.061+0000 7f6084d36640 1 -- 172.21.15.50:0/3558805901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f608006aa30 msgr2=0x7f6080101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.061+0000 7f6084d36640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f608006aa30 0x7f6080101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.061+0000 7f6084d36640 1 -- 172.21.15.50:0/3558805901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6080101cf0 msgr2=0x7f60801a6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.061+0000 7f6084d36640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6080101cf0 0x7f60801a6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.061+0000 7f6084d36640 1 -- 172.21.15.50:0/3558805901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60801a6f70 con 0x7f6080107260 2024-04-23T18:27:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.061+0000 7f6084d36640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6080107260 0x7f60801017b0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f6070006010 tx=0x7f6070004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.061+0000 7f60767fc640 1 -- 172.21.15.50:0/3558805901 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607003f030 con 0x7f6080107260 2024-04-23T18:27:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.061+0000 7f608779b640 1 -- 172.21.15.50:0/3558805901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60801a7200 con 0x7f6080107260 2024-04-23T18:27:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.061+0000 7f60767fc640 1 -- 172.21.15.50:0/3558805901 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6070003aa0 con 0x7f6080107260 2024-04-23T18:27:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.061+0000 7f60767fc640 1 -- 172.21.15.50:0/3558805901 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070035070 con 0x7f6080107260 2024-04-23T18:27:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.061+0000 7f608779b640 1 -- 172.21.15.50:0/3558805901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60801a7740 con 0x7f6080107260 2024-04-23T18:27:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.064+0000 7f60767fc640 1 -- 172.21.15.50:0/3558805901 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6070042030 con 0x7f6080107260 2024-04-23T18:27:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f608a58c740 1 Processor -- start 2024-04-23T18:27:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f608a58c740 1 -- start start 2024-04-23T18:27:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f608a58c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 0x56491935ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f608a58c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x5649193662c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f608a58c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564919366800 0x564919368bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f608a58c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5649192c0970 con 0x56491935b620 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f608a58c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56491925d4a0 con 0x564919366800 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f608a58c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5649192bc930 con 0x56491935bf40 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f6085d38640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 0x56491935ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f60767fc640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f606c0a3df0 0x7f606c0a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f6085d38640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 0x56491935ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36388/0 (socket says 172.21.15.50:36388) 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f6085d38640 1 -- 172.21.15.50:0/470390621 learned_addr learned my addr 172.21.15.50:0/470390621 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f6084d36640 1 --2- 172.21.15.50:0/470390621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564919366800 0x564919368bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f6085537640 1 --2- 172.21.15.50:0/470390621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x5649193662c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f6085d38640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f606c0a3df0 0x7f606c0a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.065+0000 7f60767fc640 1 -- 172.21.15.50:0/3558805901 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f60700f0b60 con 0x7f6080107260 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.066+0000 7f6085d38640 1 -- 172.21.15.50:0/470390621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 msgr2=0x5649193662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.066+0000 7f6085d38640 1 --2- 172.21.15.50:0/470390621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x5649193662c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.066+0000 7f6085d38640 1 -- 172.21.15.50:0/470390621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564919366800 msgr2=0x564919368bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.066+0000 7f6085d38640 1 --2- 172.21.15.50:0/470390621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564919366800 0x564919368bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.066+0000 7f6085d38640 1 -- 172.21.15.50:0/470390621 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564918fd3210 con 0x56491935b620 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.066+0000 7f6085d38640 1 --2- 172.21.15.50:0/470390621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 0x56491935ba00 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f60780049d0 tx=0x7f6078013e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f2ed8bcdc74b574 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.066+0000 7f606b7fe640 1 -- 172.21.15.50:0/470390621 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078022020 con 0x56491935b620 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.066+0000 7f606b7fe640 1 -- 172.21.15.50:0/470390621 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60780133d0 con 0x56491935b620 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.066+0000 7f606b7fe640 1 -- 172.21.15.50:0/470390621 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078010820 con 0x56491935b620 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.066+0000 7f6085d38640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f606c0a3df0 0x7f606c0a62b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6078023000 tx=0x7f6078021000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.067+0000 7f608a58c740 1 -- 172.21.15.50:0/470390621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 msgr2=0x56491935ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.067+0000 7f608a58c740 1 --2- 172.21.15.50:0/470390621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 0x56491935ba00 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f60780049d0 tx=0x7f6078013e90 comp rx=0 tx=0).stop 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.067+0000 7f608a58c740 1 -- 172.21.15.50:0/470390621 shutdown_connections 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.067+0000 7f608a58c740 1 --2- 172.21.15.50:0/470390621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564919366800 0x564919368bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.067+0000 7f608a58c740 1 --2- 172.21.15.50:0/470390621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x5649193662c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.067+0000 7f608a58c740 1 --2- 172.21.15.50:0/470390621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 0x56491935ba00 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.067+0000 7f608a58c740 1 -- 172.21.15.50:0/470390621 >> 172.21.15.50:0/470390621 conn(0x564919357b30 msgr2=0x564919357f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.067+0000 7f608a58c740 1 -- 172.21.15.50:0/470390621 shutdown_connections 2024-04-23T18:27:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.067+0000 7f608a58c740 1 -- 172.21.15.50:0/470390621 wait complete. 2024-04-23T18:27:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.067+0000 7f608a58c740 1 -- start start 2024-04-23T18:27:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.067+0000 7f608a58c740 1 Processor -- start 2024-04-23T18:27:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f608a58c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 0x564919366140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f608a58c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x564919363430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f608a58c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564919366800 0x564919363970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f608a58c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564919369430 con 0x56491935b620 2024-04-23T18:27:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f608a58c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56491925b0b0 con 0x564919366800 2024-04-23T18:27:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f608a58c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5649193695b0 con 0x56491935bf40 2024-04-23T18:27:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f608a58c740 10 client.0 _ll_register_callbacks cb 0x564919248d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:33.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6085537640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 0x564919366140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6085537640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 0x564919366140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36404/0 (socket says 172.21.15.50:36404) 2024-04-23T18:27:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6085537640 1 -- 172.21.15.50:0/2586847710 learned_addr learned my addr 172.21.15.50:0/2586847710 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6085d38640 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564919366800 0x564919363970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6084d36640 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x564919363430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6085537640 1 -- 172.21.15.50:0/2586847710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 msgr2=0x564919363430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6085537640 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x564919363430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6085537640 1 -- 172.21.15.50:0/2586847710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564919366800 msgr2=0x564919363970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6085537640 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564919366800 0x564919363970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6085537640 1 -- 172.21.15.50:0/2586847710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56491909f620 con 0x56491935b620 2024-04-23T18:27:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6085d38640 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564919366800 0x564919363970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:33.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6085537640 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 0x564919366140 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f607c00bad0 tx=0x7f607c00bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52dcfa3f07efeece server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.068+0000 7f6068ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f6068ff9640 1 -- 172.21.15.50:0/2586847710 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c003e00 con 0x56491935b620 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f6068ff9640 1 -- 172.21.15.50:0/2586847710 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f607c005020 con 0x56491935b620 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f608a58c740 1 -- 172.21.15.50:0/2586847710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60780133d0 con 0x56491935b620 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f6068ff9640 1 -- 172.21.15.50:0/2586847710 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f607c005220 con 0x56491935b620 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f608a58c740 10 client.16584 fetch_fsmap learned FSMap version 39 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f608a58c740 1 -- 172.21.15.50:0/2586847710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5649193753a0 con 0x56491935b620 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f6068ff9640 1 -- 172.21.15.50:0/2586847710 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60780133d0 con 0x56491935b620 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f6068ff9640 1 -- 172.21.15.50:0/2586847710 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f607c01b570 con 0x56491935b620 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f608a58c740 10 client.16584 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f608a58c740 10 client.16584 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f608a58c740 20 client.16584 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f608a58c740 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564919376860 0x564919378c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f608a58c740 4 client.16584 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.069+0000 7f608a58c740 1 -- 172.21.15.50:0/2586847710 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649192ce390 con 0x564919376860 2024-04-23T18:27:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.070+0000 7f6084d36640 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564919376860 0x564919378c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.070+0000 7f6068ff9640 10 client.16584 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.070+0000 7f6084d36640 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564919376860 0x564919378c50 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1a4619f177bec14 server_cookie=87a1c5ab7cec3cbe in_seq=0 out_seq=0 2024-04-23T18:27:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f6068ff9640 1 -- 172.21.15.50:0/2586847710 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5649192ce390 con 0x564919376860 2024-04-23T18:27:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f6068ff9640 10 client.16584 handle_command_reply: tid=0 2024-04-23T18:27:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f608a58c740 1 client.16584 shutdown 2024-04-23T18:27:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f608a58c740 1 -- 172.21.15.50:0/2586847710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 msgr2=0x564919366140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f608a58c740 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 0x564919366140 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f607c00bad0 tx=0x7f607c00bf90 comp rx=0 tx=0).stop 2024-04-23T18:27:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f608a58c740 1 -- 172.21.15.50:0/2586847710 shutdown_connections 2024-04-23T18:27:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f608a58c740 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564919376860 0x564919378c50 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f608a58c740 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564919366800 0x564919363970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f6068ff9640 0 client.16584 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f608a58c740 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x564919363430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f608a58c740 1 --2- 172.21.15.50:0/2586847710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935b620 0x564919366140 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f608a58c740 1 -- 172.21.15.50:0/2586847710 >> 172.21.15.50:0/2586847710 conn(0x564919357b30 msgr2=0x56491935d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f608a58c740 1 -- 172.21.15.50:0/2586847710 shutdown_connections 2024-04-23T18:27:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f608a58c740 1 -- 172.21.15.50:0/2586847710 wait complete. 2024-04-23T18:27:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.071+0000 7f608a58c740 20 client.16584 trim_cache size 0 max 16384 2024-04-23T18:27:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.086+0000 7f608a58c740 1 Processor -- start 2024-04-23T18:27:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.086+0000 7f608a58c740 1 -- start start 2024-04-23T18:27:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.086+0000 7f608a58c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56491935b620 0x5649193722f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.086+0000 7f608a58c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x56491937e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.086+0000 7f608a58c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564919366800 0x56491937eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.086+0000 7f608a58c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f60801b26e0 con 0x564919366800 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.086+0000 7f608a58c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f60801aff50 con 0x56491935b620 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f608a58c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f60801addc0 con 0x56491935bf40 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f6085d38640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564919366800 0x56491937eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f6085d38640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564919366800 0x56491937eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36414/0 (socket says 172.21.15.50:36414) 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f6085d38640 1 -- 172.21.15.50:0/3585201242 learned_addr learned my addr 172.21.15.50:0/3585201242 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f6085537640 1 --2- 172.21.15.50:0/3585201242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56491935b620 0x5649193722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f6084d36640 1 --2- 172.21.15.50:0/3585201242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x56491937e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f6085d38640 1 -- 172.21.15.50:0/3585201242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 msgr2=0x56491937e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f6085d38640 1 --2- 172.21.15.50:0/3585201242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x56491937e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f6085d38640 1 -- 172.21.15.50:0/3585201242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56491935b620 msgr2=0x5649193722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f6085d38640 1 --2- 172.21.15.50:0/3585201242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56491935b620 0x5649193722f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f6085d38640 1 -- 172.21.15.50:0/3585201242 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56491935ae20 con 0x564919366800 2024-04-23T18:27:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f6084d36640 1 --2- 172.21.15.50:0/3585201242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x56491937e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f6085d38640 1 --2- 172.21.15.50:0/3585201242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564919366800 0x56491937eaa0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6078014fd0 tx=0x7f60780137b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afc7828de790a5e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f606affd640 1 -- 172.21.15.50:0/3585201242 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078022070 con 0x564919366800 2024-04-23T18:27:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f606affd640 1 -- 172.21.15.50:0/3585201242 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60780133d0 con 0x564919366800 2024-04-23T18:27:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.087+0000 7f606affd640 1 -- 172.21.15.50:0/3585201242 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078010e60 con 0x564919366800 2024-04-23T18:27:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.088+0000 7f608a58c740 1 -- 172.21.15.50:0/3585201242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564919366800 msgr2=0x56491937eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.088+0000 7f608a58c740 1 --2- 172.21.15.50:0/3585201242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564919366800 0x56491937eaa0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6078014fd0 tx=0x7f60780137b0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.088+0000 7f608a58c740 1 -- 172.21.15.50:0/3585201242 shutdown_connections 2024-04-23T18:27:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.088+0000 7f608a58c740 1 --2- 172.21.15.50:0/3585201242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564919366800 0x56491937eaa0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.088+0000 7f608a58c740 1 --2- 172.21.15.50:0/3585201242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56491935bf40 0x56491937e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.088+0000 7f608a58c740 1 --2- 172.21.15.50:0/3585201242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56491935b620 0x5649193722f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.088+0000 7f608a58c740 1 -- 172.21.15.50:0/3585201242 >> 172.21.15.50:0/3585201242 conn(0x564919357b30 msgr2=0x564919363790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.088+0000 7f608a58c740 1 -- 172.21.15.50:0/3585201242 shutdown_connections 2024-04-23T18:27:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.088+0000 7f608a58c740 1 -- 172.21.15.50:0/3585201242 wait complete. 2024-04-23T18:27:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.088+0000 7f608a58c740 1 -- start start 2024-04-23T18:27:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.088+0000 7f608a58c740 1 Processor -- start 2024-04-23T18:27:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f608a58c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56491935b620 0x56491936e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f608a58c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935bf40 0x56491936bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f608a58c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564919366800 0x56491936c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f608a58c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f60801ae410 con 0x56491935bf40 2024-04-23T18:27:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f608a58c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f60801af880 con 0x56491935b620 2024-04-23T18:27:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f608a58c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f60801aeae0 con 0x564919366800 2024-04-23T18:27:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f608a58c740 10 client.0 _ll_register_callbacks cb 0x7f607c004400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6084d36640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935bf40 0x56491936bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6084d36640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935bf40 0x56491936bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36424/0 (socket says 172.21.15.50:36424) 2024-04-23T18:27:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6084d36640 1 -- 172.21.15.50:0/2580166133 learned_addr learned my addr 172.21.15.50:0/2580166133 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6085537640 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56491935b620 0x56491936e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6085d38640 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564919366800 0x56491936c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6084d36640 1 -- 172.21.15.50:0/2580166133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564919366800 msgr2=0x56491936c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6084d36640 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564919366800 0x56491936c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6084d36640 1 -- 172.21.15.50:0/2580166133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56491935b620 msgr2=0x56491936e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6084d36640 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56491935b620 0x56491936e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6084d36640 1 -- 172.21.15.50:0/2580166133 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56491936c700 con 0x56491935bf40 2024-04-23T18:27:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6085537640 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56491935b620 0x56491936e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6084d36640 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935bf40 0x56491936bc80 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f60700ee350 tx=0x7f607008c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb09582dd8fac1c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.089+0000 7f6068ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f6068ff9640 1 -- 172.21.15.50:0/2580166133 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070049040 con 0x56491935bf40 2024-04-23T18:27:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f6068ff9640 1 -- 172.21.15.50:0/2580166133 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f6070003c50 con 0x56491935bf40 2024-04-23T18:27:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f608a58c740 1 -- 172.21.15.50:0/2580166133 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60780133d0 con 0x56491935bf40 2024-04-23T18:27:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f6068ff9640 1 -- 172.21.15.50:0/2580166133 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f60700c0ea0 con 0x56491935bf40 2024-04-23T18:27:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f608a58c740 10 client.16590 fetch_fsmap learned FSMap version 39 2024-04-23T18:27:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f608a58c740 1 -- 172.21.15.50:0/2580166133 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5649193dca90 con 0x56491935bf40 2024-04-23T18:27:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f6068ff9640 1 -- 172.21.15.50:0/2580166133 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60780133d0 con 0x56491935bf40 2024-04-23T18:27:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f6068ff9640 1 -- 172.21.15.50:0/2580166133 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f60700bce00 con 0x56491935bf40 2024-04-23T18:27:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f608a58c740 10 client.16590 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T18:27:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f608a58c740 10 client.16590 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f608a58c740 20 client.16590 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f608a58c740 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5649193dd210 0x5649193df630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f608a58c740 4 client.16590 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:27:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.090+0000 7f608a58c740 1 -- 172.21.15.50:0/2580166133 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5649192c8200 con 0x5649193dd210 2024-04-23T18:27:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.091+0000 7f6085537640 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5649193dd210 0x5649193df630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.091+0000 7f6068ff9640 10 client.16590 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.091+0000 7f6085537640 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5649193dd210 0x5649193df630 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8877097a5c71e69 server_cookie=7f871a077f6127b9 in_seq=0 out_seq=0 2024-04-23T18:27:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f6068ff9640 1 -- 172.21.15.50:0/2580166133 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5649192c8200 con 0x5649193dd210 2024-04-23T18:27:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f6068ff9640 10 client.16590 handle_command_reply: tid=0 2024-04-23T18:27:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f608a58c740 1 client.16590 shutdown 2024-04-23T18:27:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f608a58c740 1 -- 172.21.15.50:0/2580166133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935bf40 msgr2=0x56491936bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f608a58c740 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935bf40 0x56491936bc80 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f60700ee350 tx=0x7f607008c480 comp rx=0 tx=0).stop 2024-04-23T18:27:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f608a58c740 1 -- 172.21.15.50:0/2580166133 shutdown_connections 2024-04-23T18:27:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f608a58c740 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5649193dd210 0x5649193df630 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f608a58c740 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564919366800 0x56491936c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f608a58c740 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56491935bf40 0x56491936bc80 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f608a58c740 1 --2- 172.21.15.50:0/2580166133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56491935b620 0x56491936e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f6068ff9640 0 client.16590 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f608a58c740 1 -- 172.21.15.50:0/2580166133 >> 172.21.15.50:0/2580166133 conn(0x564919357b30 msgr2=0x564919356cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f608a58c740 1 -- 172.21.15.50:0/2580166133 shutdown_connections 2024-04-23T18:27:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.129+0000 7f608a58c740 1 -- 172.21.15.50:0/2580166133 wait complete. 2024-04-23T18:27:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608a58c740 20 client.16590 trim_cache size 0 max 16384 2024-04-23T18:27:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:27:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:27:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:27:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:27:33.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608779b640 1 -- 172.21.15.50:0/3558805901 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f606c0a3df0 msgr2=0x7f606c0a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:33.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608779b640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f606c0a3df0 0x7f606c0a62b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6078023000 tx=0x7f6078021000 comp rx=0 tx=0).stop 2024-04-23T18:27:33.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608779b640 1 -- 172.21.15.50:0/3558805901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6080107260 msgr2=0x7f60801017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:33.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608779b640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6080107260 0x7f60801017b0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f6070006010 tx=0x7f6070004060 comp rx=0 tx=0).stop 2024-04-23T18:27:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608779b640 1 -- 172.21.15.50:0/3558805901 shutdown_connections 2024-04-23T18:27:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608779b640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f606c0a3df0 0x7f606c0a62b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608779b640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6080101cf0 0x7f60801a6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608779b640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6080107260 0x7f60801017b0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608779b640 1 --2- 172.21.15.50:0/3558805901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f608006aa30 0x7f6080101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608779b640 1 -- 172.21.15.50:0/3558805901 >> 172.21.15.50:0/3558805901 conn(0x7f6080076c50 msgr2=0x7f60800ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608779b640 1 -- 172.21.15.50:0/3558805901 shutdown_connections 2024-04-23T18:27:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:33.130+0000 7f608779b640 1 -- 172.21.15.50:0/3558805901 wait complete. 2024-04-23T18:27:33.221 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.09 MB/sec execute 367 sec latency 11278.843 ms 2024-04-23T18:27:34.222 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.07 MB/sec execute 368 sec latency 12278.924 ms 2024-04-23T18:27:34.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:34 smithi050 ceph-mon[29187]: pgmap v283: 97 pgs: 97 active+clean; 200 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 3.1 MiB/s wr, 197 op/s 2024-04-23T18:27:34.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:34 smithi177 ceph-mon[32308]: pgmap v283: 97 pgs: 97 active+clean; 200 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 3.1 MiB/s wr, 197 op/s 2024-04-23T18:27:34.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:34 smithi188 ceph-mon[32694]: pgmap v283: 97 pgs: 97 active+clean; 200 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 3.1 MiB/s wr, 197 op/s 2024-04-23T18:27:35.222 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.05 MB/sec execute 369 sec latency 13279.014 ms 2024-04-23T18:27:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:35 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:35.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:35 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:35.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:35 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:27:36.222 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.04 MB/sec execute 370 sec latency 14279.142 ms 2024-04-23T18:27:36.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:36 smithi050 ceph-mon[29187]: pgmap v284: 97 pgs: 97 active+clean; 200 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 1.5 MiB/s wr, 68 op/s 2024-04-23T18:27:36.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:36 smithi177 ceph-mon[32308]: pgmap v284: 97 pgs: 97 active+clean; 200 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 1.5 MiB/s wr, 68 op/s 2024-04-23T18:27:36.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:36 smithi188 ceph-mon[32694]: pgmap v284: 97 pgs: 97 active+clean; 200 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 1.5 MiB/s wr, 68 op/s 2024-04-23T18:27:37.222 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.02 MB/sec execute 371 sec latency 15279.191 ms 2024-04-23T18:27:38.222 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 6.01 MB/sec execute 372 sec latency 16279.254 ms 2024-04-23T18:27:38.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:38 smithi050 ceph-mon[29187]: pgmap v285: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 2.1 MiB/s wr, 97 op/s 2024-04-23T18:27:38.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:38 smithi177 ceph-mon[32308]: pgmap v285: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 2.1 MiB/s wr, 97 op/s 2024-04-23T18:27:38.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:38 smithi188 ceph-mon[32694]: pgmap v285: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 2.1 MiB/s wr, 97 op/s 2024-04-23T18:27:39.222 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 5.99 MB/sec execute 373 sec latency 17279.292 ms 2024-04-23T18:27:40.222 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 5.97 MB/sec execute 374 sec latency 18279.409 ms 2024-04-23T18:27:40.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:40 smithi050 ceph-mon[29187]: pgmap v286: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-23T18:27:40.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:40 smithi177 ceph-mon[32308]: pgmap v286: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-23T18:27:40.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:40 smithi188 ceph-mon[32694]: pgmap v286: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-23T18:27:41.222 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 5.96 MB/sec execute 375 sec latency 19279.516 ms 2024-04-23T18:27:42.223 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 5.94 MB/sec execute 376 sec latency 20279.673 ms 2024-04-23T18:27:42.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:42 smithi050 ceph-mon[29187]: pgmap v287: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-23T18:27:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:42 smithi177 ceph-mon[32308]: pgmap v287: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-23T18:27:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:42 smithi188 ceph-mon[32694]: pgmap v287: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-23T18:27:43.223 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 5.93 MB/sec execute 377 sec latency 21279.789 ms 2024-04-23T18:27:44.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:27:44.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.128+0000 7f8b7813d640 1 -- 172.21.15.50:0/2419705992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b70102b40 msgr2=0x7f8b70106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.128+0000 7f8b7813d640 1 --2- 172.21.15.50:0/2419705992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b70102b40 0x7f8b70106b80 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f8b60009900 tx=0x7f8b6002a410 comp rx=0 tx=0).stop 2024-04-23T18:27:44.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.128+0000 7f8b7813d640 1 -- 172.21.15.50:0/2419705992 shutdown_connections 2024-04-23T18:27:44.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.128+0000 7f8b7813d640 1 --2- 172.21.15.50:0/2419705992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b70107260 0x7f8b7010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.128+0000 7f8b7813d640 1 --2- 172.21.15.50:0/2419705992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b70102b40 0x7f8b70106b80 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.128+0000 7f8b7813d640 1 --2- 172.21.15.50:0/2419705992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b70102190 0x7f8b70102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.128+0000 7f8b7813d640 1 -- 172.21.15.50:0/2419705992 >> 172.21.15.50:0/2419705992 conn(0x7f8b70079310 msgr2=0x7f8b70079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:44.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.128+0000 7f8b7813d640 1 -- 172.21.15.50:0/2419705992 shutdown_connections 2024-04-23T18:27:44.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.128+0000 7f8b7813d640 1 -- 172.21.15.50:0/2419705992 wait complete. 2024-04-23T18:27:44.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b7813d640 1 Processor -- start 2024-04-23T18:27:44.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b7813d640 1 -- start start 2024-04-23T18:27:44.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b7813d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b70102190 0x7f8b701a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b7813d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b70107260 0x7f8b701a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b7813d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b7010e030 0x7f8b7010e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b7813d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8b701121f0 con 0x7f8b70107260 2024-04-23T18:27:44.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b7813d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b70112070 con 0x7f8b70102190 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b7813d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8b70112370 con 0x7f8b7010e030 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b756d8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b70107260 0x7f8b701a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b756d8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b70107260 0x7f8b701a3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40536/0 (socket says 172.21.15.50:40536) 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b756d8640 1 -- 172.21.15.50:0/3607753653 learned_addr learned my addr 172.21.15.50:0/3607753653 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b75ed9640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b70102190 0x7f8b701a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.129+0000 7f8b766da640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b7010e030 0x7f8b7010e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.130+0000 7f8b756d8640 1 -- 172.21.15.50:0/3607753653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b7010e030 msgr2=0x7f8b7010e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.130+0000 7f8b756d8640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b7010e030 0x7f8b7010e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.130+0000 7f8b756d8640 1 -- 172.21.15.50:0/3607753653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b70102190 msgr2=0x7f8b701a3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.130+0000 7f8b756d8640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b70102190 0x7f8b701a3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.130+0000 7f8b756d8640 1 -- 172.21.15.50:0/3607753653 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b7010ec10 con 0x7f8b70107260 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.130+0000 7f8b756d8640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b70107260 0x7f8b701a3ed0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8b60005ab0 tx=0x7f8b60003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.130+0000 7f8b66ffd640 1 -- 172.21.15.50:0/3607753653 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6003f070 con 0x7f8b70107260 2024-04-23T18:27:44.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.130+0000 7f8b66ffd640 1 -- 172.21.15.50:0/3607753653 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b60031b50 con 0x7f8b70107260 2024-04-23T18:27:44.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.130+0000 7f8b7813d640 1 -- 172.21.15.50:0/3607753653 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b701ab8b0 con 0x7f8b70107260 2024-04-23T18:27:44.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.130+0000 7f8b66ffd640 1 -- 172.21.15.50:0/3607753653 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60035030 con 0x7f8b70107260 2024-04-23T18:27:44.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.130+0000 7f8b7813d640 1 -- 172.21.15.50:0/3607753653 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b701abc60 con 0x7f8b70107260 2024-04-23T18:27:44.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.131+0000 7f8b7813d640 1 -- 172.21.15.50:0/3607753653 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b700691c0 con 0x7f8b70107260 2024-04-23T18:27:44.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.132+0000 7f8b66ffd640 1 -- 172.21.15.50:0/3607753653 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8b60042070 con 0x7f8b70107260 2024-04-23T18:27:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.133+0000 7f8b66ffd640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f8b5c0a3da0 0x7f8b5c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.133+0000 7f8b75ed9640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f8b5c0a3da0 0x7f8b5c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.133+0000 7f8b66ffd640 1 -- 172.21.15.50:0/3607753653 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f8b600f1180 con 0x7f8b70107260 2024-04-23T18:27:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.134+0000 7f8b75ed9640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f8b5c0a3da0 0x7f8b5c0a6260 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c009bd0 tx=0x7f8b6c005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.139+0000 7f8b66ffd640 1 -- 172.21.15.50:0/3607753653 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b600bd160 con 0x7f8b70107260 2024-04-23T18:27:44.223 INFO:tasks.workunit.client.0.smithi050.stdout: 1 606366 5.91 MB/sec execute 378 sec latency 22279.836 ms 2024-04-23T18:27:44.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.370+0000 7f8b7813d640 1 -- 172.21.15.50:0/3607753653 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b70102b70 con 0x7f8b70107260 2024-04-23T18:27:44.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.370+0000 7f8b66ffd640 1 -- 172.21.15.50:0/3607753653 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11881 (secure 0 0 0) 0x7f8b600c1460 con 0x7f8b70107260 2024-04-23T18:27:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:27:44.364 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:27:33.821377+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/4033440314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4033440314},{"type":"v1","addr":"172.21.15.177:6837","nonce":4033440314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:27:44.364 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 40 2024-04-23T18:27:44.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.372+0000 7f8b7813d640 1 -- 172.21.15.50:0/3607753653 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f8b5c0a3da0 msgr2=0x7f8b5c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.372+0000 7f8b7813d640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f8b5c0a3da0 0x7f8b5c0a6260 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c009bd0 tx=0x7f8b6c005f90 comp rx=0 tx=0).stop 2024-04-23T18:27:44.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.372+0000 7f8b7813d640 1 -- 172.21.15.50:0/3607753653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b70107260 msgr2=0x7f8b701a3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.372+0000 7f8b7813d640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b70107260 0x7f8b701a3ed0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8b60005ab0 tx=0x7f8b60003b90 comp rx=0 tx=0).stop 2024-04-23T18:27:44.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.372+0000 7f8b7813d640 1 -- 172.21.15.50:0/3607753653 shutdown_connections 2024-04-23T18:27:44.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.372+0000 7f8b7813d640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f8b5c0a3da0 0x7f8b5c0a6260 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.372+0000 7f8b7813d640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b7010e030 0x7f8b7010e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.372+0000 7f8b7813d640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b70107260 0x7f8b701a3ed0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.372+0000 7f8b7813d640 1 --2- 172.21.15.50:0/3607753653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b70102190 0x7f8b701a3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.372+0000 7f8b7813d640 1 -- 172.21.15.50:0/3607753653 >> 172.21.15.50:0/3607753653 conn(0x7f8b70079310 msgr2=0x7f8b700fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:44.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.372+0000 7f8b7813d640 1 -- 172.21.15.50:0/3607753653 shutdown_connections 2024-04-23T18:27:44.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.372+0000 7f8b7813d640 1 -- 172.21.15.50:0/3607753653 wait complete. 2024-04-23T18:27:44.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:44 smithi050 ceph-mon[29187]: pgmap v288: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-04-23T18:27:44.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:44 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:27:44.376 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id c31bc802 --release --await' 2024-04-23T18:27:44.377 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id c31bc802 --release --await 2024-04-23T18:27:44.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.493+0000 7fb2f8acf640 1 -- 172.21.15.50:0/119688117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2f4102190 msgr2=0x7fb2f4102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.493+0000 7fb2f8acf640 1 --2- 172.21.15.50:0/119688117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2f4102190 0x7fb2f4102570 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009900 tx=0x7fb2e402a410 comp rx=0 tx=0).stop 2024-04-23T18:27:44.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.493+0000 7fb2f8acf640 1 -- 172.21.15.50:0/119688117 shutdown_connections 2024-04-23T18:27:44.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.493+0000 7fb2f8acf640 1 --2- 172.21.15.50:0/119688117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2f4107260 0x7fb2f410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.493+0000 7fb2f8acf640 1 --2- 172.21.15.50:0/119688117 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f4102b40 0x7fb2f4106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.493+0000 7fb2f8acf640 1 --2- 172.21.15.50:0/119688117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2f4102190 0x7fb2f4102570 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.493+0000 7fb2f8acf640 1 -- 172.21.15.50:0/119688117 >> 172.21.15.50:0/119688117 conn(0x7fb2f40793a0 msgr2=0x7fb2f40797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:44.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.493+0000 7fb2f8acf640 1 -- 172.21.15.50:0/119688117 shutdown_connections 2024-04-23T18:27:44.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.493+0000 7fb2f8acf640 1 -- 172.21.15.50:0/119688117 wait complete. 2024-04-23T18:27:44.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.493+0000 7fb2f8acf640 1 Processor -- start 2024-04-23T18:27:44.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.493+0000 7fb2f8acf640 1 -- start start 2024-04-23T18:27:44.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f8acf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2f4102190 0x7fb2f419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f8acf640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f4107260 0x7fb2f419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f8acf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2f41985a0 0x7fb2f4198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f8acf640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f41121c0 con 0x7fb2f41985a0 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f8acf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f4112040 con 0x7fb2f4102190 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f8acf640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f4112340 con 0x7fb2f4107260 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2f41985a0 0x7fb2f4198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2f41985a0 0x7fb2f4198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40554/0 (socket says 172.21.15.50:40554) 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f2d9d640 1 -- 172.21.15.50:0/1353402877 learned_addr learned my addr 172.21.15.50:0/1353402877 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f259c640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2f4102190 0x7fb2f419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f4107260 0x7fb2f419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f2d9d640 1 -- 172.21.15.50:0/1353402877 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f4107260 msgr2=0x7fb2f419e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f2d9d640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f4107260 0x7fb2f419e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f2d9d640 1 -- 172.21.15.50:0/1353402877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2f4102190 msgr2=0x7fb2f419e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f2d9d640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2f4102190 0x7fb2f419e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f2d9d640 1 -- 172.21.15.50:0/1353402877 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f41990c0 con 0x7fb2f41985a0 2024-04-23T18:27:44.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f4107260 0x7fb2f419e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:44.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f2d9d640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2f41985a0 0x7fb2f4198a50 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec009fd0 tx=0x7fb2ec00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:44.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2eb7fe640 1 -- 172.21.15.50:0/1353402877 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec017070 con 0x7fb2f41985a0 2024-04-23T18:27:44.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2eb7fe640 1 -- 172.21.15.50:0/1353402877 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2ec003e20 con 0x7fb2f41985a0 2024-04-23T18:27:44.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f8acf640 1 -- 172.21.15.50:0/1353402877 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f41993b0 con 0x7fb2f41985a0 2024-04-23T18:27:44.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2eb7fe640 1 -- 172.21.15.50:0/1353402877 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec010460 con 0x7fb2f41985a0 2024-04-23T18:27:44.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.494+0000 7fb2f8acf640 1 -- 172.21.15.50:0/1353402877 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f41a55b0 con 0x7fb2f41985a0 2024-04-23T18:27:44.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.497+0000 7fb2eb7fe640 1 -- 172.21.15.50:0/1353402877 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb2ec003970 con 0x7fb2f41985a0 2024-04-23T18:27:44.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.497+0000 7fb2fb7e4740 1 Processor -- start 2024-04-23T18:27:44.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.497+0000 7fb2fb7e4740 1 -- start start 2024-04-23T18:27:44.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.497+0000 7fb2fb7e4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71c98750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.497+0000 7fb2fb7e4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71c98c90 0x55fd71ca7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.497+0000 7fb2fb7e4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71ca7890 0x55fd71ca9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.497+0000 7fb2fb7e4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55fd71bfd6d0 con 0x55fd71ca7890 2024-04-23T18:27:44.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.497+0000 7fb2fb7e4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fd71b9a1a0 con 0x55fd71c98c90 2024-04-23T18:27:44.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.497+0000 7fb2fb7e4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fd71bf9690 con 0x55fd71c98370 2024-04-23T18:27:44.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.497+0000 7fb2eb7fe640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2e00a3d20 0x7fb2e00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.497+0000 7fb2ebfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71ca7890 0x55fd71ca9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2ebfff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71c98750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2f259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71c98c90 0x55fd71ca7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2ebfff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71c98750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:50508/0 (socket says 172.21.15.50:50508) 2024-04-23T18:27:44.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2f259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71c98c90 0x55fd71ca7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:56280/0 (socket says 172.21.15.50:56280) 2024-04-23T18:27:44.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2ebfff640 1 -- 172.21.15.50:0/4220817878 learned_addr learned my addr 172.21.15.50:0/4220817878 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2f2d9d640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2e00a3d20 0x7fb2e00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2eb7fe640 1 -- 172.21.15.50:0/1353402877 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fb2ec0d0550 con 0x7fb2f41985a0 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2ebfff640 1 -- 172.21.15.50:0/4220817878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71c98c90 msgr2=0x55fd71ca7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/4220817878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71c98c90 0x55fd71ca7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2ebfff640 1 -- 172.21.15.50:0/4220817878 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71ca7890 msgr2=0x55fd71ca9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/4220817878 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71ca7890 0x55fd71ca9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2ebfff640 1 -- 172.21.15.50:0/4220817878 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd71ba45b0 con 0x55fd71c98370 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2f2d9d640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2e00a3d20 0x7fb2e00a61e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec09c1e0 tx=0x7fb2ec067030 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/4220817878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71c98750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0138c0 tx=0x7fb2dc005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ea3a1d3b71cb4c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2e8ff9640 1 -- 172.21.15.50:0/4220817878 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc010970 con 0x55fd71c98370 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2e8ff9640 1 -- 172.21.15.50:0/4220817878 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2dc01d390 con 0x55fd71c98370 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.498+0000 7fb2e8ff9640 1 -- 172.21.15.50:0/4220817878 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc01e5e0 con 0x55fd71c98370 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.499+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/4220817878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 msgr2=0x55fd71c98750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.499+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/4220817878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71c98750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0138c0 tx=0x7fb2dc005c90 comp rx=0 tx=0).stop 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.499+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/4220817878 shutdown_connections 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.499+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/4220817878 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71ca7890 0x55fd71ca9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.499+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/4220817878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71c98c90 0x55fd71ca7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.499+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/4220817878 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71c98750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.499+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/4220817878 >> 172.21.15.50:0/4220817878 conn(0x55fd71c947b0 msgr2=0x55fd71c94b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.499+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/4220817878 shutdown_connections 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.499+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/4220817878 wait complete. 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.499+0000 7fb2fb7e4740 1 -- start start 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2fb7e4740 1 Processor -- start 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2fb7e4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98370 0x55fd71ca7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2fb7e4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71c98c90 0x55fd71ca4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2fb7e4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71ca7890 0x55fd71ca4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2fb7e4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55fd71caa710 con 0x55fd71c98370 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2fb7e4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fd71b97b20 con 0x55fd71c98c90 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2fb7e4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fd71caa890 con 0x55fd71ca7890 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2ebfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98370 0x55fd71ca7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2fb7e4740 10 client.0 _ll_register_callbacks cb 0x55fd71b97530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2ebfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98370 0x55fd71ca7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40580/0 (socket says 172.21.15.50:40580) 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2ebfff640 1 -- 172.21.15.50:0/753520559 learned_addr learned my addr 172.21.15.50:0/753520559 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71ca7890 0x55fd71ca4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2f259c640 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71c98c90 0x55fd71ca4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2ebfff640 1 -- 172.21.15.50:0/753520559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71ca7890 msgr2=0x55fd71ca4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71ca7890 0x55fd71ca4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2ebfff640 1 -- 172.21.15.50:0/753520559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71c98c90 msgr2=0x55fd71ca4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71c98c90 0x55fd71ca4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2ebfff640 1 -- 172.21.15.50:0/753520559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd7190f920 con 0x55fd71c98370 2024-04-23T18:27:44.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98370 0x55fd71ca7260 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc00d4b0 tx=0x7fb2dc00d420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eed6ab56e3c1a57 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.500+0000 7fb2d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2d9ffb640 1 -- 172.21.15.50:0/753520559 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc028040 con 0x55fd71c98370 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2d9ffb640 1 -- 172.21.15.50:0/753520559 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fb2dc010770 con 0x55fd71c98370 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/753520559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2dc01d390 con 0x55fd71c98370 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2d9ffb640 1 -- 172.21.15.50:0/753520559 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb2dc035050 con 0x55fd71c98370 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2fb7e4740 10 client.16608 fetch_fsmap learned FSMap version 40 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/753520559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55fd719dbd30 con 0x55fd71c98370 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2d9ffb640 1 -- 172.21.15.50:0/753520559 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2dc01d390 con 0x55fd71c98370 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2d9ffb640 1 -- 172.21.15.50:0/753520559 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fb2dc01d070 con 0x55fd71c98370 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2fb7e4740 10 client.16608 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2fb7e4740 10 client.16608 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2fb7e4740 20 client.16608 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fd71cb78d0 0x55fd71cb9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2fb7e4740 4 client.16608 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.501+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/753520559 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd71bbb500 con 0x55fd71cb78d0 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.502+0000 7fb2f259c640 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fd71cb78d0 0x55fd71cb9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.502+0000 7fb2f259c640 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fd71cb78d0 0x55fd71cb9cc0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca119754e9899d55 server_cookie=9c861d5a5fcc37c2 in_seq=0 out_seq=0 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.502+0000 7fb2d9ffb640 10 client.16608 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2d9ffb640 1 -- 172.21.15.50:0/753520559 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd71bbb500 con 0x55fd71cb78d0 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2d9ffb640 10 client.16608 handle_command_reply: tid=0 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2fb7e4740 1 client.16608 shutdown 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/753520559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98370 msgr2=0x55fd71ca7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98370 0x55fd71ca7260 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc00d4b0 tx=0x7fb2dc00d420 comp rx=0 tx=0).stop 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/753520559 shutdown_connections 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fd71cb78d0 0x55fd71cb9cc0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71ca7890 0x55fd71ca4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71c98c90 0x55fd71ca4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2d9ffb640 0 client.16608 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:44.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/753520559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98370 0x55fd71ca7260 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/753520559 >> 172.21.15.50:0/753520559 conn(0x55fd71c947b0 msgr2=0x55fd71c9a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:44.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/753520559 shutdown_connections 2024-04-23T18:27:44.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/753520559 wait complete. 2024-04-23T18:27:44.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.503+0000 7fb2fb7e4740 20 client.16608 trim_cache size 0 max 16384 2024-04-23T18:27:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.520+0000 7fb2fb7e4740 1 Processor -- start 2024-04-23T18:27:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.520+0000 7fb2fb7e4740 1 -- start start 2024-04-23T18:27:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.520+0000 7fb2fb7e4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71c9fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2fb7e4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 0x55fd71cb3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2fb7e4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71ca7890 0x55fd71ca6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2fb7e4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f41ae3f0 con 0x55fd71c98c90 2024-04-23T18:27:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2fb7e4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f41abc00 con 0x55fd71ca7890 2024-04-23T18:27:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2fb7e4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f41a9a90 con 0x55fd71c98370 2024-04-23T18:27:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2f259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 0x55fd71cb3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2f259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 0x55fd71cb3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40586/0 (socket says 172.21.15.50:40586) 2024-04-23T18:27:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2f259c640 1 -- 172.21.15.50:0/2116881876 learned_addr learned my addr 172.21.15.50:0/2116881876 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/2116881876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71c9fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/2116881876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71ca7890 0x55fd71ca6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2f259c640 1 -- 172.21.15.50:0/2116881876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 msgr2=0x55fd71c9fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2f259c640 1 --2- 172.21.15.50:0/2116881876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71c9fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2f259c640 1 -- 172.21.15.50:0/2116881876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71ca7890 msgr2=0x55fd71ca6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2f259c640 1 --2- 172.21.15.50:0/2116881876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71ca7890 0x55fd71ca6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2f259c640 1 -- 172.21.15.50:0/2116881876 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd71c98ac0 con 0x55fd71c98c90 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2f259c640 1 --2- 172.21.15.50:0/2116881876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 0x55fd71cb3640 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb2e402a930 tx=0x7fb2e4005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562f361a92569229 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2dbfff640 1 -- 172.21.15.50:0/2116881876 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e404a070 con 0x55fd71c98c90 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2dbfff640 1 -- 172.21.15.50:0/2116881876 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2e4045dc0 con 0x55fd71c98c90 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.521+0000 7fb2dbfff640 1 -- 172.21.15.50:0/2116881876 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4035040 con 0x55fd71c98c90 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.522+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2116881876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 msgr2=0x55fd71cb3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.522+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/2116881876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 0x55fd71cb3640 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb2e402a930 tx=0x7fb2e4005c00 comp rx=0 tx=0).stop 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.522+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2116881876 shutdown_connections 2024-04-23T18:27:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.522+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/2116881876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71ca7890 0x55fd71ca6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.522+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/2116881876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 0x55fd71cb3640 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.522+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/2116881876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71c9fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.522+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2116881876 >> 172.21.15.50:0/2116881876 conn(0x55fd71c947b0 msgr2=0x55fd71c99c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.522+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2116881876 shutdown_connections 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.522+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2116881876 wait complete. 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.522+0000 7fb2fb7e4740 1 -- start start 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.522+0000 7fb2fb7e4740 1 Processor -- start 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2fb7e4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71cafd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2fb7e4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 0x55fd71ca1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2fb7e4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71ca7890 0x55fd71ca1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2fb7e4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f41aa0e0 con 0x55fd71c98c90 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2fb7e4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f41ab530 con 0x55fd71ca7890 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2fb7e4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f41aa790 con 0x55fd71c98370 2024-04-23T18:27:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2fb7e4740 10 client.0 _ll_register_callbacks cb 0x7fb2dc01d540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2f259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 0x55fd71ca1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2f259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 0x55fd71ca1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40600/0 (socket says 172.21.15.50:40600) 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2f259c640 1 -- 172.21.15.50:0/2974977541 learned_addr learned my addr 172.21.15.50:0/2974977541 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71ca7890 0x55fd71ca1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71cafd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2f259c640 1 -- 172.21.15.50:0/2974977541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 msgr2=0x55fd71cafd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2f259c640 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71cafd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2f259c640 1 -- 172.21.15.50:0/2974977541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71ca7890 msgr2=0x55fd71ca1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2f259c640 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71ca7890 0x55fd71ca1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2f259c640 1 -- 172.21.15.50:0/2974977541 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd71ca22c0 con 0x55fd71c98c90 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2f259c640 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 0x55fd71ca1810 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009540 tx=0x7fb2e4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be807c9a50e9e4c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2d9ffb640 1 -- 172.21.15.50:0/2974977541 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4053060 con 0x55fd71c98c90 2024-04-23T18:27:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2d9ffb640 1 -- 172.21.15.50:0/2974977541 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fb2e4005040 con 0x55fd71c98c90 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.523+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2974977541 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2e4045dc0 con 0x55fd71c98c90 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2d9ffb640 1 -- 172.21.15.50:0/2974977541 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb2e404a040 con 0x55fd71c98c90 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2fb7e4740 10 client.16614 fetch_fsmap learned FSMap version 40 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2974977541 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55fd71ca8e20 con 0x55fd71c98c90 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2d9ffb640 1 -- 172.21.15.50:0/2974977541 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2e4045dc0 con 0x55fd71c98c90 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2d9ffb640 1 -- 172.21.15.50:0/2974977541 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fb2e4045380 con 0x55fd71c98c90 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2fb7e4740 10 client.16614 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2fb7e4740 10 client.16614 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2fb7e4740 20 client.16614 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fd71d1e270 0x55fd71d20690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2fb7e4740 4 client.16614 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c31bc802", "release": true, "await": true}] 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2974977541 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c31bc802", "release": true, "await": true}) -- 0x55fd71c0b0f0 con 0x55fd71d1e270 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fd71d1e270 0x55fd71d20690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2ebfff640 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fd71d1e270 0x55fd71d20690 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45bb9f722ad4bb14 server_cookie=457b27121a98cf12 in_seq=0 out_seq=0 2024-04-23T18:27:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.524+0000 7fb2d9ffb640 10 client.16614 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2d9ffb640 1 -- 172.21.15.50:0/2974977541 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55fd71c0b0f0 con 0x55fd71d1e270 2024-04-23T18:27:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2d9ffb640 10 client.16614 handle_command_reply: tid=0 2024-04-23T18:27:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2fb7e4740 1 client.16614 shutdown 2024-04-23T18:27:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2974977541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 msgr2=0x55fd71ca1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 0x55fd71ca1810 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009540 tx=0x7fb2e4000bc0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2974977541 shutdown_connections 2024-04-23T18:27:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fd71d1e270 0x55fd71d20690 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd71ca7890 0x55fd71ca1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fd71c98c90 0x55fd71ca1810 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2d9ffb640 0 client.16614 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2fb7e4740 1 --2- 172.21.15.50:0/2974977541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fd71c98370 0x55fd71cafd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2974977541 >> 172.21.15.50:0/2974977541 conn(0x55fd71c947b0 msgr2=0x55fd71c9a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:44.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2974977541 shutdown_connections 2024-04-23T18:27:44.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2fb7e4740 1 -- 172.21.15.50:0/2974977541 wait complete. 2024-04-23T18:27:44.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.530+0000 7fb2fb7e4740 20 client.16614 trim_cache size 0 max 16384 2024-04-23T18:27:44.525 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:27:44.525 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 40, 2024-04-23T18:27:44.525 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:27:44.525 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 28, 2024-04-23T18:27:44.525 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:27:44.525 INFO:teuthology.orchestra.run.smithi050.stdout: "c31bc802": { 2024-04-23T18:27:44.525 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 28, 2024-04-23T18:27:44.525 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 82.0, 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:27:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:44.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:44.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:44.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:44.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:44.527 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:27:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.531+0000 7fb2f8acf640 1 -- 172.21.15.50:0/1353402877 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2e00a3d20 msgr2=0x7fb2e00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.531+0000 7fb2f8acf640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2e00a3d20 0x7fb2e00a61e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec09c1e0 tx=0x7fb2ec067030 comp rx=0 tx=0).stop 2024-04-23T18:27:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.531+0000 7fb2f8acf640 1 -- 172.21.15.50:0/1353402877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2f41985a0 msgr2=0x7fb2f4198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.531+0000 7fb2f8acf640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2f41985a0 0x7fb2f4198a50 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec009fd0 tx=0x7fb2ec00b260 comp rx=0 tx=0).stop 2024-04-23T18:27:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.531+0000 7fb2f8acf640 1 -- 172.21.15.50:0/1353402877 shutdown_connections 2024-04-23T18:27:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.531+0000 7fb2f8acf640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2e00a3d20 0x7fb2e00a61e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.531+0000 7fb2f8acf640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2f41985a0 0x7fb2f4198a50 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.531+0000 7fb2f8acf640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f4107260 0x7fb2f419e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.531+0000 7fb2f8acf640 1 --2- 172.21.15.50:0/1353402877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2f4102190 0x7fb2f419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.531+0000 7fb2f8acf640 1 -- 172.21.15.50:0/1353402877 >> 172.21.15.50:0/1353402877 conn(0x7fb2f40793a0 msgr2=0x7fb2f4100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.531+0000 7fb2f8acf640 1 -- 172.21.15.50:0/1353402877 shutdown_connections 2024-04-23T18:27:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.531+0000 7fb2f8acf640 1 -- 172.21.15.50:0/1353402877 wait complete. 2024-04-23T18:27:44.535 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c31bc802', seconds elapsed: 23.2 2024-04-23T18:27:44.535 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.05 2024-04-23T18:27:44.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:27:44.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:44 smithi177 ceph-mon[32308]: pgmap v288: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-04-23T18:27:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:44 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:27:44.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:44 smithi188 ceph-mon[32694]: pgmap v288: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-04-23T18:27:44.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:44 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:27:44.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.646+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/1094775659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2c810c520 msgr2=0x7fd2c810e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.646+0000 7fd2cd9ba640 1 --2- 172.21.15.50:0/1094775659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2c810c520 0x7fd2c810e9c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009ca0 tx=0x7fd2c002a9e0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.646+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/1094775659 shutdown_connections 2024-04-23T18:27:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.646+0000 7fd2cd9ba640 1 --2- 172.21.15.50:0/1094775659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2c810c520 0x7fd2c810e9c0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.646+0000 7fd2cd9ba640 1 --2- 172.21.15.50:0/1094775659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2c8103b00 0x7fd2c810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.646+0000 7fd2cd9ba640 1 --2- 172.21.15.50:0/1094775659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c81031e0 0x7fd2c81035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.646+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/1094775659 >> 172.21.15.50:0/1094775659 conn(0x7fd2c8077f50 msgr2=0x7fd2c8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.646+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/1094775659 shutdown_connections 2024-04-23T18:27:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.646+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/1094775659 wait complete. 2024-04-23T18:27:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2cd9ba640 1 Processor -- start 2024-04-23T18:27:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2cd9ba640 1 -- start start 2024-04-23T18:27:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2cd9ba640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c81031e0 0x7fd2c8100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2cd9ba640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2c8103b00 0x7fd2c8101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2cd9ba640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2c810c520 0x7fd2c81018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2cd9ba640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c81121b0 con 0x7fd2c8103b00 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2cd9ba640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c8112030 con 0x7fd2c810c520 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2cd9ba640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c8112330 con 0x7fd2c81031e0 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2be7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2c8103b00 0x7fd2c8101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2be7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2c8103b00 0x7fd2c8101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40612/0 (socket says 172.21.15.50:40612) 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2be7fc640 1 -- 172.21.15.50:0/83245460 learned_addr learned my addr 172.21.15.50:0/83245460 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2c6ffd640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c81031e0 0x7fd2c8100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.647+0000 7fd2c77fe640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2c810c520 0x7fd2c81018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.648+0000 7fd2be7fc640 1 -- 172.21.15.50:0/83245460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c81031e0 msgr2=0x7fd2c8100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.648+0000 7fd2be7fc640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c81031e0 0x7fd2c8100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.648+0000 7fd2be7fc640 1 -- 172.21.15.50:0/83245460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2c810c520 msgr2=0x7fd2c81018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.648+0000 7fd2be7fc640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2c810c520 0x7fd2c81018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.648+0000 7fd2be7fc640 1 -- 172.21.15.50:0/83245460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c81a6b90 con 0x7fd2c8103b00 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.648+0000 7fd2be7fc640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2c8103b00 0x7fd2c8101370 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fd2b000e9f0 tx=0x7fd2b000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:44.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.648+0000 7fd2cc9b8640 1 -- 172.21.15.50:0/83245460 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b000cd80 con 0x7fd2c8103b00 2024-04-23T18:27:44.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.648+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/83245460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c81a6e80 con 0x7fd2c8103b00 2024-04-23T18:27:44.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.648+0000 7fd2cc9b8640 1 -- 172.21.15.50:0/83245460 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2b00108e0 con 0x7fd2c8103b00 2024-04-23T18:27:44.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.648+0000 7fd2cc9b8640 1 -- 172.21.15.50:0/83245460 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0018980 con 0x7fd2c8103b00 2024-04-23T18:27:44.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.648+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/83245460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c81a7390 con 0x7fd2c8103b00 2024-04-23T18:27:44.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.649+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/83245460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c8068d70 con 0x7fd2c8103b00 2024-04-23T18:27:44.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.651+0000 7fd2cc9b8640 1 -- 172.21.15.50:0/83245460 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd2b0010430 con 0x7fd2c8103b00 2024-04-23T18:27:44.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.651+0000 7fd2cc9b8640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd2b40a3d50 0x7fd2b40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:44.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.652+0000 7fd2cc9b8640 1 -- 172.21.15.50:0/83245460 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fd2b00d0ed0 con 0x7fd2c8103b00 2024-04-23T18:27:44.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.652+0000 7fd2c6ffd640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd2b40a3d50 0x7fd2b40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:44.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.652+0000 7fd2c6ffd640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd2b40a3d50 0x7fd2b40a6210 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009bd0 tx=0x7fd2b8005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:44.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.655+0000 7fd2cc9b8640 1 -- 172.21.15.50:0/83245460 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd2b009ceb0 con 0x7fd2c8103b00 2024-04-23T18:27:44.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.891+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/83245460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2c802d5c0 con 0x7fd2c8103b00 2024-04-23T18:27:44.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.892+0000 7fd2cc9b8640 1 -- 172.21.15.50:0/83245460 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11881 (secure 0 0 0) 0x7fd2b00a11b0 con 0x7fd2c8103b00 2024-04-23T18:27:44.885 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:27:44.885 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:27:33.821377+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/4033440314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4033440314},{"type":"v1","addr":"172.21.15.177:6837","nonce":4033440314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:27:44.885 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 40 2024-04-23T18:27:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.894+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/83245460 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd2b40a3d50 msgr2=0x7fd2b40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.894+0000 7fd2cd9ba640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd2b40a3d50 0x7fd2b40a6210 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009bd0 tx=0x7fd2b8005f90 comp rx=0 tx=0).stop 2024-04-23T18:27:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.894+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/83245460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2c8103b00 msgr2=0x7fd2c8101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.894+0000 7fd2cd9ba640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2c8103b00 0x7fd2c8101370 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fd2b000e9f0 tx=0x7fd2b000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.894+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/83245460 shutdown_connections 2024-04-23T18:27:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.894+0000 7fd2cd9ba640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd2b40a3d50 0x7fd2b40a6210 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.894+0000 7fd2cd9ba640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2c810c520 0x7fd2c81018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.894+0000 7fd2cd9ba640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2c8103b00 0x7fd2c8101370 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.894+0000 7fd2cd9ba640 1 --2- 172.21.15.50:0/83245460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2c81031e0 0x7fd2c8100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:44.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.894+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/83245460 >> 172.21.15.50:0/83245460 conn(0x7fd2c8077f50 msgr2=0x7fd2c80fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:44.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.894+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/83245460 shutdown_connections 2024-04-23T18:27:44.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:44.894+0000 7fd2cd9ba640 1 -- 172.21.15.50:0/83245460 wait complete. 2024-04-23T18:27:44.906 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 82.05 --await-for 42.0' 2024-04-23T18:27:44.906 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 82.05 --await-for 42.0 2024-04-23T18:27:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.038+0000 7fbfda0cf640 1 -- 172.21.15.50:0/1875410579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd4103b50 msgr2=0x7fbfd410c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.038+0000 7fbfda0cf640 1 --2- 172.21.15.50:0/1875410579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd4103b50 0x7fbfd410c030 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009900 tx=0x7fbfc802a480 comp rx=0 tx=0).stop 2024-04-23T18:27:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.038+0000 7fbfda0cf640 1 -- 172.21.15.50:0/1875410579 shutdown_connections 2024-04-23T18:27:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.038+0000 7fbfda0cf640 1 --2- 172.21.15.50:0/1875410579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfd410c570 0x7fbfd410e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.038+0000 7fbfda0cf640 1 --2- 172.21.15.50:0/1875410579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd4103b50 0x7fbfd410c030 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.038+0000 7fbfda0cf640 1 --2- 172.21.15.50:0/1875410579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd4103230 0x7fbfd4103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.038+0000 7fbfda0cf640 1 -- 172.21.15.50:0/1875410579 >> 172.21.15.50:0/1875410579 conn(0x7fbfd4077fb0 msgr2=0x7fbfd40783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.038+0000 7fbfda0cf640 1 -- 172.21.15.50:0/1875410579 shutdown_connections 2024-04-23T18:27:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.038+0000 7fbfda0cf640 1 -- 172.21.15.50:0/1875410579 wait complete. 2024-04-23T18:27:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfda0cf640 1 Processor -- start 2024-04-23T18:27:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfda0cf640 1 -- start start 2024-04-23T18:27:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfda0cf640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfd4103230 0x7fbfd419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfda0cf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd4103b50 0x7fbfd419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfda0cf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd410c570 0x7fbfd4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfda0cf640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41121b0 con 0x7fbfd4103b50 2024-04-23T18:27:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfda0cf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd4112030 con 0x7fbfd410c570 2024-04-23T18:27:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfda0cf640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd4112330 con 0x7fbfd4103230 2024-04-23T18:27:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd4103b50 0x7fbfd419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd4103b50 0x7fbfd419e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40636/0 (socket says 172.21.15.50:40636) 2024-04-23T18:27:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfd2ffd640 1 -- 172.21.15.50:0/557565962 learned_addr learned my addr 172.21.15.50:0/557565962 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfd3fff640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd410c570 0x7fbfd4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfd37fe640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfd4103230 0x7fbfd419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfd2ffd640 1 -- 172.21.15.50:0/557565962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfd4103230 msgr2=0x7fbfd419e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfd2ffd640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfd4103230 0x7fbfd419e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfd2ffd640 1 -- 172.21.15.50:0/557565962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd410c570 msgr2=0x7fbfd4198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfd2ffd640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd410c570 0x7fbfd4198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfd2ffd640 1 -- 172.21.15.50:0/557565962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd4198d10 con 0x7fbfd4103b50 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.039+0000 7fbfd3fff640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd410c570 0x7fbfd4198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.040+0000 7fbfd2ffd640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd4103b50 0x7fbfd419e920 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8005940 tx=0x7fbfc80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.040+0000 7fbfd0ff9640 1 -- 172.21.15.50:0/557565962 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc803f030 con 0x7fbfd4103b50 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.040+0000 7fbfd0ff9640 1 -- 172.21.15.50:0/557565962 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfc80044c0 con 0x7fbfd4103b50 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.040+0000 7fbfda0cf640 1 -- 172.21.15.50:0/557565962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd4198fa0 con 0x7fbfd4103b50 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.040+0000 7fbfd0ff9640 1 -- 172.21.15.50:0/557565962 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8035070 con 0x7fbfd4103b50 2024-04-23T18:27:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.040+0000 7fbfda0cf640 1 -- 172.21.15.50:0/557565962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd41a52a0 con 0x7fbfd4103b50 2024-04-23T18:27:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.042+0000 7fbfd0ff9640 1 -- 172.21.15.50:0/557565962 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbfc8042030 con 0x7fbfd4103b50 2024-04-23T18:27:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.042+0000 7fbfdcde6740 1 Processor -- start 2024-04-23T18:27:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfdcde6740 1 -- start start 2024-04-23T18:27:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd0ff9640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbfbc0a3d20 0x7fbfbc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfdcde6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 0x562305144480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd0ff9640 1 -- 172.21.15.50:0/557565962 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fbfc80f0b60 con 0x7fbfd4103b50 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfdcde6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051449c0 0x56230514eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd3fff640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbfbc0a3d20 0x7fbfbc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfdcde6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56230514f090 0x562305151480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfdcde6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5623050a9370 con 0x56230514f090 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfdcde6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5623050462e0 con 0x5623051440a0 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfdcde6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5623050a5330 con 0x5623051449c0 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 0x562305144480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 0x562305144480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:56354/0 (socket says 172.21.15.50:56354) 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd37fe640 1 -- 172.21.15.50:0/2021406113 learned_addr learned my addr 172.21.15.50:0/2021406113 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd3fff640 1 --2- 172.21.15.50:0/2021406113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56230514f090 0x562305151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd2ffd640 1 --2- 172.21.15.50:0/2021406113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051449c0 0x56230514eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd37fe640 1 -- 172.21.15.50:0/2021406113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051449c0 msgr2=0x56230514eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd37fe640 1 --2- 172.21.15.50:0/2021406113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051449c0 0x56230514eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd37fe640 1 -- 172.21.15.50:0/2021406113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56230514f090 msgr2=0x562305151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd37fe640 1 --2- 172.21.15.50:0/2021406113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56230514f090 0x562305151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd37fe640 1 -- 172.21.15.50:0/2021406113 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562304e87d80 con 0x5623051440a0 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.043+0000 7fbfd3fff640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbfbc0a3d20 0x7fbfbc0a61e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0095f0 tx=0x7fbfcc009c40 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.044+0000 7fbfd37fe640 1 --2- 172.21.15.50:0/2021406113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 0x562305144480 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40114f0 tx=0x7fbfc40119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b97740b1a29694e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.044+0000 7fbfb9ffb640 1 -- 172.21.15.50:0/2021406113 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc401e070 con 0x5623051440a0 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.044+0000 7fbfb9ffb640 1 -- 172.21.15.50:0/2021406113 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfc4011310 con 0x5623051440a0 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.044+0000 7fbfb9ffb640 1 -- 172.21.15.50:0/2021406113 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc40198b0 con 0x5623051440a0 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.044+0000 7fbfdcde6740 1 -- 172.21.15.50:0/2021406113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 msgr2=0x562305144480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.044+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/2021406113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 0x562305144480 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40114f0 tx=0x7fbfc40119b0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.044+0000 7fbfdcde6740 1 -- 172.21.15.50:0/2021406113 shutdown_connections 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.044+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/2021406113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56230514f090 0x562305151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.044+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/2021406113 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051449c0 0x56230514eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.044+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/2021406113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 0x562305144480 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.044+0000 7fbfdcde6740 1 -- 172.21.15.50:0/2021406113 >> 172.21.15.50:0/2021406113 conn(0x5623051403c0 msgr2=0x5623051407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.044+0000 7fbfdcde6740 1 -- 172.21.15.50:0/2021406113 shutdown_connections 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfdcde6740 1 -- 172.21.15.50:0/2021406113 wait complete. 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfdcde6740 1 -- start start 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfdcde6740 1 Processor -- start 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfdcde6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 0x56230514bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfdcde6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051449c0 0x56230514c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfdcde6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56230514f090 0x56230514ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfdcde6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562305151e80 con 0x56230514f090 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfdcde6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562305043fb0 con 0x5623051440a0 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfdcde6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562305152000 con 0x5623051449c0 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfdcde6740 10 client.0 _ll_register_callbacks cb 0x5623050439a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 0x56230514bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfd3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56230514f090 0x56230514ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 0x56230514bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:56364/0 (socket says 172.21.15.50:56364) 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfd37fe640 1 -- 172.21.15.50:0/1550268951 learned_addr learned my addr 172.21.15.50:0/1550268951 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.045+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051449c0 0x56230514c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfd37fe640 1 -- 172.21.15.50:0/1550268951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051449c0 msgr2=0x56230514c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfd37fe640 1 --2- 172.21.15.50:0/1550268951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051449c0 0x56230514c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfd37fe640 1 -- 172.21.15.50:0/1550268951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56230514f090 msgr2=0x56230514ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfd37fe640 1 --2- 172.21.15.50:0/1550268951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56230514f090 0x56230514ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfd37fe640 1 -- 172.21.15.50:0/1550268951 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562304ee8510 con 0x5623051440a0 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfd3fff640 1 --2- 172.21.15.50:0/1550268951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56230514f090 0x56230514ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfd37fe640 1 --2- 172.21.15.50:0/1550268951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 0x56230514bfb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40114c0 tx=0x7fbfc4009d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7258eefc90028b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfb2ffd640 1 -- 172.21.15.50:0/1550268951 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc401e050 con 0x5623051440a0 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfdcde6740 1 -- 172.21.15.50:0/1550268951 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfc4011310 con 0x5623051440a0 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfb2ffd640 1 -- 172.21.15.50:0/1550268951 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fbfc4019a80 con 0x5623051440a0 2024-04-23T18:27:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfb2ffd640 1 -- 172.21.15.50:0/1550268951 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fbfc4033050 con 0x5623051440a0 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfdcde6740 10 client.26201 fetch_fsmap learned FSMap version 40 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfdcde6740 1 -- 172.21.15.50:0/1550268951 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x562304d7b670 con 0x5623051440a0 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.046+0000 7fbfb2ffd640 1 -- 172.21.15.50:0/1550268951 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfc4011310 con 0x5623051440a0 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.047+0000 7fbfb2ffd640 1 -- 172.21.15.50:0/1550268951 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fbfc4005040 con 0x5623051440a0 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.047+0000 7fbfdcde6740 10 client.26201 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.047+0000 7fbfdcde6740 10 client.26201 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.047+0000 7fbfdcde6740 20 client.26201 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.047+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/1550268951 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56230515f300 0x5623051616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.047+0000 7fbfdcde6740 4 client.26201 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.047+0000 7fbfdcde6740 1 -- 172.21.15.50:0/1550268951 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623050b6d90 con 0x56230515f300 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.047+0000 7fbfd2ffd640 1 --2- 172.21.15.50:0/1550268951 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56230515f300 0x5623051616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.047+0000 7fbfb2ffd640 10 client.26201 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.047+0000 7fbfd2ffd640 1 --2- 172.21.15.50:0/1550268951 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56230515f300 0x5623051616f0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2c24cda9981d1ce server_cookie=771a1284c5056d9 in_seq=0 out_seq=0 2024-04-23T18:27:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfb2ffd640 1 -- 172.21.15.50:0/1550268951 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623050b6d90 con 0x56230515f300 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfb2ffd640 10 client.26201 handle_command_reply: tid=0 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfdcde6740 1 client.26201 shutdown 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfdcde6740 1 -- 172.21.15.50:0/1550268951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 msgr2=0x56230514bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/1550268951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 0x56230514bfb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40114c0 tx=0x7fbfc4009d80 comp rx=0 tx=0).stop 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfdcde6740 1 -- 172.21.15.50:0/1550268951 shutdown_connections 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/1550268951 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56230515f300 0x5623051616f0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/1550268951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56230514f090 0x56230514ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfb2ffd640 0 client.26201 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/1550268951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051449c0 0x56230514c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/1550268951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051440a0 0x56230514bfb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfdcde6740 1 -- 172.21.15.50:0/1550268951 >> 172.21.15.50:0/1550268951 conn(0x5623051403c0 msgr2=0x5623051461c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfdcde6740 1 -- 172.21.15.50:0/1550268951 shutdown_connections 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfdcde6740 1 -- 172.21.15.50:0/1550268951 wait complete. 2024-04-23T18:27:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.049+0000 7fbfdcde6740 20 client.26201 trim_cache size 0 max 16384 2024-04-23T18:27:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.066+0000 7fbfdcde6740 1 Processor -- start 2024-04-23T18:27:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.066+0000 7fbfdcde6740 1 -- start start 2024-04-23T18:27:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfdcde6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051440a0 0x56230515ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfdcde6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051449c0 0x56230513fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfdcde6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56230514f090 0x562305166f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfdcde6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41b05b0 con 0x5623051449c0 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfdcde6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41addc0 con 0x56230514f090 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfdcde6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41abc30 con 0x5623051440a0 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051449c0 0x56230513fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051449c0 0x56230513fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40660/0 (socket says 172.21.15.50:40660) 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051440a0 0x56230515ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfd2ffd640 1 -- 172.21.15.50:0/157900415 learned_addr learned my addr 172.21.15.50:0/157900415 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfd3fff640 1 --2- 172.21.15.50:0/157900415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56230514f090 0x562305166f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfd2ffd640 1 -- 172.21.15.50:0/157900415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051440a0 msgr2=0x56230515ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfd2ffd640 1 --2- 172.21.15.50:0/157900415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051440a0 0x56230515ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfd2ffd640 1 -- 172.21.15.50:0/157900415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56230514f090 msgr2=0x562305166f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfd2ffd640 1 --2- 172.21.15.50:0/157900415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56230514f090 0x562305166f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfd2ffd640 1 -- 172.21.15.50:0/157900415 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562305161d20 con 0x5623051449c0 2024-04-23T18:27:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfd3fff640 1 --2- 172.21.15.50:0/157900415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56230514f090 0x562305166f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfd2ffd640 1 --2- 172.21.15.50:0/157900415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051449c0 0x56230513fd80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80ee300 tx=0x7fbfc802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9faa4fd64e1846d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfb97fa640 1 -- 172.21.15.50:0/157900415 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8049040 con 0x5623051449c0 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfb97fa640 1 -- 172.21.15.50:0/157900415 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfc80baea0 con 0x5623051449c0 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.067+0000 7fbfb97fa640 1 -- 172.21.15.50:0/157900415 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc80c52d0 con 0x5623051449c0 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.068+0000 7fbfdcde6740 1 -- 172.21.15.50:0/157900415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051449c0 msgr2=0x56230513fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.068+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/157900415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051449c0 0x56230513fd80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80ee300 tx=0x7fbfc802a480 comp rx=0 tx=0).stop 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.068+0000 7fbfdcde6740 1 -- 172.21.15.50:0/157900415 shutdown_connections 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.068+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/157900415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56230514f090 0x562305166f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.068+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/157900415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051449c0 0x56230513fd80 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.068+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/157900415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5623051440a0 0x56230515ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.068+0000 7fbfdcde6740 1 -- 172.21.15.50:0/157900415 >> 172.21.15.50:0/157900415 conn(0x5623051403c0 msgr2=0x56230514c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.068+0000 7fbfdcde6740 1 -- 172.21.15.50:0/157900415 shutdown_connections 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.068+0000 7fbfdcde6740 1 -- 172.21.15.50:0/157900415 wait complete. 2024-04-23T18:27:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.068+0000 7fbfdcde6740 1 -- start start 2024-04-23T18:27:45.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.068+0000 7fbfdcde6740 1 Processor -- start 2024-04-23T18:27:45.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfdcde6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051440a0 0x5623051577c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfdcde6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051449c0 0x562305152b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfdcde6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56230514f090 0x562305153270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfdcde6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41ac280 con 0x5623051440a0 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfdcde6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41ad6f0 con 0x5623051449c0 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfdcde6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd41ac950 con 0x56230514f090 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfdcde6740 10 client.0 _ll_register_callbacks cb 0x7fbfc8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051440a0 0x5623051577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051440a0 0x5623051577c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40664/0 (socket says 172.21.15.50:40664) 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfd37fe640 1 -- 172.21.15.50:0/3043354827 learned_addr learned my addr 172.21.15.50:0/3043354827 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfd2ffd640 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051449c0 0x562305152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfd3fff640 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56230514f090 0x562305153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfd37fe640 1 -- 172.21.15.50:0/3043354827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56230514f090 msgr2=0x562305153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfd37fe640 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56230514f090 0x562305153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfd37fe640 1 -- 172.21.15.50:0/3043354827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051449c0 msgr2=0x562305152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfd37fe640 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051449c0 0x562305152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfd37fe640 1 -- 172.21.15.50:0/3043354827 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562305146d20 con 0x5623051440a0 2024-04-23T18:27:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfd37fe640 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051440a0 0x5623051577c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4000f80 tx=0x7fbfc4019540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8689022cffb9fd8d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.069+0000 7fbfb2ffd640 1 -- 172.21.15.50:0/3043354827 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc403c060 con 0x5623051440a0 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfb2ffd640 1 -- 172.21.15.50:0/3043354827 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fbfc4005680 con 0x5623051440a0 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfdcde6740 1 -- 172.21.15.50:0/3043354827 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfc80baea0 con 0x5623051440a0 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfb2ffd640 1 -- 172.21.15.50:0/3043354827 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fbfc401e040 con 0x5623051440a0 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfdcde6740 10 client.16638 fetch_fsmap learned FSMap version 40 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfdcde6740 1 -- 172.21.15.50:0/3043354827 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5623051c5770 con 0x5623051440a0 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfb2ffd640 1 -- 172.21.15.50:0/3043354827 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfc80baea0 con 0x5623051440a0 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfb2ffd640 1 -- 172.21.15.50:0/3043354827 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fbfc4019880 con 0x5623051440a0 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfdcde6740 10 client.16638 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfdcde6740 10 client.16638 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfdcde6740 20 client.16638 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623051c5ed0 0x5623051c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfdcde6740 4 client.16638 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.05, "await_for": 42.0}] 2024-04-23T18:27:45.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfdcde6740 1 -- 172.21.15.50:0/3043354827 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.05, "await_for": 42.0}) -- 0x5623050b0c00 con 0x5623051c5ed0 2024-04-23T18:27:45.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.070+0000 7fbfd2ffd640 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623051c5ed0 0x5623051c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:45.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.071+0000 7fbfd2ffd640 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623051c5ed0 0x5623051c82f0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f1fd4a849071a61 server_cookie=112c38b417ccd34c in_seq=0 out_seq=0 2024-04-23T18:27:45.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.071+0000 7fbfb2ffd640 10 client.16638 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfb2ffd640 1 -- 172.21.15.50:0/3043354827 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5623050b0c00 con 0x5623051c5ed0 2024-04-23T18:27:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfb2ffd640 10 client.16638 handle_command_reply: tid=0 2024-04-23T18:27:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfdcde6740 1 client.16638 shutdown 2024-04-23T18:27:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfdcde6740 1 -- 172.21.15.50:0/3043354827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051440a0 msgr2=0x5623051577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051440a0 0x5623051577c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4000f80 tx=0x7fbfc4019540 comp rx=0 tx=0).stop 2024-04-23T18:27:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfdcde6740 1 -- 172.21.15.50:0/3043354827 shutdown_connections 2024-04-23T18:27:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5623051c5ed0 0x5623051c82f0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56230514f090 0x562305153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfb2ffd640 0 client.16638 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:45.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623051449c0 0x562305152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfdcde6740 1 --2- 172.21.15.50:0/3043354827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5623051440a0 0x5623051577c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfdcde6740 1 -- 172.21.15.50:0/3043354827 >> 172.21.15.50:0/3043354827 conn(0x5623051403c0 msgr2=0x56230514bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:45.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.155+0000 7fbfdcde6740 1 -- 172.21.15.50:0/3043354827 shutdown_connections 2024-04-23T18:27:45.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfdcde6740 1 -- 172.21.15.50:0/3043354827 wait complete. 2024-04-23T18:27:45.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfdcde6740 20 client.16638 trim_cache size 0 max 16384 2024-04-23T18:27:45.150 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:27:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 40, 2024-04-23T18:27:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:27:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 30, 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "46f1ed70": { 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 30, 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 82.0, 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:27:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:27:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:27:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:27:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:27:45.152 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:27:45.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfda0cf640 1 -- 172.21.15.50:0/557565962 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbfbc0a3d20 msgr2=0x7fbfbc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:45.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfda0cf640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbfbc0a3d20 0x7fbfbc0a61e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0095f0 tx=0x7fbfcc009c40 comp rx=0 tx=0).stop 2024-04-23T18:27:45.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfda0cf640 1 -- 172.21.15.50:0/557565962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd4103b50 msgr2=0x7fbfd419e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:45.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfda0cf640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd4103b50 0x7fbfd419e920 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8005940 tx=0x7fbfc80040c0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfda0cf640 1 -- 172.21.15.50:0/557565962 shutdown_connections 2024-04-23T18:27:45.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfda0cf640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbfbc0a3d20 0x7fbfbc0a61e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfda0cf640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd410c570 0x7fbfd4198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfda0cf640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd4103b50 0x7fbfd419e920 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfda0cf640 1 --2- 172.21.15.50:0/557565962 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfd4103230 0x7fbfd419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:45.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfda0cf640 1 -- 172.21.15.50:0/557565962 >> 172.21.15.50:0/557565962 conn(0x7fbfd4077fb0 msgr2=0x7fbfd40fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:45.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfda0cf640 1 -- 172.21.15.50:0/557565962 shutdown_connections 2024-04-23T18:27:45.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:27:45.156+0000 7fbfda0cf640 1 -- 172.21.15.50:0/557565962 wait complete. 2024-04-23T18:27:45.159 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:45 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:27:45.159 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:45 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:27:45.159 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:45 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:27:45.159 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:45 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3607753653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:45.159 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:45 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/83245460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:45.160 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '46f1ed70', quiesce took 0.6 seconds. Will release after: 22.05 2024-04-23T18:27:45.223 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.92 MB/sec execute 379 sec latency 22586.300 ms 2024-04-23T18:27:45.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:45 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:27:45.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:45 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:27:45.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:45 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:27:45.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:45 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3607753653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:45.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:45 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/83245460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:45.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:45 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:27:45.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:45 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:27:45.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:45 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:27:45.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:45 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3607753653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:45.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:45 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/83245460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:27:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.220+0000 7fb8a07ef640 1 -- 172.21.15.50:0/1290208359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898101aa0 msgr2=0x7fb898105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.220+0000 7fb8a07ef640 1 --2- 172.21.15.50:0/1290208359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898101aa0 0x7fb898105a90 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb888009900 tx=0x7fb88802a410 comp rx=0 tx=0).stop 2024-04-23T18:27:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 -- 172.21.15.50:0/1290208359 shutdown_connections 2024-04-23T18:27:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 --2- 172.21.15.50:0/1290208359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb898106170 0x7fb89810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 --2- 172.21.15.50:0/1290208359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898101aa0 0x7fb898105a90 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 --2- 172.21.15.50:0/1290208359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8981010f0 0x7fb8981014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 -- 172.21.15.50:0/1290208359 >> 172.21.15.50:0/1290208359 conn(0x7fb898077fd0 msgr2=0x7fb8980783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 -- 172.21.15.50:0/1290208359 shutdown_connections 2024-04-23T18:27:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 -- 172.21.15.50:0/1290208359 wait complete. 2024-04-23T18:27:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 Processor -- start 2024-04-23T18:27:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 -- start start 2024-04-23T18:27:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8981010f0 0x7fb898198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898106170 0x7fb898198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb89819cfa0 0x7fb89819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981121f0 con 0x7fb898106170 2024-04-23T18:27:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb898112070 con 0x7fb8981010f0 2024-04-23T18:27:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.221+0000 7fb8a07ef640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb898112370 con 0x7fb89819cfa0 2024-04-23T18:27:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb89dd8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898106170 0x7fb898198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb89dd8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898106170 0x7fb898198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40676/0 (socket says 172.21.15.50:40676) 2024-04-23T18:27:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb89dd8a640 1 -- 172.21.15.50:0/3547163968 learned_addr learned my addr 172.21.15.50:0/3547163968 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb89e58b640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8981010f0 0x7fb898198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb89819cfa0 0x7fb89819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb89dd8a640 1 -- 172.21.15.50:0/3547163968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb89819cfa0 msgr2=0x7fb89819d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb89dd8a640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb89819cfa0 0x7fb89819d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb89dd8a640 1 -- 172.21.15.50:0/3547163968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8981010f0 msgr2=0x7fb898198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb89dd8a640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8981010f0 0x7fb898198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb89dd8a640 1 -- 172.21.15.50:0/3547163968 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89819da90 con 0x7fb898106170 2024-04-23T18:27:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb89e58b640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8981010f0 0x7fb898198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:27:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb89dd8a640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898106170 0x7fb898198c70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb888006010 tx=0x7fb888004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb88f7fe640 1 -- 172.21.15.50:0/3547163968 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88803f030 con 0x7fb898106170 2024-04-23T18:27:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb88f7fe640 1 -- 172.21.15.50:0/3547163968 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb888003aa0 con 0x7fb898106170 2024-04-23T18:27:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb8a07ef640 1 -- 172.21.15.50:0/3547163968 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb89819dd20 con 0x7fb898106170 2024-04-23T18:27:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb88f7fe640 1 -- 172.21.15.50:0/3547163968 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888035070 con 0x7fb898106170 2024-04-23T18:27:46.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.222+0000 7fb8a07ef640 1 -- 172.21.15.50:0/3547163968 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8981a54e0 con 0x7fb898106170 2024-04-23T18:27:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.224+0000 7fb88f7fe640 1 -- 172.21.15.50:0/3547163968 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb888042030 con 0x7fb898106170 2024-04-23T18:27:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb8a35d7740 1 Processor -- start 2024-04-23T18:27:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb8a35d7740 1 -- start start 2024-04-23T18:27:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb8a35d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36b6620 0x5592f36b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb8a35d7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6f40 0x5592f36c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb8a35d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 0x5592f36c3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb8a35d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5592f361b970 con 0x5592f36c1800 2024-04-23T18:27:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb8a35d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5592f35b84a0 con 0x5592f36b6620 2024-04-23T18:27:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb8a35d7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5592f3617930 con 0x5592f36b6f40 2024-04-23T18:27:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb89dd8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 0x5592f36c3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb89dd8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 0x5592f36c3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40686/0 (socket says 172.21.15.50:40686) 2024-04-23T18:27:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb89dd8a640 1 -- 172.21.15.50:0/3599729255 learned_addr learned my addr 172.21.15.50:0/3599729255 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/3599729255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36b6620 0x5592f36b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.225+0000 7fb89e58b640 1 --2- 172.21.15.50:0/3599729255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6f40 0x5592f36c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb88f7fe640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb8840a3dc0 0x7fb8840a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb89ed8c640 1 -- 172.21.15.50:0/3599729255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6f40 msgr2=0x5592f36c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/3599729255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6f40 0x5592f36c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb89ed8c640 1 -- 172.21.15.50:0/3599729255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 msgr2=0x5592f36c3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/3599729255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 0x5592f36c3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb89ed8c640 1 -- 172.21.15.50:0/3599729255 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592f332e210 con 0x5592f36b6620 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb89dd8a640 1 --2- 172.21.15.50:0/3599729255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 0x5592f36c3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb8840a3dc0 0x7fb8840a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb88f7fe640 1 -- 172.21.15.50:0/3547163968 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fb8880f0d50 con 0x7fb898106170 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/3599729255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36b6620 0x5592f36b6a00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb89000bc60 tx=0x7fb89000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2db56f590ea1bd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb8840a3dc0 0x7fb8840a6280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb890020000 tx=0x7fb89001d000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb88cff9640 1 -- 172.21.15.50:0/3599729255 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb890026480 con 0x5592f36b6620 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb88cff9640 1 -- 172.21.15.50:0/3599729255 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb89001b8f0 con 0x5592f36b6620 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.226+0000 7fb88cff9640 1 -- 172.21.15.50:0/3599729255 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89001c840 con 0x5592f36b6620 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.227+0000 7fb8a35d7740 1 -- 172.21.15.50:0/3599729255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36b6620 msgr2=0x5592f36b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.227+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/3599729255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36b6620 0x5592f36b6a00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb89000bc60 tx=0x7fb89000f730 comp rx=0 tx=0).stop 2024-04-23T18:27:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.227+0000 7fb8a35d7740 1 -- 172.21.15.50:0/3599729255 shutdown_connections 2024-04-23T18:27:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.227+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/3599729255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 0x5592f36c3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.227+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/3599729255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6f40 0x5592f36c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.227+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/3599729255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36b6620 0x5592f36b6a00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.227+0000 7fb8a35d7740 1 -- 172.21.15.50:0/3599729255 >> 172.21.15.50:0/3599729255 conn(0x5592f36b2b30 msgr2=0x5592f36b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.227+0000 7fb8a35d7740 1 -- 172.21.15.50:0/3599729255 shutdown_connections 2024-04-23T18:27:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.227+0000 7fb8a35d7740 1 -- 172.21.15.50:0/3599729255 wait complete. 2024-04-23T18:27:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.227+0000 7fb8a35d7740 1 -- start start 2024-04-23T18:27:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb8a35d7740 1 Processor -- start 2024-04-23T18:27:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb8a35d7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6620 0x5592f36c11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb8a35d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 0x5592f36be4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb8a35d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36c1800 0x5592f36bea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb8a35d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5592f36c4430 con 0x5592f36b6f40 2024-04-23T18:27:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb8a35d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5592f35b60b0 con 0x5592f36c1800 2024-04-23T18:27:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb8a35d7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5592f36c45b0 con 0x5592f36b6620 2024-04-23T18:27:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb8a35d7740 10 client.0 _ll_register_callbacks cb 0x5592f35a3d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb89dd8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 0x5592f36be4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb89dd8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 0x5592f36be4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40698/0 (socket says 172.21.15.50:40698) 2024-04-23T18:27:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb89dd8a640 1 -- 172.21.15.50:0/2636043445 learned_addr learned my addr 172.21.15.50:0/2636043445 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36c1800 0x5592f36bea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb89e58b640 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6620 0x5592f36c11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb89dd8a640 1 -- 172.21.15.50:0/2636043445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6620 msgr2=0x5592f36c11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb89dd8a640 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6620 0x5592f36c11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb89dd8a640 1 -- 172.21.15.50:0/2636043445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36c1800 msgr2=0x5592f36bea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb89dd8a640 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36c1800 0x5592f36bea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb89dd8a640 1 -- 172.21.15.50:0/2636043445 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592f33fa620 con 0x5592f36b6f40 2024-04-23T18:27:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb89dd8a640 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 0x5592f36be4a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb888004650 tx=0x7fb888003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fed4887e6b65ac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb881ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.228+0000 7fb881ffb640 1 -- 172.21.15.50:0/2636043445 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888049050 con 0x5592f36b6f40 2024-04-23T18:27:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb881ffb640 1 -- 172.21.15.50:0/2636043445 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fb8880ee6b0 con 0x5592f36b6f40 2024-04-23T18:27:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb8a35d7740 1 -- 172.21.15.50:0/2636043445 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb89001b8f0 con 0x5592f36b6f40 2024-04-23T18:27:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb881ffb640 1 -- 172.21.15.50:0/2636043445 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb8880ee8b0 con 0x5592f36b6f40 2024-04-23T18:27:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb8a35d7740 10 client.16650 fetch_fsmap learned FSMap version 40 2024-04-23T18:27:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb8a35d7740 1 -- 172.21.15.50:0/2636043445 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5592f36d03a0 con 0x5592f36b6f40 2024-04-23T18:27:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb881ffb640 1 -- 172.21.15.50:0/2636043445 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb89001b8f0 con 0x5592f36b6f40 2024-04-23T18:27:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb881ffb640 1 -- 172.21.15.50:0/2636043445 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fb8880dd0b0 con 0x5592f36b6f40 2024-04-23T18:27:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb8a35d7740 10 client.16650 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:27:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb8a35d7740 10 client.16650 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb8a35d7740 20 client.16650 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5592f36d1800 0x5592f36d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb8a35d7740 4 client.16650 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb8a35d7740 1 -- 172.21.15.50:0/2636043445 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592f3629390 con 0x5592f36d1800 2024-04-23T18:27:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.229+0000 7fb89e58b640 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5592f36d1800 0x5592f36d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.230+0000 7fb89e58b640 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5592f36d1800 0x5592f36d3c60 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=926634cc9e924a6 server_cookie=8a96a5ed26769735 in_seq=0 out_seq=0 2024-04-23T18:27:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.230+0000 7fb881ffb640 10 client.16650 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:46.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.90 MB/sec execute 380 sec latency 1142.073 ms 2024-04-23T18:27:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.230+0000 7fb881ffb640 1 -- 172.21.15.50:0/2636043445 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5592f3629390 con 0x5592f36d1800 2024-04-23T18:27:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.230+0000 7fb881ffb640 10 client.16650 handle_command_reply: tid=0 2024-04-23T18:27:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb8a35d7740 1 client.16650 shutdown 2024-04-23T18:27:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb8a35d7740 1 -- 172.21.15.50:0/2636043445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 msgr2=0x5592f36be4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 0x5592f36be4a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb888004650 tx=0x7fb888003f50 comp rx=0 tx=0).stop 2024-04-23T18:27:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb8a35d7740 1 -- 172.21.15.50:0/2636043445 shutdown_connections 2024-04-23T18:27:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5592f36d1800 0x5592f36d3c60 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36c1800 0x5592f36bea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 0x5592f36be4a0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb881ffb640 0 client.16650 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/2636043445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6620 0x5592f36c11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb8a35d7740 1 -- 172.21.15.50:0/2636043445 >> 172.21.15.50:0/2636043445 conn(0x5592f36b2b30 msgr2=0x5592f36b8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb8a35d7740 1 -- 172.21.15.50:0/2636043445 shutdown_connections 2024-04-23T18:27:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb8a35d7740 1 -- 172.21.15.50:0/2636043445 wait complete. 2024-04-23T18:27:46.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.231+0000 7fb8a35d7740 20 client.16650 trim_cache size 0 max 16384 2024-04-23T18:27:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.246+0000 7fb8a35d7740 1 Processor -- start 2024-04-23T18:27:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.246+0000 7fb8a35d7740 1 -- start start 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.246+0000 7fb8a35d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36b6620 0x5592f36cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.246+0000 7fb8a35d7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6f40 0x5592f36b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.246+0000 7fb8a35d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 0x5592f36c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.246+0000 7fb8a35d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981b0530 con 0x5592f36c1800 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.246+0000 7fb8a35d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981add40 con 0x5592f36b6620 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.246+0000 7fb8a35d7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981abbb0 con 0x5592f36b6f40 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.246+0000 7fb89ed8c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 0x5592f36c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.246+0000 7fb89ed8c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 0x5592f36c0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40712/0 (socket says 172.21.15.50:40712) 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.246+0000 7fb89ed8c640 1 -- 172.21.15.50:0/574820827 learned_addr learned my addr 172.21.15.50:0/574820827 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.247+0000 7fb89e58b640 1 --2- 172.21.15.50:0/574820827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36b6620 0x5592f36cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.247+0000 7fb89dd8a640 1 --2- 172.21.15.50:0/574820827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6f40 0x5592f36b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.247+0000 7fb89ed8c640 1 -- 172.21.15.50:0/574820827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6f40 msgr2=0x5592f36b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.247+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/574820827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6f40 0x5592f36b1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.247+0000 7fb89ed8c640 1 -- 172.21.15.50:0/574820827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36b6620 msgr2=0x5592f36cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.247+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/574820827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36b6620 0x5592f36cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.247+0000 7fb89ed8c640 1 -- 172.21.15.50:0/574820827 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592f36bc9f0 con 0x5592f36c1800 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.247+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/574820827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 0x5592f36c0ce0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb89000bc60 tx=0x7fb890012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f84d3d8d4ce0c2e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.247+0000 7fb883fff640 1 -- 172.21.15.50:0/574820827 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb890036070 con 0x5592f36c1800 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.247+0000 7fb883fff640 1 -- 172.21.15.50:0/574820827 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb89000b490 con 0x5592f36c1800 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.247+0000 7fb883fff640 1 -- 172.21.15.50:0/574820827 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8900266e0 con 0x5592f36c1800 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 -- 172.21.15.50:0/574820827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 msgr2=0x5592f36c0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/574820827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 0x5592f36c0ce0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb89000bc60 tx=0x7fb890012f90 comp rx=0 tx=0).stop 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 -- 172.21.15.50:0/574820827 shutdown_connections 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/574820827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36c1800 0x5592f36c0ce0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/574820827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6f40 0x5592f36b1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/574820827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36b6620 0x5592f36cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 -- 172.21.15.50:0/574820827 >> 172.21.15.50:0/574820827 conn(0x5592f36b2b30 msgr2=0x5592f36b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 -- 172.21.15.50:0/574820827 shutdown_connections 2024-04-23T18:27:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 -- 172.21.15.50:0/574820827 wait complete. 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 -- start start 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 Processor -- start 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6620 0x5592f36bb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 0x5592f36bb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36c1800 0x5592f36bbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981ac200 con 0x5592f36b6f40 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981ad670 con 0x5592f36c1800 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb8981ac8d0 con 0x5592f36b6620 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb8a35d7740 10 client.0 _ll_register_callbacks cb 0x7fb888004550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb89dd8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 0x5592f36bb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb89dd8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 0x5592f36bb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40728/0 (socket says 172.21.15.50:40728) 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb89dd8a640 1 -- 172.21.15.50:0/1020057342 learned_addr learned my addr 172.21.15.50:0/1020057342 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.248+0000 7fb89e58b640 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6620 0x5592f36bb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36c1800 0x5592f36bbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb89dd8a640 1 -- 172.21.15.50:0/1020057342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6620 msgr2=0x5592f36bb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb89dd8a640 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6620 0x5592f36bb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb89dd8a640 1 -- 172.21.15.50:0/1020057342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36c1800 msgr2=0x5592f36bbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb89dd8a640 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36c1800 0x5592f36bbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb89dd8a640 1 -- 172.21.15.50:0/1020057342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592f3736f40 con 0x5592f36b6f40 2024-04-23T18:27:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb89ed8c640 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36c1800 0x5592f36bbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb89dd8a640 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 0x5592f36bb7b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb8880b9650 tx=0x7fb8880ee8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36685ed8a2b4ef30 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb881ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb881ffb640 1 -- 172.21.15.50:0/1020057342 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880f42f0 con 0x5592f36b6f40 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb881ffb640 1 -- 172.21.15.50:0/1020057342 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fb8880c5460 con 0x5592f36b6f40 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb8a35d7740 1 -- 172.21.15.50:0/1020057342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb89000b490 con 0x5592f36b6f40 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb881ffb640 1 -- 172.21.15.50:0/1020057342 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb88803b460 con 0x5592f36b6f40 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb8a35d7740 10 client.16656 fetch_fsmap learned FSMap version 40 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb8a35d7740 1 -- 172.21.15.50:0/1020057342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5592f37381d0 con 0x5592f36b6f40 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb881ffb640 1 -- 172.21.15.50:0/1020057342 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb89000b490 con 0x5592f36b6f40 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb881ffb640 1 -- 172.21.15.50:0/1020057342 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fb8880bcee0 con 0x5592f36b6f40 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb8a35d7740 10 client.16656 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb8a35d7740 10 client.16656 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.249+0000 7fb8a35d7740 20 client.16656 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.250+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5592f36d1800 0x5592f373a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.250+0000 7fb8a35d7740 4 client.16656 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.250+0000 7fb8a35d7740 1 -- 172.21.15.50:0/1020057342 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5592f3623200 con 0x5592f36d1800 2024-04-23T18:27:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.250+0000 7fb89e58b640 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5592f36d1800 0x5592f373a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.250+0000 7fb89e58b640 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5592f36d1800 0x5592f373a7d0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=585c094e235f3d6d server_cookie=e8d399d114e42a5f in_seq=0 out_seq=0 2024-04-23T18:27:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.250+0000 7fb881ffb640 10 client.16656 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:46.276 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb881ffb640 1 -- 172.21.15.50:0/1020057342 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5592f3623200 con 0x5592f36d1800 2024-04-23T18:27:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb881ffb640 10 client.16656 handle_command_reply: tid=0 2024-04-23T18:27:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb8a35d7740 1 client.16656 shutdown 2024-04-23T18:27:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb8a35d7740 1 -- 172.21.15.50:0/1020057342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 msgr2=0x5592f36bb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 0x5592f36bb7b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb8880b9650 tx=0x7fb8880ee8e0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb8a35d7740 1 -- 172.21.15.50:0/1020057342 shutdown_connections 2024-04-23T18:27:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5592f36d1800 0x5592f373a7d0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592f36c1800 0x5592f36bbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5592f36b6f40 0x5592f36bb7b0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb8a35d7740 1 --2- 172.21.15.50:0/1020057342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5592f36b6620 0x5592f36bb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb881ffb640 0 client.16656 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb8a35d7740 1 -- 172.21.15.50:0/1020057342 >> 172.21.15.50:0/1020057342 conn(0x5592f36b2b30 msgr2=0x5592f36b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb8a35d7740 1 -- 172.21.15.50:0/1020057342 shutdown_connections 2024-04-23T18:27:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb8a35d7740 1 -- 172.21.15.50:0/1020057342 wait complete. 2024-04-23T18:27:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.283+0000 7fb8a35d7740 20 client.16656 trim_cache size 0 max 16384 2024-04-23T18:27:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:27:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:27:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:27:46.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:27:46.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.284+0000 7fb8a07ef640 1 -- 172.21.15.50:0/3547163968 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb8840a3dc0 msgr2=0x7fb8840a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:46.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.284+0000 7fb8a07ef640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb8840a3dc0 0x7fb8840a6280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb890020000 tx=0x7fb89001d000 comp rx=0 tx=0).stop 2024-04-23T18:27:46.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.284+0000 7fb8a07ef640 1 -- 172.21.15.50:0/3547163968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898106170 msgr2=0x7fb898198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:46.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.284+0000 7fb8a07ef640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898106170 0x7fb898198c70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb888006010 tx=0x7fb888004060 comp rx=0 tx=0).stop 2024-04-23T18:27:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.284+0000 7fb8a07ef640 1 -- 172.21.15.50:0/3547163968 shutdown_connections 2024-04-23T18:27:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.284+0000 7fb8a07ef640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb8840a3dc0 0x7fb8840a6280 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.284+0000 7fb8a07ef640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb89819cfa0 0x7fb89819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.284+0000 7fb8a07ef640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898106170 0x7fb898198c70 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.284+0000 7fb8a07ef640 1 --2- 172.21.15.50:0/3547163968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8981010f0 0x7fb898198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.284+0000 7fb8a07ef640 1 -- 172.21.15.50:0/3547163968 >> 172.21.15.50:0/3547163968 conn(0x7fb898077fd0 msgr2=0x7fb8980fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.284+0000 7fb8a07ef640 1 -- 172.21.15.50:0/3547163968 shutdown_connections 2024-04-23T18:27:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:46.284+0000 7fb8a07ef640 1 -- 172.21.15.50:0/3547163968 wait complete. 2024-04-23T18:27:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:46 smithi050 ceph-mon[29187]: pgmap v289: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 1.2 MiB/s wr, 61 op/s 2024-04-23T18:27:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:46 smithi177 ceph-mon[32308]: pgmap v289: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 1.2 MiB/s wr, 61 op/s 2024-04-23T18:27:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:46 smithi188 ceph-mon[32694]: pgmap v289: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 1.2 MiB/s wr, 61 op/s 2024-04-23T18:27:47.223 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.89 MB/sec execute 381 sec latency 2142.179 ms 2024-04-23T18:27:48.223 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.87 MB/sec execute 382 sec latency 3142.326 ms 2024-04-23T18:27:48.381 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:48 smithi188 ceph-mon[32694]: pgmap v290: 97 pgs: 97 active+clean; 196 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-04-23T18:27:48.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:48 smithi050 ceph-mon[29187]: pgmap v290: 97 pgs: 97 active+clean; 196 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-04-23T18:27:48.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:48 smithi177 ceph-mon[32308]: pgmap v290: 97 pgs: 97 active+clean; 196 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-04-23T18:27:49.223 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.86 MB/sec execute 383 sec latency 4142.430 ms 2024-04-23T18:27:50.223 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.84 MB/sec execute 384 sec latency 5142.540 ms 2024-04-23T18:27:50.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:50 smithi050 ceph-mon[29187]: pgmap v291: 97 pgs: 97 active+clean; 196 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-04-23T18:27:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:50 smithi177 ceph-mon[32308]: pgmap v291: 97 pgs: 97 active+clean; 196 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-04-23T18:27:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:50 smithi188 ceph-mon[32694]: pgmap v291: 97 pgs: 97 active+clean; 196 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-04-23T18:27:51.223 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.83 MB/sec execute 385 sec latency 6142.611 ms 2024-04-23T18:27:52.225 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.81 MB/sec execute 386 sec latency 7142.677 ms 2024-04-23T18:27:52.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:52 smithi050 ceph-mon[29187]: pgmap v292: 97 pgs: 97 active+clean; 196 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-04-23T18:27:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:52 smithi177 ceph-mon[32308]: pgmap v292: 97 pgs: 97 active+clean; 196 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-04-23T18:27:52.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:52 smithi188 ceph-mon[32694]: pgmap v292: 97 pgs: 97 active+clean; 196 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-04-23T18:27:53.223 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.80 MB/sec execute 387 sec latency 8142.788 ms 2024-04-23T18:27:54.224 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.78 MB/sec execute 388 sec latency 9142.851 ms 2024-04-23T18:27:54.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:54 smithi050 ceph-mon[29187]: pgmap v293: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 2.1 MiB/s wr, 103 op/s 2024-04-23T18:27:54.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:54 smithi177 ceph-mon[32308]: pgmap v293: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 2.1 MiB/s wr, 103 op/s 2024-04-23T18:27:54.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:54 smithi188 ceph-mon[32694]: pgmap v293: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 2.1 MiB/s wr, 103 op/s 2024-04-23T18:27:55.224 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.77 MB/sec execute 389 sec latency 10142.931 ms 2024-04-23T18:27:56.224 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.75 MB/sec execute 390 sec latency 11142.988 ms 2024-04-23T18:27:56.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:56 smithi050 ceph-mon[29187]: pgmap v294: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 1.4 MiB/s wr, 71 op/s 2024-04-23T18:27:56.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:56 smithi177 ceph-mon[32308]: pgmap v294: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 1.4 MiB/s wr, 71 op/s 2024-04-23T18:27:56.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:56 smithi188 ceph-mon[32694]: pgmap v294: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 1.4 MiB/s wr, 71 op/s 2024-04-23T18:27:57.224 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.74 MB/sec execute 391 sec latency 12143.094 ms 2024-04-23T18:27:58.224 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.72 MB/sec execute 392 sec latency 13143.155 ms 2024-04-23T18:27:58.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:27:58 smithi050 ceph-mon[29187]: pgmap v295: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-04-23T18:27:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:27:58 smithi177 ceph-mon[32308]: pgmap v295: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-04-23T18:27:58.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:27:58 smithi188 ceph-mon[32694]: pgmap v295: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-04-23T18:27:59.224 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.71 MB/sec execute 393 sec latency 14143.269 ms 2024-04-23T18:27:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.383+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/206320963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a48102b70 msgr2=0x7f4a48106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.383+0000 7f4a4c9fe640 1 --2- 172.21.15.50:0/206320963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a48102b70 0x7f4a48106bb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009900 tx=0x7f4a3c02a410 comp rx=0 tx=0).stop 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.383+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/206320963 shutdown_connections 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.383+0000 7f4a4c9fe640 1 --2- 172.21.15.50:0/206320963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a48107290 0x7f4a4810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.383+0000 7f4a4c9fe640 1 --2- 172.21.15.50:0/206320963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a48102b70 0x7f4a48106bb0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.383+0000 7f4a4c9fe640 1 --2- 172.21.15.50:0/206320963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a481021c0 0x7f4a481025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.383+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/206320963 >> 172.21.15.50:0/206320963 conn(0x7f4a480fda20 msgr2=0x7f4a480ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.383+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/206320963 shutdown_connections 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.383+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/206320963 wait complete. 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.384+0000 7f4a4c9fe640 1 Processor -- start 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.384+0000 7f4a4c9fe640 1 -- start start 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.384+0000 7f4a4c9fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a481021c0 0x7f4a481986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.384+0000 7f4a4c9fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a48107290 0x7f4a48198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.384+0000 7f4a4c9fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4819cf60 0x7f4a4819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.384+0000 7f4a4c9fe640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4a48112200 con 0x7f4a4819cf60 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.384+0000 7f4a4c9fe640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4a48112080 con 0x7f4a481021c0 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.384+0000 7f4a4c9fe640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4a48112380 con 0x7f4a48107290 2024-04-23T18:27:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a46d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4819cf60 0x7f4a4819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a46d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4819cf60 0x7f4a4819d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54690/0 (socket says 172.21.15.50:54690) 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a46d9d640 1 -- 172.21.15.50:0/1933455431 learned_addr learned my addr 172.21.15.50:0/1933455431 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a4659c640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a481021c0 0x7f4a481986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a48107290 0x7f4a48198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a46d9d640 1 -- 172.21.15.50:0/1933455431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a48107290 msgr2=0x7f4a48198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a46d9d640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a48107290 0x7f4a48198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a46d9d640 1 -- 172.21.15.50:0/1933455431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a481021c0 msgr2=0x7f4a481986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a46d9d640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a481021c0 0x7f4a481986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a46d9d640 1 -- 172.21.15.50:0/1933455431 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4819da50 con 0x7f4a4819cf60 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a46d9d640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4819cf60 0x7f4a4819d410 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4a4000eb10 tx=0x7f4a4000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a377fe640 1 -- 172.21.15.50:0/1933455431 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a40004b20 con 0x7f4a4819cf60 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/1933455431 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a4819dd40 con 0x7f4a4819cf60 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a377fe640 1 -- 172.21.15.50:0/1933455431 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a40004cc0 con 0x7f4a4819cf60 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a377fe640 1 -- 172.21.15.50:0/1933455431 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a40010670 con 0x7f4a4819cf60 2024-04-23T18:27:59.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.385+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/1933455431 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a481a5500 con 0x7f4a4819cf60 2024-04-23T18:27:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.387+0000 7f4a377fe640 1 -- 172.21.15.50:0/1933455431 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4a40005020 con 0x7f4a4819cf60 2024-04-23T18:27:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.388+0000 7f4a4f7eb740 1 Processor -- start 2024-04-23T18:27:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.388+0000 7f4a4f7eb740 1 -- start start 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.388+0000 7f4a4f7eb740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42620 0x56417cd42a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.388+0000 7f4a4f7eb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42f40 0x56417cd4d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.388+0000 7f4a4f7eb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 0x56417cd4fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.388+0000 7f4a4f7eb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56417cca7970 con 0x56417cd4d800 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.388+0000 7f4a4f7eb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56417cc444a0 con 0x56417cd42f40 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.388+0000 7f4a4f7eb740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56417cca3930 con 0x56417cd42620 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.388+0000 7f4a377fe640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f4a300a3dc0 0x7f4a300a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a45d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 0x56417cd4fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a45d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 0x56417cd4fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54706/0 (socket says 172.21.15.50:54706) 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a45d9b640 1 -- 172.21.15.50:0/3807841477 learned_addr learned my addr 172.21.15.50:0/3807841477 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/3807841477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42620 0x56417cd42a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a46d9d640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f4a300a3dc0 0x7f4a300a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a4659c640 1 --2- 172.21.15.50:0/3807841477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42f40 0x56417cd4d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a45d9b640 1 -- 172.21.15.50:0/3807841477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42620 msgr2=0x56417cd42a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/3807841477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42620 0x56417cd42a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a45d9b640 1 -- 172.21.15.50:0/3807841477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42f40 msgr2=0x56417cd4d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/3807841477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42f40 0x56417cd4d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a377fe640 1 -- 172.21.15.50:0/1933455431 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f4a400d0ee0 con 0x7f4a4819cf60 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a45d9b640 1 -- 172.21.15.50:0/3807841477 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56417c9ba210 con 0x56417cd4d800 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/3807841477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 0x56417cd4fbf0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0057c0 tx=0x7f4a3c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef394f11fa0ad8b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a46d9d640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f4a300a3dc0 0x7f4a300a6280 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4a400bbfd0 tx=0x7f4a4006e850 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a34ff9640 1 -- 172.21.15.50:0/3807841477 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c035040 con 0x56417cd4d800 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a34ff9640 1 -- 172.21.15.50:0/3807841477 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a3c003d50 con 0x56417cd4d800 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.389+0000 7f4a34ff9640 1 -- 172.21.15.50:0/3807841477 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c03db90 con 0x56417cd4d800 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.390+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3807841477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 msgr2=0x56417cd4fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.390+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/3807841477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 0x56417cd4fbf0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0057c0 tx=0x7f4a3c000bc0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.390+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3807841477 shutdown_connections 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.390+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/3807841477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 0x56417cd4fbf0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.390+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/3807841477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42f40 0x56417cd4d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.390+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/3807841477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42620 0x56417cd42a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.390+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3807841477 >> 172.21.15.50:0/3807841477 conn(0x56417cd3eb30 msgr2=0x56417cd3ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.390+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3807841477 shutdown_connections 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.390+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3807841477 wait complete. 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4f7eb740 1 -- start start 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4f7eb740 1 Processor -- start 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4f7eb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42620 0x56417cd4d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4f7eb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42f40 0x56417cd4a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4f7eb740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd4d800 0x56417cd4ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4f7eb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56417cd503d0 con 0x56417cd42f40 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4f7eb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56417cc420b0 con 0x56417cd42620 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4f7eb740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56417cd50550 con 0x56417cd4d800 2024-04-23T18:27:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4f7eb740 10 client.0 _ll_register_callbacks cb 0x56417cc2fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42f40 0x56417cd4a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42f40 0x56417cd4a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54710/0 (socket says 172.21.15.50:54710) 2024-04-23T18:27:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4659c640 1 -- 172.21.15.50:0/1684781808 learned_addr learned my addr 172.21.15.50:0/1684781808 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42620 0x56417cd4d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd4d800 0x56417cd4ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4659c640 1 -- 172.21.15.50:0/1684781808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd4d800 msgr2=0x56417cd4ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4659c640 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd4d800 0x56417cd4ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4659c640 1 -- 172.21.15.50:0/1684781808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42620 msgr2=0x56417cd4d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4659c640 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42620 0x56417cd4d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.391+0000 7f4a4659c640 1 -- 172.21.15.50:0/1684781808 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56417ca86620 con 0x56417cd42f40 2024-04-23T18:27:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.392+0000 7f4a4659c640 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42f40 0x56417cd4a4f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800bee0 tx=0x7f4a38012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfdc11ca32bae817 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.392+0000 7f4a2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.392+0000 7f4a2dffb640 1 -- 172.21.15.50:0/1684781808 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a38004230 con 0x56417cd42f40 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.392+0000 7f4a2dffb640 1 -- 172.21.15.50:0/1684781808 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f4a380043d0 con 0x56417cd42f40 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.392+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/1684781808 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a3c003d50 con 0x56417cd42f40 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.392+0000 7f4a2dffb640 1 -- 172.21.15.50:0/1684781808 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f4a380106a0 con 0x56417cd42f40 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.392+0000 7f4a4f7eb740 10 client.16668 fetch_fsmap learned FSMap version 40 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.392+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/1684781808 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56417cd5c3e0 con 0x56417cd42f40 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.392+0000 7f4a2dffb640 1 -- 172.21.15.50:0/1684781808 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a3c003d50 con 0x56417cd42f40 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.392+0000 7f4a2dffb640 1 -- 172.21.15.50:0/1684781808 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7f4a38003b80 con 0x56417cd42f40 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.392+0000 7f4a4f7eb740 10 client.16668 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.392+0000 7f4a4f7eb740 10 client.16668 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.393+0000 7f4a4f7eb740 20 client.16668 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.393+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56417cd5d8f0 0x56417cd5fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.393+0000 7f4a4f7eb740 4 client.16668 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:27:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.393+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/1684781808 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56417ccb5390 con 0x56417cd5d8f0 2024-04-23T18:27:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.393+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56417cd5d8f0 0x56417cd5fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.393+0000 7f4a2dffb640 10 client.16668 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.393+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56417cd5d8f0 0x56417cd5fd20 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=530e2639bfa8168f server_cookie=a685ccee9a8e0071 in_seq=0 out_seq=0 2024-04-23T18:27:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a2dffb640 1 -- 172.21.15.50:0/1684781808 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56417ccb5390 con 0x56417cd5d8f0 2024-04-23T18:27:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a2dffb640 10 client.16668 handle_command_reply: tid=0 2024-04-23T18:27:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a4f7eb740 1 client.16668 shutdown 2024-04-23T18:27:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/1684781808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42f40 msgr2=0x56417cd4a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42f40 0x56417cd4a4f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800bee0 tx=0x7f4a38012820 comp rx=0 tx=0).stop 2024-04-23T18:27:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/1684781808 shutdown_connections 2024-04-23T18:27:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56417cd5d8f0 0x56417cd5fd20 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd4d800 0x56417cd4ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a2dffb640 0 client.16668 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42f40 0x56417cd4a4f0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/1684781808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42620 0x56417cd4d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/1684781808 >> 172.21.15.50:0/1684781808 conn(0x56417cd3eb30 msgr2=0x56417cd3dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/1684781808 shutdown_connections 2024-04-23T18:27:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/1684781808 wait complete. 2024-04-23T18:27:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.394+0000 7f4a4f7eb740 20 client.16668 trim_cache size 0 max 16384 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.409+0000 7f4a4f7eb740 1 Processor -- start 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.409+0000 7f4a4f7eb740 1 -- start start 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a4f7eb740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42620 0x56417cd59290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a4f7eb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42f40 0x56417cd3db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a4f7eb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 0x56417cd4cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a4f7eb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4a481b04d0 con 0x56417cd4d800 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a4f7eb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4a481adce0 con 0x56417cd42f40 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a4f7eb740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4a481abb50 con 0x56417cd42620 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a45d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 0x56417cd4cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a45d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 0x56417cd4cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54716/0 (socket says 172.21.15.50:54716) 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a45d9b640 1 -- 172.21.15.50:0/2280032171 learned_addr learned my addr 172.21.15.50:0/2280032171 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/2280032171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42620 0x56417cd59290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a4659c640 1 --2- 172.21.15.50:0/2280032171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42f40 0x56417cd3db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a45d9b640 1 -- 172.21.15.50:0/2280032171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42620 msgr2=0x56417cd59290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/2280032171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42620 0x56417cd59290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a45d9b640 1 -- 172.21.15.50:0/2280032171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42f40 msgr2=0x56417cd3db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/2280032171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42f40 0x56417cd3db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a45d9b640 1 -- 172.21.15.50:0/2280032171 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56417cd39090 con 0x56417cd4d800 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a4659c640 1 --2- 172.21.15.50:0/2280032171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42f40 0x56417cd3db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/2280032171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 0x56417cd4cd30 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0057c0 tx=0x7f4a3c040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b702b01146ff9cc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a2ffff640 1 -- 172.21.15.50:0/2280032171 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c053040 con 0x56417cd4d800 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.410+0000 7f4a2ffff640 1 -- 172.21.15.50:0/2280032171 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a3c0562d0 con 0x56417cd4d800 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.411+0000 7f4a2ffff640 1 -- 172.21.15.50:0/2280032171 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c035040 con 0x56417cd4d800 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.411+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/2280032171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 msgr2=0x56417cd4cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.411+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/2280032171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 0x56417cd4cd30 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0057c0 tx=0x7f4a3c040040 comp rx=0 tx=0).stop 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.411+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/2280032171 shutdown_connections 2024-04-23T18:27:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.411+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/2280032171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd4d800 0x56417cd4cd30 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.411+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/2280032171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd42f40 0x56417cd3db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.411+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/2280032171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42620 0x56417cd59290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.411+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/2280032171 >> 172.21.15.50:0/2280032171 conn(0x56417cd3eb30 msgr2=0x56417cd3d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.411+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/2280032171 shutdown_connections 2024-04-23T18:27:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.411+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/2280032171 wait complete. 2024-04-23T18:27:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.411+0000 7f4a4f7eb740 1 -- start start 2024-04-23T18:27:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.411+0000 7f4a4f7eb740 1 Processor -- start 2024-04-23T18:27:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a4f7eb740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42620 0x56417cd51490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a4f7eb740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42f40 0x56417cd519d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a4f7eb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd4d800 0x56417cd51f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a4f7eb740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4a481ac1a0 con 0x56417cd42620 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a4f7eb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4a481ad610 con 0x56417cd4d800 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a4f7eb740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4a481ac870 con 0x56417cd42f40 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a4f7eb740 10 client.0 _ll_register_callbacks cb 0x7f4a3800baf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a45d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42620 0x56417cd51490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a45d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42620 0x56417cd51490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54730/0 (socket says 172.21.15.50:54730) 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a45d9b640 1 -- 172.21.15.50:0/3971591182 learned_addr learned my addr 172.21.15.50:0/3971591182 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a4659c640 1 --2- 172.21.15.50:0/3971591182 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42f40 0x56417cd519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a45d9b640 1 -- 172.21.15.50:0/3971591182 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42f40 msgr2=0x56417cd519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/3971591182 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42f40 0x56417cd519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a45d9b640 1 -- 172.21.15.50:0/3971591182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd4d800 msgr2=0x56417cd51f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/3971591182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd4d800 0x56417cd51f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a45d9b640 1 -- 172.21.15.50:0/3971591182 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56417cdc3100 con 0x56417cd42620 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a45d9b640 1 --2- 172.21.15.50:0/3971591182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42620 0x56417cd51490 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c02ab60 tx=0x7f4a3c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16228c0a13d9fff server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:27:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a2dffb640 1 -- 172.21.15.50:0/3971591182 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c05c070 con 0x56417cd42620 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a2dffb640 1 -- 172.21.15.50:0/3971591182 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f4a3c0568e0 con 0x56417cd42620 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.412+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3971591182 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a3c0562d0 con 0x56417cd42620 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.413+0000 7f4a2dffb640 1 -- 172.21.15.50:0/3971591182 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f4a3c068050 con 0x56417cd42620 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.413+0000 7f4a4f7eb740 10 client.16674 fetch_fsmap learned FSMap version 40 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.413+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3971591182 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56417cdc4360 con 0x56417cd42620 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.413+0000 7f4a2dffb640 1 -- 172.21.15.50:0/3971591182 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a3c0562d0 con 0x56417cd42620 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.413+0000 7f4a2dffb640 1 -- 172.21.15.50:0/3971591182 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7f4a3c03a040 con 0x56417cd42620 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.413+0000 7f4a4f7eb740 10 client.16674 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.413+0000 7f4a4f7eb740 10 client.16674 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.413+0000 7f4a4f7eb740 20 client.16674 populate_metadata read hostname 'smithi050' 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.413+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/3971591182 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56417cd5d8f0 0x56417cdc6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.413+0000 7f4a4f7eb740 4 client.16674 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.413+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3971591182 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56417ccaf200 con 0x56417cd5d8f0 2024-04-23T18:27:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.413+0000 7f4a4659c640 1 --2- 172.21.15.50:0/3971591182 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56417cd5d8f0 0x56417cdc6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:27:59.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.414+0000 7f4a2dffb640 10 client.16674 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:59.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.414+0000 7f4a4659c640 1 --2- 172.21.15.50:0/3971591182 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56417cd5d8f0 0x56417cdc6540 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe40be442017e43d server_cookie=bc4a7569079d79ae in_seq=0 out_seq=0 2024-04-23T18:27:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a2dffb640 1 -- 172.21.15.50:0/3971591182 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56417ccaf200 con 0x56417cd5d8f0 2024-04-23T18:27:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a2dffb640 10 client.16674 handle_command_reply: tid=0 2024-04-23T18:27:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4f7eb740 1 client.16674 shutdown 2024-04-23T18:27:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3971591182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42620 msgr2=0x56417cd51490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/3971591182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42620 0x56417cd51490 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c02ab60 tx=0x7f4a3c000bc0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3971591182 shutdown_connections 2024-04-23T18:27:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/3971591182 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56417cd5d8f0 0x56417cdc6540 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/3971591182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417cd4d800 0x56417cd51f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/3971591182 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56417cd42f40 0x56417cd519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4f7eb740 1 --2- 172.21.15.50:0/3971591182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56417cd42620 0x56417cd51490 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a2dffb640 0 client.16674 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:27:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3971591182 >> 172.21.15.50:0/3971591182 conn(0x56417cd3eb30 msgr2=0x56417cd3dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3971591182 shutdown_connections 2024-04-23T18:27:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4f7eb740 1 -- 172.21.15.50:0/3971591182 wait complete. 2024-04-23T18:27:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4f7eb740 20 client.16674 trim_cache size 0 max 16384 2024-04-23T18:27:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:27:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:27:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:27:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:27:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/1933455431 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f4a300a3dc0 msgr2=0x7f4a300a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4c9fe640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f4a300a3dc0 0x7f4a300a6280 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4a400bbfd0 tx=0x7f4a4006e850 comp rx=0 tx=0).stop 2024-04-23T18:27:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/1933455431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4819cf60 msgr2=0x7f4a4819d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:27:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4c9fe640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4819cf60 0x7f4a4819d410 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4a4000eb10 tx=0x7f4a4000c2d0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/1933455431 shutdown_connections 2024-04-23T18:27:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4c9fe640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f4a300a3dc0 0x7f4a300a6280 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4c9fe640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a4819cf60 0x7f4a4819d410 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4c9fe640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4a48107290 0x7f4a48198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4c9fe640 1 --2- 172.21.15.50:0/1933455431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a481021c0 0x7f4a481986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:27:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/1933455431 >> 172.21.15.50:0/1933455431 conn(0x7f4a480fda20 msgr2=0x7f4a480ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:27:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/1933455431 shutdown_connections 2024-04-23T18:27:59.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:27:59.439+0000 7f4a4c9fe640 1 -- 172.21.15.50:0/1933455431 wait complete. 2024-04-23T18:28:00.224 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.70 MB/sec execute 394 sec latency 15143.378 ms 2024-04-23T18:28:00.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:00 smithi050 ceph-mon[29187]: pgmap v296: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 795 KiB/s wr, 37 op/s 2024-04-23T18:28:00.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:00 smithi177 ceph-mon[32308]: pgmap v296: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 795 KiB/s wr, 37 op/s 2024-04-23T18:28:00.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:00 smithi188 ceph-mon[32694]: pgmap v296: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 795 KiB/s wr, 37 op/s 2024-04-23T18:28:01.224 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.68 MB/sec execute 395 sec latency 16143.467 ms 2024-04-23T18:28:02.224 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.67 MB/sec execute 396 sec latency 17143.570 ms 2024-04-23T18:28:02.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:02 smithi050 ceph-mon[29187]: pgmap v297: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 795 KiB/s wr, 37 op/s 2024-04-23T18:28:02.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:02 smithi177 ceph-mon[32308]: pgmap v297: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 795 KiB/s wr, 37 op/s 2024-04-23T18:28:02.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:02 smithi188 ceph-mon[32694]: pgmap v297: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 795 KiB/s wr, 37 op/s 2024-04-23T18:28:03.224 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.65 MB/sec execute 397 sec latency 18143.669 ms 2024-04-23T18:28:04.224 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.64 MB/sec execute 398 sec latency 19143.807 ms 2024-04-23T18:28:04.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:04 smithi050 ceph-mon[29187]: pgmap v298: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 872 KiB/s wr, 66 op/s 2024-04-23T18:28:04.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:04 smithi177 ceph-mon[32308]: pgmap v298: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 872 KiB/s wr, 66 op/s 2024-04-23T18:28:04.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:04 smithi188 ceph-mon[32694]: pgmap v298: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 872 KiB/s wr, 66 op/s 2024-04-23T18:28:05.225 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.62 MB/sec execute 399 sec latency 20143.896 ms 2024-04-23T18:28:06.225 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.61 MB/sec execute 400 sec latency 21143.994 ms 2024-04-23T18:28:06.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:06 smithi050 ceph-mon[29187]: pgmap v299: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 289 KiB/s wr, 41 op/s 2024-04-23T18:28:06.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:06 smithi177 ceph-mon[32308]: pgmap v299: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 289 KiB/s wr, 41 op/s 2024-04-23T18:28:06.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:06 smithi188 ceph-mon[32694]: pgmap v299: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 289 KiB/s wr, 41 op/s 2024-04-23T18:28:07.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:28:07.224 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607436 5.60 MB/sec execute 401 sec latency 22144.044 ms 2024-04-23T18:28:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.334+0000 7fee209bb640 1 -- 172.21.15.50:0/3998862413 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee1c106190 msgr2=0x7fee1c10dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.334+0000 7fee209bb640 1 --2- 172.21.15.50:0/3998862413 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee1c106190 0x7fee1c10dd60 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fee14009ca0 tx=0x7fee1402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.334+0000 7fee209bb640 1 -- 172.21.15.50:0/3998862413 shutdown_connections 2024-04-23T18:28:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.334+0000 7fee209bb640 1 --2- 172.21.15.50:0/3998862413 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee1c106190 0x7fee1c10dd60 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.334+0000 7fee209bb640 1 --2- 172.21.15.50:0/3998862413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee1c101ac0 0x7fee1c105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.334+0000 7fee209bb640 1 --2- 172.21.15.50:0/3998862413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee1c101110 0x7fee1c1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.334+0000 7fee209bb640 1 -- 172.21.15.50:0/3998862413 >> 172.21.15.50:0/3998862413 conn(0x7fee1c077fb0 msgr2=0x7fee1c0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.334+0000 7fee209bb640 1 -- 172.21.15.50:0/3998862413 shutdown_connections 2024-04-23T18:28:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.334+0000 7fee209bb640 1 -- 172.21.15.50:0/3998862413 wait complete. 2024-04-23T18:28:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee209bb640 1 Processor -- start 2024-04-23T18:28:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee209bb640 1 -- start start 2024-04-23T18:28:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee209bb640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee1c101110 0x7fee1c197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee209bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee1c106190 0x7fee1c1978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee209bb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee1c197e10 0x7fee1c1a6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee209bb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fee1c111100 con 0x7fee1c106190 2024-04-23T18:28:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee209bb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee1c110f80 con 0x7fee1c197e10 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee209bb640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fee1c111280 con 0x7fee1c101110 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee19d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee1c106190 0x7fee1c1978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee19d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee1c106190 0x7fee1c1978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53332/0 (socket says 172.21.15.50:53332) 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee19d9b640 1 -- 172.21.15.50:0/3027948769 learned_addr learned my addr 172.21.15.50:0/3027948769 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee1a59c640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee1c101110 0x7fee1c197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee1ad9d640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee1c197e10 0x7fee1c1a6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee19d9b640 1 -- 172.21.15.50:0/3027948769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee1c101110 msgr2=0x7fee1c197390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee19d9b640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee1c101110 0x7fee1c197390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee19d9b640 1 -- 172.21.15.50:0/3027948769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee1c197e10 msgr2=0x7fee1c1a6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.335+0000 7fee19d9b640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee1c197e10 0x7fee1c1a6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.336+0000 7fee19d9b640 1 -- 172.21.15.50:0/3027948769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee1c1a6fa0 con 0x7fee1c106190 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.336+0000 7fee19d9b640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee1c106190 0x7fee1c1978d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fee0400d740 tx=0x7fee0400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.336+0000 7fee137fe640 1 -- 172.21.15.50:0/3027948769 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04017070 con 0x7fee1c106190 2024-04-23T18:28:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.336+0000 7fee137fe640 1 -- 172.21.15.50:0/3027948769 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee0400de50 con 0x7fee1c106190 2024-04-23T18:28:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.336+0000 7fee209bb640 1 -- 172.21.15.50:0/3027948769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee1c1a7290 con 0x7fee1c106190 2024-04-23T18:28:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.336+0000 7fee137fe640 1 -- 172.21.15.50:0/3027948769 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04004ce0 con 0x7fee1c106190 2024-04-23T18:28:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.336+0000 7fee209bb640 1 -- 172.21.15.50:0/3027948769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee1c1a7760 con 0x7fee1c106190 2024-04-23T18:28:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.337+0000 7fee209bb640 1 -- 172.21.15.50:0/3027948769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee1c0691c0 con 0x7fee1c106190 2024-04-23T18:28:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.338+0000 7fee137fe640 1 -- 172.21.15.50:0/3027948769 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fee04004e80 con 0x7fee1c106190 2024-04-23T18:28:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.339+0000 7fee137fe640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee080a3d20 0x7fee080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.339+0000 7fee1a59c640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee080a3d20 0x7fee080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.339+0000 7fee137fe640 1 -- 172.21.15.50:0/3027948769 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fee040d00f0 con 0x7fee1c106190 2024-04-23T18:28:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.340+0000 7fee1a59c640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee080a3d20 0x7fee080a61e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00bbe0 tx=0x7fee0c00b390 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:07.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.344+0000 7fee137fe640 1 -- 172.21.15.50:0/3027948769 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee0409c0d0 con 0x7fee1c106190 2024-04-23T18:28:07.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.571+0000 7fee209bb640 1 -- 172.21.15.50:0/3027948769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee1c101ac0 con 0x7fee1c106190 2024-04-23T18:28:07.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.572+0000 7fee137fe640 1 -- 172.21.15.50:0/3027948769 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11881 (secure 0 0 0) 0x7fee040a03d0 con 0x7fee1c106190 2024-04-23T18:28:07.565 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:28:07.565 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:27:33.821377+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/4033440314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4033440314},{"type":"v1","addr":"172.21.15.177:6837","nonce":4033440314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:28:07.565 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 40 2024-04-23T18:28:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.574+0000 7fee209bb640 1 -- 172.21.15.50:0/3027948769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee080a3d20 msgr2=0x7fee080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.574+0000 7fee209bb640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee080a3d20 0x7fee080a61e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00bbe0 tx=0x7fee0c00b390 comp rx=0 tx=0).stop 2024-04-23T18:28:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.574+0000 7fee209bb640 1 -- 172.21.15.50:0/3027948769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee1c106190 msgr2=0x7fee1c1978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.574+0000 7fee209bb640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee1c106190 0x7fee1c1978d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fee0400d740 tx=0x7fee0400dc00 comp rx=0 tx=0).stop 2024-04-23T18:28:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.574+0000 7fee209bb640 1 -- 172.21.15.50:0/3027948769 shutdown_connections 2024-04-23T18:28:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.574+0000 7fee209bb640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee080a3d20 0x7fee080a61e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.574+0000 7fee209bb640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee1c197e10 0x7fee1c1a6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.574+0000 7fee209bb640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee1c106190 0x7fee1c1978d0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.574+0000 7fee209bb640 1 --2- 172.21.15.50:0/3027948769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee1c101110 0x7fee1c197390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.574+0000 7fee209bb640 1 -- 172.21.15.50:0/3027948769 >> 172.21.15.50:0/3027948769 conn(0x7fee1c077fb0 msgr2=0x7fee1c0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:07.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.574+0000 7fee209bb640 1 -- 172.21.15.50:0/3027948769 shutdown_connections 2024-04-23T18:28:07.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.574+0000 7fee209bb640 1 -- 172.21.15.50:0/3027948769 wait complete. 2024-04-23T18:28:07.578 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id 46f1ed70 --release --await' 2024-04-23T18:28:07.578 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id 46f1ed70 --release --await 2024-04-23T18:28:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.733+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/1466541200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf8102b40 msgr2=0x7f1bf8106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.733+0000 7f1bfcbf9640 1 --2- 172.21.15.50:0/1466541200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf8102b40 0x7f1bf8106b80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f1be0009900 tx=0x7f1be002a410 comp rx=0 tx=0).stop 2024-04-23T18:28:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.733+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/1466541200 shutdown_connections 2024-04-23T18:28:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.733+0000 7f1bfcbf9640 1 --2- 172.21.15.50:0/1466541200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bf8107260 0x7f1bf810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.733+0000 7f1bfcbf9640 1 --2- 172.21.15.50:0/1466541200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf8102b40 0x7f1bf8106b80 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.733+0000 7f1bfcbf9640 1 --2- 172.21.15.50:0/1466541200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bf8102190 0x7f1bf8102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.733+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/1466541200 >> 172.21.15.50:0/1466541200 conn(0x7f1bf80793a0 msgr2=0x7f1bf80797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.733+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/1466541200 shutdown_connections 2024-04-23T18:28:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.733+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/1466541200 wait complete. 2024-04-23T18:28:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.734+0000 7f1bfcbf9640 1 Processor -- start 2024-04-23T18:28:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.734+0000 7f1bfcbf9640 1 -- start start 2024-04-23T18:28:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.734+0000 7f1bfcbf9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf8102190 0x7f1bf8198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.734+0000 7f1bfcbf9640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bf8107260 0x7f1bf8198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.734+0000 7f1bfcbf9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bf819d020 0x7f1bf819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.734+0000 7f1bfcbf9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81121a0 con 0x7f1bf8102190 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.734+0000 7f1bfcbf9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf8112020 con 0x7f1bf819d020 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.734+0000 7f1bfcbf9640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf8112320 con 0x7f1bf8107260 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.734+0000 7f1bf659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf8102190 0x7f1bf8198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.734+0000 7f1bf659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf8102190 0x7f1bf8198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53356/0 (socket says 172.21.15.50:53356) 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.734+0000 7f1bf659c640 1 -- 172.21.15.50:0/2968332312 learned_addr learned my addr 172.21.15.50:0/2968332312 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1beffff640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bf8107260 0x7f1bf8198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bf6d9d640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bf819d020 0x7f1bf819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bf659c640 1 -- 172.21.15.50:0/2968332312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bf8107260 msgr2=0x7f1bf8198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bf659c640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bf8107260 0x7f1bf8198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bf659c640 1 -- 172.21.15.50:0/2968332312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bf819d020 msgr2=0x7f1bf819d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bf659c640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bf819d020 0x7f1bf819d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bf659c640 1 -- 172.21.15.50:0/2968332312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf819db10 con 0x7f1bf8102190 2024-04-23T18:28:07.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bf6d9d640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bf819d020 0x7f1bf819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:28:07.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bf659c640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf8102190 0x7f1bf8198760 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f1be800d950 tx=0x7f1be800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:07.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bef7fe640 1 -- 172.21.15.50:0/2968332312 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8017070 con 0x7f1bf8102190 2024-04-23T18:28:07.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bef7fe640 1 -- 172.21.15.50:0/2968332312 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1be8009c30 con 0x7f1bf8102190 2024-04-23T18:28:07.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/2968332312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf819de00 con 0x7f1bf8102190 2024-04-23T18:28:07.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bef7fe640 1 -- 172.21.15.50:0/2968332312 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be800f620 con 0x7f1bf8102190 2024-04-23T18:28:07.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.735+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/2968332312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf81a55c0 con 0x7f1bf8102190 2024-04-23T18:28:07.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.737+0000 7f1bef7fe640 1 -- 172.21.15.50:0/2968332312 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1be8010460 con 0x7f1bf8102190 2024-04-23T18:28:07.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bff9e0740 1 Processor -- start 2024-04-23T18:28:07.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bff9e0740 1 -- start start 2024-04-23T18:28:07.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bff9e0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005390 0x563da3005770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bff9e0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 0x563da3014350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bff9e0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3016c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bff9e0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563da2f6a6d0 con 0x563da3005cb0 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bff9e0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563da2f071a0 con 0x563da3005390 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bff9e0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563da2f66690 con 0x563da3014890 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bf659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 0x563da3014350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bf659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 0x563da3014350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53372/0 (socket says 172.21.15.50:53372) 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bf659c640 1 -- 172.21.15.50:0/3430283647 learned_addr learned my addr 172.21.15.50:0/3430283647 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bf6d9d640 1 --2- 172.21.15.50:0/3430283647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005390 0x563da3005770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1beffff640 1 --2- 172.21.15.50:0/3430283647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3016c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.738+0000 7f1bef7fe640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1be40a3d20 0x7f1be40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1bf6d9d640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1be40a3d20 0x7f1be40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1bef7fe640 1 -- 172.21.15.50:0/2968332312 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f1be80d04b0 con 0x7f1bf8102190 2024-04-23T18:28:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1bf659c640 1 -- 172.21.15.50:0/3430283647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 msgr2=0x563da3016c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1bf659c640 1 --2- 172.21.15.50:0/3430283647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3016c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1bf659c640 1 -- 172.21.15.50:0/3430283647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005390 msgr2=0x563da3005770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1bf659c640 1 --2- 172.21.15.50:0/3430283647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005390 0x563da3005770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1bf659c640 1 -- 172.21.15.50:0/3430283647 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563da2f115b0 con 0x563da3005cb0 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1beffff640 1 --2- 172.21.15.50:0/3430283647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3016c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1bf659c640 1 --2- 172.21.15.50:0/3430283647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 0x563da3014350 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f1be80cdb00 tx=0x7f1be8010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d740d40e67fce4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1bf6d9d640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1be40a3d20 0x7f1be40a61e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1bf00092f0 tx=0x7f1bf0004ab0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1becff9640 1 -- 172.21.15.50:0/3430283647 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8020050 con 0x563da3005cb0 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1becff9640 1 -- 172.21.15.50:0/3430283647 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1be809cae0 con 0x563da3005cb0 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.739+0000 7f1becff9640 1 -- 172.21.15.50:0/3430283647 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be80d3af0 con 0x563da3005cb0 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.740+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3430283647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 msgr2=0x563da3014350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.740+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/3430283647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 0x563da3014350 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f1be80cdb00 tx=0x7f1be8010cb0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.740+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3430283647 shutdown_connections 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.740+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/3430283647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3016c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.740+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/3430283647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 0x563da3014350 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.740+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/3430283647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005390 0x563da3005770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.740+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3430283647 >> 172.21.15.50:0/3430283647 conn(0x563da30017b0 msgr2=0x563da3001b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.740+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3430283647 shutdown_connections 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.740+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3430283647 wait complete. 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.740+0000 7f1bff9e0740 1 -- start start 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.740+0000 7f1bff9e0740 1 Processor -- start 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bff9e0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 0x563da3014310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bff9e0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005cb0 0x563da3011600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bff9e0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3011d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bff9e0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563da3017630 con 0x563da3005390 2024-04-23T18:28:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bff9e0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563da2f04b20 con 0x563da3005cb0 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bff9e0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563da30177b0 con 0x563da3014890 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bff9e0740 10 client.0 _ll_register_callbacks cb 0x563da2f04530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bf659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 0x563da3014310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bf659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 0x563da3014310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53380/0 (socket says 172.21.15.50:53380) 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bf659c640 1 -- 172.21.15.50:0/2672596579 learned_addr learned my addr 172.21.15.50:0/2672596579 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bf6d9d640 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3011d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1beffff640 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005cb0 0x563da3011600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bf659c640 1 -- 172.21.15.50:0/2672596579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 msgr2=0x563da3011d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bf659c640 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3011d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bf659c640 1 -- 172.21.15.50:0/2672596579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005cb0 msgr2=0x563da3011600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bf659c640 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005cb0 0x563da3011600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bf659c640 1 -- 172.21.15.50:0/2672596579 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563da2c7c920 con 0x563da3005390 2024-04-23T18:28:07.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1beffff640 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005cb0 0x563da3011600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bf659c640 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 0x563da3014310 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1be8098db0 tx=0x7f1be80ce040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eccc6db90f52560 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bddffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bddffb640 1 -- 172.21.15.50:0/2672596579 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8098bf0 con 0x563da3005390 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bddffb640 1 -- 172.21.15.50:0/2672596579 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f1be809ba40 con 0x563da3005390 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.741+0000 7f1bff9e0740 1 -- 172.21.15.50:0/2672596579 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1be809cae0 con 0x563da3005390 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.742+0000 7f1bddffb640 1 -- 172.21.15.50:0/2672596579 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1be809c3e0 con 0x563da3005390 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.742+0000 7f1bff9e0740 10 client.16692 fetch_fsmap learned FSMap version 40 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.742+0000 7f1bff9e0740 1 -- 172.21.15.50:0/2672596579 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x563da2d48d30 con 0x563da3005390 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.742+0000 7f1bddffb640 1 -- 172.21.15.50:0/2672596579 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1be809cae0 con 0x563da3005390 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.742+0000 7f1bddffb640 1 -- 172.21.15.50:0/2672596579 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7f1be809b2a0 con 0x563da3005390 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.742+0000 7f1bff9e0740 10 client.16692 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.742+0000 7f1bff9e0740 10 client.16692 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.742+0000 7f1bff9e0740 20 client.16692 populate_metadata read hostname 'smithi050' 2024-04-23T18:28:07.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.742+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da3024880 0x563da3026c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.742+0000 7f1bff9e0740 4 client.16692 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.742+0000 7f1bff9e0740 1 -- 172.21.15.50:0/2672596579 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563da2f28500 con 0x563da3024880 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.742+0000 7f1beffff640 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da3024880 0x563da3026c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.743+0000 7f1bddffb640 10 client.16692 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.743+0000 7f1beffff640 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da3024880 0x563da3026c70 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e663f4a7bf1bb75 server_cookie=c133ce28db7ba7df in_seq=0 out_seq=0 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.743+0000 7f1bddffb640 1 -- 172.21.15.50:0/2672596579 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563da2f28500 con 0x563da3024880 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.743+0000 7f1bddffb640 10 client.16692 handle_command_reply: tid=0 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.743+0000 7f1bff9e0740 1 client.16692 shutdown 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.743+0000 7f1bff9e0740 1 -- 172.21.15.50:0/2672596579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 msgr2=0x563da3014310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.743+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 0x563da3014310 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1be8098db0 tx=0x7f1be80ce040 comp rx=0 tx=0).stop 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.744+0000 7f1bff9e0740 1 -- 172.21.15.50:0/2672596579 shutdown_connections 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.744+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da3024880 0x563da3026c70 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.744+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3011d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.744+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005cb0 0x563da3011600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.744+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/2672596579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 0x563da3014310 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.744+0000 7f1bddffb640 0 client.16692 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:07.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.744+0000 7f1bff9e0740 1 -- 172.21.15.50:0/2672596579 >> 172.21.15.50:0/2672596579 conn(0x563da30017b0 msgr2=0x563da30075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:07.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.744+0000 7f1bff9e0740 1 -- 172.21.15.50:0/2672596579 shutdown_connections 2024-04-23T18:28:07.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.744+0000 7f1bff9e0740 1 -- 172.21.15.50:0/2672596579 wait complete. 2024-04-23T18:28:07.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.744+0000 7f1bff9e0740 20 client.16692 trim_cache size 0 max 16384 2024-04-23T18:28:07.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.761+0000 7f1bff9e0740 1 Processor -- start 2024-04-23T18:28:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.761+0000 7f1bff9e0740 1 -- start start 2024-04-23T18:28:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.761+0000 7f1bff9e0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 0x563da3013e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.761+0000 7f1bff9e0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005cb0 0x563da300e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.761+0000 7f1bff9e0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da300edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.761+0000 7f1bff9e0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81ae340 con 0x563da3005390 2024-04-23T18:28:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.761+0000 7f1bff9e0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81abb50 con 0x563da3005cb0 2024-04-23T18:28:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.761+0000 7f1bff9e0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81a99c0 con 0x563da3014890 2024-04-23T18:28:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.761+0000 7f1bf659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 0x563da3013e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.761+0000 7f1bf659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 0x563da3013e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53388/0 (socket says 172.21.15.50:53388) 2024-04-23T18:28:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.761+0000 7f1bf659c640 1 -- 172.21.15.50:0/1275957619 learned_addr learned my addr 172.21.15.50:0/1275957619 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.762+0000 7f1bf6d9d640 1 --2- 172.21.15.50:0/1275957619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da300edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.762+0000 7f1beffff640 1 --2- 172.21.15.50:0/1275957619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005cb0 0x563da300e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.762+0000 7f1bf659c640 1 -- 172.21.15.50:0/1275957619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 msgr2=0x563da300edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.762+0000 7f1bf659c640 1 --2- 172.21.15.50:0/1275957619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da300edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.762+0000 7f1bf659c640 1 -- 172.21.15.50:0/1275957619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005cb0 msgr2=0x563da300e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.762+0000 7f1bf659c640 1 --2- 172.21.15.50:0/1275957619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005cb0 0x563da300e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.762+0000 7f1bf659c640 1 -- 172.21.15.50:0/1275957619 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563da302cab0 con 0x563da3005390 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.762+0000 7f1bf659c640 1 --2- 172.21.15.50:0/1275957619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 0x563da3013e40 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1be8098840 tx=0x7f1be8010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc0a08241d3477f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.762+0000 7f1bdffff640 1 -- 172.21.15.50:0/1275957619 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8020070 con 0x563da3005390 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.762+0000 7f1bdffff640 1 -- 172.21.15.50:0/1275957619 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1be8009de0 con 0x563da3005390 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.762+0000 7f1bdffff640 1 -- 172.21.15.50:0/1275957619 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8098bb0 con 0x563da3005390 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 -- 172.21.15.50:0/1275957619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 msgr2=0x563da3013e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/1275957619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 0x563da3013e40 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1be8098840 tx=0x7f1be8010cb0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 -- 172.21.15.50:0/1275957619 shutdown_connections 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/1275957619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da300edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/1275957619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005cb0 0x563da300e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/1275957619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005390 0x563da3013e40 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 -- 172.21.15.50:0/1275957619 >> 172.21.15.50:0/1275957619 conn(0x563da30017b0 msgr2=0x563da3001b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 -- 172.21.15.50:0/1275957619 shutdown_connections 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 -- 172.21.15.50:0/1275957619 wait complete. 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 -- start start 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 Processor -- start 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005390 0x563da300e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 0x563da300ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3089a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81aa010 con 0x563da3005cb0 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81ab480 con 0x563da3005390 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf81aa6e0 con 0x563da3014890 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bff9e0740 10 client.0 _ll_register_callbacks cb 0x7f1be0005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1beffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 0x563da300ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1beffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 0x563da300ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53404/0 (socket says 172.21.15.50:53404) 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1beffff640 1 -- 172.21.15.50:0/3010700753 learned_addr learned my addr 172.21.15.50:0/3010700753 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bf6d9d640 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3089a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.763+0000 7f1bf659c640 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005390 0x563da300e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1beffff640 1 -- 172.21.15.50:0/3010700753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 msgr2=0x563da3089a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1beffff640 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3089a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1beffff640 1 -- 172.21.15.50:0/3010700753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005390 msgr2=0x563da300e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1beffff640 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005390 0x563da300e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1beffff640 1 -- 172.21.15.50:0/3010700753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563da3089f90 con 0x563da3005cb0 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bf659c640 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005390 0x563da300e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1beffff640 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 0x563da300ee80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f1be0000c00 tx=0x7f1be0004130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb89e5f7c68d2f3f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bddffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:28:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bddffb640 1 -- 172.21.15.50:0/3010700753 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be0035040 con 0x563da3005cb0 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3010700753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1be8009de0 con 0x563da3005cb0 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bddffb640 1 -- 172.21.15.50:0/3010700753 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f1be0044d50 con 0x563da3005cb0 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bddffb640 1 -- 172.21.15.50:0/3010700753 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1be0053050 con 0x563da3005cb0 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bff9e0740 10 client.16698 fetch_fsmap learned FSMap version 40 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3010700753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x563da308b290 con 0x563da3005cb0 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bddffb640 1 -- 172.21.15.50:0/3010700753 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1be8009de0 con 0x563da3005cb0 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bddffb640 1 -- 172.21.15.50:0/3010700753 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7f1be00445b0 con 0x563da3005cb0 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bff9e0740 10 client.16698 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bff9e0740 10 client.16698 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bff9e0740 20 client.16698 populate_metadata read hostname 'smithi050' 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da308ba40 0x563da308de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bff9e0740 4 client.16698 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "46f1ed70", "release": true, "await": true}] 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.764+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3010700753 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "46f1ed70", "release": true, "await": true}) -- 0x563da2f780f0 con 0x563da308ba40 2024-04-23T18:28:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.765+0000 7f1bf659c640 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da308ba40 0x563da308de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:07.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.765+0000 7f1bddffb640 10 client.16698 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:07.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.765+0000 7f1bf659c640 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da308ba40 0x563da308de60 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a71b731559d5bbab server_cookie=d7f2ee23bc18b068 in_seq=0 out_seq=0 2024-04-23T18:28:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.771+0000 7f1bddffb640 1 -- 172.21.15.50:0/3010700753 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x563da2f780f0 con 0x563da308ba40 2024-04-23T18:28:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.771+0000 7f1bddffb640 10 client.16698 handle_command_reply: tid=0 2024-04-23T18:28:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.771+0000 7f1bff9e0740 1 client.16698 shutdown 2024-04-23T18:28:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.771+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3010700753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 msgr2=0x563da300ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.771+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 0x563da300ee80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f1be0000c00 tx=0x7f1be0004130 comp rx=0 tx=0).stop 2024-04-23T18:28:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.771+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3010700753 shutdown_connections 2024-04-23T18:28:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.771+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563da308ba40 0x563da308de60 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.771+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563da3014890 0x563da3089a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.771+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563da3005cb0 0x563da300ee80 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.771+0000 7f1bddffb640 0 client.16698 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:07.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.771+0000 7f1bff9e0740 1 --2- 172.21.15.50:0/3010700753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563da3005390 0x563da300e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.771+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3010700753 >> 172.21.15.50:0/3010700753 conn(0x563da30017b0 msgr2=0x563da301fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:07.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3010700753 shutdown_connections 2024-04-23T18:28:07.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bff9e0740 1 -- 172.21.15.50:0/3010700753 wait complete. 2024-04-23T18:28:07.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bff9e0740 20 client.16698 trim_cache size 0 max 16384 2024-04-23T18:28:07.766 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:28:07.766 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 40, 2024-04-23T18:28:07.766 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 32, 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: "46f1ed70": { 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 32, 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 82.0, 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:28:07.767 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:28:07.768 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:28:07.768 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:28:07.768 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:28:07.768 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:28:07.768 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:28:07.768 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:28:07.768 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:28:07.768 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:28:07.768 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:28:07.768 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:28:07.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/2968332312 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1be40a3d20 msgr2=0x7f1be40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bfcbf9640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1be40a3d20 0x7f1be40a61e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1bf00092f0 tx=0x7f1bf0004ab0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/2968332312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf8102190 msgr2=0x7f1bf8198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:07.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bfcbf9640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf8102190 0x7f1bf8198760 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f1be800d950 tx=0x7f1be800de10 comp rx=0 tx=0).stop 2024-04-23T18:28:07.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/2968332312 shutdown_connections 2024-04-23T18:28:07.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bfcbf9640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1be40a3d20 0x7f1be40a61e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bfcbf9640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bf819d020 0x7f1bf819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bfcbf9640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bf8107260 0x7f1bf8198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bfcbf9640 1 --2- 172.21.15.50:0/2968332312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bf8102190 0x7f1bf8198760 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:07.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/2968332312 >> 172.21.15.50:0/2968332312 conn(0x7f1bf80793a0 msgr2=0x7f1bf81000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:07.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/2968332312 shutdown_connections 2024-04-23T18:28:07.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:28:07.772+0000 7f1bfcbf9640 1 -- 172.21.15.50:0/2968332312 wait complete. 2024-04-23T18:28:07.776 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '46f1ed70', seconds elapsed: 23.2 2024-04-23T18:28:08.225 INFO:tasks.workunit.client.0.smithi050.stdout: 1 607621 5.58 MB/sec execute 402 sec latency 22689.133 ms 2024-04-23T18:28:08.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:08 smithi050 ceph-mon[29187]: pgmap v300: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 290 KiB/s wr, 43 op/s 2024-04-23T18:28:08.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:08 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3027948769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:28:08.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:08 smithi177 ceph-mon[32308]: pgmap v300: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 290 KiB/s wr, 43 op/s 2024-04-23T18:28:08.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:08 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3027948769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:28:08.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:08 smithi188 ceph-mon[32694]: pgmap v300: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 290 KiB/s wr, 43 op/s 2024-04-23T18:28:08.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:08 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3027948769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:28:09.225 INFO:tasks.workunit.client.0.smithi050.stdout: 1 609036 5.58 MB/sec execute 403 sec latency 69.329 ms 2024-04-23T18:28:10.225 INFO:tasks.workunit.client.0.smithi050.stdout: 1 609782 5.58 MB/sec execute 404 sec latency 785.646 ms 2024-04-23T18:28:10.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:10 smithi050 ceph-mon[29187]: pgmap v301: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 31 op/s 2024-04-23T18:28:10.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:10 smithi050 ceph-mon[29187]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T18:28:10.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:10 smithi177 ceph-mon[32308]: pgmap v301: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 31 op/s 2024-04-23T18:28:10.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:10 smithi177 ceph-mon[32308]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T18:28:10.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:10 smithi188 ceph-mon[32694]: pgmap v301: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 31 op/s 2024-04-23T18:28:10.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:10 smithi188 ceph-mon[32694]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T18:28:11.225 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610836 5.59 MB/sec execute 405 sec latency 581.497 ms 2024-04-23T18:28:12.225 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610836 5.57 MB/sec execute 406 sec latency 1581.610 ms 2024-04-23T18:28:12.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:12 smithi050 ceph-mon[29187]: pgmap v303: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 92 KiB/s wr, 37 op/s 2024-04-23T18:28:12.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.540+0000 7f845d2d0640 1 -- 172.21.15.50:0/4198954811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f845810c5d0 msgr2=0x7f845810ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.540+0000 7f845d2d0640 1 --2- 172.21.15.50:0/4198954811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f845810c5d0 0x7f845810ea00 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f8450009ca0 tx=0x7f845002a950 comp rx=0 tx=0).stop 2024-04-23T18:28:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.540+0000 7f845d2d0640 1 -- 172.21.15.50:0/4198954811 shutdown_connections 2024-04-23T18:28:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.540+0000 7f845d2d0640 1 --2- 172.21.15.50:0/4198954811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f845810c5d0 0x7f845810ea00 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.540+0000 7f845d2d0640 1 --2- 172.21.15.50:0/4198954811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8458103b20 0x7f845810c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.540+0000 7f845d2d0640 1 --2- 172.21.15.50:0/4198954811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8458103200 0x7f84581035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.540+0000 7f845d2d0640 1 -- 172.21.15.50:0/4198954811 >> 172.21.15.50:0/4198954811 conn(0x7f84580771d0 msgr2=0x7f84580775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.540+0000 7f845d2d0640 1 -- 172.21.15.50:0/4198954811 shutdown_connections 2024-04-23T18:28:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.540+0000 7f845d2d0640 1 -- 172.21.15.50:0/4198954811 wait complete. 2024-04-23T18:28:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f845d2d0640 1 Processor -- start 2024-04-23T18:28:12.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f845d2d0640 1 -- start start 2024-04-23T18:28:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f845d2d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8458103200 0x7f8458198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f845d2d0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8458103b20 0x7f8458198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f845d2d0640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f845810c5d0 0x7f845819cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f845d2d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f84581121f0 con 0x7f8458103200 2024-04-23T18:28:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f845d2d0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8458112070 con 0x7f8458103b20 2024-04-23T18:28:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f845d2d0640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8458112370 con 0x7f845810c5d0 2024-04-23T18:28:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f8456ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8458103200 0x7f8458198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f8456ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8458103200 0x7f8458198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34228/0 (socket says 172.21.15.50:34228) 2024-04-23T18:28:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f8456ffd640 1 -- 172.21.15.50:0/3299353736 learned_addr learned my addr 172.21.15.50:0/3299353736 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f84567fc640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8458103b20 0x7f8458198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.541+0000 7f84577fe640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f845810c5d0 0x7f845819cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.542+0000 7f8456ffd640 1 -- 172.21.15.50:0/3299353736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f845810c5d0 msgr2=0x7f845819cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.542+0000 7f8456ffd640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f845810c5d0 0x7f845819cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.542+0000 7f8456ffd640 1 -- 172.21.15.50:0/3299353736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8458103b20 msgr2=0x7f8458198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.542+0000 7f8456ffd640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8458103b20 0x7f8458198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.542+0000 7f8456ffd640 1 -- 172.21.15.50:0/3299353736 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845819d630 con 0x7f8458103200 2024-04-23T18:28:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.542+0000 7f8456ffd640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8458103200 0x7f8458198690 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f844800d770 tx=0x7f844800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.542+0000 7f843ffff640 1 -- 172.21.15.50:0/3299353736 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8448017070 con 0x7f8458103200 2024-04-23T18:28:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.542+0000 7f845d2d0640 1 -- 172.21.15.50:0/3299353736 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f845819d920 con 0x7f8458103200 2024-04-23T18:28:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.542+0000 7f843ffff640 1 -- 172.21.15.50:0/3299353736 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84480054d0 con 0x7f8458103200 2024-04-23T18:28:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.542+0000 7f843ffff640 1 -- 172.21.15.50:0/3299353736 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8448004c30 con 0x7f8458103200 2024-04-23T18:28:12.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.542+0000 7f845d2d0640 1 -- 172.21.15.50:0/3299353736 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84581a51b0 con 0x7f8458103200 2024-04-23T18:28:12.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:12 smithi177 ceph-mon[32308]: pgmap v303: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 92 KiB/s wr, 37 op/s 2024-04-23T18:28:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.545+0000 7f843ffff640 1 -- 172.21.15.50:0/3299353736 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8448004e10 con 0x7f8458103200 2024-04-23T18:28:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.545+0000 7f845ffde740 1 Processor -- start 2024-04-23T18:28:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.545+0000 7f845ffde740 1 -- start start 2024-04-23T18:28:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.545+0000 7f843ffff640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f84400a3dc0 0x7f84400a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.545+0000 7f845ffde740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 0x55e870f49a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.545+0000 7f845ffde740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49f40 0x55e870f542c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.545+0000 7f845ffde740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f54800 0x55e870f56bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.545+0000 7f845ffde740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e870eae970 con 0x55e870f49620 2024-04-23T18:28:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.545+0000 7f845ffde740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e870e4b4a0 con 0x55e870f49f40 2024-04-23T18:28:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.545+0000 7f845ffde740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e870eaa930 con 0x55e870f54800 2024-04-23T18:28:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f8456ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 0x55e870f49a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f84577fe640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f84400a3dc0 0x7f84400a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f843ffff640 1 -- 172.21.15.50:0/3299353736 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8586+0+0 (secure 0 0 0) 0x7f84480d0260 con 0x7f8458103200 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f84567fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49f40 0x55e870f542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f8456ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 0x55e870f49a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34244/0 (socket says 172.21.15.50:34244) 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f8456ffd640 1 -- 172.21.15.50:0/1144503461 learned_addr learned my addr 172.21.15.50:0/1144503461 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f84577fe640 1 --2- 172.21.15.50:0/1144503461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f54800 0x55e870f56bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f8456ffd640 1 -- 172.21.15.50:0/1144503461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f54800 msgr2=0x55e870f56bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f8456ffd640 1 --2- 172.21.15.50:0/1144503461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f54800 0x55e870f56bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f8456ffd640 1 -- 172.21.15.50:0/1144503461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49f40 msgr2=0x55e870f542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f8456ffd640 1 --2- 172.21.15.50:0/1144503461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49f40 0x55e870f542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f8456ffd640 1 -- 172.21.15.50:0/1144503461 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e870bc1210 con 0x55e870f49620 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f84567fc640 1 --2- 172.21.15.50:0/1144503461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49f40 0x55e870f542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f8456ffd640 1 --2- 172.21.15.50:0/1144503461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 0x55e870f49a00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f84480986d0 tx=0x7f8448098640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40da48e072a0785 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f84577fe640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f84400a3dc0 0x7f84400a6280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8450005e30 tx=0x7f8450005f60 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f843d7fa640 1 -- 172.21.15.50:0/1144503461 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8448020040 con 0x55e870f49620 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f843d7fa640 1 -- 172.21.15.50:0/1144503461 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84480a4e30 con 0x55e870f49620 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.546+0000 7f843d7fa640 1 -- 172.21.15.50:0/1144503461 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84480cdd80 con 0x55e870f49620 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.547+0000 7f845ffde740 1 -- 172.21.15.50:0/1144503461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 msgr2=0x55e870f49a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.547+0000 7f845ffde740 1 --2- 172.21.15.50:0/1144503461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 0x55e870f49a00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f84480986d0 tx=0x7f8448098640 comp rx=0 tx=0).stop 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.547+0000 7f845ffde740 1 -- 172.21.15.50:0/1144503461 shutdown_connections 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.547+0000 7f845ffde740 1 --2- 172.21.15.50:0/1144503461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f54800 0x55e870f56bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.547+0000 7f845ffde740 1 --2- 172.21.15.50:0/1144503461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49f40 0x55e870f542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.547+0000 7f845ffde740 1 --2- 172.21.15.50:0/1144503461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 0x55e870f49a00 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.547+0000 7f845ffde740 1 -- 172.21.15.50:0/1144503461 >> 172.21.15.50:0/1144503461 conn(0x55e870f45b30 msgr2=0x55e870f45f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.547+0000 7f845ffde740 1 -- 172.21.15.50:0/1144503461 shutdown_connections 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.547+0000 7f845ffde740 1 -- 172.21.15.50:0/1144503461 wait complete. 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f845ffde740 1 -- start start 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f845ffde740 1 Processor -- start 2024-04-23T18:28:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f845ffde740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 0x55e870f541b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f845ffde740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f49f40 0x55e870f514a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f845ffde740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f54800 0x55e870f51a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f845ffde740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55e870f57430 con 0x55e870f49620 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f845ffde740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e870e490b0 con 0x55e870f54800 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f845ffde740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e870f575b0 con 0x55e870f49f40 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f845ffde740 10 client.0 _ll_register_callbacks cb 0x55e870e36d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f8456ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 0x55e870f541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f8456ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 0x55e870f541b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34258/0 (socket says 172.21.15.50:34258) 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f8456ffd640 1 -- 172.21.15.50:0/3439813004 learned_addr learned my addr 172.21.15.50:0/3439813004 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f84577fe640 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f54800 0x55e870f51a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f84567fc640 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f49f40 0x55e870f514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f8456ffd640 1 -- 172.21.15.50:0/3439813004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f49f40 msgr2=0x55e870f514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f8456ffd640 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f49f40 0x55e870f514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f8456ffd640 1 -- 172.21.15.50:0/3439813004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f54800 msgr2=0x55e870f51a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f8456ffd640 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f54800 0x55e870f51a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f8456ffd640 1 -- 172.21.15.50:0/3439813004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e870c8d620 con 0x55e870f49620 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f84567fc640 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f49f40 0x55e870f514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.548+0000 7f84577fe640 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f54800 0x55e870f51a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f8456ffd640 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 0x55e870f541b0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f84480cdf70 tx=0x7f8448098b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb33bff7985e414 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f843aff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f843aff5640 1 -- 172.21.15.50:0/3439813004 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8448020040 con 0x55e870f49620 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f843aff5640 1 -- 172.21.15.50:0/3439813004 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8586+0+0 (secure 0 0 0) 0x7f84480a49a0 con 0x55e870f49620 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f845ffde740 1 -- 172.21.15.50:0/3439813004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84480a4e30 con 0x55e870f49620 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f843aff5640 1 -- 172.21.15.50:0/3439813004 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f84480a0360 con 0x55e870f49620 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f845ffde740 10 client.16710 fetch_fsmap learned FSMap version 40 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f845ffde740 1 -- 172.21.15.50:0/3439813004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e870f633a0 con 0x55e870f49620 2024-04-23T18:28:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f843aff5640 1 -- 172.21.15.50:0/3439813004 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84480a4e30 con 0x55e870f49620 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f843aff5640 1 -- 172.21.15.50:0/3439813004 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7f84480bcc20 con 0x55e870f49620 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f845ffde740 10 client.16710 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f845ffde740 10 client.16710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f845ffde740 20 client.16710 populate_metadata read hostname 'smithi050' 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f845ffde740 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e870f646f0 0x55e870f66ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f845ffde740 4 client.16710 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.549+0000 7f845ffde740 1 -- 172.21.15.50:0/3439813004 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e870ebc390 con 0x55e870f646f0 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.550+0000 7f84567fc640 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e870f646f0 0x55e870f66ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.550+0000 7f843aff5640 10 client.16710 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.550+0000 7f84567fc640 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e870f646f0 0x55e870f66ae0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a42fef617d5ccc4a server_cookie=476a804ba471082d in_seq=0 out_seq=0 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f843aff5640 1 -- 172.21.15.50:0/3439813004 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e870ebc390 con 0x55e870f646f0 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f843aff5640 10 client.16710 handle_command_reply: tid=0 2024-04-23T18:28:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f845ffde740 1 client.16710 shutdown 2024-04-23T18:28:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f845ffde740 1 -- 172.21.15.50:0/3439813004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 msgr2=0x55e870f541b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f845ffde740 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 0x55e870f541b0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f84480cdf70 tx=0x7f8448098b10 comp rx=0 tx=0).stop 2024-04-23T18:28:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f845ffde740 1 -- 172.21.15.50:0/3439813004 shutdown_connections 2024-04-23T18:28:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f845ffde740 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e870f646f0 0x55e870f66ae0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f845ffde740 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f54800 0x55e870f51a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f843aff5640 0 client.16710 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f845ffde740 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f49f40 0x55e870f514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f845ffde740 1 --2- 172.21.15.50:0/3439813004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49620 0x55e870f541b0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f845ffde740 1 -- 172.21.15.50:0/3439813004 >> 172.21.15.50:0/3439813004 conn(0x55e870f45b30 msgr2=0x55e870f4b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f845ffde740 1 -- 172.21.15.50:0/3439813004 shutdown_connections 2024-04-23T18:28:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f845ffde740 1 -- 172.21.15.50:0/3439813004 wait complete. 2024-04-23T18:28:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.551+0000 7f845ffde740 20 client.16710 trim_cache size 0 max 16384 2024-04-23T18:28:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.566+0000 7f845ffde740 1 Processor -- start 2024-04-23T18:28:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.566+0000 7f845ffde740 1 -- start start 2024-04-23T18:28:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.566+0000 7f845ffde740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f45b30 0x55e870f602f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.566+0000 7f845ffde740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49620 0x55e870f44b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.566+0000 7f845ffde740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49f40 0x55e870f53ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.566+0000 7f845ffde740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f84581b05a0 con 0x55e870f49f40 2024-04-23T18:28:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.566+0000 7f845ffde740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f84581addb0 con 0x55e870f49620 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f845ffde740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f84581abc20 con 0x55e870f45b30 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f84577fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49f40 0x55e870f53ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f84577fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49f40 0x55e870f53ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34270/0 (socket says 172.21.15.50:34270) 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f84577fe640 1 -- 172.21.15.50:0/3435137634 learned_addr learned my addr 172.21.15.50:0/3435137634 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f84567fc640 1 --2- 172.21.15.50:0/3435137634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49620 0x55e870f44b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f8456ffd640 1 --2- 172.21.15.50:0/3435137634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f45b30 0x55e870f602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f84577fe640 1 -- 172.21.15.50:0/3435137634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f45b30 msgr2=0x55e870f602f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f84577fe640 1 --2- 172.21.15.50:0/3435137634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f45b30 0x55e870f602f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f84577fe640 1 -- 172.21.15.50:0/3435137634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49620 msgr2=0x55e870f44b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f84577fe640 1 --2- 172.21.15.50:0/3435137634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49620 0x55e870f44b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f84577fe640 1 -- 172.21.15.50:0/3435137634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e870f51db0 con 0x55e870f49f40 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f84567fc640 1 --2- 172.21.15.50:0/3435137634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49620 0x55e870f44b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:28:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f8456ffd640 1 --2- 172.21.15.50:0/3435137634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f45b30 0x55e870f602f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f84577fe640 1 --2- 172.21.15.50:0/3435137634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49f40 0x55e870f53ce0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f8450035000 tx=0x7f8450049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b399158e404aec1b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f843cff9640 1 -- 172.21.15.50:0/3435137634 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8450034040 con 0x55e870f49f40 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f843cff9640 1 -- 172.21.15.50:0/3435137634 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f845004f070 con 0x55e870f49f40 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.567+0000 7f843cff9640 1 -- 172.21.15.50:0/3435137634 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8450004680 con 0x55e870f49f40 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.568+0000 7f845ffde740 1 -- 172.21.15.50:0/3435137634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49f40 msgr2=0x55e870f53ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.568+0000 7f845ffde740 1 --2- 172.21.15.50:0/3435137634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49f40 0x55e870f53ce0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f8450035000 tx=0x7f8450049000 comp rx=0 tx=0).stop 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.568+0000 7f845ffde740 1 -- 172.21.15.50:0/3435137634 shutdown_connections 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.568+0000 7f845ffde740 1 --2- 172.21.15.50:0/3435137634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f49f40 0x55e870f53ce0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.568+0000 7f845ffde740 1 --2- 172.21.15.50:0/3435137634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49620 0x55e870f44b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.568+0000 7f845ffde740 1 --2- 172.21.15.50:0/3435137634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f45b30 0x55e870f602f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.568+0000 7f845ffde740 1 -- 172.21.15.50:0/3435137634 >> 172.21.15.50:0/3435137634 conn(0x55e870f612d0 msgr2=0x55e870f44420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.568+0000 7f845ffde740 1 -- 172.21.15.50:0/3435137634 shutdown_connections 2024-04-23T18:28:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.568+0000 7f845ffde740 1 -- 172.21.15.50:0/3435137634 wait complete. 2024-04-23T18:28:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.568+0000 7f845ffde740 1 -- start start 2024-04-23T18:28:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.568+0000 7f845ffde740 1 Processor -- start 2024-04-23T18:28:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f845ffde740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f45b30 0x55e870f5cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f845ffde740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f49620 0x55e870f4df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f845ffde740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49f40 0x55e870f4e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f845ffde740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f84581ac270 con 0x55e870f45b30 2024-04-23T18:28:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f845ffde740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f84581ad6e0 con 0x55e870f49f40 2024-04-23T18:28:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f845ffde740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f84581ac940 con 0x55e870f49620 2024-04-23T18:28:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f845ffde740 10 client.0 _ll_register_callbacks cb 0x7f844800de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f8456ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f45b30 0x55e870f5cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f8456ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f45b30 0x55e870f5cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34274/0 (socket says 172.21.15.50:34274) 2024-04-23T18:28:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f8456ffd640 1 -- 172.21.15.50:0/883035802 learned_addr learned my addr 172.21.15.50:0/883035802 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f84567fc640 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f49620 0x55e870f4df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f84577fe640 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49f40 0x55e870f4e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f8456ffd640 1 -- 172.21.15.50:0/883035802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f49620 msgr2=0x55e870f4df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f8456ffd640 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f49620 0x55e870f4df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f8456ffd640 1 -- 172.21.15.50:0/883035802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49f40 msgr2=0x55e870f4e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f8456ffd640 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49f40 0x55e870f4e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f8456ffd640 1 -- 172.21.15.50:0/883035802 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e870f4ecd0 con 0x55e870f45b30 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f84577fe640 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49f40 0x55e870f4e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f8456ffd640 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f45b30 0x55e870f5cc70 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f8448098180 tx=0x7f8448004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647f7499c583f033 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f843aff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f843aff5640 1 -- 172.21.15.50:0/883035802 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84480c5030 con 0x55e870f45b30 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f843aff5640 1 -- 172.21.15.50:0/883035802 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8586+0+0 (secure 0 0 0) 0x7f84480a49a0 con 0x55e870f45b30 2024-04-23T18:28:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.569+0000 7f845ffde740 1 -- 172.21.15.50:0/883035802 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f845004f070 con 0x55e870f45b30 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.570+0000 7f843aff5640 1 -- 172.21.15.50:0/883035802 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f84480b5030 con 0x55e870f45b30 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.570+0000 7f845ffde740 10 client.16716 fetch_fsmap learned FSMap version 40 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.570+0000 7f845ffde740 1 -- 172.21.15.50:0/883035802 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e870fcab60 con 0x55e870f45b30 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.570+0000 7f843aff5640 1 -- 172.21.15.50:0/883035802 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f845004f070 con 0x55e870f45b30 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.570+0000 7f843aff5640 1 -- 172.21.15.50:0/883035802 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7f84480bcb00 con 0x55e870f45b30 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.570+0000 7f845ffde740 10 client.16716 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.570+0000 7f845ffde740 10 client.16716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.570+0000 7f845ffde740 20 client.16716 populate_metadata read hostname 'smithi050' 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.570+0000 7f845ffde740 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e870f54800 0x55e870fcd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.570+0000 7f845ffde740 4 client.16716 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.570+0000 7f845ffde740 1 -- 172.21.15.50:0/883035802 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e870eb6200 con 0x55e870f54800 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.570+0000 7f84567fc640 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e870f54800 0x55e870fcd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.571+0000 7f843aff5640 10 client.16716 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:12.571+0000 7f84567fc640 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e870f54800 0x55e870fcd100 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7258b07c6ff4fa6 server_cookie=b0ae5f26cd1b38b2 in_seq=0 out_seq=0 2024-04-23T18:28:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:12 smithi188 ceph-mon[32694]: pgmap v303: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 92 KiB/s wr, 37 op/s 2024-04-23T18:28:13.225 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610836 5.56 MB/sec execute 407 sec latency 2581.715 ms 2024-04-23T18:28:14.225 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610836 5.55 MB/sec execute 408 sec latency 3581.864 ms 2024-04-23T18:28:14.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:14 smithi050 ceph-mon[29187]: pgmap v304: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 480 KiB/s wr, 73 op/s 2024-04-23T18:28:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:14 smithi177 ceph-mon[32308]: pgmap v304: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 480 KiB/s wr, 73 op/s 2024-04-23T18:28:14.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:14 smithi188 ceph-mon[32694]: pgmap v304: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 480 KiB/s wr, 73 op/s 2024-04-23T18:28:15.225 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610836 5.53 MB/sec execute 409 sec latency 4581.961 ms 2024-04-23T18:28:16.225 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610836 5.52 MB/sec execute 410 sec latency 5582.069 ms 2024-04-23T18:28:16.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:16 smithi050 ceph-mon[29187]: pgmap v305: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 480 KiB/s wr, 73 op/s 2024-04-23T18:28:16.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:16 smithi050 ceph-mon[29187]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T18:28:16.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:16 smithi177 ceph-mon[32308]: pgmap v305: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 480 KiB/s wr, 73 op/s 2024-04-23T18:28:16.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:16 smithi177 ceph-mon[32308]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T18:28:16.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:16 smithi188 ceph-mon[32694]: pgmap v305: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 480 KiB/s wr, 73 op/s 2024-04-23T18:28:16.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:16 smithi188 ceph-mon[32694]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T18:28:17.226 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610836 5.51 MB/sec execute 411 sec latency 6582.141 ms 2024-04-23T18:28:18.226 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610836 5.49 MB/sec execute 412 sec latency 7582.193 ms 2024-04-23T18:28:18.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:18 smithi050 ceph-mon[29187]: pgmap v307: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 994 KiB/s wr, 116 op/s 2024-04-23T18:28:18.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:18 smithi177 ceph-mon[32308]: pgmap v307: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 994 KiB/s wr, 116 op/s 2024-04-23T18:28:18.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:18 smithi188 ceph-mon[32694]: pgmap v307: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 994 KiB/s wr, 116 op/s 2024-04-23T18:28:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.855+0000 7f843aff5640 1 -- 172.21.15.50:0/883035802 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e870eb6200 con 0x55e870f54800 2024-04-23T18:28:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.855+0000 7f843aff5640 10 client.16716 handle_command_reply: tid=0 2024-04-23T18:28:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.855+0000 7f845ffde740 1 client.16716 shutdown 2024-04-23T18:28:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.855+0000 7f845ffde740 1 -- 172.21.15.50:0/883035802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f45b30 msgr2=0x55e870f5cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.855+0000 7f845ffde740 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f45b30 0x55e870f5cc70 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f8448098180 tx=0x7f8448004730 comp rx=0 tx=0).stop 2024-04-23T18:28:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.855+0000 7f845ffde740 1 -- 172.21.15.50:0/883035802 shutdown_connections 2024-04-23T18:28:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.855+0000 7f845ffde740 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55e870f54800 0x55e870fcd100 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.856+0000 7f845ffde740 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e870f49f40 0x55e870f4e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.856+0000 7f845ffde740 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e870f49620 0x55e870f4df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.856+0000 7f845ffde740 1 --2- 172.21.15.50:0/883035802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55e870f45b30 0x55e870f5cc70 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.856+0000 7f845ffde740 1 -- 172.21.15.50:0/883035802 >> 172.21.15.50:0/883035802 conn(0x55e870f612d0 msgr2=0x55e870f4b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.856+0000 7f843aff5640 0 client.16716 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.856+0000 7f845ffde740 1 -- 172.21.15.50:0/883035802 shutdown_connections 2024-04-23T18:28:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.856+0000 7f845ffde740 1 -- 172.21.15.50:0/883035802 wait complete. 2024-04-23T18:28:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.856+0000 7f845ffde740 20 client.16716 trim_cache size 0 max 16384 2024-04-23T18:28:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:28:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:28:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:28:18.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:28:18.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.857+0000 7f845d2d0640 1 -- 172.21.15.50:0/3299353736 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f84400a3dc0 msgr2=0x7f84400a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:18.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.857+0000 7f845d2d0640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f84400a3dc0 0x7f84400a6280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8450005e30 tx=0x7f8450005f60 comp rx=0 tx=0).stop 2024-04-23T18:28:18.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.857+0000 7f845d2d0640 1 -- 172.21.15.50:0/3299353736 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8458103200 msgr2=0x7f8458198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:18.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.857+0000 7f845d2d0640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8458103200 0x7f8458198690 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f844800d770 tx=0x7f844800dc30 comp rx=0 tx=0).stop 2024-04-23T18:28:18.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.857+0000 7f845d2d0640 1 -- 172.21.15.50:0/3299353736 shutdown_connections 2024-04-23T18:28:18.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.857+0000 7f845d2d0640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f84400a3dc0 0x7f84400a6280 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:18.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.857+0000 7f845d2d0640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f845810c5d0 0x7f845819cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:18.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.857+0000 7f845d2d0640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8458103b20 0x7f8458198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:18.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.857+0000 7f845d2d0640 1 --2- 172.21.15.50:0/3299353736 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8458103200 0x7f8458198690 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:18.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.857+0000 7f845d2d0640 1 -- 172.21.15.50:0/3299353736 >> 172.21.15.50:0/3299353736 conn(0x7f84580771d0 msgr2=0x7f84580fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:18.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.857+0000 7f845d2d0640 1 -- 172.21.15.50:0/3299353736 shutdown_connections 2024-04-23T18:28:18.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:18.857+0000 7f845d2d0640 1 -- 172.21.15.50:0/3299353736 wait complete. 2024-04-23T18:28:19.226 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610874 5.48 MB/sec execute 413 sec latency 8202.817 ms 2024-04-23T18:28:20.226 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610899 5.47 MB/sec execute 414 sec latency 1138.493 ms 2024-04-23T18:28:20.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:20 smithi050 ceph-mon[29187]: pgmap v308: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 837 KiB/s wr, 98 op/s 2024-04-23T18:28:20.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:20 smithi177 ceph-mon[32308]: pgmap v308: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 837 KiB/s wr, 98 op/s 2024-04-23T18:28:20.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:20 smithi188 ceph-mon[32694]: pgmap v308: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 837 KiB/s wr, 98 op/s 2024-04-23T18:28:21.226 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610899 5.45 MB/sec execute 415 sec latency 1195.759 ms 2024-04-23T18:28:22.226 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610899 5.44 MB/sec execute 416 sec latency 2195.908 ms 2024-04-23T18:28:22.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:22 smithi050 ceph-mon[29187]: pgmap v309: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 795 KiB/s wr, 93 op/s 2024-04-23T18:28:22.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:22 smithi177 ceph-mon[32308]: pgmap v309: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 795 KiB/s wr, 93 op/s 2024-04-23T18:28:22.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:22 smithi188 ceph-mon[32694]: pgmap v309: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 795 KiB/s wr, 93 op/s 2024-04-23T18:28:23.226 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610899 5.43 MB/sec execute 417 sec latency 3196.010 ms 2024-04-23T18:28:24.226 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610899 5.41 MB/sec execute 418 sec latency 4196.067 ms 2024-04-23T18:28:24.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:24 smithi050 ceph-mon[29187]: pgmap v310: 97 pgs: 97 active+clean; 194 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 364 KiB/s wr, 34 op/s 2024-04-23T18:28:24.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:24 smithi177 ceph-mon[32308]: pgmap v310: 97 pgs: 97 active+clean; 194 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 364 KiB/s wr, 34 op/s 2024-04-23T18:28:24.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:24 smithi188 ceph-mon[32694]: pgmap v310: 97 pgs: 97 active+clean; 194 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 364 KiB/s wr, 34 op/s 2024-04-23T18:28:25.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 610980 5.40 MB/sec execute 419 sec latency 4912.236 ms 2024-04-23T18:28:26.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 611009 5.39 MB/sec execute 420 sec latency 898.850 ms 2024-04-23T18:28:26.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:26 smithi050 ceph-mon[29187]: pgmap v311: 97 pgs: 97 active+clean; 194 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 364 KiB/s wr, 34 op/s 2024-04-23T18:28:26.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:26 smithi177 ceph-mon[32308]: pgmap v311: 97 pgs: 97 active+clean; 194 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 364 KiB/s wr, 34 op/s 2024-04-23T18:28:26.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:26 smithi188 ceph-mon[32694]: pgmap v311: 97 pgs: 97 active+clean; 194 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 364 KiB/s wr, 34 op/s 2024-04-23T18:28:27.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 611009 5.38 MB/sec execute 421 sec latency 1310.617 ms 2024-04-23T18:28:28.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 611009 5.36 MB/sec execute 422 sec latency 2310.742 ms 2024-04-23T18:28:28.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:28 smithi050 ceph-mon[29187]: pgmap v312: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 641 KiB/s wr, 45 op/s 2024-04-23T18:28:28.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:28 smithi177 ceph-mon[32308]: pgmap v312: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 641 KiB/s wr, 45 op/s 2024-04-23T18:28:28.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:28 smithi188 ceph-mon[32694]: pgmap v312: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 641 KiB/s wr, 45 op/s 2024-04-23T18:28:29.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 611009 5.35 MB/sec execute 423 sec latency 3310.852 ms 2024-04-23T18:28:30.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 611262 5.34 MB/sec execute 424 sec latency 4030.080 ms 2024-04-23T18:28:30.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:30 smithi050 ceph-mon[29187]: pgmap v313: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 302 KiB/s wr, 20 op/s 2024-04-23T18:28:30.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:30 smithi177 ceph-mon[32308]: pgmap v313: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 302 KiB/s wr, 20 op/s 2024-04-23T18:28:30.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:30 smithi188 ceph-mon[32694]: pgmap v313: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 302 KiB/s wr, 20 op/s 2024-04-23T18:28:31.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 612586 5.34 MB/sec execute 425 sec latency 74.852 ms 2024-04-23T18:28:31.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:31 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:31.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:31 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:31.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:31 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:31.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.960+0000 7fea54c7c640 1 -- 172.21.15.50:0/2204258474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea501010f0 msgr2=0x7fea501014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:31.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.960+0000 7fea54c7c640 1 --2- 172.21.15.50:0/2204258474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea501010f0 0x7fea501014d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fea40009900 tx=0x7fea4002a410 comp rx=0 tx=0).stop 2024-04-23T18:28:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.960+0000 7fea54c7c640 1 -- 172.21.15.50:0/2204258474 shutdown_connections 2024-04-23T18:28:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.960+0000 7fea54c7c640 1 --2- 172.21.15.50:0/2204258474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea50106170 0x7fea5010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.960+0000 7fea54c7c640 1 --2- 172.21.15.50:0/2204258474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea50101aa0 0x7fea50105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.960+0000 7fea54c7c640 1 --2- 172.21.15.50:0/2204258474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea501010f0 0x7fea501014d0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.960+0000 7fea54c7c640 1 -- 172.21.15.50:0/2204258474 >> 172.21.15.50:0/2204258474 conn(0x7fea50077fd0 msgr2=0x7fea500783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.960+0000 7fea54c7c640 1 -- 172.21.15.50:0/2204258474 shutdown_connections 2024-04-23T18:28:31.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.960+0000 7fea54c7c640 1 -- 172.21.15.50:0/2204258474 wait complete. 2024-04-23T18:28:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea54c7c640 1 Processor -- start 2024-04-23T18:28:31.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea54c7c640 1 -- start start 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea54c7c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea501010f0 0x7fea50102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea54c7c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea50106170 0x7fea50103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea54c7c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea501035b0 0x7fea501a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea54c7c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fea501110e0 con 0x7fea50106170 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea54c7c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fea50110f60 con 0x7fea501035b0 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea54c7c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fea50111260 con 0x7fea501010f0 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea4dd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea50106170 0x7fea50103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea4dd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea50106170 0x7fea50103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38890/0 (socket says 172.21.15.50:38890) 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea4dd9b640 1 -- 172.21.15.50:0/3022951446 learned_addr learned my addr 172.21.15.50:0/3022951446 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea4e59c640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea501010f0 0x7fea50102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.961+0000 7fea4ed9d640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea501035b0 0x7fea501a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.962+0000 7fea4dd9b640 1 -- 172.21.15.50:0/3022951446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea501010f0 msgr2=0x7fea50102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:31.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.962+0000 7fea4dd9b640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea501010f0 0x7fea50102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.962+0000 7fea4dd9b640 1 -- 172.21.15.50:0/3022951446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea501035b0 msgr2=0x7fea501a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:31.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.962+0000 7fea4dd9b640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea501035b0 0x7fea501a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.962+0000 7fea4dd9b640 1 -- 172.21.15.50:0/3022951446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea501a6fe0 con 0x7fea50106170 2024-04-23T18:28:31.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.962+0000 7fea4dd9b640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea50106170 0x7fea50103070 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fea44005fd0 tx=0x7fea4400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:31.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.962+0000 7fea3f7fe640 1 -- 172.21.15.50:0/3022951446 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea44017070 con 0x7fea50106170 2024-04-23T18:28:31.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.962+0000 7fea3f7fe640 1 -- 172.21.15.50:0/3022951446 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea44009c30 con 0x7fea50106170 2024-04-23T18:28:31.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.962+0000 7fea54c7c640 1 -- 172.21.15.50:0/3022951446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea501a72d0 con 0x7fea50106170 2024-04-23T18:28:31.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.962+0000 7fea3f7fe640 1 -- 172.21.15.50:0/3022951446 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea44003d30 con 0x7fea50106170 2024-04-23T18:28:31.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.962+0000 7fea54c7c640 1 -- 172.21.15.50:0/3022951446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea501a7820 con 0x7fea50106170 2024-04-23T18:28:31.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.964+0000 7fea3f7fe640 1 -- 172.21.15.50:0/3022951446 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fea44003ed0 con 0x7fea50106170 2024-04-23T18:28:31.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.965+0000 7fea57991740 1 Processor -- start 2024-04-23T18:28:31.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.965+0000 7fea57991740 1 -- start start 2024-04-23T18:28:31.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.965+0000 7fea57991740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1a3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.965+0000 7fea57991740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1ae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.965+0000 7fea57991740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1ae800 0x55632b1b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.965+0000 7fea57991740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55632b108970 con 0x55632b1a3f40 2024-04-23T18:28:31.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.965+0000 7fea57991740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55632b0a54a0 con 0x55632b1ae800 2024-04-23T18:28:31.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.965+0000 7fea57991740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55632b104930 con 0x55632b1a3620 2024-04-23T18:28:31.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1ae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea3f7fe640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fea380a3df0 0x7fea380a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1ae2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38898/0 (socket says 172.21.15.50:38898) 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4e59c640 1 -- 172.21.15.50:0/2936789572 learned_addr learned my addr 172.21.15.50:0/2936789572 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4ed9d640 1 --2- 172.21.15.50:0/2936789572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1a3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4dd9b640 1 --2- 172.21.15.50:0/2936789572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1ae800 0x55632b1b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4ed9d640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fea380a3df0 0x7fea380a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea3f7fe640 1 -- 172.21.15.50:0/3022951446 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7fea440d0400 con 0x7fea50106170 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4e59c640 1 -- 172.21.15.50:0/2936789572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 msgr2=0x55632b1a3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4e59c640 1 --2- 172.21.15.50:0/2936789572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1a3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4e59c640 1 -- 172.21.15.50:0/2936789572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1ae800 msgr2=0x55632b1b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4e59c640 1 --2- 172.21.15.50:0/2936789572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1ae800 0x55632b1b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4e59c640 1 -- 172.21.15.50:0/2936789572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55632ae1b210 con 0x55632b1a3f40 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4e59c640 1 --2- 172.21.15.50:0/2936789572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1ae2c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fea40009900 tx=0x7fea40032e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c51ebcbdb90773 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:31.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea4ed9d640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fea380a3df0 0x7fea380a62b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fea4800a780 tx=0x7fea4800a4d0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.966+0000 7fea3cff9640 1 -- 172.21.15.50:0/2936789572 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40040070 con 0x55632b1a3f40 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.967+0000 7fea3cff9640 1 -- 172.21.15.50:0/2936789572 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea400316a0 con 0x55632b1a3f40 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.967+0000 7fea3cff9640 1 -- 172.21.15.50:0/2936789572 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40035040 con 0x55632b1a3f40 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.967+0000 7fea57991740 1 -- 172.21.15.50:0/2936789572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 msgr2=0x55632b1ae2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.967+0000 7fea57991740 1 --2- 172.21.15.50:0/2936789572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1ae2c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fea40009900 tx=0x7fea40032e60 comp rx=0 tx=0).stop 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.967+0000 7fea57991740 1 -- 172.21.15.50:0/2936789572 shutdown_connections 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.967+0000 7fea57991740 1 --2- 172.21.15.50:0/2936789572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1ae800 0x55632b1b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.967+0000 7fea57991740 1 --2- 172.21.15.50:0/2936789572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1ae2c0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.967+0000 7fea57991740 1 --2- 172.21.15.50:0/2936789572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1a3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.967+0000 7fea57991740 1 -- 172.21.15.50:0/2936789572 >> 172.21.15.50:0/2936789572 conn(0x55632b19fb30 msgr2=0x55632b19ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.967+0000 7fea57991740 1 -- 172.21.15.50:0/2936789572 shutdown_connections 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.967+0000 7fea57991740 1 -- 172.21.15.50:0/2936789572 wait complete. 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea57991740 1 -- start start 2024-04-23T18:28:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea57991740 1 Processor -- start 2024-04-23T18:28:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea57991740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1a3620 0x55632b1ae150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea57991740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea57991740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1ae800 0x55632b1ab9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea57991740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55632b1b13d0 con 0x55632b1a3f40 2024-04-23T18:28:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea57991740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55632b0a30b0 con 0x55632b1a3620 2024-04-23T18:28:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea57991740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55632b1b1550 con 0x55632b1ae800 2024-04-23T18:28:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea57991740 10 client.0 _ll_register_callbacks cb 0x55632b090d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea4dd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea4dd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1ab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38900/0 (socket says 172.21.15.50:38900) 2024-04-23T18:28:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea4dd9b640 1 -- 172.21.15.50:0/936289774 learned_addr learned my addr 172.21.15.50:0/936289774 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea4ed9d640 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1ae800 0x55632b1ab9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.968+0000 7fea4e59c640 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1a3620 0x55632b1ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea4dd9b640 1 -- 172.21.15.50:0/936289774 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1ae800 msgr2=0x55632b1ab9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea4dd9b640 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1ae800 0x55632b1ab9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea4dd9b640 1 -- 172.21.15.50:0/936289774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1a3620 msgr2=0x55632b1ae150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea4dd9b640 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1a3620 0x55632b1ae150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea4dd9b640 1 -- 172.21.15.50:0/936289774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55632aee7620 con 0x55632b1a3f40 2024-04-23T18:28:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea4dd9b640 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1ab440 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fea44009610 tx=0x7fea44017400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52881c0801b18d91 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:28:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea35ffb640 1 -- 172.21.15.50:0/936289774 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4409c370 con 0x55632b1a3f40 2024-04-23T18:28:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea35ffb640 1 -- 172.21.15.50:0/936289774 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7fea440989e0 con 0x55632b1a3f40 2024-04-23T18:28:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea57991740 1 -- 172.21.15.50:0/936289774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea400316a0 con 0x55632b1a3f40 2024-04-23T18:28:31.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea35ffb640 1 -- 172.21.15.50:0/936289774 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fea440a0720 con 0x55632b1a3f40 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea57991740 10 client.16728 fetch_fsmap learned FSMap version 41 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea57991740 1 -- 172.21.15.50:0/936289774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55632b1bd3b0 con 0x55632b1a3f40 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.969+0000 7fea35ffb640 1 -- 172.21.15.50:0/936289774 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea400316a0 con 0x55632b1a3f40 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.970+0000 7fea35ffb640 1 -- 172.21.15.50:0/936289774 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7fea440041c0 con 0x55632b1a3f40 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.970+0000 7fea57991740 10 client.16728 fetch_fsmap finished waiting for FSMap version 41 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.970+0000 7fea57991740 10 client.16728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.970+0000 7fea57991740 20 client.16728 populate_metadata read hostname 'smithi050' 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.970+0000 7fea57991740 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55632b1be8e0 0x55632b1c0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.970+0000 7fea57991740 4 client.16728 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.970+0000 7fea57991740 1 -- 172.21.15.50:0/936289774 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55632b116390 con 0x55632b1be8e0 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.970+0000 7fea4e59c640 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55632b1be8e0 0x55632b1c0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.970+0000 7fea4e59c640 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55632b1be8e0 0x55632b1c0d80 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6bb06bde685e523 server_cookie=d8fcdeaf1f0f9c4c in_seq=0 out_seq=0 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.970+0000 7fea35ffb640 10 client.16728 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:31.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea35ffb640 1 -- 172.21.15.50:0/936289774 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55632b116390 con 0x55632b1be8e0 2024-04-23T18:28:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea35ffb640 10 client.16728 handle_command_reply: tid=0 2024-04-23T18:28:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea57991740 1 client.16728 shutdown 2024-04-23T18:28:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea57991740 1 -- 172.21.15.50:0/936289774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 msgr2=0x55632b1ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea57991740 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1ab440 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fea44009610 tx=0x7fea44017400 comp rx=0 tx=0).stop 2024-04-23T18:28:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea57991740 1 -- 172.21.15.50:0/936289774 shutdown_connections 2024-04-23T18:28:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea57991740 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55632b1be8e0 0x55632b1c0d80 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea57991740 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1ae800 0x55632b1ab9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea35ffb640 0 client.16728 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea57991740 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1ab440 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea57991740 1 --2- 172.21.15.50:0/936289774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1a3620 0x55632b1ae150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea57991740 1 -- 172.21.15.50:0/936289774 >> 172.21.15.50:0/936289774 conn(0x55632b19fb30 msgr2=0x55632b1a5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:31.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea57991740 1 -- 172.21.15.50:0/936289774 shutdown_connections 2024-04-23T18:28:31.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.971+0000 7fea57991740 1 -- 172.21.15.50:0/936289774 wait complete. 2024-04-23T18:28:31.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.972+0000 7fea57991740 20 client.16728 trim_cache size 0 max 16384 2024-04-23T18:28:31.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.986+0000 7fea57991740 1 Processor -- start 2024-04-23T18:28:31.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea57991740 1 -- start start 2024-04-23T18:28:31.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea57991740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1ba290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea57991740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1a3f40 0x55632b1adc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea57991740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1ae800 0x55632b1c6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea57991740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fea501b2670 con 0x55632b1ae800 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea57991740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fea501afe80 con 0x55632b1a3f40 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea57991740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fea501adcf0 con 0x55632b1a3620 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea4ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1ae800 0x55632b1c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea4ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1ae800 0x55632b1c6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38904/0 (socket says 172.21.15.50:38904) 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea4ed9d640 1 -- 172.21.15.50:0/246189339 learned_addr learned my addr 172.21.15.50:0/246189339 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea4dd9b640 1 --2- 172.21.15.50:0/246189339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1a3f40 0x55632b1adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea4e59c640 1 --2- 172.21.15.50:0/246189339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1ba290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea4ed9d640 1 -- 172.21.15.50:0/246189339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 msgr2=0x55632b1ba290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea4ed9d640 1 --2- 172.21.15.50:0/246189339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1ba290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea4ed9d640 1 -- 172.21.15.50:0/246189339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1a3f40 msgr2=0x55632b1adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea4ed9d640 1 --2- 172.21.15.50:0/246189339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1a3f40 0x55632b1adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea4ed9d640 1 -- 172.21.15.50:0/246189339 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55632b1ac2c0 con 0x55632b1ae800 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea4dd9b640 1 --2- 172.21.15.50:0/246189339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1a3f40 0x55632b1adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.987+0000 7fea4ed9d640 1 --2- 172.21.15.50:0/246189339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1ae800 0x55632b1c6560 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fea48011cc0 tx=0x7fea4800a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f735a771ea358caa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea37fff640 1 -- 172.21.15.50:0/246189339 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48004b20 con 0x55632b1ae800 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea37fff640 1 -- 172.21.15.50:0/246189339 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea48004cc0 con 0x55632b1ae800 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea37fff640 1 -- 172.21.15.50:0/246189339 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48021430 con 0x55632b1ae800 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea57991740 1 -- 172.21.15.50:0/246189339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1ae800 msgr2=0x55632b1c6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea57991740 1 --2- 172.21.15.50:0/246189339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1ae800 0x55632b1c6560 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fea48011cc0 tx=0x7fea4800a890 comp rx=0 tx=0).stop 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea57991740 1 -- 172.21.15.50:0/246189339 shutdown_connections 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea57991740 1 --2- 172.21.15.50:0/246189339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1ae800 0x55632b1c6560 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea57991740 1 --2- 172.21.15.50:0/246189339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1a3f40 0x55632b1adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea57991740 1 --2- 172.21.15.50:0/246189339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1ba290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea57991740 1 -- 172.21.15.50:0/246189339 >> 172.21.15.50:0/246189339 conn(0x55632b19fb30 msgr2=0x55632b1a6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea57991740 1 -- 172.21.15.50:0/246189339 shutdown_connections 2024-04-23T18:28:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea57991740 1 -- 172.21.15.50:0/246189339 wait complete. 2024-04-23T18:28:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.988+0000 7fea57991740 1 -- start start 2024-04-23T18:28:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea57991740 1 Processor -- start 2024-04-23T18:28:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea57991740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea57991740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea57991740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1ae800 0x55632b1b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea57991740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fea501ae340 con 0x55632b1a3f40 2024-04-23T18:28:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea57991740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fea501af7b0 con 0x55632b1ae800 2024-04-23T18:28:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea57991740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fea501aea10 con 0x55632b1a3620 2024-04-23T18:28:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea57991740 10 client.0 _ll_register_callbacks cb 0x7fea4409beb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea4dd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea4dd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1b29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38912/0 (socket says 172.21.15.50:38912) 2024-04-23T18:28:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea4dd9b640 1 -- 172.21.15.50:0/3632867249 learned_addr learned my addr 172.21.15.50:0/3632867249 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea4ed9d640 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1ae800 0x55632b1b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea4dd9b640 1 -- 172.21.15.50:0/3632867249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 msgr2=0x55632b1b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea4e59c640 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea4dd9b640 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1b24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea4dd9b640 1 -- 172.21.15.50:0/3632867249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1ae800 msgr2=0x55632b1b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea4dd9b640 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1ae800 0x55632b1b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea4dd9b640 1 -- 172.21.15.50:0/3632867249 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55632b224070 con 0x55632b1a3f40 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea4e59c640 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1b24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea4dd9b640 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1b29e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fea4409c300 tx=0x7fea44005ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48536c51188c1333 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.989+0000 7fea35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea35ffb640 1 -- 172.21.15.50:0/3632867249 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea440a4fe0 con 0x55632b1a3f40 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea35ffb640 1 -- 172.21.15.50:0/3632867249 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7fea440cdda0 con 0x55632b1a3f40 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea57991740 1 -- 172.21.15.50:0/3632867249 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea48004cc0 con 0x55632b1a3f40 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea35ffb640 1 -- 172.21.15.50:0/3632867249 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fea4409c370 con 0x55632b1a3f40 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea57991740 10 client.16734 fetch_fsmap learned FSMap version 41 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea57991740 1 -- 172.21.15.50:0/3632867249 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55632b225380 con 0x55632b1a3f40 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea35ffb640 1 -- 172.21.15.50:0/3632867249 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea48004cc0 con 0x55632b1a3f40 2024-04-23T18:28:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea35ffb640 1 -- 172.21.15.50:0/3632867249 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7fea44004230 con 0x55632b1a3f40 2024-04-23T18:28:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea57991740 10 client.16734 fetch_fsmap finished waiting for FSMap version 41 2024-04-23T18:28:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea57991740 10 client.16734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea57991740 20 client.16734 populate_metadata read hostname 'smithi050' 2024-04-23T18:28:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea57991740 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55632b225940 0x55632b227d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea57991740 4 client.16734 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:28:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea57991740 1 -- 172.21.15.50:0/3632867249 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55632b110200 con 0x55632b225940 2024-04-23T18:28:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.990+0000 7fea4e59c640 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55632b225940 0x55632b227d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.991+0000 7fea35ffb640 10 client.16734 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:31.991+0000 7fea4e59c640 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55632b225940 0x55632b227d60 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aec52c996ad579a server_cookie=9af6928e7847b330 in_seq=0 out_seq=0 2024-04-23T18:28:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.031+0000 7fea35ffb640 1 -- 172.21.15.50:0/3632867249 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55632b110200 con 0x55632b225940 2024-04-23T18:28:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.031+0000 7fea35ffb640 10 client.16734 handle_command_reply: tid=0 2024-04-23T18:28:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.031+0000 7fea57991740 1 client.16734 shutdown 2024-04-23T18:28:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.032+0000 7fea57991740 1 -- 172.21.15.50:0/3632867249 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 msgr2=0x55632b1b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.032+0000 7fea57991740 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1b29e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fea4409c300 tx=0x7fea44005ab0 comp rx=0 tx=0).stop 2024-04-23T18:28:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.032+0000 7fea57991740 1 -- 172.21.15.50:0/3632867249 shutdown_connections 2024-04-23T18:28:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.032+0000 7fea57991740 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55632b225940 0x55632b227d60 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.032+0000 7fea57991740 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55632b1ae800 0x55632b1b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.032+0000 7fea57991740 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55632b1a3f40 0x55632b1b29e0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.032+0000 7fea57991740 1 --2- 172.21.15.50:0/3632867249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55632b1a3620 0x55632b1b24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.032+0000 7fea57991740 1 -- 172.21.15.50:0/3632867249 >> 172.21.15.50:0/3632867249 conn(0x55632b19fb30 msgr2=0x55632b1b9280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.032+0000 7fea35ffb640 0 client.16734 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.032+0000 7fea57991740 1 -- 172.21.15.50:0/3632867249 shutdown_connections 2024-04-23T18:28:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.032+0000 7fea57991740 1 -- 172.21.15.50:0/3632867249 wait complete. 2024-04-23T18:28:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.032+0000 7fea57991740 20 client.16734 trim_cache size 0 max 16384 2024-04-23T18:28:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:28:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:28:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:28:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:28:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.033+0000 7fea54c7c640 1 -- 172.21.15.50:0/3022951446 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fea380a3df0 msgr2=0x7fea380a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.033+0000 7fea54c7c640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fea380a3df0 0x7fea380a62b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fea4800a780 tx=0x7fea4800a4d0 comp rx=0 tx=0).stop 2024-04-23T18:28:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.033+0000 7fea54c7c640 1 -- 172.21.15.50:0/3022951446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea50106170 msgr2=0x7fea50103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.033+0000 7fea54c7c640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea50106170 0x7fea50103070 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fea44005fd0 tx=0x7fea4400dde0 comp rx=0 tx=0).stop 2024-04-23T18:28:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.033+0000 7fea54c7c640 1 -- 172.21.15.50:0/3022951446 shutdown_connections 2024-04-23T18:28:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.033+0000 7fea54c7c640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fea380a3df0 0x7fea380a62b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.033+0000 7fea54c7c640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea501035b0 0x7fea501a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.033+0000 7fea54c7c640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea50106170 0x7fea50103070 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.033+0000 7fea54c7c640 1 --2- 172.21.15.50:0/3022951446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea501010f0 0x7fea50102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.033+0000 7fea54c7c640 1 -- 172.21.15.50:0/3022951446 >> 172.21.15.50:0/3022951446 conn(0x7fea50077fd0 msgr2=0x7fea500699b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.033+0000 7fea54c7c640 1 -- 172.21.15.50:0/3022951446 shutdown_connections 2024-04-23T18:28:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:32.033+0000 7fea54c7c640 1 -- 172.21.15.50:0/3022951446 wait complete. 2024-04-23T18:28:32.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 613262 5.33 MB/sec execute 426 sec latency 869.114 ms 2024-04-23T18:28:32.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:32 smithi177 ceph-mon[32308]: pgmap v314: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 302 KiB/s wr, 20 op/s 2024-04-23T18:28:32.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:32 smithi188 ceph-mon[32694]: pgmap v314: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 302 KiB/s wr, 20 op/s 2024-04-23T18:28:32.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:32 smithi050 ceph-mon[29187]: pgmap v314: 97 pgs: 97 active+clean; 197 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 302 KiB/s wr, 20 op/s 2024-04-23T18:28:33.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 614906 5.35 MB/sec execute 427 sec latency 13.690 ms 2024-04-23T18:28:34.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 616129 5.35 MB/sec execute 428 sec latency 58.594 ms 2024-04-23T18:28:34.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:34 smithi177 ceph-mon[32308]: pgmap v315: 97 pgs: 97 active+clean; 199 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 588 KiB/s wr, 49 op/s 2024-04-23T18:28:34.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:34 smithi188 ceph-mon[32694]: pgmap v315: 97 pgs: 97 active+clean; 199 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 588 KiB/s wr, 49 op/s 2024-04-23T18:28:34.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:34 smithi050 ceph-mon[29187]: pgmap v315: 97 pgs: 97 active+clean; 199 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 588 KiB/s wr, 49 op/s 2024-04-23T18:28:35.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 616880 5.34 MB/sec execute 429 sec latency 778.690 ms 2024-04-23T18:28:36.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 618603 5.36 MB/sec execute 430 sec latency 9.465 ms 2024-04-23T18:28:36.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:36 smithi177 ceph-mon[32308]: pgmap v316: 97 pgs: 97 active+clean; 199 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 547 KiB/s wr, 41 op/s 2024-04-23T18:28:36.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:36 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:36.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:36 smithi188 ceph-mon[32694]: pgmap v316: 97 pgs: 97 active+clean; 199 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 547 KiB/s wr, 41 op/s 2024-04-23T18:28:36.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:36 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:36.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:36 smithi050 ceph-mon[29187]: pgmap v316: 97 pgs: 97 active+clean; 199 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 547 KiB/s wr, 41 op/s 2024-04-23T18:28:36.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:36 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:37.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 619675 5.35 MB/sec execute 431 sec latency 388.571 ms 2024-04-23T18:28:37.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:37 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:37.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:37 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:37.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:37 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:38.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 619675 5.34 MB/sec execute 432 sec latency 1388.715 ms 2024-04-23T18:28:38.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:38 smithi177 ceph-mon[32308]: pgmap v317: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-04-23T18:28:38.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:38 smithi188 ceph-mon[32694]: pgmap v317: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-04-23T18:28:38.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:38 smithi050 ceph-mon[29187]: pgmap v317: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-04-23T18:28:39.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 620586 5.34 MB/sec execute 433 sec latency 1993.574 ms 2024-04-23T18:28:40.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 622317 5.35 MB/sec execute 434 sec latency 20.942 ms 2024-04-23T18:28:40.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:40 smithi177 ceph-mon[32308]: pgmap v318: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 905 KiB/s wr, 116 op/s 2024-04-23T18:28:40.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:40 smithi188 ceph-mon[32694]: pgmap v318: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 905 KiB/s wr, 116 op/s 2024-04-23T18:28:40.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:40 smithi050 ceph-mon[29187]: pgmap v318: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 905 KiB/s wr, 116 op/s 2024-04-23T18:28:41.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 623219 5.35 MB/sec execute 435 sec latency 297.882 ms 2024-04-23T18:28:42.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 623219 5.34 MB/sec execute 436 sec latency 1297.918 ms 2024-04-23T18:28:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:42 smithi177 ceph-mon[32308]: pgmap v319: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 905 KiB/s wr, 116 op/s 2024-04-23T18:28:42.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:42 smithi188 ceph-mon[32694]: pgmap v319: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 905 KiB/s wr, 116 op/s 2024-04-23T18:28:42.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:42 smithi050 ceph-mon[29187]: pgmap v319: 97 pgs: 97 active+clean; 203 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 905 KiB/s wr, 116 op/s 2024-04-23T18:28:43.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 623219 5.33 MB/sec execute 437 sec latency 2297.955 ms 2024-04-23T18:28:44.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 624122 5.32 MB/sec execute 438 sec latency 2901.984 ms 2024-04-23T18:28:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:44 smithi177 ceph-mon[32308]: pgmap v320: 97 pgs: 97 active+clean; 206 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-04-23T18:28:44.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:44 smithi188 ceph-mon[32694]: pgmap v320: 97 pgs: 97 active+clean; 206 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-04-23T18:28:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:44 smithi050 ceph-mon[29187]: pgmap v320: 97 pgs: 97 active+clean; 206 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-04-23T18:28:45.127 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.132+0000 7faec2dc6640 1 -- 172.21.15.50:0/686160632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc101aa0 msgr2=0x7faebc105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.132+0000 7faec2dc6640 1 --2- 172.21.15.50:0/686160632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc101aa0 0x7faebc105a90 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009ca0 tx=0x7faeb402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.132+0000 7faec2dc6640 1 -- 172.21.15.50:0/686160632 shutdown_connections 2024-04-23T18:28:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.132+0000 7faec2dc6640 1 --2- 172.21.15.50:0/686160632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc106170 0x7faebc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.132+0000 7faec2dc6640 1 --2- 172.21.15.50:0/686160632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc101aa0 0x7faebc105a90 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.132+0000 7faec2dc6640 1 --2- 172.21.15.50:0/686160632 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faebc1010f0 0x7faebc1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.132+0000 7faec2dc6640 1 -- 172.21.15.50:0/686160632 >> 172.21.15.50:0/686160632 conn(0x7faebc077fd0 msgr2=0x7faebc0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.132+0000 7faec2dc6640 1 -- 172.21.15.50:0/686160632 shutdown_connections 2024-04-23T18:28:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.132+0000 7faec2dc6640 1 -- 172.21.15.50:0/686160632 wait complete. 2024-04-23T18:28:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.133+0000 7faec2dc6640 1 Processor -- start 2024-04-23T18:28:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.133+0000 7faec2dc6640 1 -- start start 2024-04-23T18:28:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.133+0000 7faec2dc6640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faebc1010f0 0x7faebc1975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.133+0000 7faec2dc6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc106170 0x7faebc197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.133+0000 7faec2dc6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc19be70 0x7faebc19c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.133+0000 7faec2dc6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1110b0 con 0x7faebc106170 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.133+0000 7faec2dc6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faebc110f30 con 0x7faebc19be70 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.133+0000 7faec2dc6640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faebc111230 con 0x7faebc1010f0 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.133+0000 7faebbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc106170 0x7faebc197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faec1363640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc19be70 0x7faebc19c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faebbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc106170 0x7faebc197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55524/0 (socket says 172.21.15.50:55524) 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faec0b62640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faebc1010f0 0x7faebc1975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faebbfff640 1 -- 172.21.15.50:0/3525912639 learned_addr learned my addr 172.21.15.50:0/3525912639 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faebbfff640 1 -- 172.21.15.50:0/3525912639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faebc1010f0 msgr2=0x7faebc1975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faebbfff640 1 --2- 172.21.15.50:0/3525912639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faebc1010f0 0x7faebc1975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faebbfff640 1 -- 172.21.15.50:0/3525912639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc19be70 msgr2=0x7faebc19c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faebbfff640 1 --2- 172.21.15.50:0/3525912639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc19be70 0x7faebc19c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faebbfff640 1 -- 172.21.15.50:0/3525912639 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faebc19c960 con 0x7faebc106170 2024-04-23T18:28:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faebbfff640 1 --2- 172.21.15.50:0/3525912639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc106170 0x7faebc197af0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7faeb4033040 tx=0x7faeb40316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faeb9ffb640 1 -- 172.21.15.50:0/3525912639 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4003b70 con 0x7faebc106170 2024-04-23T18:28:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faec2dc6640 1 -- 172.21.15.50:0/3525912639 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faebc19cbf0 con 0x7faebc106170 2024-04-23T18:28:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faeb9ffb640 1 -- 172.21.15.50:0/3525912639 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeb4003d10 con 0x7faebc106170 2024-04-23T18:28:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faeb9ffb640 1 -- 172.21.15.50:0/3525912639 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb402fa80 con 0x7faebc106170 2024-04-23T18:28:45.130 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.134+0000 7faec2dc6640 1 -- 172.21.15.50:0/3525912639 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faebc1a4440 con 0x7faebc106170 2024-04-23T18:28:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.137+0000 7faec5bb8740 1 Processor -- start 2024-04-23T18:28:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faec5bb8740 1 -- start start 2024-04-23T18:28:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faeb9ffb640 1 -- 172.21.15.50:0/3525912639 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faeb402fc20 con 0x7faebc106170 2024-04-23T18:28:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faec5bb8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e13620 0x564488e13a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faec5bb8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e1e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faec5bb8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e1e800 0x564488e20bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faec5bb8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564488d78970 con 0x564488e1e800 2024-04-23T18:28:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faec5bb8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564488d154a0 con 0x564488e13f40 2024-04-23T18:28:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faec5bb8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564488d74930 con 0x564488e13620 2024-04-23T18:28:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faec1363640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e13620 0x564488e13a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faec1363640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e13620 0x564488e13a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:58274/0 (socket says 172.21.15.50:58274) 2024-04-23T18:28:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faec1363640 1 -- 172.21.15.50:0/2937744253 learned_addr learned my addr 172.21.15.50:0/2937744253 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faebbfff640 1 --2- 172.21.15.50:0/2937744253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e1e800 0x564488e20bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faec0b62640 1 --2- 172.21.15.50:0/2937744253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e1e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.138+0000 7faeb9ffb640 1 --2- 172.21.15.50:0/3525912639 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faea40a3dc0 0x7faea40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.139+0000 7faebbfff640 1 -- 172.21.15.50:0/2937744253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e13620 msgr2=0x564488e13a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.139+0000 7faebbfff640 1 --2- 172.21.15.50:0/2937744253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e13620 0x564488e13a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.139+0000 7faebbfff640 1 -- 172.21.15.50:0/2937744253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 msgr2=0x564488e1e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.139+0000 7faebbfff640 1 --2- 172.21.15.50:0/2937744253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e1e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.139+0000 7faec1363640 1 --2- 172.21.15.50:0/3525912639 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faea40a3dc0 0x7faea40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.139+0000 7faebbfff640 1 -- 172.21.15.50:0/2937744253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564488a8b210 con 0x564488e1e800 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.139+0000 7faebbfff640 1 --2- 172.21.15.50:0/2937744253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e1e800 0x564488e20bf0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7faeb40ef2a0 tx=0x7faeb40bd630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c40e724c122e74 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.139+0000 7faeb9ffb640 1 -- 172.21.15.50:0/3525912639 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7faeb40f1a50 con 0x7faebc106170 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.139+0000 7faec1363640 1 --2- 172.21.15.50:0/3525912639 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faea40a3dc0 0x7faea40a6280 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faeac0045e0 tx=0x7faeac0046c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.139+0000 7faea2ffd640 1 -- 172.21.15.50:0/2937744253 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb40098d0 con 0x564488e1e800 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.139+0000 7faea2ffd640 1 -- 172.21.15.50:0/2937744253 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeb4036030 con 0x564488e1e800 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.139+0000 7faea2ffd640 1 -- 172.21.15.50:0/2937744253 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb40be1f0 con 0x564488e1e800 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.140+0000 7faec5bb8740 1 -- 172.21.15.50:0/2937744253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e1e800 msgr2=0x564488e20bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.140+0000 7faec5bb8740 1 --2- 172.21.15.50:0/2937744253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e1e800 0x564488e20bf0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7faeb40ef2a0 tx=0x7faeb40bd630 comp rx=0 tx=0).stop 2024-04-23T18:28:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.140+0000 7faec5bb8740 1 -- 172.21.15.50:0/2937744253 shutdown_connections 2024-04-23T18:28:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.140+0000 7faec5bb8740 1 --2- 172.21.15.50:0/2937744253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e1e800 0x564488e20bf0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.140+0000 7faec5bb8740 1 --2- 172.21.15.50:0/2937744253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e1e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.140+0000 7faec5bb8740 1 --2- 172.21.15.50:0/2937744253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e13620 0x564488e13a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.140+0000 7faec5bb8740 1 -- 172.21.15.50:0/2937744253 >> 172.21.15.50:0/2937744253 conn(0x564488e0fb30 msgr2=0x564488e0ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.140+0000 7faec5bb8740 1 -- 172.21.15.50:0/2937744253 shutdown_connections 2024-04-23T18:28:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.140+0000 7faec5bb8740 1 -- 172.21.15.50:0/2937744253 wait complete. 2024-04-23T18:28:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.140+0000 7faec5bb8740 1 -- start start 2024-04-23T18:28:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec5bb8740 1 Processor -- start 2024-04-23T18:28:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec5bb8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13620 0x564488e1e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec5bb8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e1b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec5bb8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e1e800 0x564488e1b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec5bb8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564488d78970 con 0x564488e13620 2024-04-23T18:28:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec5bb8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564488d130b0 con 0x564488e13f40 2024-04-23T18:28:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec5bb8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564488e211f0 con 0x564488e1e800 2024-04-23T18:28:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec5bb8740 10 client.0 _ll_register_callbacks cb 0x564488d00d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec0b62640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13620 0x564488e1e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec1363640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e1e800 0x564488e1b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faebbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e1b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec0b62640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13620 0x564488e1e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55540/0 (socket says 172.21.15.50:55540) 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec0b62640 1 -- 172.21.15.50:0/1605047197 learned_addr learned my addr 172.21.15.50:0/1605047197 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec1363640 1 -- 172.21.15.50:0/1605047197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 msgr2=0x564488e1b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec1363640 1 --2- 172.21.15.50:0/1605047197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e1b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec1363640 1 -- 172.21.15.50:0/1605047197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13620 msgr2=0x564488e1e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec1363640 1 --2- 172.21.15.50:0/1605047197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13620 0x564488e1e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.141+0000 7faec1363640 1 -- 172.21.15.50:0/1605047197 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564488b57620 con 0x564488e1e800 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.142+0000 7faec1363640 1 --2- 172.21.15.50:0/1605047197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e1e800 0x564488e1b940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7faeac01f6a0 tx=0x7faeac004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b817eb84f471ed78 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.142+0000 7fae9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.142+0000 7fae9bfff640 1 -- 172.21.15.50:0/1605047197 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac01d450 con 0x564488e1e800 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.142+0000 7faec5bb8740 1 -- 172.21.15.50:0/1605047197 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faeb4036030 con 0x564488e1e800 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.142+0000 7fae9bfff640 1 -- 172.21.15.50:0/1605047197 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7faeac01d5f0 con 0x564488e1e800 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.142+0000 7fae9bfff640 1 -- 172.21.15.50:0/1605047197 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7faeac01d7f0 con 0x564488e1e800 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.142+0000 7faec5bb8740 10 client.26353 fetch_fsmap learned FSMap version 43 2024-04-23T18:28:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.142+0000 7faec5bb8740 1 -- 172.21.15.50:0/1605047197 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x564488e2cf30 con 0x564488e1e800 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.142+0000 7fae9bfff640 1 -- 172.21.15.50:0/1605047197 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeb4036030 con 0x564488e1e800 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.142+0000 7fae9bfff640 1 -- 172.21.15.50:0/1605047197 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7faeac01cac0 con 0x564488e1e800 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.143+0000 7faec5bb8740 10 client.26353 fetch_fsmap finished waiting for FSMap version 43 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.143+0000 7faec5bb8740 10 client.26353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.143+0000 7faec5bb8740 20 client.26353 populate_metadata read hostname 'smithi050' 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.143+0000 7faec5bb8740 1 --2- 172.21.15.50:0/1605047197 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564488e2e3f0 0x564488e307e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.143+0000 7faec5bb8740 4 client.26353 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.143+0000 7faec5bb8740 1 -- 172.21.15.50:0/1605047197 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564488d86390 con 0x564488e2e3f0 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.143+0000 7faec0b62640 1 --2- 172.21.15.50:0/1605047197 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564488e2e3f0 0x564488e307e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.143+0000 7faec0b62640 1 --2- 172.21.15.50:0/1605047197 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564488e2e3f0 0x564488e307e0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cbbae34d865d972 server_cookie=557cf09d253f42ab in_seq=0 out_seq=0 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.143+0000 7fae9bfff640 10 client.26353 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7fae9bfff640 1 -- 172.21.15.50:0/1605047197 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564488d86390 con 0x564488e2e3f0 2024-04-23T18:28:45.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7fae9bfff640 10 client.26353 handle_command_reply: tid=0 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7faec5bb8740 1 client.26353 shutdown 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7faec5bb8740 1 -- 172.21.15.50:0/1605047197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e1e800 msgr2=0x564488e1b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7faec5bb8740 1 --2- 172.21.15.50:0/1605047197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e1e800 0x564488e1b940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7faeac01f6a0 tx=0x7faeac004730 comp rx=0 tx=0).stop 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7faec5bb8740 1 -- 172.21.15.50:0/1605047197 shutdown_connections 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7faec5bb8740 1 --2- 172.21.15.50:0/1605047197 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564488e2e3f0 0x564488e307e0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7faec5bb8740 1 --2- 172.21.15.50:0/1605047197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e1e800 0x564488e1b940 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7faec5bb8740 1 --2- 172.21.15.50:0/1605047197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e1b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7faec5bb8740 1 --2- 172.21.15.50:0/1605047197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13620 0x564488e1e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7fae9bfff640 0 client.26353 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7faec5bb8740 1 -- 172.21.15.50:0/1605047197 >> 172.21.15.50:0/1605047197 conn(0x564488e0fb30 msgr2=0x564488e158b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7faec5bb8740 1 -- 172.21.15.50:0/1605047197 shutdown_connections 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7faec5bb8740 1 -- 172.21.15.50:0/1605047197 wait complete. 2024-04-23T18:28:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.144+0000 7faec5bb8740 20 client.26353 trim_cache size 0 max 16384 2024-04-23T18:28:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec5bb8740 1 Processor -- start 2024-04-23T18:28:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec5bb8740 1 -- start start 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec5bb8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e0fb30 0x564488e29e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec5bb8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13620 0x564488e360d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec5bb8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13f40 0x564488e36610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec5bb8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1ae310 con 0x564488e13f40 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec5bb8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1abb50 con 0x564488e13620 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec5bb8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1a99c0 con 0x564488e0fb30 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec1363640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13f40 0x564488e36610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec1363640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13f40 0x564488e36610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55552/0 (socket says 172.21.15.50:55552) 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec1363640 1 -- 172.21.15.50:0/3542045275 learned_addr learned my addr 172.21.15.50:0/3542045275 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec0b62640 1 --2- 172.21.15.50:0/3542045275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e0fb30 0x564488e29e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faebbfff640 1 --2- 172.21.15.50:0/3542045275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13620 0x564488e360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec1363640 1 -- 172.21.15.50:0/3542045275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e0fb30 msgr2=0x564488e29e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec1363640 1 --2- 172.21.15.50:0/3542045275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e0fb30 0x564488e29e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec1363640 1 -- 172.21.15.50:0/3542045275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13620 msgr2=0x564488e360d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec1363640 1 --2- 172.21.15.50:0/3542045275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13620 0x564488e360d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.159+0000 7faec1363640 1 -- 172.21.15.50:0/3542045275 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564488e30e40 con 0x564488e13f40 2024-04-23T18:28:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.160+0000 7faec1363640 1 --2- 172.21.15.50:0/3542045275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13f40 0x564488e36610 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7faeac01cf60 tx=0x7faeac02a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac7c44ef7369cbd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.160+0000 7faea27fc640 1 -- 172.21.15.50:0/3542045275 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac037020 con 0x564488e13f40 2024-04-23T18:28:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.160+0000 7faea27fc640 1 -- 172.21.15.50:0/3542045275 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeac01de40 con 0x564488e13f40 2024-04-23T18:28:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.160+0000 7faea27fc640 1 -- 172.21.15.50:0/3542045275 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac02abd0 con 0x564488e13f40 2024-04-23T18:28:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.160+0000 7faec5bb8740 1 -- 172.21.15.50:0/3542045275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13f40 msgr2=0x564488e36610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.160+0000 7faec5bb8740 1 --2- 172.21.15.50:0/3542045275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13f40 0x564488e36610 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7faeac01cf60 tx=0x7faeac02a760 comp rx=0 tx=0).stop 2024-04-23T18:28:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.160+0000 7faec5bb8740 1 -- 172.21.15.50:0/3542045275 shutdown_connections 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.160+0000 7faec5bb8740 1 --2- 172.21.15.50:0/3542045275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13f40 0x564488e36610 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.160+0000 7faec5bb8740 1 --2- 172.21.15.50:0/3542045275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13620 0x564488e360d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.160+0000 7faec5bb8740 1 --2- 172.21.15.50:0/3542045275 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e0fb30 0x564488e29e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.160+0000 7faec5bb8740 1 -- 172.21.15.50:0/3542045275 >> 172.21.15.50:0/3542045275 conn(0x564488e1baa0 msgr2=0x564488e0ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.160+0000 7faec5bb8740 1 -- 172.21.15.50:0/3542045275 shutdown_connections 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec5bb8740 1 -- 172.21.15.50:0/3542045275 wait complete. 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec5bb8740 1 -- start start 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec5bb8740 1 Processor -- start 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec5bb8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e0fb30 0x564488e24130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec5bb8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13620 0x564488e24670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec5bb8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e93950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec5bb8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1aa010 con 0x564488e13620 2024-04-23T18:28:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec5bb8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1ab480 con 0x564488e13f40 2024-04-23T18:28:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec5bb8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1aa6e0 con 0x564488e0fb30 2024-04-23T18:28:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec5bb8740 10 client.0 _ll_register_callbacks cb 0x7faeac004060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec0b62640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e0fb30 0x564488e24130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faebbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13620 0x564488e24670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec0b62640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e0fb30 0x564488e24130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:58292/0 (socket says 172.21.15.50:58292) 2024-04-23T18:28:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec0b62640 1 -- 172.21.15.50:0/383339164 learned_addr learned my addr 172.21.15.50:0/383339164 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.161+0000 7faec1363640 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e93950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7faec0b62640 1 -- 172.21.15.50:0/383339164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 msgr2=0x564488e93950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7faec0b62640 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e93950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7faec0b62640 1 -- 172.21.15.50:0/383339164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13620 msgr2=0x564488e24670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7faec0b62640 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13620 0x564488e24670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7faec0b62640 1 -- 172.21.15.50:0/383339164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564488e93e90 con 0x564488e0fb30 2024-04-23T18:28:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7faebbfff640 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13620 0x564488e24670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7faec1363640 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e93950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7faec0b62640 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e0fb30 0x564488e24130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x564488e31040 tx=0x7faeb00206e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fffd1eef86039a18 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7fae9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:28:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7fae9bfff640 1 -- 172.21.15.50:0/383339164 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb0010c90 con 0x564488e0fb30 2024-04-23T18:28:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7faec5bb8740 1 -- 172.21.15.50:0/383339164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564488d80200 con 0x564488e0fb30 2024-04-23T18:28:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7fae9bfff640 1 -- 172.21.15.50:0/383339164 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7faeb0005020 con 0x564488e0fb30 2024-04-23T18:28:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7fae9bfff640 1 -- 172.21.15.50:0/383339164 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7faeb0010e30 con 0x564488e0fb30 2024-04-23T18:28:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7faec5bb8740 10 client.26359 fetch_fsmap learned FSMap version 43 2024-04-23T18:28:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7faec5bb8740 1 -- 172.21.15.50:0/383339164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x564488e95080 con 0x564488e0fb30 2024-04-23T18:28:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.162+0000 7fae9bfff640 1 -- 172.21.15.50:0/383339164 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564488d80200 con 0x564488e0fb30 2024-04-23T18:28:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.163+0000 7fae9bfff640 1 -- 172.21.15.50:0/383339164 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7faeb0028410 con 0x564488e0fb30 2024-04-23T18:28:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.163+0000 7faec5bb8740 10 client.26359 fetch_fsmap finished waiting for FSMap version 43 2024-04-23T18:28:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.163+0000 7faec5bb8740 10 client.26359 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.163+0000 7faec5bb8740 20 client.26359 populate_metadata read hostname 'smithi050' 2024-04-23T18:28:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.163+0000 7faec5bb8740 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564488e1e800 0x564488e978d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.163+0000 7faec5bb8740 4 client.26359 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:28:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.163+0000 7faec5bb8740 1 -- 172.21.15.50:0/383339164 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564488cfc5f0 con 0x564488e1e800 2024-04-23T18:28:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.163+0000 7faebbfff640 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564488e1e800 0x564488e978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.163+0000 7fae9bfff640 10 client.26359 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.163+0000 7faebbfff640 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564488e1e800 0x564488e978d0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13f0d87282617eb4 server_cookie=cad42b49337e3ea4 in_seq=0 out_seq=0 2024-04-23T18:28:45.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7fae9bfff640 1 -- 172.21.15.50:0/383339164 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564488cfc5f0 con 0x564488e1e800 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7fae9bfff640 10 client.26359 handle_command_reply: tid=0 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7faec5bb8740 1 client.26359 shutdown 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7faec5bb8740 1 -- 172.21.15.50:0/383339164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e0fb30 msgr2=0x564488e24130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7faec5bb8740 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e0fb30 0x564488e24130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x564488e31040 tx=0x7faeb00206e0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7faec5bb8740 1 -- 172.21.15.50:0/383339164 shutdown_connections 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7faec5bb8740 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564488e1e800 0x564488e978d0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7faec5bb8740 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564488e13f40 0x564488e93950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7faec5bb8740 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564488e13620 0x564488e24670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7fae9bfff640 0 client.26359 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7faec5bb8740 1 --2- 172.21.15.50:0/383339164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564488e0fb30 0x564488e24130 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7faec5bb8740 1 -- 172.21.15.50:0/383339164 >> 172.21.15.50:0/383339164 conn(0x564488e1baa0 msgr2=0x564488e28e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7faec5bb8740 1 -- 172.21.15.50:0/383339164 shutdown_connections 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7faec5bb8740 1 -- 172.21.15.50:0/383339164 wait complete. 2024-04-23T18:28:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.207+0000 7faec5bb8740 20 client.26359 trim_cache size 0 max 16384 2024-04-23T18:28:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:28:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:28:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:28:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:28:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.208+0000 7faec2dc6640 1 -- 172.21.15.50:0/3525912639 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faea40a3dc0 msgr2=0x7faea40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.208+0000 7faec2dc6640 1 --2- 172.21.15.50:0/3525912639 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faea40a3dc0 0x7faea40a6280 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faeac0045e0 tx=0x7faeac0046c0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.208+0000 7faec2dc6640 1 -- 172.21.15.50:0/3525912639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc106170 msgr2=0x7faebc197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.208+0000 7faec2dc6640 1 --2- 172.21.15.50:0/3525912639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc106170 0x7faebc197af0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7faeb4033040 tx=0x7faeb40316f0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.208+0000 7faec2dc6640 1 -- 172.21.15.50:0/3525912639 shutdown_connections 2024-04-23T18:28:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.208+0000 7faec2dc6640 1 --2- 172.21.15.50:0/3525912639 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faea40a3dc0 0x7faea40a6280 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.208+0000 7faec2dc6640 1 --2- 172.21.15.50:0/3525912639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faebc19be70 0x7faebc19c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.208+0000 7faec2dc6640 1 --2- 172.21.15.50:0/3525912639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faebc106170 0x7faebc197af0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.208+0000 7faec2dc6640 1 --2- 172.21.15.50:0/3525912639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faebc1010f0 0x7faebc1975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.208+0000 7faec2dc6640 1 -- 172.21.15.50:0/3525912639 >> 172.21.15.50:0/3525912639 conn(0x7faebc077fd0 msgr2=0x7faebc0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.208+0000 7faec2dc6640 1 -- 172.21.15.50:0/3525912639 shutdown_connections 2024-04-23T18:28:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:45.208+0000 7faec2dc6640 1 -- 172.21.15.50:0/3525912639 wait complete. 2024-04-23T18:28:45.227 INFO:tasks.workunit.client.0.smithi050.stdout: 1 625673 5.33 MB/sec execute 439 sec latency 19.434 ms 2024-04-23T18:28:45.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:45 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:28:45.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:45 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:28:45.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:45 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:28:46.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 626768 5.33 MB/sec execute 440 sec latency 58.007 ms 2024-04-23T18:28:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:46 smithi177 ceph-mon[32308]: pgmap v321: 97 pgs: 97 active+clean; 206 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 960 KiB/s wr, 132 op/s 2024-04-23T18:28:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:46 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:46 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:46 smithi188 ceph-mon[32694]: pgmap v321: 97 pgs: 97 active+clean; 206 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 960 KiB/s wr, 132 op/s 2024-04-23T18:28:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:46 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:46 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:46.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:46 smithi050 ceph-mon[29187]: pgmap v321: 97 pgs: 97 active+clean; 206 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 960 KiB/s wr, 132 op/s 2024-04-23T18:28:46.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:46 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:46.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:46 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 626768 5.32 MB/sec execute 441 sec latency 1051.916 ms 2024-04-23T18:28:47.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:47.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:28:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:28:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:47.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:28:47.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:28:47.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:47.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:28:47.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:28:47.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:28:47.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:28:48.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 626768 5.31 MB/sec execute 442 sec latency 2051.948 ms 2024-04-23T18:28:48.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:48 smithi177 ceph-mon[32308]: pgmap v322: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 197 op/s 2024-04-23T18:28:48.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:48 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-04-23T18:28:48.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:48 smithi177 ceph-mon[32308]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:28:48.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:48 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-23T18:28:48.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:48 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-23T18:28:48.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:48 smithi177 ceph-mon[32308]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-23T18:28:48.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:48 smithi177 ceph-mon[32308]: Unable to set osd_memory_target on smithi188 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T18:28:48.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:48 smithi188 ceph-mon[32694]: pgmap v322: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 197 op/s 2024-04-23T18:28:48.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:48 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-04-23T18:28:48.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:48 smithi188 ceph-mon[32694]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:28:48.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:48 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-23T18:28:48.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:48 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-23T18:28:48.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:48 smithi188 ceph-mon[32694]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-23T18:28:48.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:48 smithi188 ceph-mon[32694]: Unable to set osd_memory_target on smithi188 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T18:28:48.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:48 smithi050 ceph-mon[29187]: pgmap v322: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 197 op/s 2024-04-23T18:28:48.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:48 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-04-23T18:28:48.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:48 smithi050 ceph-mon[29187]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:28:48.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:48 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-23T18:28:48.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:48 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-23T18:28:48.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:48 smithi050 ceph-mon[29187]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-23T18:28:48.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:48 smithi050 ceph-mon[29187]: Unable to set osd_memory_target on smithi188 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T18:28:49.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 627658 5.30 MB/sec execute 443 sec latency 2655.704 ms 2024-04-23T18:28:50.231 INFO:tasks.workunit.client.0.smithi050.stdout: 1 629338 5.32 MB/sec execute 444 sec latency 20.745 ms 2024-04-23T18:28:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:50 smithi177 ceph-mon[32308]: pgmap v323: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 674 KiB/s wr, 109 op/s 2024-04-23T18:28:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:50 smithi188 ceph-mon[32694]: pgmap v323: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 674 KiB/s wr, 109 op/s 2024-04-23T18:28:50.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:50 smithi050 ceph-mon[29187]: pgmap v323: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 674 KiB/s wr, 109 op/s 2024-04-23T18:28:51.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 630309 5.32 MB/sec execute 445 sec latency 254.447 ms 2024-04-23T18:28:52.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 630309 5.30 MB/sec execute 446 sec latency 1254.562 ms 2024-04-23T18:28:52.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:52 smithi050 ceph-mon[29187]: pgmap v324: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 674 KiB/s wr, 109 op/s 2024-04-23T18:28:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:52 smithi177 ceph-mon[32308]: pgmap v324: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 674 KiB/s wr, 109 op/s 2024-04-23T18:28:52.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:52 smithi188 ceph-mon[32694]: pgmap v324: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 674 KiB/s wr, 109 op/s 2024-04-23T18:28:53.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 630309 5.29 MB/sec execute 447 sec latency 2254.616 ms 2024-04-23T18:28:54.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 631117 5.29 MB/sec execute 448 sec latency 2868.058 ms 2024-04-23T18:28:54.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:54 smithi177 ceph-mon[32308]: pgmap v325: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 985 KiB/s wr, 151 op/s 2024-04-23T18:28:54.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:54 smithi188 ceph-mon[32694]: pgmap v325: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 985 KiB/s wr, 151 op/s 2024-04-23T18:28:54.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:54 smithi050 ceph-mon[29187]: pgmap v325: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 985 KiB/s wr, 151 op/s 2024-04-23T18:28:55.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 632766 5.30 MB/sec execute 449 sec latency 9.925 ms 2024-04-23T18:28:56.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 634551 5.30 MB/sec execute 450 sec latency 19.232 ms 2024-04-23T18:28:56.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:56 smithi177 ceph-mon[32308]: pgmap v326: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 645 KiB/s wr, 106 op/s 2024-04-23T18:28:56.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:56 smithi188 ceph-mon[32694]: pgmap v326: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 645 KiB/s wr, 106 op/s 2024-04-23T18:28:56.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:56 smithi050 ceph-mon[29187]: pgmap v326: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 645 KiB/s wr, 106 op/s 2024-04-23T18:28:57.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 636246 5.32 MB/sec execute 451 sec latency 17.250 ms 2024-04-23T18:28:58.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 637677 5.32 MB/sec execute 452 sec latency 52.184 ms 2024-04-23T18:28:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.305+0000 7fb2e7031640 1 -- 172.21.15.50:0/2620895211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2e0101aa0 msgr2=0x7fb2e0105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.305+0000 7fb2e7031640 1 --2- 172.21.15.50:0/2620895211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2e0101aa0 0x7fb2e0105a90 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009ca0 tx=0x7fb2d002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.305+0000 7fb2e7031640 1 -- 172.21.15.50:0/2620895211 shutdown_connections 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.305+0000 7fb2e7031640 1 --2- 172.21.15.50:0/2620895211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e0106170 0x7fb2e010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.305+0000 7fb2e7031640 1 --2- 172.21.15.50:0/2620895211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2e0101aa0 0x7fb2e0105a90 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.305+0000 7fb2e7031640 1 --2- 172.21.15.50:0/2620895211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2e01010f0 0x7fb2e01014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.305+0000 7fb2e7031640 1 -- 172.21.15.50:0/2620895211 >> 172.21.15.50:0/2620895211 conn(0x7fb2e0077fd0 msgr2=0x7fb2e00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.305+0000 7fb2e7031640 1 -- 172.21.15.50:0/2620895211 shutdown_connections 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.305+0000 7fb2e7031640 1 -- 172.21.15.50:0/2620895211 wait complete. 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.305+0000 7fb2e7031640 1 Processor -- start 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.305+0000 7fb2e7031640 1 -- start start 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2e7031640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e01010f0 0x7fb2e0102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2e7031640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2e0106170 0x7fb2e0103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2e7031640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2e01035b0 0x7fb2e01a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2e7031640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01110b0 con 0x7fb2e0106170 2024-04-23T18:28:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2e7031640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e0110f30 con 0x7fb2e01010f0 2024-04-23T18:28:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2e7031640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e0111230 con 0x7fb2e01035b0 2024-04-23T18:28:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2dffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2e0106170 0x7fb2e0103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2dffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2e0106170 0x7fb2e0103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36584/0 (socket says 172.21.15.50:36584) 2024-04-23T18:28:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2dffff640 1 -- 172.21.15.50:0/1650556114 learned_addr learned my addr 172.21.15.50:0/1650556114 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2e4dcd640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e01010f0 0x7fb2e0102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2e55ce640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2e01035b0 0x7fb2e01a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2dffff640 1 -- 172.21.15.50:0/1650556114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2e01035b0 msgr2=0x7fb2e01a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2dffff640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2e01035b0 0x7fb2e01a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2dffff640 1 -- 172.21.15.50:0/1650556114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e01010f0 msgr2=0x7fb2e0102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2dffff640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e01010f0 0x7fb2e0102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2dffff640 1 -- 172.21.15.50:0/1650556114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e01a6fc0 con 0x7fb2e0106170 2024-04-23T18:28:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2e4dcd640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e01010f0 0x7fb2e0102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2dffff640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2e0106170 0x7fb2e0103070 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb2d002ade0 tx=0x7fb2d0031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2ddffb640 1 -- 172.21.15.50:0/1650556114 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0003b70 con 0x7fb2e0106170 2024-04-23T18:28:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2ddffb640 1 -- 172.21.15.50:0/1650556114 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2d0003d10 con 0x7fb2e0106170 2024-04-23T18:28:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2ddffb640 1 -- 172.21.15.50:0/1650556114 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d002fa50 con 0x7fb2e0106170 2024-04-23T18:28:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.306+0000 7fb2e7031640 1 -- 172.21.15.50:0/1650556114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e01a7250 con 0x7fb2e0106170 2024-04-23T18:28:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.307+0000 7fb2e7031640 1 -- 172.21.15.50:0/1650556114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e01a77f0 con 0x7fb2e0106170 2024-04-23T18:28:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e9e28740 1 Processor -- start 2024-04-23T18:28:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2ddffb640 1 -- 172.21.15.50:0/1650556114 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb2d0004580 con 0x7fb2e0106170 2024-04-23T18:28:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e9e28740 1 -- start start 2024-04-23T18:28:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e9e28740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e9e28740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3f40 0x5638a25ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e9e28740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25ce800 0x5638a25d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e9e28740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5638a2528970 con 0x5638a25c3f40 2024-04-23T18:28:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e9e28740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5638a24c54a0 con 0x5638a25ce800 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e9e28740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5638a2524930 con 0x5638a25c3620 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e4dcd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3f40 0x5638a25ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e4dcd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3f40 0x5638a25ce2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36588/0 (socket says 172.21.15.50:36588) 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e4dcd640 1 -- 172.21.15.50:0/1808470472 learned_addr learned my addr 172.21.15.50:0/1808470472 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2dffff640 1 --2- 172.21.15.50:0/1808470472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25ce800 0x5638a25d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e55ce640 1 --2- 172.21.15.50:0/1808470472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e4dcd640 1 -- 172.21.15.50:0/1808470472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 msgr2=0x5638a25c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e4dcd640 1 --2- 172.21.15.50:0/1808470472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e4dcd640 1 -- 172.21.15.50:0/1808470472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25ce800 msgr2=0x5638a25d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e4dcd640 1 --2- 172.21.15.50:0/1808470472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25ce800 0x5638a25d0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2ddffb640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2cc0a3e70 0x7fb2cc0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e4dcd640 1 -- 172.21.15.50:0/1808470472 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638a223b210 con 0x5638a25c3f40 2024-04-23T18:28:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e55ce640 1 --2- 172.21.15.50:0/1808470472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2dffff640 1 --2- 172.21.15.50:0/1808470472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25ce800 0x5638a25d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e55ce640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2cc0a3e70 0x7fb2cc0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.311+0000 7fb2ddffb640 1 -- 172.21.15.50:0/1650556114 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7fb2d00f1a40 con 0x7fb2e0106170 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.310+0000 7fb2e4dcd640 1 --2- 172.21.15.50:0/1808470472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3f40 0x5638a25ce2c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80115c0 tx=0x7fb2d8011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4fc30b89dd7e9ca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.311+0000 7fb2d6ffd640 1 -- 172.21.15.50:0/1808470472 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d800f640 con 0x5638a25c3f40 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.311+0000 7fb2d6ffd640 1 -- 172.21.15.50:0/1808470472 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2d800fc90 con 0x5638a25c3f40 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.311+0000 7fb2d6ffd640 1 -- 172.21.15.50:0/1808470472 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8019ca0 con 0x5638a25c3f40 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.311+0000 7fb2e9e28740 1 -- 172.21.15.50:0/1808470472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3f40 msgr2=0x5638a25ce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.311+0000 7fb2e55ce640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2cc0a3e70 0x7fb2cc0a6330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800b600 tx=0x7fb2c800b2e0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.311+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/1808470472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3f40 0x5638a25ce2c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80115c0 tx=0x7fb2d8011a80 comp rx=0 tx=0).stop 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.311+0000 7fb2e9e28740 1 -- 172.21.15.50:0/1808470472 shutdown_connections 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.311+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/1808470472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25ce800 0x5638a25d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/1808470472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3f40 0x5638a25ce2c0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/1808470472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 -- 172.21.15.50:0/1808470472 >> 172.21.15.50:0/1808470472 conn(0x5638a25bfb30 msgr2=0x5638a25bff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 -- 172.21.15.50:0/1808470472 shutdown_connections 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 -- 172.21.15.50:0/1808470472 wait complete. 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 -- start start 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 Processor -- start 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25c3f40 0x5638a25cb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 0x5638a25cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5638a25d1430 con 0x5638a25ce800 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5638a24c30b0 con 0x5638a25c3f40 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5638a25d15b0 con 0x5638a25c3620 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.312+0000 7fb2e9e28740 10 client.0 _ll_register_callbacks cb 0x5638a24b0d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2e4dcd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2dffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25c3f40 0x5638a25cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2e55ce640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 0x5638a25cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2e4dcd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25ce160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:39638/0 (socket says 172.21.15.50:39638) 2024-04-23T18:28:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2e4dcd640 1 -- 172.21.15.50:0/2977543128 learned_addr learned my addr 172.21.15.50:0/2977543128 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2e4dcd640 1 -- 172.21.15.50:0/2977543128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25c3f40 msgr2=0x5638a25cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2e4dcd640 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25c3f40 0x5638a25cb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2e4dcd640 1 -- 172.21.15.50:0/2977543128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 msgr2=0x5638a25cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2e4dcd640 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 0x5638a25cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2e4dcd640 1 -- 172.21.15.50:0/2977543128 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638a2307620 con 0x5638a25c3620 2024-04-23T18:28:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2dffff640 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25c3f40 0x5638a25cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2e55ce640 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 0x5638a25cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2e4dcd640 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25ce160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8005740 tx=0x7fb2d8005770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=339b0590bea98726 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2bffff640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:28:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2bffff640 1 -- 172.21.15.50:0/2977543128 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8026020 con 0x5638a25c3620 2024-04-23T18:28:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2bffff640 1 -- 172.21.15.50:0/2977543128 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7fb2d8019c30 con 0x5638a25c3620 2024-04-23T18:28:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.313+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2977543128 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2d800fc90 con 0x5638a25c3620 2024-04-23T18:28:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.314+0000 7fb2bffff640 1 -- 172.21.15.50:0/2977543128 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb2d8031050 con 0x5638a25c3620 2024-04-23T18:28:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.314+0000 7fb2e9e28740 10 client.26368 fetch_fsmap learned FSMap version 44 2024-04-23T18:28:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.314+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2977543128 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5638a25dd2d0 con 0x5638a25c3620 2024-04-23T18:28:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.314+0000 7fb2bffff640 1 -- 172.21.15.50:0/2977543128 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2d800fc90 con 0x5638a25c3620 2024-04-23T18:28:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.314+0000 7fb2bffff640 1 -- 172.21.15.50:0/2977543128 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fb2d80035f0 con 0x5638a25c3620 2024-04-23T18:28:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.314+0000 7fb2e9e28740 10 client.26368 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.314+0000 7fb2e9e28740 10 client.26368 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.314+0000 7fb2e9e28740 20 client.26368 populate_metadata read hostname 'smithi050' 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.314+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5638a25de730 0x5638a25e0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.314+0000 7fb2e9e28740 4 client.26368 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.314+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2977543128 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638a2536390 con 0x5638a25de730 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.314+0000 7fb2dffff640 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5638a25de730 0x5638a25e0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.315+0000 7fb2dffff640 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5638a25de730 0x5638a25e0b20 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15b34f31075b4f13 server_cookie=c8f54ace1c32682b in_seq=0 out_seq=0 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.315+0000 7fb2bffff640 10 client.26368 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.315+0000 7fb2bffff640 1 -- 172.21.15.50:0/2977543128 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638a2536390 con 0x5638a25de730 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.315+0000 7fb2bffff640 10 client.26368 handle_command_reply: tid=0 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.315+0000 7fb2e9e28740 1 client.26368 shutdown 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.316+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2977543128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 msgr2=0x5638a25ce160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.316+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25ce160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8005740 tx=0x7fb2d8005770 comp rx=0 tx=0).stop 2024-04-23T18:28:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.316+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2977543128 shutdown_connections 2024-04-23T18:28:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.316+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5638a25de730 0x5638a25e0b20 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.316+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 0x5638a25cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.316+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25c3f40 0x5638a25cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.316+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/2977543128 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25ce160 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.316+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2977543128 >> 172.21.15.50:0/2977543128 conn(0x5638a25bfb30 msgr2=0x5638a25c5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.316+0000 7fb2bffff640 0 client.26368 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.316+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2977543128 shutdown_connections 2024-04-23T18:28:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.316+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2977543128 wait complete. 2024-04-23T18:28:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.316+0000 7fb2e9e28740 20 client.26368 trim_cache size 0 max 16384 2024-04-23T18:28:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.332+0000 7fb2e9e28740 1 Processor -- start 2024-04-23T18:28:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.332+0000 7fb2e9e28740 1 -- start start 2024-04-23T18:28:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.332+0000 7fb2e9e28740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e9e28740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25c3f40 0x5638a25cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e9e28740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 0x5638a25e6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e9e28740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01b26f0 con 0x5638a25ce800 2024-04-23T18:28:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e9e28740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01aff60 con 0x5638a25c3f40 2024-04-23T18:28:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e9e28740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01addd0 con 0x5638a25c3620 2024-04-23T18:28:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e55ce640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 0x5638a25e6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e55ce640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 0x5638a25e6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36594/0 (socket says 172.21.15.50:36594) 2024-04-23T18:28:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2dffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25c3f40 0x5638a25cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e4dcd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e55ce640 1 -- 172.21.15.50:0/3252972143 learned_addr learned my addr 172.21.15.50:0/3252972143 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:58.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e55ce640 1 -- 172.21.15.50:0/3252972143 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 msgr2=0x5638a25cb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:58.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e55ce640 1 --2- 172.21.15.50:0/3252972143 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25cb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e55ce640 1 -- 172.21.15.50:0/3252972143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25c3f40 msgr2=0x5638a25cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:58.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e55ce640 1 --2- 172.21.15.50:0/3252972143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25c3f40 0x5638a25cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e55ce640 1 -- 172.21.15.50:0/3252972143 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638a25c04b0 con 0x5638a25ce800 2024-04-23T18:28:58.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2dffff640 1 --2- 172.21.15.50:0/3252972143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25c3f40 0x5638a25cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:28:58.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2e55ce640 1 --2- 172.21.15.50:0/3252972143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 0x5638a25e6300 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8003df0 tx=0x7fb2c8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7139477b0a2456f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:58.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2d67fc640 1 -- 172.21.15.50:0/3252972143 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8029440 con 0x5638a25ce800 2024-04-23T18:28:58.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2d67fc640 1 -- 172.21.15.50:0/3252972143 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c801f8d0 con 0x5638a25ce800 2024-04-23T18:28:58.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.333+0000 7fb2d67fc640 1 -- 172.21.15.50:0/3252972143 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c801b890 con 0x5638a25ce800 2024-04-23T18:28:58.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.334+0000 7fb2e9e28740 1 -- 172.21.15.50:0/3252972143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 msgr2=0x5638a25e6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:58.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.334+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/3252972143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 0x5638a25e6300 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8003df0 tx=0x7fb2c8014000 comp rx=0 tx=0).stop 2024-04-23T18:28:58.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.334+0000 7fb2e9e28740 1 -- 172.21.15.50:0/3252972143 shutdown_connections 2024-04-23T18:28:58.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.334+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/3252972143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25ce800 0x5638a25e6300 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.334+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/3252972143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25c3f40 0x5638a25cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.334+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/3252972143 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3620 0x5638a25cb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.334+0000 7fb2e9e28740 1 -- 172.21.15.50:0/3252972143 >> 172.21.15.50:0/3252972143 conn(0x5638a25bfb30 msgr2=0x5638a25cace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:58.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.334+0000 7fb2e9e28740 1 -- 172.21.15.50:0/3252972143 shutdown_connections 2024-04-23T18:28:58.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.334+0000 7fb2e9e28740 1 -- 172.21.15.50:0/3252972143 wait complete. 2024-04-23T18:28:58.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e9e28740 1 -- start start 2024-04-23T18:28:58.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e9e28740 1 Processor -- start 2024-04-23T18:28:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e9e28740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3620 0x5638a25d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e9e28740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3f40 0x5638a25d37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e9e28740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25ce800 0x5638a25d3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e9e28740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01ae420 con 0x5638a25c3620 2024-04-23T18:28:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e9e28740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01af890 con 0x5638a25ce800 2024-04-23T18:28:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e9e28740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01aeaf0 con 0x5638a25c3f40 2024-04-23T18:28:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e9e28740 10 client.0 _ll_register_callbacks cb 0x7fb2d8005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e4dcd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3620 0x5638a25d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e4dcd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3620 0x5638a25d3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36606/0 (socket says 172.21.15.50:36606) 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e4dcd640 1 -- 172.21.15.50:0/2010402768 learned_addr learned my addr 172.21.15.50:0/2010402768 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e55ce640 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25ce800 0x5638a25d3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2dffff640 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3f40 0x5638a25d37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e4dcd640 1 -- 172.21.15.50:0/2010402768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3f40 msgr2=0x5638a25d37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e4dcd640 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3f40 0x5638a25d37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e4dcd640 1 -- 172.21.15.50:0/2010402768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25ce800 msgr2=0x5638a25d3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e4dcd640 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25ce800 0x5638a25d3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.335+0000 7fb2e4dcd640 1 -- 172.21.15.50:0/2010402768 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638a2642ba0 con 0x5638a25c3620 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.336+0000 7fb2e4dcd640 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3620 0x5638a25d3260 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80115c0 tx=0x7fb2d8003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb072f913d861c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.336+0000 7fb2bffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:28:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.336+0000 7fb2bffff640 1 -- 172.21.15.50:0/2010402768 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d803a060 con 0x5638a25c3620 2024-04-23T18:28:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.336+0000 7fb2bffff640 1 -- 172.21.15.50:0/2010402768 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7fb2d80052b0 con 0x5638a25c3620 2024-04-23T18:28:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.336+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2010402768 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2c801f8d0 con 0x5638a25c3620 2024-04-23T18:28:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.336+0000 7fb2bffff640 1 -- 172.21.15.50:0/2010402768 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb2d8026040 con 0x5638a25c3620 2024-04-23T18:28:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.336+0000 7fb2e9e28740 10 client.16770 fetch_fsmap learned FSMap version 44 2024-04-23T18:28:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.336+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2010402768 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5638a2643e90 con 0x5638a25c3620 2024-04-23T18:28:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.336+0000 7fb2bffff640 1 -- 172.21.15.50:0/2010402768 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c801f8d0 con 0x5638a25c3620 2024-04-23T18:28:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.337+0000 7fb2bffff640 1 -- 172.21.15.50:0/2010402768 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fb2d800fc80 con 0x5638a25c3620 2024-04-23T18:28:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.337+0000 7fb2e9e28740 10 client.16770 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:28:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.337+0000 7fb2e9e28740 10 client.16770 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:28:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.337+0000 7fb2e9e28740 20 client.16770 populate_metadata read hostname 'smithi050' 2024-04-23T18:28:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.337+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5638a26445e0 0x5638a2646a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:28:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.337+0000 7fb2e9e28740 4 client.16770 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:28:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.337+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2010402768 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5638a2530200 con 0x5638a26445e0 2024-04-23T18:28:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.337+0000 7fb2dffff640 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5638a26445e0 0x5638a2646a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:28:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.337+0000 7fb2dffff640 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5638a26445e0 0x5638a2646a00 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b532040ff88c23ad server_cookie=5adf8706a6b700d6 in_seq=0 out_seq=0 2024-04-23T18:28:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.337+0000 7fb2bffff640 10 client.16770 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.375+0000 7fb2bffff640 1 -- 172.21.15.50:0/2010402768 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5638a2530200 con 0x5638a26445e0 2024-04-23T18:28:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.375+0000 7fb2bffff640 10 client.16770 handle_command_reply: tid=0 2024-04-23T18:28:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2e9e28740 1 client.16770 shutdown 2024-04-23T18:28:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2010402768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3620 msgr2=0x5638a25d3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3620 0x5638a25d3260 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80115c0 tx=0x7fb2d8003820 comp rx=0 tx=0).stop 2024-04-23T18:28:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2010402768 shutdown_connections 2024-04-23T18:28:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5638a26445e0 0x5638a2646a00 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638a25ce800 0x5638a25d3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5638a25c3f40 0x5638a25d37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2e9e28740 1 --2- 172.21.15.50:0/2010402768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5638a25c3620 0x5638a25d3260 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2bffff640 0 client.16770 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:28:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2010402768 >> 172.21.15.50:0/2010402768 conn(0x5638a25bfb30 msgr2=0x5638a25be420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2010402768 shutdown_connections 2024-04-23T18:28:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2e9e28740 1 -- 172.21.15.50:0/2010402768 wait complete. 2024-04-23T18:28:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.376+0000 7fb2e9e28740 20 client.16770 trim_cache size 0 max 16384 2024-04-23T18:28:58.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:28:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:28:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:28:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:28:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.377+0000 7fb2e7031640 1 -- 172.21.15.50:0/1650556114 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2cc0a3e70 msgr2=0x7fb2cc0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.377+0000 7fb2e7031640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2cc0a3e70 0x7fb2cc0a6330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800b600 tx=0x7fb2c800b2e0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.377+0000 7fb2e7031640 1 -- 172.21.15.50:0/1650556114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2e0106170 msgr2=0x7fb2e0103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:28:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.377+0000 7fb2e7031640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2e0106170 0x7fb2e0103070 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb2d002ade0 tx=0x7fb2d0031770 comp rx=0 tx=0).stop 2024-04-23T18:28:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.377+0000 7fb2e7031640 1 -- 172.21.15.50:0/1650556114 shutdown_connections 2024-04-23T18:28:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.377+0000 7fb2e7031640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb2cc0a3e70 0x7fb2cc0a6330 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.377+0000 7fb2e7031640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2e01035b0 0x7fb2e01a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.377+0000 7fb2e7031640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2e0106170 0x7fb2e0103070 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.377+0000 7fb2e7031640 1 --2- 172.21.15.50:0/1650556114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e01010f0 0x7fb2e0102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:28:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.377+0000 7fb2e7031640 1 -- 172.21.15.50:0/1650556114 >> 172.21.15.50:0/1650556114 conn(0x7fb2e0077fd0 msgr2=0x7fb2e0069940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:28:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.377+0000 7fb2e7031640 1 -- 172.21.15.50:0/1650556114 shutdown_connections 2024-04-23T18:28:58.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:28:58.377+0000 7fb2e7031640 1 -- 172.21.15.50:0/1650556114 wait complete. 2024-04-23T18:28:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:28:58 smithi177 ceph-mon[32308]: pgmap v327: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 157 op/s 2024-04-23T18:28:58.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:28:58 smithi188 ceph-mon[32694]: pgmap v327: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 157 op/s 2024-04-23T18:28:58.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:28:58 smithi050 ceph-mon[29187]: pgmap v327: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 157 op/s 2024-04-23T18:28:59.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 639542 5.34 MB/sec execute 453 sec latency 98.237 ms 2024-04-23T18:29:00.228 INFO:tasks.workunit.client.0.smithi050.stdout: 1 640764 5.34 MB/sec execute 454 sec latency 19.594 ms 2024-04-23T18:29:00.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:00 smithi177 ceph-mon[32308]: pgmap v328: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 732 KiB/s wr, 93 op/s 2024-04-23T18:29:00.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:00 smithi188 ceph-mon[32694]: pgmap v328: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 732 KiB/s wr, 93 op/s 2024-04-23T18:29:00.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:00 smithi050 ceph-mon[29187]: pgmap v328: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 732 KiB/s wr, 93 op/s 2024-04-23T18:29:01.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 642873 5.35 MB/sec execute 455 sec latency 30.645 ms 2024-04-23T18:29:02.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 643837 5.35 MB/sec execute 456 sec latency 221.282 ms 2024-04-23T18:29:02.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:02 smithi177 ceph-mon[32308]: pgmap v329: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 732 KiB/s wr, 93 op/s 2024-04-23T18:29:02.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:02 smithi188 ceph-mon[32694]: pgmap v329: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 732 KiB/s wr, 93 op/s 2024-04-23T18:29:02.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:02 smithi050 ceph-mon[29187]: pgmap v329: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 732 KiB/s wr, 93 op/s 2024-04-23T18:29:03.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 645821 5.36 MB/sec execute 457 sec latency 16.842 ms 2024-04-23T18:29:04.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 647457 5.37 MB/sec execute 458 sec latency 22.277 ms 2024-04-23T18:29:04.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:04 smithi177 ceph-mon[32308]: pgmap v330: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 212 op/s 2024-04-23T18:29:04.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:04 smithi188 ceph-mon[32694]: pgmap v330: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 212 op/s 2024-04-23T18:29:04.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:04 smithi050 ceph-mon[29187]: pgmap v330: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 212 op/s 2024-04-23T18:29:05.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 649112 5.37 MB/sec execute 459 sec latency 71.284 ms 2024-04-23T18:29:05.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:05 smithi177 ceph-mon[32308]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T18:29:05.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:05 smithi188 ceph-mon[32694]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T18:29:05.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:05 smithi050 ceph-mon[29187]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T18:29:06.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 650572 5.38 MB/sec execute 460 sec latency 41.248 ms 2024-04-23T18:29:06.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:06 smithi177 ceph-mon[32308]: pgmap v332: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 204 op/s 2024-04-23T18:29:06.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:06 smithi188 ceph-mon[32694]: pgmap v332: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 204 op/s 2024-04-23T18:29:06.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:06 smithi050 ceph-mon[29187]: pgmap v332: 97 pgs: 97 active+clean; 208 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.5 MiB/s wr, 204 op/s 2024-04-23T18:29:07.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 652299 5.38 MB/sec execute 461 sec latency 44.241 ms 2024-04-23T18:29:08.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 654033 5.40 MB/sec execute 462 sec latency 9.340 ms 2024-04-23T18:29:08.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:08 smithi177 ceph-mon[32308]: pgmap v333: 97 pgs: 97 active+clean; 221 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-04-23T18:29:08.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:08 smithi188 ceph-mon[32694]: pgmap v333: 97 pgs: 97 active+clean; 221 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-04-23T18:29:08.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:08 smithi050 ceph-mon[29187]: pgmap v333: 97 pgs: 97 active+clean; 221 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-04-23T18:29:09.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 655840 5.40 MB/sec execute 463 sec latency 18.561 ms 2024-04-23T18:29:10.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 657459 5.42 MB/sec execute 464 sec latency 13.735 ms 2024-04-23T18:29:10.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:10 smithi177 ceph-mon[32308]: pgmap v334: 97 pgs: 97 active+clean; 221 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-04-23T18:29:10.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:10 smithi188 ceph-mon[32694]: pgmap v334: 97 pgs: 97 active+clean; 221 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-04-23T18:29:10.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:10 smithi050 ceph-mon[29187]: pgmap v334: 97 pgs: 97 active+clean; 221 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-04-23T18:29:11.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 659075 5.42 MB/sec execute 465 sec latency 27.822 ms 2024-04-23T18:29:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.470+0000 7ff59037c640 1 -- 172.21.15.50:0/3925306862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff588102b90 msgr2=0x7ff588106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.470+0000 7ff59037c640 1 --2- 172.21.15.50:0/3925306862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff588102b90 0x7ff588106bd0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009900 tx=0x7ff57c02a410 comp rx=0 tx=0).stop 2024-04-23T18:29:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.470+0000 7ff59037c640 1 -- 172.21.15.50:0/3925306862 shutdown_connections 2024-04-23T18:29:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.470+0000 7ff59037c640 1 --2- 172.21.15.50:0/3925306862 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5881072b0 0x7ff58810ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.470+0000 7ff59037c640 1 --2- 172.21.15.50:0/3925306862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff588102b90 0x7ff588106bd0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.470+0000 7ff59037c640 1 --2- 172.21.15.50:0/3925306862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5881021e0 0x7ff5881025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.470+0000 7ff59037c640 1 -- 172.21.15.50:0/3925306862 >> 172.21.15.50:0/3925306862 conn(0x7ff5880fda00 msgr2=0x7ff5880ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.470+0000 7ff59037c640 1 -- 172.21.15.50:0/3925306862 shutdown_connections 2024-04-23T18:29:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.470+0000 7ff59037c640 1 -- 172.21.15.50:0/3925306862 wait complete. 2024-04-23T18:29:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.470+0000 7ff59037c640 1 Processor -- start 2024-04-23T18:29:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff59037c640 1 -- start start 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff59037c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5881021e0 0x7ff588198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff59037c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5881072b0 0x7ff588198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff59037c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff58819cfd0 0x7ff58819d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff59037c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5881121f0 con 0x7ff58819cfd0 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff59037c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff588112070 con 0x7ff5881021e0 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff59037c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff588112370 con 0x7ff5881072b0 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58e919640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff58819cfd0 0x7ff58819d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58d917640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5881072b0 0x7ff588198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58e919640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff58819cfd0 0x7ff58819d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53794/0 (socket says 172.21.15.50:53794) 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58e118640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5881021e0 0x7ff588198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58e919640 1 -- 172.21.15.50:0/185112392 learned_addr learned my addr 172.21.15.50:0/185112392 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58e919640 1 -- 172.21.15.50:0/185112392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5881072b0 msgr2=0x7ff588198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58e919640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5881072b0 0x7ff588198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58e919640 1 -- 172.21.15.50:0/185112392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5881021e0 msgr2=0x7ff588198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58e919640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5881021e0 0x7ff588198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58e919640 1 -- 172.21.15.50:0/185112392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff58819dac0 con 0x7ff58819cfd0 2024-04-23T18:29:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58d917640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5881072b0 0x7ff588198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58e118640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5881021e0 0x7ff588198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.471+0000 7ff58e919640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff58819cfd0 0x7ff58819d480 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ff58000eb10 tx=0x7ff58000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.472+0000 7ff57b7fe640 1 -- 172.21.15.50:0/185112392 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580004b20 con 0x7ff58819cfd0 2024-04-23T18:29:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.472+0000 7ff57b7fe640 1 -- 172.21.15.50:0/185112392 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff580004cc0 con 0x7ff58819cfd0 2024-04-23T18:29:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.472+0000 7ff59037c640 1 -- 172.21.15.50:0/185112392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff58819ddb0 con 0x7ff58819cfd0 2024-04-23T18:29:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.472+0000 7ff57b7fe640 1 -- 172.21.15.50:0/185112392 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580010670 con 0x7ff58819cfd0 2024-04-23T18:29:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.472+0000 7ff59037c640 1 -- 172.21.15.50:0/185112392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5881a5570 con 0x7ff58819cfd0 2024-04-23T18:29:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.474+0000 7ff57b7fe640 1 -- 172.21.15.50:0/185112392 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff580005020 con 0x7ff58819cfd0 2024-04-23T18:29:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.474+0000 7ff593168740 1 Processor -- start 2024-04-23T18:29:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff593168740 1 -- start start 2024-04-23T18:29:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff593168740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2dba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff593168740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff593168740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff593168740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bd6f240970 con 0x55bd6f2db620 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff57b7fe640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5740a3e40 0x7ff5740a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff593168740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bd6f1dd4a0 con 0x55bd6f2e6800 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff593168740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bd6f23c930 con 0x55bd6f2dbf40 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff58d917640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2dba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff58d917640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2dba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53804/0 (socket says 172.21.15.50:53804) 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff58d917640 1 -- 172.21.15.50:0/2477092029 learned_addr learned my addr 172.21.15.50:0/2477092029 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff58e118640 1 --2- 172.21.15.50:0/2477092029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff57b7fe640 1 -- 172.21.15.50:0/185112392 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7ff5800d0d10 con 0x7ff58819cfd0 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff58e919640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5740a3e40 0x7ff5740a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff58d917640 1 --2- 172.21.15.50:0/2477092029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff58e118640 1 -- 172.21.15.50:0/2477092029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2e6800 msgr2=0x55bd6f2e8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff58e118640 1 --2- 172.21.15.50:0/2477092029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff58e118640 1 -- 172.21.15.50:0/2477092029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 msgr2=0x55bd6f2dba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff58e118640 1 --2- 172.21.15.50:0/2477092029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2dba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff58e118640 1 -- 172.21.15.50:0/2477092029 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd6ef53210 con 0x55bd6f2dbf40 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.475+0000 7ff58d917640 1 --2- 172.21.15.50:0/2477092029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2dba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.476+0000 7ff58e919640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5740a3e40 0x7ff5740a6300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff5800049d0 tx=0x7ff58006e680 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.476+0000 7ff58e118640 1 --2- 172.21.15.50:0/2477092029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e62c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff584009fd0 tx=0x7ff5840118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4885314b1ef2d91 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.476+0000 7ff578bfe640 1 -- 172.21.15.50:0/2477092029 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff584004310 con 0x55bd6f2dbf40 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.476+0000 7ff578bfe640 1 -- 172.21.15.50:0/2477092029 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5840044b0 con 0x55bd6f2dbf40 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.476+0000 7ff578bfe640 1 -- 172.21.15.50:0/2477092029 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff584004e50 con 0x55bd6f2dbf40 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 -- 172.21.15.50:0/2477092029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2dbf40 msgr2=0x55bd6f2e62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 --2- 172.21.15.50:0/2477092029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e62c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff584009fd0 tx=0x7ff5840118d0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 -- 172.21.15.50:0/2477092029 shutdown_connections 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 --2- 172.21.15.50:0/2477092029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 --2- 172.21.15.50:0/2477092029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e62c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 --2- 172.21.15.50:0/2477092029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2dba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 -- 172.21.15.50:0/2477092029 >> 172.21.15.50:0/2477092029 conn(0x55bd6f2d7b30 msgr2=0x55bd6f2d7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 -- 172.21.15.50:0/2477092029 shutdown_connections 2024-04-23T18:29:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 -- 172.21.15.50:0/2477092029 wait complete. 2024-04-23T18:29:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 -- start start 2024-04-23T18:29:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 Processor -- start 2024-04-23T18:29:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2db620 0x55bd6f2e60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bd6f2e9430 con 0x55bd6f2e6800 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bd6f1db0b0 con 0x55bd6f2dbf40 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bd6f2e95b0 con 0x55bd6f2db620 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff593168740 10 client.0 _ll_register_callbacks cb 0x55bd6f1c8d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.477+0000 7ff58d917640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff58d917640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53814/0 (socket says 172.21.15.50:53814) 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff58d917640 1 -- 172.21.15.50:0/216507335 learned_addr learned my addr 172.21.15.50:0/216507335 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff58e118640 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff58d917640 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2db620 0x55bd6f2e60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff58d917640 1 -- 172.21.15.50:0/216507335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2db620 msgr2=0x55bd6f2e60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff58d917640 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2db620 0x55bd6f2e60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff58d917640 1 -- 172.21.15.50:0/216507335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2dbf40 msgr2=0x55bd6f2e33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff58d917640 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff58d917640 1 -- 172.21.15.50:0/216507335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd6f01f620 con 0x55bd6f2e6800 2024-04-23T18:29:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff58e118640 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff58d917640 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e3920 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff57c000c00 tx=0x7ff57c031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10464b34acf6718 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff571ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff571ffb640 1 -- 172.21.15.50:0/216507335 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c035040 con 0x55bd6f2e6800 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff571ffb640 1 -- 172.21.15.50:0/216507335 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7ff57c0037f0 con 0x55bd6f2e6800 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff593168740 1 -- 172.21.15.50:0/216507335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5840044b0 con 0x55bd6f2e6800 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff571ffb640 1 -- 172.21.15.50:0/216507335 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7ff57c051050 con 0x55bd6f2e6800 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff593168740 10 client.16782 fetch_fsmap learned FSMap version 44 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.478+0000 7ff593168740 1 -- 172.21.15.50:0/216507335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55bd6f2f5300 con 0x55bd6f2e6800 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.479+0000 7ff571ffb640 1 -- 172.21.15.50:0/216507335 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5840044b0 con 0x55bd6f2e6800 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.479+0000 7ff571ffb640 1 -- 172.21.15.50:0/216507335 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7ff57c03c960 con 0x55bd6f2e6800 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.479+0000 7ff593168740 10 client.16782 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.479+0000 7ff593168740 10 client.16782 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.479+0000 7ff593168740 20 client.16782 populate_metadata read hostname 'smithi050' 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.479+0000 7ff593168740 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd6f2f67c0 0x55bd6f2f8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.479+0000 7ff593168740 4 client.16782 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.479+0000 7ff593168740 1 -- 172.21.15.50:0/216507335 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd6f24e390 con 0x55bd6f2f67c0 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.479+0000 7ff58e118640 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd6f2f67c0 0x55bd6f2f8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.479+0000 7ff58e118640 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd6f2f67c0 0x55bd6f2f8bb0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a6a44218397401e server_cookie=e24c8e2a96e4920c in_seq=0 out_seq=0 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.479+0000 7ff571ffb640 10 client.16782 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff571ffb640 1 -- 172.21.15.50:0/216507335 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bd6f24e390 con 0x55bd6f2f67c0 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff571ffb640 10 client.16782 handle_command_reply: tid=0 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff593168740 1 client.16782 shutdown 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff593168740 1 -- 172.21.15.50:0/216507335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2e6800 msgr2=0x55bd6f2e3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff593168740 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e3920 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff57c000c00 tx=0x7ff57c031400 comp rx=0 tx=0).stop 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff593168740 1 -- 172.21.15.50:0/216507335 shutdown_connections 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff593168740 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd6f2f67c0 0x55bd6f2f8bb0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff593168740 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e3920 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff593168740 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff571ffb640 0 client.16782 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:11.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff593168740 1 --2- 172.21.15.50:0/216507335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2db620 0x55bd6f2e60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff593168740 1 -- 172.21.15.50:0/216507335 >> 172.21.15.50:0/216507335 conn(0x55bd6f2d7b30 msgr2=0x55bd6f2dd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:11.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff593168740 1 -- 172.21.15.50:0/216507335 shutdown_connections 2024-04-23T18:29:11.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff593168740 1 -- 172.21.15.50:0/216507335 wait complete. 2024-04-23T18:29:11.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.480+0000 7ff593168740 20 client.16782 trim_cache size 0 max 16384 2024-04-23T18:29:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff593168740 1 Processor -- start 2024-04-23T18:29:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff593168740 1 -- start start 2024-04-23T18:29:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff593168740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2e5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff593168740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2dd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff593168740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2e6800 0x55bd6f2fe4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff593168740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5881b0510 con 0x55bd6f2db620 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff593168740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff5881add80 con 0x55bd6f2e6800 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff593168740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff5881abbf0 con 0x55bd6f2dbf40 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff58d917640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2e5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff58d917640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2e5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53822/0 (socket says 172.21.15.50:53822) 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff58d917640 1 -- 172.21.15.50:0/2173790365 learned_addr learned my addr 172.21.15.50:0/2173790365 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff58e118640 1 --2- 172.21.15.50:0/2173790365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2dd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff58d917640 1 --2- 172.21.15.50:0/2173790365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2e6800 0x55bd6f2fe4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff58d917640 1 -- 172.21.15.50:0/2173790365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2dbf40 msgr2=0x55bd6f2dd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff58d917640 1 --2- 172.21.15.50:0/2173790365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2dd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff58d917640 1 -- 172.21.15.50:0/2173790365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2e6800 msgr2=0x55bd6f2fe4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff58d917640 1 --2- 172.21.15.50:0/2173790365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2e6800 0x55bd6f2fe4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff58d917640 1 -- 172.21.15.50:0/2173790365 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd6f2e1950 con 0x55bd6f2db620 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff58d917640 1 --2- 172.21.15.50:0/2173790365 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2e5c20 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff57c02a930 tx=0x7ff57c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbe1a677678f454 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff573fff640 1 -- 172.21.15.50:0/2173790365 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c035040 con 0x55bd6f2db620 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff573fff640 1 -- 172.21.15.50:0/2173790365 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff57c03c700 con 0x55bd6f2db620 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.496+0000 7ff573fff640 1 -- 172.21.15.50:0/2173790365 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c0037f0 con 0x55bd6f2db620 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.497+0000 7ff593168740 1 -- 172.21.15.50:0/2173790365 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 msgr2=0x55bd6f2e5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.497+0000 7ff593168740 1 --2- 172.21.15.50:0/2173790365 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2e5c20 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff57c02a930 tx=0x7ff57c02a410 comp rx=0 tx=0).stop 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.497+0000 7ff593168740 1 -- 172.21.15.50:0/2173790365 shutdown_connections 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.497+0000 7ff593168740 1 --2- 172.21.15.50:0/2173790365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2e6800 0x55bd6f2fe4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.497+0000 7ff593168740 1 --2- 172.21.15.50:0/2173790365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2dd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.497+0000 7ff593168740 1 --2- 172.21.15.50:0/2173790365 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2e5c20 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.497+0000 7ff593168740 1 -- 172.21.15.50:0/2173790365 >> 172.21.15.50:0/2173790365 conn(0x55bd6f2d7b30 msgr2=0x55bd6f2dcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.497+0000 7ff593168740 1 -- 172.21.15.50:0/2173790365 shutdown_connections 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.497+0000 7ff593168740 1 -- 172.21.15.50:0/2173790365 wait complete. 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.497+0000 7ff593168740 1 -- start start 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.497+0000 7ff593168740 1 Processor -- start 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff593168740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2dffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff593168740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff593168740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff593168740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5881ac240 con 0x55bd6f2db620 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff593168740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff5881ad6b0 con 0x55bd6f2dbf40 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff593168740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff5881ac910 con 0x55bd6f2e6800 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff593168740 10 client.0 _ll_register_callbacks cb 0x7ff57c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff58d917640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2dffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff58d917640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2dffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53828/0 (socket says 172.21.15.50:53828) 2024-04-23T18:29:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff58d917640 1 -- 172.21.15.50:0/3428783953 learned_addr learned my addr 172.21.15.50:0/3428783953 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff58e118640 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff58d917640 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff58d917640 1 -- 172.21.15.50:0/3428783953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2e6800 msgr2=0x55bd6f2e0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff58d917640 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff58d917640 1 -- 172.21.15.50:0/3428783953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2dbf40 msgr2=0x55bd6f2e04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff58d917640 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff58d917640 1 -- 172.21.15.50:0/3428783953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd6f35bd30 con 0x55bd6f2db620 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff58d917640 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2dffb0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0319f0 tx=0x7ff57c058040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8856e0d45e2bd0ac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff571ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff571ffb640 1 -- 172.21.15.50:0/3428783953 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c035070 con 0x55bd6f2db620 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff571ffb640 1 -- 172.21.15.50:0/3428783953 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7ff57c05d040 con 0x55bd6f2db620 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff593168740 1 -- 172.21.15.50:0/3428783953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff57c03c700 con 0x55bd6f2db620 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff571ffb640 1 -- 172.21.15.50:0/3428783953 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7ff57c031a90 con 0x55bd6f2db620 2024-04-23T18:29:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.498+0000 7ff593168740 10 client.16788 fetch_fsmap learned FSMap version 44 2024-04-23T18:29:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.499+0000 7ff593168740 1 -- 172.21.15.50:0/3428783953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55bd6f35cef0 con 0x55bd6f2db620 2024-04-23T18:29:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.499+0000 7ff571ffb640 1 -- 172.21.15.50:0/3428783953 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff57c03c700 con 0x55bd6f2db620 2024-04-23T18:29:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.499+0000 7ff571ffb640 1 -- 172.21.15.50:0/3428783953 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7ff57c003ba0 con 0x55bd6f2db620 2024-04-23T18:29:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.499+0000 7ff593168740 10 client.16788 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:29:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.499+0000 7ff593168740 10 client.16788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.499+0000 7ff593168740 20 client.16788 populate_metadata read hostname 'smithi050' 2024-04-23T18:29:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.499+0000 7ff593168740 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd6f35d5c0 0x55bd6f35f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.499+0000 7ff593168740 4 client.16788 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:29:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.499+0000 7ff593168740 1 -- 172.21.15.50:0/3428783953 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bd6f248200 con 0x55bd6f35d5c0 2024-04-23T18:29:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.499+0000 7ff58e118640 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd6f35d5c0 0x55bd6f35f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.499+0000 7ff571ffb640 10 client.16788 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.499+0000 7ff58e118640 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd6f35d5c0 0x55bd6f35f9e0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32f45e6646f274b2 server_cookie=91baec5a46fd02b0 in_seq=0 out_seq=0 2024-04-23T18:29:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.540+0000 7ff571ffb640 1 -- 172.21.15.50:0/3428783953 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bd6f248200 con 0x55bd6f35d5c0 2024-04-23T18:29:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.540+0000 7ff571ffb640 10 client.16788 handle_command_reply: tid=0 2024-04-23T18:29:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.540+0000 7ff593168740 1 client.16788 shutdown 2024-04-23T18:29:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.540+0000 7ff593168740 1 -- 172.21.15.50:0/3428783953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 msgr2=0x55bd6f2dffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.540+0000 7ff593168740 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2dffb0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0319f0 tx=0x7ff57c058040 comp rx=0 tx=0).stop 2024-04-23T18:29:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.540+0000 7ff593168740 1 -- 172.21.15.50:0/3428783953 shutdown_connections 2024-04-23T18:29:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.540+0000 7ff593168740 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd6f35d5c0 0x55bd6f35f9e0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.540+0000 7ff593168740 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd6f2e6800 0x55bd6f2e0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.540+0000 7ff593168740 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd6f2dbf40 0x55bd6f2e04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.540+0000 7ff593168740 1 --2- 172.21.15.50:0/3428783953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd6f2db620 0x55bd6f2dffb0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.540+0000 7ff593168740 1 -- 172.21.15.50:0/3428783953 >> 172.21.15.50:0/3428783953 conn(0x55bd6f2d7b30 msgr2=0x55bd6f2dd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.540+0000 7ff571ffb640 0 client.16788 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.541+0000 7ff593168740 1 -- 172.21.15.50:0/3428783953 shutdown_connections 2024-04-23T18:29:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.541+0000 7ff593168740 1 -- 172.21.15.50:0/3428783953 wait complete. 2024-04-23T18:29:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.541+0000 7ff593168740 20 client.16788 trim_cache size 0 max 16384 2024-04-23T18:29:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:29:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:29:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:29:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:29:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.542+0000 7ff59037c640 1 -- 172.21.15.50:0/185112392 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5740a3e40 msgr2=0x7ff5740a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.542+0000 7ff59037c640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5740a3e40 0x7ff5740a6300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff5800049d0 tx=0x7ff58006e680 comp rx=0 tx=0).stop 2024-04-23T18:29:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.542+0000 7ff59037c640 1 -- 172.21.15.50:0/185112392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff58819cfd0 msgr2=0x7ff58819d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.542+0000 7ff59037c640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff58819cfd0 0x7ff58819d480 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ff58000eb10 tx=0x7ff58000c2d0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.542+0000 7ff59037c640 1 -- 172.21.15.50:0/185112392 shutdown_connections 2024-04-23T18:29:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.542+0000 7ff59037c640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5740a3e40 0x7ff5740a6300 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.542+0000 7ff59037c640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff58819cfd0 0x7ff58819d480 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.542+0000 7ff59037c640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5881072b0 0x7ff588198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.542+0000 7ff59037c640 1 --2- 172.21.15.50:0/185112392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5881021e0 0x7ff588198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.542+0000 7ff59037c640 1 -- 172.21.15.50:0/185112392 >> 172.21.15.50:0/185112392 conn(0x7ff5880fda00 msgr2=0x7ff5880fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.542+0000 7ff59037c640 1 -- 172.21.15.50:0/185112392 shutdown_connections 2024-04-23T18:29:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:11.543+0000 7ff59037c640 1 -- 172.21.15.50:0/185112392 wait complete. 2024-04-23T18:29:12.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 660868 5.43 MB/sec execute 466 sec latency 43.151 ms 2024-04-23T18:29:12.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:12 smithi050 ceph-mon[29187]: pgmap v335: 97 pgs: 97 active+clean; 221 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-04-23T18:29:12.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:12 smithi050 ceph-mon[29187]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T18:29:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:12 smithi177 ceph-mon[32308]: pgmap v335: 97 pgs: 97 active+clean; 221 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-04-23T18:29:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:12 smithi177 ceph-mon[32308]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T18:29:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:12 smithi188 ceph-mon[32694]: pgmap v335: 97 pgs: 97 active+clean; 221 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 262 op/s 2024-04-23T18:29:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:12 smithi188 ceph-mon[32694]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T18:29:13.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 662139 5.43 MB/sec execute 467 sec latency 18.030 ms 2024-04-23T18:29:14.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 664312 5.45 MB/sec execute 468 sec latency 22.189 ms 2024-04-23T18:29:14.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:14 smithi050 ceph-mon[29187]: pgmap v337: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.4 MiB/s wr, 314 op/s 2024-04-23T18:29:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:14 smithi177 ceph-mon[32308]: pgmap v337: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.4 MiB/s wr, 314 op/s 2024-04-23T18:29:14.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:14 smithi188 ceph-mon[32694]: pgmap v337: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.4 MiB/s wr, 314 op/s 2024-04-23T18:29:15.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 665631 5.45 MB/sec execute 469 sec latency 22.245 ms 2024-04-23T18:29:16.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 667736 5.47 MB/sec execute 470 sec latency 25.923 ms 2024-04-23T18:29:16.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:16 smithi050 ceph-mon[29187]: pgmap v338: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 264 op/s 2024-04-23T18:29:16.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:16 smithi177 ceph-mon[32308]: pgmap v338: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 264 op/s 2024-04-23T18:29:16.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:16 smithi188 ceph-mon[32694]: pgmap v338: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 264 op/s 2024-04-23T18:29:17.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 669075 5.47 MB/sec execute 471 sec latency 24.621 ms 2024-04-23T18:29:18.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 670996 5.48 MB/sec execute 472 sec latency 13.455 ms 2024-04-23T18:29:18.355 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:18 smithi177 ceph-mon[32308]: pgmap v339: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-23T18:29:18.380 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:18 smithi188 ceph-mon[32694]: pgmap v339: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-23T18:29:18.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:18 smithi050 ceph-mon[29187]: pgmap v339: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-23T18:29:19.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 671975 5.48 MB/sec execute 473 sec latency 25.336 ms 2024-04-23T18:29:20.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 673309 5.48 MB/sec execute 474 sec latency 24.288 ms 2024-04-23T18:29:20.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:20 smithi050 ceph-mon[29187]: pgmap v340: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-23T18:29:20.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:20 smithi177 ceph-mon[32308]: pgmap v340: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-23T18:29:20.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:20 smithi188 ceph-mon[32694]: pgmap v340: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-23T18:29:21.229 INFO:tasks.workunit.client.0.smithi050.stdout: 1 675122 5.49 MB/sec execute 475 sec latency 33.545 ms 2024-04-23T18:29:22.230 INFO:tasks.workunit.client.0.smithi050.stdout: 1 676627 5.49 MB/sec execute 476 sec latency 24.633 ms 2024-04-23T18:29:22.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:22 smithi050 ceph-mon[29187]: pgmap v341: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-23T18:29:22.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:22 smithi177 ceph-mon[32308]: pgmap v341: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-23T18:29:22.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:22 smithi188 ceph-mon[32694]: pgmap v341: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-04-23T18:29:23.230 INFO:tasks.workunit.client.0.smithi050.stdout: 1 678537 5.51 MB/sec execute 477 sec latency 15.272 ms 2024-04-23T18:29:24.230 INFO:tasks.workunit.client.0.smithi050.stdout: 1 679561 5.50 MB/sec execute 478 sec latency 27.007 ms 2024-04-23T18:29:24.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:24 smithi050 ceph-mon[29187]: pgmap v342: 97 pgs: 97 active+clean; 228 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.1 MiB/s wr, 323 op/s 2024-04-23T18:29:24.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:24 smithi177 ceph-mon[32308]: pgmap v342: 97 pgs: 97 active+clean; 228 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.1 MiB/s wr, 323 op/s 2024-04-23T18:29:24.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:24 smithi188 ceph-mon[32694]: pgmap v342: 97 pgs: 97 active+clean; 228 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.1 MiB/s wr, 323 op/s 2024-04-23T18:29:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.635+0000 7f843f4cb640 1 -- 172.21.15.50:0/3247798132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438101130 msgr2=0x7f84381015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.635+0000 7f843f4cb640 1 --2- 172.21.15.50:0/3247798132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438101130 0x7f84381015b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f8428009930 tx=0x7f842802a420 comp rx=0 tx=0).stop 2024-04-23T18:29:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.635+0000 7f843f4cb640 1 -- 172.21.15.50:0/3247798132 shutdown_connections 2024-04-23T18:29:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.635+0000 7f843f4cb640 1 --2- 172.21.15.50:0/3247798132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8438101af0 0x7f843810dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.635+0000 7f843f4cb640 1 --2- 172.21.15.50:0/3247798132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438101130 0x7f84381015b0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.635+0000 7f843f4cb640 1 --2- 172.21.15.50:0/3247798132 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8438106f90 0x7f8438107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.635+0000 7f843f4cb640 1 -- 172.21.15.50:0/3247798132 >> 172.21.15.50:0/3247798132 conn(0x7f8438077c30 msgr2=0x7f8438078040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.636+0000 7f843f4cb640 1 -- 172.21.15.50:0/3247798132 shutdown_connections 2024-04-23T18:29:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.636+0000 7f843f4cb640 1 -- 172.21.15.50:0/3247798132 wait complete. 2024-04-23T18:29:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.636+0000 7f843f4cb640 1 Processor -- start 2024-04-23T18:29:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.636+0000 7f843f4cb640 1 -- start start 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.636+0000 7f843f4cb640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8438101130 0x7f84381986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.636+0000 7f843f4cb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438101af0 0x7f8438198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.636+0000 7f843f4cb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8438106f90 0x7f843819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.636+0000 7f843f4cb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8438111090 con 0x7f8438101af0 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.636+0000 7f843f4cb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8438110f10 con 0x7f8438106f90 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.636+0000 7f843f4cb640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8438111210 con 0x7f8438101130 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843ca66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438101af0 0x7f8438198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843d267640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8438101130 0x7f84381986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843d267640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8438101130 0x7f84381986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:57698/0 (socket says 172.21.15.50:57698) 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843ca66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438101af0 0x7f8438198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56294/0 (socket says 172.21.15.50:56294) 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843da68640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8438106f90 0x7f843819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843d267640 1 -- 172.21.15.50:0/1953017149 learned_addr learned my addr 172.21.15.50:0/1953017149 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843ca66640 1 -- 172.21.15.50:0/1953017149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8438101130 msgr2=0x7f84381986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:24.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843ca66640 1 --2- 172.21.15.50:0/1953017149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8438101130 0x7f84381986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843ca66640 1 -- 172.21.15.50:0/1953017149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8438106f90 msgr2=0x7f843819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:24.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843ca66640 1 --2- 172.21.15.50:0/1953017149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8438106f90 0x7f843819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843ca66640 1 -- 172.21.15.50:0/1953017149 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f843819d6b0 con 0x7f8438101af0 2024-04-23T18:29:24.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843ca66640 1 --2- 172.21.15.50:0/1953017149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438101af0 0x7f8438198c00 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f8428031440 tx=0x7f8428004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:24.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f842e7fc640 1 -- 172.21.15.50:0/1953017149 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842803f070 con 0x7f8438101af0 2024-04-23T18:29:24.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f842e7fc640 1 -- 172.21.15.50:0/1953017149 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8428003ca0 con 0x7f8438101af0 2024-04-23T18:29:24.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843f4cb640 1 -- 172.21.15.50:0/1953017149 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f843819d940 con 0x7f8438101af0 2024-04-23T18:29:24.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f842e7fc640 1 -- 172.21.15.50:0/1953017149 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8428035070 con 0x7f8438101af0 2024-04-23T18:29:24.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.637+0000 7f843f4cb640 1 -- 172.21.15.50:0/1953017149 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84381a5230 con 0x7f8438101af0 2024-04-23T18:29:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.640+0000 7f842e7fc640 1 -- 172.21.15.50:0/1953017149 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f842803b4f0 con 0x7f8438101af0 2024-04-23T18:29:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.640+0000 7f84421d7740 1 Processor -- start 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.640+0000 7f84421d7740 1 -- start start 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.640+0000 7f84421d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f630a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.640+0000 7f84421d7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 0x55fc1f63b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.640+0000 7f84421d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f63b800 0x55fc1f63dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.640+0000 7f84421d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55fc1f595970 con 0x55fc1f63b800 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.640+0000 7f84421d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fc1f5324a0 con 0x55fc1f630620 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.640+0000 7f84421d7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fc1f591930 con 0x55fc1f630f40 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843d267640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 0x55fc1f63b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843da68640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f630a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843ca66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f63b800 0x55fc1f63dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843d267640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 0x55fc1f63b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:57708/0 (socket says 172.21.15.50:57708) 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843d267640 1 -- 172.21.15.50:0/374454760 learned_addr learned my addr 172.21.15.50:0/374454760 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843da68640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f630a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:45704/0 (socket says 172.21.15.50:45704) 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843d267640 1 -- 172.21.15.50:0/374454760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 msgr2=0x55fc1f630a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843d267640 1 --2- 172.21.15.50:0/374454760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f630a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843d267640 1 -- 172.21.15.50:0/374454760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f63b800 msgr2=0x55fc1f63dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843d267640 1 --2- 172.21.15.50:0/374454760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f63b800 0x55fc1f63dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f842e7fc640 1 --2- 172.21.15.50:0/1953017149 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f84240a3e40 0x7f84240a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843d267640 1 -- 172.21.15.50:0/374454760 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc1f2a8210 con 0x55fc1f630f40 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843da68640 1 --2- 172.21.15.50:0/1953017149 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f84240a3e40 0x7f84240a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f842e7fc640 1 -- 172.21.15.50:0/1953017149 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7f84280f0b20 con 0x7f8438101af0 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f843d267640 1 --2- 172.21.15.50:0/374454760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 0x55fc1f63b2c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8434009da0 tx=0x7f8434011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b970346405411c4f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f84237fe640 1 -- 172.21.15.50:0/374454760 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843401e070 con 0x55fc1f630f40 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f84237fe640 1 -- 172.21.15.50:0/374454760 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8434019ae0 con 0x55fc1f630f40 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.641+0000 7f84237fe640 1 -- 172.21.15.50:0/374454760 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843400fa90 con 0x55fc1f630f40 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.642+0000 7f84421d7740 1 -- 172.21.15.50:0/374454760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 msgr2=0x55fc1f63b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.642+0000 7f843da68640 1 --2- 172.21.15.50:0/1953017149 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f84240a3e40 0x7f84240a6300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8430005fd0 tx=0x7f84300096a0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.642+0000 7f84421d7740 1 --2- 172.21.15.50:0/374454760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 0x55fc1f63b2c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8434009da0 tx=0x7f8434011810 comp rx=0 tx=0).stop 2024-04-23T18:29:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.642+0000 7f84421d7740 1 -- 172.21.15.50:0/374454760 shutdown_connections 2024-04-23T18:29:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.642+0000 7f84421d7740 1 --2- 172.21.15.50:0/374454760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f63b800 0x55fc1f63dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.642+0000 7f84421d7740 1 --2- 172.21.15.50:0/374454760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 0x55fc1f63b2c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.642+0000 7f84421d7740 1 --2- 172.21.15.50:0/374454760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f630a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.642+0000 7f84421d7740 1 -- 172.21.15.50:0/374454760 >> 172.21.15.50:0/374454760 conn(0x55fc1f62cb30 msgr2=0x55fc1f62cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.642+0000 7f84421d7740 1 -- 172.21.15.50:0/374454760 shutdown_connections 2024-04-23T18:29:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.642+0000 7f84421d7740 1 -- 172.21.15.50:0/374454760 wait complete. 2024-04-23T18:29:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f84421d7740 1 -- start start 2024-04-23T18:29:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f84421d7740 1 Processor -- start 2024-04-23T18:29:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f84421d7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630620 0x55fc1f63b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f84421d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 0x55fc1f6383d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f84421d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f63b800 0x55fc1f638910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f84421d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55fc1f63e430 con 0x55fc1f630f40 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f84421d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fc1f5300b0 con 0x55fc1f63b800 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f84421d7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55fc1f63e5b0 con 0x55fc1f630620 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f84421d7740 10 client.0 _ll_register_callbacks cb 0x55fc1f51dd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f843ca66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 0x55fc1f6383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f843d267640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630620 0x55fc1f63b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f843d267640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630620 0x55fc1f63b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:57724/0 (socket says 172.21.15.50:57724) 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f843d267640 1 -- 172.21.15.50:0/2289331820 learned_addr learned my addr 172.21.15.50:0/2289331820 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.643+0000 7f843da68640 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f63b800 0x55fc1f638910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f843d267640 1 -- 172.21.15.50:0/2289331820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f63b800 msgr2=0x55fc1f638910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f843d267640 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f63b800 0x55fc1f638910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f843d267640 1 -- 172.21.15.50:0/2289331820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 msgr2=0x55fc1f6383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f843d267640 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 0x55fc1f6383d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f843d267640 1 -- 172.21.15.50:0/2289331820 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc1f374620 con 0x55fc1f630620 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f843ca66640 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 0x55fc1f6383d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f843da68640 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f63b800 0x55fc1f638910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f843d267640 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630620 0x55fc1f63b0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f843400f610 tx=0x7f8434019660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef53fae727ae8366 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f8420ff9640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f8420ff9640 1 -- 172.21.15.50:0/2289331820 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843401e070 con 0x55fc1f630620 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f84421d7740 1 -- 172.21.15.50:0/2289331820 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8434019ae0 con 0x55fc1f630620 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f8420ff9640 1 -- 172.21.15.50:0/2289331820 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7f8434005230 con 0x55fc1f630620 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f8420ff9640 1 -- 172.21.15.50:0/2289331820 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f8434031050 con 0x55fc1f630620 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f84421d7740 10 client.26404 fetch_fsmap learned FSMap version 44 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.644+0000 7f84421d7740 1 -- 172.21.15.50:0/2289331820 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55fc1f64a2d0 con 0x55fc1f630620 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.645+0000 7f8420ff9640 1 -- 172.21.15.50:0/2289331820 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8434019ae0 con 0x55fc1f630620 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.645+0000 7f8420ff9640 1 -- 172.21.15.50:0/2289331820 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7f84340046c0 con 0x55fc1f630620 2024-04-23T18:29:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.645+0000 7f84421d7740 10 client.26404 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:29:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.645+0000 7f84421d7740 10 client.26404 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.645+0000 7f84421d7740 20 client.26404 populate_metadata read hostname 'smithi050' 2024-04-23T18:29:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.645+0000 7f84421d7740 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fc1f64b6e0 0x55fc1f64dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.645+0000 7f84421d7740 4 client.26404 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:29:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.645+0000 7f84421d7740 1 -- 172.21.15.50:0/2289331820 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc1f5a3390 con 0x55fc1f64b6e0 2024-04-23T18:29:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.646+0000 7f843ca66640 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fc1f64b6e0 0x55fc1f64dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.648+0000 7f843ca66640 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fc1f64b6e0 0x55fc1f64dad0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72c1a6afc447fcd5 server_cookie=4390d2fb23d4b975 in_seq=0 out_seq=0 2024-04-23T18:29:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.648+0000 7f8420ff9640 10 client.26404 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.649+0000 7f8420ff9640 1 -- 172.21.15.50:0/2289331820 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc1f5a3390 con 0x55fc1f64b6e0 2024-04-23T18:29:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.649+0000 7f8420ff9640 10 client.26404 handle_command_reply: tid=0 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.649+0000 7f84421d7740 1 client.26404 shutdown 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.650+0000 7f84421d7740 1 -- 172.21.15.50:0/2289331820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630620 msgr2=0x55fc1f63b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.650+0000 7f84421d7740 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630620 0x55fc1f63b0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f843400f610 tx=0x7f8434019660 comp rx=0 tx=0).stop 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.650+0000 7f84421d7740 1 -- 172.21.15.50:0/2289331820 shutdown_connections 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.650+0000 7f84421d7740 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fc1f64b6e0 0x55fc1f64dad0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.650+0000 7f84421d7740 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f63b800 0x55fc1f638910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.650+0000 7f8420ff9640 0 client.26404 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.650+0000 7f84421d7740 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 0x55fc1f6383d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.650+0000 7f84421d7740 1 --2- 172.21.15.50:0/2289331820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630620 0x55fc1f63b0e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.650+0000 7f84421d7740 1 -- 172.21.15.50:0/2289331820 >> 172.21.15.50:0/2289331820 conn(0x55fc1f62cb30 msgr2=0x55fc1f632860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.650+0000 7f84421d7740 1 -- 172.21.15.50:0/2289331820 shutdown_connections 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.650+0000 7f84421d7740 1 -- 172.21.15.50:0/2289331820 wait complete. 2024-04-23T18:29:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.650+0000 7f84421d7740 20 client.26404 trim_cache size 0 max 16384 2024-04-23T18:29:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.665+0000 7f84421d7740 1 Processor -- start 2024-04-23T18:29:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f84421d7740 1 -- start start 2024-04-23T18:29:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f84421d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f6472f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f84421d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 0x55fc1f633af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f84421d7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f63b800 0x55fc1f62bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f84421d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f84381ae240 con 0x55fc1f630f40 2024-04-23T18:29:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f84421d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f84381abab0 con 0x55fc1f630620 2024-04-23T18:29:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f84421d7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f84381a9920 con 0x55fc1f63b800 2024-04-23T18:29:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f843ca66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 0x55fc1f633af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f843ca66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 0x55fc1f633af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56314/0 (socket says 172.21.15.50:56314) 2024-04-23T18:29:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f843ca66640 1 -- 172.21.15.50:0/275655254 learned_addr learned my addr 172.21.15.50:0/275655254 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f843d267640 1 --2- 172.21.15.50:0/275655254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f6472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f843da68640 1 --2- 172.21.15.50:0/275655254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f63b800 0x55fc1f62bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f843ca66640 1 -- 172.21.15.50:0/275655254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f63b800 msgr2=0x55fc1f62bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f843ca66640 1 --2- 172.21.15.50:0/275655254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f63b800 0x55fc1f62bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f843ca66640 1 -- 172.21.15.50:0/275655254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 msgr2=0x55fc1f6472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f843ca66640 1 --2- 172.21.15.50:0/275655254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f6472f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f843ca66640 1 -- 172.21.15.50:0/275655254 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc1f63dfb0 con 0x55fc1f630f40 2024-04-23T18:29:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f843ca66640 1 --2- 172.21.15.50:0/275655254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 0x55fc1f633af0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f84280ee2c0 tx=0x7f842803b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f702a027c0a93f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.666+0000 7f8422ffd640 1 -- 172.21.15.50:0/275655254 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8428048040 con 0x55fc1f630f40 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.667+0000 7f8422ffd640 1 -- 172.21.15.50:0/275655254 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84280f5040 con 0x55fc1f630f40 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.667+0000 7f8422ffd640 1 -- 172.21.15.50:0/275655254 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84280baef0 con 0x55fc1f630f40 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.667+0000 7f84421d7740 1 -- 172.21.15.50:0/275655254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 msgr2=0x55fc1f633af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.667+0000 7f84421d7740 1 --2- 172.21.15.50:0/275655254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 0x55fc1f633af0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f84280ee2c0 tx=0x7f842803b890 comp rx=0 tx=0).stop 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.667+0000 7f84421d7740 1 -- 172.21.15.50:0/275655254 shutdown_connections 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.667+0000 7f84421d7740 1 --2- 172.21.15.50:0/275655254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f63b800 0x55fc1f62bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.667+0000 7f84421d7740 1 --2- 172.21.15.50:0/275655254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f630f40 0x55fc1f633af0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.667+0000 7f84421d7740 1 --2- 172.21.15.50:0/275655254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f6472f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.667+0000 7f84421d7740 1 -- 172.21.15.50:0/275655254 >> 172.21.15.50:0/275655254 conn(0x55fc1f62cb30 msgr2=0x55fc1f62b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.667+0000 7f84421d7740 1 -- 172.21.15.50:0/275655254 shutdown_connections 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.667+0000 7f84421d7740 1 -- 172.21.15.50:0/275655254 wait complete. 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.667+0000 7f84421d7740 1 -- start start 2024-04-23T18:29:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f84421d7740 1 Processor -- start 2024-04-23T18:29:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f84421d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f63e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f84421d7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 0x55fc1f63ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f84421d7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f63b800 0x55fc1f63f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f84421d7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f84381a9f70 con 0x55fc1f63b800 2024-04-23T18:29:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f84421d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f84381ab3e0 con 0x55fc1f630620 2024-04-23T18:29:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f84421d7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f84381aa640 con 0x55fc1f630f40 2024-04-23T18:29:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f84421d7740 10 client.0 _ll_register_callbacks cb 0x7f84280f4540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f843d267640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f63e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f843da68640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f63b800 0x55fc1f63f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f843d267640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f63e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:45732/0 (socket says 172.21.15.50:45732) 2024-04-23T18:29:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f843d267640 1 -- 172.21.15.50:0/553660811 learned_addr learned my addr 172.21.15.50:0/553660811 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f843ca66640 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 0x55fc1f63ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f843da68640 1 -- 172.21.15.50:0/553660811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 msgr2=0x55fc1f63ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f843da68640 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 0x55fc1f63ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f843da68640 1 -- 172.21.15.50:0/553660811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 msgr2=0x55fc1f63e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f843da68640 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f63e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f843da68640 1 -- 172.21.15.50:0/553660811 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc1f6afae0 con 0x55fc1f63b800 2024-04-23T18:29:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f843ca66640 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 0x55fc1f63ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.668+0000 7f843d267640 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f63e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f843da68640 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f63b800 0x55fc1f63f240 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f8430012fd0 tx=0x7f8430004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7fe12da731ab0b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f8420ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:29:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f8420ff9640 1 -- 172.21.15.50:0/553660811 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843002c650 con 0x55fc1f63b800 2024-04-23T18:29:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f8420ff9640 1 -- 172.21.15.50:0/553660811 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7f843002c7f0 con 0x55fc1f63b800 2024-04-23T18:29:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f84421d7740 1 -- 172.21.15.50:0/553660811 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84280f5040 con 0x55fc1f63b800 2024-04-23T18:29:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f8420ff9640 1 -- 172.21.15.50:0/553660811 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f843003a050 con 0x55fc1f63b800 2024-04-23T18:29:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f84421d7740 10 client.16806 fetch_fsmap learned FSMap version 44 2024-04-23T18:29:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f84421d7740 1 -- 172.21.15.50:0/553660811 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55fc1f6b0d90 con 0x55fc1f63b800 2024-04-23T18:29:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f8420ff9640 1 -- 172.21.15.50:0/553660811 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84280f5040 con 0x55fc1f63b800 2024-04-23T18:29:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f8420ff9640 1 -- 172.21.15.50:0/553660811 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7f843001e970 con 0x55fc1f63b800 2024-04-23T18:29:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f84421d7740 10 client.16806 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:29:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f84421d7740 10 client.16806 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f84421d7740 20 client.16806 populate_metadata read hostname 'smithi050' 2024-04-23T18:29:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f84421d7740 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fc1f64b6e0 0x55fc1f6b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f84421d7740 4 client.16806 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:29:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.669+0000 7f84421d7740 1 -- 172.21.15.50:0/553660811 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fc1f59d200 con 0x55fc1f64b6e0 2024-04-23T18:29:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.670+0000 7f843d267640 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fc1f64b6e0 0x55fc1f6b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.670+0000 7f843d267640 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fc1f64b6e0 0x55fc1f6b35d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a774ce352d4b7347 server_cookie=b96d66f89553b8b9 in_seq=0 out_seq=0 2024-04-23T18:29:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.670+0000 7f8420ff9640 10 client.16806 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f8420ff9640 1 -- 172.21.15.50:0/553660811 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fc1f59d200 con 0x55fc1f64b6e0 2024-04-23T18:29:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f8420ff9640 10 client.16806 handle_command_reply: tid=0 2024-04-23T18:29:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f84421d7740 1 client.16806 shutdown 2024-04-23T18:29:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f84421d7740 1 -- 172.21.15.50:0/553660811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f63b800 msgr2=0x55fc1f63f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f84421d7740 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f63b800 0x55fc1f63f240 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f8430012fd0 tx=0x7f8430004a40 comp rx=0 tx=0).stop 2024-04-23T18:29:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f84421d7740 1 -- 172.21.15.50:0/553660811 shutdown_connections 2024-04-23T18:29:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f84421d7740 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55fc1f64b6e0 0x55fc1f6b35d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f84421d7740 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55fc1f63b800 0x55fc1f63f240 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f84421d7740 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55fc1f630f40 0x55fc1f63ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f84421d7740 1 --2- 172.21.15.50:0/553660811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc1f630620 0x55fc1f63e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f84421d7740 1 -- 172.21.15.50:0/553660811 >> 172.21.15.50:0/553660811 conn(0x55fc1f62cb30 msgr2=0x55fc1f637b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f8420ff9640 0 client.16806 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f84421d7740 1 -- 172.21.15.50:0/553660811 shutdown_connections 2024-04-23T18:29:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f84421d7740 1 -- 172.21.15.50:0/553660811 wait complete. 2024-04-23T18:29:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.723+0000 7f84421d7740 20 client.16806 trim_cache size 0 max 16384 2024-04-23T18:29:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:29:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:29:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:29:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:29:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.724+0000 7f843f4cb640 1 -- 172.21.15.50:0/1953017149 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f84240a3e40 msgr2=0x7f84240a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.724+0000 7f843f4cb640 1 --2- 172.21.15.50:0/1953017149 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f84240a3e40 0x7f84240a6300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8430005fd0 tx=0x7f84300096a0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.724+0000 7f843f4cb640 1 -- 172.21.15.50:0/1953017149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438101af0 msgr2=0x7f8438198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.724+0000 7f843f4cb640 1 --2- 172.21.15.50:0/1953017149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438101af0 0x7f8438198c00 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f8428031440 tx=0x7f8428004270 comp rx=0 tx=0).stop 2024-04-23T18:29:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.724+0000 7f843f4cb640 1 -- 172.21.15.50:0/1953017149 shutdown_connections 2024-04-23T18:29:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.724+0000 7f843f4cb640 1 --2- 172.21.15.50:0/1953017149 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f84240a3e40 0x7f84240a6300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.724+0000 7f843f4cb640 1 --2- 172.21.15.50:0/1953017149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8438106f90 0x7f843819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.724+0000 7f843f4cb640 1 --2- 172.21.15.50:0/1953017149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438101af0 0x7f8438198c00 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.724+0000 7f843f4cb640 1 --2- 172.21.15.50:0/1953017149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8438101130 0x7f84381986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.724+0000 7f843f4cb640 1 -- 172.21.15.50:0/1953017149 >> 172.21.15.50:0/1953017149 conn(0x7f8438077c30 msgr2=0x7f843806a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.724+0000 7f843f4cb640 1 -- 172.21.15.50:0/1953017149 shutdown_connections 2024-04-23T18:29:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:24.724+0000 7f843f4cb640 1 -- 172.21.15.50:0/1953017149 wait complete. 2024-04-23T18:29:25.230 INFO:tasks.workunit.client.0.smithi050.stdout: 1 680305 5.50 MB/sec execute 479 sec latency 214.757 ms 2024-04-23T18:29:26.230 INFO:tasks.workunit.client.0.smithi050.stdout: 1 682225 5.51 MB/sec execute 480 sec latency 13.022 ms 2024-04-23T18:29:26.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:26 smithi050 ceph-mon[29187]: pgmap v343: 97 pgs: 97 active+clean; 228 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-04-23T18:29:26.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:26 smithi177 ceph-mon[32308]: pgmap v343: 97 pgs: 97 active+clean; 228 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-04-23T18:29:26.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:26 smithi188 ceph-mon[32694]: pgmap v343: 97 pgs: 97 active+clean; 228 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-04-23T18:29:27.230 INFO:tasks.workunit.client.0.smithi050.stdout: 1 683549 5.51 MB/sec execute 481 sec latency 25.577 ms 2024-04-23T18:29:28.230 INFO:tasks.workunit.client.0.smithi050.stdout: 1 685707 5.53 MB/sec execute 482 sec latency 20.728 ms 2024-04-23T18:29:28.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:28 smithi050 ceph-mon[29187]: pgmap v344: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T18:29:28.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:28 smithi177 ceph-mon[32308]: pgmap v344: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T18:29:28.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:28 smithi188 ceph-mon[32694]: pgmap v344: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T18:29:29.230 INFO:tasks.workunit.client.0.smithi050.stdout: 1 686994 5.53 MB/sec execute 483 sec latency 23.333 ms 2024-04-23T18:29:29.677 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.4 2024-04-23T18:29:29.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:29:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.796+0000 7fac54f95640 1 -- 172.21.15.50:0/4026393270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac50101a50 msgr2=0x7fac50105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.796+0000 7fac54f95640 1 --2- 172.21.15.50:0/4026393270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac50101a50 0x7fac50105a90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fac44009860 tx=0x7fac4402a340 comp rx=0 tx=0).stop 2024-04-23T18:29:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.796+0000 7fac54f95640 1 -- 172.21.15.50:0/4026393270 shutdown_connections 2024-04-23T18:29:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.796+0000 7fac54f95640 1 --2- 172.21.15.50:0/4026393270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac50106170 0x7fac5010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.796+0000 7fac54f95640 1 --2- 172.21.15.50:0/4026393270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac50101a50 0x7fac50105a90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.796+0000 7fac54f95640 1 --2- 172.21.15.50:0/4026393270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac501010a0 0x7fac50101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.796+0000 7fac54f95640 1 -- 172.21.15.50:0/4026393270 >> 172.21.15.50:0/4026393270 conn(0x7fac50077f50 msgr2=0x7fac50078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.796+0000 7fac54f95640 1 -- 172.21.15.50:0/4026393270 shutdown_connections 2024-04-23T18:29:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.796+0000 7fac54f95640 1 -- 172.21.15.50:0/4026393270 wait complete. 2024-04-23T18:29:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac54f95640 1 Processor -- start 2024-04-23T18:29:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac54f95640 1 -- start start 2024-04-23T18:29:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac54f95640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac501010a0 0x7fac50197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac54f95640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac50106170 0x7fac50197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac54f95640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac5019bed0 0x7fac5019c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac54f95640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fac501110b0 con 0x7fac5019bed0 2024-04-23T18:29:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac54f95640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac50110f30 con 0x7fac501010a0 2024-04-23T18:29:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac54f95640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fac50111230 con 0x7fac50106170 2024-04-23T18:29:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac4ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac5019bed0 0x7fac5019c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac4dd9b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac50106170 0x7fac50197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac4dd9b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac50106170 0x7fac50197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:57744/0 (socket says 172.21.15.50:57744) 2024-04-23T18:29:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac4dd9b640 1 -- 172.21.15.50:0/1717221387 learned_addr learned my addr 172.21.15.50:0/1717221387 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.797+0000 7fac4e59c640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac501010a0 0x7fac50197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.798+0000 7fac4dd9b640 1 -- 172.21.15.50:0/1717221387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac501010a0 msgr2=0x7fac50197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.798+0000 7fac4dd9b640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac501010a0 0x7fac50197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.798+0000 7fac4dd9b640 1 -- 172.21.15.50:0/1717221387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac5019bed0 msgr2=0x7fac5019c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.798+0000 7fac4dd9b640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac5019bed0 0x7fac5019c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.798+0000 7fac4dd9b640 1 -- 172.21.15.50:0/1717221387 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5019c9c0 con 0x7fac50106170 2024-04-23T18:29:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.798+0000 7fac4ed9d640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac5019bed0 0x7fac5019c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.798+0000 7fac4dd9b640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac50106170 0x7fac50197b50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fac44000c00 tx=0x7fac44032d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.798+0000 7fac3f7fe640 1 -- 172.21.15.50:0/1717221387 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44040020 con 0x7fac50106170 2024-04-23T18:29:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.798+0000 7fac3f7fe640 1 -- 172.21.15.50:0/1717221387 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac44030d50 con 0x7fac50106170 2024-04-23T18:29:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.798+0000 7fac54f95640 1 -- 172.21.15.50:0/1717221387 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5019cc50 con 0x7fac50106170 2024-04-23T18:29:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.798+0000 7fac3f7fe640 1 -- 172.21.15.50:0/1717221387 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44037040 con 0x7fac50106170 2024-04-23T18:29:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.798+0000 7fac54f95640 1 -- 172.21.15.50:0/1717221387 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac50069370 con 0x7fac50106170 2024-04-23T18:29:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.799+0000 7fac54f95640 1 -- 172.21.15.50:0/1717221387 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac50068d70 con 0x7fac50106170 2024-04-23T18:29:29.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.800+0000 7fac3f7fe640 1 -- 172.21.15.50:0/1717221387 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fac44043020 con 0x7fac50106170 2024-04-23T18:29:29.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.801+0000 7fac3f7fe640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fac380a3d20 0x7fac380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:29.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.801+0000 7fac4e59c640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fac380a3d20 0x7fac380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:29.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.801+0000 7fac3f7fe640 1 -- 172.21.15.50:0/1717221387 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7fac440f08d0 con 0x7fac50106170 2024-04-23T18:29:29.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.801+0000 7fac4e59c640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fac380a3d20 0x7fac380a61e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fac48005fd0 tx=0x7fac48009260 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:29.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:29.805+0000 7fac3f7fe640 1 -- 172.21.15.50:0/1717221387 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac4404a020 con 0x7fac50106170 2024-04-23T18:29:30.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.057+0000 7fac54f95640 1 -- 172.21.15.50:0/1717221387 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac50198960 con 0x7fac50106170 2024-04-23T18:29:30.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.057+0000 7fac3f7fe640 1 -- 172.21.15.50:0/1717221387 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11881 (secure 0 0 0) 0x7fac50198960 con 0x7fac50106170 2024-04-23T18:29:30.050 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:29:30.051 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:28:45.479730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/4033440314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4033440314},{"type":"v1","addr":"172.21.15.177:6837","nonce":4033440314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:29:30.051 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 44 2024-04-23T18:29:30.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.059+0000 7fac54f95640 1 -- 172.21.15.50:0/1717221387 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fac380a3d20 msgr2=0x7fac380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.059+0000 7fac54f95640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fac380a3d20 0x7fac380a61e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fac48005fd0 tx=0x7fac48009260 comp rx=0 tx=0).stop 2024-04-23T18:29:30.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.059+0000 7fac54f95640 1 -- 172.21.15.50:0/1717221387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac50106170 msgr2=0x7fac50197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.059+0000 7fac54f95640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac50106170 0x7fac50197b50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fac44000c00 tx=0x7fac44032d40 comp rx=0 tx=0).stop 2024-04-23T18:29:30.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.060+0000 7fac54f95640 1 -- 172.21.15.50:0/1717221387 shutdown_connections 2024-04-23T18:29:30.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.060+0000 7fac54f95640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fac380a3d20 0x7fac380a61e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.060+0000 7fac54f95640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac5019bed0 0x7fac5019c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.060+0000 7fac54f95640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac50106170 0x7fac50197b50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.060+0000 7fac54f95640 1 --2- 172.21.15.50:0/1717221387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac501010a0 0x7fac50197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.060+0000 7fac54f95640 1 -- 172.21.15.50:0/1717221387 >> 172.21.15.50:0/1717221387 conn(0x7fac50077f50 msgr2=0x7fac500ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.060+0000 7fac54f95640 1 -- 172.21.15.50:0/1717221387 shutdown_connections 2024-04-23T18:29:30.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.060+0000 7fac54f95640 1 -- 172.21.15.50:0/1717221387 wait complete. 2024-04-23T18:29:30.064 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 78.4 --await-for 42.0' 2024-04-23T18:29:30.065 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 78.4 --await-for 42.0 2024-04-23T18:29:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.193+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1022949447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5ac107ff0 msgr2=0x7fb5ac1083d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.193+0000 7fb5b3b75640 1 --2- 172.21.15.50:0/1022949447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5ac107ff0 0x7fb5ac1083d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8009890 tx=0x7fb5a802a380 comp rx=0 tx=0).stop 2024-04-23T18:29:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.193+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1022949447 shutdown_connections 2024-04-23T18:29:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.193+0000 7fb5b3b75640 1 --2- 172.21.15.50:0/1022949447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5ac06acc0 0x7fb5ac10ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.193+0000 7fb5b3b75640 1 --2- 172.21.15.50:0/1022949447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5ac06a320 0x7fb5ac06a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.193+0000 7fb5b3b75640 1 --2- 172.21.15.50:0/1022949447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5ac107ff0 0x7fb5ac1083d0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.193+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1022949447 >> 172.21.15.50:0/1022949447 conn(0x7fb5ac076f20 msgr2=0x7fb5ac077330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.193+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1022949447 shutdown_connections 2024-04-23T18:29:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1022949447 wait complete. 2024-04-23T18:29:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b3b75640 1 Processor -- start 2024-04-23T18:29:30.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b3b75640 1 -- start start 2024-04-23T18:29:30.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b3b75640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5ac06a320 0x7fb5ac198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b3b75640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5ac06acc0 0x7fb5ac198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b3b75640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5ac107ff0 0x7fb5ac19cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b3b75640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1121a0 con 0x7fb5ac06acc0 2024-04-23T18:29:30.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b3b75640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac112020 con 0x7fb5ac06a320 2024-04-23T18:29:30.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b3b75640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac112320 con 0x7fb5ac107ff0 2024-04-23T18:29:30.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b1110640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5ac06acc0 0x7fb5ac198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b1110640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5ac06acc0 0x7fb5ac198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56366/0 (socket says 172.21.15.50:56366) 2024-04-23T18:29:30.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b1110640 1 -- 172.21.15.50:0/1208655004 learned_addr learned my addr 172.21.15.50:0/1208655004 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b1911640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5ac06a320 0x7fb5ac198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.194+0000 7fb5b2112640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5ac107ff0 0x7fb5ac19cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.195+0000 7fb5b1110640 1 -- 172.21.15.50:0/1208655004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5ac107ff0 msgr2=0x7fb5ac19cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.195+0000 7fb5b1110640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5ac107ff0 0x7fb5ac19cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.195+0000 7fb5b1110640 1 -- 172.21.15.50:0/1208655004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5ac06a320 msgr2=0x7fb5ac198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.195+0000 7fb5b1110640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5ac06a320 0x7fb5ac198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.195+0000 7fb5b1110640 1 -- 172.21.15.50:0/1208655004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac19d6e0 con 0x7fb5ac06acc0 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.195+0000 7fb5b1110640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5ac06acc0 0x7fb5ac198c60 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00e9c0 tx=0x7fb59c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.195+0000 7fb5a2ffd640 1 -- 172.21.15.50:0/1208655004 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c00ccc0 con 0x7fb5ac06acc0 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.195+0000 7fb5a2ffd640 1 -- 172.21.15.50:0/1208655004 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb59c0054d0 con 0x7fb5ac06acc0 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.195+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1208655004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ac19d9d0 con 0x7fb5ac06acc0 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.195+0000 7fb5a2ffd640 1 -- 172.21.15.50:0/1208655004 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c010430 con 0x7fb5ac06acc0 2024-04-23T18:29:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.195+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1208655004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ac06b970 con 0x7fb5ac06acc0 2024-04-23T18:29:30.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.198+0000 7fb5a2ffd640 1 -- 172.21.15.50:0/1208655004 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb59c004fb0 con 0x7fb5ac06acc0 2024-04-23T18:29:30.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.198+0000 7fb5b6885740 1 Processor -- start 2024-04-23T18:29:30.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.198+0000 7fb5b6885740 1 -- start start 2024-04-23T18:29:30.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b6885740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500ae480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b6885740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b6885740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x5617500bb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b6885740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561750013370 con 0x5617500ae9c0 2024-04-23T18:29:30.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b6885740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56174ffb02e0 con 0x5617500b9090 2024-04-23T18:29:30.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b6885740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56175000f330 con 0x5617500ae0a0 2024-04-23T18:29:30.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5a2ffd640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb5980a3d20 0x7fb5980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b1110640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x5617500bb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b1911640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b1911640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56382/0 (socket says 172.21.15.50:56382) 2024-04-23T18:29:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b1911640 1 -- 172.21.15.50:0/1315605254 learned_addr learned my addr 172.21.15.50:0/1315605254 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b2112640 1 --2- 172.21.15.50:0/1315605254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500ae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b2112640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb5980a3d20 0x7fb5980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5a2ffd640 1 -- 172.21.15.50:0/1208655004 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7fb59c0d0a50 con 0x7fb5ac06acc0 2024-04-23T18:29:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b1110640 1 -- 172.21.15.50:0/1315605254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 msgr2=0x5617500ae480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b1110640 1 --2- 172.21.15.50:0/1315605254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500ae480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b1110640 1 -- 172.21.15.50:0/1315605254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 msgr2=0x5617500b8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b1110640 1 --2- 172.21.15.50:0/1315605254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b1110640 1 -- 172.21.15.50:0/1315605254 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56174fdf1d80 con 0x5617500b9090 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b1911640 1 --2- 172.21.15.50:0/1315605254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b2112640 1 --2- 172.21.15.50:0/1315605254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500ae480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b1110640 1 --2- 172.21.15.50:0/1315605254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x5617500bb480 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0ce190 tx=0x7fb59c004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96f6e955fcc8db41 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.199+0000 7fb5b2112640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb5980a3d20 0x7fb5980a61e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4005fd0 tx=0x7fb5a4009600 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.200+0000 7fb597fff640 1 -- 172.21.15.50:0/1315605254 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c09c9e0 con 0x5617500b9090 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.200+0000 7fb597fff640 1 -- 172.21.15.50:0/1315605254 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb59c0a0d90 con 0x5617500b9090 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.200+0000 7fb597fff640 1 -- 172.21.15.50:0/1315605254 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c0a51c0 con 0x5617500b9090 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.200+0000 7fb5b6885740 1 -- 172.21.15.50:0/1315605254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 msgr2=0x5617500bb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.200+0000 7fb5b6885740 1 --2- 172.21.15.50:0/1315605254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x5617500bb480 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0ce190 tx=0x7fb59c004a50 comp rx=0 tx=0).stop 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.200+0000 7fb5b6885740 1 -- 172.21.15.50:0/1315605254 shutdown_connections 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.200+0000 7fb5b6885740 1 --2- 172.21.15.50:0/1315605254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x5617500bb480 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.200+0000 7fb5b6885740 1 --2- 172.21.15.50:0/1315605254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.200+0000 7fb5b6885740 1 --2- 172.21.15.50:0/1315605254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500ae480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.200+0000 7fb5b6885740 1 -- 172.21.15.50:0/1315605254 >> 172.21.15.50:0/1315605254 conn(0x5617500aa3c0 msgr2=0x5617500aa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.201+0000 7fb5b6885740 1 -- 172.21.15.50:0/1315605254 shutdown_connections 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.201+0000 7fb5b6885740 1 -- 172.21.15.50:0/1315605254 wait complete. 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.201+0000 7fb5b6885740 1 -- start start 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.201+0000 7fb5b6885740 1 Processor -- start 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.201+0000 7fb5b6885740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500b6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.201+0000 7fb5b6885740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.201+0000 7fb5b6885740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x5617500b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.201+0000 7fb5b6885740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5617500bbdc0 con 0x5617500ae9c0 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.201+0000 7fb5b6885740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56174ffadfb0 con 0x5617500b9090 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.201+0000 7fb5b6885740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5617500bbf40 con 0x5617500ae0a0 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.201+0000 7fb5b6885740 10 client.0 _ll_register_callbacks cb 0x56174ffad9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5b1911640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5b1911640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500b6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:57774/0 (socket says 172.21.15.50:57774) 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5b1911640 1 -- 172.21.15.50:0/1113448769 learned_addr learned my addr 172.21.15.50:0/1113448769 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5b1110640 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5b2112640 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x5617500b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5b1911640 1 -- 172.21.15.50:0/1113448769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 msgr2=0x5617500b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5b1911640 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x5617500b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5b1911640 1 -- 172.21.15.50:0/1113448769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 msgr2=0x5617500b65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5b1911640 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5b1911640 1 -- 172.21.15.50:0/1113448769 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56174fe52510 con 0x5617500ae0a0 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5b1911640 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500b6070 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8009890 tx=0x7fb5a8009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c5c59c094a12a8a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5957fa640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5957fa640 1 -- 172.21.15.50:0/1113448769 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8037040 con 0x5617500ae0a0 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5957fa640 1 -- 172.21.15.50:0/1113448769 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7fb5a8030450 con 0x5617500ae0a0 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.202+0000 7fb5b6885740 1 -- 172.21.15.50:0/1113448769 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb59c0a0d90 con 0x5617500ae0a0 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.203+0000 7fb5957fa640 1 -- 172.21.15.50:0/1113448769 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb5a804d050 con 0x5617500ae0a0 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.203+0000 7fb5b6885740 10 client.26428 fetch_fsmap learned FSMap version 44 2024-04-23T18:29:30.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.203+0000 7fb5b6885740 1 -- 172.21.15.50:0/1113448769 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x56174fce5670 con 0x5617500ae0a0 2024-04-23T18:29:30.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.203+0000 7fb5957fa640 1 -- 172.21.15.50:0/1113448769 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb59c0a0d90 con 0x5617500ae0a0 2024-04-23T18:29:30.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.203+0000 7fb5957fa640 1 -- 172.21.15.50:0/1113448769 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fb5a8032440 con 0x5617500ae0a0 2024-04-23T18:29:30.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.203+0000 7fb5b6885740 10 client.26428 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:29:30.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.203+0000 7fb5b6885740 10 client.26428 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:29:30.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.203+0000 7fb5b6885740 20 client.26428 populate_metadata read hostname 'smithi050' 2024-04-23T18:29:30.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.203+0000 7fb5b6885740 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5617500c93a0 0x5617500cb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.203+0000 7fb5b6885740 4 client.26428 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:29:30.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.203+0000 7fb5b6885740 1 -- 172.21.15.50:0/1113448769 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561750020d90 con 0x5617500c93a0 2024-04-23T18:29:30.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.203+0000 7fb5b1110640 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5617500c93a0 0x5617500cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.204+0000 7fb5957fa640 10 client.26428 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:30.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.204+0000 7fb5b1110640 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5617500c93a0 0x5617500cb860 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6c39d8c2e2ddce6 server_cookie=49899413288bd885 in_seq=0 out_seq=0 2024-04-23T18:29:30.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.205+0000 7fb5957fa640 1 -- 172.21.15.50:0/1113448769 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561750020d90 con 0x5617500c93a0 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.205+0000 7fb5957fa640 10 client.26428 handle_command_reply: tid=0 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.205+0000 7fb5b6885740 1 client.26428 shutdown 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.205+0000 7fb5b6885740 1 -- 172.21.15.50:0/1113448769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 msgr2=0x5617500b6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.205+0000 7fb5b6885740 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500b6070 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8009890 tx=0x7fb5a8009490 comp rx=0 tx=0).stop 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.206+0000 7fb5b6885740 1 -- 172.21.15.50:0/1113448769 shutdown_connections 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.206+0000 7fb5b6885740 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5617500c93a0 0x5617500cb860 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.206+0000 7fb5b6885740 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x5617500b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.206+0000 7fb5957fa640 0 client.26428 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.206+0000 7fb5b6885740 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.206+0000 7fb5b6885740 1 --2- 172.21.15.50:0/1113448769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500b6070 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.206+0000 7fb5b6885740 1 -- 172.21.15.50:0/1113448769 >> 172.21.15.50:0/1113448769 conn(0x5617500aa3c0 msgr2=0x5617500b01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.206+0000 7fb5b6885740 1 -- 172.21.15.50:0/1113448769 shutdown_connections 2024-04-23T18:29:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.206+0000 7fb5b6885740 1 -- 172.21.15.50:0/1113448769 wait complete. 2024-04-23T18:29:30.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.206+0000 7fb5b6885740 20 client.26428 trim_cache size 0 max 16384 2024-04-23T18:29:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b6885740 1 Processor -- start 2024-04-23T18:29:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b6885740 1 -- start start 2024-04-23T18:29:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b6885740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500ae0a0 0x5617500c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b6885740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae9c0 0x5617500d1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b6885740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500b9090 0x5617500d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b6885740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1b6af0 con 0x5617500b9090 2024-04-23T18:29:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b6885740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac072230 con 0x5617500ae0a0 2024-04-23T18:29:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b6885740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac0700a0 con 0x5617500ae9c0 2024-04-23T18:29:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b1110640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae9c0 0x5617500d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b2112640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500b9090 0x5617500d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b1110640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae9c0 0x5617500d1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:57788/0 (socket says 172.21.15.50:57788) 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b1110640 1 -- 172.21.15.50:0/3299070859 learned_addr learned my addr 172.21.15.50:0/3299070859 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b2112640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500b9090 0x5617500d1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56404/0 (socket says 172.21.15.50:56404) 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b1911640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500ae0a0 0x5617500c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b1911640 1 -- 172.21.15.50:0/3299070859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae9c0 msgr2=0x5617500d1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b1911640 1 --2- 172.21.15.50:0/3299070859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae9c0 0x5617500d1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b1911640 1 -- 172.21.15.50:0/3299070859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500b9090 msgr2=0x5617500d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b1911640 1 --2- 172.21.15.50:0/3299070859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500b9090 0x5617500d1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.225+0000 7fb5b1911640 1 -- 172.21.15.50:0/3299070859 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617500bba50 con 0x5617500ae0a0 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.226+0000 7fb5b1110640 1 --2- 172.21.15.50:0/3299070859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae9c0 0x5617500d1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.226+0000 7fb5b1911640 1 --2- 172.21.15.50:0/3299070859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500ae0a0 0x5617500c4c80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8000c00 tx=0x7fb5a80303b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5462c10f6d581a23 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.226+0000 7fb5977fe640 1 -- 172.21.15.50:0/3299070859 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8056070 con 0x5617500ae0a0 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.226+0000 7fb5977fe640 1 -- 172.21.15.50:0/3299070859 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5a8030900 con 0x5617500ae0a0 2024-04-23T18:29:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.226+0000 7fb5977fe640 1 -- 172.21.15.50:0/3299070859 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8037030 con 0x5617500ae0a0 2024-04-23T18:29:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 -- 172.21.15.50:0/3299070859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500ae0a0 msgr2=0x5617500c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 --2- 172.21.15.50:0/3299070859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500ae0a0 0x5617500c4c80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8000c00 tx=0x7fb5a80303b0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 -- 172.21.15.50:0/3299070859 shutdown_connections 2024-04-23T18:29:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 --2- 172.21.15.50:0/3299070859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500b9090 0x5617500d1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 --2- 172.21.15.50:0/3299070859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae9c0 0x5617500d1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 --2- 172.21.15.50:0/3299070859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500ae0a0 0x5617500c4c80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 -- 172.21.15.50:0/3299070859 >> 172.21.15.50:0/3299070859 conn(0x5617500aa3c0 msgr2=0x5617500a9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 -- 172.21.15.50:0/3299070859 shutdown_connections 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 -- 172.21.15.50:0/3299070859 wait complete. 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 -- start start 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 Processor -- start 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.227+0000 7fb5b6885740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x56175012e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b6885740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac0706f0 con 0x5617500ae9c0 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b6885740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac071b60 con 0x5617500b9090 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b6885740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac070dc0 con 0x5617500ae0a0 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b6885740 10 client.0 _ll_register_callbacks cb 0x7fb5a8005cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b1110640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b2112640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x56175012e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b1110640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56418/0 (socket says 172.21.15.50:56418) 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b1110640 1 -- 172.21.15.50:0/2297169239 learned_addr learned my addr 172.21.15.50:0/2297169239 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b1911640 1 --2- 172.21.15.50:0/2297169239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b2112640 1 -- 172.21.15.50:0/2297169239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 msgr2=0x5617500b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b2112640 1 --2- 172.21.15.50:0/2297169239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b2112640 1 -- 172.21.15.50:0/2297169239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 msgr2=0x5617500b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b2112640 1 --2- 172.21.15.50:0/2297169239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b2112640 1 -- 172.21.15.50:0/2297169239 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56175012eee0 con 0x5617500b9090 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b2112640 1 --2- 172.21.15.50:0/2297169239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x56175012e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x5617500d1bb0 tx=0x7fb5a4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9f1db48632237f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5957fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5957fa640 1 -- 172.21.15.50:0/2297169239 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4017b30 con 0x5617500b9090 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5957fa640 1 -- 172.21.15.50:0/2297169239 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7fb5a4017cd0 con 0x5617500b9090 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.228+0000 7fb5b6885740 1 -- 172.21.15.50:0/2297169239 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5a8030900 con 0x5617500b9090 2024-04-23T18:29:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.229+0000 7fb5957fa640 1 -- 172.21.15.50:0/2297169239 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb5a4004b20 con 0x5617500b9090 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.229+0000 7fb5b6885740 10 client.26384 fetch_fsmap learned FSMap version 44 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.229+0000 7fb5b6885740 1 -- 172.21.15.50:0/2297169239 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5617501301c0 con 0x5617500b9090 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.229+0000 7fb5957fa640 1 -- 172.21.15.50:0/2297169239 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5a8030900 con 0x5617500b9090 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.229+0000 7fb5957fa640 1 -- 172.21.15.50:0/2297169239 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fb5a400f040 con 0x5617500b9090 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.229+0000 7fb5b6885740 10 client.26384 fetch_fsmap finished waiting for FSMap version 44 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.229+0000 7fb5b6885740 10 client.26384 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.229+0000 7fb5b6885740 20 client.26384 populate_metadata read hostname 'smithi050' 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.229+0000 7fb5b6885740 1 --2- 172.21.15.50:0/2297169239 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561750130a80 0x561750132ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.229+0000 7fb5b6885740 4 client.26384 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 78.4, "await_for": 42.0}] 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.229+0000 7fb5b6885740 1 -- 172.21.15.50:0/2297169239 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 78.4, "await_for": 42.0}) -- 0x56175001ac00 con 0x561750130a80 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.229+0000 7fb5b1911640 1 --2- 172.21.15.50:0/2297169239 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561750130a80 0x561750132ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.230+0000 7fb5b1911640 1 --2- 172.21.15.50:0/2297169239 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561750130a80 0x561750132ea0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1b605d2a39bbd89 server_cookie=37b056dad56be8ba in_seq=0 out_seq=0 2024-04-23T18:29:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.230+0000 7fb5957fa640 10 client.26384 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:30.230 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688887 5.54 MB/sec execute 484 sec latency 13.266 ms 2024-04-23T18:29:30.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.299+0000 7fb5957fa640 1 -- 172.21.15.50:0/2297169239 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56175001ac00 con 0x561750130a80 2024-04-23T18:29:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.299+0000 7fb5957fa640 10 client.26384 handle_command_reply: tid=0 2024-04-23T18:29:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.299+0000 7fb5b6885740 1 client.26384 shutdown 2024-04-23T18:29:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.300+0000 7fb5b6885740 1 -- 172.21.15.50:0/2297169239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 msgr2=0x56175012e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.300+0000 7fb5b6885740 1 --2- 172.21.15.50:0/2297169239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x56175012e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x5617500d1bb0 tx=0x7fb5a4004a40 comp rx=0 tx=0).stop 2024-04-23T18:29:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.300+0000 7fb5b6885740 1 -- 172.21.15.50:0/2297169239 shutdown_connections 2024-04-23T18:29:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.300+0000 7fb5b6885740 1 --2- 172.21.15.50:0/2297169239 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561750130a80 0x561750132ea0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.300+0000 7fb5b6885740 1 --2- 172.21.15.50:0/2297169239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617500b9090 0x56175012e9a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.300+0000 7fb5b6885740 1 --2- 172.21.15.50:0/2297169239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5617500ae9c0 0x5617500b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.300+0000 7fb5b6885740 1 --2- 172.21.15.50:0/2297169239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5617500ae0a0 0x5617500b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.300+0000 7fb5957fa640 0 client.26384 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.300+0000 7fb5b6885740 1 -- 172.21.15.50:0/2297169239 >> 172.21.15.50:0/2297169239 conn(0x5617500aa3c0 msgr2=0x5617500b5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.300+0000 7fb5b6885740 1 -- 172.21.15.50:0/2297169239 shutdown_connections 2024-04-23T18:29:30.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.300+0000 7fb5b6885740 1 -- 172.21.15.50:0/2297169239 wait complete. 2024-04-23T18:29:30.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.300+0000 7fb5b6885740 20 client.26384 trim_cache size 0 max 16384 2024-04-23T18:29:30.296 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:29:30.296 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 44, 2024-04-23T18:29:30.297 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:29:30.297 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 34, 2024-04-23T18:29:30.297 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:29:30.297 INFO:teuthology.orchestra.run.smithi050.stdout: "99593c95": { 2024-04-23T18:29:30.298 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 34, 2024-04-23T18:29:30.298 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:29:30.299 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:29:30.299 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:29:30.299 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:29:30.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:29:30.300 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:29:30.300 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 78.4, 2024-04-23T18:29:30.300 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:29:30.300 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:29:30.301 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:29:30.301 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:29:30.301 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:29:30.302 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:29:30.302 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:29:30.302 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:29:30.302 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:29:30.302 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:29:30.302 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:29:30.302 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:29:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.301+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1208655004 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb5980a3d20 msgr2=0x7fb5980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.301+0000 7fb5b3b75640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb5980a3d20 0x7fb5980a61e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4005fd0 tx=0x7fb5a4009600 comp rx=0 tx=0).stop 2024-04-23T18:29:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.301+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1208655004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5ac06acc0 msgr2=0x7fb5ac198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.301+0000 7fb5b3b75640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5ac06acc0 0x7fb5ac198c60 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00e9c0 tx=0x7fb59c00ee80 comp rx=0 tx=0).stop 2024-04-23T18:29:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.301+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1208655004 shutdown_connections 2024-04-23T18:29:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.301+0000 7fb5b3b75640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb5980a3d20 0x7fb5980a61e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.301+0000 7fb5b3b75640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5ac107ff0 0x7fb5ac19cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.301+0000 7fb5b3b75640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5ac06acc0 0x7fb5ac198c60 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.301+0000 7fb5b3b75640 1 --2- 172.21.15.50:0/1208655004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5ac06a320 0x7fb5ac198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.301+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1208655004 >> 172.21.15.50:0/1208655004 conn(0x7fb5ac076f20 msgr2=0x7fb5ac0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.301+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1208655004 shutdown_connections 2024-04-23T18:29:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:30.301+0000 7fb5b3b75640 1 -- 172.21.15.50:0/1208655004 wait complete. 2024-04-23T18:29:30.309 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '99593c95', quiesce took 0.6 seconds. Will release after: 18.4 2024-04-23T18:29:30.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:30 smithi050 ceph-mon[29187]: pgmap v345: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-23T18:29:30.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:30 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1717221387' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:29:30.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:30 smithi177 ceph-mon[32308]: pgmap v345: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-23T18:29:30.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:30 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1717221387' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:29:30.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:30 smithi188 ceph-mon[32694]: pgmap v345: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-23T18:29:30.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:30 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1717221387' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:29:31.230 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.53 MB/sec execute 485 sec latency 984.853 ms 2024-04-23T18:29:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.52 MB/sec execute 486 sec latency 1984.957 ms 2024-04-23T18:29:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:32 smithi050 ceph-mon[29187]: pgmap v346: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-23T18:29:32.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:32 smithi177 ceph-mon[32308]: pgmap v346: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-23T18:29:32.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:32 smithi188 ceph-mon[32694]: pgmap v346: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-23T18:29:33.230 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.51 MB/sec execute 487 sec latency 2985.019 ms 2024-04-23T18:29:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.50 MB/sec execute 488 sec latency 3985.111 ms 2024-04-23T18:29:34.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:34 smithi050 ceph-mon[29187]: pgmap v347: 97 pgs: 97 active+clean; 235 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-04-23T18:29:34.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:34 smithi177 ceph-mon[32308]: pgmap v347: 97 pgs: 97 active+clean; 235 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-04-23T18:29:34.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:34 smithi188 ceph-mon[32694]: pgmap v347: 97 pgs: 97 active+clean; 235 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-04-23T18:29:35.231 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.49 MB/sec execute 489 sec latency 4985.205 ms 2024-04-23T18:29:36.231 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.48 MB/sec execute 490 sec latency 5985.300 ms 2024-04-23T18:29:36.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:36 smithi050 ceph-mon[29187]: pgmap v348: 97 pgs: 97 active+clean; 235 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 189 op/s 2024-04-23T18:29:36.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:36 smithi177 ceph-mon[32308]: pgmap v348: 97 pgs: 97 active+clean; 235 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 189 op/s 2024-04-23T18:29:36.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:36 smithi188 ceph-mon[32694]: pgmap v348: 97 pgs: 97 active+clean; 235 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 189 op/s 2024-04-23T18:29:37.231 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.47 MB/sec execute 491 sec latency 6985.406 ms 2024-04-23T18:29:37.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:37 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:37.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:37 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:37.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:37 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.823+0000 7fe44ddad640 1 -- 172.21.15.50:0/1733418190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe448107290 msgr2=0x7fe44810ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.823+0000 7fe44ddad640 1 --2- 172.21.15.50:0/1733418190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe448107290 0x7fe44810ee60 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fe440009ca0 tx=0x7fe44002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.824+0000 7fe44ddad640 1 -- 172.21.15.50:0/1733418190 shutdown_connections 2024-04-23T18:29:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.824+0000 7fe44ddad640 1 --2- 172.21.15.50:0/1733418190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe448107290 0x7fe44810ee60 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.824+0000 7fe44ddad640 1 --2- 172.21.15.50:0/1733418190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe448102b70 0x7fe448106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.824+0000 7fe44ddad640 1 --2- 172.21.15.50:0/1733418190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4481021c0 0x7fe4481025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.824+0000 7fe44ddad640 1 -- 172.21.15.50:0/1733418190 >> 172.21.15.50:0/1733418190 conn(0x7fe4480fda20 msgr2=0x7fe4480ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.824+0000 7fe44ddad640 1 -- 172.21.15.50:0/1733418190 shutdown_connections 2024-04-23T18:29:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.824+0000 7fe44ddad640 1 -- 172.21.15.50:0/1733418190 wait complete. 2024-04-23T18:29:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.824+0000 7fe44ddad640 1 Processor -- start 2024-04-23T18:29:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe44ddad640 1 -- start start 2024-04-23T18:29:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe44ddad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4481021c0 0x7fe448198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe44ddad640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe448107290 0x7fe448198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe44ddad640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe44819cf90 0x7fe44819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe44ddad640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe4481121e0 con 0x7fe4481021c0 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe44ddad640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe448112060 con 0x7fe44819cf90 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe44ddad640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe448112360 con 0x7fe448107290 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe4477fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4481021c0 0x7fe448198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe4477fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4481021c0 0x7fe448198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55652/0 (socket says 172.21.15.50:55652) 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe4477fe640 1 -- 172.21.15.50:0/950198553 learned_addr learned my addr 172.21.15.50:0/950198553 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe447fff640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe44819cf90 0x7fe44819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe4477fe640 1 -- 172.21.15.50:0/950198553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe448107290 msgr2=0x7fe448198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe4477fe640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe448107290 0x7fe448198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe4477fe640 1 -- 172.21.15.50:0/950198553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe44819cf90 msgr2=0x7fe44819d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe4477fe640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe44819cf90 0x7fe44819d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe4477fe640 1 -- 172.21.15.50:0/950198553 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe44819da80 con 0x7fe4481021c0 2024-04-23T18:29:37.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe447fff640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe44819cf90 0x7fe44819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:29:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.825+0000 7fe4477fe640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4481021c0 0x7fe448198720 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fe43800e9f0 tx=0x7fe43800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.826+0000 7fe444ff9640 1 -- 172.21.15.50:0/950198553 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43800cd80 con 0x7fe4481021c0 2024-04-23T18:29:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.826+0000 7fe444ff9640 1 -- 172.21.15.50:0/950198553 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4380108e0 con 0x7fe4481021c0 2024-04-23T18:29:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.826+0000 7fe44ddad640 1 -- 172.21.15.50:0/950198553 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe44819dd70 con 0x7fe4481021c0 2024-04-23T18:29:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.826+0000 7fe444ff9640 1 -- 172.21.15.50:0/950198553 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe438018980 con 0x7fe4481021c0 2024-04-23T18:29:37.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.826+0000 7fe44ddad640 1 -- 172.21.15.50:0/950198553 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4481a5560 con 0x7fe4481021c0 2024-04-23T18:29:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.828+0000 7fe444ff9640 1 -- 172.21.15.50:0/950198553 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe438010430 con 0x7fe4481021c0 2024-04-23T18:29:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.828+0000 7fe450b9d740 1 Processor -- start 2024-04-23T18:29:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.828+0000 7fe450b9d740 1 -- start start 2024-04-23T18:29:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe450b9d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 0x55602af6fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe450b9d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af6ff40 0x55602af7a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe450b9d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af7a800 0x55602af7cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe450b9d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55602aed4970 con 0x55602af6f620 2024-04-23T18:29:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe450b9d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55602ae714a0 con 0x55602af6ff40 2024-04-23T18:29:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe450b9d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55602aed0930 con 0x55602af7a800 2024-04-23T18:29:37.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe447fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 0x55602af6fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe444ff9640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe4300a3dc0 0x7fe4300a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe447fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 0x55602af6fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55654/0 (socket says 172.21.15.50:55654) 2024-04-23T18:29:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe447fff640 1 -- 172.21.15.50:0/918993908 learned_addr learned my addr 172.21.15.50:0/918993908 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe4477fe640 1 --2- 172.21.15.50:0/918993908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af6ff40 0x55602af7a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe446ffd640 1 --2- 172.21.15.50:0/918993908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af7a800 0x55602af7cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe447fff640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe4300a3dc0 0x7fe4300a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe444ff9640 1 -- 172.21.15.50:0/950198553 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7fe4380d0c50 con 0x7fe4481021c0 2024-04-23T18:29:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe447fff640 1 -- 172.21.15.50:0/918993908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af7a800 msgr2=0x55602af7cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe447fff640 1 --2- 172.21.15.50:0/918993908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af7a800 0x55602af7cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe447fff640 1 -- 172.21.15.50:0/918993908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af6ff40 msgr2=0x55602af7a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe447fff640 1 --2- 172.21.15.50:0/918993908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af6ff40 0x55602af7a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.829+0000 7fe447fff640 1 -- 172.21.15.50:0/918993908 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55602abe7210 con 0x55602af6f620 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.830+0000 7fe447fff640 1 --2- 172.21.15.50:0/918993908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 0x55602af6fa00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fe440005e30 tx=0x7fe440031da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90cfb4f225f6d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.830+0000 7fe42dffb640 1 -- 172.21.15.50:0/918993908 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440004b40 con 0x55602af6f620 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.830+0000 7fe42dffb640 1 -- 172.21.15.50:0/918993908 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe44004e070 con 0x55602af6f620 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.830+0000 7fe447fff640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe4300a3dc0 0x7fe4300a6280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe44004d000 tx=0x7fe44004b000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.830+0000 7fe42dffb640 1 -- 172.21.15.50:0/918993908 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44002f660 con 0x55602af6f620 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.830+0000 7fe450b9d740 1 -- 172.21.15.50:0/918993908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 msgr2=0x55602af6fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.830+0000 7fe450b9d740 1 --2- 172.21.15.50:0/918993908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 0x55602af6fa00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fe440005e30 tx=0x7fe440031da0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.831+0000 7fe450b9d740 1 -- 172.21.15.50:0/918993908 shutdown_connections 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.831+0000 7fe450b9d740 1 --2- 172.21.15.50:0/918993908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af7a800 0x55602af7cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.831+0000 7fe450b9d740 1 --2- 172.21.15.50:0/918993908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af6ff40 0x55602af7a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.831+0000 7fe450b9d740 1 --2- 172.21.15.50:0/918993908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 0x55602af6fa00 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.831+0000 7fe450b9d740 1 -- 172.21.15.50:0/918993908 >> 172.21.15.50:0/918993908 conn(0x55602af6bb30 msgr2=0x55602af6bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.831+0000 7fe450b9d740 1 -- 172.21.15.50:0/918993908 shutdown_connections 2024-04-23T18:29:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.831+0000 7fe450b9d740 1 -- 172.21.15.50:0/918993908 wait complete. 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.831+0000 7fe450b9d740 1 -- start start 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.831+0000 7fe450b9d740 1 Processor -- start 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.831+0000 7fe450b9d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af6f620 0x55602af7a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.831+0000 7fe450b9d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 0x55602af774b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe450b9d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af7a800 0x55602af77ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe450b9d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55602af7d430 con 0x55602af6ff40 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe450b9d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55602ae6f0b0 con 0x55602af7a800 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe450b9d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55602af7d5b0 con 0x55602af6f620 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe450b9d740 10 client.0 _ll_register_callbacks cb 0x55602ae5cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe446ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 0x55602af774b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe446ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 0x55602af774b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55658/0 (socket says 172.21.15.50:55658) 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe446ffd640 1 -- 172.21.15.50:0/1069959514 learned_addr learned my addr 172.21.15.50:0/1069959514 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe4477fe640 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af6f620 0x55602af7a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe447fff640 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af7a800 0x55602af77ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe446ffd640 1 -- 172.21.15.50:0/1069959514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af6f620 msgr2=0x55602af7a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe446ffd640 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af6f620 0x55602af7a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe446ffd640 1 -- 172.21.15.50:0/1069959514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af7a800 msgr2=0x55602af77ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe446ffd640 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af7a800 0x55602af77ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe446ffd640 1 -- 172.21.15.50:0/1069959514 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55602acb3620 con 0x55602af6ff40 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe446ffd640 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 0x55602af774b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe43c011470 tx=0x7fe43c011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18d58a7e73e89f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe426ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe426ffd640 1 -- 172.21.15.50:0/1069959514 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c005020 con 0x55602af6ff40 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe426ffd640 1 -- 172.21.15.50:0/1069959514 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7fe43c0051c0 con 0x55602af6ff40 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.832+0000 7fe450b9d740 1 -- 172.21.15.50:0/1069959514 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe44004e070 con 0x55602af6ff40 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.833+0000 7fe426ffd640 1 -- 172.21.15.50:0/1069959514 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe43c0053c0 con 0x55602af6ff40 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.833+0000 7fe450b9d740 10 client.16842 fetch_fsmap learned FSMap version 45 2024-04-23T18:29:37.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.833+0000 7fe450b9d740 1 -- 172.21.15.50:0/1069959514 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55602af89470 con 0x55602af6ff40 2024-04-23T18:29:37.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.833+0000 7fe426ffd640 1 -- 172.21.15.50:0/1069959514 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe44004e070 con 0x55602af6ff40 2024-04-23T18:29:37.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.833+0000 7fe426ffd640 1 -- 172.21.15.50:0/1069959514 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7fe43c0036f0 con 0x55602af6ff40 2024-04-23T18:29:37.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.833+0000 7fe450b9d740 10 client.16842 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:29:37.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.833+0000 7fe450b9d740 10 client.16842 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:37.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.833+0000 7fe450b9d740 20 client.16842 populate_metadata read hostname 'smithi050' 2024-04-23T18:29:37.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.833+0000 7fe450b9d740 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55602af8a9f0 0x55602af8ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.833+0000 7fe450b9d740 4 client.16842 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:29:37.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.833+0000 7fe450b9d740 1 -- 172.21.15.50:0/1069959514 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55602aee2390 con 0x55602af8a9f0 2024-04-23T18:29:37.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.833+0000 7fe4477fe640 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55602af8a9f0 0x55602af8ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.834+0000 7fe4477fe640 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55602af8a9f0 0x55602af8ce30 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24ff0a56b403e9c0 server_cookie=302d9aef43780c73 in_seq=0 out_seq=0 2024-04-23T18:29:37.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.834+0000 7fe426ffd640 10 client.16842 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:37.832 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe426ffd640 1 -- 172.21.15.50:0/1069959514 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55602aee2390 con 0x55602af8a9f0 2024-04-23T18:29:37.832 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe426ffd640 10 client.16842 handle_command_reply: tid=0 2024-04-23T18:29:37.832 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe450b9d740 1 client.16842 shutdown 2024-04-23T18:29:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe450b9d740 1 -- 172.21.15.50:0/1069959514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 msgr2=0x55602af774b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe450b9d740 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 0x55602af774b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe43c011470 tx=0x7fe43c011930 comp rx=0 tx=0).stop 2024-04-23T18:29:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe450b9d740 1 -- 172.21.15.50:0/1069959514 shutdown_connections 2024-04-23T18:29:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe450b9d740 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55602af8a9f0 0x55602af8ce30 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe450b9d740 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af7a800 0x55602af77ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe450b9d740 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 0x55602af774b0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe426ffd640 0 client.16842 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe450b9d740 1 --2- 172.21.15.50:0/1069959514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af6f620 0x55602af7a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe450b9d740 1 -- 172.21.15.50:0/1069959514 >> 172.21.15.50:0/1069959514 conn(0x55602af6bb30 msgr2=0x55602af718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe450b9d740 1 -- 172.21.15.50:0/1069959514 shutdown_connections 2024-04-23T18:29:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe450b9d740 1 -- 172.21.15.50:0/1069959514 wait complete. 2024-04-23T18:29:37.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.835+0000 7fe450b9d740 20 client.16842 trim_cache size 0 max 16384 2024-04-23T18:29:37.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.850+0000 7fe450b9d740 1 Processor -- start 2024-04-23T18:29:37.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.850+0000 7fe450b9d740 1 -- start start 2024-04-23T18:29:37.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe450b9d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af6f620 0x55602af862f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe450b9d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 0x55602af72260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe450b9d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af7a800 0x55602af79cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe450b9d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe4481b0560 con 0x55602af6ff40 2024-04-23T18:29:37.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe450b9d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4481adda0 con 0x55602af7a800 2024-04-23T18:29:37.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe450b9d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe4481abc10 con 0x55602af6f620 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe446ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 0x55602af72260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe446ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 0x55602af72260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55672/0 (socket says 172.21.15.50:55672) 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe446ffd640 1 -- 172.21.15.50:0/3866330688 learned_addr learned my addr 172.21.15.50:0/3866330688 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe446ffd640 1 -- 172.21.15.50:0/3866330688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af6f620 msgr2=0x55602af862f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe446ffd640 1 --2- 172.21.15.50:0/3866330688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af6f620 0x55602af862f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe446ffd640 1 -- 172.21.15.50:0/3866330688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af7a800 msgr2=0x55602af79cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe447fff640 1 --2- 172.21.15.50:0/3866330688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af7a800 0x55602af79cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe446ffd640 1 --2- 172.21.15.50:0/3866330688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af7a800 0x55602af79cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe446ffd640 1 -- 172.21.15.50:0/3866330688 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55602af6ee20 con 0x55602af6ff40 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe447fff640 1 --2- 172.21.15.50:0/3866330688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af7a800 0x55602af79cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe446ffd640 1 --2- 172.21.15.50:0/3866330688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 0x55602af72260 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe43c004020 tx=0x7fe43c003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e60e1d65b926c51 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe42d7fa640 1 -- 172.21.15.50:0/3866330688 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c030070 con 0x55602af6ff40 2024-04-23T18:29:37.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe42d7fa640 1 -- 172.21.15.50:0/3866330688 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe43c005bf0 con 0x55602af6ff40 2024-04-23T18:29:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.851+0000 7fe42d7fa640 1 -- 172.21.15.50:0/3866330688 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c00f990 con 0x55602af6ff40 2024-04-23T18:29:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.852+0000 7fe450b9d740 1 -- 172.21.15.50:0/3866330688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 msgr2=0x55602af72260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.852+0000 7fe450b9d740 1 --2- 172.21.15.50:0/3866330688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 0x55602af72260 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe43c004020 tx=0x7fe43c003ee0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.852+0000 7fe450b9d740 1 -- 172.21.15.50:0/3866330688 shutdown_connections 2024-04-23T18:29:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.852+0000 7fe450b9d740 1 --2- 172.21.15.50:0/3866330688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af7a800 0x55602af79cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.852+0000 7fe450b9d740 1 --2- 172.21.15.50:0/3866330688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6ff40 0x55602af72260 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.852+0000 7fe450b9d740 1 --2- 172.21.15.50:0/3866330688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af6f620 0x55602af862f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.852+0000 7fe450b9d740 1 -- 172.21.15.50:0/3866330688 >> 172.21.15.50:0/3866330688 conn(0x55602af6bb30 msgr2=0x55602af764e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.852+0000 7fe450b9d740 1 -- 172.21.15.50:0/3866330688 shutdown_connections 2024-04-23T18:29:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.852+0000 7fe450b9d740 1 -- 172.21.15.50:0/3866330688 wait complete. 2024-04-23T18:29:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.852+0000 7fe450b9d740 1 -- start start 2024-04-23T18:29:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe450b9d740 1 Processor -- start 2024-04-23T18:29:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe450b9d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 0x55602af7e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe450b9d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af6ff40 0x55602af7e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe450b9d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af7a800 0x55602af7ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe450b9d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe4481ac260 con 0x55602af6f620 2024-04-23T18:29:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe450b9d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4481ad6d0 con 0x55602af6ff40 2024-04-23T18:29:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe450b9d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe4481ac930 con 0x55602af7a800 2024-04-23T18:29:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe450b9d740 10 client.0 _ll_register_callbacks cb 0x7fe43c009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe4477fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 0x55602af7e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe4477fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 0x55602af7e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55684/0 (socket says 172.21.15.50:55684) 2024-04-23T18:29:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe4477fe640 1 -- 172.21.15.50:0/374617174 learned_addr learned my addr 172.21.15.50:0/374617174 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe4477fe640 1 -- 172.21.15.50:0/374617174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af7a800 msgr2=0x55602af7ec00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe4477fe640 1 --2- 172.21.15.50:0/374617174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af7a800 0x55602af7ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe4477fe640 1 -- 172.21.15.50:0/374617174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af6ff40 msgr2=0x55602af7e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe4477fe640 1 --2- 172.21.15.50:0/374617174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af6ff40 0x55602af7e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe4477fe640 1 -- 172.21.15.50:0/374617174 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55602afeff70 con 0x55602af6f620 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe4477fe640 1 --2- 172.21.15.50:0/374617174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 0x55602af7e180 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fe4380995b0 tx=0x7fe43806c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c01fdeffd868bf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.853+0000 7fe426ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe426ffd640 1 -- 172.21.15.50:0/374617174 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4380a0f90 con 0x55602af6f620 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe426ffd640 1 -- 172.21.15.50:0/374617174 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7fe4380ce610 con 0x55602af6f620 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe450b9d740 1 -- 172.21.15.50:0/374617174 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe43c005bf0 con 0x55602af6f620 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe426ffd640 1 -- 172.21.15.50:0/374617174 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe4380a5850 con 0x55602af6f620 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe450b9d740 10 client.16848 fetch_fsmap learned FSMap version 45 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe450b9d740 1 -- 172.21.15.50:0/374617174 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55602aff1190 con 0x55602af6f620 2024-04-23T18:29:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe426ffd640 1 -- 172.21.15.50:0/374617174 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe43c005bf0 con 0x55602af6f620 2024-04-23T18:29:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe426ffd640 1 -- 172.21.15.50:0/374617174 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7fe438018b20 con 0x55602af6f620 2024-04-23T18:29:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe450b9d740 10 client.16848 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:29:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe450b9d740 10 client.16848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe450b9d740 20 client.16848 populate_metadata read hostname 'smithi050' 2024-04-23T18:29:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe450b9d740 1 --2- 172.21.15.50:0/374617174 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55602aff18d0 0x55602aff3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe450b9d740 4 client.16848 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:29:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.854+0000 7fe450b9d740 1 -- 172.21.15.50:0/374617174 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55602aedc200 con 0x55602aff18d0 2024-04-23T18:29:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.856+0000 7fe446ffd640 1 --2- 172.21.15.50:0/374617174 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55602aff18d0 0x55602aff3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:37.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.860+0000 7fe446ffd640 1 --2- 172.21.15.50:0/374617174 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55602aff18d0 0x55602aff3d10 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f81242770d56e5b server_cookie=1114c2530f70f82a in_seq=0 out_seq=0 2024-04-23T18:29:37.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.860+0000 7fe426ffd640 10 client.16848 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.901+0000 7fe426ffd640 1 -- 172.21.15.50:0/374617174 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55602aedc200 con 0x55602aff18d0 2024-04-23T18:29:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.901+0000 7fe426ffd640 10 client.16848 handle_command_reply: tid=0 2024-04-23T18:29:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.901+0000 7fe450b9d740 1 client.16848 shutdown 2024-04-23T18:29:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe450b9d740 1 -- 172.21.15.50:0/374617174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 msgr2=0x55602af7e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe450b9d740 1 --2- 172.21.15.50:0/374617174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 0x55602af7e180 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fe4380995b0 tx=0x7fe43806c580 comp rx=0 tx=0).stop 2024-04-23T18:29:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe450b9d740 1 -- 172.21.15.50:0/374617174 shutdown_connections 2024-04-23T18:29:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe450b9d740 1 --2- 172.21.15.50:0/374617174 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55602aff18d0 0x55602aff3d10 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe450b9d740 1 --2- 172.21.15.50:0/374617174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55602af7a800 0x55602af7ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe450b9d740 1 --2- 172.21.15.50:0/374617174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55602af6ff40 0x55602af7e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe450b9d740 1 --2- 172.21.15.50:0/374617174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55602af6f620 0x55602af7e180 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe426ffd640 0 client.16848 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe450b9d740 1 -- 172.21.15.50:0/374617174 >> 172.21.15.50:0/374617174 conn(0x55602af6bb30 msgr2=0x55602af6a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe450b9d740 1 -- 172.21.15.50:0/374617174 shutdown_connections 2024-04-23T18:29:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe450b9d740 1 -- 172.21.15.50:0/374617174 wait complete. 2024-04-23T18:29:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe450b9d740 20 client.16848 trim_cache size 0 max 16384 2024-04-23T18:29:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:29:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:29:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:29:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:29:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe44ddad640 1 -- 172.21.15.50:0/950198553 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe4300a3dc0 msgr2=0x7fe4300a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe44ddad640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe4300a3dc0 0x7fe4300a6280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe44004d000 tx=0x7fe44004b000 comp rx=0 tx=0).stop 2024-04-23T18:29:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe44ddad640 1 -- 172.21.15.50:0/950198553 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4481021c0 msgr2=0x7fe448198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe44ddad640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4481021c0 0x7fe448198720 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fe43800e9f0 tx=0x7fe43800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe44ddad640 1 -- 172.21.15.50:0/950198553 shutdown_connections 2024-04-23T18:29:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe44ddad640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe4300a3dc0 0x7fe4300a6280 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe44ddad640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe44819cf90 0x7fe44819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe44ddad640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe448107290 0x7fe448198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe44ddad640 1 --2- 172.21.15.50:0/950198553 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4481021c0 0x7fe448198720 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.902+0000 7fe44ddad640 1 -- 172.21.15.50:0/950198553 >> 172.21.15.50:0/950198553 conn(0x7fe4480fda20 msgr2=0x7fe4480ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.903+0000 7fe44ddad640 1 -- 172.21.15.50:0/950198553 shutdown_connections 2024-04-23T18:29:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:37.903+0000 7fe44ddad640 1 -- 172.21.15.50:0/950198553 wait complete. 2024-04-23T18:29:38.231 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.45 MB/sec execute 492 sec latency 7985.489 ms 2024-04-23T18:29:38.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:38 smithi050 ceph-mon[29187]: pgmap v349: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-04-23T18:29:38.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:38 smithi177 ceph-mon[32308]: pgmap v349: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-04-23T18:29:38.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:38 smithi188 ceph-mon[32694]: pgmap v349: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-04-23T18:29:39.231 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.44 MB/sec execute 493 sec latency 8985.639 ms 2024-04-23T18:29:40.231 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.43 MB/sec execute 494 sec latency 9985.752 ms 2024-04-23T18:29:40.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:40 smithi050 ceph-mon[29187]: pgmap v350: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-04-23T18:29:40.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:40 smithi177 ceph-mon[32308]: pgmap v350: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-04-23T18:29:40.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:40 smithi188 ceph-mon[32694]: pgmap v350: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-04-23T18:29:41.231 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.42 MB/sec execute 495 sec latency 10985.822 ms 2024-04-23T18:29:42.231 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.41 MB/sec execute 496 sec latency 11985.944 ms 2024-04-23T18:29:42.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:42 smithi050 ceph-mon[29187]: pgmap v351: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-04-23T18:29:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:42 smithi177 ceph-mon[32308]: pgmap v351: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-04-23T18:29:42.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:42 smithi188 ceph-mon[32694]: pgmap v351: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.7 MiB/s wr, 113 op/s 2024-04-23T18:29:43.231 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.40 MB/sec execute 497 sec latency 12986.103 ms 2024-04-23T18:29:44.231 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.39 MB/sec execute 498 sec latency 13986.142 ms 2024-04-23T18:29:44.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:44 smithi050 ceph-mon[29187]: pgmap v352: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.8 MiB/s wr, 139 op/s 2024-04-23T18:29:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:44 smithi177 ceph-mon[32308]: pgmap v352: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.8 MiB/s wr, 139 op/s 2024-04-23T18:29:44.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:44 smithi188 ceph-mon[32694]: pgmap v352: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.8 MiB/s wr, 139 op/s 2024-04-23T18:29:45.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.38 MB/sec execute 499 sec latency 14986.253 ms 2024-04-23T18:29:46.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.37 MB/sec execute 500 sec latency 15986.380 ms 2024-04-23T18:29:46.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:46 smithi050 ceph-mon[29187]: pgmap v353: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 659 KiB/s wr, 53 op/s 2024-04-23T18:29:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:46 smithi177 ceph-mon[32308]: pgmap v353: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 659 KiB/s wr, 53 op/s 2024-04-23T18:29:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:46 smithi188 ceph-mon[32694]: pgmap v353: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 659 KiB/s wr, 53 op/s 2024-04-23T18:29:47.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.36 MB/sec execute 501 sec latency 16986.444 ms 2024-04-23T18:29:47.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:47 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:29:47.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:47 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:29:47.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:47 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:29:48.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688890 5.35 MB/sec execute 502 sec latency 17986.489 ms 2024-04-23T18:29:48.437 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:48 smithi177 ceph-mon[32308]: pgmap v354: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-23T18:29:48.437 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:48 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:29:48.437 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:48 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:29:48.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:48 smithi050 ceph-mon[29187]: pgmap v354: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-23T18:29:48.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:48 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:29:48.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:48 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:29:48.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:48 smithi188 ceph-mon[32694]: pgmap v354: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-23T18:29:48.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:48 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:29:48.480 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:48 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:29:48.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:29:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.823+0000 7f6fab233640 1 -- 172.21.15.50:0/2314468095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa4102bf0 msgr2=0x7f6fa4106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.823+0000 7f6fab233640 1 --2- 172.21.15.50:0/2314468095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa4102bf0 0x7f6fa4106ba0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009900 tx=0x7f6f8c02a410 comp rx=0 tx=0).stop 2024-04-23T18:29:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.823+0000 7f6fab233640 1 -- 172.21.15.50:0/2314468095 shutdown_connections 2024-04-23T18:29:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.823+0000 7f6fab233640 1 --2- 172.21.15.50:0/2314468095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fa4107280 0x7f6fa410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.823+0000 7f6fab233640 1 --2- 172.21.15.50:0/2314468095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa4102bf0 0x7f6fa4106ba0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.823+0000 7f6fab233640 1 --2- 172.21.15.50:0/2314468095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fa4102240 0x7f6fa4102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.823+0000 7f6fab233640 1 -- 172.21.15.50:0/2314468095 >> 172.21.15.50:0/2314468095 conn(0x7f6fa40790e0 msgr2=0x7f6fa40794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.823+0000 7f6fab233640 1 -- 172.21.15.50:0/2314468095 shutdown_connections 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.823+0000 7f6fab233640 1 -- 172.21.15.50:0/2314468095 wait complete. 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.823+0000 7f6fab233640 1 Processor -- start 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.823+0000 7f6fab233640 1 -- start start 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fab233640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fa4102240 0x7f6fa41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fab233640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fa4107280 0x7f6fa4198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fab233640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa419cfa0 0x7f6fa419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fab233640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa41121c0 con 0x7f6fa419cfa0 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fab233640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa4112040 con 0x7f6fa4102240 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fab233640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa4112340 con 0x7f6fa4107280 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa97d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa419cfa0 0x7f6fa419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa97d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa419cfa0 0x7f6fa419d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56360/0 (socket says 172.21.15.50:56360) 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa97d0640 1 -- 172.21.15.50:0/689216040 learned_addr learned my addr 172.21.15.50:0/689216040 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa3fff640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fa4107280 0x7f6fa4198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa8fcf640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fa4102240 0x7f6fa41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa97d0640 1 -- 172.21.15.50:0/689216040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fa4107280 msgr2=0x7f6fa4198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa97d0640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fa4107280 0x7f6fa4198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa97d0640 1 -- 172.21.15.50:0/689216040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fa4102240 msgr2=0x7f6fa41986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa97d0640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fa4102240 0x7f6fa41986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa97d0640 1 -- 172.21.15.50:0/689216040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa419da90 con 0x7f6fa419cfa0 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa97d0640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa419cfa0 0x7f6fa419d450 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00cce0 tx=0x7f6f9c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa1ffb640 1 -- 172.21.15.50:0/689216040 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c00f450 con 0x7f6fa419cfa0 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fab233640 1 -- 172.21.15.50:0/689216040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa419dd80 con 0x7f6fa419cfa0 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa1ffb640 1 -- 172.21.15.50:0/689216040 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f9c00faa0 con 0x7f6fa419cfa0 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.824+0000 7f6fa1ffb640 1 -- 172.21.15.50:0/689216040 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c018b70 con 0x7f6fa419cfa0 2024-04-23T18:29:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.825+0000 7f6fab233640 1 -- 172.21.15.50:0/689216040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa41a54f0 con 0x7f6fa419cfa0 2024-04-23T18:29:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.826+0000 7f6fab233640 1 -- 172.21.15.50:0/689216040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa40691c0 con 0x7f6fa419cfa0 2024-04-23T18:29:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.827+0000 7f6fa1ffb640 1 -- 172.21.15.50:0/689216040 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6f9c018d10 con 0x7f6fa419cfa0 2024-04-23T18:29:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.827+0000 7f6fa1ffb640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6f900a3d20 0x7f6f900a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.827+0000 7f6fa1ffb640 1 -- 172.21.15.50:0/689216040 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7f6f9c0d0b50 con 0x7f6fa419cfa0 2024-04-23T18:29:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.827+0000 7f6fa8fcf640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6f900a3d20 0x7f6f900a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.828+0000 7f6fa8fcf640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6f900a3d20 0x7f6f900a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6f94009bd0 tx=0x7f6f94005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:48.833+0000 7f6fa1ffb640 1 -- 172.21.15.50:0/689216040 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6f9c014030 con 0x7f6fa419cfa0 2024-04-23T18:29:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.057+0000 7f6fab233640 1 -- 172.21.15.50:0/689216040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fa4102bf0 con 0x7f6fa419cfa0 2024-04-23T18:29:49.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.057+0000 7f6fa1ffb640 1 -- 172.21.15.50:0/689216040 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11882 (secure 0 0 0) 0x7f6f9c09d1e0 con 0x7f6fa419cfa0 2024-04-23T18:29:49.051 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:29:49.051 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:29:35.481028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/4033440314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4033440314},{"type":"v1","addr":"172.21.15.177:6837","nonce":4033440314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:29:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 45 2024-04-23T18:29:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.060+0000 7f6fab233640 1 -- 172.21.15.50:0/689216040 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6f900a3d20 msgr2=0x7f6f900a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.060+0000 7f6fab233640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6f900a3d20 0x7f6f900a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6f94009bd0 tx=0x7f6f94005f90 comp rx=0 tx=0).stop 2024-04-23T18:29:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.060+0000 7f6fab233640 1 -- 172.21.15.50:0/689216040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa419cfa0 msgr2=0x7f6fa419d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.060+0000 7f6fab233640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa419cfa0 0x7f6fa419d450 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00cce0 tx=0x7f6f9c00a620 comp rx=0 tx=0).stop 2024-04-23T18:29:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.060+0000 7f6fab233640 1 -- 172.21.15.50:0/689216040 shutdown_connections 2024-04-23T18:29:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.060+0000 7f6fab233640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6f900a3d20 0x7f6f900a61e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.060+0000 7f6fab233640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa419cfa0 0x7f6fa419d450 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.060+0000 7f6fab233640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fa4107280 0x7f6fa4198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.060+0000 7f6fab233640 1 --2- 172.21.15.50:0/689216040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fa4102240 0x7f6fa41986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.060+0000 7f6fab233640 1 -- 172.21.15.50:0/689216040 >> 172.21.15.50:0/689216040 conn(0x7f6fa40790e0 msgr2=0x7f6fa4100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:49.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.060+0000 7f6fab233640 1 -- 172.21.15.50:0/689216040 shutdown_connections 2024-04-23T18:29:49.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.060+0000 7f6fab233640 1 -- 172.21.15.50:0/689216040 wait complete. 2024-04-23T18:29:49.064 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id 99593c95 --release --await' 2024-04-23T18:29:49.064 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id 99593c95 --release --await 2024-04-23T18:29:49.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.181+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1036115022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1010a0 msgr2=0x7fb5dc101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:49.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.181+0000 7fb5e45e2640 1 --2- 172.21.15.50:0/1036115022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1010a0 0x7fb5dc101480 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009900 tx=0x7fb5d802a410 comp rx=0 tx=0).stop 2024-04-23T18:29:49.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.181+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1036115022 shutdown_connections 2024-04-23T18:29:49.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.181+0000 7fb5e45e2640 1 --2- 172.21.15.50:0/1036115022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5dc106170 0x7fb5dc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.181+0000 7fb5e45e2640 1 --2- 172.21.15.50:0/1036115022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5dc101a50 0x7fb5dc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.181+0000 7fb5e45e2640 1 --2- 172.21.15.50:0/1036115022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1010a0 0x7fb5dc101480 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.181+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1036115022 >> 172.21.15.50:0/1036115022 conn(0x7fb5dc077eb0 msgr2=0x7fb5dc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:49.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.181+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1036115022 shutdown_connections 2024-04-23T18:29:49.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.181+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1036115022 wait complete. 2024-04-23T18:29:49.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e45e2640 1 Processor -- start 2024-04-23T18:29:49.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e45e2640 1 -- start start 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e45e2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1010a0 0x7fb5dc19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e45e2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5dc106170 0x7fb5dc19e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e45e2640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5dc198610 0x7fb5dc198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e45e2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb5dc1121f0 con 0x7fb5dc1010a0 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e45e2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb5dc112070 con 0x7fb5dc106170 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e45e2640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb5dc112370 con 0x7fb5dc198610 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e237e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1010a0 0x7fb5dc19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e237e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1010a0 0x7fb5dc19e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56382/0 (socket says 172.21.15.50:56382) 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e237e640 1 -- 172.21.15.50:0/1016555888 learned_addr learned my addr 172.21.15.50:0/1016555888 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e1b7d640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5dc106170 0x7fb5dc19e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5dc198610 0x7fb5dc198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e237e640 1 -- 172.21.15.50:0/1016555888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5dc198610 msgr2=0x7fb5dc198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e237e640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5dc198610 0x7fb5dc198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e237e640 1 -- 172.21.15.50:0/1016555888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5dc106170 msgr2=0x7fb5dc19e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:49.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e237e640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5dc106170 0x7fb5dc19e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e237e640 1 -- 172.21.15.50:0/1016555888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5dc199130 con 0x7fb5dc1010a0 2024-04-23T18:29:49.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5dc198610 0x7fb5dc198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:29:49.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.182+0000 7fb5e237e640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1010a0 0x7fb5dc19e460 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8005940 tx=0x7fb5d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:49.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.183+0000 7fb5d37fe640 1 -- 172.21.15.50:0/1016555888 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d803f030 con 0x7fb5dc1010a0 2024-04-23T18:29:49.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.183+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1016555888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5dc1993c0 con 0x7fb5dc1010a0 2024-04-23T18:29:49.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.183+0000 7fb5d37fe640 1 -- 172.21.15.50:0/1016555888 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5d8003cb0 con 0x7fb5dc1010a0 2024-04-23T18:29:49.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.183+0000 7fb5d37fe640 1 -- 172.21.15.50:0/1016555888 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8035070 con 0x7fb5dc1010a0 2024-04-23T18:29:49.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.183+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1016555888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5dc1a55b0 con 0x7fb5dc1010a0 2024-04-23T18:29:49.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.185+0000 7fb5d37fe640 1 -- 172.21.15.50:0/1016555888 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb5d8042030 con 0x7fb5dc1010a0 2024-04-23T18:29:49.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.185+0000 7fb5e73d4740 1 Processor -- start 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.185+0000 7fb5e73d4740 1 -- start start 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e73d4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78f390 0x55c2dd78f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e73d4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e73d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd79e890 0x55c2dd7a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e73d4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c2dd6f46d0 con 0x55c2dd78f390 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e73d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c2dd6911a0 con 0x55c2dd79e890 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e73d4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c2dd6f0690 con 0x55c2dd78fcb0 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e2b7f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78f390 0x55c2dd78f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e2b7f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78f390 0x55c2dd78f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56390/0 (socket says 172.21.15.50:56390) 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e2b7f640 1 -- 172.21.15.50:0/4002848297 learned_addr learned my addr 172.21.15.50:0/4002848297 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e1b7d640 1 --2- 172.21.15.50:0/4002848297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd79e890 0x55c2dd7a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e237e640 1 --2- 172.21.15.50:0/4002848297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5d37fe640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb5c80a3d20 0x7fb5c80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e2b7f640 1 -- 172.21.15.50:0/4002848297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78fcb0 msgr2=0x55c2dd79e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/4002848297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e2b7f640 1 -- 172.21.15.50:0/4002848297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd79e890 msgr2=0x55c2dd7a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/4002848297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd79e890 0x55c2dd7a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e2b7f640 1 -- 172.21.15.50:0/4002848297 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2dd69b5b0 con 0x55c2dd78f390 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5d37fe640 1 -- 172.21.15.50:0/1016555888 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7fb5d80f1160 con 0x7fb5dc1010a0 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/4002848297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78f390 0x55c2dd78f770 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4009e40 tx=0x7fb5d400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd4035a6902a2a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.186+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb5c80a3d20 0x7fb5c80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.187+0000 7fb5d0bfe640 1 -- 172.21.15.50:0/4002848297 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d401a680 con 0x55c2dd78f390 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.187+0000 7fb5d0bfe640 1 -- 172.21.15.50:0/4002848297 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5d4011440 con 0x55c2dd78f390 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.187+0000 7fb5d0bfe640 1 -- 172.21.15.50:0/4002848297 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4019600 con 0x55c2dd78f390 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.187+0000 7fb5e73d4740 1 -- 172.21.15.50:0/4002848297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78f390 msgr2=0x55c2dd78f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.187+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb5c80a3d20 0x7fb5c80a61e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4009fd0 tx=0x7fb5d4019d10 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.187+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/4002848297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78f390 0x55c2dd78f770 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4009e40 tx=0x7fb5d400f730 comp rx=0 tx=0).stop 2024-04-23T18:29:49.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.187+0000 7fb5e73d4740 1 -- 172.21.15.50:0/4002848297 shutdown_connections 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.187+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/4002848297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd79e890 0x55c2dd7a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.187+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/4002848297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.187+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/4002848297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78f390 0x55c2dd78f770 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.187+0000 7fb5e73d4740 1 -- 172.21.15.50:0/4002848297 >> 172.21.15.50:0/4002848297 conn(0x55c2dd78b7b0 msgr2=0x55c2dd78bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e73d4740 1 -- 172.21.15.50:0/4002848297 shutdown_connections 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e73d4740 1 -- 172.21.15.50:0/4002848297 wait complete. 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e73d4740 1 -- start start 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e73d4740 1 Processor -- start 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e73d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd78f390 0x55c2dd79e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e73d4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e73d4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 0x55c2dd79bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e73d4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c2dd7a16f0 con 0x55c2dd79e890 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e73d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c2dd68eb20 con 0x55c2dd78f390 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e73d4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c2dd7a1870 con 0x55c2dd78fcb0 2024-04-23T18:29:49.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e73d4740 10 client.0 _ll_register_callbacks cb 0x55c2dd68e530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e2b7f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 0x55c2dd79bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e1b7d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e1b7d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:34494/0 (socket says 172.21.15.50:34494) 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e1b7d640 1 -- 172.21.15.50:0/3673396104 learned_addr learned my addr 172.21.15.50:0/3673396104 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.188+0000 7fb5e237e640 1 --2- 172.21.15.50:0/3673396104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd78f390 0x55c2dd79e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5e2b7f640 1 -- 172.21.15.50:0/3673396104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78fcb0 msgr2=0x55c2dd79b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/3673396104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5e2b7f640 1 -- 172.21.15.50:0/3673396104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd78f390 msgr2=0x55c2dd79e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/3673396104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd78f390 0x55c2dd79e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5e2b7f640 1 -- 172.21.15.50:0/3673396104 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2dd406920 con 0x55c2dd79e890 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/3673396104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 0x55c2dd79bb80 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4011dd0 tx=0x7fb5d401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67c4bbb3ab14d04 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5c5ffb640 1 -- 172.21.15.50:0/3673396104 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4035060 con 0x55c2dd79e890 2024-04-23T18:29:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5c5ffb640 1 -- 172.21.15.50:0/3673396104 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7fb5d4011820 con 0x55c2dd79e890 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5e73d4740 1 -- 172.21.15.50:0/3673396104 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5d4011440 con 0x55c2dd79e890 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5c5ffb640 1 -- 172.21.15.50:0/3673396104 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb5d402c040 con 0x55c2dd79e890 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5e73d4740 10 client.16866 fetch_fsmap learned FSMap version 45 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5e73d4740 1 -- 172.21.15.50:0/3673396104 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55c2dd4d2d30 con 0x55c2dd79e890 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5c5ffb640 1 -- 172.21.15.50:0/3673396104 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5d4011440 con 0x55c2dd79e890 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.189+0000 7fb5c5ffb640 1 -- 172.21.15.50:0/3673396104 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7fb5d40108f0 con 0x55c2dd79e890 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.190+0000 7fb5e73d4740 10 client.16866 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.190+0000 7fb5e73d4740 10 client.16866 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.190+0000 7fb5e73d4740 20 client.16866 populate_metadata read hostname 'smithi050' 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.190+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/3673396104 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c2dd7ae9b0 0x55c2dd7b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.190+0000 7fb5e73d4740 4 client.16866 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.190+0000 7fb5e73d4740 1 -- 172.21.15.50:0/3673396104 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2dd6b2500 con 0x55c2dd7ae9b0 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.190+0000 7fb5e237e640 1 --2- 172.21.15.50:0/3673396104 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c2dd7ae9b0 0x55c2dd7b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.190+0000 7fb5e237e640 1 --2- 172.21.15.50:0/3673396104 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c2dd7ae9b0 0x55c2dd7b0da0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ad4d8bb6c705ef0 server_cookie=ac89888dc0a49f92 in_seq=0 out_seq=0 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.190+0000 7fb5c5ffb640 10 client.16866 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5c5ffb640 1 -- 172.21.15.50:0/3673396104 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c2dd6b2500 con 0x55c2dd7ae9b0 2024-04-23T18:29:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5c5ffb640 10 client.16866 handle_command_reply: tid=0 2024-04-23T18:29:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5e73d4740 1 client.16866 shutdown 2024-04-23T18:29:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5e73d4740 1 -- 172.21.15.50:0/3673396104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 msgr2=0x55c2dd79bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/3673396104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 0x55c2dd79bb80 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4011dd0 tx=0x7fb5d401a000 comp rx=0 tx=0).stop 2024-04-23T18:29:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5e73d4740 1 -- 172.21.15.50:0/3673396104 shutdown_connections 2024-04-23T18:29:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/3673396104 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c2dd7ae9b0 0x55c2dd7b0da0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/3673396104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 0x55c2dd79bb80 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5c5ffb640 0 client.16866 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/3673396104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/3673396104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd78f390 0x55c2dd79e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5e73d4740 1 -- 172.21.15.50:0/3673396104 >> 172.21.15.50:0/3673396104 conn(0x55c2dd78b7b0 msgr2=0x55c2dd7915e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5e73d4740 1 -- 172.21.15.50:0/3673396104 shutdown_connections 2024-04-23T18:29:49.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5e73d4740 1 -- 172.21.15.50:0/3673396104 wait complete. 2024-04-23T18:29:49.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.191+0000 7fb5e73d4740 20 client.16866 trim_cache size 0 max 16384 2024-04-23T18:29:49.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e73d4740 1 Processor -- start 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e73d4740 1 -- start start 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e73d4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78f390 0x55c2dd78a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e73d4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e73d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd79e890 0x55c2dd7b6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e73d4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb5dc1ae440 con 0x55c2dd78fcb0 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e73d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb5dc1abc80 con 0x55c2dd79e890 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e73d4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb5dc1a9af0 con 0x55c2dd78f390 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e1b7d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e1b7d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56414/0 (socket says 172.21.15.50:56414) 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e1b7d640 1 -- 172.21.15.50:0/2699809782 learned_addr learned my addr 172.21.15.50:0/2699809782 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/2699809782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd79e890 0x55c2dd7b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e237e640 1 --2- 172.21.15.50:0/2699809782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78f390 0x55c2dd78a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e1b7d640 1 -- 172.21.15.50:0/2699809782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78f390 msgr2=0x55c2dd78a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e1b7d640 1 --2- 172.21.15.50:0/2699809782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78f390 0x55c2dd78a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e1b7d640 1 -- 172.21.15.50:0/2699809782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd79e890 msgr2=0x55c2dd7b6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e1b7d640 1 --2- 172.21.15.50:0/2699809782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd79e890 0x55c2dd7b6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e1b7d640 1 -- 172.21.15.50:0/2699809782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2dd785de0 con 0x55c2dd78fcb0 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/2699809782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd79e890 0x55c2dd7b6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.209+0000 7fb5e1b7d640 1 --2- 172.21.15.50:0/2699809782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79ddb0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc004610 tx=0x7fb5cc00daf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53861e2bc63698fe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5c7fff640 1 -- 172.21.15.50:0/2699809782 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc005020 con 0x55c2dd78fcb0 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5c7fff640 1 -- 172.21.15.50:0/2699809782 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5cc0037f0 con 0x55c2dd78fcb0 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5c7fff640 1 -- 172.21.15.50:0/2699809782 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc004b60 con 0x55c2dd78fcb0 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5e73d4740 1 -- 172.21.15.50:0/2699809782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78fcb0 msgr2=0x55c2dd79ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/2699809782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79ddb0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc004610 tx=0x7fb5cc00daf0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5e73d4740 1 -- 172.21.15.50:0/2699809782 shutdown_connections 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/2699809782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd79e890 0x55c2dd7b6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/2699809782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd78fcb0 0x55c2dd79ddb0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/2699809782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78f390 0x55c2dd78a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5e73d4740 1 -- 172.21.15.50:0/2699809782 >> 172.21.15.50:0/2699809782 conn(0x55c2dd78b7b0 msgr2=0x55c2dd78a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5e73d4740 1 -- 172.21.15.50:0/2699809782 shutdown_connections 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5e73d4740 1 -- 172.21.15.50:0/2699809782 wait complete. 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.210+0000 7fb5e73d4740 1 -- start start 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e73d4740 1 Processor -- start 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e73d4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78f390 0x55c2dd7a6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e73d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd78fcb0 0x55c2dd7a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e73d4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 0x55c2dd7a2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e73d4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb5dc1aa140 con 0x55c2dd79e890 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e73d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb5dc1ab5b0 con 0x55c2dd78fcb0 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e73d4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb5dc1aa810 con 0x55c2dd78f390 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e73d4740 10 client.0 _ll_register_callbacks cb 0x7fb5d4009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e2b7f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 0x55c2dd7a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e2b7f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 0x55c2dd7a2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56426/0 (socket says 172.21.15.50:56426) 2024-04-23T18:29:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e2b7f640 1 -- 172.21.15.50:0/784089598 learned_addr learned my addr 172.21.15.50:0/784089598 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e237e640 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78f390 0x55c2dd7a6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e1b7d640 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd78fcb0 0x55c2dd7a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e2b7f640 1 -- 172.21.15.50:0/784089598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78f390 msgr2=0x55c2dd7a6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78f390 0x55c2dd7a6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e2b7f640 1 -- 172.21.15.50:0/784089598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd78fcb0 msgr2=0x55c2dd7a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd78fcb0 0x55c2dd7a1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e2b7f640 1 -- 172.21.15.50:0/784089598 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2dd7a2540 con 0x55c2dd79e890 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e1b7d640 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd78fcb0 0x55c2dd7a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5e2b7f640 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 0x55c2dd7a2000 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4009e40 tx=0x7fb5d4010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eadd48bea034459d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.211+0000 7fb5c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5c5ffb640 1 -- 172.21.15.50:0/784089598 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4046070 con 0x55c2dd79e890 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5c5ffb640 1 -- 172.21.15.50:0/784089598 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7fb5d400fc30 con 0x55c2dd79e890 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5e73d4740 1 -- 172.21.15.50:0/784089598 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5cc0037f0 con 0x55c2dd79e890 2024-04-23T18:29:49.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5c5ffb640 1 -- 172.21.15.50:0/784089598 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb5d404f050 con 0x55c2dd79e890 2024-04-23T18:29:49.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5e73d4740 10 client.16872 fetch_fsmap learned FSMap version 45 2024-04-23T18:29:49.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5e73d4740 1 -- 172.21.15.50:0/784089598 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55c2dd815a30 con 0x55c2dd79e890 2024-04-23T18:29:49.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5c5ffb640 1 -- 172.21.15.50:0/784089598 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5cc0037f0 con 0x55c2dd79e890 2024-04-23T18:29:49.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5c5ffb640 1 -- 172.21.15.50:0/784089598 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7fb5d4049020 con 0x55c2dd79e890 2024-04-23T18:29:49.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5e73d4740 10 client.16872 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:29:49.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5e73d4740 10 client.16872 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:29:49.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5e73d4740 20 client.16872 populate_metadata read hostname 'smithi050' 2024-04-23T18:29:49.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c2dd816110 0x55c2dd818550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:49.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5e73d4740 4 client.16872 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "99593c95", "release": true, "await": true}] 2024-04-23T18:29:49.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5e73d4740 1 -- 172.21.15.50:0/784089598 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "99593c95", "release": true, "await": true}) -- 0x55c2dd7020f0 con 0x55c2dd816110 2024-04-23T18:29:49.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.212+0000 7fb5e237e640 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c2dd816110 0x55c2dd818550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:49.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.213+0000 7fb5c5ffb640 10 client.16872 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:49.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.213+0000 7fb5e237e640 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c2dd816110 0x55c2dd818550 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc80aac7549842b7 server_cookie=dab7c15b411327ca in_seq=0 out_seq=0 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.217+0000 7fb5c5ffb640 1 -- 172.21.15.50:0/784089598 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55c2dd7020f0 con 0x55c2dd816110 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.217+0000 7fb5c5ffb640 10 client.16872 handle_command_reply: tid=0 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.217+0000 7fb5e73d4740 1 client.16872 shutdown 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e73d4740 1 -- 172.21.15.50:0/784089598 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 msgr2=0x55c2dd7a2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 0x55c2dd7a2000 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4009e40 tx=0x7fb5d4010c00 comp rx=0 tx=0).stop 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e73d4740 1 -- 172.21.15.50:0/784089598 shutdown_connections 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c2dd816110 0x55c2dd818550 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c2dd79e890 0x55c2dd7a2000 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c2dd78fcb0 0x55c2dd7a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e73d4740 1 --2- 172.21.15.50:0/784089598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c2dd78f390 0x55c2dd7a6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e73d4740 1 -- 172.21.15.50:0/784089598 >> 172.21.15.50:0/784089598 conn(0x55c2dd78b7b0 msgr2=0x55c2dd791510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5c5ffb640 0 client.16872 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:49.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e73d4740 1 -- 172.21.15.50:0/784089598 shutdown_connections 2024-04-23T18:29:49.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e73d4740 1 -- 172.21.15.50:0/784089598 wait complete. 2024-04-23T18:29:49.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e73d4740 20 client.16872 trim_cache size 0 max 16384 2024-04-23T18:29:49.213 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:29:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 45, 2024-04-23T18:29:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:29:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 36, 2024-04-23T18:29:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:29:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: "99593c95": { 2024-04-23T18:29:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 36, 2024-04-23T18:29:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:29:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:29:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:29:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:29:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:29:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:29:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 78.4, 2024-04-23T18:29:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:29:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:29:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:29:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:29:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:29:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:29:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:29:49.215 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:29:49.215 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:29:49.215 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:29:49.215 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:29:49.215 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:29:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1016555888 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb5c80a3d20 msgr2=0x7fb5c80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e45e2640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb5c80a3d20 0x7fb5c80a61e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4009fd0 tx=0x7fb5d4019d10 comp rx=0 tx=0).stop 2024-04-23T18:29:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1016555888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1010a0 msgr2=0x7fb5dc19e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e45e2640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1010a0 0x7fb5dc19e460 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8005940 tx=0x7fb5d80042f0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.218+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1016555888 shutdown_connections 2024-04-23T18:29:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.219+0000 7fb5e45e2640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb5c80a3d20 0x7fb5c80a61e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.219+0000 7fb5e45e2640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5dc198610 0x7fb5dc198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.219+0000 7fb5e45e2640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5dc106170 0x7fb5dc19e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.219+0000 7fb5e45e2640 1 --2- 172.21.15.50:0/1016555888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1010a0 0x7fb5dc19e460 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.219+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1016555888 >> 172.21.15.50:0/1016555888 conn(0x7fb5dc077eb0 msgr2=0x7fb5dc107340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.219+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1016555888 shutdown_connections 2024-04-23T18:29:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:29:49.219+0000 7fb5e45e2640 1 -- 172.21.15.50:0/1016555888 wait complete. 2024-04-23T18:29:49.223 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '99593c95', seconds elapsed: 19.5 2024-04-23T18:29:49.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 688909 5.33 MB/sec execute 503 sec latency 18971.138 ms 2024-04-23T18:29:49.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:49 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:29:49.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:49 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:29:49.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:49 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:29:49.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:49 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/689216040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:29:49.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:49 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:29:49.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:49 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:29:49.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:49 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:29:49.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:49 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/689216040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:29:49.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:49 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:29:49.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:49 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:29:49.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:49 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:29:49.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:49 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/689216040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:29:50.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 690017 5.33 MB/sec execute 504 sec latency 74.584 ms 2024-04-23T18:29:50.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:50 smithi050 ceph-mon[29187]: pgmap v355: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 700 KiB/s wr, 54 op/s 2024-04-23T18:29:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:50 smithi177 ceph-mon[32308]: pgmap v355: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 700 KiB/s wr, 54 op/s 2024-04-23T18:29:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:50 smithi188 ceph-mon[32694]: pgmap v355: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 700 KiB/s wr, 54 op/s 2024-04-23T18:29:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.996+0000 7f5468c8a640 1 -- 172.21.15.50:0/2625520102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5464103200 msgr2=0x7f54641035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.996+0000 7f5468c8a640 1 --2- 172.21.15.50:0/2625520102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5464103200 0x7f54641035e0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f5454009900 tx=0x7f545402a480 comp rx=0 tx=0).stop 2024-04-23T18:29:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.996+0000 7f5468c8a640 1 -- 172.21.15.50:0/2625520102 shutdown_connections 2024-04-23T18:29:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.996+0000 7f5468c8a640 1 --2- 172.21.15.50:0/2625520102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f546410c5d0 0x7f546410ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.996+0000 7f5468c8a640 1 --2- 172.21.15.50:0/2625520102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5464103b20 0x7f546410c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.996+0000 7f5468c8a640 1 --2- 172.21.15.50:0/2625520102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5464103200 0x7f54641035e0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.996+0000 7f5468c8a640 1 -- 172.21.15.50:0/2625520102 >> 172.21.15.50:0/2625520102 conn(0x7f54640771d0 msgr2=0x7f54640775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.996+0000 7f5468c8a640 1 -- 172.21.15.50:0/2625520102 shutdown_connections 2024-04-23T18:29:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.996+0000 7f5468c8a640 1 -- 172.21.15.50:0/2625520102 wait complete. 2024-04-23T18:29:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5468c8a640 1 Processor -- start 2024-04-23T18:29:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5468c8a640 1 -- start start 2024-04-23T18:29:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5468c8a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5464103200 0x7f5464198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5468c8a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5464103b20 0x7f5464198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5468c8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f546410c5d0 0x7f546419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5468c8a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5464112270 con 0x7f546410c5d0 2024-04-23T18:29:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5468c8a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f54641120f0 con 0x7f5464103b20 2024-04-23T18:29:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5468c8a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f54641123f0 con 0x7f5464103200 2024-04-23T18:29:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5462d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f546410c5d0 0x7f546419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5462d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f546410c5d0 0x7f546419cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56450/0 (socket says 172.21.15.50:56450) 2024-04-23T18:29:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5462d9d640 1 -- 172.21.15.50:0/2292684844 learned_addr learned my addr 172.21.15.50:0/2292684844 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5461d9b640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5464103b20 0x7f5464198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f546259c640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5464103200 0x7f5464198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5462d9d640 1 -- 172.21.15.50:0/2292684844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5464103200 msgr2=0x7f5464198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5462d9d640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5464103200 0x7f5464198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5462d9d640 1 -- 172.21.15.50:0/2292684844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5464103b20 msgr2=0x7f5464198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5462d9d640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5464103b20 0x7f5464198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5462d9d640 1 -- 172.21.15.50:0/2292684844 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f546419d6b0 con 0x7f546410c5d0 2024-04-23T18:29:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.997+0000 7f5461d9b640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5464103b20 0x7f5464198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:29:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.998+0000 7f5462d9d640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f546410c5d0 0x7f546419cf80 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f545c00dac0 tx=0x7f545c00df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.998+0000 7f54537fe640 1 -- 172.21.15.50:0/2292684844 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c017070 con 0x7f546410c5d0 2024-04-23T18:29:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.998+0000 7f54537fe640 1 -- 172.21.15.50:0/2292684844 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f545c00be30 con 0x7f546410c5d0 2024-04-23T18:29:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.998+0000 7f5468c8a640 1 -- 172.21.15.50:0/2292684844 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f546419d9a0 con 0x7f546410c5d0 2024-04-23T18:29:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.998+0000 7f54537fe640 1 -- 172.21.15.50:0/2292684844 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c010460 con 0x7f546410c5d0 2024-04-23T18:29:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:50.998+0000 7f5468c8a640 1 -- 172.21.15.50:0/2292684844 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54641a5230 con 0x7f546410c5d0 2024-04-23T18:29:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.000+0000 7f54537fe640 1 -- 172.21.15.50:0/2292684844 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f545c0040b0 con 0x7f546410c5d0 2024-04-23T18:29:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.000+0000 7f546b99f740 1 Processor -- start 2024-04-23T18:29:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.000+0000 7f546b99f740 1 -- start start 2024-04-23T18:29:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f546b99f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x558121383920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f546b99f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383e60 0x5581213926d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f546b99f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121392c10 0x558121395000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f546b99f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5581212e8970 con 0x558121383540 2024-04-23T18:29:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f546b99f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5581212854a0 con 0x558121392c10 2024-04-23T18:29:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f546b99f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5581212e4930 con 0x558121383e60 2024-04-23T18:29:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f54537fe640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f544c0a3dc0 0x7f544c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f5461d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x558121383920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f5461d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x558121383920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56452/0 (socket says 172.21.15.50:56452) 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f5461d9b640 1 -- 172.21.15.50:0/3321757481 learned_addr learned my addr 172.21.15.50:0/3321757481 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f5462d9d640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f544c0a3dc0 0x7f544c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f546259c640 1 --2- 172.21.15.50:0/3321757481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383e60 0x5581213926d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f5461d9b640 1 --2- 172.21.15.50:0/3321757481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121392c10 0x558121395000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f54537fe640 1 -- 172.21.15.50:0/2292684844 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7f545c0d0110 con 0x7f546410c5d0 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f5461d9b640 1 -- 172.21.15.50:0/3321757481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383e60 msgr2=0x5581213926d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f5461d9b640 1 --2- 172.21.15.50:0/3321757481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383e60 0x5581213926d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f5461d9b640 1 -- 172.21.15.50:0/3321757481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121392c10 msgr2=0x558121395000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f5461d9b640 1 --2- 172.21.15.50:0/3321757481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121392c10 0x558121395000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f5461d9b640 1 -- 172.21.15.50:0/3321757481 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558120ffb210 con 0x558121383540 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.001+0000 7f5461d9b640 1 --2- 172.21.15.50:0/3321757481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x558121383920 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f54580135e0 tx=0x7f54580097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160135c8fae314d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.002+0000 7f5462d9d640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f544c0a3dc0 0x7f544c0a6280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f545c0cd9a0 tx=0x7f545c06db80 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.002+0000 7f5450ff9640 1 -- 172.21.15.50:0/3321757481 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458011470 con 0x558121383540 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.002+0000 7f5450ff9640 1 -- 172.21.15.50:0/3321757481 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f545800f950 con 0x558121383540 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.002+0000 7f5450ff9640 1 -- 172.21.15.50:0/3321757481 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458010630 con 0x558121383540 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.002+0000 7f546b99f740 1 -- 172.21.15.50:0/3321757481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 msgr2=0x558121383920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.002+0000 7f546b99f740 1 --2- 172.21.15.50:0/3321757481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x558121383920 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f54580135e0 tx=0x7f54580097d0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.002+0000 7f546b99f740 1 -- 172.21.15.50:0/3321757481 shutdown_connections 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.002+0000 7f546b99f740 1 --2- 172.21.15.50:0/3321757481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121392c10 0x558121395000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.002+0000 7f546b99f740 1 --2- 172.21.15.50:0/3321757481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383e60 0x5581213926d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.002+0000 7f546b99f740 1 --2- 172.21.15.50:0/3321757481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x558121383920 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.002+0000 7f546b99f740 1 -- 172.21.15.50:0/3321757481 >> 172.21.15.50:0/3321757481 conn(0x55812137fb30 msgr2=0x55812137ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f546b99f740 1 -- 172.21.15.50:0/3321757481 shutdown_connections 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f546b99f740 1 -- 172.21.15.50:0/3321757481 wait complete. 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f546b99f740 1 -- start start 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f546b99f740 1 Processor -- start 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f546b99f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x558121392480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f546b99f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383e60 0x55812138f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f546b99f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121392c10 0x55812138fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f546b99f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558121395840 con 0x558121383540 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f546b99f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5581212830b0 con 0x558121392c10 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f546b99f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5581213959c0 con 0x558121383e60 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f546b99f740 10 client.0 _ll_register_callbacks cb 0x558121270d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f5461d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x558121392480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f5461d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x558121392480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56458/0 (socket says 172.21.15.50:56458) 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f5461d9b640 1 -- 172.21.15.50:0/801244231 learned_addr learned my addr 172.21.15.50:0/801244231 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.003+0000 7f546259c640 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383e60 0x55812138f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f5461d9b640 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121392c10 0x55812138fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f5461d9b640 1 -- 172.21.15.50:0/801244231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383e60 msgr2=0x55812138f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f5461d9b640 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383e60 0x55812138f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f5461d9b640 1 -- 172.21.15.50:0/801244231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121392c10 msgr2=0x55812138fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f5461d9b640 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121392c10 0x55812138fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f5461d9b640 1 -- 172.21.15.50:0/801244231 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581210c7620 con 0x558121383540 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f5461d9b640 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x558121392480 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f54580132e0 tx=0x7f5458009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40769be7541525a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f5449ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f5449ffb640 1 -- 172.21.15.50:0/801244231 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458005450 con 0x558121383540 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f5449ffb640 1 -- 172.21.15.50:0/801244231 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7f54580055f0 con 0x558121383540 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f546b99f740 1 -- 172.21.15.50:0/801244231 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f545800f950 con 0x558121383540 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f5449ffb640 1 -- 172.21.15.50:0/801244231 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f5458035050 con 0x558121383540 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f546b99f740 10 client.16884 fetch_fsmap learned FSMap version 45 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f546b99f740 1 -- 172.21.15.50:0/801244231 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5581213a16b0 con 0x558121383540 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.004+0000 7f5449ffb640 1 -- 172.21.15.50:0/801244231 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f545800f950 con 0x558121383540 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.005+0000 7f5449ffb640 1 -- 172.21.15.50:0/801244231 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7f5458011590 con 0x558121383540 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.005+0000 7f546b99f740 10 client.16884 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.005+0000 7f546b99f740 10 client.16884 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.005+0000 7f546b99f740 20 client.16884 populate_metadata read hostname 'smithi050' 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.005+0000 7f546b99f740 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5581213a2ac0 0x5581213a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.005+0000 7f546b99f740 4 client.16884 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.005+0000 7f546b99f740 1 -- 172.21.15.50:0/801244231 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581212f6390 con 0x5581213a2ac0 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.005+0000 7f546259c640 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5581213a2ac0 0x5581213a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.005+0000 7f546259c640 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5581213a2ac0 0x5581213a4eb0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2a55045bbe0d144 server_cookie=6ceeebe33fc71099 in_seq=0 out_seq=0 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.005+0000 7f5449ffb640 10 client.16884 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f5449ffb640 1 -- 172.21.15.50:0/801244231 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581212f6390 con 0x5581213a2ac0 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f5449ffb640 10 client.16884 handle_command_reply: tid=0 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f546b99f740 1 client.16884 shutdown 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f546b99f740 1 -- 172.21.15.50:0/801244231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 msgr2=0x558121392480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f546b99f740 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x558121392480 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f54580132e0 tx=0x7f5458009650 comp rx=0 tx=0).stop 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f546b99f740 1 -- 172.21.15.50:0/801244231 shutdown_connections 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f546b99f740 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5581213a2ac0 0x5581213a4eb0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f546b99f740 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121392c10 0x55812138fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f5449ffb640 0 client.16884 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f546b99f740 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383e60 0x55812138f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f546b99f740 1 --2- 172.21.15.50:0/801244231 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x558121392480 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f546b99f740 1 -- 172.21.15.50:0/801244231 >> 172.21.15.50:0/801244231 conn(0x55812137fb30 msgr2=0x5581213859a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.006+0000 7f546b99f740 1 -- 172.21.15.50:0/801244231 shutdown_connections 2024-04-23T18:29:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.007+0000 7f546b99f740 1 -- 172.21.15.50:0/801244231 wait complete. 2024-04-23T18:29:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.007+0000 7f546b99f740 20 client.16884 trim_cache size 0 max 16384 2024-04-23T18:29:51.018 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.021+0000 7f546b99f740 1 Processor -- start 2024-04-23T18:29:51.018 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.021+0000 7f546b99f740 1 -- start start 2024-04-23T18:29:51.018 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f546b99f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x55812139e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f546b99f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121383e60 0x55812138ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f546b99f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121392c10 0x558121391fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f546b99f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f54641ae320 con 0x558121383540 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f546b99f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f54641abb60 con 0x558121383e60 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f546b99f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f54641a99f0 con 0x558121392c10 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f5461d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x55812139e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f5461d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x55812139e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56464/0 (socket says 172.21.15.50:56464) 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f5461d9b640 1 -- 172.21.15.50:0/3887924634 learned_addr learned my addr 172.21.15.50:0/3887924634 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f5461d9b640 1 -- 172.21.15.50:0/3887924634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121392c10 msgr2=0x558121391fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f5461d9b640 1 --2- 172.21.15.50:0/3887924634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121392c10 0x558121391fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f5461d9b640 1 -- 172.21.15.50:0/3887924634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121383e60 msgr2=0x55812138ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f546259c640 1 --2- 172.21.15.50:0/3887924634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121383e60 0x55812138ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f5461d9b640 1 --2- 172.21.15.50:0/3887924634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121383e60 0x55812138ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f5461d9b640 1 -- 172.21.15.50:0/3887924634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55812137f960 con 0x558121383540 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f546259c640 1 --2- 172.21.15.50:0/3887924634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121383e60 0x55812138ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f5461d9b640 1 --2- 172.21.15.50:0/3887924634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x55812139e700 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f54580059f0 tx=0x7f54580095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55331992c6dcfb7c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f544bfff640 1 -- 172.21.15.50:0/3887924634 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54580096c0 con 0x558121383540 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f544bfff640 1 -- 172.21.15.50:0/3887924634 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5458023070 con 0x558121383540 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.022+0000 7f544bfff640 1 -- 172.21.15.50:0/3887924634 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54580054b0 con 0x558121383540 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.023+0000 7f546b99f740 1 -- 172.21.15.50:0/3887924634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 msgr2=0x55812139e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.023+0000 7f546b99f740 1 --2- 172.21.15.50:0/3887924634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x55812139e700 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f54580059f0 tx=0x7f54580095e0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.023+0000 7f546b99f740 1 -- 172.21.15.50:0/3887924634 shutdown_connections 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.023+0000 7f546b99f740 1 --2- 172.21.15.50:0/3887924634 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121392c10 0x558121391fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.023+0000 7f546b99f740 1 --2- 172.21.15.50:0/3887924634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121383e60 0x55812138ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.023+0000 7f546b99f740 1 --2- 172.21.15.50:0/3887924634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121383540 0x55812139e700 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.023+0000 7f546b99f740 1 -- 172.21.15.50:0/3887924634 >> 172.21.15.50:0/3887924634 conn(0x55812137fb30 msgr2=0x55812138a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.023+0000 7f546b99f740 1 -- 172.21.15.50:0/3887924634 shutdown_connections 2024-04-23T18:29:51.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.023+0000 7f546b99f740 1 -- 172.21.15.50:0/3887924634 wait complete. 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.023+0000 7f546b99f740 1 -- start start 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.023+0000 7f546b99f740 1 Processor -- start 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f546b99f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383540 0x55812139ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f546b99f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121383e60 0x55812139b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f546b99f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121392c10 0x55812138bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f546b99f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f54641aa040 con 0x558121392c10 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f546b99f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f54641ab490 con 0x558121383e60 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f546b99f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f54641aa6f0 con 0x558121383540 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f546b99f740 10 client.0 _ll_register_callbacks cb 0x7f545800fb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5461d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121392c10 0x55812138bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5461d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121392c10 0x55812138bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56472/0 (socket says 172.21.15.50:56472) 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5461d9b640 1 -- 172.21.15.50:0/927537045 learned_addr learned my addr 172.21.15.50:0/927537045 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f546259c640 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121383e60 0x55812139b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5461d9b640 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383540 0x55812139ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5461d9b640 1 -- 172.21.15.50:0/927537045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383540 msgr2=0x55812139ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5461d9b640 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383540 0x55812139ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5461d9b640 1 -- 172.21.15.50:0/927537045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121383e60 msgr2=0x55812139b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5461d9b640 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121383e60 0x55812139b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5461d9b640 1 -- 172.21.15.50:0/927537045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55812138c380 con 0x558121392c10 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5461d9b640 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121392c10 0x55812138bb80 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f545800a010 tx=0x7f545800fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbda9bffcb0837f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5449ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5449ffb640 1 -- 172.21.15.50:0/927537045 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458013270 con 0x558121392c10 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5449ffb640 1 -- 172.21.15.50:0/927537045 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7f5458005250 con 0x558121392c10 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f546b99f740 1 -- 172.21.15.50:0/927537045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5458023070 con 0x558121392c10 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f5449ffb640 1 -- 172.21.15.50:0/927537045 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f545804a050 con 0x558121392c10 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f546b99f740 10 client.16890 fetch_fsmap learned FSMap version 45 2024-04-23T18:29:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.024+0000 7f546b99f740 1 -- 172.21.15.50:0/927537045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x558121408860 con 0x558121392c10 2024-04-23T18:29:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.025+0000 7f5449ffb640 1 -- 172.21.15.50:0/927537045 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5458023070 con 0x558121392c10 2024-04-23T18:29:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.025+0000 7f5449ffb640 1 -- 172.21.15.50:0/927537045 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7f5458004ab0 con 0x558121392c10 2024-04-23T18:29:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.025+0000 7f546b99f740 10 client.16890 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T18:29:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.025+0000 7f546b99f740 10 client.16890 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:29:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.025+0000 7f546b99f740 20 client.16890 populate_metadata read hostname 'smithi050' 2024-04-23T18:29:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.025+0000 7f546b99f740 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5581213a2ac0 0x55812140af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:29:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.025+0000 7f546b99f740 4 client.16890 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:29:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.025+0000 7f546b99f740 1 -- 172.21.15.50:0/927537045 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558121390040 con 0x5581213a2ac0 2024-04-23T18:29:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.025+0000 7f546259c640 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5581213a2ac0 0x55812140af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:29:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.025+0000 7f5449ffb640 10 client.16890 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.025+0000 7f546259c640 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5581213a2ac0 0x55812140af10 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=706d2e05de1e1b4d server_cookie=20622924176df4ec in_seq=0 out_seq=0 2024-04-23T18:29:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.068+0000 7f5449ffb640 1 -- 172.21.15.50:0/927537045 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558121390040 con 0x5581213a2ac0 2024-04-23T18:29:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.068+0000 7f5449ffb640 10 client.16890 handle_command_reply: tid=0 2024-04-23T18:29:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.068+0000 7f546b99f740 1 client.16890 shutdown 2024-04-23T18:29:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.068+0000 7f546b99f740 1 -- 172.21.15.50:0/927537045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121392c10 msgr2=0x55812138bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.068+0000 7f546b99f740 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121392c10 0x55812138bb80 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f545800a010 tx=0x7f545800fc20 comp rx=0 tx=0).stop 2024-04-23T18:29:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.069+0000 7f546b99f740 1 -- 172.21.15.50:0/927537045 shutdown_connections 2024-04-23T18:29:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.069+0000 7f546b99f740 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5581213a2ac0 0x55812140af10 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.069+0000 7f546b99f740 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558121392c10 0x55812138bb80 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.069+0000 7f546b99f740 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558121383e60 0x55812139b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.069+0000 7f5449ffb640 0 client.16890 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:29:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.069+0000 7f546b99f740 1 --2- 172.21.15.50:0/927537045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558121383540 0x55812139ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.069+0000 7f546b99f740 1 -- 172.21.15.50:0/927537045 >> 172.21.15.50:0/927537045 conn(0x55812137fb30 msgr2=0x558121385000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.069+0000 7f546b99f740 1 -- 172.21.15.50:0/927537045 shutdown_connections 2024-04-23T18:29:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.069+0000 7f546b99f740 1 -- 172.21.15.50:0/927537045 wait complete. 2024-04-23T18:29:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.069+0000 7f546b99f740 20 client.16890 trim_cache size 0 max 16384 2024-04-23T18:29:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:29:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:29:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:29:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:29:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.069+0000 7f5468c8a640 1 -- 172.21.15.50:0/2292684844 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f544c0a3dc0 msgr2=0x7f544c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.069+0000 7f5468c8a640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f544c0a3dc0 0x7f544c0a6280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f545c0cd9a0 tx=0x7f545c06db80 comp rx=0 tx=0).stop 2024-04-23T18:29:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.070+0000 7f5468c8a640 1 -- 172.21.15.50:0/2292684844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f546410c5d0 msgr2=0x7f546419cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:29:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.070+0000 7f5468c8a640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f546410c5d0 0x7f546419cf80 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f545c00dac0 tx=0x7f545c00df80 comp rx=0 tx=0).stop 2024-04-23T18:29:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.070+0000 7f5468c8a640 1 -- 172.21.15.50:0/2292684844 shutdown_connections 2024-04-23T18:29:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.070+0000 7f5468c8a640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f544c0a3dc0 0x7f544c0a6280 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.070+0000 7f5468c8a640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f546410c5d0 0x7f546419cf80 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.070+0000 7f5468c8a640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5464103b20 0x7f5464198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.070+0000 7f5468c8a640 1 --2- 172.21.15.50:0/2292684844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5464103200 0x7f5464198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:29:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.070+0000 7f5468c8a640 1 -- 172.21.15.50:0/2292684844 >> 172.21.15.50:0/2292684844 conn(0x7f54640771d0 msgr2=0x7f54640fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:29:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.070+0000 7f5468c8a640 1 -- 172.21.15.50:0/2292684844 shutdown_connections 2024-04-23T18:29:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:29:51.070+0000 7f5468c8a640 1 -- 172.21.15.50:0/2292684844 wait complete. 2024-04-23T18:29:51.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 690454 5.33 MB/sec execute 505 sec latency 480.961 ms 2024-04-23T18:29:52.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 692490 5.34 MB/sec execute 506 sec latency 40.187 ms 2024-04-23T18:29:52.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:52 smithi050 ceph-mon[29187]: pgmap v356: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 700 KiB/s wr, 54 op/s 2024-04-23T18:29:52.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:52 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:52 smithi177 ceph-mon[32308]: pgmap v356: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 700 KiB/s wr, 54 op/s 2024-04-23T18:29:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:52 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:52.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:52 smithi188 ceph-mon[32694]: pgmap v356: 97 pgs: 97 active+clean; 223 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 700 KiB/s wr, 54 op/s 2024-04-23T18:29:52.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:52 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:29:53.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 693560 5.34 MB/sec execute 507 sec latency 104.080 ms 2024-04-23T18:29:54.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 694183 5.33 MB/sec execute 508 sec latency 697.203 ms 2024-04-23T18:29:54.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:54 smithi050 ceph-mon[29187]: pgmap v357: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.6 MiB/s wr, 117 op/s 2024-04-23T18:29:54.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:54 smithi177 ceph-mon[32308]: pgmap v357: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.6 MiB/s wr, 117 op/s 2024-04-23T18:29:54.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:54 smithi188 ceph-mon[32694]: pgmap v357: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.6 MiB/s wr, 117 op/s 2024-04-23T18:29:55.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 696176 5.35 MB/sec execute 509 sec latency 18.211 ms 2024-04-23T18:29:56.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 697441 5.35 MB/sec execute 510 sec latency 20.128 ms 2024-04-23T18:29:56.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:56 smithi050 ceph-mon[29187]: pgmap v358: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.5 MiB/s wr, 91 op/s 2024-04-23T18:29:56.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:56 smithi177 ceph-mon[32308]: pgmap v358: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.5 MiB/s wr, 91 op/s 2024-04-23T18:29:56.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:56 smithi188 ceph-mon[32694]: pgmap v358: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.5 MiB/s wr, 91 op/s 2024-04-23T18:29:57.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 698917 5.35 MB/sec execute 511 sec latency 55.966 ms 2024-04-23T18:29:58.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 700154 5.36 MB/sec execute 512 sec latency 18.476 ms 2024-04-23T18:29:58.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:29:58 smithi050 ceph-mon[29187]: pgmap v359: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 183 op/s 2024-04-23T18:29:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:29:58 smithi177 ceph-mon[32308]: pgmap v359: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 183 op/s 2024-04-23T18:29:58.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:29:58 smithi188 ceph-mon[32694]: pgmap v359: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 183 op/s 2024-04-23T18:29:59.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 701930 5.36 MB/sec execute 513 sec latency 30.340 ms 2024-04-23T18:30:00.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 703515 5.37 MB/sec execute 514 sec latency 15.997 ms 2024-04-23T18:30:00.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:00 smithi050 ceph-mon[29187]: pgmap v360: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 154 op/s 2024-04-23T18:30:00.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:00 smithi050 ceph-mon[29187]: overall HEALTH_OK 2024-04-23T18:30:00.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:00 smithi177 ceph-mon[32308]: pgmap v360: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 154 op/s 2024-04-23T18:30:00.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:00 smithi177 ceph-mon[32308]: overall HEALTH_OK 2024-04-23T18:30:00.787 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:00 smithi188 ceph-mon[32694]: pgmap v360: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 154 op/s 2024-04-23T18:30:00.787 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:00 smithi188 ceph-mon[32694]: overall HEALTH_OK 2024-04-23T18:30:01.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 704626 5.37 MB/sec execute 515 sec latency 24.517 ms 2024-04-23T18:30:02.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 705579 5.37 MB/sec execute 516 sec latency 43.867 ms 2024-04-23T18:30:02.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:02 smithi050 ceph-mon[29187]: pgmap v361: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 154 op/s 2024-04-23T18:30:02.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:02 smithi177 ceph-mon[32308]: pgmap v361: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 154 op/s 2024-04-23T18:30:02.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:02 smithi188 ceph-mon[32694]: pgmap v361: 97 pgs: 97 active+clean; 226 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 154 op/s 2024-04-23T18:30:03.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 707056 5.38 MB/sec execute 517 sec latency 134.809 ms 2024-04-23T18:30:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.168+0000 7fdb8bafd640 1 -- 172.21.15.50:0/3434110057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb84106170 msgr2=0x7fdb8410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.168+0000 7fdb8bafd640 1 --2- 172.21.15.50:0/3434110057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb84106170 0x7fdb8410dd40 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009ca0 tx=0x7fdb7c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.168+0000 7fdb8bafd640 1 -- 172.21.15.50:0/3434110057 shutdown_connections 2024-04-23T18:30:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.168+0000 7fdb8bafd640 1 --2- 172.21.15.50:0/3434110057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb84106170 0x7fdb8410dd40 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.168+0000 7fdb8bafd640 1 --2- 172.21.15.50:0/3434110057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb84101aa0 0x7fdb84105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.168+0000 7fdb8bafd640 1 --2- 172.21.15.50:0/3434110057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb841010f0 0x7fdb841014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.168+0000 7fdb8bafd640 1 -- 172.21.15.50:0/3434110057 >> 172.21.15.50:0/3434110057 conn(0x7fdb84077fd0 msgr2=0x7fdb840783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.168+0000 7fdb8bafd640 1 -- 172.21.15.50:0/3434110057 shutdown_connections 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.168+0000 7fdb8bafd640 1 -- 172.21.15.50:0/3434110057 wait complete. 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb8bafd640 1 Processor -- start 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb8bafd640 1 -- start start 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb8bafd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb841010f0 0x7fdb84198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb8bafd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb84106170 0x7fdb84198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb8bafd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb8419bf80 0x7fdb8419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb8bafd640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841110e0 con 0x7fdb8419bf80 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb8bafd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84110f60 con 0x7fdb841010f0 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb8bafd640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84111260 con 0x7fdb84106170 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb8a09a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb8419bf80 0x7fdb8419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb89899640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb841010f0 0x7fdb84198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb8a09a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb8419bf80 0x7fdb8419c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53780/0 (socket says 172.21.15.50:53780) 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb8a09a640 1 -- 172.21.15.50:0/98201030 learned_addr learned my addr 172.21.15.50:0/98201030 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb89098640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb84106170 0x7fdb84198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb89899640 1 -- 172.21.15.50:0/98201030 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb84106170 msgr2=0x7fdb84198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb89899640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb84106170 0x7fdb84198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb89899640 1 -- 172.21.15.50:0/98201030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb8419bf80 msgr2=0x7fdb8419c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb89899640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb8419bf80 0x7fdb8419c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb89899640 1 -- 172.21.15.50:0/98201030 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb8419ca70 con 0x7fdb841010f0 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb89098640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb84106170 0x7fdb84198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.169+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb8419bf80 0x7fdb8419c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.170+0000 7fdb89899640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb841010f0 0x7fdb84198770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdb80009bc0 tx=0x7fdb8000b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.170+0000 7fdb7affd640 1 -- 172.21.15.50:0/98201030 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8001e070 con 0x7fdb841010f0 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.170+0000 7fdb8bafd640 1 -- 172.21.15.50:0/98201030 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb8419cd60 con 0x7fdb841010f0 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.170+0000 7fdb7affd640 1 -- 172.21.15.50:0/98201030 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb800198c0 con 0x7fdb841010f0 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.170+0000 7fdb7affd640 1 -- 172.21.15.50:0/98201030 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb800187d0 con 0x7fdb841010f0 2024-04-23T18:30:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.170+0000 7fdb8bafd640 1 -- 172.21.15.50:0/98201030 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb84102d50 con 0x7fdb841010f0 2024-04-23T18:30:04.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.172+0000 7fdb8e80e740 1 Processor -- start 2024-04-23T18:30:04.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8e80e740 1 -- start start 2024-04-23T18:30:04.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb7affd640 1 -- 172.21.15.50:0/98201030 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdb80021070 con 0x7fdb841010f0 2024-04-23T18:30:04.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8e80e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363cf620 0x5620363cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8e80e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 0x5620363da2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8e80e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363da800 0x5620363dcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8e80e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562036334970 con 0x5620363cff40 2024-04-23T18:30:04.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8e80e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5620362d14a0 con 0x5620363cf620 2024-04-23T18:30:04.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8e80e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562036330930 con 0x5620363da800 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb89899640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 0x5620363da2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb89098640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363da800 0x5620363dcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8a09a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363cf620 0x5620363cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb89098640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363da800 0x5620363dcbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:45470/0 (socket says 172.21.15.50:45470) 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb89899640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 0x5620363da2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53784/0 (socket says 172.21.15.50:53784) 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb89098640 1 -- 172.21.15.50:0/1924916822 learned_addr learned my addr 172.21.15.50:0/1924916822 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8a09a640 1 -- 172.21.15.50:0/1924916822 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363da800 msgr2=0x5620363dcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/1924916822 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363da800 0x5620363dcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8a09a640 1 -- 172.21.15.50:0/1924916822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 msgr2=0x5620363da2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/1924916822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 0x5620363da2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb8a09a640 1 -- 172.21.15.50:0/1924916822 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562036047210 con 0x5620363cf620 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.173+0000 7fdb89899640 1 --2- 172.21.15.50:0/1924916822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 0x5620363da2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.174+0000 7fdb7affd640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdb700a3dc0 0x7fdb700a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.174+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/1924916822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363cf620 0x5620363cfa00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c005cc0 tx=0x7fdb7c005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11fbd2390f826d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:04.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.174+0000 7fdb89899640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdb700a3dc0 0x7fdb700a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.174+0000 7fdb6ffff640 1 -- 172.21.15.50:0/1924916822 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c031450 con 0x5620363cf620 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.174+0000 7fdb7affd640 1 -- 172.21.15.50:0/98201030 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7fdb800d3f90 con 0x7fdb841010f0 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.174+0000 7fdb6ffff640 1 -- 172.21.15.50:0/1924916822 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb7c036040 con 0x5620363cf620 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.174+0000 7fdb6ffff640 1 -- 172.21.15.50:0/1924916822 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c004400 con 0x5620363cf620 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.175+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1924916822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363cf620 msgr2=0x5620363cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.175+0000 7fdb89899640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdb700a3dc0 0x7fdb700a6280 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x5620363d0bc0 tx=0x7fdb800d7990 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.175+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/1924916822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363cf620 0x5620363cfa00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c005cc0 tx=0x7fdb7c005f80 comp rx=0 tx=0).stop 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.175+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1924916822 shutdown_connections 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.175+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/1924916822 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363da800 0x5620363dcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.175+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/1924916822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 0x5620363da2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.175+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/1924916822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363cf620 0x5620363cfa00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.175+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1924916822 >> 172.21.15.50:0/1924916822 conn(0x5620363cbb30 msgr2=0x5620363cbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.175+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1924916822 shutdown_connections 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.175+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1924916822 wait complete. 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.175+0000 7fdb8e80e740 1 -- start start 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.175+0000 7fdb8e80e740 1 Processor -- start 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb8e80e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 0x5620363da1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb8e80e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363cff40 0x5620363d74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb8e80e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363da800 0x5620363d7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb8e80e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5620363dd250 con 0x5620363cf620 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb8e80e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5620362cf0b0 con 0x5620363da800 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb8e80e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5620363dd3d0 con 0x5620363cff40 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb8e80e740 10 client.0 _ll_register_callbacks cb 0x5620362bcd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb8a09a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 0x5620363da1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb8a09a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 0x5620363da1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53792/0 (socket says 172.21.15.50:53792) 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb8a09a640 1 -- 172.21.15.50:0/1428832855 learned_addr learned my addr 172.21.15.50:0/1428832855 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:04.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb89098640 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363cff40 0x5620363d74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363da800 0x5620363d7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb89098640 1 -- 172.21.15.50:0/1428832855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363da800 msgr2=0x5620363d7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb89098640 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363da800 0x5620363d7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb89098640 1 -- 172.21.15.50:0/1428832855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 msgr2=0x5620363da1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb89098640 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 0x5620363da1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb89098640 1 -- 172.21.15.50:0/1428832855 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562036113620 con 0x5620363cff40 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 0x5620363da1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb89098640 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363cff40 0x5620363d74b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5620363cccc0 tx=0x7fdb7400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55ed6f9bd6c075d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.176+0000 7fdb6d7fa640 1 -- 172.21.15.50:0/1428832855 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7400fe60 con 0x5620363cff40 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1428832855 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb7c036040 con 0x5620363cff40 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb6d7fa640 1 -- 172.21.15.50:0/1428832855 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8542+0+0 (secure 0 0 0) 0x7fdb740120d0 con 0x5620363cff40 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb6d7fa640 1 -- 172.21.15.50:0/1428832855 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fdb74004020 con 0x5620363cff40 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb8e80e740 10 client.26488 fetch_fsmap learned FSMap version 46 2024-04-23T18:30:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1428832855 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5620363e9290 con 0x5620363cff40 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb6d7fa640 1 -- 172.21.15.50:0/1428832855 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb7c036040 con 0x5620363cff40 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb6d7fa640 1 -- 172.21.15.50:0/1428832855 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 46) ==== 8362+0+0 (secure 0 0 0) 0x7fdb7401c470 con 0x5620363cff40 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb8e80e740 10 client.26488 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb8e80e740 10 client.26488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb8e80e740 20 client.26488 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5620363ea750 0x5620363ecb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb8e80e740 4 client.26488 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1428832855 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562036342390 con 0x5620363ea750 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.177+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5620363ea750 0x5620363ecb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.178+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5620363ea750 0x5620363ecb40 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1cc44ed2c549b32 server_cookie=c8c4d34441043c15 in_seq=0 out_seq=0 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.178+0000 7fdb6d7fa640 10 client.26488 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.178+0000 7fdb6d7fa640 1 -- 172.21.15.50:0/1428832855 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562036342390 con 0x5620363ea750 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.178+0000 7fdb6d7fa640 10 client.26488 handle_command_reply: tid=0 2024-04-23T18:30:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb8e80e740 1 client.26488 shutdown 2024-04-23T18:30:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1428832855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363cff40 msgr2=0x5620363d74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363cff40 0x5620363d74b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5620363cccc0 tx=0x7fdb7400f790 comp rx=0 tx=0).stop 2024-04-23T18:30:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1428832855 shutdown_connections 2024-04-23T18:30:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5620363ea750 0x5620363ecb40 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363da800 0x5620363d7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb6d7fa640 0 client.26488 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363cff40 0x5620363d74b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/1428832855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 0x5620363da1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1428832855 >> 172.21.15.50:0/1428832855 conn(0x5620363cbb30 msgr2=0x5620363d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1428832855 shutdown_connections 2024-04-23T18:30:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb8e80e740 1 -- 172.21.15.50:0/1428832855 wait complete. 2024-04-23T18:30:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.179+0000 7fdb8e80e740 20 client.26488 trim_cache size 0 max 16384 2024-04-23T18:30:04.205 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.211+0000 7fdb8e80e740 1 Processor -- start 2024-04-23T18:30:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.213+0000 7fdb8e80e740 1 -- start start 2024-04-23T18:30:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.213+0000 7fdb8e80e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363cf620 0x5620363e6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.213+0000 7fdb8e80e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 0x5620363d2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.213+0000 7fdb8e80e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363da800 0x5620363d9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.213+0000 7fdb8e80e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841ae220 con 0x5620363cff40 2024-04-23T18:30:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.213+0000 7fdb8e80e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841aba30 con 0x5620363da800 2024-04-23T18:30:04.206 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.213+0000 7fdb8e80e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841a98c0 con 0x5620363cf620 2024-04-23T18:30:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.214+0000 7fdb89098640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 0x5620363d2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.214+0000 7fdb89098640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 0x5620363d2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53798/0 (socket says 172.21.15.50:53798) 2024-04-23T18:30:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.214+0000 7fdb89098640 1 -- 172.21.15.50:0/3121293714 learned_addr learned my addr 172.21.15.50:0/3121293714 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.214+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/3121293714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363cf620 0x5620363e6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.214+0000 7fdb89098640 1 -- 172.21.15.50:0/3121293714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363cf620 msgr2=0x5620363e6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.214+0000 7fdb89098640 1 --2- 172.21.15.50:0/3121293714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363cf620 0x5620363e6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.207 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.214+0000 7fdb89098640 1 -- 172.21.15.50:0/3121293714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363da800 msgr2=0x5620363d9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.214+0000 7fdb89098640 1 --2- 172.21.15.50:0/3121293714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363da800 0x5620363d9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.214+0000 7fdb89098640 1 -- 172.21.15.50:0/3121293714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620363ed1c0 con 0x5620363cff40 2024-04-23T18:30:04.208 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.215+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/3121293714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363cf620 0x5620363e6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:30:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.217+0000 7fdb89098640 1 --2- 172.21.15.50:0/3121293714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 0x5620363d2c10 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fdb74001eb0 tx=0x7fdb7401dee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1049d60880340c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.217+0000 7fdb6f7fe640 1 -- 172.21.15.50:0/3121293714 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb74028030 con 0x5620363cff40 2024-04-23T18:30:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.217+0000 7fdb6f7fe640 1 -- 172.21.15.50:0/3121293714 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb740120d0 con 0x5620363cff40 2024-04-23T18:30:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.217+0000 7fdb6f7fe640 1 -- 172.21.15.50:0/3121293714 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb74005470 con 0x5620363cff40 2024-04-23T18:30:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.218+0000 7fdb8e80e740 1 -- 172.21.15.50:0/3121293714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 msgr2=0x5620363d2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.218+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/3121293714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 0x5620363d2c10 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fdb74001eb0 tx=0x7fdb7401dee0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.219+0000 7fdb8e80e740 1 -- 172.21.15.50:0/3121293714 shutdown_connections 2024-04-23T18:30:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.219+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/3121293714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363da800 0x5620363d9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.219+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/3121293714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cff40 0x5620363d2c10 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.219+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/3121293714 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363cf620 0x5620363e6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.219+0000 7fdb8e80e740 1 -- 172.21.15.50:0/3121293714 >> 172.21.15.50:0/3121293714 conn(0x5620363cbb30 msgr2=0x5620363d1860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.220+0000 7fdb8e80e740 1 -- 172.21.15.50:0/3121293714 shutdown_connections 2024-04-23T18:30:04.214 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.222+0000 7fdb8e80e740 1 -- 172.21.15.50:0/3121293714 wait complete. 2024-04-23T18:30:04.215 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.222+0000 7fdb8e80e740 1 -- start start 2024-04-23T18:30:04.215 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.223+0000 7fdb8e80e740 1 Processor -- start 2024-04-23T18:30:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.223+0000 7fdb8e80e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 0x5620363ddfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.224+0000 7fdb8e80e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363cff40 0x5620363de4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.224+0000 7fdb8e80e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363da800 0x5620363dea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.224+0000 7fdb8a09a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 0x5620363ddfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.224+0000 7fdb8a09a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 0x5620363ddfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53810/0 (socket says 172.21.15.50:53810) 2024-04-23T18:30:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.224+0000 7fdb8e80e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841a9f10 con 0x5620363cf620 2024-04-23T18:30:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.224+0000 7fdb8e80e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841ab360 con 0x5620363cff40 2024-04-23T18:30:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.224+0000 7fdb8e80e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841aa5c0 con 0x5620363da800 2024-04-23T18:30:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.224+0000 7fdb8e80e740 10 client.0 _ll_register_callbacks cb 0x7fdb74012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.224+0000 7fdb8a09a640 1 -- 172.21.15.50:0/2349808702 learned_addr learned my addr 172.21.15.50:0/2349808702 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.226+0000 7fdb89098640 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363cff40 0x5620363de4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.226+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363da800 0x5620363dea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.227+0000 7fdb8a09a640 1 -- 172.21.15.50:0/2349808702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363da800 msgr2=0x5620363dea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.227+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363da800 0x5620363dea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.227+0000 7fdb8a09a640 1 -- 172.21.15.50:0/2349808702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363cff40 msgr2=0x5620363de4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.227+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363cff40 0x5620363de4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.227+0000 7fdb8a09a640 1 -- 172.21.15.50:0/2349808702 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620364584d0 con 0x5620363cf620 2024-04-23T18:30:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.230+0000 7fdb8a09a640 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 0x5620363ddfa0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c000f80 tx=0x7fdb7c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dff903ef2b25cd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.230+0000 7fdb6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.231+0000 7fdb6d7fa640 1 -- 172.21.15.50:0/2349808702 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c0316e0 con 0x5620363cf620 2024-04-23T18:30:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.231+0000 7fdb6d7fa640 1 -- 172.21.15.50:0/2349808702 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8586+0+0 (secure 0 0 0) 0x7fdb7c031880 con 0x5620363cf620 2024-04-23T18:30:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.231+0000 7fdb8e80e740 1 -- 172.21.15.50:0/2349808702 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb740120d0 con 0x5620363cf620 2024-04-23T18:30:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.233+0000 7fdb6d7fa640 1 -- 172.21.15.50:0/2349808702 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fdb7c059140 con 0x5620363cf620 2024-04-23T18:30:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.233+0000 7fdb8e80e740 10 client.16908 fetch_fsmap learned FSMap version 46 2024-04-23T18:30:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.233+0000 7fdb8e80e740 1 -- 172.21.15.50:0/2349808702 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x562036459690 con 0x5620363cf620 2024-04-23T18:30:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.234+0000 7fdb6d7fa640 1 -- 172.21.15.50:0/2349808702 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb740120d0 con 0x5620363cf620 2024-04-23T18:30:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.234+0000 7fdb6d7fa640 1 -- 172.21.15.50:0/2349808702 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 46) ==== 8362+0+0 (secure 0 0 0) 0x7fdb7c05f030 con 0x5620363cf620 2024-04-23T18:30:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.234+0000 7fdb8e80e740 10 client.16908 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:30:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.234+0000 7fdb8e80e740 10 client.16908 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.234+0000 7fdb8e80e740 20 client.16908 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.234+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562036459a30 0x56203645be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.234+0000 7fdb8e80e740 4 client.16908 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:30:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.234+0000 7fdb8e80e740 1 -- 172.21.15.50:0/2349808702 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56203633c200 con 0x562036459a30 2024-04-23T18:30:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.235+0000 7fdb89098640 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562036459a30 0x56203645be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:04.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.238+0000 7fdb89098640 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562036459a30 0x56203645be20 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbb932e4a8970502 server_cookie=adaedbd6af0d0ed9 in_seq=0 out_seq=0 2024-04-23T18:30:04.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.238+0000 7fdb6d7fa640 10 client.16908 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:04.232 INFO:tasks.workunit.client.0.smithi050.stdout: 1 708449 5.38 MB/sec execute 518 sec latency 30.354 ms 2024-04-23T18:30:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.333+0000 7fdb6d7fa640 1 -- 172.21.15.50:0/2349808702 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56203633c200 con 0x562036459a30 2024-04-23T18:30:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.333+0000 7fdb6d7fa640 10 client.16908 handle_command_reply: tid=0 2024-04-23T18:30:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb8e80e740 1 client.16908 shutdown 2024-04-23T18:30:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb8e80e740 1 -- 172.21.15.50:0/2349808702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 msgr2=0x5620363ddfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 0x5620363ddfa0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c000f80 tx=0x7fdb7c0095c0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb8e80e740 1 -- 172.21.15.50:0/2349808702 shutdown_connections 2024-04-23T18:30:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562036459a30 0x56203645be20 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5620363da800 0x5620363dea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620363cff40 0x5620363de4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb8e80e740 1 --2- 172.21.15.50:0/2349808702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5620363cf620 0x5620363ddfa0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb8e80e740 1 -- 172.21.15.50:0/2349808702 >> 172.21.15.50:0/2349808702 conn(0x5620363cbb30 msgr2=0x5620363e5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb6d7fa640 0 client.16908 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb8e80e740 1 -- 172.21.15.50:0/2349808702 shutdown_connections 2024-04-23T18:30:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb8e80e740 1 -- 172.21.15.50:0/2349808702 wait complete. 2024-04-23T18:30:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.334+0000 7fdb8e80e740 20 client.16908 trim_cache size 0 max 16384 2024-04-23T18:30:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:30:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:30:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:30:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:30:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.335+0000 7fdb8bafd640 1 -- 172.21.15.50:0/98201030 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdb700a3dc0 msgr2=0x7fdb700a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.335+0000 7fdb8bafd640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdb700a3dc0 0x7fdb700a6280 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x5620363d0bc0 tx=0x7fdb800d7990 comp rx=0 tx=0).stop 2024-04-23T18:30:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.336+0000 7fdb8bafd640 1 -- 172.21.15.50:0/98201030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb841010f0 msgr2=0x7fdb84198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.336+0000 7fdb8bafd640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb841010f0 0x7fdb84198770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdb80009bc0 tx=0x7fdb8000b690 comp rx=0 tx=0).stop 2024-04-23T18:30:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.336+0000 7fdb8bafd640 1 -- 172.21.15.50:0/98201030 shutdown_connections 2024-04-23T18:30:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.336+0000 7fdb8bafd640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdb700a3dc0 0x7fdb700a6280 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.336+0000 7fdb8bafd640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb8419bf80 0x7fdb8419c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.336+0000 7fdb8bafd640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb84106170 0x7fdb84198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.336+0000 7fdb8bafd640 1 --2- 172.21.15.50:0/98201030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb841010f0 0x7fdb84198770 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:04.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.336+0000 7fdb8bafd640 1 -- 172.21.15.50:0/98201030 >> 172.21.15.50:0/98201030 conn(0x7fdb84077fd0 msgr2=0x7fdb84069970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:04.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.336+0000 7fdb8bafd640 1 -- 172.21.15.50:0/98201030 shutdown_connections 2024-04-23T18:30:04.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:04.336+0000 7fdb8bafd640 1 -- 172.21.15.50:0/98201030 wait complete. 2024-04-23T18:30:04.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:04 smithi050 ceph-mon[29187]: pgmap v362: 97 pgs: 97 active+clean; 232 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 264 op/s 2024-04-23T18:30:04.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:04 smithi177 ceph-mon[32308]: pgmap v362: 97 pgs: 97 active+clean; 232 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 264 op/s 2024-04-23T18:30:04.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:04 smithi188 ceph-mon[32694]: pgmap v362: 97 pgs: 97 active+clean; 232 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 264 op/s 2024-04-23T18:30:05.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 710302 5.39 MB/sec execute 519 sec latency 55.382 ms 2024-04-23T18:30:05.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:05 smithi050 ceph-mon[29187]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T18:30:05.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:05 smithi177 ceph-mon[32308]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T18:30:05.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:05 smithi188 ceph-mon[32694]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T18:30:06.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 711608 5.39 MB/sec execute 520 sec latency 20.316 ms 2024-04-23T18:30:06.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:06 smithi050 ceph-mon[29187]: pgmap v364: 97 pgs: 97 active+clean; 232 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 241 op/s 2024-04-23T18:30:06.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:06 smithi177 ceph-mon[32308]: pgmap v364: 97 pgs: 97 active+clean; 232 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 241 op/s 2024-04-23T18:30:06.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:06 smithi188 ceph-mon[32694]: pgmap v364: 97 pgs: 97 active+clean; 232 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 241 op/s 2024-04-23T18:30:07.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 713748 5.41 MB/sec execute 521 sec latency 18.191 ms 2024-04-23T18:30:08.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 714948 5.40 MB/sec execute 522 sec latency 17.430 ms 2024-04-23T18:30:08.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:08 smithi050 ceph-mon[29187]: pgmap v365: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 294 op/s 2024-04-23T18:30:08.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:08 smithi177 ceph-mon[32308]: pgmap v365: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 294 op/s 2024-04-23T18:30:08.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:08 smithi188 ceph-mon[32694]: pgmap v365: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 294 op/s 2024-04-23T18:30:09.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 716680 5.41 MB/sec execute 523 sec latency 21.081 ms 2024-04-23T18:30:10.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 718058 5.42 MB/sec execute 524 sec latency 21.762 ms 2024-04-23T18:30:10.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:10 smithi050 ceph-mon[29187]: pgmap v366: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 294 op/s 2024-04-23T18:30:10.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:10 smithi177 ceph-mon[32308]: pgmap v366: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 294 op/s 2024-04-23T18:30:10.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:10 smithi188 ceph-mon[32694]: pgmap v366: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 294 op/s 2024-04-23T18:30:11.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 719232 5.42 MB/sec execute 525 sec latency 21.200 ms 2024-04-23T18:30:12.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 721158 5.43 MB/sec execute 526 sec latency 13.567 ms 2024-04-23T18:30:12.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:12 smithi050 ceph-mon[29187]: pgmap v367: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 294 op/s 2024-04-23T18:30:12.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:12 smithi050 ceph-mon[29187]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T18:30:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:12 smithi177 ceph-mon[32308]: pgmap v367: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 294 op/s 2024-04-23T18:30:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:12 smithi177 ceph-mon[32308]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T18:30:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:12 smithi188 ceph-mon[32694]: pgmap v367: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 294 op/s 2024-04-23T18:30:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:12 smithi188 ceph-mon[32694]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T18:30:13.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 722352 5.43 MB/sec execute 527 sec latency 37.426 ms 2024-04-23T18:30:14.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 724365 5.44 MB/sec execute 528 sec latency 15.771 ms 2024-04-23T18:30:14.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:14 smithi050 ceph-mon[29187]: pgmap v369: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 301 op/s 2024-04-23T18:30:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:14 smithi177 ceph-mon[32308]: pgmap v369: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 301 op/s 2024-04-23T18:30:14.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:14 smithi188 ceph-mon[32694]: pgmap v369: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 301 op/s 2024-04-23T18:30:15.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 725636 5.44 MB/sec execute 529 sec latency 20.587 ms 2024-04-23T18:30:16.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 727773 5.45 MB/sec execute 530 sec latency 21.565 ms 2024-04-23T18:30:16.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:16 smithi050 ceph-mon[29187]: pgmap v370: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 256 op/s 2024-04-23T18:30:16.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:16 smithi177 ceph-mon[32308]: pgmap v370: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 256 op/s 2024-04-23T18:30:16.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:16 smithi188 ceph-mon[32694]: pgmap v370: 97 pgs: 97 active+clean; 234 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 256 op/s 2024-04-23T18:30:17.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 729197 5.46 MB/sec execute 531 sec latency 18.313 ms 2024-04-23T18:30:17.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.436+0000 7fa0337ad640 1 -- 172.21.15.50:0/2445497205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa02c10e940 msgr2=0x7fa02c110d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:17.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.436+0000 7fa0337ad640 1 --2- 172.21.15.50:0/2445497205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa02c10e940 0x7fa02c110d30 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa024009ca0 tx=0x7fa02402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.436+0000 7fa0337ad640 1 -- 172.21.15.50:0/2445497205 shutdown_connections 2024-04-23T18:30:17.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.436+0000 7fa0337ad640 1 --2- 172.21.15.50:0/2445497205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa02c10e940 0x7fa02c110d30 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.436+0000 7fa0337ad640 1 --2- 172.21.15.50:0/2445497205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa02c06ad50 0x7fa02c06b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.436+0000 7fa0337ad640 1 --2- 172.21.15.50:0/2445497205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa02c102fa0 0x7fa02c06a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.436+0000 7fa0337ad640 1 -- 172.21.15.50:0/2445497205 >> 172.21.15.50:0/2445497205 conn(0x7fa02c076c50 msgr2=0x7fa02c077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:17.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.437+0000 7fa0337ad640 1 -- 172.21.15.50:0/2445497205 shutdown_connections 2024-04-23T18:30:17.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.437+0000 7fa0337ad640 1 -- 172.21.15.50:0/2445497205 wait complete. 2024-04-23T18:30:17.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.437+0000 7fa0337ad640 1 Processor -- start 2024-04-23T18:30:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.437+0000 7fa0337ad640 1 -- start start 2024-04-23T18:30:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.437+0000 7fa0337ad640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa02c06ad50 0x7fa02c1a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.437+0000 7fa0337ad640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa02c102fa0 0x7fa02c1a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.437+0000 7fa0337ad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa02c10e940 0x7fa02c19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.437+0000 7fa0337ad640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1143b0 con 0x7fa02c10e940 2024-04-23T18:30:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.437+0000 7fa0337ad640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c114230 con 0x7fa02c06ad50 2024-04-23T18:30:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.437+0000 7fa0337ad640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c114530 con 0x7fa02c102fa0 2024-04-23T18:30:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa031d4a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa02c10e940 0x7fa02c19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa031d4a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa02c10e940 0x7fa02c19a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36094/0 (socket says 172.21.15.50:36094) 2024-04-23T18:30:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa031d4a640 1 -- 172.21.15.50:0/2021848968 learned_addr learned my addr 172.21.15.50:0/2021848968 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa030d48640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa02c102fa0 0x7fa02c1a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa031549640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa02c06ad50 0x7fa02c1a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa031d4a640 1 -- 172.21.15.50:0/2021848968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa02c102fa0 msgr2=0x7fa02c1a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa031d4a640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa02c102fa0 0x7fa02c1a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa031d4a640 1 -- 172.21.15.50:0/2021848968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa02c06ad50 msgr2=0x7fa02c1a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa031d4a640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa02c06ad50 0x7fa02c1a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa031d4a640 1 -- 172.21.15.50:0/2021848968 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa02c19ad70 con 0x7fa02c10e940 2024-04-23T18:30:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa031d4a640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa02c10e940 0x7fa02c19a5f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa024009430 tx=0x7fa024031660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa0227fc640 1 -- 172.21.15.50:0/2021848968 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa024003b70 con 0x7fa02c10e940 2024-04-23T18:30:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa0227fc640 1 -- 172.21.15.50:0/2021848968 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa024003d10 con 0x7fa02c10e940 2024-04-23T18:30:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa0337ad640 1 -- 172.21.15.50:0/2021848968 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa02c19b000 con 0x7fa02c10e940 2024-04-23T18:30:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa0227fc640 1 -- 172.21.15.50:0/2021848968 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02402fa50 con 0x7fa02c10e940 2024-04-23T18:30:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.438+0000 7fa0337ad640 1 -- 172.21.15.50:0/2021848968 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa02c1a7300 con 0x7fa02c10e940 2024-04-23T18:30:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.440+0000 7fa0227fc640 1 -- 172.21.15.50:0/2021848968 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa02402fbf0 con 0x7fa02c10e940 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa0365a3740 1 Processor -- start 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa0365a3740 1 -- start start 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa0365a3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a6967a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa0365a3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 0x5610a69722c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa0365a3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6972800 0x5610a6974bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa0365a3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5610a68cc970 con 0x5610a6967f40 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa0365a3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5610a68694a0 con 0x5610a6972800 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa0365a3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5610a68c8930 con 0x5610a6967620 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa0227fc640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0180a3dc0 0x7fa0180a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa030d48640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a6967a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa031549640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 0x5610a69722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa030d48640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a6967a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:46182/0 (socket says 172.21.15.50:46182) 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa030d48640 1 -- 172.21.15.50:0/1295441336 learned_addr learned my addr 172.21.15.50:0/1295441336 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa030d48640 1 --2- 172.21.15.50:0/1295441336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6972800 0x5610a6974bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.441+0000 7fa031d4a640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0180a3dc0 0x7fa0180a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.442+0000 7fa031549640 1 -- 172.21.15.50:0/1295441336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 msgr2=0x5610a6967a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.442+0000 7fa031549640 1 --2- 172.21.15.50:0/1295441336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a6967a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.442+0000 7fa031549640 1 -- 172.21.15.50:0/1295441336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6972800 msgr2=0x5610a6974bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.442+0000 7fa031549640 1 --2- 172.21.15.50:0/1295441336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6972800 0x5610a6974bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.442+0000 7fa031549640 1 -- 172.21.15.50:0/1295441336 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610a65df210 con 0x5610a6967f40 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.442+0000 7fa030d48640 1 --2- 172.21.15.50:0/1295441336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a6967a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.442+0000 7fa0227fc640 1 -- 172.21.15.50:0/2021848968 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7fa0240f1980 con 0x7fa02c10e940 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.442+0000 7fa031549640 1 --2- 172.21.15.50:0/1295441336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 0x5610a69722c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa028009c20 tx=0x7fa028011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4872a19e19d7e295 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.442+0000 7fa031d4a640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0180a3dc0 0x7fa0180a6280 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa0240ef280 tx=0x7fa0240b2eb0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.442+0000 7fa0177fe640 1 -- 172.21.15.50:0/1295441336 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028004050 con 0x5610a6967f40 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.442+0000 7fa0177fe640 1 -- 172.21.15.50:0/1295441336 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0280054d0 con 0x5610a6967f40 2024-04-23T18:30:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.442+0000 7fa0177fe640 1 -- 172.21.15.50:0/1295441336 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028004d00 con 0x5610a6967f40 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.443+0000 7fa0365a3740 1 -- 172.21.15.50:0/1295441336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 msgr2=0x5610a69722c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.443+0000 7fa0365a3740 1 --2- 172.21.15.50:0/1295441336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 0x5610a69722c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa028009c20 tx=0x7fa028011810 comp rx=0 tx=0).stop 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.443+0000 7fa0365a3740 1 -- 172.21.15.50:0/1295441336 shutdown_connections 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.443+0000 7fa0365a3740 1 --2- 172.21.15.50:0/1295441336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6972800 0x5610a6974bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.443+0000 7fa0365a3740 1 --2- 172.21.15.50:0/1295441336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 0x5610a69722c0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.443+0000 7fa0365a3740 1 --2- 172.21.15.50:0/1295441336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a6967a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.443+0000 7fa0365a3740 1 -- 172.21.15.50:0/1295441336 >> 172.21.15.50:0/1295441336 conn(0x5610a6963b30 msgr2=0x5610a6963f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.443+0000 7fa0365a3740 1 -- 172.21.15.50:0/1295441336 shutdown_connections 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.443+0000 7fa0365a3740 1 -- 172.21.15.50:0/1295441336 wait complete. 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.443+0000 7fa0365a3740 1 -- start start 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.443+0000 7fa0365a3740 1 Processor -- start 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa0365a3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a6972170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa0365a3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 0x5610a696f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa0365a3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6972800 0x5610a696f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa0365a3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5610a6975430 con 0x5610a6967f40 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa0365a3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5610a68670b0 con 0x5610a6972800 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa0365a3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5610a69755b0 con 0x5610a6967620 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa0365a3740 10 client.0 _ll_register_callbacks cb 0x5610a6854d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa031549640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 0x5610a696f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa031549640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 0x5610a696f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36116/0 (socket says 172.21.15.50:36116) 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa030d48640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a6972170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa031549640 1 -- 172.21.15.50:0/3209564189 learned_addr learned my addr 172.21.15.50:0/3209564189 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa030d48640 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6972800 0x5610a696f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa031549640 1 -- 172.21.15.50:0/3209564189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 msgr2=0x5610a6972170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa031549640 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a6972170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa031549640 1 -- 172.21.15.50:0/3209564189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6972800 msgr2=0x5610a696f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa031549640 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6972800 0x5610a696f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa031549640 1 -- 172.21.15.50:0/3209564189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610a66ab620 con 0x5610a6967f40 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa030d48640 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a6972170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa031549640 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 0x5610a696f460 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa02801b040 tx=0x7fa028004720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50cd798c80996198 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa014ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa014ff9640 1 -- 172.21.15.50:0/3209564189 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028029070 con 0x5610a6967f40 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa014ff9640 1 -- 172.21.15.50:0/3209564189 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7fa028003a00 con 0x5610a6967f40 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.444+0000 7fa0365a3740 1 -- 172.21.15.50:0/3209564189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0280054d0 con 0x5610a6967f40 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.445+0000 7fa014ff9640 1 -- 172.21.15.50:0/3209564189 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fa028035050 con 0x5610a6967f40 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.445+0000 7fa0365a3740 10 client.16920 fetch_fsmap learned FSMap version 46 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.445+0000 7fa0365a3740 1 -- 172.21.15.50:0/3209564189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5610a69813a0 con 0x5610a6967f40 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.445+0000 7fa014ff9640 1 -- 172.21.15.50:0/3209564189 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0280054d0 con 0x5610a6967f40 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.445+0000 7fa014ff9640 1 -- 172.21.15.50:0/3209564189 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 46) ==== 8362+0+0 (secure 0 0 0) 0x7fa028005160 con 0x5610a6967f40 2024-04-23T18:30:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.445+0000 7fa0365a3740 10 client.16920 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.445+0000 7fa0365a3740 10 client.16920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.445+0000 7fa0365a3740 20 client.16920 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.445+0000 7fa0365a3740 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5610a6982750 0x5610a6984b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.445+0000 7fa0365a3740 4 client.16920 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.445+0000 7fa0365a3740 1 -- 172.21.15.50:0/3209564189 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610a68da390 con 0x5610a6982750 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.445+0000 7fa030d48640 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5610a6982750 0x5610a6984b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.446+0000 7fa030d48640 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5610a6982750 0x5610a6984b40 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=237565c8d305ad20 server_cookie=d15d099ca0d343fb in_seq=0 out_seq=0 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.446+0000 7fa014ff9640 10 client.16920 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa014ff9640 1 -- 172.21.15.50:0/3209564189 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5610a68da390 con 0x5610a6982750 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa014ff9640 10 client.16920 handle_command_reply: tid=0 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa0365a3740 1 client.16920 shutdown 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa0365a3740 1 -- 172.21.15.50:0/3209564189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 msgr2=0x5610a696f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa0365a3740 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 0x5610a696f460 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa02801b040 tx=0x7fa028004720 comp rx=0 tx=0).stop 2024-04-23T18:30:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa0365a3740 1 -- 172.21.15.50:0/3209564189 shutdown_connections 2024-04-23T18:30:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa0365a3740 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5610a6982750 0x5610a6984b40 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa0365a3740 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6972800 0x5610a696f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa0365a3740 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6967f40 0x5610a696f460 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa014ff9640 0 client.16920 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa0365a3740 1 --2- 172.21.15.50:0/3209564189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a6972170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa0365a3740 1 -- 172.21.15.50:0/3209564189 >> 172.21.15.50:0/3209564189 conn(0x5610a6963b30 msgr2=0x5610a6969860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa0365a3740 1 -- 172.21.15.50:0/3209564189 shutdown_connections 2024-04-23T18:30:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa0365a3740 1 -- 172.21.15.50:0/3209564189 wait complete. 2024-04-23T18:30:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.447+0000 7fa0365a3740 20 client.16920 trim_cache size 0 max 16384 2024-04-23T18:30:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.463+0000 7fa0365a3740 1 Processor -- start 2024-04-23T18:30:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.463+0000 7fa0365a3740 1 -- start start 2024-04-23T18:30:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.463+0000 7fa0365a3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a697e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa0365a3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6967f40 0x5610a6971ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa0365a3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6972800 0x5610a698a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa0365a3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1b2670 con 0x5610a6972800 2024-04-23T18:30:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa0365a3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1afe80 con 0x5610a6967f40 2024-04-23T18:30:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa0365a3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1adcf0 con 0x5610a6967620 2024-04-23T18:30:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa030d48640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6972800 0x5610a698a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa030d48640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a697e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa030d48640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a697e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:46186/0 (socket says 172.21.15.50:46186) 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa030d48640 1 -- 172.21.15.50:0/3867415516 learned_addr learned my addr 172.21.15.50:0/3867415516 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa031549640 1 --2- 172.21.15.50:0/3867415516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6967f40 0x5610a6971ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa030d48640 1 -- 172.21.15.50:0/3867415516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6967f40 msgr2=0x5610a6971ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa030d48640 1 --2- 172.21.15.50:0/3867415516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6967f40 0x5610a6971ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa030d48640 1 -- 172.21.15.50:0/3867415516 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6972800 msgr2=0x5610a698a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa030d48640 1 --2- 172.21.15.50:0/3867415516 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6972800 0x5610a698a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa030d48640 1 -- 172.21.15.50:0/3867415516 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610a696e410 con 0x5610a6967620 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa030d48640 1 --2- 172.21.15.50:0/3867415516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a697e2f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009fd0 tx=0x7fa01c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcc460910e312d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa016ffd640 1 -- 172.21.15.50:0/3867415516 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c004220 con 0x5610a6967620 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa016ffd640 1 -- 172.21.15.50:0/3867415516 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa01c024e50 con 0x5610a6967620 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.464+0000 7fa016ffd640 1 -- 172.21.15.50:0/3867415516 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c010bf0 con 0x5610a6967620 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.465+0000 7fa0365a3740 1 -- 172.21.15.50:0/3867415516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 msgr2=0x5610a697e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.465+0000 7fa0365a3740 1 --2- 172.21.15.50:0/3867415516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a697e2f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009fd0 tx=0x7fa01c004730 comp rx=0 tx=0).stop 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.465+0000 7fa0365a3740 1 -- 172.21.15.50:0/3867415516 shutdown_connections 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.465+0000 7fa0365a3740 1 --2- 172.21.15.50:0/3867415516 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6972800 0x5610a698a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.465+0000 7fa0365a3740 1 --2- 172.21.15.50:0/3867415516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6967f40 0x5610a6971ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.465+0000 7fa0365a3740 1 --2- 172.21.15.50:0/3867415516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967620 0x5610a697e2f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.465+0000 7fa0365a3740 1 -- 172.21.15.50:0/3867415516 >> 172.21.15.50:0/3867415516 conn(0x5610a6963b30 msgr2=0x5610a696aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.465+0000 7fa0365a3740 1 -- 172.21.15.50:0/3867415516 shutdown_connections 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.465+0000 7fa0365a3740 1 -- 172.21.15.50:0/3867415516 wait complete. 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa0365a3740 1 -- start start 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa0365a3740 1 Processor -- start 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa0365a3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6967620 0x5610a697ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa0365a3740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967f40 0x5610a6975f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa0365a3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6972800 0x5610a69764f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa0365a3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1ae340 con 0x5610a6972800 2024-04-23T18:30:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa0365a3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1af7b0 con 0x5610a6967620 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa0365a3740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c1aea10 con 0x5610a6967f40 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa0365a3740 10 client.0 _ll_register_callbacks cb 0x7fa028004410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa030d48640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6972800 0x5610a69764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa030d48640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6972800 0x5610a69764f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36134/0 (socket says 172.21.15.50:36134) 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa030d48640 1 -- 172.21.15.50:0/2162149657 learned_addr learned my addr 172.21.15.50:0/2162149657 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa031549640 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967f40 0x5610a6975f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.466+0000 7fa030d48640 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6967620 0x5610a697ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa031549640 1 -- 172.21.15.50:0/2162149657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6967620 msgr2=0x5610a697ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa031549640 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6967620 0x5610a697ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa031549640 1 -- 172.21.15.50:0/2162149657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6972800 msgr2=0x5610a69764f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa031549640 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6972800 0x5610a69764f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa031549640 1 -- 172.21.15.50:0/2162149657 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610a6976d70 con 0x5610a6967f40 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa031549640 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967f40 0x5610a6975f90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa028001eb0 tx=0x7fa02802e650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1e8b4c2c1c7712c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa014ff9640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa014ff9640 1 -- 172.21.15.50:0/2162149657 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028029050 con 0x5610a6967f40 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa0365a3740 1 -- 172.21.15.50:0/2162149657 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa01c024e50 con 0x5610a6967f40 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa014ff9640 1 -- 172.21.15.50:0/2162149657 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7fa02802ead0 con 0x5610a6967f40 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa014ff9640 1 -- 172.21.15.50:0/2162149657 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fa02802ecd0 con 0x5610a6967f40 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa0365a3740 10 client.26509 fetch_fsmap learned FSMap version 46 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa0365a3740 1 -- 172.21.15.50:0/2162149657 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5610a69e8c80 con 0x5610a6967f40 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.467+0000 7fa014ff9640 1 -- 172.21.15.50:0/2162149657 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa01c024e50 con 0x5610a6967f40 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.468+0000 7fa014ff9640 1 -- 172.21.15.50:0/2162149657 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 46) ==== 8362+0+0 (secure 0 0 0) 0x7fa028003b30 con 0x5610a6967f40 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.468+0000 7fa0365a3740 10 client.26509 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.468+0000 7fa0365a3740 10 client.26509 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.468+0000 7fa0365a3740 20 client.26509 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.468+0000 7fa0365a3740 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5610a69e9360 0x5610a69eb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.468+0000 7fa0365a3740 4 client.26509 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.468+0000 7fa0365a3740 1 -- 172.21.15.50:0/2162149657 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5610a68d4200 con 0x5610a69e9360 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.468+0000 7fa030d48640 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5610a69e9360 0x5610a69eb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.468+0000 7fa014ff9640 10 client.26509 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.468+0000 7fa030d48640 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5610a69e9360 0x5610a69eb780 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c43cd055d847954 server_cookie=cafa60d54bf05d43 in_seq=0 out_seq=0 2024-04-23T18:30:17.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.529+0000 7fa014ff9640 1 -- 172.21.15.50:0/2162149657 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5610a68d4200 con 0x5610a69e9360 2024-04-23T18:30:17.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.529+0000 7fa014ff9640 10 client.26509 handle_command_reply: tid=0 2024-04-23T18:30:17.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.529+0000 7fa0365a3740 1 client.26509 shutdown 2024-04-23T18:30:17.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.529+0000 7fa0365a3740 1 -- 172.21.15.50:0/2162149657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967f40 msgr2=0x5610a6975f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:17.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.529+0000 7fa0365a3740 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967f40 0x5610a6975f90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa028001eb0 tx=0x7fa02802e650 comp rx=0 tx=0).stop 2024-04-23T18:30:17.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.529+0000 7fa0365a3740 1 -- 172.21.15.50:0/2162149657 shutdown_connections 2024-04-23T18:30:17.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.529+0000 7fa0365a3740 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5610a69e9360 0x5610a69eb780 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.530+0000 7fa0365a3740 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5610a6972800 0x5610a69764f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.530+0000 7fa014ff9640 0 client.26509 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.530+0000 7fa0365a3740 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5610a6967f40 0x5610a6975f90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.530+0000 7fa0365a3740 1 --2- 172.21.15.50:0/2162149657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a6967620 0x5610a697ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.530+0000 7fa0365a3740 1 -- 172.21.15.50:0/2162149657 >> 172.21.15.50:0/2162149657 conn(0x5610a6963b30 msgr2=0x5610a697d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.530+0000 7fa0365a3740 1 -- 172.21.15.50:0/2162149657 shutdown_connections 2024-04-23T18:30:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.530+0000 7fa0365a3740 1 -- 172.21.15.50:0/2162149657 wait complete. 2024-04-23T18:30:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.530+0000 7fa0365a3740 20 client.26509 trim_cache size 0 max 16384 2024-04-23T18:30:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:30:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:30:17.524 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:30:17.524 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:30:17.524 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.531+0000 7fa0337ad640 1 -- 172.21.15.50:0/2021848968 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0180a3dc0 msgr2=0x7fa0180a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:17.524 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.531+0000 7fa0337ad640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0180a3dc0 0x7fa0180a6280 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa0240ef280 tx=0x7fa0240b2eb0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.524 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.531+0000 7fa0337ad640 1 -- 172.21.15.50:0/2021848968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa02c10e940 msgr2=0x7fa02c19a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:17.525 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.531+0000 7fa0337ad640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa02c10e940 0x7fa02c19a5f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa024009430 tx=0x7fa024031660 comp rx=0 tx=0).stop 2024-04-23T18:30:17.525 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.531+0000 7fa0337ad640 1 -- 172.21.15.50:0/2021848968 shutdown_connections 2024-04-23T18:30:17.525 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.531+0000 7fa0337ad640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa0180a3dc0 0x7fa0180a6280 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.525 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.531+0000 7fa0337ad640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa02c10e940 0x7fa02c19a5f0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.525 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.531+0000 7fa0337ad640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa02c102fa0 0x7fa02c1a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.525 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.531+0000 7fa0337ad640 1 --2- 172.21.15.50:0/2021848968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa02c06ad50 0x7fa02c1a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:17.525 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.531+0000 7fa0337ad640 1 -- 172.21.15.50:0/2021848968 >> 172.21.15.50:0/2021848968 conn(0x7fa02c076c50 msgr2=0x7fa02c10f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:17.525 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.531+0000 7fa0337ad640 1 -- 172.21.15.50:0/2021848968 shutdown_connections 2024-04-23T18:30:17.525 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:17.531+0000 7fa0337ad640 1 -- 172.21.15.50:0/2021848968 wait complete. 2024-04-23T18:30:18.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 730861 5.47 MB/sec execute 532 sec latency 33.769 ms 2024-04-23T18:30:18.380 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:18 smithi188 ceph-mon[32694]: pgmap v371: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-23T18:30:18.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:18 smithi050 ceph-mon[29187]: pgmap v371: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-23T18:30:18.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:18 smithi177 ceph-mon[32308]: pgmap v371: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-23T18:30:19.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 732308 5.47 MB/sec execute 533 sec latency 20.709 ms 2024-04-23T18:30:20.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 733944 5.47 MB/sec execute 534 sec latency 18.243 ms 2024-04-23T18:30:20.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:20 smithi050 ceph-mon[29187]: pgmap v372: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-23T18:30:20.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:20 smithi177 ceph-mon[32308]: pgmap v372: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-23T18:30:20.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:20 smithi188 ceph-mon[32694]: pgmap v372: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-23T18:30:21.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 735595 5.48 MB/sec execute 535 sec latency 10.827 ms 2024-04-23T18:30:22.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 737479 5.49 MB/sec execute 536 sec latency 18.495 ms 2024-04-23T18:30:22.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:22 smithi050 ceph-mon[29187]: pgmap v373: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-23T18:30:22.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:22 smithi177 ceph-mon[32308]: pgmap v373: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-23T18:30:22.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:22 smithi188 ceph-mon[32694]: pgmap v373: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-23T18:30:23.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 739062 5.50 MB/sec execute 537 sec latency 15.415 ms 2024-04-23T18:30:23.395 INFO:tasks.quiescer.fs.[cephfs]:Total duration (47.9) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:30:23.395 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.95 2024-04-23T18:30:23.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:30:23.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.522+0000 7fedc4ecf640 1 -- 172.21.15.50:0/1055178713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc0101a50 msgr2=0x7fedc0105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.522+0000 7fedc4ecf640 1 --2- 172.21.15.50:0/1055178713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc0101a50 0x7fedc0105a90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009860 tx=0x7fedb402a340 comp rx=0 tx=0).stop 2024-04-23T18:30:23.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.523+0000 7fedc4ecf640 1 -- 172.21.15.50:0/1055178713 shutdown_connections 2024-04-23T18:30:23.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.523+0000 7fedc4ecf640 1 --2- 172.21.15.50:0/1055178713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc0106170 0x7fedc010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.523+0000 7fedc4ecf640 1 --2- 172.21.15.50:0/1055178713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc0101a50 0x7fedc0105a90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.523+0000 7fedc4ecf640 1 --2- 172.21.15.50:0/1055178713 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedc01010a0 0x7fedc0101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.523+0000 7fedc4ecf640 1 -- 172.21.15.50:0/1055178713 >> 172.21.15.50:0/1055178713 conn(0x7fedc0077f50 msgr2=0x7fedc0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:23.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.523+0000 7fedc4ecf640 1 -- 172.21.15.50:0/1055178713 shutdown_connections 2024-04-23T18:30:23.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.523+0000 7fedc4ecf640 1 -- 172.21.15.50:0/1055178713 wait complete. 2024-04-23T18:30:23.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.523+0000 7fedc4ecf640 1 Processor -- start 2024-04-23T18:30:23.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.523+0000 7fedc4ecf640 1 -- start start 2024-04-23T18:30:23.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedc4ecf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc01010a0 0x7fedc0102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedc4ecf640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedc0106170 0x7fedc0103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedc4ecf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc01035b0 0x7fedc01a6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedc4ecf640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fedc01110b0 con 0x7fedc01035b0 2024-04-23T18:30:23.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedc4ecf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fedc0110f30 con 0x7fedc01010a0 2024-04-23T18:30:23.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedc4ecf640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fedc0111230 con 0x7fedc0106170 2024-04-23T18:30:23.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedbed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc01035b0 0x7fedc01a6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedbe59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc01010a0 0x7fedc0102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedbdd9b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedc0106170 0x7fedc0103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedbed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc01035b0 0x7fedc01a6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58558/0 (socket says 172.21.15.50:58558) 2024-04-23T18:30:23.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedbed9d640 1 -- 172.21.15.50:0/3932538597 learned_addr learned my addr 172.21.15.50:0/3932538597 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:23.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedbed9d640 1 -- 172.21.15.50:0/3932538597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedc0106170 msgr2=0x7fedc0103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedbed9d640 1 --2- 172.21.15.50:0/3932538597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedc0106170 0x7fedc0103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedbed9d640 1 -- 172.21.15.50:0/3932538597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc01010a0 msgr2=0x7fedc0102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedbed9d640 1 --2- 172.21.15.50:0/3932538597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc01010a0 0x7fedc0102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedbed9d640 1 -- 172.21.15.50:0/3932538597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc01a7020 con 0x7fedc01035b0 2024-04-23T18:30:23.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedbed9d640 1 --2- 172.21.15.50:0/3932538597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc01035b0 0x7fedc01a6920 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fedb800eb10 tx=0x7fedb800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.524+0000 7fedaf7fe640 1 -- 172.21.15.50:0/3932538597 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb800ce20 con 0x7fedc01035b0 2024-04-23T18:30:23.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.525+0000 7fedaf7fe640 1 -- 172.21.15.50:0/3932538597 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedb8005020 con 0x7fedc01035b0 2024-04-23T18:30:23.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.525+0000 7fedc4ecf640 1 -- 172.21.15.50:0/3932538597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedc01a7310 con 0x7fedc01035b0 2024-04-23T18:30:23.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.525+0000 7fedaf7fe640 1 -- 172.21.15.50:0/3932538597 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8010690 con 0x7fedc01035b0 2024-04-23T18:30:23.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.525+0000 7fedc4ecf640 1 -- 172.21.15.50:0/3932538597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedc01a77e0 con 0x7fedc01035b0 2024-04-23T18:30:23.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.526+0000 7fedc4ecf640 1 -- 172.21.15.50:0/3932538597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedc0068d70 con 0x7fedc01035b0 2024-04-23T18:30:23.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.527+0000 7fedaf7fe640 1 -- 172.21.15.50:0/3932538597 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fedb8005510 con 0x7fedc01035b0 2024-04-23T18:30:23.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.528+0000 7fedaf7fe640 1 --2- 172.21.15.50:0/3932538597 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feda80a3d50 0x7feda80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.528+0000 7fedbe59c640 1 --2- 172.21.15.50:0/3932538597 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feda80a3d50 0x7feda80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.528+0000 7fedaf7fe640 1 -- 172.21.15.50:0/3932538597 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7fedb80d0f60 con 0x7fedc01035b0 2024-04-23T18:30:23.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.528+0000 7fedbe59c640 1 --2- 172.21.15.50:0/3932538597 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feda80a3d50 0x7feda80a6210 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fedb0005fd0 tx=0x7fedb0009260 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.531+0000 7fedaf7fe640 1 -- 172.21.15.50:0/3932538597 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fedb8053ff0 con 0x7fedc01035b0 2024-04-23T18:30:23.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.778+0000 7fedc4ecf640 1 -- 172.21.15.50:0/3932538597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedc010a3b0 con 0x7fedc01035b0 2024-04-23T18:30:23.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.779+0000 7fedaf7fe640 1 -- 172.21.15.50:0/3932538597 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+11884 (secure 0 0 0) 0x7fedb809d590 con 0x7fedc01035b0 2024-04-23T18:30:23.772 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:30:23.772 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":46,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:29:50.480612+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/4033440314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4033440314},{"type":"v1","addr":"172.21.15.177:6837","nonce":4033440314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:30:23.772 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 46 2024-04-23T18:30:23.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.781+0000 7fedc4ecf640 1 -- 172.21.15.50:0/3932538597 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feda80a3d50 msgr2=0x7feda80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.781+0000 7fedc4ecf640 1 --2- 172.21.15.50:0/3932538597 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feda80a3d50 0x7feda80a6210 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fedb0005fd0 tx=0x7fedb0009260 comp rx=0 tx=0).stop 2024-04-23T18:30:23.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.781+0000 7fedc4ecf640 1 -- 172.21.15.50:0/3932538597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc01035b0 msgr2=0x7fedc01a6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.781+0000 7fedc4ecf640 1 --2- 172.21.15.50:0/3932538597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc01035b0 0x7fedc01a6920 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fedb800eb10 tx=0x7fedb800ef10 comp rx=0 tx=0).stop 2024-04-23T18:30:23.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.781+0000 7fedc4ecf640 1 -- 172.21.15.50:0/3932538597 shutdown_connections 2024-04-23T18:30:23.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.781+0000 7fedc4ecf640 1 --2- 172.21.15.50:0/3932538597 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7feda80a3d50 0x7feda80a6210 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.781+0000 7fedc4ecf640 1 --2- 172.21.15.50:0/3932538597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc01035b0 0x7fedc01a6920 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.781+0000 7fedc4ecf640 1 --2- 172.21.15.50:0/3932538597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedc0106170 0x7fedc0103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.781+0000 7fedc4ecf640 1 --2- 172.21.15.50:0/3932538597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fedc01010a0 0x7fedc0102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.782+0000 7fedc4ecf640 1 -- 172.21.15.50:0/3932538597 >> 172.21.15.50:0/3932538597 conn(0x7fedc0077f50 msgr2=0x7fedc01001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:23.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.782+0000 7fedc4ecf640 1 -- 172.21.15.50:0/3932538597 shutdown_connections 2024-04-23T18:30:23.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.782+0000 7fedc4ecf640 1 -- 172.21.15.50:0/3932538597 wait complete. 2024-04-23T18:30:23.786 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 83.95 --await-for 42.0' 2024-04-23T18:30:23.786 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 83.95 --await-for 42.0 2024-04-23T18:30:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.911+0000 7f43e1ea3640 1 -- 172.21.15.50:0/1896712321 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43dc107280 msgr2=0x7f43dc10ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.911+0000 7f43e1ea3640 1 --2- 172.21.15.50:0/1896712321 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43dc107280 0x7f43dc10ee50 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009ca0 tx=0x7f43d402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.911+0000 7f43e1ea3640 1 -- 172.21.15.50:0/1896712321 shutdown_connections 2024-04-23T18:30:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.911+0000 7f43e1ea3640 1 --2- 172.21.15.50:0/1896712321 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43dc107280 0x7f43dc10ee50 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.911+0000 7f43e1ea3640 1 --2- 172.21.15.50:0/1896712321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43dc102bf0 0x7f43dc106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.911+0000 7f43e1ea3640 1 --2- 172.21.15.50:0/1896712321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43dc102240 0x7f43dc102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.911+0000 7f43e1ea3640 1 -- 172.21.15.50:0/1896712321 >> 172.21.15.50:0/1896712321 conn(0x7f43dc0790e0 msgr2=0x7f43dc0794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.911+0000 7f43e1ea3640 1 -- 172.21.15.50:0/1896712321 shutdown_connections 2024-04-23T18:30:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.911+0000 7f43e1ea3640 1 -- 172.21.15.50:0/1896712321 wait complete. 2024-04-23T18:30:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43e1ea3640 1 Processor -- start 2024-04-23T18:30:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43e1ea3640 1 -- start start 2024-04-23T18:30:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43e1ea3640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43dc102240 0x7f43dc198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43e1ea3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43dc107280 0x7f43dc198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43e1ea3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43dc19cf40 0x7f43dc19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43e1ea3640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1121e0 con 0x7f43dc107280 2024-04-23T18:30:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43e1ea3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc112060 con 0x7f43dc19cf40 2024-04-23T18:30:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43e1ea3640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc112360 con 0x7f43dc102240 2024-04-23T18:30:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43d3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43dc107280 0x7f43dc198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43d3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43dc107280 0x7f43dc198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58576/0 (socket says 172.21.15.50:58576) 2024-04-23T18:30:23.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43d3fff640 1 -- 172.21.15.50:0/3955707358 learned_addr learned my addr 172.21.15.50:0/3955707358 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:23.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43dbfff640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43dc19cf40 0x7f43dc19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43db7fe640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43dc102240 0x7f43dc198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43d3fff640 1 -- 172.21.15.50:0/3955707358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43dc102240 msgr2=0x7f43dc198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43d3fff640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43dc102240 0x7f43dc198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43d3fff640 1 -- 172.21.15.50:0/3955707358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43dc19cf40 msgr2=0x7f43dc19d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43d3fff640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43dc19cf40 0x7f43dc19d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.912+0000 7f43d3fff640 1 -- 172.21.15.50:0/3955707358 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43dc19da30 con 0x7f43dc107280 2024-04-23T18:30:23.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.913+0000 7f43d3fff640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43dc107280 0x7f43dc198bc0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f43c400d950 tx=0x7f43c400dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.913+0000 7f43d97fa640 1 -- 172.21.15.50:0/3955707358 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c4017070 con 0x7f43dc107280 2024-04-23T18:30:23.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.913+0000 7f43e1ea3640 1 -- 172.21.15.50:0/3955707358 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43dc19dd20 con 0x7f43dc107280 2024-04-23T18:30:23.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.913+0000 7f43e1ea3640 1 -- 172.21.15.50:0/3955707358 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43dc1a5560 con 0x7f43dc107280 2024-04-23T18:30:23.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.913+0000 7f43d97fa640 1 -- 172.21.15.50:0/3955707358 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43c4009c30 con 0x7f43dc107280 2024-04-23T18:30:23.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.913+0000 7f43d97fa640 1 -- 172.21.15.50:0/3955707358 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c40047a0 con 0x7f43dc107280 2024-04-23T18:30:23.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.915+0000 7f43d97fa640 1 -- 172.21.15.50:0/3955707358 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f43c4004020 con 0x7f43dc107280 2024-04-23T18:30:23.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.915+0000 7f43e4bb6740 1 Processor -- start 2024-04-23T18:30:23.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.915+0000 7f43e4bb6740 1 -- start start 2024-04-23T18:30:23.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.915+0000 7f43e4bb6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f73dd0 0x556cb2f741b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.915+0000 7f43e4bb6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f74780 0x556cb2f82f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.915+0000 7f43e4bb6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f834a0 0x556cb2f85890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.915+0000 7f43e4bb6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556cb2ed9370 con 0x556cb2f834a0 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.915+0000 7f43e4bb6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556cb2e762e0 con 0x556cb2f73dd0 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.915+0000 7f43e4bb6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556cb2ed5330 con 0x556cb2f74780 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f74780 0x556cb2f82f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43d3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f834a0 0x556cb2f85890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f74780 0x556cb2f82f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:60484/0 (socket says 172.21.15.50:60484) 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43db7fe640 1 -- 172.21.15.50:0/3335370818 learned_addr learned my addr 172.21.15.50:0/3335370818 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43dbfff640 1 --2- 172.21.15.50:0/3335370818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f73dd0 0x556cb2f741b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43d97fa640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f43c80a3d20 0x7f43c80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43db7fe640 1 -- 172.21.15.50:0/3335370818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f73dd0 msgr2=0x556cb2f741b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43db7fe640 1 --2- 172.21.15.50:0/3335370818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f73dd0 0x556cb2f741b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43db7fe640 1 -- 172.21.15.50:0/3335370818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f834a0 msgr2=0x556cb2f85890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43db7fe640 1 --2- 172.21.15.50:0/3335370818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f834a0 0x556cb2f85890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43db7fe640 1 -- 172.21.15.50:0/3335370818 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cb2cb7d80 con 0x556cb2f74780 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43d3fff640 1 --2- 172.21.15.50:0/3335370818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f834a0 0x556cb2f85890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43dbfff640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f43c80a3d20 0x7f43c80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43d97fa640 1 -- 172.21.15.50:0/3955707358 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f43c40d0340 con 0x7f43dc107280 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.916+0000 7f43db7fe640 1 --2- 172.21.15.50:0/3335370818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f74780 0x556cb2f82f60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00ad30 tx=0x7f43cc00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=660220c71364ff67 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43d1ffb640 1 -- 172.21.15.50:0/3335370818 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc010840 con 0x556cb2f74780 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43d1ffb640 1 -- 172.21.15.50:0/3335370818 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43cc014070 con 0x556cb2f74780 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43d1ffb640 1 -- 172.21.15.50:0/3335370818 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc018a80 con 0x556cb2f74780 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43e4bb6740 1 -- 172.21.15.50:0/3335370818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f74780 msgr2=0x556cb2f82f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43dbfff640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f43c80a3d20 0x7f43c80a61e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f43d4006010 tx=0x7f43d40095c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/3335370818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f74780 0x556cb2f82f60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00ad30 tx=0x7f43cc00f790 comp rx=0 tx=0).stop 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43e4bb6740 1 -- 172.21.15.50:0/3335370818 shutdown_connections 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/3335370818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f834a0 0x556cb2f85890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/3335370818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f74780 0x556cb2f82f60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/3335370818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f73dd0 0x556cb2f741b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43e4bb6740 1 -- 172.21.15.50:0/3335370818 >> 172.21.15.50:0/3335370818 conn(0x556cb2f703c0 msgr2=0x556cb2f707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43e4bb6740 1 -- 172.21.15.50:0/3335370818 shutdown_connections 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.917+0000 7f43e4bb6740 1 -- 172.21.15.50:0/3335370818 wait complete. 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43e4bb6740 1 -- start start 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43e4bb6740 1 Processor -- start 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43e4bb6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f73dd0 0x556cb2f803a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43e4bb6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f74780 0x556cb2f808e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43e4bb6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f834a0 0x556cb2f80e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43e4bb6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556cb2f86260 con 0x556cb2f74780 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43e4bb6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556cb2e73fb0 con 0x556cb2f73dd0 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43e4bb6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556cb2f863e0 con 0x556cb2f834a0 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43e4bb6740 10 client.0 _ll_register_callbacks cb 0x556cb2e739a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43d3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f74780 0x556cb2f808e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43d3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f74780 0x556cb2f808e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58598/0 (socket says 172.21.15.50:58598) 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f73dd0 0x556cb2f803a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43dbfff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f834a0 0x556cb2f80e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.918+0000 7f43d3fff640 1 -- 172.21.15.50:0/546370603 learned_addr learned my addr 172.21.15.50:0/546370603 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43d3fff640 1 -- 172.21.15.50:0/546370603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f834a0 msgr2=0x556cb2f80e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43d3fff640 1 --2- 172.21.15.50:0/546370603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f834a0 0x556cb2f80e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43d3fff640 1 -- 172.21.15.50:0/546370603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f73dd0 msgr2=0x556cb2f803a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43d3fff640 1 --2- 172.21.15.50:0/546370603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f73dd0 0x556cb2f803a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43d3fff640 1 -- 172.21.15.50:0/546370603 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cb2d18510 con 0x556cb2f74780 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43db7fe640 1 --2- 172.21.15.50:0/546370603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f73dd0 0x556cb2f803a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43d3fff640 1 --2- 172.21.15.50:0/546370603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f74780 0x556cb2f808e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009610 tx=0x7f43c400b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b471b9b45aff1ad server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43baffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43baffd640 1 -- 172.21.15.50:0/546370603 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c409bc30 con 0x556cb2f74780 2024-04-23T18:30:23.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43baffd640 1 -- 172.21.15.50:0/546370603 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f43c40cdd00 con 0x556cb2f74780 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43e4bb6740 1 -- 172.21.15.50:0/546370603 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43cc014070 con 0x556cb2f74780 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43baffd640 1 -- 172.21.15.50:0/546370603 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f43c400be60 con 0x556cb2f74780 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43e4bb6740 10 client.16944 fetch_fsmap learned FSMap version 46 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43e4bb6740 1 -- 172.21.15.50:0/546370603 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x556cb2bab670 con 0x556cb2f74780 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.919+0000 7f43baffd640 1 -- 172.21.15.50:0/546370603 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43cc014070 con 0x556cb2f74780 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.920+0000 7f43baffd640 1 -- 172.21.15.50:0/546370603 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 46) ==== 8362+0+0 (secure 0 0 0) 0x7f43c4098940 con 0x556cb2f74780 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.920+0000 7f43e4bb6740 10 client.16944 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.920+0000 7f43e4bb6740 10 client.16944 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.920+0000 7f43e4bb6740 20 client.16944 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.920+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/546370603 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cb2f936e0 0x556cb2f95b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.920+0000 7f43e4bb6740 4 client.16944 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.920+0000 7f43e4bb6740 1 -- 172.21.15.50:0/546370603 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cb2ee6d90 con 0x556cb2f936e0 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.920+0000 7f43db7fe640 1 --2- 172.21.15.50:0/546370603 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cb2f936e0 0x556cb2f95b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.922+0000 7f43db7fe640 1 --2- 172.21.15.50:0/546370603 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cb2f936e0 0x556cb2f95b70 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66a42209a8de64c6 server_cookie=1c839d7c97a2dc09 in_seq=0 out_seq=0 2024-04-23T18:30:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.922+0000 7f43baffd640 10 client.16944 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43baffd640 1 -- 172.21.15.50:0/546370603 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556cb2ee6d90 con 0x556cb2f936e0 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43baffd640 10 client.16944 handle_command_reply: tid=0 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43e4bb6740 1 client.16944 shutdown 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43e4bb6740 1 -- 172.21.15.50:0/546370603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f74780 msgr2=0x556cb2f808e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/546370603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f74780 0x556cb2f808e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009610 tx=0x7f43c400b6c0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43e4bb6740 1 -- 172.21.15.50:0/546370603 shutdown_connections 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/546370603 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cb2f936e0 0x556cb2f95b70 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/546370603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f834a0 0x556cb2f80e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43baffd640 0 client.16944 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/546370603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f74780 0x556cb2f808e0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/546370603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f73dd0 0x556cb2f803a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43e4bb6740 1 -- 172.21.15.50:0/546370603 >> 172.21.15.50:0/546370603 conn(0x556cb2f703c0 msgr2=0x556cb2f76230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:23.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43e4bb6740 1 -- 172.21.15.50:0/546370603 shutdown_connections 2024-04-23T18:30:23.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43e4bb6740 1 -- 172.21.15.50:0/546370603 wait complete. 2024-04-23T18:30:23.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.923+0000 7f43e4bb6740 20 client.16944 trim_cache size 0 max 16384 2024-04-23T18:30:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.941+0000 7f43e4bb6740 1 Processor -- start 2024-04-23T18:30:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.941+0000 7f43e4bb6740 1 -- start start 2024-04-23T18:30:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.941+0000 7f43e4bb6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 0x556cb2f8f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.941+0000 7f43e4bb6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f74780 0x556cb2f6fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.941+0000 7f43e4bb6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f834a0 0x556cb2f9b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.941+0000 7f43e4bb6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1ae3f0 con 0x556cb2f73dd0 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.941+0000 7f43e4bb6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1abc00 con 0x556cb2f834a0 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.941+0000 7f43e4bb6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1a9a70 con 0x556cb2f74780 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.942+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 0x556cb2f8f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.942+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 0x556cb2f8f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58614/0 (socket says 172.21.15.50:58614) 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.942+0000 7f43db7fe640 1 -- 172.21.15.50:0/2016991975 learned_addr learned my addr 172.21.15.50:0/2016991975 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.942+0000 7f43db7fe640 1 -- 172.21.15.50:0/2016991975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f74780 msgr2=0x556cb2f6fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.942+0000 7f43db7fe640 1 --2- 172.21.15.50:0/2016991975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f74780 0x556cb2f6fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.942+0000 7f43db7fe640 1 -- 172.21.15.50:0/2016991975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f834a0 msgr2=0x556cb2f9b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.942+0000 7f43db7fe640 1 --2- 172.21.15.50:0/2016991975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f834a0 0x556cb2f9b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.942+0000 7f43db7fe640 1 -- 172.21.15.50:0/2016991975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cb2f74c50 con 0x556cb2f73dd0 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.942+0000 7f43db7fe640 1 --2- 172.21.15.50:0/2016991975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 0x556cb2f8f160 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f43cc018630 tx=0x7f43cc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7fa37ef18f63194 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.942+0000 7f43d17fa640 1 -- 172.21.15.50:0/2016991975 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc02f070 con 0x556cb2f73dd0 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.942+0000 7f43d17fa640 1 -- 172.21.15.50:0/2016991975 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43cc014050 con 0x556cb2f73dd0 2024-04-23T18:30:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.942+0000 7f43d17fa640 1 -- 172.21.15.50:0/2016991975 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc02a600 con 0x556cb2f73dd0 2024-04-23T18:30:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.943+0000 7f43e4bb6740 1 -- 172.21.15.50:0/2016991975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 msgr2=0x556cb2f8f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.943+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/2016991975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 0x556cb2f8f160 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f43cc018630 tx=0x7f43cc00f790 comp rx=0 tx=0).stop 2024-04-23T18:30:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.943+0000 7f43e4bb6740 1 -- 172.21.15.50:0/2016991975 shutdown_connections 2024-04-23T18:30:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.943+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/2016991975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f834a0 0x556cb2f9b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.943+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/2016991975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f74780 0x556cb2f6fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.943+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/2016991975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 0x556cb2f8f160 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.943+0000 7f43e4bb6740 1 -- 172.21.15.50:0/2016991975 >> 172.21.15.50:0/2016991975 conn(0x556cb2f703c0 msgr2=0x556cb2f81bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.943+0000 7f43e4bb6740 1 -- 172.21.15.50:0/2016991975 shutdown_connections 2024-04-23T18:30:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.943+0000 7f43e4bb6740 1 -- 172.21.15.50:0/2016991975 wait complete. 2024-04-23T18:30:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43e4bb6740 1 -- start start 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43e4bb6740 1 Processor -- start 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43e4bb6740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 0x556cb2f87790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43e4bb6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f74780 0x556cb2f87cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43e4bb6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f834a0 0x556cb2ff8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43e4bb6740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1aa0c0 con 0x556cb2f73dd0 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43e4bb6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1ab530 con 0x556cb2f74780 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43e4bb6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1aa790 con 0x556cb2f834a0 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43e4bb6740 10 client.0 _ll_register_callbacks cb 0x7f43c4005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 0x556cb2f87790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 0x556cb2f87790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58630/0 (socket says 172.21.15.50:58630) 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43db7fe640 1 -- 172.21.15.50:0/1290722445 learned_addr learned my addr 172.21.15.50:0/1290722445 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.944+0000 7f43dbfff640 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f834a0 0x556cb2ff8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43d3fff640 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f74780 0x556cb2f87cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43db7fe640 1 -- 172.21.15.50:0/1290722445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f834a0 msgr2=0x556cb2ff8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43db7fe640 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f834a0 0x556cb2ff8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43db7fe640 1 -- 172.21.15.50:0/1290722445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f74780 msgr2=0x556cb2f87cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43db7fe640 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f74780 0x556cb2f87cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43db7fe640 1 -- 172.21.15.50:0/1290722445 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cb2ff91f0 con 0x556cb2f73dd0 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43db7fe640 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 0x556cb2f87790 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00fc90 tx=0x7f43cc005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179a5eebd1328e15 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43baffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43baffd640 1 -- 172.21.15.50:0/1290722445 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc02f040 con 0x556cb2f73dd0 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43baffd640 1 -- 172.21.15.50:0/1290722445 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f43cc005550 con 0x556cb2f73dd0 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43e4bb6740 1 -- 172.21.15.50:0/1290722445 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43cc014050 con 0x556cb2f73dd0 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43baffd640 1 -- 172.21.15.50:0/1290722445 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f43cc043050 con 0x556cb2f73dd0 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43e4bb6740 10 client.16950 fetch_fsmap learned FSMap version 46 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43e4bb6740 1 -- 172.21.15.50:0/1290722445 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x556cb2ffa4a0 con 0x556cb2f73dd0 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.945+0000 7f43baffd640 1 -- 172.21.15.50:0/1290722445 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43cc014050 con 0x556cb2f73dd0 2024-04-23T18:30:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.946+0000 7f43baffd640 1 -- 172.21.15.50:0/1290722445 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 46) ==== 8362+0+0 (secure 0 0 0) 0x7f43cc0108e0 con 0x556cb2f73dd0 2024-04-23T18:30:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.946+0000 7f43e4bb6740 10 client.16950 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:30:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.946+0000 7f43e4bb6740 10 client.16950 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:30:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.946+0000 7f43e4bb6740 20 client.16950 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.946+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cb2ffaa30 0x556cb2ffce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.946+0000 7f43e4bb6740 4 client.16950 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.95, "await_for": 42.0}] 2024-04-23T18:30:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.946+0000 7f43e4bb6740 1 -- 172.21.15.50:0/1290722445 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.95, "await_for": 42.0}) -- 0x556cb2ee0c00 con 0x556cb2ffaa30 2024-04-23T18:30:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.946+0000 7f43d3fff640 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cb2ffaa30 0x556cb2ffce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.946+0000 7f43baffd640 10 client.16950 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:23.946+0000 7f43d3fff640 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cb2ffaa30 0x556cb2ffce50 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d88d6366f2d2562 server_cookie=d3a2f2258f0698ea in_seq=0 out_seq=0 2024-04-23T18:30:24.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.016+0000 7f43baffd640 1 -- 172.21.15.50:0/1290722445 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x556cb2ee0c00 con 0x556cb2ffaa30 2024-04-23T18:30:24.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.016+0000 7f43baffd640 10 client.16950 handle_command_reply: tid=0 2024-04-23T18:30:24.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.016+0000 7f43e4bb6740 1 client.16950 shutdown 2024-04-23T18:30:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e4bb6740 1 -- 172.21.15.50:0/1290722445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 msgr2=0x556cb2f87790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 0x556cb2f87790 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00fc90 tx=0x7f43cc005900 comp rx=0 tx=0).stop 2024-04-23T18:30:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e4bb6740 1 -- 172.21.15.50:0/1290722445 shutdown_connections 2024-04-23T18:30:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556cb2ffaa30 0x556cb2ffce50 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556cb2f834a0 0x556cb2ff8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cb2f74780 0x556cb2f87cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43baffd640 0 client.16950 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e4bb6740 1 --2- 172.21.15.50:0/1290722445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556cb2f73dd0 0x556cb2f87790 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e4bb6740 1 -- 172.21.15.50:0/1290722445 >> 172.21.15.50:0/1290722445 conn(0x556cb2f703c0 msgr2=0x556cb2f75980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e4bb6740 1 -- 172.21.15.50:0/1290722445 shutdown_connections 2024-04-23T18:30:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e4bb6740 1 -- 172.21.15.50:0/1290722445 wait complete. 2024-04-23T18:30:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e4bb6740 20 client.16950 trim_cache size 0 max 16384 2024-04-23T18:30:24.012 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:30:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 46, 2024-04-23T18:30:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:30:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 38, 2024-04-23T18:30:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:30:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "a5988bee": { 2024-04-23T18:30:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 38, 2024-04-23T18:30:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:30:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:30:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 84.0, 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:24.014 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:30:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e1ea3640 1 -- 172.21.15.50:0/3955707358 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f43c80a3d20 msgr2=0x7f43c80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e1ea3640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f43c80a3d20 0x7f43c80a61e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f43d4006010 tx=0x7f43d40095c0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e1ea3640 1 -- 172.21.15.50:0/3955707358 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43dc107280 msgr2=0x7f43dc198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.017+0000 7f43e1ea3640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43dc107280 0x7f43dc198bc0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f43c400d950 tx=0x7f43c400dd50 comp rx=0 tx=0).stop 2024-04-23T18:30:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.018+0000 7f43e1ea3640 1 -- 172.21.15.50:0/3955707358 shutdown_connections 2024-04-23T18:30:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.018+0000 7f43e1ea3640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f43c80a3d20 0x7f43c80a61e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.018+0000 7f43e1ea3640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43dc19cf40 0x7f43dc19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.018+0000 7f43e1ea3640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43dc107280 0x7f43dc198bc0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.018+0000 7f43e1ea3640 1 --2- 172.21.15.50:0/3955707358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f43dc102240 0x7f43dc198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.018+0000 7f43e1ea3640 1 -- 172.21.15.50:0/3955707358 >> 172.21.15.50:0/3955707358 conn(0x7f43dc0790e0 msgr2=0x7f43dc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.018+0000 7f43e1ea3640 1 -- 172.21.15.50:0/3955707358 shutdown_connections 2024-04-23T18:30:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:24.018+0000 7f43e1ea3640 1 -- 172.21.15.50:0/3955707358 wait complete. 2024-04-23T18:30:24.021 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a5988bee', quiesce took 0.6 seconds. Will release after: 23.95 2024-04-23T18:30:24.233 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.50 MB/sec execute 538 sec latency 227.037 ms 2024-04-23T18:30:24.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:24 smithi050 ceph-mon[29187]: pgmap v374: 97 pgs: 97 active+clean; 237 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.2 MiB/s wr, 279 op/s 2024-04-23T18:30:24.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:24 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3932538597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:24.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:24 smithi177 ceph-mon[32308]: pgmap v374: 97 pgs: 97 active+clean; 237 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.2 MiB/s wr, 279 op/s 2024-04-23T18:30:24.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:24 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3932538597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:24.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:24 smithi188 ceph-mon[32694]: pgmap v374: 97 pgs: 97 active+clean; 237 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.2 MiB/s wr, 279 op/s 2024-04-23T18:30:24.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:24 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3932538597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:25.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.49 MB/sec execute 539 sec latency 1227.111 ms 2024-04-23T18:30:26.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.48 MB/sec execute 540 sec latency 2227.162 ms 2024-04-23T18:30:26.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:26 smithi050 ceph-mon[29187]: pgmap v375: 97 pgs: 97 active+clean; 237 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 192 op/s 2024-04-23T18:30:26.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:26 smithi177 ceph-mon[32308]: pgmap v375: 97 pgs: 97 active+clean; 237 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 192 op/s 2024-04-23T18:30:26.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:26 smithi188 ceph-mon[32694]: pgmap v375: 97 pgs: 97 active+clean; 237 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 192 op/s 2024-04-23T18:30:27.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.47 MB/sec execute 541 sec latency 3227.273 ms 2024-04-23T18:30:28.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.46 MB/sec execute 542 sec latency 4227.390 ms 2024-04-23T18:30:28.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:28 smithi050 ceph-mon[29187]: pgmap v376: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 251 op/s 2024-04-23T18:30:28.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:28 smithi177 ceph-mon[32308]: pgmap v376: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 251 op/s 2024-04-23T18:30:28.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:28 smithi188 ceph-mon[32694]: pgmap v376: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 251 op/s 2024-04-23T18:30:29.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.45 MB/sec execute 543 sec latency 5227.512 ms 2024-04-23T18:30:30.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.44 MB/sec execute 544 sec latency 6227.638 ms 2024-04-23T18:30:30.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:30 smithi050 ceph-mon[29187]: pgmap v377: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-04-23T18:30:30.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:30 smithi177 ceph-mon[32308]: pgmap v377: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-04-23T18:30:30.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:30 smithi188 ceph-mon[32694]: pgmap v377: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-04-23T18:30:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.631+0000 7f3873e7c640 1 -- 172.21.15.50:0/427306306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f386c107290 msgr2=0x7f386c10ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.631+0000 7f3873e7c640 1 --2- 172.21.15.50:0/427306306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f386c107290 0x7f386c10ee60 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f3864009ca0 tx=0x7f386402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.631+0000 7f3873e7c640 1 -- 172.21.15.50:0/427306306 shutdown_connections 2024-04-23T18:30:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.631+0000 7f3873e7c640 1 --2- 172.21.15.50:0/427306306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f386c107290 0x7f386c10ee60 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.631+0000 7f3873e7c640 1 --2- 172.21.15.50:0/427306306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f386c102b70 0x7f386c106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.631+0000 7f3873e7c640 1 --2- 172.21.15.50:0/427306306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f386c1021c0 0x7f386c1025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.631+0000 7f3873e7c640 1 -- 172.21.15.50:0/427306306 >> 172.21.15.50:0/427306306 conn(0x7f386c0fda20 msgr2=0x7f386c0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.631+0000 7f3873e7c640 1 -- 172.21.15.50:0/427306306 shutdown_connections 2024-04-23T18:30:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.631+0000 7f3873e7c640 1 -- 172.21.15.50:0/427306306 wait complete. 2024-04-23T18:30:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.632+0000 7f3873e7c640 1 Processor -- start 2024-04-23T18:30:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.632+0000 7f3873e7c640 1 -- start start 2024-04-23T18:30:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.632+0000 7f3873e7c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f386c1021c0 0x7f386c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.632+0000 7f3873e7c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f386c107290 0x7f386c198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.632+0000 7f3873e7c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f386c19cf10 0x7f386c19d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.632+0000 7f3873e7c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1121e0 con 0x7f386c1021c0 2024-04-23T18:30:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.632+0000 7f3873e7c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f386c112060 con 0x7f386c19cf10 2024-04-23T18:30:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.632+0000 7f3873e7c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f386c112360 con 0x7f386c107290 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.632+0000 7f3871c18640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f386c1021c0 0x7f386c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.632+0000 7f3871c18640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f386c1021c0 0x7f386c1986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58650/0 (socket says 172.21.15.50:58650) 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.632+0000 7f3871c18640 1 -- 172.21.15.50:0/819294855 learned_addr learned my addr 172.21.15.50:0/819294855 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3871417640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f386c107290 0x7f386c198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3872419640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f386c19cf10 0x7f386c19d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3871c18640 1 -- 172.21.15.50:0/819294855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f386c107290 msgr2=0x7f386c198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3871c18640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f386c107290 0x7f386c198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3871c18640 1 -- 172.21.15.50:0/819294855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f386c19cf10 msgr2=0x7f386c19d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3871c18640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f386c19cf10 0x7f386c19d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3871c18640 1 -- 172.21.15.50:0/819294855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f386c19da00 con 0x7f386c1021c0 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3871417640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f386c107290 0x7f386c198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3872419640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f386c19cf10 0x7f386c19d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3871c18640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f386c1021c0 0x7f386c1986a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f386800d950 tx=0x7f386800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3862ffd640 1 -- 172.21.15.50:0/819294855 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3868017070 con 0x7f386c1021c0 2024-04-23T18:30:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3862ffd640 1 -- 172.21.15.50:0/819294855 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3868009c30 con 0x7f386c1021c0 2024-04-23T18:30:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3873e7c640 1 -- 172.21.15.50:0/819294855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f386c19dcf0 con 0x7f386c1021c0 2024-04-23T18:30:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3862ffd640 1 -- 172.21.15.50:0/819294855 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386800f620 con 0x7f386c1021c0 2024-04-23T18:30:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.633+0000 7f3873e7c640 1 -- 172.21.15.50:0/819294855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f386c1a5530 con 0x7f386c1021c0 2024-04-23T18:30:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.635+0000 7f3862ffd640 1 -- 172.21.15.50:0/819294855 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3868010460 con 0x7f386c1021c0 2024-04-23T18:30:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.636+0000 7f3862ffd640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f38580a3e40 0x7f38580a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.636+0000 7f3871417640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f38580a3e40 0x7f38580a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.636+0000 7f3862ffd640 1 -- 172.21.15.50:0/819294855 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f38680d04e0 con 0x7f386c1021c0 2024-04-23T18:30:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.636+0000 7f3876b90740 1 Processor -- start 2024-04-23T18:30:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3876b90740 1 -- start start 2024-04-23T18:30:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3871417640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f38580a3e40 0x7f38580a6300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f385c005fd0 tx=0x7f385c005850 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3876b90740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6a620 0x55cfeea6aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3876b90740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea6af40 0x55cfeea752c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3876b90740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea75800 0x55cfeea77bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3876b90740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cfee9cf970 con 0x55cfeea6a620 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3876b90740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cfee96c4a0 con 0x55cfeea75800 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3876b90740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55cfee9cb930 con 0x55cfeea6af40 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3871c18640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6a620 0x55cfeea6aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3871c18640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6a620 0x55cfeea6aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58666/0 (socket says 172.21.15.50:58666) 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3871c18640 1 -- 172.21.15.50:0/3995601583 learned_addr learned my addr 172.21.15.50:0/3995601583 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3872419640 1 --2- 172.21.15.50:0/3995601583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea75800 0x55cfeea77bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3871417640 1 --2- 172.21.15.50:0/3995601583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea6af40 0x55cfeea752c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.637+0000 7f3871c18640 1 -- 172.21.15.50:0/3995601583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea6af40 msgr2=0x55cfeea752c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.638+0000 7f3871c18640 1 --2- 172.21.15.50:0/3995601583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea6af40 0x55cfeea752c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.638+0000 7f3871c18640 1 -- 172.21.15.50:0/3995601583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea75800 msgr2=0x55cfeea77bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.638+0000 7f3871c18640 1 --2- 172.21.15.50:0/3995601583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea75800 0x55cfeea77bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.638+0000 7f3871c18640 1 -- 172.21.15.50:0/3995601583 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfee6e2210 con 0x55cfeea6a620 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.638+0000 7f3871c18640 1 --2- 172.21.15.50:0/3995601583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6a620 0x55cfeea6aa00 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f38680a4b50 tx=0x7f386809c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e20097c1a1edaa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.638+0000 7f3857fff640 1 -- 172.21.15.50:0/3995601583 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3868020070 con 0x55cfeea6a620 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.638+0000 7f3857fff640 1 -- 172.21.15.50:0/3995601583 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38680a07c0 con 0x55cfeea6a620 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.638+0000 7f3857fff640 1 -- 172.21.15.50:0/3995601583 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3868098360 con 0x55cfeea6a620 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.639+0000 7f3876b90740 1 -- 172.21.15.50:0/3995601583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6a620 msgr2=0x55cfeea6aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.639+0000 7f3876b90740 1 --2- 172.21.15.50:0/3995601583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6a620 0x55cfeea6aa00 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f38680a4b50 tx=0x7f386809c310 comp rx=0 tx=0).stop 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.639+0000 7f3876b90740 1 -- 172.21.15.50:0/3995601583 shutdown_connections 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.639+0000 7f3876b90740 1 --2- 172.21.15.50:0/3995601583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea75800 0x55cfeea77bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.639+0000 7f3876b90740 1 --2- 172.21.15.50:0/3995601583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea6af40 0x55cfeea752c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.639+0000 7f3876b90740 1 --2- 172.21.15.50:0/3995601583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6a620 0x55cfeea6aa00 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.639+0000 7f3876b90740 1 -- 172.21.15.50:0/3995601583 >> 172.21.15.50:0/3995601583 conn(0x55cfeea66b30 msgr2=0x55cfeea66f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.639+0000 7f3876b90740 1 -- 172.21.15.50:0/3995601583 shutdown_connections 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.639+0000 7f3876b90740 1 -- 172.21.15.50:0/3995601583 wait complete. 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.639+0000 7f3876b90740 1 -- start start 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.639+0000 7f3876b90740 1 Processor -- start 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3876b90740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea6a620 0x55cfeea75150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3876b90740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea72440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3876b90740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea75800 0x55cfeea729a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3876b90740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cfeea78370 con 0x55cfeea6af40 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3876b90740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cfee96a0b0 con 0x55cfeea6a620 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3876b90740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55cfeea784f0 con 0x55cfeea75800 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3876b90740 10 client.0 _ll_register_callbacks cb 0x55cfee957d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3871417640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea72440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3871417640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea72440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58672/0 (socket says 172.21.15.50:58672) 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3871417640 1 -- 172.21.15.50:0/373750759 learned_addr learned my addr 172.21.15.50:0/373750759 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3871c18640 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea6a620 0x55cfeea75150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3872419640 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea75800 0x55cfeea729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3871417640 1 -- 172.21.15.50:0/373750759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea75800 msgr2=0x55cfeea729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3871417640 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea75800 0x55cfeea729a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3871417640 1 -- 172.21.15.50:0/373750759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea6a620 msgr2=0x55cfeea75150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3871417640 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea6a620 0x55cfeea75150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3871417640 1 -- 172.21.15.50:0/373750759 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfee7ae620 con 0x55cfeea6af40 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f3871417640 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea72440 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f385c011470 tx=0x7f385c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4630fba55b2c83a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.640+0000 7f38557fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f38557fa640 1 -- 172.21.15.50:0/373750759 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c004220 con 0x55cfeea6af40 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f3876b90740 1 -- 172.21.15.50:0/373750759 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38680a07c0 con 0x55cfeea6af40 2024-04-23T18:30:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f38557fa640 1 -- 172.21.15.50:0/373750759 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f385c0043c0 con 0x55cfeea6af40 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f38557fa640 1 -- 172.21.15.50:0/373750759 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f385c01f700 con 0x55cfeea6af40 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f3876b90740 10 client.16962 fetch_fsmap learned FSMap version 46 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f3876b90740 1 -- 172.21.15.50:0/373750759 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55cfeea84350 con 0x55cfeea6af40 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f38557fa640 1 -- 172.21.15.50:0/373750759 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38680a07c0 con 0x55cfeea6af40 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f38557fa640 1 -- 172.21.15.50:0/373750759 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 46) ==== 8362+0+0 (secure 0 0 0) 0x7f385c003a80 con 0x55cfeea6af40 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f3876b90740 10 client.16962 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f3876b90740 10 client.16962 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f3876b90740 20 client.16962 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f3876b90740 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cfeea85860 0x55cfeea87c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f3876b90740 4 client.16962 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.641+0000 7f3876b90740 1 -- 172.21.15.50:0/373750759 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfee9dd390 con 0x55cfeea85860 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.642+0000 7f3871c18640 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cfeea85860 0x55cfeea87c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.642+0000 7f3871c18640 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cfeea85860 0x55cfeea87c70 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74b0304663a3c7d1 server_cookie=8c1f3a62fbe9e4e4 in_seq=0 out_seq=0 2024-04-23T18:30:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.642+0000 7f38557fa640 10 client.16962 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f38557fa640 1 -- 172.21.15.50:0/373750759 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cfee9dd390 con 0x55cfeea85860 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f38557fa640 10 client.16962 handle_command_reply: tid=0 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f3876b90740 1 client.16962 shutdown 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f3876b90740 1 -- 172.21.15.50:0/373750759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 msgr2=0x55cfeea72440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f3876b90740 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea72440 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f385c011470 tx=0x7f385c0046c0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f3876b90740 1 -- 172.21.15.50:0/373750759 shutdown_connections 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f3876b90740 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cfeea85860 0x55cfeea87c70 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f3876b90740 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea75800 0x55cfeea729a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f3876b90740 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea72440 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f3876b90740 1 --2- 172.21.15.50:0/373750759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea6a620 0x55cfeea75150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f38557fa640 0 client.16962 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f3876b90740 1 -- 172.21.15.50:0/373750759 >> 172.21.15.50:0/373750759 conn(0x55cfeea66b30 msgr2=0x55cfeea6c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f3876b90740 1 -- 172.21.15.50:0/373750759 shutdown_connections 2024-04-23T18:30:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f3876b90740 1 -- 172.21.15.50:0/373750759 wait complete. 2024-04-23T18:30:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.643+0000 7f3876b90740 20 client.16962 trim_cache size 0 max 16384 2024-04-23T18:30:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.659+0000 7f3876b90740 1 Processor -- start 2024-04-23T18:30:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.659+0000 7f3876b90740 1 -- start start 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3876b90740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea6a620 0x55cfeea74c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3876b90740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea7d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3876b90740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea75800 0x55cfeea7dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3876b90740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1b0570 con 0x55cfeea6af40 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3876b90740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1add80 con 0x55cfeea75800 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3876b90740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1abbf0 con 0x55cfeea6a620 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3871417640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea7d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3871c18640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea6a620 0x55cfeea74c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3871417640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea7d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58684/0 (socket says 172.21.15.50:58684) 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3871417640 1 -- 172.21.15.50:0/2354412394 learned_addr learned my addr 172.21.15.50:0/2354412394 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3871c18640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea6a620 0x55cfeea74c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:60554/0 (socket says 172.21.15.50:60554) 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3872419640 1 --2- 172.21.15.50:0/2354412394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea75800 0x55cfeea7dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3871417640 1 -- 172.21.15.50:0/2354412394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea6a620 msgr2=0x55cfeea74c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3871417640 1 --2- 172.21.15.50:0/2354412394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea6a620 0x55cfeea74c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3871417640 1 -- 172.21.15.50:0/2354412394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea75800 msgr2=0x55cfeea7dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3871417640 1 --2- 172.21.15.50:0/2354412394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea75800 0x55cfeea7dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3871417640 1 -- 172.21.15.50:0/2354412394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfeea72d10 con 0x55cfeea6af40 2024-04-23T18:30:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.660+0000 7f3872419640 1 --2- 172.21.15.50:0/2354412394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea75800 0x55cfeea7dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.661+0000 7f3871417640 1 --2- 172.21.15.50:0/2354412394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea7d740 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f385c003f70 tx=0x7f385c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd08fc68ddfd784 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.661+0000 7f38577fe640 1 -- 172.21.15.50:0/2354412394 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c033040 con 0x55cfeea6af40 2024-04-23T18:30:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.661+0000 7f38577fe640 1 -- 172.21.15.50:0/2354412394 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f385c01f8a0 con 0x55cfeea6af40 2024-04-23T18:30:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.661+0000 7f38577fe640 1 -- 172.21.15.50:0/2354412394 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c0039b0 con 0x55cfeea6af40 2024-04-23T18:30:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.662+0000 7f3876b90740 1 -- 172.21.15.50:0/2354412394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 msgr2=0x55cfeea7d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.662+0000 7f3876b90740 1 --2- 172.21.15.50:0/2354412394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea7d740 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f385c003f70 tx=0x7f385c00b4f0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.662+0000 7f3876b90740 1 -- 172.21.15.50:0/2354412394 shutdown_connections 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.662+0000 7f3876b90740 1 --2- 172.21.15.50:0/2354412394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea75800 0x55cfeea7dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.662+0000 7f3876b90740 1 --2- 172.21.15.50:0/2354412394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea7d740 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.662+0000 7f3876b90740 1 --2- 172.21.15.50:0/2354412394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea6a620 0x55cfeea74c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.662+0000 7f3876b90740 1 -- 172.21.15.50:0/2354412394 >> 172.21.15.50:0/2354412394 conn(0x55cfeea66b30 msgr2=0x55cfeea80220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.662+0000 7f3876b90740 1 -- 172.21.15.50:0/2354412394 shutdown_connections 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.662+0000 7f3876b90740 1 -- 172.21.15.50:0/2354412394 wait complete. 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.662+0000 7f3876b90740 1 -- start start 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.662+0000 7f3876b90740 1 Processor -- start 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.663+0000 7f3876b90740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea6a620 0x55cfeea7b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.663+0000 7f3876b90740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea7b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.663+0000 7f3876b90740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea75800 0x55cfeea7be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.663+0000 7f3876b90740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1ac240 con 0x55cfeea6af40 2024-04-23T18:30:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.663+0000 7f3876b90740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1ad6b0 con 0x55cfeea6a620 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.663+0000 7f3876b90740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1ac910 con 0x55cfeea75800 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.663+0000 7f3876b90740 10 client.0 _ll_register_callbacks cb 0x7f385c00bd10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.663+0000 7f3871417640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea7b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.663+0000 7f3871417640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea7b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58690/0 (socket says 172.21.15.50:58690) 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.663+0000 7f3871417640 1 -- 172.21.15.50:0/31605780 learned_addr learned my addr 172.21.15.50:0/31605780 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.663+0000 7f3872419640 1 --2- 172.21.15.50:0/31605780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea75800 0x55cfeea7be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.664+0000 7f3871417640 1 -- 172.21.15.50:0/31605780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea75800 msgr2=0x55cfeea7be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.664+0000 7f3871417640 1 --2- 172.21.15.50:0/31605780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea75800 0x55cfeea7be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.664+0000 7f3871417640 1 -- 172.21.15.50:0/31605780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea6a620 msgr2=0x55cfeea7b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.664+0000 7f3871417640 1 --2- 172.21.15.50:0/31605780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea6a620 0x55cfeea7b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.664+0000 7f3871417640 1 -- 172.21.15.50:0/31605780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfeeaef460 con 0x55cfeea6af40 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.664+0000 7f3871417640 1 --2- 172.21.15.50:0/31605780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea7b940 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f385c027540 tx=0x7f385c005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8582025021aaca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.664+0000 7f38557fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.664+0000 7f38557fa640 1 -- 172.21.15.50:0/31605780 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c033030 con 0x55cfeea6af40 2024-04-23T18:30:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.664+0000 7f38557fa640 1 -- 172.21.15.50:0/31605780 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f385c045ac0 con 0x55cfeea6af40 2024-04-23T18:30:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.665+0000 7f3876b90740 1 -- 172.21.15.50:0/31605780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f385c01f8a0 con 0x55cfeea6af40 2024-04-23T18:30:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.665+0000 7f38557fa640 1 -- 172.21.15.50:0/31605780 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f385c054050 con 0x55cfeea6af40 2024-04-23T18:30:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.665+0000 7f3876b90740 10 client.16968 fetch_fsmap learned FSMap version 46 2024-04-23T18:30:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.665+0000 7f3876b90740 1 -- 172.21.15.50:0/31605780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55cfeeae1da0 con 0x55cfeea6af40 2024-04-23T18:30:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.665+0000 7f38557fa640 1 -- 172.21.15.50:0/31605780 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f385c049030 con 0x55cfeea6af40 2024-04-23T18:30:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.665+0000 7f38557fa640 1 -- 172.21.15.50:0/31605780 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 46) ==== 8362+0+0 (secure 0 0 0) 0x7f385c036070 con 0x55cfeea6af40 2024-04-23T18:30:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.665+0000 7f3876b90740 10 client.16968 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:30:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.665+0000 7f3876b90740 10 client.16968 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.665+0000 7f3876b90740 20 client.16968 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.665+0000 7f3876b90740 1 --2- 172.21.15.50:0/31605780 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cfeeae24a0 0x55cfeeae48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.665+0000 7f3876b90740 4 client.16968 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:30:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.665+0000 7f3876b90740 1 -- 172.21.15.50:0/31605780 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f385c01f8a0 con 0x55cfeeae24a0 2024-04-23T18:30:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.666+0000 7f3871c18640 1 --2- 172.21.15.50:0/31605780 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cfeeae24a0 0x55cfeeae48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.666+0000 7f38557fa640 10 client.16968 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.666+0000 7f3871c18640 1 --2- 172.21.15.50:0/31605780 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cfeeae24a0 0x55cfeeae48c0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a06ad49909c68bf server_cookie=1adb5c8c83c50350 in_seq=0 out_seq=0 2024-04-23T18:30:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.741+0000 7f38557fa640 1 -- 172.21.15.50:0/31605780 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f385c01f8a0 con 0x55cfeeae24a0 2024-04-23T18:30:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.741+0000 7f38557fa640 10 client.16968 handle_command_reply: tid=0 2024-04-23T18:30:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.741+0000 7f3876b90740 1 client.16968 shutdown 2024-04-23T18:30:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.741+0000 7f3876b90740 1 -- 172.21.15.50:0/31605780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 msgr2=0x55cfeea7b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.741+0000 7f3876b90740 1 --2- 172.21.15.50:0/31605780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea7b940 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f385c027540 tx=0x7f385c005bf0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3876b90740 1 -- 172.21.15.50:0/31605780 shutdown_connections 2024-04-23T18:30:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3876b90740 1 --2- 172.21.15.50:0/31605780 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cfeeae24a0 0x55cfeeae48c0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f38557fa640 0 client.16968 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3876b90740 1 --2- 172.21.15.50:0/31605780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cfeea75800 0x55cfeea7be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3876b90740 1 --2- 172.21.15.50:0/31605780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cfeea6af40 0x55cfeea7b940 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3876b90740 1 --2- 172.21.15.50:0/31605780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfeea6a620 0x55cfeea7b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3876b90740 1 -- 172.21.15.50:0/31605780 >> 172.21.15.50:0/31605780 conn(0x55cfeea66b30 msgr2=0x55cfeea80ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3876b90740 1 -- 172.21.15.50:0/31605780 shutdown_connections 2024-04-23T18:30:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3876b90740 1 -- 172.21.15.50:0/31605780 wait complete. 2024-04-23T18:30:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3876b90740 20 client.16968 trim_cache size 0 max 16384 2024-04-23T18:30:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:30:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:30:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:30:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:30:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3873e7c640 1 -- 172.21.15.50:0/819294855 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f38580a3e40 msgr2=0x7f38580a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3873e7c640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f38580a3e40 0x7f38580a6300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f385c005fd0 tx=0x7f385c005850 comp rx=0 tx=0).stop 2024-04-23T18:30:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3873e7c640 1 -- 172.21.15.50:0/819294855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f386c1021c0 msgr2=0x7f386c1986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3873e7c640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f386c1021c0 0x7f386c1986a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f386800d950 tx=0x7f386800de10 comp rx=0 tx=0).stop 2024-04-23T18:30:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3873e7c640 1 -- 172.21.15.50:0/819294855 shutdown_connections 2024-04-23T18:30:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3873e7c640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f38580a3e40 0x7f38580a6300 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3873e7c640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f386c19cf10 0x7f386c19d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3873e7c640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f386c107290 0x7f386c198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.742+0000 7f3873e7c640 1 --2- 172.21.15.50:0/819294855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f386c1021c0 0x7f386c1986a0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.743+0000 7f3873e7c640 1 -- 172.21.15.50:0/819294855 >> 172.21.15.50:0/819294855 conn(0x7f386c0fda20 msgr2=0x7f386c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.743+0000 7f3873e7c640 1 -- 172.21.15.50:0/819294855 shutdown_connections 2024-04-23T18:30:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:30.743+0000 7f3873e7c640 1 -- 172.21.15.50:0/819294855 wait complete. 2024-04-23T18:30:31.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.43 MB/sec execute 545 sec latency 7227.754 ms 2024-04-23T18:30:32.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.42 MB/sec execute 546 sec latency 8227.806 ms 2024-04-23T18:30:32.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:32 smithi050 ceph-mon[29187]: pgmap v378: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-04-23T18:30:32.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:32 smithi177 ceph-mon[32308]: pgmap v378: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-04-23T18:30:32.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:32 smithi188 ceph-mon[32694]: pgmap v378: 97 pgs: 97 active+clean; 241 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-04-23T18:30:33.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.41 MB/sec execute 547 sec latency 9227.909 ms 2024-04-23T18:30:34.234 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.40 MB/sec execute 548 sec latency 10227.998 ms 2024-04-23T18:30:34.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:34 smithi050 ceph-mon[29187]: pgmap v379: 97 pgs: 97 active+clean; 237 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 207 op/s 2024-04-23T18:30:34.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:34 smithi177 ceph-mon[32308]: pgmap v379: 97 pgs: 97 active+clean; 237 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 207 op/s 2024-04-23T18:30:34.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:34 smithi188 ceph-mon[32694]: pgmap v379: 97 pgs: 97 active+clean; 237 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 207 op/s 2024-04-23T18:30:35.235 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.39 MB/sec execute 549 sec latency 11228.133 ms 2024-04-23T18:30:36.235 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.38 MB/sec execute 550 sec latency 12228.244 ms 2024-04-23T18:30:36.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:36 smithi050 ceph-mon[29187]: pgmap v380: 97 pgs: 97 active+clean; 237 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 1.2 MiB/s wr, 98 op/s 2024-04-23T18:30:36.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:36 smithi177 ceph-mon[32308]: pgmap v380: 97 pgs: 97 active+clean; 237 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 1.2 MiB/s wr, 98 op/s 2024-04-23T18:30:36.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:36 smithi188 ceph-mon[32694]: pgmap v380: 97 pgs: 97 active+clean; 237 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 1.2 MiB/s wr, 98 op/s 2024-04-23T18:30:37.235 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.37 MB/sec execute 551 sec latency 13228.396 ms 2024-04-23T18:30:38.235 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.36 MB/sec execute 552 sec latency 14228.513 ms 2024-04-23T18:30:38.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:38 smithi050 ceph-mon[29187]: pgmap v381: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.8 MiB/s wr, 120 op/s 2024-04-23T18:30:38.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:38 smithi177 ceph-mon[32308]: pgmap v381: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.8 MiB/s wr, 120 op/s 2024-04-23T18:30:38.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:38 smithi188 ceph-mon[32694]: pgmap v381: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.8 MiB/s wr, 120 op/s 2024-04-23T18:30:39.235 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.35 MB/sec execute 553 sec latency 15228.625 ms 2024-04-23T18:30:40.235 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.34 MB/sec execute 554 sec latency 16228.709 ms 2024-04-23T18:30:40.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:40 smithi050 ceph-mon[29187]: pgmap v382: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 1.1 MiB/s wr, 61 op/s 2024-04-23T18:30:40.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:40 smithi177 ceph-mon[32308]: pgmap v382: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 1.1 MiB/s wr, 61 op/s 2024-04-23T18:30:40.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:40 smithi188 ceph-mon[32694]: pgmap v382: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 1.1 MiB/s wr, 61 op/s 2024-04-23T18:30:41.235 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.33 MB/sec execute 555 sec latency 17228.767 ms 2024-04-23T18:30:42.235 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.32 MB/sec execute 556 sec latency 18228.894 ms 2024-04-23T18:30:42.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:42 smithi050 ceph-mon[29187]: pgmap v383: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 1.1 MiB/s wr, 61 op/s 2024-04-23T18:30:42.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:42 smithi177 ceph-mon[32308]: pgmap v383: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 1.1 MiB/s wr, 61 op/s 2024-04-23T18:30:42.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:42 smithi188 ceph-mon[32694]: pgmap v383: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 1.1 MiB/s wr, 61 op/s 2024-04-23T18:30:43.235 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.31 MB/sec execute 557 sec latency 19229.025 ms 2024-04-23T18:30:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.843+0000 7f5d30efd640 1 -- 172.21.15.50:0/4064009694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d2c100d80 msgr2=0x7f5d2c101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.843+0000 7f5d30efd640 1 --2- 172.21.15.50:0/4064009694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d2c100d80 0x7f5d2c101160 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009ca0 tx=0x7f5d1c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.843+0000 7f5d30efd640 1 -- 172.21.15.50:0/4064009694 shutdown_connections 2024-04-23T18:30:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.843+0000 7f5d30efd640 1 --2- 172.21.15.50:0/4064009694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d2c10e930 0x7f5d2c110d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.843+0000 7f5d30efd640 1 --2- 172.21.15.50:0/4064009694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d2c1016a0 0x7f5d2c10e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.843+0000 7f5d30efd640 1 --2- 172.21.15.50:0/4064009694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d2c100d80 0x7f5d2c101160 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.843+0000 7f5d30efd640 1 -- 172.21.15.50:0/4064009694 >> 172.21.15.50:0/4064009694 conn(0x7f5d2c077fd0 msgr2=0x7f5d2c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.843+0000 7f5d30efd640 1 -- 172.21.15.50:0/4064009694 shutdown_connections 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.843+0000 7f5d30efd640 1 -- 172.21.15.50:0/4064009694 wait complete. 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d30efd640 1 Processor -- start 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d30efd640 1 -- start start 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d30efd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d2c100d80 0x7f5d2c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d30efd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d2c1016a0 0x7f5d2c1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d30efd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d2c10e930 0x7f5d2c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d30efd640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c1143c0 con 0x7f5d2c10e930 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d30efd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c114240 con 0x7f5d2c100d80 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d30efd640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c114540 con 0x7f5d2c1016a0 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d2ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d2c10e930 0x7f5d2c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d2ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d2c10e930 0x7f5d2c19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43032/0 (socket says 172.21.15.50:43032) 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d2ad9d640 1 -- 172.21.15.50:0/4234391997 learned_addr learned my addr 172.21.15.50:0/4234391997 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d2c100d80 0x7f5d2c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d2c1016a0 0x7f5d2c1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d2ad9d640 1 -- 172.21.15.50:0/4234391997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d2c1016a0 msgr2=0x7f5d2c1a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d2ad9d640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d2c1016a0 0x7f5d2c1a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d2ad9d640 1 -- 172.21.15.50:0/4234391997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d2c100d80 msgr2=0x7f5d2c1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d2ad9d640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d2c100d80 0x7f5d2c1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d2ad9d640 1 -- 172.21.15.50:0/4234391997 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2c19b020 con 0x7f5d2c10e930 2024-04-23T18:30:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.844+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d2c1016a0 0x7f5d2c1a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.845+0000 7f5d2ad9d640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d2c10e930 0x7f5d2c19a760 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400e970 tx=0x7f5d2400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.845+0000 7f5d1b7fe640 1 -- 172.21.15.50:0/4234391997 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2400ccc0 con 0x7f5d2c10e930 2024-04-23T18:30:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.845+0000 7f5d1b7fe640 1 -- 172.21.15.50:0/4234391997 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d240054d0 con 0x7f5d2c10e930 2024-04-23T18:30:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.845+0000 7f5d30efd640 1 -- 172.21.15.50:0/4234391997 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d2c19b310 con 0x7f5d2c10e930 2024-04-23T18:30:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.845+0000 7f5d1b7fe640 1 -- 172.21.15.50:0/4234391997 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d24010430 con 0x7f5d2c10e930 2024-04-23T18:30:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.845+0000 7f5d30efd640 1 -- 172.21.15.50:0/4234391997 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d2c1a75d0 con 0x7f5d2c10e930 2024-04-23T18:30:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.847+0000 7f5d1b7fe640 1 -- 172.21.15.50:0/4234391997 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5d24005020 con 0x7f5d2c10e930 2024-04-23T18:30:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d33c0e740 1 Processor -- start 2024-04-23T18:30:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d33c0e740 1 -- start start 2024-04-23T18:30:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d33c0e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbba3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d33c0e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d33c0e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbb0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d33c0e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5582cbb08970 con 0x5582cbba3f40 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d33c0e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5582cbaa54a0 con 0x5582cbbae800 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d33c0e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5582cbb04930 con 0x5582cbba3620 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d1b7fe640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5d140a3dc0 0x7f5d140a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d2a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d29d9b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbba3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d2a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbae2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43040/0 (socket says 172.21.15.50:43040) 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.848+0000 7f5d2a59c640 1 -- 172.21.15.50:0/2915313338 learned_addr learned my addr 172.21.15.50:0/2915313338 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d2ad9d640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5d140a3dc0 0x7f5d140a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/2915313338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbb0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d1b7fe640 1 -- 172.21.15.50:0/4234391997 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f5d240d0b90 con 0x7f5d2c10e930 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d2a59c640 1 -- 172.21.15.50:0/2915313338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 msgr2=0x5582cbba3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/2915313338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbba3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d2a59c640 1 -- 172.21.15.50:0/2915313338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 msgr2=0x5582cbbb0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/2915313338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbb0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d2a59c640 1 -- 172.21.15.50:0/2915313338 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582cb81b210 con 0x5582cbba3f40 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/2915313338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbb0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/2915313338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbae2c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c035000 tx=0x7f5d1c0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594cc4f518fbb004 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d2ad9d640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5d140a3dc0 0x7f5d140a6280 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5d2409c820 tx=0x7f5d2406e560 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d18ff9640 1 -- 172.21.15.50:0/2915313338 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c031cf0 con 0x5582cbba3f40 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d18ff9640 1 -- 172.21.15.50:0/2915313338 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d1c036030 con 0x5582cbba3f40 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.849+0000 7f5d18ff9640 1 -- 172.21.15.50:0/2915313338 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c004260 con 0x5582cbba3f40 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.850+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2915313338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 msgr2=0x5582cbbae2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.850+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/2915313338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbae2c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c035000 tx=0x7f5d1c0093f0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.850+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2915313338 shutdown_connections 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.850+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/2915313338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbb0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.850+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/2915313338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbae2c0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.850+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/2915313338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbba3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.850+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2915313338 >> 172.21.15.50:0/2915313338 conn(0x5582cbb9fb30 msgr2=0x5582cbb9ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.850+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2915313338 shutdown_connections 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.850+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2915313338 wait complete. 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.850+0000 7f5d33c0e740 1 -- start start 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d33c0e740 1 Processor -- start 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d33c0e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbbae080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d33c0e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbab370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d33c0e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d33c0e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5582cbbb1250 con 0x5582cbba3f40 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d33c0e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5582cbaa30b0 con 0x5582cbbae800 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d33c0e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5582cbbb13d0 con 0x5582cbba3620 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d33c0e740 10 client.0 _ll_register_callbacks cb 0x5582cba90d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d2a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbab370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d2a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbab370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43048/0 (socket says 172.21.15.50:43048) 2024-04-23T18:30:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d2a59c640 1 -- 172.21.15.50:0/2760807077 learned_addr learned my addr 172.21.15.50:0/2760807077 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbbae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d2a59c640 1 -- 172.21.15.50:0/2760807077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 msgr2=0x5582cbbae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbbae080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d2a59c640 1 -- 172.21.15.50:0/2760807077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 msgr2=0x5582cbbab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbab8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d2a59c640 1 -- 172.21.15.50:0/2760807077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582cb8e7620 con 0x5582cbba3f40 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbab370 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009a60 tx=0x7f5d1c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74aa7ab5f6145a86 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.851+0000 7f5d11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d11ffb640 1 -- 172.21.15.50:0/2760807077 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c052020 con 0x5582cbba3f40 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d11ffb640 1 -- 172.21.15.50:0/2760807077 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f5d1c031650 con 0x5582cbba3f40 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2760807077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d1c036030 con 0x5582cbba3f40 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d11ffb640 1 -- 172.21.15.50:0/2760807077 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f5d1c04a040 con 0x5582cbba3f40 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d33c0e740 10 client.16980 fetch_fsmap learned FSMap version 46 2024-04-23T18:30:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2760807077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5582cbbb1250 con 0x5582cbba3f40 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d11ffb640 1 -- 172.21.15.50:0/2760807077 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d1c036030 con 0x5582cbba3f40 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d11ffb640 1 -- 172.21.15.50:0/2760807077 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 46) ==== 8362+0+0 (secure 0 0 0) 0x7f5d1c04d030 con 0x5582cbba3f40 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d33c0e740 10 client.16980 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d33c0e740 10 client.16980 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d33c0e740 20 client.16980 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582cbbbe2d0 0x5582cbbc06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d33c0e740 4 client.16980 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.852+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2760807077 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582cbb16390 con 0x5582cbbbe2d0 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.853+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582cbbbe2d0 0x5582cbbc06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.853+0000 7f5d11ffb640 10 client.16980 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.853+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582cbbbe2d0 0x5582cbbc06c0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95793b1c3e4d109b server_cookie=c3363ecb2fd36068 in_seq=0 out_seq=0 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d11ffb640 1 -- 172.21.15.50:0/2760807077 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582cbb16390 con 0x5582cbbbe2d0 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d11ffb640 10 client.16980 handle_command_reply: tid=0 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d33c0e740 1 client.16980 shutdown 2024-04-23T18:30:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2760807077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 msgr2=0x5582cbbab370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbab370 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009a60 tx=0x7f5d1c003d50 comp rx=0 tx=0).stop 2024-04-23T18:30:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2760807077 shutdown_connections 2024-04-23T18:30:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582cbbbe2d0 0x5582cbbc06c0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbab370 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/2760807077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbbae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d11ffb640 0 client.16980 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2760807077 >> 172.21.15.50:0/2760807077 conn(0x5582cbb9fb30 msgr2=0x5582cbba5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2760807077 shutdown_connections 2024-04-23T18:30:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d33c0e740 1 -- 172.21.15.50:0/2760807077 wait complete. 2024-04-23T18:30:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.854+0000 7f5d33c0e740 20 client.16980 trim_cache size 0 max 16384 2024-04-23T18:30:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d33c0e740 1 Processor -- start 2024-04-23T18:30:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d33c0e740 1 -- start start 2024-04-23T18:30:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d33c0e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbbba110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d33c0e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbaa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d33c0e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbb9eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d33c0e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c1b2730 con 0x5582cbba3f40 2024-04-23T18:30:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d33c0e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c1aff40 con 0x5582cbbae800 2024-04-23T18:30:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d33c0e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c1addb0 con 0x5582cbba3620 2024-04-23T18:30:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d2a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbaa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d2a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbaa4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43052/0 (socket says 172.21.15.50:43052) 2024-04-23T18:30:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d2a59c640 1 -- 172.21.15.50:0/3437762861 learned_addr learned my addr 172.21.15.50:0/3437762861 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/3437762861 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbbba110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/3437762861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbb9eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d2a59c640 1 -- 172.21.15.50:0/3437762861 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 msgr2=0x5582cbbba110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/3437762861 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbbba110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d2a59c640 1 -- 172.21.15.50:0/3437762861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 msgr2=0x5582cbb9eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.869+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/3437762861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbb9eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d2a59c640 1 -- 172.21.15.50:0/3437762861 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582cbbc0a30 con 0x5582cbba3f40 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/3437762861 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbbba110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/3437762861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbaa4d0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c0040e0 tx=0x7f5d1c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b98f875d95c4c50e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d13fff640 1 -- 172.21.15.50:0/3437762861 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c052070 con 0x5582cbba3f40 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d13fff640 1 -- 172.21.15.50:0/3437762861 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d1c036070 con 0x5582cbba3f40 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d13fff640 1 -- 172.21.15.50:0/3437762861 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c02f660 con 0x5582cbba3f40 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d33c0e740 1 -- 172.21.15.50:0/3437762861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 msgr2=0x5582cbbaa4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/3437762861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbaa4d0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c0040e0 tx=0x7f5d1c02bfd0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d33c0e740 1 -- 172.21.15.50:0/3437762861 shutdown_connections 2024-04-23T18:30:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/3437762861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbb9eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/3437762861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3f40 0x5582cbbaa4d0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/3437762861 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3620 0x5582cbbba110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d33c0e740 1 -- 172.21.15.50:0/3437762861 >> 172.21.15.50:0/3437762861 conn(0x5582cbb9fb30 msgr2=0x5582cbb9e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d33c0e740 1 -- 172.21.15.50:0/3437762861 shutdown_connections 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.870+0000 7f5d33c0e740 1 -- 172.21.15.50:0/3437762861 wait complete. 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d33c0e740 1 -- start start 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d33c0e740 1 Processor -- start 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d33c0e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3620 0x5582cbbb62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d33c0e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3f40 0x5582cbbb6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d33c0e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbb3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d33c0e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c1ae400 con 0x5582cbba3620 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d33c0e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c1af870 con 0x5582cbbae800 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d33c0e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c1aead0 con 0x5582cbba3f40 2024-04-23T18:30:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d33c0e740 10 client.0 _ll_register_callbacks cb 0x7f5d1c031e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d29d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3620 0x5582cbbb62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d29d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3620 0x5582cbbb62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43066/0 (socket says 172.21.15.50:43066) 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d29d9b640 1 -- 172.21.15.50:0/4089594174 learned_addr learned my addr 172.21.15.50:0/4089594174 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/4089594174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3f40 0x5582cbbb6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d29d9b640 1 -- 172.21.15.50:0/4089594174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3f40 msgr2=0x5582cbbb6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/4089594174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3f40 0x5582cbbb6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.871+0000 7f5d29d9b640 1 -- 172.21.15.50:0/4089594174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 msgr2=0x5582cbbb3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.872+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/4089594174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbb3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.872+0000 7f5d29d9b640 1 -- 172.21.15.50:0/4089594174 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582cbbb3f90 con 0x5582cbba3620 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.872+0000 7f5d29d9b640 1 --2- 172.21.15.50:0/4089594174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3620 0x5582cbbb62e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000c4e0 tx=0x7f5d200229d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c689f7137d53c247 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.872+0000 7f5d11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.872+0000 7f5d11ffb640 1 -- 172.21.15.50:0/4089594174 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20026020 con 0x5582cbba3620 2024-04-23T18:30:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.872+0000 7f5d11ffb640 1 -- 172.21.15.50:0/4089594174 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f5d20014e10 con 0x5582cbba3620 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.872+0000 7f5d33c0e740 1 -- 172.21.15.50:0/4089594174 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d1c036070 con 0x5582cbba3620 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.872+0000 7f5d11ffb640 1 -- 172.21.15.50:0/4089594174 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f5d20031050 con 0x5582cbba3620 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.872+0000 7f5d33c0e740 10 client.16986 fetch_fsmap learned FSMap version 46 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.872+0000 7f5d33c0e740 1 -- 172.21.15.50:0/4089594174 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5582cbc24360 con 0x5582cbba3620 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.872+0000 7f5d11ffb640 1 -- 172.21.15.50:0/4089594174 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d1c036070 con 0x5582cbba3620 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.872+0000 7f5d11ffb640 1 -- 172.21.15.50:0/4089594174 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 46) ==== 8362+0+0 (secure 0 0 0) 0x7f5d20022040 con 0x5582cbba3620 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.873+0000 7f5d33c0e740 10 client.16986 fetch_fsmap finished waiting for FSMap version 46 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.873+0000 7f5d33c0e740 10 client.16986 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.873+0000 7f5d33c0e740 20 client.16986 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.873+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/4089594174 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582cbc24b40 0x5582cbc26f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.873+0000 7f5d33c0e740 4 client.16986 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.873+0000 7f5d33c0e740 1 -- 172.21.15.50:0/4089594174 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5582cbb10200 con 0x5582cbc24b40 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.873+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/4089594174 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582cbc24b40 0x5582cbc26f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.873+0000 7f5d2a59c640 1 --2- 172.21.15.50:0/4089594174 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582cbc24b40 0x5582cbc26f60 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=305520ad0854ade2 server_cookie=e55fcc8fd3624d65 in_seq=0 out_seq=0 2024-04-23T18:30:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.873+0000 7f5d11ffb640 10 client.16986 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.905+0000 7f5d11ffb640 1 -- 172.21.15.50:0/4089594174 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5582cbb10200 con 0x5582cbc24b40 2024-04-23T18:30:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d11ffb640 10 client.16986 handle_command_reply: tid=0 2024-04-23T18:30:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d33c0e740 1 client.16986 shutdown 2024-04-23T18:30:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d33c0e740 1 -- 172.21.15.50:0/4089594174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3620 msgr2=0x5582cbbb62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/4089594174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3620 0x5582cbbb62e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000c4e0 tx=0x7f5d200229d0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d33c0e740 1 -- 172.21.15.50:0/4089594174 shutdown_connections 2024-04-23T18:30:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/4089594174 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5582cbc24b40 0x5582cbc26f60 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/4089594174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5582cbbae800 0x5582cbbb3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/4089594174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5582cbba3f40 0x5582cbbb6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d33c0e740 1 --2- 172.21.15.50:0/4089594174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5582cbba3620 0x5582cbbb62e0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d11ffb640 0 client.16986 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d33c0e740 1 -- 172.21.15.50:0/4089594174 >> 172.21.15.50:0/4089594174 conn(0x5582cbb9fb30 msgr2=0x5582cbb9e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d33c0e740 1 -- 172.21.15.50:0/4089594174 shutdown_connections 2024-04-23T18:30:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d33c0e740 1 -- 172.21.15.50:0/4089594174 wait complete. 2024-04-23T18:30:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.906+0000 7f5d33c0e740 20 client.16986 trim_cache size 0 max 16384 2024-04-23T18:30:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:30:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:30:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:30:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:30:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.907+0000 7f5d30efd640 1 -- 172.21.15.50:0/4234391997 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5d140a3dc0 msgr2=0x7f5d140a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.907+0000 7f5d30efd640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5d140a3dc0 0x7f5d140a6280 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5d2409c820 tx=0x7f5d2406e560 comp rx=0 tx=0).stop 2024-04-23T18:30:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.907+0000 7f5d30efd640 1 -- 172.21.15.50:0/4234391997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d2c10e930 msgr2=0x7f5d2c19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.907+0000 7f5d30efd640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d2c10e930 0x7f5d2c19a760 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400e970 tx=0x7f5d2400ee30 comp rx=0 tx=0).stop 2024-04-23T18:30:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.907+0000 7f5d30efd640 1 -- 172.21.15.50:0/4234391997 shutdown_connections 2024-04-23T18:30:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.907+0000 7f5d30efd640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5d140a3dc0 0x7f5d140a6280 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.907+0000 7f5d30efd640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d2c10e930 0x7f5d2c19a760 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.907+0000 7f5d30efd640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d2c1016a0 0x7f5d2c1a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.907+0000 7f5d30efd640 1 --2- 172.21.15.50:0/4234391997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d2c100d80 0x7f5d2c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.907+0000 7f5d30efd640 1 -- 172.21.15.50:0/4234391997 >> 172.21.15.50:0/4234391997 conn(0x7f5d2c077fd0 msgr2=0x7f5d2c10fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.907+0000 7f5d30efd640 1 -- 172.21.15.50:0/4234391997 shutdown_connections 2024-04-23T18:30:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:43.907+0000 7f5d30efd640 1 -- 172.21.15.50:0/4234391997 wait complete. 2024-04-23T18:30:44.236 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.30 MB/sec execute 558 sec latency 20229.130 ms 2024-04-23T18:30:44.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:44 smithi050 ceph-mon[29187]: pgmap v384: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-04-23T18:30:44.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:44 smithi177 ceph-mon[32308]: pgmap v384: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-04-23T18:30:44.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:44 smithi188 ceph-mon[32694]: pgmap v384: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-04-23T18:30:45.236 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.29 MB/sec execute 559 sec latency 21229.273 ms 2024-04-23T18:30:46.236 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.28 MB/sec execute 560 sec latency 22229.370 ms 2024-04-23T18:30:46.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:46 smithi050 ceph-mon[29187]: pgmap v385: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 1.1 MiB/s wr, 44 op/s 2024-04-23T18:30:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:46 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:30:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:46 smithi177 ceph-mon[32308]: pgmap v385: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 1.1 MiB/s wr, 44 op/s 2024-04-23T18:30:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:46 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:30:46.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:46 smithi188 ceph-mon[32694]: pgmap v385: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 1.1 MiB/s wr, 44 op/s 2024-04-23T18:30:46.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:46 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:30:47.236 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.27 MB/sec execute 561 sec latency 23229.523 ms 2024-04-23T18:30:47.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:30:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.096+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/2837559375 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdec103230 msgr2=0x7fcdec103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.096+0000 7fcdf2c4e640 1 --2- 172.21.15.50:0/2837559375 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdec103230 0x7fcdec103610 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4009900 tx=0x7fcdd402a480 comp rx=0 tx=0).stop 2024-04-23T18:30:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.096+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/2837559375 shutdown_connections 2024-04-23T18:30:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.096+0000 7fcdf2c4e640 1 --2- 172.21.15.50:0/2837559375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdec10c570 0x7fcdec10e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.096+0000 7fcdf2c4e640 1 --2- 172.21.15.50:0/2837559375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdec103b50 0x7fcdec10c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.096+0000 7fcdf2c4e640 1 --2- 172.21.15.50:0/2837559375 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdec103230 0x7fcdec103610 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.096+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/2837559375 >> 172.21.15.50:0/2837559375 conn(0x7fcdec077fb0 msgr2=0x7fcdec0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.096+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/2837559375 shutdown_connections 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.096+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/2837559375 wait complete. 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdf2c4e640 1 Processor -- start 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdf2c4e640 1 -- start start 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdf2c4e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdec103230 0x7fcdec100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdf2c4e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdec103b50 0x7fcdec1013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdf2c4e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdec10c570 0x7fcdec101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdf2c4e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec1121d0 con 0x7fcdec103230 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdf2c4e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec112050 con 0x7fcdec103b50 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdf2c4e640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec112350 con 0x7fcdec10c570 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdf09ea640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdec103230 0x7fcdec100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdf09ea640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdec103230 0x7fcdec100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43082/0 (socket says 172.21.15.50:43082) 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdf09ea640 1 -- 172.21.15.50:0/3623826505 learned_addr learned my addr 172.21.15.50:0/3623826505 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdf11eb640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdec10c570 0x7fcdec101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.097+0000 7fcdebfff640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdec103b50 0x7fcdec1013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.098+0000 7fcdf09ea640 1 -- 172.21.15.50:0/3623826505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdec10c570 msgr2=0x7fcdec101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.098+0000 7fcdf09ea640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdec10c570 0x7fcdec101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.098+0000 7fcdf09ea640 1 -- 172.21.15.50:0/3623826505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdec103b50 msgr2=0x7fcdec1013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.098+0000 7fcdf09ea640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdec103b50 0x7fcdec1013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.098+0000 7fcdf09ea640 1 -- 172.21.15.50:0/3623826505 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdec1a6c90 con 0x7fcdec103230 2024-04-23T18:30:48.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.098+0000 7fcdf09ea640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdec103230 0x7fcdec100e80 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4005940 tx=0x7fcdd4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:48.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.098+0000 7fcde9ffb640 1 -- 172.21.15.50:0/3623826505 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd403f030 con 0x7fcdec103230 2024-04-23T18:30:48.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.098+0000 7fcde9ffb640 1 -- 172.21.15.50:0/3623826505 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd4003aa0 con 0x7fcdec103230 2024-04-23T18:30:48.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.098+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/3623826505 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdec1a6f20 con 0x7fcdec103230 2024-04-23T18:30:48.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.098+0000 7fcde9ffb640 1 -- 172.21.15.50:0/3623826505 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4035070 con 0x7fcdec103230 2024-04-23T18:30:48.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.098+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/3623826505 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdec1a7380 con 0x7fcdec103230 2024-04-23T18:30:48.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.099+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/3623826505 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdec0691c0 con 0x7fcdec103230 2024-04-23T18:30:48.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.100+0000 7fcde9ffb640 1 -- 172.21.15.50:0/3623826505 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcdd4042030 con 0x7fcdec103230 2024-04-23T18:30:48.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.101+0000 7fcde9ffb640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcdd80a3d20 0x7fcdd80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.101+0000 7fcde9ffb640 1 -- 172.21.15.50:0/3623826505 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7fcdd40f0ea0 con 0x7fcdec103230 2024-04-23T18:30:48.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.101+0000 7fcdebfff640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcdd80a3d20 0x7fcdd80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.102+0000 7fcdebfff640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcdd80a3d20 0x7fcdd80a61e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcdec102910 tx=0x7fcddc009500 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:48.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.106+0000 7fcde9ffb640 1 -- 172.21.15.50:0/3623826505 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcdd40bce80 con 0x7fcdec103230 2024-04-23T18:30:48.236 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740180 5.26 MB/sec execute 562 sec latency 24229.585 ms 2024-04-23T18:30:48.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.361+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/3623826505 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdec107160 con 0x7fcdec103230 2024-04-23T18:30:48.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.362+0000 7fcde9ffb640 1 -- 172.21.15.50:0/3623826505 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+11882 (secure 0 0 0) 0x7fcdd40c1180 con 0x7fcdec103230 2024-04-23T18:30:48.355 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:30:48.355 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:30:44.945124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/4033440314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4033440314},{"type":"v1","addr":"172.21.15.177:6837","nonce":4033440314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:30:48.355 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 47 2024-04-23T18:30:48.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.364+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/3623826505 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcdd80a3d20 msgr2=0x7fcdd80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.364+0000 7fcdf2c4e640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcdd80a3d20 0x7fcdd80a61e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcdec102910 tx=0x7fcddc009500 comp rx=0 tx=0).stop 2024-04-23T18:30:48.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.364+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/3623826505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdec103230 msgr2=0x7fcdec100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.364+0000 7fcdf2c4e640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdec103230 0x7fcdec100e80 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4005940 tx=0x7fcdd4004270 comp rx=0 tx=0).stop 2024-04-23T18:30:48.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.364+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/3623826505 shutdown_connections 2024-04-23T18:30:48.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.364+0000 7fcdf2c4e640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcdd80a3d20 0x7fcdd80a61e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.364+0000 7fcdf2c4e640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdec10c570 0x7fcdec101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.364+0000 7fcdf2c4e640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdec103b50 0x7fcdec1013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.364+0000 7fcdf2c4e640 1 --2- 172.21.15.50:0/3623826505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdec103230 0x7fcdec100e80 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.364+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/3623826505 >> 172.21.15.50:0/3623826505 conn(0x7fcdec077fb0 msgr2=0x7fcdec0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:48.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.364+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/3623826505 shutdown_connections 2024-04-23T18:30:48.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.364+0000 7fcdf2c4e640 1 -- 172.21.15.50:0/3623826505 wait complete. 2024-04-23T18:30:48.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:48 smithi050 ceph-mon[29187]: pgmap v386: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-04-23T18:30:48.367 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id a5988bee --release --await' 2024-04-23T18:30:48.367 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id a5988bee --release --await 2024-04-23T18:30:48.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:48 smithi188 ceph-mon[32694]: pgmap v386: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-04-23T18:30:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.484+0000 7f2457b08640 1 -- 172.21.15.50:0/544573461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2450102b40 msgr2=0x7f2450106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.484+0000 7f2457b08640 1 --2- 172.21.15.50:0/544573461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2450102b40 0x7f2450106b80 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2440009900 tx=0x7f244002a410 comp rx=0 tx=0).stop 2024-04-23T18:30:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.484+0000 7f2457b08640 1 -- 172.21.15.50:0/544573461 shutdown_connections 2024-04-23T18:30:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.484+0000 7f2457b08640 1 --2- 172.21.15.50:0/544573461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2450107260 0x7f245010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.484+0000 7f2457b08640 1 --2- 172.21.15.50:0/544573461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2450102b40 0x7f2450106b80 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.484+0000 7f2457b08640 1 --2- 172.21.15.50:0/544573461 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2450102190 0x7f2450102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.484+0000 7f2457b08640 1 -- 172.21.15.50:0/544573461 >> 172.21.15.50:0/544573461 conn(0x7f24500793a0 msgr2=0x7f24500797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.484+0000 7f2457b08640 1 -- 172.21.15.50:0/544573461 shutdown_connections 2024-04-23T18:30:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.484+0000 7f2457b08640 1 -- 172.21.15.50:0/544573461 wait complete. 2024-04-23T18:30:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f2457b08640 1 Processor -- start 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f2457b08640 1 -- start start 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f2457b08640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2450102190 0x7f245019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f2457b08640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2450107260 0x7f245019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f2457b08640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2450198620 0x7f2450198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f2457b08640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f24501121a0 con 0x7f2450102190 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f2457b08640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2450112020 con 0x7f2450107260 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f2457b08640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2450112320 con 0x7f2450198620 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24558a4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2450102190 0x7f245019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24558a4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2450102190 0x7f245019e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43088/0 (socket says 172.21.15.50:43088) 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24558a4640 1 -- 172.21.15.50:0/4083904322 learned_addr learned my addr 172.21.15.50:0/4083904322 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24560a5640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2450198620 0x7f2450198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24550a3640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2450107260 0x7f245019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24558a4640 1 -- 172.21.15.50:0/4083904322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2450198620 msgr2=0x7f2450198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24558a4640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2450198620 0x7f2450198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24558a4640 1 -- 172.21.15.50:0/4083904322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2450107260 msgr2=0x7f245019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24558a4640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2450107260 0x7f245019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24558a4640 1 -- 172.21.15.50:0/4083904322 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2450199140 con 0x7f2450102190 2024-04-23T18:30:48.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24560a5640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2450198620 0x7f2450198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:48.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24550a3640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2450107260 0x7f245019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:30:48.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.485+0000 7f24558a4640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2450102190 0x7f245019e420 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f244c00e9f0 tx=0x7f244c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:48.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.486+0000 7f2446ffd640 1 -- 172.21.15.50:0/4083904322 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c00cd80 con 0x7f2450102190 2024-04-23T18:30:48.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.486+0000 7f2446ffd640 1 -- 172.21.15.50:0/4083904322 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f244c0054d0 con 0x7f2450102190 2024-04-23T18:30:48.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.486+0000 7f2457b08640 1 -- 172.21.15.50:0/4083904322 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2450199430 con 0x7f2450102190 2024-04-23T18:30:48.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.486+0000 7f2446ffd640 1 -- 172.21.15.50:0/4083904322 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c010690 con 0x7f2450102190 2024-04-23T18:30:48.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.486+0000 7f2457b08640 1 -- 172.21.15.50:0/4083904322 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24501a55e0 con 0x7f2450102190 2024-04-23T18:30:48.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.488+0000 7f2446ffd640 1 -- 172.21.15.50:0/4083904322 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f244c004e90 con 0x7f2450102190 2024-04-23T18:30:48.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.488+0000 7f245a81d740 1 Processor -- start 2024-04-23T18:30:48.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f245a81d740 1 -- start start 2024-04-23T18:30:48.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f245a81d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8deb490 0x5615f8deb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f245a81d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8debdb0 0x5615f8df5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f245a81d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8df6480 0x5615f8df8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f245a81d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5615f8d506d0 con 0x5615f8debdb0 2024-04-23T18:30:48.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f245a81d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5615f8ced1a0 con 0x5615f8df6480 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f245a81d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5615f8d4c690 con 0x5615f8deb490 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f2446ffd640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f243c0a3d20 0x7f243c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f24558a4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8debdb0 0x5615f8df5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f24558a4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8debdb0 0x5615f8df5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43094/0 (socket says 172.21.15.50:43094) 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f24558a4640 1 -- 172.21.15.50:0/1105552554 learned_addr learned my addr 172.21.15.50:0/1105552554 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f24560a5640 1 --2- 172.21.15.50:0/1105552554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8deb490 0x5615f8deb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f24560a5640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f243c0a3d20 0x7f243c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f24550a3640 1 --2- 172.21.15.50:0/1105552554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8df6480 0x5615f8df8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f2446ffd640 1 -- 172.21.15.50:0/4083904322 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f244c0d0d70 con 0x7f2450102190 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f24558a4640 1 -- 172.21.15.50:0/1105552554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8deb490 msgr2=0x5615f8deb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f24558a4640 1 --2- 172.21.15.50:0/1105552554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8deb490 0x5615f8deb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f24558a4640 1 -- 172.21.15.50:0/1105552554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8df6480 msgr2=0x5615f8df8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f24558a4640 1 --2- 172.21.15.50:0/1105552554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8df6480 0x5615f8df8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.489+0000 7f24558a4640 1 -- 172.21.15.50:0/1105552554 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615f8cf75b0 con 0x5615f8debdb0 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.490+0000 7f24558a4640 1 --2- 172.21.15.50:0/1105552554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8debdb0 0x5615f8df5f40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f244c0ce450 tx=0x7f244c004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b756e2267829e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.490+0000 7f24560a5640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f243c0a3d20 0x7f243c0a61e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2448009370 tx=0x7f2448009240 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.490+0000 7f243bfff640 1 -- 172.21.15.50:0/1105552554 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c09d400 con 0x5615f8debdb0 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.490+0000 7f243bfff640 1 -- 172.21.15.50:0/1105552554 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f244c09cd00 con 0x5615f8debdb0 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.490+0000 7f243bfff640 1 -- 172.21.15.50:0/1105552554 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c0ce730 con 0x5615f8debdb0 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.490+0000 7f245a81d740 1 -- 172.21.15.50:0/1105552554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8debdb0 msgr2=0x5615f8df5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.490+0000 7f245a81d740 1 --2- 172.21.15.50:0/1105552554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8debdb0 0x5615f8df5f40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f244c0ce450 tx=0x7f244c004a90 comp rx=0 tx=0).stop 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 -- 172.21.15.50:0/1105552554 shutdown_connections 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 --2- 172.21.15.50:0/1105552554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8df6480 0x5615f8df8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 --2- 172.21.15.50:0/1105552554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8debdb0 0x5615f8df5f40 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 --2- 172.21.15.50:0/1105552554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8deb490 0x5615f8deb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 -- 172.21.15.50:0/1105552554 >> 172.21.15.50:0/1105552554 conn(0x5615f8de77b0 msgr2=0x5615f8de7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 -- 172.21.15.50:0/1105552554 shutdown_connections 2024-04-23T18:30:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 -- 172.21.15.50:0/1105552554 wait complete. 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 -- start start 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 Processor -- start 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 0x5615f8df5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8debdb0 0x5615f8df31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8df6480 0x5615f8df38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5615f8df9250 con 0x5615f8deb490 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5615f8ceab20 con 0x5615f8df6480 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5615f8df93d0 con 0x5615f8debdb0 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.491+0000 7f245a81d740 10 client.0 _ll_register_callbacks cb 0x5615f8cea530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24558a4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 0x5615f8df5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24558a4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 0x5615f8df5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43102/0 (socket says 172.21.15.50:43102) 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24558a4640 1 -- 172.21.15.50:0/2175629756 learned_addr learned my addr 172.21.15.50:0/2175629756 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24550a3640 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8debdb0 0x5615f8df31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24558a4640 1 -- 172.21.15.50:0/2175629756 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8debdb0 msgr2=0x5615f8df31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24558a4640 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8debdb0 0x5615f8df31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24558a4640 1 -- 172.21.15.50:0/2175629756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8df6480 msgr2=0x5615f8df38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24560a5640 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8df6480 0x5615f8df38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24558a4640 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8df6480 0x5615f8df38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24558a4640 1 -- 172.21.15.50:0/2175629756 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615f8a62920 con 0x5615f8deb490 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24558a4640 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 0x5615f8df5ec0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f244c0996d0 tx=0x7f244c016fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f492a42a3c6900cb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24397fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24397fa640 1 -- 172.21.15.50:0/2175629756 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c0b5070 con 0x5615f8deb490 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24397fa640 1 -- 172.21.15.50:0/2175629756 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f244c0a54e0 con 0x5615f8deb490 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f245a81d740 1 -- 172.21.15.50:0/2175629756 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f244c09cd00 con 0x5615f8deb490 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24397fa640 1 -- 172.21.15.50:0/2175629756 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f244c098fa0 con 0x5615f8deb490 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f245a81d740 10 client.17004 fetch_fsmap learned FSMap version 47 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f245a81d740 1 -- 172.21.15.50:0/2175629756 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5615f8b2ed30 con 0x5615f8deb490 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.492+0000 7f24397fa640 1 -- 172.21.15.50:0/2175629756 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f244c09cd00 con 0x5615f8deb490 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.493+0000 7f24397fa640 1 -- 172.21.15.50:0/2175629756 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 47) ==== 8358+0+0 (secure 0 0 0) 0x7f244c09c150 con 0x5615f8deb490 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.493+0000 7f245a81d740 10 client.17004 fetch_fsmap finished waiting for FSMap version 47 2024-04-23T18:30:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.493+0000 7f245a81d740 10 client.17004 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.493+0000 7f245a81d740 20 client.17004 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.493+0000 7f245a81d740 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f8e06400 0x5615f8e087f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.493+0000 7f245a81d740 4 client.17004 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.493+0000 7f245a81d740 1 -- 172.21.15.50:0/2175629756 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615f8d0e500 con 0x5615f8e06400 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.493+0000 7f24550a3640 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f8e06400 0x5615f8e087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.493+0000 7f24550a3640 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f8e06400 0x5615f8e087f0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=894016f77849a25f server_cookie=577c846f81c4b711 in_seq=0 out_seq=0 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.493+0000 7f24397fa640 10 client.17004 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f24397fa640 1 -- 172.21.15.50:0/2175629756 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5615f8d0e500 con 0x5615f8e06400 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f24397fa640 10 client.17004 handle_command_reply: tid=0 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f245a81d740 1 client.17004 shutdown 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f245a81d740 1 -- 172.21.15.50:0/2175629756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 msgr2=0x5615f8df5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f245a81d740 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 0x5615f8df5ec0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f244c0996d0 tx=0x7f244c016fc0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f245a81d740 1 -- 172.21.15.50:0/2175629756 shutdown_connections 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f245a81d740 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f8e06400 0x5615f8e087f0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f245a81d740 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8df6480 0x5615f8df38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f245a81d740 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8debdb0 0x5615f8df31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f245a81d740 1 --2- 172.21.15.50:0/2175629756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 0x5615f8df5ec0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f24397fa640 0 client.17004 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:48.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f245a81d740 1 -- 172.21.15.50:0/2175629756 >> 172.21.15.50:0/2175629756 conn(0x5615f8de77b0 msgr2=0x5615f8ded5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:48.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f245a81d740 1 -- 172.21.15.50:0/2175629756 shutdown_connections 2024-04-23T18:30:48.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f245a81d740 1 -- 172.21.15.50:0/2175629756 wait complete. 2024-04-23T18:30:48.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.494+0000 7f245a81d740 20 client.17004 trim_cache size 0 max 16384 2024-04-23T18:30:48.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f245a81d740 1 Processor -- start 2024-04-23T18:30:48.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f245a81d740 1 -- start start 2024-04-23T18:30:48.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f245a81d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8deb490 0x5615f8df20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f245a81d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8debdb0 0x5615f8df59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f245a81d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8df6480 0x5615f8df5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f245a81d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f24501b0650 con 0x5615f8df6480 2024-04-23T18:30:48.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f245a81d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f24501ade60 con 0x5615f8debdb0 2024-04-23T18:30:48.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f245a81d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f24501abcd0 con 0x5615f8deb490 2024-04-23T18:30:48.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f24560a5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8df6480 0x5615f8df5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f24560a5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8df6480 0x5615f8df5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43114/0 (socket says 172.21.15.50:43114) 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f24560a5640 1 -- 172.21.15.50:0/3109176177 learned_addr learned my addr 172.21.15.50:0/3109176177 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f24558a4640 1 --2- 172.21.15.50:0/3109176177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8deb490 0x5615f8df20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f24550a3640 1 --2- 172.21.15.50:0/3109176177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8debdb0 0x5615f8df59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f24560a5640 1 -- 172.21.15.50:0/3109176177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8deb490 msgr2=0x5615f8df20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f24560a5640 1 --2- 172.21.15.50:0/3109176177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8deb490 0x5615f8df20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f24560a5640 1 -- 172.21.15.50:0/3109176177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8debdb0 msgr2=0x5615f8df59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f24560a5640 1 --2- 172.21.15.50:0/3109176177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8debdb0 0x5615f8df59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f24560a5640 1 -- 172.21.15.50:0/3109176177 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615f8df3c60 con 0x5615f8df6480 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.512+0000 7f24550a3640 1 --2- 172.21.15.50:0/3109176177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8debdb0 0x5615f8df59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f24560a5640 1 --2- 172.21.15.50:0/3109176177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8df6480 0x5615f8df5f30 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f2448014660 tx=0x7f2448009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba4e811c86c5b408 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f243b7fe640 1 -- 172.21.15.50:0/3109176177 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2448025070 con 0x5615f8df6480 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f243b7fe640 1 -- 172.21.15.50:0/3109176177 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f244800fe70 con 0x5615f8df6480 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f243b7fe640 1 -- 172.21.15.50:0/3109176177 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24480043e0 con 0x5615f8df6480 2024-04-23T18:30:48.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f245a81d740 1 -- 172.21.15.50:0/3109176177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8df6480 msgr2=0x5615f8df5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f245a81d740 1 --2- 172.21.15.50:0/3109176177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8df6480 0x5615f8df5f30 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f2448014660 tx=0x7f2448009710 comp rx=0 tx=0).stop 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f245a81d740 1 -- 172.21.15.50:0/3109176177 shutdown_connections 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f245a81d740 1 --2- 172.21.15.50:0/3109176177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8df6480 0x5615f8df5f30 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f245a81d740 1 --2- 172.21.15.50:0/3109176177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8debdb0 0x5615f8df59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f245a81d740 1 --2- 172.21.15.50:0/3109176177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8deb490 0x5615f8df20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f245a81d740 1 -- 172.21.15.50:0/3109176177 >> 172.21.15.50:0/3109176177 conn(0x5615f8de77b0 msgr2=0x5615f8e01eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f245a81d740 1 -- 172.21.15.50:0/3109176177 shutdown_connections 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.513+0000 7f245a81d740 1 -- 172.21.15.50:0/3109176177 wait complete. 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f245a81d740 1 -- start start 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f245a81d740 1 Processor -- start 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f245a81d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 0x5615f8e078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f245a81d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8debdb0 0x5615f8e07e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f245a81d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8df6480 0x5615f8e6b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f245a81d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f24501ac320 con 0x5615f8deb490 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f245a81d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f24501ad790 con 0x5615f8debdb0 2024-04-23T18:30:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f245a81d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f24501ac9f0 con 0x5615f8df6480 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f245a81d740 10 client.0 _ll_register_callbacks cb 0x7f244c009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f24558a4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 0x5615f8e078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f24558a4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 0x5615f8e078f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43122/0 (socket says 172.21.15.50:43122) 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f24558a4640 1 -- 172.21.15.50:0/1538437308 learned_addr learned my addr 172.21.15.50:0/1538437308 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f24560a5640 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8df6480 0x5615f8e6b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f24550a3640 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8debdb0 0x5615f8e07e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f24558a4640 1 -- 172.21.15.50:0/1538437308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8df6480 msgr2=0x5615f8e6b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f24558a4640 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8df6480 0x5615f8e6b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f24558a4640 1 -- 172.21.15.50:0/1538437308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8debdb0 msgr2=0x5615f8e07e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f24558a4640 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8debdb0 0x5615f8e07e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f24558a4640 1 -- 172.21.15.50:0/1538437308 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615f8e6bb10 con 0x5615f8deb490 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.514+0000 7f24550a3640 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8debdb0 0x5615f8e07e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f24558a4640 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 0x5615f8e078f0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f244c0a5440 tx=0x7f244c0a57c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e3d1f4b44e2799 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f24397fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:48.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f24397fa640 1 -- 172.21.15.50:0/1538437308 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c0bb060 con 0x5615f8deb490 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f24397fa640 1 -- 172.21.15.50:0/1538437308 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f244c09b000 con 0x5615f8deb490 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f245a81d740 1 -- 172.21.15.50:0/1538437308 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f244800fe70 con 0x5615f8deb490 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f24397fa640 1 -- 172.21.15.50:0/1538437308 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f244c0b5070 con 0x5615f8deb490 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f245a81d740 10 client.17010 fetch_fsmap learned FSMap version 47 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f245a81d740 1 -- 172.21.15.50:0/1538437308 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5615f8e6cd30 con 0x5615f8deb490 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f24397fa640 1 -- 172.21.15.50:0/1538437308 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f244800fe70 con 0x5615f8deb490 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f24397fa640 1 -- 172.21.15.50:0/1538437308 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 47) ==== 8358+0+0 (secure 0 0 0) 0x7f244c0bd0c0 con 0x5615f8deb490 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f245a81d740 10 client.17010 fetch_fsmap finished waiting for FSMap version 47 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f245a81d740 10 client.17010 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f245a81d740 20 client.17010 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f245a81d740 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f8e6d320 0x5615f8e6f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f245a81d740 4 client.17010 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a5988bee", "release": true, "await": true}] 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.515+0000 7f245a81d740 1 -- 172.21.15.50:0/1538437308 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a5988bee", "release": true, "await": true}) -- 0x5615f8d5e0f0 con 0x5615f8e6d320 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.516+0000 7f24550a3640 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f8e6d320 0x5615f8e6f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.516+0000 7f24550a3640 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f8e6d320 0x5615f8e6f740 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbe622f0d46adeb7 server_cookie=d0c5f3ada4ed9f4a in_seq=0 out_seq=0 2024-04-23T18:30:48.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.516+0000 7f24397fa640 10 client.17010 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:48.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f24397fa640 1 -- 172.21.15.50:0/1538437308 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5615f8d5e0f0 con 0x5615f8e6d320 2024-04-23T18:30:48.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f24397fa640 10 client.17010 handle_command_reply: tid=0 2024-04-23T18:30:48.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f245a81d740 1 client.17010 shutdown 2024-04-23T18:30:48.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f245a81d740 1 -- 172.21.15.50:0/1538437308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 msgr2=0x5615f8e078f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f245a81d740 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 0x5615f8e078f0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f244c0a5440 tx=0x7f244c0a57c0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f245a81d740 1 -- 172.21.15.50:0/1538437308 shutdown_connections 2024-04-23T18:30:48.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f245a81d740 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5615f8e6d320 0x5615f8e6f740 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f245a81d740 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5615f8df6480 0x5615f8e6b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f245a81d740 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615f8debdb0 0x5615f8e07e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f245a81d740 1 --2- 172.21.15.50:0/1538437308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5615f8deb490 0x5615f8e078f0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f24397fa640 0 client.17010 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:48.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f245a81d740 1 -- 172.21.15.50:0/1538437308 >> 172.21.15.50:0/1538437308 conn(0x5615f8de77b0 msgr2=0x5615f8decbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:48.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f245a81d740 1 -- 172.21.15.50:0/1538437308 shutdown_connections 2024-04-23T18:30:48.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f245a81d740 1 -- 172.21.15.50:0/1538437308 wait complete. 2024-04-23T18:30:48.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.522+0000 7f245a81d740 20 client.17010 trim_cache size 0 max 16384 2024-04-23T18:30:48.516 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:30:48.516 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 47, 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 40, 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "a5988bee": { 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 40, 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 84.0, 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:30:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:30:48.518 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:30:48.518 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:30:48.518 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:30:48.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:48.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:48.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:48.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:48.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:48.518 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:30:48.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.523+0000 7f2457b08640 1 -- 172.21.15.50:0/4083904322 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f243c0a3d20 msgr2=0x7f243c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.523+0000 7f2457b08640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f243c0a3d20 0x7f243c0a61e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2448009370 tx=0x7f2448009240 comp rx=0 tx=0).stop 2024-04-23T18:30:48.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.523+0000 7f2457b08640 1 -- 172.21.15.50:0/4083904322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2450102190 msgr2=0x7f245019e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.523+0000 7f2457b08640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2450102190 0x7f245019e420 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f244c00e9f0 tx=0x7f244c00edf0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.523+0000 7f2457b08640 1 -- 172.21.15.50:0/4083904322 shutdown_connections 2024-04-23T18:30:48.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.523+0000 7f2457b08640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f243c0a3d20 0x7f243c0a61e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.523+0000 7f2457b08640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2450198620 0x7f2450198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.523+0000 7f2457b08640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2450107260 0x7f245019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.523+0000 7f2457b08640 1 --2- 172.21.15.50:0/4083904322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2450102190 0x7f245019e420 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.523+0000 7f2457b08640 1 -- 172.21.15.50:0/4083904322 >> 172.21.15.50:0/4083904322 conn(0x7f24500793a0 msgr2=0x7f24501000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:48.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.523+0000 7f2457b08640 1 -- 172.21.15.50:0/4083904322 shutdown_connections 2024-04-23T18:30:48.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.523+0000 7f2457b08640 1 -- 172.21.15.50:0/4083904322 wait complete. 2024-04-23T18:30:48.527 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a5988bee', seconds elapsed: 25.1 2024-04-23T18:30:48.527 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.95 2024-04-23T18:30:48.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:30:48.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:48 smithi177 ceph-mon[32308]: pgmap v386: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-04-23T18:30:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.670+0000 7f2fe36e1640 1 -- 172.21.15.50:0/1244585926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc101a50 msgr2=0x7f2fdc105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.670+0000 7f2fe36e1640 1 --2- 172.21.15.50:0/1244585926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc101a50 0x7f2fdc105a90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009900 tx=0x7f2fcc02a410 comp rx=0 tx=0).stop 2024-04-23T18:30:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.670+0000 7f2fe36e1640 1 -- 172.21.15.50:0/1244585926 shutdown_connections 2024-04-23T18:30:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.670+0000 7f2fe36e1640 1 --2- 172.21.15.50:0/1244585926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc106170 0x7f2fdc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.670+0000 7f2fe36e1640 1 --2- 172.21.15.50:0/1244585926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc101a50 0x7f2fdc105a90 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.670+0000 7f2fe36e1640 1 --2- 172.21.15.50:0/1244585926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fdc1010a0 0x7f2fdc101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.670+0000 7f2fe36e1640 1 -- 172.21.15.50:0/1244585926 >> 172.21.15.50:0/1244585926 conn(0x7f2fdc077f50 msgr2=0x7f2fdc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.670+0000 7f2fe36e1640 1 -- 172.21.15.50:0/1244585926 shutdown_connections 2024-04-23T18:30:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.670+0000 7f2fe36e1640 1 -- 172.21.15.50:0/1244585926 wait complete. 2024-04-23T18:30:48.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe36e1640 1 Processor -- start 2024-04-23T18:30:48.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe36e1640 1 -- start start 2024-04-23T18:30:48.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe36e1640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc1010a0 0x7f2fdc1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe36e1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fdc106170 0x7f2fdc197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe36e1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc19bf60 0x7f2fdc19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe36e1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2fdc1110b0 con 0x7f2fdc19bf60 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe36e1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2fdc110f30 con 0x7f2fdc106170 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe36e1640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2fdc111230 con 0x7f2fdc1010a0 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe1c7e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc19bf60 0x7f2fdc19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe1c7e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc19bf60 0x7f2fdc19c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43126/0 (socket says 172.21.15.50:43126) 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe147d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc1010a0 0x7f2fdc1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe1c7e640 1 -- 172.21.15.50:0/650634382 learned_addr learned my addr 172.21.15.50:0/650634382 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.671+0000 7f2fe0c7c640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fdc106170 0x7f2fdc197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.672+0000 7f2fe1c7e640 1 -- 172.21.15.50:0/650634382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc1010a0 msgr2=0x7f2fdc1976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.672+0000 7f2fe1c7e640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc1010a0 0x7f2fdc1976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.672+0000 7f2fe1c7e640 1 -- 172.21.15.50:0/650634382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fdc106170 msgr2=0x7f2fdc197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.672+0000 7f2fe1c7e640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fdc106170 0x7f2fdc197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.672+0000 7f2fe1c7e640 1 -- 172.21.15.50:0/650634382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fdc19ca50 con 0x7f2fdc19bf60 2024-04-23T18:30:48.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.672+0000 7f2fe1c7e640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc19bf60 0x7f2fdc19c410 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400eb10 tx=0x7f2fd400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:48.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.672+0000 7f2fd27fc640 1 -- 172.21.15.50:0/650634382 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4004b20 con 0x7f2fdc19bf60 2024-04-23T18:30:48.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.672+0000 7f2fe36e1640 1 -- 172.21.15.50:0/650634382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fdc19cd40 con 0x7f2fdc19bf60 2024-04-23T18:30:48.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.672+0000 7f2fe36e1640 1 -- 172.21.15.50:0/650634382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fdc069420 con 0x7f2fdc19bf60 2024-04-23T18:30:48.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.672+0000 7f2fd27fc640 1 -- 172.21.15.50:0/650634382 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fd4004cc0 con 0x7f2fdc19bf60 2024-04-23T18:30:48.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.672+0000 7f2fd27fc640 1 -- 172.21.15.50:0/650634382 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4010690 con 0x7f2fdc19bf60 2024-04-23T18:30:48.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.674+0000 7f2fe36e1640 1 -- 172.21.15.50:0/650634382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fc0005290 con 0x7f2fdc19bf60 2024-04-23T18:30:48.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.675+0000 7f2fd27fc640 1 -- 172.21.15.50:0/650634382 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2fd40108f0 con 0x7f2fdc19bf60 2024-04-23T18:30:48.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.676+0000 7f2fd27fc640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2fc80a3da0 0x7f2fc80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:48.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.676+0000 7f2fd27fc640 1 -- 172.21.15.50:0/650634382 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f2fd40d0d40 con 0x7f2fdc19bf60 2024-04-23T18:30:48.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.676+0000 7f2fe147d640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2fc80a3da0 0x7f2fc80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:48.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.677+0000 7f2fe147d640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2fc80a3da0 0x7f2fc80a6260 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009bd0 tx=0x7f2fd8005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:48.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.682+0000 7f2fd27fc640 1 -- 172.21.15.50:0/650634382 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2fd4053ff0 con 0x7f2fdc19bf60 2024-04-23T18:30:48.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.943+0000 7f2fe36e1640 1 -- 172.21.15.50:0/650634382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fc0005580 con 0x7f2fdc19bf60 2024-04-23T18:30:48.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.944+0000 7f2fd27fc640 1 -- 172.21.15.50:0/650634382 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+11882 (secure 0 0 0) 0x7f2fd409d3d0 con 0x7f2fdc19bf60 2024-04-23T18:30:48.937 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:30:48.937 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:30:44.945124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/4033440314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4033440314},{"type":"v1","addr":"172.21.15.177:6837","nonce":4033440314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:30:48.937 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 47 2024-04-23T18:30:48.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.946+0000 7f2fe36e1640 1 -- 172.21.15.50:0/650634382 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2fc80a3da0 msgr2=0x7f2fc80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.946+0000 7f2fe36e1640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2fc80a3da0 0x7f2fc80a6260 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009bd0 tx=0x7f2fd8005f90 comp rx=0 tx=0).stop 2024-04-23T18:30:48.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.946+0000 7f2fe36e1640 1 -- 172.21.15.50:0/650634382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc19bf60 msgr2=0x7f2fdc19c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:48.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.946+0000 7f2fe36e1640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc19bf60 0x7f2fdc19c410 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400eb10 tx=0x7f2fd400c2d0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.946+0000 7f2fe36e1640 1 -- 172.21.15.50:0/650634382 shutdown_connections 2024-04-23T18:30:48.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.946+0000 7f2fe36e1640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2fc80a3da0 0x7f2fc80a6260 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.946+0000 7f2fe36e1640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc19bf60 0x7f2fdc19c410 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.946+0000 7f2fe36e1640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fdc106170 0x7f2fdc197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.946+0000 7f2fe36e1640 1 --2- 172.21.15.50:0/650634382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2fdc1010a0 0x7f2fdc1976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:48.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.946+0000 7f2fe36e1640 1 -- 172.21.15.50:0/650634382 >> 172.21.15.50:0/650634382 conn(0x7f2fdc077f50 msgr2=0x7f2fdc0ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:48.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.947+0000 7f2fe36e1640 1 -- 172.21.15.50:0/650634382 shutdown_connections 2024-04-23T18:30:48.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:48.947+0000 7f2fe36e1640 1 -- 172.21.15.50:0/650634382 wait complete. 2024-04-23T18:30:48.951 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 83.95 --await-for 42.0' 2024-04-23T18:30:48.952 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 83.95 --await-for 42.0 2024-04-23T18:30:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.083+0000 7f3076de8640 1 -- 172.21.15.50:0/3220123626 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307010c580 msgr2=0x7f307010e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.083+0000 7f3076de8640 1 --2- 172.21.15.50:0/3220123626 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307010c580 0x7f307010e9b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f3068009ca0 tx=0x7f306802a950 comp rx=0 tx=0).stop 2024-04-23T18:30:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.083+0000 7f3076de8640 1 -- 172.21.15.50:0/3220123626 shutdown_connections 2024-04-23T18:30:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.083+0000 7f3076de8640 1 --2- 172.21.15.50:0/3220123626 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307010c580 0x7f307010e9b0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.083+0000 7f3076de8640 1 --2- 172.21.15.50:0/3220123626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3070103ad0 0x7f307010bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.083+0000 7f3076de8640 1 --2- 172.21.15.50:0/3220123626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30701031b0 0x7f3070103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.083+0000 7f3076de8640 1 -- 172.21.15.50:0/3220123626 >> 172.21.15.50:0/3220123626 conn(0x7f30700771d0 msgr2=0x7f30700775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.083+0000 7f3076de8640 1 -- 172.21.15.50:0/3220123626 shutdown_connections 2024-04-23T18:30:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.083+0000 7f3076de8640 1 -- 172.21.15.50:0/3220123626 wait complete. 2024-04-23T18:30:49.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3076de8640 1 Processor -- start 2024-04-23T18:30:49.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3076de8640 1 -- start start 2024-04-23T18:30:49.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3076de8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30701031b0 0x7f3070198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3076de8640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3070103ad0 0x7f3070198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3076de8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307010c580 0x7f307019cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3076de8640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3070112180 con 0x7f307010c580 2024-04-23T18:30:49.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3076de8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3070112000 con 0x7f30701031b0 2024-04-23T18:30:49.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3076de8640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3070112300 con 0x7f3070103ad0 2024-04-23T18:30:49.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3075385640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307010c580 0x7f307019cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3074b84640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30701031b0 0x7f3070198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3075385640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307010c580 0x7f307019cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43156/0 (socket says 172.21.15.50:43156) 2024-04-23T18:30:49.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3074b84640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30701031b0 0x7f3070198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:46192/0 (socket says 172.21.15.50:46192) 2024-04-23T18:30:49.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3075385640 1 -- 172.21.15.50:0/1807931101 learned_addr learned my addr 172.21.15.50:0/1807931101 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:49.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f306ffff640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3070103ad0 0x7f3070198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3075385640 1 -- 172.21.15.50:0/1807931101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3070103ad0 msgr2=0x7f3070198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3075385640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3070103ad0 0x7f3070198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3075385640 1 -- 172.21.15.50:0/1807931101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30701031b0 msgr2=0x7f3070198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3075385640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30701031b0 0x7f3070198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3075385640 1 -- 172.21.15.50:0/1807931101 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307019d6f0 con 0x7f307010c580 2024-04-23T18:30:49.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.084+0000 7f3074b84640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30701031b0 0x7f3070198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:49.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.085+0000 7f3075385640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307010c580 0x7f307019cfc0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3068004af0 tx=0x7f3068031f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.085+0000 7f306dffb640 1 -- 172.21.15.50:0/1807931101 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068034040 con 0x7f307010c580 2024-04-23T18:30:49.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.085+0000 7f306dffb640 1 -- 172.21.15.50:0/1807931101 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30680308f0 con 0x7f307010c580 2024-04-23T18:30:49.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.085+0000 7f3076de8640 1 -- 172.21.15.50:0/1807931101 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f307019d980 con 0x7f307010c580 2024-04-23T18:30:49.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.085+0000 7f306dffb640 1 -- 172.21.15.50:0/1807931101 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30680045d0 con 0x7f307010c580 2024-04-23T18:30:49.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.085+0000 7f3076de8640 1 -- 172.21.15.50:0/1807931101 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30701a5270 con 0x7f307010c580 2024-04-23T18:30:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.087+0000 7f306dffb640 1 -- 172.21.15.50:0/1807931101 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3068037020 con 0x7f307010c580 2024-04-23T18:30:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.088+0000 7f3079bd0740 1 Processor -- start 2024-04-23T18:30:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.088+0000 7f3079bd0740 1 -- start start 2024-04-23T18:30:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.088+0000 7f306dffb640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f30600a3d20 0x7f30600a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.088+0000 7f3079bd0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6cc7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.088+0000 7f3079bd0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6cd1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3079bd0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6cd4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3079bd0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cce6c2c370 con 0x55cce6cd2090 2024-04-23T18:30:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3079bd0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cce6bc92e0 con 0x55cce6cc70a0 2024-04-23T18:30:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3079bd0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55cce6c28330 con 0x55cce6cc79c0 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f306ffff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6cd1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f306ffff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6cd1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:54026/0 (socket says 172.21.15.50:54026) 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f306ffff640 1 -- 172.21.15.50:0/3044653392 learned_addr learned my addr 172.21.15.50:0/3044653392 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3074b84640 1 --2- 172.21.15.50:0/3044653392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6cc7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3075385640 1 --2- 172.21.15.50:0/3044653392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6cd4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f306ffff640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f30600a3d20 0x7f30600a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3074b84640 1 -- 172.21.15.50:0/3044653392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 msgr2=0x55cce6cd1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3074b84640 1 --2- 172.21.15.50:0/3044653392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6cd1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3074b84640 1 -- 172.21.15.50:0/3044653392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 msgr2=0x55cce6cd4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3074b84640 1 --2- 172.21.15.50:0/3044653392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6cd4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3074b84640 1 -- 172.21.15.50:0/3044653392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cce6a0ad80 con 0x55cce6cc70a0 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3075385640 1 --2- 172.21.15.50:0/3044653392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6cd4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f306dffb640 1 -- 172.21.15.50:0/1807931101 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f30680f13b0 con 0x7f307010c580 2024-04-23T18:30:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.089+0000 7f3074b84640 1 --2- 172.21.15.50:0/3044653392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6cc7480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3064009fd0 tx=0x7f30640118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f28eb39ba32c4863 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.090+0000 7f306ffff640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f30600a3d20 0x7f30600a61e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55cce6cc8660 tx=0x7f305c015040 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.090+0000 7f305affd640 1 -- 172.21.15.50:0/3044653392 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064004020 con 0x55cce6cc70a0 2024-04-23T18:30:49.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.090+0000 7f305affd640 1 -- 172.21.15.50:0/3044653392 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30640054d0 con 0x55cce6cc70a0 2024-04-23T18:30:49.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.090+0000 7f305affd640 1 -- 172.21.15.50:0/3044653392 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064004bb0 con 0x55cce6cc70a0 2024-04-23T18:30:49.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.091+0000 7f3079bd0740 1 -- 172.21.15.50:0/3044653392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 msgr2=0x55cce6cc7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.091+0000 7f3079bd0740 1 --2- 172.21.15.50:0/3044653392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6cc7480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3064009fd0 tx=0x7f30640118d0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.091+0000 7f3079bd0740 1 -- 172.21.15.50:0/3044653392 shutdown_connections 2024-04-23T18:30:49.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.091+0000 7f3079bd0740 1 --2- 172.21.15.50:0/3044653392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6cd4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.091+0000 7f3079bd0740 1 --2- 172.21.15.50:0/3044653392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6cd1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.091+0000 7f3079bd0740 1 --2- 172.21.15.50:0/3044653392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6cc7480 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.091+0000 7f3079bd0740 1 -- 172.21.15.50:0/3044653392 >> 172.21.15.50:0/3044653392 conn(0x55cce6cc33c0 msgr2=0x55cce6cc37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.091+0000 7f3079bd0740 1 -- 172.21.15.50:0/3044653392 shutdown_connections 2024-04-23T18:30:49.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.091+0000 7f3079bd0740 1 -- 172.21.15.50:0/3044653392 wait complete. 2024-04-23T18:30:49.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.091+0000 7f3079bd0740 1 -- start start 2024-04-23T18:30:49.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.091+0000 7f3079bd0740 1 Processor -- start 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.092+0000 7f3079bd0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6ccef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.092+0000 7f3079bd0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6ccf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.092+0000 7f3079bd0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6ccfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.092+0000 7f3079bd0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55cce6cd4e80 con 0x55cce6cd2090 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.092+0000 7f3079bd0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cce6bc6fb0 con 0x55cce6cc70a0 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.092+0000 7f3079bd0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55cce6cd5000 con 0x55cce6cc79c0 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.092+0000 7f3075385640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6ccfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.092+0000 7f3074b84640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6ccef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.092+0000 7f3075385640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6ccfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43182/0 (socket says 172.21.15.50:43182) 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.092+0000 7f3075385640 1 -- 172.21.15.50:0/3494179540 learned_addr learned my addr 172.21.15.50:0/3494179540 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.092+0000 7f3079bd0740 10 client.0 _ll_register_callbacks cb 0x55cce6bc69a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.092+0000 7f306ffff640 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6ccf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f3075385640 1 -- 172.21.15.50:0/3494179540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 msgr2=0x55cce6ccf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f3075385640 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6ccf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f3075385640 1 -- 172.21.15.50:0/3494179540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 msgr2=0x55cce6ccef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f3075385640 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6ccef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f3075385640 1 -- 172.21.15.50:0/3494179540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cce6a6b510 con 0x55cce6cd2090 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f3074b84640 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6ccef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f3075385640 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6ccfa10 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f30680eecb0 tx=0x7f30680bcf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d1efa83e5389b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f304ffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f304ffff640 1 -- 172.21.15.50:0/3494179540 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306802c040 con 0x55cce6cd2090 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f304ffff640 1 -- 172.21.15.50:0/3494179540 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f30680b9ac0 con 0x55cce6cd2090 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f3079bd0740 1 -- 172.21.15.50:0/3494179540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30640054d0 con 0x55cce6cd2090 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f304ffff640 1 -- 172.21.15.50:0/3494179540 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f3068037310 con 0x55cce6cd2090 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f3079bd0740 10 client.17025 fetch_fsmap learned FSMap version 47 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.093+0000 7f3079bd0740 1 -- 172.21.15.50:0/3494179540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55cce68fe670 con 0x55cce6cd2090 2024-04-23T18:30:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.094+0000 7f304ffff640 1 -- 172.21.15.50:0/3494179540 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30640054d0 con 0x55cce6cd2090 2024-04-23T18:30:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.094+0000 7f304ffff640 1 -- 172.21.15.50:0/3494179540 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 47) ==== 8358+0+0 (secure 0 0 0) 0x7f306804d030 con 0x55cce6cd2090 2024-04-23T18:30:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.094+0000 7f3079bd0740 10 client.17025 fetch_fsmap finished waiting for FSMap version 47 2024-04-23T18:30:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.094+0000 7f3079bd0740 10 client.17025 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:30:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.094+0000 7f3079bd0740 20 client.17025 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.094+0000 7f3079bd0740 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cce6ce23b0 0x55cce6ce4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.094+0000 7f3079bd0740 4 client.17025 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.094+0000 7f3079bd0740 1 -- 172.21.15.50:0/3494179540 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cce6c39d90 con 0x55cce6ce23b0 2024-04-23T18:30:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.094+0000 7f3074b84640 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cce6ce23b0 0x55cce6ce4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.095+0000 7f304ffff640 10 client.17025 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.095+0000 7f3074b84640 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cce6ce23b0 0x55cce6ce4810 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b55a45799db9baa server_cookie=b9e10bfe64337eec in_seq=0 out_seq=0 2024-04-23T18:30:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.095+0000 7f304ffff640 1 -- 172.21.15.50:0/3494179540 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cce6c39d90 con 0x55cce6ce23b0 2024-04-23T18:30:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.095+0000 7f304ffff640 10 client.17025 handle_command_reply: tid=0 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f3079bd0740 1 client.17025 shutdown 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f3079bd0740 1 -- 172.21.15.50:0/3494179540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 msgr2=0x55cce6ccfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f3079bd0740 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6ccfa10 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f30680eecb0 tx=0x7f30680bcf90 comp rx=0 tx=0).stop 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f3079bd0740 1 -- 172.21.15.50:0/3494179540 shutdown_connections 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f3079bd0740 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cce6ce23b0 0x55cce6ce4810 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f304ffff640 0 client.17025 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f3079bd0740 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6ccfa10 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f3079bd0740 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6ccf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f3079bd0740 1 --2- 172.21.15.50:0/3494179540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6ccef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f3079bd0740 1 -- 172.21.15.50:0/3494179540 >> 172.21.15.50:0/3494179540 conn(0x55cce6cc33c0 msgr2=0x55cce6cc91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f3079bd0740 1 -- 172.21.15.50:0/3494179540 shutdown_connections 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f3079bd0740 1 -- 172.21.15.50:0/3494179540 wait complete. 2024-04-23T18:30:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.096+0000 7f3079bd0740 20 client.17025 trim_cache size 0 max 16384 2024-04-23T18:30:49.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.118+0000 7f3079bd0740 1 Processor -- start 2024-04-23T18:30:49.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.118+0000 7f3079bd0740 1 -- start start 2024-04-23T18:30:49.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.118+0000 7f3079bd0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6cddd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.118+0000 7f3079bd0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6cc9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.118+0000 7f3079bd0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6cea120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.118+0000 7f3079bd0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f30701b0470 con 0x55cce6cd2090 2024-04-23T18:30:49.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.118+0000 7f3079bd0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f30701adc80 con 0x55cce6cc70a0 2024-04-23T18:30:49.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.118+0000 7f3079bd0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f30701abaf0 con 0x55cce6cc79c0 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.118+0000 7f3075385640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6cea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.118+0000 7f3075385640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6cea120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43188/0 (socket says 172.21.15.50:43188) 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.118+0000 7f3075385640 1 -- 172.21.15.50:0/2328195293 learned_addr learned my addr 172.21.15.50:0/2328195293 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.118+0000 7f3074b84640 1 --2- 172.21.15.50:0/2328195293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6cddd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.119+0000 7f306ffff640 1 --2- 172.21.15.50:0/2328195293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6cc9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.119+0000 7f3075385640 1 -- 172.21.15.50:0/2328195293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 msgr2=0x55cce6cc9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.119+0000 7f3075385640 1 --2- 172.21.15.50:0/2328195293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6cc9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.119+0000 7f3075385640 1 -- 172.21.15.50:0/2328195293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 msgr2=0x55cce6cddd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.119+0000 7f3075385640 1 --2- 172.21.15.50:0/2328195293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6cddd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.119+0000 7f3075385640 1 -- 172.21.15.50:0/2328195293 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cce6ccdf40 con 0x55cce6cd2090 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.119+0000 7f3075385640 1 --2- 172.21.15.50:0/2328195293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6cea120 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f30680b98b0 tx=0x7f30680095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a27627fb6ca557b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.119+0000 7f305a7fc640 1 -- 172.21.15.50:0/2328195293 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306802c040 con 0x55cce6cd2090 2024-04-23T18:30:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.119+0000 7f305a7fc640 1 -- 172.21.15.50:0/2328195293 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30680098d0 con 0x55cce6cd2090 2024-04-23T18:30:49.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.119+0000 7f305a7fc640 1 -- 172.21.15.50:0/2328195293 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30680dc1e0 con 0x55cce6cd2090 2024-04-23T18:30:49.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.120+0000 7f3079bd0740 1 -- 172.21.15.50:0/2328195293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 msgr2=0x55cce6cea120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.120+0000 7f3079bd0740 1 --2- 172.21.15.50:0/2328195293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6cea120 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f30680b98b0 tx=0x7f30680095c0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.120+0000 7f3079bd0740 1 -- 172.21.15.50:0/2328195293 shutdown_connections 2024-04-23T18:30:49.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.120+0000 7f3079bd0740 1 --2- 172.21.15.50:0/2328195293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cd2090 0x55cce6cea120 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.120+0000 7f3079bd0740 1 --2- 172.21.15.50:0/2328195293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc79c0 0x55cce6cc9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.120+0000 7f3079bd0740 1 --2- 172.21.15.50:0/2328195293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cc70a0 0x55cce6cddd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.120+0000 7f3079bd0740 1 -- 172.21.15.50:0/2328195293 >> 172.21.15.50:0/2328195293 conn(0x55cce6cc33c0 msgr2=0x55cce6cd07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.120+0000 7f3079bd0740 1 -- 172.21.15.50:0/2328195293 shutdown_connections 2024-04-23T18:30:49.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.120+0000 7f3079bd0740 1 -- 172.21.15.50:0/2328195293 wait complete. 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.120+0000 7f3079bd0740 1 -- start start 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.120+0000 7f3079bd0740 1 Processor -- start 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f3079bd0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc70a0 0x55cce6cd60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f3079bd0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cc79c0 0x55cce6cd65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f3079bd0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cd2090 0x55cce6d47990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f3079bd0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f30701ac140 con 0x55cce6cc79c0 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f3079bd0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f30701ad5b0 con 0x55cce6cd2090 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f3079bd0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f30701ac810 con 0x55cce6cc70a0 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f3079bd0740 10 client.0 _ll_register_callbacks cb 0x7f30680b97e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f306ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cc79c0 0x55cce6cd65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f306ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cc79c0 0x55cce6cd65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43194/0 (socket says 172.21.15.50:43194) 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f306ffff640 1 -- 172.21.15.50:0/453092277 learned_addr learned my addr 172.21.15.50:0/453092277 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f3074b84640 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc70a0 0x55cce6cd60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f3075385640 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cd2090 0x55cce6d47990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f306ffff640 1 -- 172.21.15.50:0/453092277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc70a0 msgr2=0x55cce6cd60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f306ffff640 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc70a0 0x55cce6cd60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f306ffff640 1 -- 172.21.15.50:0/453092277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cd2090 msgr2=0x55cce6d47990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f306ffff640 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cd2090 0x55cce6d47990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.121+0000 7f306ffff640 1 -- 172.21.15.50:0/453092277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cce6d47ed0 con 0x55cce6cc79c0 2024-04-23T18:30:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f306ffff640 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cc79c0 0x55cce6cd65e0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f305c0058c0 tx=0x7f305c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab5d959a6cbfa32 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f304ffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f304ffff640 1 -- 172.21.15.50:0/453092277 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c02e060 con 0x55cce6cc79c0 2024-04-23T18:30:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f304ffff640 1 -- 172.21.15.50:0/453092277 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f305c01f800 con 0x55cce6cc79c0 2024-04-23T18:30:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f3079bd0740 1 -- 172.21.15.50:0/453092277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cce6c33c00 con 0x55cce6cc79c0 2024-04-23T18:30:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f304ffff640 1 -- 172.21.15.50:0/453092277 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f305c01fa00 con 0x55cce6cc79c0 2024-04-23T18:30:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f3079bd0740 10 client.17031 fetch_fsmap learned FSMap version 47 2024-04-23T18:30:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f3079bd0740 1 -- 172.21.15.50:0/453092277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55cce6d490a0 con 0x55cce6cc79c0 2024-04-23T18:30:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f304ffff640 1 -- 172.21.15.50:0/453092277 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55cce6c33c00 con 0x55cce6cc79c0 2024-04-23T18:30:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f304ffff640 1 -- 172.21.15.50:0/453092277 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 47) ==== 8358+0+0 (secure 0 0 0) 0x7f305c028030 con 0x55cce6cc79c0 2024-04-23T18:30:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f3079bd0740 10 client.17031 fetch_fsmap finished waiting for FSMap version 47 2024-04-23T18:30:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f3079bd0740 10 client.17031 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:30:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.122+0000 7f3079bd0740 20 client.17031 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.123+0000 7f3079bd0740 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cce6ce23b0 0x55cce6d4b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.123+0000 7f3079bd0740 4 client.17031 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.95, "await_for": 42.0}] 2024-04-23T18:30:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.123+0000 7f3079bd0740 1 -- 172.21.15.50:0/453092277 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.95, "await_for": 42.0}) -- 0x55cce6bafca0 con 0x55cce6ce23b0 2024-04-23T18:30:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.123+0000 7f3074b84640 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cce6ce23b0 0x55cce6d4b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.123+0000 7f304ffff640 10 client.17031 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.123+0000 7f3074b84640 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cce6ce23b0 0x55cce6d4b990 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35b79cbe7c4984f7 server_cookie=1611dfe186bb73f9 in_seq=0 out_seq=0 2024-04-23T18:30:49.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f304ffff640 1 -- 172.21.15.50:0/453092277 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55cce6bafca0 con 0x55cce6ce23b0 2024-04-23T18:30:49.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f304ffff640 10 client.17031 handle_command_reply: tid=0 2024-04-23T18:30:49.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f3079bd0740 1 client.17031 shutdown 2024-04-23T18:30:49.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f3079bd0740 1 -- 172.21.15.50:0/453092277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cc79c0 msgr2=0x55cce6cd65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f3079bd0740 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cc79c0 0x55cce6cd65e0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f305c0058c0 tx=0x7f305c004730 comp rx=0 tx=0).stop 2024-04-23T18:30:49.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f3079bd0740 1 -- 172.21.15.50:0/453092277 shutdown_connections 2024-04-23T18:30:49.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f3079bd0740 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55cce6ce23b0 0x55cce6d4b990 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f3079bd0740 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cce6cd2090 0x55cce6d47990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f3079bd0740 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55cce6cc79c0 0x55cce6cd65e0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f3079bd0740 1 --2- 172.21.15.50:0/453092277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55cce6cc70a0 0x55cce6cd60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f304ffff640 0 client.17031 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:49.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f3079bd0740 1 -- 172.21.15.50:0/453092277 >> 172.21.15.50:0/453092277 conn(0x55cce6cc33c0 msgr2=0x55cce6cd1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f3079bd0740 1 -- 172.21.15.50:0/453092277 shutdown_connections 2024-04-23T18:30:49.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f3079bd0740 1 -- 172.21.15.50:0/453092277 wait complete. 2024-04-23T18:30:49.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.204+0000 7f3079bd0740 20 client.17031 trim_cache size 0 max 16384 2024-04-23T18:30:49.198 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:30:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 47, 2024-04-23T18:30:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:30:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 42, 2024-04-23T18:30:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "d0f64ea1": { 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 42, 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 84.0, 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:30:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:30:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:30:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:30:49.200 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:30:49.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.205+0000 7f3076de8640 1 -- 172.21.15.50:0/1807931101 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f30600a3d20 msgr2=0x7f30600a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.205+0000 7f3076de8640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f30600a3d20 0x7f30600a61e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55cce6cc8660 tx=0x7f305c015040 comp rx=0 tx=0).stop 2024-04-23T18:30:49.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.205+0000 7f3076de8640 1 -- 172.21.15.50:0/1807931101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307010c580 msgr2=0x7f307019cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:49.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.205+0000 7f3076de8640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307010c580 0x7f307019cfc0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3068004af0 tx=0x7f3068031f50 comp rx=0 tx=0).stop 2024-04-23T18:30:49.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.205+0000 7f3076de8640 1 -- 172.21.15.50:0/1807931101 shutdown_connections 2024-04-23T18:30:49.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.205+0000 7f3076de8640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f30600a3d20 0x7f30600a61e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.205+0000 7f3076de8640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307010c580 0x7f307019cfc0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.205+0000 7f3076de8640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3070103ad0 0x7f3070198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.205+0000 7f3076de8640 1 --2- 172.21.15.50:0/1807931101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30701031b0 0x7f3070198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:49.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.205+0000 7f3076de8640 1 -- 172.21.15.50:0/1807931101 >> 172.21.15.50:0/1807931101 conn(0x7f30700771d0 msgr2=0x7f30700fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:49.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.205+0000 7f3076de8640 1 -- 172.21.15.50:0/1807931101 shutdown_connections 2024-04-23T18:30:49.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:30:49.205+0000 7f3076de8640 1 -- 172.21.15.50:0/1807931101 wait complete. 2024-04-23T18:30:49.211 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:49 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3623826505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:49.212 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:49 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:30:49.212 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:49 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/650634382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:49.213 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'd0f64ea1', quiesce took 0.7 seconds. Will release after: 23.95 2024-04-23T18:30:49.236 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.26 MB/sec execute 563 sec latency 24513.719 ms 2024-04-23T18:30:49.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:49 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3623826505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:49.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:49 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:30:49.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:49 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/650634382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:49.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:49 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3623826505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:49.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:49 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:30:49.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:49 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/650634382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:30:50.236 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.25 MB/sec execute 564 sec latency 1087.830 ms 2024-04-23T18:30:50.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:50 smithi050 ceph-mon[29187]: pgmap v387: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-23T18:30:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:50 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:50 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:50 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:50 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:50 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:30:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:50 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:30:50.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:50 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:50 smithi177 ceph-mon[32308]: pgmap v387: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-23T18:30:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:50 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:50 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:50 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:50 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:50 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:30:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:50 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:30:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:50 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:50 smithi188 ceph-mon[32694]: pgmap v387: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-23T18:30:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:50 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:50 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:50 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:50 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:50 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:30:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:50 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:30:50.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:50 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:30:51.236 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.24 MB/sec execute 565 sec latency 2087.936 ms 2024-04-23T18:30:52.236 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.23 MB/sec execute 566 sec latency 3087.996 ms 2024-04-23T18:30:52.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:52 smithi050 ceph-mon[29187]: pgmap v388: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-23T18:30:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:52 smithi177 ceph-mon[32308]: pgmap v388: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-23T18:30:52.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:52 smithi188 ceph-mon[32694]: pgmap v388: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-23T18:30:53.236 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.22 MB/sec execute 567 sec latency 4088.092 ms 2024-04-23T18:30:54.237 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.21 MB/sec execute 568 sec latency 5088.161 ms 2024-04-23T18:30:54.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:54 smithi050 ceph-mon[29187]: pgmap v389: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-04-23T18:30:54.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:54 smithi177 ceph-mon[32308]: pgmap v389: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-04-23T18:30:54.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:54 smithi188 ceph-mon[32694]: pgmap v389: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-04-23T18:30:55.237 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.20 MB/sec execute 569 sec latency 6088.509 ms 2024-04-23T18:30:56.237 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.19 MB/sec execute 570 sec latency 7088.641 ms 2024-04-23T18:30:56.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:56 smithi050 ceph-mon[29187]: pgmap v390: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 67 op/s 2024-04-23T18:30:56.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:56 smithi177 ceph-mon[32308]: pgmap v390: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 67 op/s 2024-04-23T18:30:56.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:56 smithi188 ceph-mon[32694]: pgmap v390: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 67 op/s 2024-04-23T18:30:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.008+0000 7f792cfe5640 1 -- 172.21.15.50:0/399876340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79281021c0 msgr2=0x7f79281025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.008+0000 7f792cfe5640 1 --2- 172.21.15.50:0/399876340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79281021c0 0x7f79281025a0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f7918009900 tx=0x7f791802a410 comp rx=0 tx=0).stop 2024-04-23T18:30:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.008+0000 7f792cfe5640 1 -- 172.21.15.50:0/399876340 shutdown_connections 2024-04-23T18:30:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.008+0000 7f792cfe5640 1 --2- 172.21.15.50:0/399876340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7928107290 0x7f792810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.008+0000 7f792cfe5640 1 --2- 172.21.15.50:0/399876340 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7928102b70 0x7f7928106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.008+0000 7f792cfe5640 1 --2- 172.21.15.50:0/399876340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79281021c0 0x7f79281025a0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.008+0000 7f792cfe5640 1 -- 172.21.15.50:0/399876340 >> 172.21.15.50:0/399876340 conn(0x7f79280fda20 msgr2=0x7f79280ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.008+0000 7f792cfe5640 1 -- 172.21.15.50:0/399876340 shutdown_connections 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.008+0000 7f792cfe5640 1 -- 172.21.15.50:0/399876340 wait complete. 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.009+0000 7f792cfe5640 1 Processor -- start 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.009+0000 7f792cfe5640 1 -- start start 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.009+0000 7f792cfe5640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79281021c0 0x7f7928198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.009+0000 7f792cfe5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7928107290 0x7f7928198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f792cfe5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f792819cf40 0x7f792819d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f792cfe5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7928112240 con 0x7f792819cf40 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f792cfe5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f79281120c0 con 0x7f7928107290 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f792cfe5640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f79281123c0 con 0x7f79281021c0 2024-04-23T18:30:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f7926d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f792819cf40 0x7f792819d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f7926d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f792819cf40 0x7f792819d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33584/0 (socket says 172.21.15.50:33584) 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f7926d9d640 1 -- 172.21.15.50:0/2459719495 learned_addr learned my addr 172.21.15.50:0/2459719495 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f792659c640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79281021c0 0x7f7928198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f791ffff640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7928107290 0x7f7928198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f7926d9d640 1 -- 172.21.15.50:0/2459719495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79281021c0 msgr2=0x7f7928198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f7926d9d640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79281021c0 0x7f7928198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f7926d9d640 1 -- 172.21.15.50:0/2459719495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7928107290 msgr2=0x7f7928198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f7926d9d640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7928107290 0x7f7928198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f7926d9d640 1 -- 172.21.15.50:0/2459719495 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792819da30 con 0x7f792819cf40 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f7926d9d640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f792819cf40 0x7f792819d3f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f792000cce0 tx=0x7f792000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f791f7fe640 1 -- 172.21.15.50:0/2459719495 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792000f450 con 0x7f792819cf40 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f791f7fe640 1 -- 172.21.15.50:0/2459719495 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f792000faa0 con 0x7f792819cf40 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f792cfe5640 1 -- 172.21.15.50:0/2459719495 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f792819dd20 con 0x7f792819cf40 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f791f7fe640 1 -- 172.21.15.50:0/2459719495 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7920018b70 con 0x7f792819cf40 2024-04-23T18:30:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.010+0000 7f792cfe5640 1 -- 172.21.15.50:0/2459719495 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79281a54c0 con 0x7f792819cf40 2024-04-23T18:30:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.013+0000 7f791f7fe640 1 -- 172.21.15.50:0/2459719495 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7920018d10 con 0x7f792819cf40 2024-04-23T18:30:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.013+0000 7f792fdd9740 1 Processor -- start 2024-04-23T18:30:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f792fdd9740 1 -- start start 2024-04-23T18:30:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f792fdd9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045d1540 0x557b045d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f792fdd9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1e60 0x557b045e06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f792fdd9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 0x557b045e3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f792fdd9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557b04536970 con 0x557b045e0c10 2024-04-23T18:30:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f792fdd9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557b044d34a0 con 0x557b045d1e60 2024-04-23T18:30:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f792fdd9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557b04532930 con 0x557b045d1540 2024-04-23T18:30:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f791f7fe640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f79140a3dc0 0x7f79140a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f791ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 0x557b045e3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f791ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 0x557b045e3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33594/0 (socket says 172.21.15.50:33594) 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f791ffff640 1 -- 172.21.15.50:0/3142873781 learned_addr learned my addr 172.21.15.50:0/3142873781 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f791ffff640 1 --2- 172.21.15.50:0/3142873781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045d1540 0x557b045d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f792659c640 1 --2- 172.21.15.50:0/3142873781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1e60 0x557b045e06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f7926d9d640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f79140a3dc0 0x7f79140a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f791f7fe640 1 -- 172.21.15.50:0/2459719495 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f79200d0c60 con 0x7f792819cf40 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f791ffff640 1 -- 172.21.15.50:0/3142873781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045d1540 msgr2=0x557b045d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f791ffff640 1 --2- 172.21.15.50:0/3142873781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045d1540 0x557b045d1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f791ffff640 1 -- 172.21.15.50:0/3142873781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1e60 msgr2=0x557b045e06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f791ffff640 1 --2- 172.21.15.50:0/3142873781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1e60 0x557b045e06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.014+0000 7f791ffff640 1 -- 172.21.15.50:0/3142873781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b04249210 con 0x557b045e0c10 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.015+0000 7f791ffff640 1 --2- 172.21.15.50:0/3142873781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 0x557b045e3000 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f7910014990 tx=0x7f791000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1bbd9cbe0d8969 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.015+0000 7f7926d9d640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f79140a3dc0 0x7f79140a6280 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f792009c890 tx=0x7f792006f680 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.015+0000 7f791cff9640 1 -- 172.21.15.50:0/3142873781 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79100142d0 con 0x557b045e0c10 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.015+0000 7f791cff9640 1 -- 172.21.15.50:0/3142873781 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79100054d0 con 0x557b045e0c10 2024-04-23T18:30:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.015+0000 7f791cff9640 1 -- 172.21.15.50:0/3142873781 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7910004d40 con 0x557b045e0c10 2024-04-23T18:30:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.016+0000 7f792fdd9740 1 -- 172.21.15.50:0/3142873781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 msgr2=0x557b045e3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.016+0000 7f792fdd9740 1 --2- 172.21.15.50:0/3142873781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 0x557b045e3000 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f7910014990 tx=0x7f791000c490 comp rx=0 tx=0).stop 2024-04-23T18:30:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.016+0000 7f792fdd9740 1 -- 172.21.15.50:0/3142873781 shutdown_connections 2024-04-23T18:30:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.016+0000 7f792fdd9740 1 --2- 172.21.15.50:0/3142873781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 0x557b045e3000 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.016+0000 7f792fdd9740 1 --2- 172.21.15.50:0/3142873781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1e60 0x557b045e06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.016+0000 7f792fdd9740 1 --2- 172.21.15.50:0/3142873781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045d1540 0x557b045d1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.016+0000 7f792fdd9740 1 -- 172.21.15.50:0/3142873781 >> 172.21.15.50:0/3142873781 conn(0x557b045cdb30 msgr2=0x557b045cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.016+0000 7f792fdd9740 1 -- 172.21.15.50:0/3142873781 shutdown_connections 2024-04-23T18:30:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.016+0000 7f792fdd9740 1 -- 172.21.15.50:0/3142873781 wait complete. 2024-04-23T18:30:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.016+0000 7f792fdd9740 1 -- start start 2024-04-23T18:30:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.016+0000 7f792fdd9740 1 Processor -- start 2024-04-23T18:30:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792fdd9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1540 0x557b045e05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792fdd9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 0x557b045dd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792fdd9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045e0c10 0x557b045dde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792fdd9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557b045e3840 con 0x557b045d1e60 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792fdd9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557b044d10b0 con 0x557b045d1540 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792fdd9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557b045e39c0 con 0x557b045e0c10 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792fdd9740 10 client.0 _ll_register_callbacks cb 0x557b044bed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 0x557b045dd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 0x557b045dd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33604/0 (socket says 172.21.15.50:33604) 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792659c640 1 -- 172.21.15.50:0/713722187 learned_addr learned my addr 172.21.15.50:0/713722187 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f791ffff640 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1540 0x557b045e05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f791ffff640 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045e0c10 0x557b045dde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792659c640 1 -- 172.21.15.50:0/713722187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045e0c10 msgr2=0x557b045dde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792659c640 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045e0c10 0x557b045dde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792659c640 1 -- 172.21.15.50:0/713722187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1540 msgr2=0x557b045e05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792659c640 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1540 0x557b045e05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792659c640 1 -- 172.21.15.50:0/713722187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b04315620 con 0x557b045d1e60 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792659c640 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 0x557b045dd8d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7918009230 tx=0x7f7918005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70a8a774de9f5fd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f790dffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f790dffb640 1 -- 172.21.15.50:0/713722187 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918035040 con 0x557b045d1e60 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f790dffb640 1 -- 172.21.15.50:0/713722187 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f7918004770 con 0x557b045d1e60 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.017+0000 7f792fdd9740 1 -- 172.21.15.50:0/713722187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79100054d0 con 0x557b045d1e60 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.018+0000 7f790dffb640 1 -- 172.21.15.50:0/713722187 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f7918004970 con 0x557b045d1e60 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.018+0000 7f792fdd9740 10 client.17043 fetch_fsmap learned FSMap version 47 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.018+0000 7f792fdd9740 1 -- 172.21.15.50:0/713722187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x557b045ef7d0 con 0x557b045d1e60 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.018+0000 7f790dffb640 1 -- 172.21.15.50:0/713722187 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79100054d0 con 0x557b045d1e60 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.018+0000 7f790dffb640 1 -- 172.21.15.50:0/713722187 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 47) ==== 8358+0+0 (secure 0 0 0) 0x7f79180314a0 con 0x557b045d1e60 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.018+0000 7f792fdd9740 10 client.17043 fetch_fsmap finished waiting for FSMap version 47 2024-04-23T18:30:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.018+0000 7f792fdd9740 10 client.17043 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.018+0000 7f792fdd9740 20 client.17043 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.018+0000 7f792fdd9740 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557b045f0c90 0x557b045f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.018+0000 7f792fdd9740 4 client.17043 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.018+0000 7f792fdd9740 1 -- 172.21.15.50:0/713722187 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b04544390 con 0x557b045f0c90 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.018+0000 7f791ffff640 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557b045f0c90 0x557b045f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.019+0000 7f791ffff640 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557b045f0c90 0x557b045f30c0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5a9ab5055b87e4d server_cookie=92cdb1d2b1768079 in_seq=0 out_seq=0 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.019+0000 7f790dffb640 10 client.17043 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.019+0000 7f790dffb640 1 -- 172.21.15.50:0/713722187 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b04544390 con 0x557b045f0c90 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.019+0000 7f790dffb640 10 client.17043 handle_command_reply: tid=0 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.019+0000 7f792fdd9740 1 client.17043 shutdown 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.020+0000 7f792fdd9740 1 -- 172.21.15.50:0/713722187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 msgr2=0x557b045dd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.020+0000 7f792fdd9740 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 0x557b045dd8d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7918009230 tx=0x7f7918005040 comp rx=0 tx=0).stop 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.020+0000 7f792fdd9740 1 -- 172.21.15.50:0/713722187 shutdown_connections 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.020+0000 7f792fdd9740 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557b045f0c90 0x557b045f30c0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.020+0000 7f792fdd9740 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045e0c10 0x557b045dde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.020+0000 7f792fdd9740 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 0x557b045dd8d0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.020+0000 7f790dffb640 0 client.17043 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.020+0000 7f792fdd9740 1 --2- 172.21.15.50:0/713722187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1540 0x557b045e05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.020+0000 7f792fdd9740 1 -- 172.21.15.50:0/713722187 >> 172.21.15.50:0/713722187 conn(0x557b045cdb30 msgr2=0x557b045d39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.020+0000 7f792fdd9740 1 -- 172.21.15.50:0/713722187 shutdown_connections 2024-04-23T18:30:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.020+0000 7f792fdd9740 1 -- 172.21.15.50:0/713722187 wait complete. 2024-04-23T18:30:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.020+0000 7f792fdd9740 20 client.17043 trim_cache size 0 max 16384 2024-04-23T18:30:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792fdd9740 1 Processor -- start 2024-04-23T18:30:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792fdd9740 1 -- start start 2024-04-23T18:30:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792fdd9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1540 0x557b045ec700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792fdd9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 0x557b045f89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792fdd9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045e0c10 0x557b045f8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792fdd9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f79281ae220 con 0x557b045d1e60 2024-04-23T18:30:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792fdd9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f79281aba30 con 0x557b045d1540 2024-04-23T18:30:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792fdd9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f79281a98a0 con 0x557b045e0c10 2024-04-23T18:30:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 0x557b045f89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 0x557b045f89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33612/0 (socket says 172.21.15.50:33612) 2024-04-23T18:30:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792659c640 1 -- 172.21.15.50:0/1619675572 learned_addr learned my addr 172.21.15.50:0/1619675572 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f791ffff640 1 --2- 172.21.15.50:0/1619675572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045e0c10 0x557b045f8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792659c640 1 -- 172.21.15.50:0/1619675572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045e0c10 msgr2=0x557b045f8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.035+0000 7f792659c640 1 --2- 172.21.15.50:0/1619675572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045e0c10 0x557b045f8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f792659c640 1 -- 172.21.15.50:0/1619675572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1540 msgr2=0x557b045ec700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f792659c640 1 --2- 172.21.15.50:0/1619675572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1540 0x557b045ec700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f792659c640 1 -- 172.21.15.50:0/1619675572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b045dc8a0 con 0x557b045d1e60 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f791ffff640 1 --2- 172.21.15.50:0/1619675572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045e0c10 0x557b045f8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f792659c640 1 --2- 172.21.15.50:0/1619675572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 0x557b045f89d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f7918031870 tx=0x7f7918004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e322771acaecb847 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f790ffff640 1 -- 172.21.15.50:0/1619675572 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918035070 con 0x557b045d1e60 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f790ffff640 1 -- 172.21.15.50:0/1619675572 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7918041070 con 0x557b045d1e60 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f790ffff640 1 -- 172.21.15.50:0/1619675572 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918053040 con 0x557b045d1e60 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f792fdd9740 1 -- 172.21.15.50:0/1619675572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 msgr2=0x557b045f89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f792fdd9740 1 --2- 172.21.15.50:0/1619675572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 0x557b045f89d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f7918031870 tx=0x7f7918004610 comp rx=0 tx=0).stop 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f792fdd9740 1 -- 172.21.15.50:0/1619675572 shutdown_connections 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f792fdd9740 1 --2- 172.21.15.50:0/1619675572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045e0c10 0x557b045f8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f792fdd9740 1 --2- 172.21.15.50:0/1619675572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045d1e60 0x557b045f89d0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.036+0000 7f792fdd9740 1 --2- 172.21.15.50:0/1619675572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1540 0x557b045ec700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792fdd9740 1 -- 172.21.15.50:0/1619675572 >> 172.21.15.50:0/1619675572 conn(0x557b045cdb30 msgr2=0x557b045d8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792fdd9740 1 -- 172.21.15.50:0/1619675572 shutdown_connections 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792fdd9740 1 -- 172.21.15.50:0/1619675572 wait complete. 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792fdd9740 1 -- start start 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792fdd9740 1 Processor -- start 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792fdd9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045d1540 0x557b045dacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792fdd9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1e60 0x557b045db200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792fdd9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 0x557b04656240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792fdd9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f79281a9ef0 con 0x557b045e0c10 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792fdd9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f79281ab360 con 0x557b045d1e60 2024-04-23T18:30:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792fdd9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f79281aa5c0 con 0x557b045d1540 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792fdd9740 10 client.0 _ll_register_callbacks cb 0x7f79180042c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f791ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 0x557b04656240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f791ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 0x557b04656240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33618/0 (socket says 172.21.15.50:33618) 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f791ffff640 1 -- 172.21.15.50:0/4125904712 learned_addr learned my addr 172.21.15.50:0/4125904712 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.037+0000 7f792659c640 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1e60 0x557b045db200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f791ffff640 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045d1540 0x557b045dacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f791ffff640 1 -- 172.21.15.50:0/4125904712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045d1540 msgr2=0x557b045dacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f791ffff640 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045d1540 0x557b045dacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f791ffff640 1 -- 172.21.15.50:0/4125904712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1e60 msgr2=0x557b045db200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f791ffff640 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1e60 0x557b045db200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f791ffff640 1 -- 172.21.15.50:0/4125904712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b045de200 con 0x557b045e0c10 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f791ffff640 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 0x557b04656240 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7910014990 tx=0x7f7910023040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d235a6ecb7f0ef9f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f790dffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:30:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f790dffb640 1 -- 172.21.15.50:0/4125904712 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7910014110 con 0x557b045e0c10 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f790dffb640 1 -- 172.21.15.50:0/4125904712 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8542+0+0 (secure 0 0 0) 0x7f7910004640 con 0x557b045e0c10 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f792fdd9740 1 -- 172.21.15.50:0/4125904712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7918041070 con 0x557b045e0c10 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f790dffb640 1 -- 172.21.15.50:0/4125904712 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f79100142b0 con 0x557b045e0c10 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f792fdd9740 10 client.17049 fetch_fsmap learned FSMap version 47 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f792fdd9740 1 -- 172.21.15.50:0/4125904712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x557b04657800 con 0x557b045e0c10 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.038+0000 7f790dffb640 1 -- 172.21.15.50:0/4125904712 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7918041070 con 0x557b045e0c10 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.039+0000 7f790dffb640 1 -- 172.21.15.50:0/4125904712 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 47) ==== 8358+0+0 (secure 0 0 0) 0x7f791000f730 con 0x557b045e0c10 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.039+0000 7f792fdd9740 10 client.17049 fetch_fsmap finished waiting for FSMap version 47 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.039+0000 7f792fdd9740 10 client.17049 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.039+0000 7f792fdd9740 20 client.17049 populate_metadata read hostname 'smithi050' 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.039+0000 7f792fdd9740 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557b04657dc0 0x557b0465a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.039+0000 7f792fdd9740 4 client.17049 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:30:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.039+0000 7f792fdd9740 1 -- 172.21.15.50:0/4125904712 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557b0453e200 con 0x557b04657dc0 2024-04-23T18:30:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.039+0000 7f792659c640 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557b04657dc0 0x557b0465a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:30:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.039+0000 7f792659c640 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557b04657dc0 0x557b0465a1e0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8da04e61f876737 server_cookie=f2be7411a0764cb0 in_seq=0 out_seq=0 2024-04-23T18:30:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.039+0000 7f790dffb640 10 client.17049 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f790dffb640 1 -- 172.21.15.50:0/4125904712 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557b0453e200 con 0x557b04657dc0 2024-04-23T18:30:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f790dffb640 10 client.17049 handle_command_reply: tid=0 2024-04-23T18:30:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f792fdd9740 1 client.17049 shutdown 2024-04-23T18:30:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f792fdd9740 1 -- 172.21.15.50:0/4125904712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 msgr2=0x557b04656240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f792fdd9740 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 0x557b04656240 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7910014990 tx=0x7f7910023040 comp rx=0 tx=0).stop 2024-04-23T18:30:57.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f792fdd9740 1 -- 172.21.15.50:0/4125904712 shutdown_connections 2024-04-23T18:30:57.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f792fdd9740 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557b04657dc0 0x557b0465a1e0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f792fdd9740 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557b045e0c10 0x557b04656240 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f792fdd9740 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b045d1e60 0x557b045db200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f790dffb640 0 client.17049 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:30:57.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f792fdd9740 1 --2- 172.21.15.50:0/4125904712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557b045d1540 0x557b045dacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f792fdd9740 1 -- 172.21.15.50:0/4125904712 >> 172.21.15.50:0/4125904712 conn(0x557b045cdb30 msgr2=0x557b045cc420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:57.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f792fdd9740 1 -- 172.21.15.50:0/4125904712 shutdown_connections 2024-04-23T18:30:57.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f792fdd9740 1 -- 172.21.15.50:0/4125904712 wait complete. 2024-04-23T18:30:57.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.073+0000 7f792fdd9740 20 client.17049 trim_cache size 0 max 16384 2024-04-23T18:30:57.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:30:57.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:30:57.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:30:57.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:30:57.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.074+0000 7f792cfe5640 1 -- 172.21.15.50:0/2459719495 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f79140a3dc0 msgr2=0x7f79140a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:57.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.074+0000 7f792cfe5640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f79140a3dc0 0x7f79140a6280 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f792009c890 tx=0x7f792006f680 comp rx=0 tx=0).stop 2024-04-23T18:30:57.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.074+0000 7f792cfe5640 1 -- 172.21.15.50:0/2459719495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f792819cf40 msgr2=0x7f792819d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:30:57.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.074+0000 7f792cfe5640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f792819cf40 0x7f792819d3f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f792000cce0 tx=0x7f792000a620 comp rx=0 tx=0).stop 2024-04-23T18:30:57.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.074+0000 7f792cfe5640 1 -- 172.21.15.50:0/2459719495 shutdown_connections 2024-04-23T18:30:57.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.074+0000 7f792cfe5640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f79140a3dc0 0x7f79140a6280 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.074+0000 7f792cfe5640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f792819cf40 0x7f792819d3f0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.074+0000 7f792cfe5640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7928107290 0x7f7928198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.074+0000 7f792cfe5640 1 --2- 172.21.15.50:0/2459719495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79281021c0 0x7f7928198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:30:57.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.074+0000 7f792cfe5640 1 -- 172.21.15.50:0/2459719495 >> 172.21.15.50:0/2459719495 conn(0x7f79280fda20 msgr2=0x7f79280fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:30:57.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.074+0000 7f792cfe5640 1 -- 172.21.15.50:0/2459719495 shutdown_connections 2024-04-23T18:30:57.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:30:57.074+0000 7f792cfe5640 1 -- 172.21.15.50:0/2459719495 wait complete. 2024-04-23T18:30:57.237 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.18 MB/sec execute 571 sec latency 8088.769 ms 2024-04-23T18:30:58.237 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.17 MB/sec execute 572 sec latency 9088.898 ms 2024-04-23T18:30:58.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:30:58 smithi050 ceph-mon[29187]: pgmap v391: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-04-23T18:30:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:30:58 smithi177 ceph-mon[32308]: pgmap v391: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-04-23T18:30:58.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:30:58 smithi188 ceph-mon[32694]: pgmap v391: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-04-23T18:30:59.237 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.17 MB/sec execute 573 sec latency 10089.002 ms 2024-04-23T18:31:00.237 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.16 MB/sec execute 574 sec latency 11089.055 ms 2024-04-23T18:31:00.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:00 smithi050 ceph-mon[29187]: pgmap v392: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.3 MiB/s wr, 63 op/s 2024-04-23T18:31:00.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:00 smithi177 ceph-mon[32308]: pgmap v392: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.3 MiB/s wr, 63 op/s 2024-04-23T18:31:00.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:00 smithi188 ceph-mon[32694]: pgmap v392: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.3 MiB/s wr, 63 op/s 2024-04-23T18:31:01.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.15 MB/sec execute 575 sec latency 12089.177 ms 2024-04-23T18:31:02.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.14 MB/sec execute 576 sec latency 13089.300 ms 2024-04-23T18:31:02.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:02 smithi177 ceph-mon[32308]: pgmap v393: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.3 MiB/s wr, 63 op/s 2024-04-23T18:31:02.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:02 smithi188 ceph-mon[32694]: pgmap v393: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.3 MiB/s wr, 63 op/s 2024-04-23T18:31:02.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:02 smithi050 ceph-mon[29187]: pgmap v393: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.3 MiB/s wr, 63 op/s 2024-04-23T18:31:03.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.13 MB/sec execute 577 sec latency 14089.396 ms 2024-04-23T18:31:04.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.12 MB/sec execute 578 sec latency 15089.490 ms 2024-04-23T18:31:04.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:04 smithi050 ceph-mon[29187]: pgmap v394: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-23T18:31:04.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:04 smithi177 ceph-mon[32308]: pgmap v394: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-23T18:31:04.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:04 smithi188 ceph-mon[32694]: pgmap v394: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-23T18:31:05.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.11 MB/sec execute 579 sec latency 16089.582 ms 2024-04-23T18:31:06.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.10 MB/sec execute 580 sec latency 17089.634 ms 2024-04-23T18:31:06.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:06 smithi177 ceph-mon[32308]: pgmap v395: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 775 KiB/s wr, 57 op/s 2024-04-23T18:31:06.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:06 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:31:06.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:06 smithi188 ceph-mon[32694]: pgmap v395: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 775 KiB/s wr, 57 op/s 2024-04-23T18:31:06.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:06 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:31:06.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:06 smithi050 ceph-mon[29187]: pgmap v395: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 775 KiB/s wr, 57 op/s 2024-04-23T18:31:06.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:06 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:31:07.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.09 MB/sec execute 581 sec latency 18089.723 ms 2024-04-23T18:31:08.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.09 MB/sec execute 582 sec latency 19089.780 ms 2024-04-23T18:31:08.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:08 smithi177 ceph-mon[32308]: pgmap v396: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 785 KiB/s wr, 60 op/s 2024-04-23T18:31:08.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:08 smithi188 ceph-mon[32694]: pgmap v396: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 785 KiB/s wr, 60 op/s 2024-04-23T18:31:08.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:08 smithi050 ceph-mon[29187]: pgmap v396: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 785 KiB/s wr, 60 op/s 2024-04-23T18:31:09.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.08 MB/sec execute 583 sec latency 20089.831 ms 2024-04-23T18:31:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.172+0000 7f2646544640 1 -- 172.21.15.50:0/1022710700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2640101aa0 msgr2=0x7f2640105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.172+0000 7f2646544640 1 --2- 172.21.15.50:0/1022710700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2640101aa0 0x7f2640105a90 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f2634009900 tx=0x7f263402a410 comp rx=0 tx=0).stop 2024-04-23T18:31:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.172+0000 7f2646544640 1 -- 172.21.15.50:0/1022710700 shutdown_connections 2024-04-23T18:31:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.172+0000 7f2646544640 1 --2- 172.21.15.50:0/1022710700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2640106170 0x7f264010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.172+0000 7f2646544640 1 --2- 172.21.15.50:0/1022710700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2640101aa0 0x7f2640105a90 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.172+0000 7f2646544640 1 --2- 172.21.15.50:0/1022710700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26401010f0 0x7f26401014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.172+0000 7f2646544640 1 -- 172.21.15.50:0/1022710700 >> 172.21.15.50:0/1022710700 conn(0x7f2640077fd0 msgr2=0x7f26400783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.172+0000 7f2646544640 1 -- 172.21.15.50:0/1022710700 shutdown_connections 2024-04-23T18:31:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.172+0000 7f2646544640 1 -- 172.21.15.50:0/1022710700 wait complete. 2024-04-23T18:31:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f2646544640 1 Processor -- start 2024-04-23T18:31:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f2646544640 1 -- start start 2024-04-23T18:31:10.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f2646544640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26401010f0 0x7f2640199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f2646544640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2640106170 0x7f2640199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f2646544640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2640199fe0 0x7f26401a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f2646544640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2640111090 con 0x7f26401010f0 2024-04-23T18:31:10.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f2646544640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2640110f10 con 0x7f2640106170 2024-04-23T18:31:10.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f2646544640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2640111210 con 0x7f2640199fe0 2024-04-23T18:31:10.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f263ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26401010f0 0x7f2640199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f263ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26401010f0 0x7f2640199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50300/0 (socket says 172.21.15.50:50300) 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f263ffff640 1 -- 172.21.15.50:0/446207077 learned_addr learned my addr 172.21.15.50:0/446207077 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f263f7fe640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2640106170 0x7f2640199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.173+0000 7f2644ae1640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2640199fe0 0x7f26401a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.174+0000 7f263ffff640 1 -- 172.21.15.50:0/446207077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2640199fe0 msgr2=0x7f26401a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.174+0000 7f263ffff640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2640199fe0 0x7f26401a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.174+0000 7f263ffff640 1 -- 172.21.15.50:0/446207077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2640106170 msgr2=0x7f2640199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.174+0000 7f263ffff640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2640106170 0x7f2640199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.174+0000 7f263ffff640 1 -- 172.21.15.50:0/446207077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26401a4d90 con 0x7f26401010f0 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.174+0000 7f263ffff640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26401010f0 0x7f2640199510 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f263000d950 tx=0x7f263000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.174+0000 7f263d7fa640 1 -- 172.21.15.50:0/446207077 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2630017030 con 0x7f26401010f0 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.174+0000 7f263d7fa640 1 -- 172.21.15.50:0/446207077 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2630004020 con 0x7f26401010f0 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.174+0000 7f263d7fa640 1 -- 172.21.15.50:0/446207077 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26300047a0 con 0x7f26401010f0 2024-04-23T18:31:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.174+0000 7f2646544640 1 -- 172.21.15.50:0/446207077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26401a5080 con 0x7f26401010f0 2024-04-23T18:31:10.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.174+0000 7f2646544640 1 -- 172.21.15.50:0/446207077 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26401a55a0 con 0x7f26401010f0 2024-04-23T18:31:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.176+0000 7f263d7fa640 1 -- 172.21.15.50:0/446207077 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2630009a30 con 0x7f26401010f0 2024-04-23T18:31:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f2649265740 1 Processor -- start 2024-04-23T18:31:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f2649265740 1 -- start start 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f2649265740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5a620 0x5654d5f5aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f2649265740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 0x5654d5f652c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f2649265740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f65800 0x5654d5f67bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f2649265740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5654d5ebf970 con 0x5654d5f5af40 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f2649265740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5654d5e5c4a0 con 0x5654d5f65800 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f2649265740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5654d5ebb930 con 0x5654d5f5a620 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f263d7fa640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f26280a3dc0 0x7f26280a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f263ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 0x5654d5f652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f2644ae1640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f26280a3dc0 0x7f26280a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f263ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 0x5654d5f652c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50314/0 (socket says 172.21.15.50:50314) 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f263f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f65800 0x5654d5f67bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f263ffff640 1 -- 172.21.15.50:0/3953535567 learned_addr learned my addr 172.21.15.50:0/3953535567 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.177+0000 7f2644ae1640 1 --2- 172.21.15.50:0/3953535567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5a620 0x5654d5f5aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.178+0000 7f263d7fa640 1 -- 172.21.15.50:0/446207077 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8586+0+0 (secure 0 0 0) 0x7f26300d0210 con 0x7f26401010f0 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.178+0000 7f263f7fe640 1 -- 172.21.15.50:0/3953535567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5a620 msgr2=0x5654d5f5aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.178+0000 7f263f7fe640 1 --2- 172.21.15.50:0/3953535567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5a620 0x5654d5f5aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.178+0000 7f263f7fe640 1 -- 172.21.15.50:0/3953535567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 msgr2=0x5654d5f652c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.178+0000 7f263f7fe640 1 --2- 172.21.15.50:0/3953535567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 0x5654d5f652c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.178+0000 7f263f7fe640 1 -- 172.21.15.50:0/3953535567 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654d5bd2210 con 0x5654d5f65800 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.178+0000 7f2644ae1640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f26280a3dc0 0x7f26280a6280 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f26380098b0 tx=0x7f2638009d90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.178+0000 7f263f7fe640 1 --2- 172.21.15.50:0/3953535567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f65800 0x5654d5f67bf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2634006010 tx=0x7f2634005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=512eebbe7a33fa70 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.178+0000 7f26267fc640 1 -- 172.21.15.50:0/3953535567 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634040070 con 0x5654d5f65800 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.178+0000 7f26267fc640 1 -- 172.21.15.50:0/3953535567 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2634004030 con 0x5654d5f65800 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.178+0000 7f26267fc640 1 -- 172.21.15.50:0/3953535567 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634035040 con 0x5654d5f65800 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.179+0000 7f2649265740 1 -- 172.21.15.50:0/3953535567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f65800 msgr2=0x5654d5f67bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.179+0000 7f2649265740 1 --2- 172.21.15.50:0/3953535567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f65800 0x5654d5f67bf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2634006010 tx=0x7f2634005c00 comp rx=0 tx=0).stop 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.179+0000 7f2649265740 1 -- 172.21.15.50:0/3953535567 shutdown_connections 2024-04-23T18:31:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.179+0000 7f2649265740 1 --2- 172.21.15.50:0/3953535567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f65800 0x5654d5f67bf0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.179+0000 7f2649265740 1 --2- 172.21.15.50:0/3953535567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 0x5654d5f652c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.179+0000 7f2649265740 1 --2- 172.21.15.50:0/3953535567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5a620 0x5654d5f5aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.179+0000 7f2649265740 1 -- 172.21.15.50:0/3953535567 >> 172.21.15.50:0/3953535567 conn(0x5654d5f56b30 msgr2=0x5654d5f56f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.179+0000 7f2649265740 1 -- 172.21.15.50:0/3953535567 shutdown_connections 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.179+0000 7f2649265740 1 -- 172.21.15.50:0/3953535567 wait complete. 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.179+0000 7f2649265740 1 -- start start 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f2649265740 1 Processor -- start 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f2649265740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5a620 0x5654d5f651c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f2649265740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f5af40 0x5654d5f624b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f2649265740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f65800 0x5654d5f62ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f2649265740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5654d5f683d0 con 0x5654d5f5a620 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f2649265740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5654d5e5a0b0 con 0x5654d5f5af40 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f2649265740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5654d5f68550 con 0x5654d5f65800 2024-04-23T18:31:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f2649265740 10 client.0 _ll_register_callbacks cb 0x5654d5e47d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f263ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5a620 0x5654d5f651c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f263ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5a620 0x5654d5f651c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50322/0 (socket says 172.21.15.50:50322) 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f263ffff640 1 -- 172.21.15.50:0/1954800903 learned_addr learned my addr 172.21.15.50:0/1954800903 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f263f7fe640 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f5af40 0x5654d5f624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f2644ae1640 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f65800 0x5654d5f62ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f263ffff640 1 -- 172.21.15.50:0/1954800903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f65800 msgr2=0x5654d5f62ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.180+0000 7f263ffff640 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f65800 0x5654d5f62ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f263ffff640 1 -- 172.21.15.50:0/1954800903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f5af40 msgr2=0x5654d5f624b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f263ffff640 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f5af40 0x5654d5f624b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f263ffff640 1 -- 172.21.15.50:0/1954800903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654d5c9e620 con 0x5654d5f5a620 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f263f7fe640 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f5af40 0x5654d5f624b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f263ffff640 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5a620 0x5654d5f651c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f26300cd9d0 tx=0x7f263006dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d53ca619ad3273 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f261f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f261f7fe640 1 -- 172.21.15.50:0/1954800903 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26300a3ae0 con 0x5654d5f5a620 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f261f7fe640 1 -- 172.21.15.50:0/1954800903 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8586+0+0 (secure 0 0 0) 0x7f26300cdba0 con 0x5654d5f5a620 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f2649265740 1 -- 172.21.15.50:0/1954800903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2634004030 con 0x5654d5f5a620 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f261f7fe640 1 -- 172.21.15.50:0/1954800903 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f263009c870 con 0x5654d5f5a620 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f2649265740 10 client.17061 fetch_fsmap learned FSMap version 48 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f2649265740 1 -- 172.21.15.50:0/1954800903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5654d5f74360 con 0x5654d5f5a620 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.181+0000 7f261f7fe640 1 -- 172.21.15.50:0/1954800903 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2634004030 con 0x5654d5f5a620 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.182+0000 7f261f7fe640 1 -- 172.21.15.50:0/1954800903 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7f2630096ce0 con 0x5654d5f5a620 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.182+0000 7f2649265740 10 client.17061 fetch_fsmap finished waiting for FSMap version 48 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.182+0000 7f2649265740 10 client.17061 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.182+0000 7f2649265740 20 client.17061 populate_metadata read hostname 'smithi050' 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.182+0000 7f2649265740 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5654d5f75840 0x5654d5f77c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.182+0000 7f2649265740 4 client.17061 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.182+0000 7f2649265740 1 -- 172.21.15.50:0/1954800903 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654d5ecd390 con 0x5654d5f75840 2024-04-23T18:31:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.182+0000 7f263f7fe640 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5654d5f75840 0x5654d5f77c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.182+0000 7f263f7fe640 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5654d5f75840 0x5654d5f77c90 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=357f90eabe08df1b server_cookie=d4c06a6bf616eeb in_seq=0 out_seq=0 2024-04-23T18:31:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.182+0000 7f261f7fe640 10 client.17061 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f261f7fe640 1 -- 172.21.15.50:0/1954800903 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5654d5ecd390 con 0x5654d5f75840 2024-04-23T18:31:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f261f7fe640 10 client.17061 handle_command_reply: tid=0 2024-04-23T18:31:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f2649265740 1 client.17061 shutdown 2024-04-23T18:31:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f2649265740 1 -- 172.21.15.50:0/1954800903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5a620 msgr2=0x5654d5f651c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f2649265740 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5a620 0x5654d5f651c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f26300cd9d0 tx=0x7f263006dbb0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f2649265740 1 -- 172.21.15.50:0/1954800903 shutdown_connections 2024-04-23T18:31:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f2649265740 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5654d5f75840 0x5654d5f77c90 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f2649265740 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f65800 0x5654d5f62ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f2649265740 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f5af40 0x5654d5f624b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f261f7fe640 0 client.17061 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f2649265740 1 --2- 172.21.15.50:0/1954800903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5a620 0x5654d5f651c0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f2649265740 1 -- 172.21.15.50:0/1954800903 >> 172.21.15.50:0/1954800903 conn(0x5654d5f56b30 msgr2=0x5654d5f5c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f2649265740 1 -- 172.21.15.50:0/1954800903 shutdown_connections 2024-04-23T18:31:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f2649265740 1 -- 172.21.15.50:0/1954800903 wait complete. 2024-04-23T18:31:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.183+0000 7f2649265740 20 client.17061 trim_cache size 0 max 16384 2024-04-23T18:31:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.198+0000 7f2649265740 1 Processor -- start 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.198+0000 7f2649265740 1 -- start start 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.198+0000 7f2649265740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f5a620 0x5654d5f71290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.198+0000 7f2649265740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5af40 0x5654d5f61670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2649265740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f65800 0x5654d5f55b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2649265740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f26401b0620 con 0x5654d5f65800 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2649265740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f26401ade30 con 0x5654d5f5a620 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2649265740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f26401abca0 con 0x5654d5f5af40 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2644ae1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f65800 0x5654d5f55b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2644ae1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f65800 0x5654d5f55b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50332/0 (socket says 172.21.15.50:50332) 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2644ae1640 1 -- 172.21.15.50:0/3446335364 learned_addr learned my addr 172.21.15.50:0/3446335364 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f263ffff640 1 --2- 172.21.15.50:0/3446335364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f5a620 0x5654d5f71290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f263f7fe640 1 --2- 172.21.15.50:0/3446335364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5af40 0x5654d5f61670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2644ae1640 1 -- 172.21.15.50:0/3446335364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5af40 msgr2=0x5654d5f61670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2644ae1640 1 --2- 172.21.15.50:0/3446335364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5af40 0x5654d5f61670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2644ae1640 1 -- 172.21.15.50:0/3446335364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f5a620 msgr2=0x5654d5f71290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2644ae1640 1 --2- 172.21.15.50:0/3446335364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f5a620 0x5654d5f71290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2644ae1640 1 -- 172.21.15.50:0/3446335364 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654d5f60a60 con 0x5654d5f65800 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f263ffff640 1 --2- 172.21.15.50:0/3446335364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f5a620 0x5654d5f71290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2644ae1640 1 --2- 172.21.15.50:0/3446335364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f65800 0x5654d5f55b30 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f2638010b70 tx=0x7f2638004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9520f88ffabbee23 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2625ffb640 1 -- 172.21.15.50:0/3446335364 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2638004b20 con 0x5654d5f65800 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2625ffb640 1 -- 172.21.15.50:0/3446335364 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2638004cc0 con 0x5654d5f65800 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.199+0000 7f2625ffb640 1 -- 172.21.15.50:0/3446335364 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2638005610 con 0x5654d5f65800 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.200+0000 7f2649265740 1 -- 172.21.15.50:0/3446335364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f65800 msgr2=0x5654d5f55b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.200+0000 7f2649265740 1 --2- 172.21.15.50:0/3446335364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f65800 0x5654d5f55b30 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f2638010b70 tx=0x7f2638004ab0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.200+0000 7f2649265740 1 -- 172.21.15.50:0/3446335364 shutdown_connections 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.200+0000 7f2649265740 1 --2- 172.21.15.50:0/3446335364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f65800 0x5654d5f55b30 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.200+0000 7f2649265740 1 --2- 172.21.15.50:0/3446335364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5af40 0x5654d5f61670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.200+0000 7f2649265740 1 --2- 172.21.15.50:0/3446335364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f5a620 0x5654d5f71290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.200+0000 7f2649265740 1 -- 172.21.15.50:0/3446335364 >> 172.21.15.50:0/3446335364 conn(0x5654d5f56b30 msgr2=0x5654d5f55420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.200+0000 7f2649265740 1 -- 172.21.15.50:0/3446335364 shutdown_connections 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.200+0000 7f2649265740 1 -- 172.21.15.50:0/3446335364 wait complete. 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.200+0000 7f2649265740 1 -- start start 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f2649265740 1 Processor -- start 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f2649265740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5a620 0x5654d5f5f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f2649265740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 0x5654d5f5f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f2649265740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f65800 0x5654d5fd7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f2649265740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f26401ac2f0 con 0x5654d5f5af40 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f2649265740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f26401ad760 con 0x5654d5f65800 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f2649265740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f26401ac9c0 con 0x5654d5f5a620 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f2649265740 10 client.0 _ll_register_callbacks cb 0x7f26300d00f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f263f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 0x5654d5f5f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f263f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 0x5654d5f5f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50340/0 (socket says 172.21.15.50:50340) 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f263f7fe640 1 -- 172.21.15.50:0/1021860596 learned_addr learned my addr 172.21.15.50:0/1021860596 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f263ffff640 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5a620 0x5654d5f5f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f2644ae1640 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f65800 0x5654d5fd7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f263f7fe640 1 -- 172.21.15.50:0/1021860596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5a620 msgr2=0x5654d5f5f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f263f7fe640 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5a620 0x5654d5f5f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f263f7fe640 1 -- 172.21.15.50:0/1021860596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f65800 msgr2=0x5654d5fd7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f263f7fe640 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f65800 0x5654d5fd7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f263f7fe640 1 -- 172.21.15.50:0/1021860596 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654d5fd76a0 con 0x5654d5f5af40 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.201+0000 7f263ffff640 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5a620 0x5654d5f5f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f263f7fe640 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 0x5654d5f5f950 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f26340096c0 tx=0x7f26340092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae50f579d1072343 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f261f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:31:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f261f7fe640 1 -- 172.21.15.50:0/1021860596 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634040040 con 0x5654d5f5af40 2024-04-23T18:31:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f261f7fe640 1 -- 172.21.15.50:0/1021860596 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8586+0+0 (secure 0 0 0) 0x7f2634004460 con 0x5654d5f5af40 2024-04-23T18:31:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f2649265740 1 -- 172.21.15.50:0/1021860596 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2638004cc0 con 0x5654d5f5af40 2024-04-23T18:31:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f261f7fe640 1 -- 172.21.15.50:0/1021860596 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f263405f050 con 0x5654d5f5af40 2024-04-23T18:31:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f2649265740 10 client.17067 fetch_fsmap learned FSMap version 48 2024-04-23T18:31:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f2649265740 1 -- 172.21.15.50:0/1021860596 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5654d5fdfc50 con 0x5654d5f5af40 2024-04-23T18:31:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f261f7fe640 1 -- 172.21.15.50:0/1021860596 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2638004cc0 con 0x5654d5f5af40 2024-04-23T18:31:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f261f7fe640 1 -- 172.21.15.50:0/1021860596 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7f263404e070 con 0x5654d5f5af40 2024-04-23T18:31:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f2649265740 10 client.17067 fetch_fsmap finished waiting for FSMap version 48 2024-04-23T18:31:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f2649265740 10 client.17067 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:31:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f2649265740 20 client.17067 populate_metadata read hostname 'smithi050' 2024-04-23T18:31:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f2649265740 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5654d5fe00f0 0x5654d5fe24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f2649265740 4 client.17067 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:31:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.202+0000 7f2649265740 1 -- 172.21.15.50:0/1021860596 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5654d5ec7200 con 0x5654d5fe00f0 2024-04-23T18:31:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.203+0000 7f263ffff640 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5654d5fe00f0 0x5654d5fe24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.203+0000 7f263ffff640 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5654d5fe00f0 0x5654d5fe24e0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=170bebe6c329bf11 server_cookie=42694f7f1620426e in_seq=0 out_seq=0 2024-04-23T18:31:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.203+0000 7f261f7fe640 10 client.17067 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.225+0000 7f261f7fe640 1 -- 172.21.15.50:0/1021860596 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5654d5ec7200 con 0x5654d5fe00f0 2024-04-23T18:31:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.225+0000 7f261f7fe640 10 client.17067 handle_command_reply: tid=0 2024-04-23T18:31:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.225+0000 7f2649265740 1 client.17067 shutdown 2024-04-23T18:31:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2649265740 1 -- 172.21.15.50:0/1021860596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 msgr2=0x5654d5f5f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2649265740 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 0x5654d5f5f950 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f26340096c0 tx=0x7f26340092d0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2649265740 1 -- 172.21.15.50:0/1021860596 shutdown_connections 2024-04-23T18:31:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2649265740 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5654d5fe00f0 0x5654d5fe24e0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2649265740 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654d5f65800 0x5654d5fd7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2649265740 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5654d5f5af40 0x5654d5f5f950 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f261f7fe640 0 client.17067 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2649265740 1 --2- 172.21.15.50:0/1021860596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5654d5f5a620 0x5654d5f5f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2649265740 1 -- 172.21.15.50:0/1021860596 >> 172.21.15.50:0/1021860596 conn(0x5654d5f56b30 msgr2=0x5654d5f55cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2649265740 1 -- 172.21.15.50:0/1021860596 shutdown_connections 2024-04-23T18:31:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2649265740 1 -- 172.21.15.50:0/1021860596 wait complete. 2024-04-23T18:31:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2649265740 20 client.17067 trim_cache size 0 max 16384 2024-04-23T18:31:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:31:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:31:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:31:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:31:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2646544640 1 -- 172.21.15.50:0/446207077 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f26280a3dc0 msgr2=0x7f26280a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2646544640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f26280a3dc0 0x7f26280a6280 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f26380098b0 tx=0x7f2638009d90 comp rx=0 tx=0).stop 2024-04-23T18:31:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2646544640 1 -- 172.21.15.50:0/446207077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26401010f0 msgr2=0x7f2640199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.226+0000 7f2646544640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26401010f0 0x7f2640199510 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f263000d950 tx=0x7f263000de10 comp rx=0 tx=0).stop 2024-04-23T18:31:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.227+0000 7f2646544640 1 -- 172.21.15.50:0/446207077 shutdown_connections 2024-04-23T18:31:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.227+0000 7f2646544640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f26280a3dc0 0x7f26280a6280 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.227+0000 7f2646544640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2640199fe0 0x7f26401a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.227+0000 7f2646544640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2640106170 0x7f2640199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.227+0000 7f2646544640 1 --2- 172.21.15.50:0/446207077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26401010f0 0x7f2640199510 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.227+0000 7f2646544640 1 -- 172.21.15.50:0/446207077 >> 172.21.15.50:0/446207077 conn(0x7f2640077fd0 msgr2=0x7f26400ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.227+0000 7f2646544640 1 -- 172.21.15.50:0/446207077 shutdown_connections 2024-04-23T18:31:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:10.227+0000 7f2646544640 1 -- 172.21.15.50:0/446207077 wait complete. 2024-04-23T18:31:10.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.07 MB/sec execute 584 sec latency 21089.914 ms 2024-04-23T18:31:10.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:10 smithi177 ceph-mon[32308]: pgmap v397: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 89 KiB/s wr, 35 op/s 2024-04-23T18:31:10.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:10 smithi177 ceph-mon[32308]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T18:31:10.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:10 smithi188 ceph-mon[32694]: pgmap v397: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 89 KiB/s wr, 35 op/s 2024-04-23T18:31:10.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:10 smithi188 ceph-mon[32694]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T18:31:10.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:10 smithi050 ceph-mon[29187]: pgmap v397: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 89 KiB/s wr, 35 op/s 2024-04-23T18:31:10.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:10 smithi050 ceph-mon[29187]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T18:31:11.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.06 MB/sec execute 585 sec latency 22090.029 ms 2024-04-23T18:31:12.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.05 MB/sec execute 586 sec latency 23090.075 ms 2024-04-23T18:31:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:12 smithi177 ceph-mon[32308]: pgmap v399: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 107 KiB/s wr, 42 op/s 2024-04-23T18:31:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:12 smithi188 ceph-mon[32694]: pgmap v399: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 107 KiB/s wr, 42 op/s 2024-04-23T18:31:12.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:12 smithi050 ceph-mon[29187]: pgmap v399: 97 pgs: 97 active+clean; 235 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 107 KiB/s wr, 42 op/s 2024-04-23T18:31:13.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:31:13.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 740986 5.04 MB/sec execute 587 sec latency 24090.126 ms 2024-04-23T18:31:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.289+0000 7f65b7267640 1 -- 172.21.15.50:0/2642676637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65b01021b0 msgr2=0x7f65b0102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.289+0000 7f65b7267640 1 --2- 172.21.15.50:0/2642676637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65b01021b0 0x7f65b0102590 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f65ac009900 tx=0x7f65ac02a410 comp rx=0 tx=0).stop 2024-04-23T18:31:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.289+0000 7f65b7267640 1 -- 172.21.15.50:0/2642676637 shutdown_connections 2024-04-23T18:31:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.289+0000 7f65b7267640 1 --2- 172.21.15.50:0/2642676637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b0107280 0x7f65b010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.289+0000 7f65b7267640 1 --2- 172.21.15.50:0/2642676637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65b0102b60 0x7f65b0106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.289+0000 7f65b7267640 1 --2- 172.21.15.50:0/2642676637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65b01021b0 0x7f65b0102590 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.289+0000 7f65b7267640 1 -- 172.21.15.50:0/2642676637 >> 172.21.15.50:0/2642676637 conn(0x7f65b00fd9f0 msgr2=0x7f65b00ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.289+0000 7f65b7267640 1 -- 172.21.15.50:0/2642676637 shutdown_connections 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.289+0000 7f65b7267640 1 -- 172.21.15.50:0/2642676637 wait complete. 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.290+0000 7f65b7267640 1 Processor -- start 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.290+0000 7f65b7267640 1 -- start start 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.290+0000 7f65b7267640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65b01021b0 0x7f65b019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.290+0000 7f65b7267640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b0107280 0x7f65b019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.290+0000 7f65b7267640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65b0198600 0x7f65b0198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.290+0000 7f65b7267640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f65b01121e0 con 0x7f65b01021b0 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.290+0000 7f65b7267640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f65b0112060 con 0x7f65b0107280 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.290+0000 7f65b7267640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f65b0112360 con 0x7f65b0198600 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.290+0000 7f65b5003640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65b01021b0 0x7f65b019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.290+0000 7f65b5003640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65b01021b0 0x7f65b019e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41422/0 (socket says 172.21.15.50:41422) 2024-04-23T18:31:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.290+0000 7f65b5003640 1 -- 172.21.15.50:0/4167191097 learned_addr learned my addr 172.21.15.50:0/4167191097 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.290+0000 7f65b5804640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65b0198600 0x7f65b0198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65b4802640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b0107280 0x7f65b019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65b5003640 1 -- 172.21.15.50:0/4167191097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65b0198600 msgr2=0x7f65b0198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65b5003640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65b0198600 0x7f65b0198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65b5003640 1 -- 172.21.15.50:0/4167191097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b0107280 msgr2=0x7f65b019e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65b5003640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b0107280 0x7f65b019e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65b5003640 1 -- 172.21.15.50:0/4167191097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b0199120 con 0x7f65b01021b0 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65b4802640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b0107280 0x7f65b019e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65b5003640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65b01021b0 0x7f65b019e400 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f65ac005940 tx=0x7f65ac004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65a67fc640 1 -- 172.21.15.50:0/4167191097 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac03f030 con 0x7f65b01021b0 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65b7267640 1 -- 172.21.15.50:0/4167191097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b01993b0 con 0x7f65b01021b0 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65a67fc640 1 -- 172.21.15.50:0/4167191097 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65ac003aa0 con 0x7f65b01021b0 2024-04-23T18:31:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65a67fc640 1 -- 172.21.15.50:0/4167191097 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac035070 con 0x7f65b01021b0 2024-04-23T18:31:13.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.291+0000 7f65b7267640 1 -- 172.21.15.50:0/4167191097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b01a55b0 con 0x7f65b01021b0 2024-04-23T18:31:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.292+0000 7f65b7267640 1 -- 172.21.15.50:0/4167191097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b0068d70 con 0x7f65b01021b0 2024-04-23T18:31:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.293+0000 7f65a67fc640 1 -- 172.21.15.50:0/4167191097 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f65ac042030 con 0x7f65b01021b0 2024-04-23T18:31:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.294+0000 7f65a67fc640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f659c0a3d20 0x7f659c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.295+0000 7f65b4802640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f659c0a3d20 0x7f659c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.295+0000 7f65a67fc640 1 -- 172.21.15.50:0/4167191097 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8586+0+0 (secure 0 0 0) 0x7f65ac0f0f20 con 0x7f65b01021b0 2024-04-23T18:31:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.295+0000 7f65b4802640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f659c0a3d20 0x7f659c0a61e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f65b00fdf10 tx=0x7f65a0005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:13.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.299+0000 7f65a67fc640 1 -- 172.21.15.50:0/4167191097 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f65ac0bd580 con 0x7f65b01021b0 2024-04-23T18:31:13.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.529+0000 7f65b7267640 1 -- 172.21.15.50:0/4167191097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65b010a510 con 0x7f65b01021b0 2024-04-23T18:31:13.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.530+0000 7f65a67fc640 1 -- 172.21.15.50:0/4167191097 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11881 (secure 0 0 0) 0x7f65ac0bce80 con 0x7f65b01021b0 2024-04-23T18:31:13.523 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:31:13.523 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:31:04.945499+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/4033440314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4033440314},{"type":"v1","addr":"172.21.15.177:6837","nonce":4033440314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:31:13.524 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 48 2024-04-23T18:31:13.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.532+0000 7f65b7267640 1 -- 172.21.15.50:0/4167191097 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f659c0a3d20 msgr2=0x7f659c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.532+0000 7f65b7267640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f659c0a3d20 0x7f659c0a61e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f65b00fdf10 tx=0x7f65a0005f90 comp rx=0 tx=0).stop 2024-04-23T18:31:13.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.532+0000 7f65b7267640 1 -- 172.21.15.50:0/4167191097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65b01021b0 msgr2=0x7f65b019e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.532+0000 7f65b7267640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65b01021b0 0x7f65b019e400 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f65ac005940 tx=0x7f65ac004270 comp rx=0 tx=0).stop 2024-04-23T18:31:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.532+0000 7f65b7267640 1 -- 172.21.15.50:0/4167191097 shutdown_connections 2024-04-23T18:31:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.532+0000 7f65b7267640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f659c0a3d20 0x7f659c0a61e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.532+0000 7f65b7267640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65b0198600 0x7f65b0198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.532+0000 7f65b7267640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65b0107280 0x7f65b019e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.532+0000 7f65b7267640 1 --2- 172.21.15.50:0/4167191097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65b01021b0 0x7f65b019e400 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.532+0000 7f65b7267640 1 -- 172.21.15.50:0/4167191097 >> 172.21.15.50:0/4167191097 conn(0x7f65b00fd9f0 msgr2=0x7f65b00fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.532+0000 7f65b7267640 1 -- 172.21.15.50:0/4167191097 shutdown_connections 2024-04-23T18:31:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.532+0000 7f65b7267640 1 -- 172.21.15.50:0/4167191097 wait complete. 2024-04-23T18:31:13.536 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id d0f64ea1 --release --await' 2024-04-23T18:31:13.536 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id d0f64ea1 --release --await 2024-04-23T18:31:13.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.651+0000 7fc6b4b87640 1 -- 172.21.15.50:0/1925928049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b0101a40 msgr2=0x7fc6b0105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.651+0000 7fc6b4b87640 1 --2- 172.21.15.50:0/1925928049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b0101a40 0x7fc6b0105a80 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009900 tx=0x7fc6a402a410 comp rx=0 tx=0).stop 2024-04-23T18:31:13.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.652+0000 7fc6b4b87640 1 -- 172.21.15.50:0/1925928049 shutdown_connections 2024-04-23T18:31:13.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.652+0000 7fc6b4b87640 1 --2- 172.21.15.50:0/1925928049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b0106160 0x7fc6b010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.652+0000 7fc6b4b87640 1 --2- 172.21.15.50:0/1925928049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b0101a40 0x7fc6b0105a80 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.652+0000 7fc6b4b87640 1 --2- 172.21.15.50:0/1925928049 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6b0101090 0x7fc6b0101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.652+0000 7fc6b4b87640 1 -- 172.21.15.50:0/1925928049 >> 172.21.15.50:0/1925928049 conn(0x7fc6b0077f50 msgr2=0x7fc6b0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:13.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.652+0000 7fc6b4b87640 1 -- 172.21.15.50:0/1925928049 shutdown_connections 2024-04-23T18:31:13.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.652+0000 7fc6b4b87640 1 -- 172.21.15.50:0/1925928049 wait complete. 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.652+0000 7fc6b4b87640 1 Processor -- start 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.652+0000 7fc6b4b87640 1 -- start start 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6b4b87640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b0101090 0x7fc6b01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6b4b87640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6b0106160 0x7fc6b0198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6b4b87640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b019cfa0 0x7fc6b019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6b4b87640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01121a0 con 0x7fc6b0101090 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6b4b87640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b0112020 con 0x7fc6b019cfa0 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6b4b87640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b0112320 con 0x7fc6b0106160 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6ae59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b0101090 0x7fc6b01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6ae59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b0101090 0x7fc6b01986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41442/0 (socket says 172.21.15.50:41442) 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6ae59c640 1 -- 172.21.15.50:0/929953885 learned_addr learned my addr 172.21.15.50:0/929953885 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b019cfa0 0x7fc6b019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6add9b640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6b0106160 0x7fc6b0198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6ae59c640 1 -- 172.21.15.50:0/929953885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6b0106160 msgr2=0x7fc6b0198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6b0106160 0x7fc6b0198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6ae59c640 1 -- 172.21.15.50:0/929953885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b019cfa0 msgr2=0x7fc6b019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b019cfa0 0x7fc6b019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6ae59c640 1 -- 172.21.15.50:0/929953885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b019da90 con 0x7fc6b0101090 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b019cfa0 0x7fc6b019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6add9b640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6b0106160 0x7fc6b0198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b0101090 0x7fc6b01986e0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000cce0 tx=0x7fc6a000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc69f7fe640 1 -- 172.21.15.50:0/929953885 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a000f650 con 0x7fc6b0101090 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc69f7fe640 1 -- 172.21.15.50:0/929953885 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6a000fca0 con 0x7fc6b0101090 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6b4b87640 1 -- 172.21.15.50:0/929953885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6b019dd80 con 0x7fc6b0101090 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc69f7fe640 1 -- 172.21.15.50:0/929953885 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0018b40 con 0x7fc6b0101090 2024-04-23T18:31:13.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.653+0000 7fc6b4b87640 1 -- 172.21.15.50:0/929953885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6b01a5570 con 0x7fc6b0101090 2024-04-23T18:31:13.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.656+0000 7fc69f7fe640 1 -- 172.21.15.50:0/929953885 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc6a000f7f0 con 0x7fc6b0101090 2024-04-23T18:31:13.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.656+0000 7fc6b797b740 1 Processor -- start 2024-04-23T18:31:13.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.656+0000 7fc6b797b740 1 -- start start 2024-04-23T18:31:13.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.656+0000 7fc69f7fe640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc6980a3d20 0x7fc6980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.656+0000 7fc6b797b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e49490 0x556bb8e49870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.656+0000 7fc6b797b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49db0 0x556bb8e53f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.656+0000 7fc6b797b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 0x556bb8e56870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.656+0000 7fc6b797b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556bb8dae6d0 con 0x556bb8e54480 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.656+0000 7fc6b797b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556bb8d4b1a0 con 0x556bb8e49490 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.656+0000 7fc6b797b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556bb8daa690 con 0x556bb8e49db0 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.656+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc6980a3d20 0x7fc6980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.656+0000 7fc69f7fe640 1 -- 172.21.15.50:0/929953885 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8586+0+0 (secure 0 0 0) 0x7fc6a00d0dd0 con 0x7fc6b0101090 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6aed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 0x556bb8e56870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6add9b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49db0 0x556bb8e53f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6aed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 0x556bb8e56870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41452/0 (socket says 172.21.15.50:41452) 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6add9b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49db0 0x556bb8e53f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:49798/0 (socket says 172.21.15.50:49798) 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/3382959583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e49490 0x556bb8e49870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6aed9d640 1 -- 172.21.15.50:0/3382959583 learned_addr learned my addr 172.21.15.50:0/3382959583 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6aed9d640 1 -- 172.21.15.50:0/3382959583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49db0 msgr2=0x556bb8e53f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/3382959583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49db0 0x556bb8e53f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6aed9d640 1 -- 172.21.15.50:0/3382959583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e49490 msgr2=0x556bb8e49870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/3382959583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e49490 0x556bb8e49870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6aed9d640 1 -- 172.21.15.50:0/3382959583 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bb8d555b0 con 0x556bb8e54480 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6add9b640 1 --2- 172.21.15.50:0/3382959583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49db0 0x556bb8e53f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc6980a3d20 0x7fc6980a61e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80149e0 tx=0x7fc6a800cba0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/3382959583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 0x556bb8e56870 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80117b0 tx=0x7fc6a801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd9cb517402ddb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc69cff9640 1 -- 172.21.15.50:0/3382959583 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8022070 con 0x556bb8e54480 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc69cff9640 1 -- 172.21.15.50:0/3382959583 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6a800fc60 con 0x556bb8e54480 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.657+0000 7fc69cff9640 1 -- 172.21.15.50:0/3382959583 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8012900 con 0x556bb8e54480 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.658+0000 7fc6b797b740 1 -- 172.21.15.50:0/3382959583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 msgr2=0x556bb8e56870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.658+0000 7fc6b797b740 1 --2- 172.21.15.50:0/3382959583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 0x556bb8e56870 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80117b0 tx=0x7fc6a801b000 comp rx=0 tx=0).stop 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.658+0000 7fc6b797b740 1 -- 172.21.15.50:0/3382959583 shutdown_connections 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.658+0000 7fc6b797b740 1 --2- 172.21.15.50:0/3382959583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 0x556bb8e56870 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.658+0000 7fc6b797b740 1 --2- 172.21.15.50:0/3382959583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49db0 0x556bb8e53f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.658+0000 7fc6b797b740 1 --2- 172.21.15.50:0/3382959583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e49490 0x556bb8e49870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.658+0000 7fc6b797b740 1 -- 172.21.15.50:0/3382959583 >> 172.21.15.50:0/3382959583 conn(0x556bb8e457b0 msgr2=0x556bb8e45b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.658+0000 7fc6b797b740 1 -- 172.21.15.50:0/3382959583 shutdown_connections 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.658+0000 7fc6b797b740 1 -- 172.21.15.50:0/3382959583 wait complete. 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.658+0000 7fc6b797b740 1 -- start start 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6b797b740 1 Processor -- start 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6b797b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e53e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6b797b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e49db0 0x556bb8e51170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6b797b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 0x556bb8e51780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6b797b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556bb8e572b0 con 0x556bb8e54480 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6b797b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556bb8d48b20 con 0x556bb8e49db0 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6b797b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556bb8e57430 con 0x556bb8e49490 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6b797b740 10 client.0 _ll_register_callbacks cb 0x556bb8d48530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6aed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 0x556bb8e51780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6aed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 0x556bb8e51780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41466/0 (socket says 172.21.15.50:41466) 2024-04-23T18:31:13.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6aed9d640 1 -- 172.21.15.50:0/2998430490 learned_addr learned my addr 172.21.15.50:0/2998430490 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:13.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e53e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6add9b640 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e49db0 0x556bb8e51170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6aed9d640 1 -- 172.21.15.50:0/2998430490 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 msgr2=0x556bb8e53e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e53e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6aed9d640 1 -- 172.21.15.50:0/2998430490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e49db0 msgr2=0x556bb8e51170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e49db0 0x556bb8e51170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6aed9d640 1 -- 172.21.15.50:0/2998430490 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bb8ac0920 con 0x556bb8e54480 2024-04-23T18:31:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 0x556bb8e51780 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800d010 tx=0x7fc6a8005650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30682f1bce39a70c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.659+0000 7fc695ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:31:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc695ffb640 1 -- 172.21.15.50:0/2998430490 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8022070 con 0x556bb8e54480 2024-04-23T18:31:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc695ffb640 1 -- 172.21.15.50:0/2998430490 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8586+0+0 (secure 0 0 0) 0x7fc6a8004680 con 0x556bb8e54480 2024-04-23T18:31:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc6b797b740 1 -- 172.21.15.50:0/2998430490 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6a800fc60 con 0x556bb8e54480 2024-04-23T18:31:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc695ffb640 1 -- 172.21.15.50:0/2998430490 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc6a800fe10 con 0x556bb8e54480 2024-04-23T18:31:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc6b797b740 10 client.17085 fetch_fsmap learned FSMap version 48 2024-04-23T18:31:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc6b797b740 1 -- 172.21.15.50:0/2998430490 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x556bb8b8cd30 con 0x556bb8e54480 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc695ffb640 1 -- 172.21.15.50:0/2998430490 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6a800fc60 con 0x556bb8e54480 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc695ffb640 1 -- 172.21.15.50:0/2998430490 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7fc6a802e020 con 0x556bb8e54480 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc6b797b740 10 client.17085 fetch_fsmap finished waiting for FSMap version 48 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc6b797b740 10 client.17085 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc6b797b740 20 client.17085 populate_metadata read hostname 'smithi050' 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556bb8e64580 0x556bb8e66970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc6b797b740 4 client.17085 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc6b797b740 1 -- 172.21.15.50:0/2998430490 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bb8d6c500 con 0x556bb8e64580 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.660+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556bb8e64580 0x556bb8e66970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.661+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556bb8e64580 0x556bb8e66970 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c473ad55fb2676dc server_cookie=f414583077061c05 in_seq=0 out_seq=0 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.661+0000 7fc695ffb640 10 client.17085 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc695ffb640 1 -- 172.21.15.50:0/2998430490 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556bb8d6c500 con 0x556bb8e64580 2024-04-23T18:31:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc695ffb640 10 client.17085 handle_command_reply: tid=0 2024-04-23T18:31:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc6b797b740 1 client.17085 shutdown 2024-04-23T18:31:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc6b797b740 1 -- 172.21.15.50:0/2998430490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 msgr2=0x556bb8e51780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 0x556bb8e51780 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800d010 tx=0x7fc6a8005650 comp rx=0 tx=0).stop 2024-04-23T18:31:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc6b797b740 1 -- 172.21.15.50:0/2998430490 shutdown_connections 2024-04-23T18:31:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556bb8e64580 0x556bb8e66970 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e54480 0x556bb8e51780 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e49db0 0x556bb8e51170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc695ffb640 0 client.17085 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2998430490 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e53e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc6b797b740 1 -- 172.21.15.50:0/2998430490 >> 172.21.15.50:0/2998430490 conn(0x556bb8e457b0 msgr2=0x556bb8e4b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc6b797b740 1 -- 172.21.15.50:0/2998430490 shutdown_connections 2024-04-23T18:31:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc6b797b740 1 -- 172.21.15.50:0/2998430490 wait complete. 2024-04-23T18:31:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.662+0000 7fc6b797b740 20 client.17085 trim_cache size 0 max 16384 2024-04-23T18:31:13.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.679+0000 7fc6b797b740 1 Processor -- start 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.679+0000 7fc6b797b740 1 -- start start 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.679+0000 7fc6b797b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e539b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.679+0000 7fc6b797b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 0x556bb8e6c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6b797b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e54480 0x556bb8e6c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6b797b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01b0580 con 0x556bb8e49db0 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6b797b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01add90 con 0x556bb8e54480 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6b797b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01abc00 con 0x556bb8e49490 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6add9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 0x556bb8e6c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6add9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 0x556bb8e6c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41480/0 (socket says 172.21.15.50:41480) 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6add9b640 1 -- 172.21.15.50:0/2451957153 learned_addr learned my addr 172.21.15.50:0/2451957153 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6add9b640 1 -- 172.21.15.50:0/2451957153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 msgr2=0x556bb8e539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/2451957153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6add9b640 1 --2- 172.21.15.50:0/2451957153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/2451957153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e54480 0x556bb8e6c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6add9b640 1 -- 172.21.15.50:0/2451957153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e54480 msgr2=0x556bb8e6c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6add9b640 1 --2- 172.21.15.50:0/2451957153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e54480 0x556bb8e6c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6add9b640 1 -- 172.21.15.50:0/2451957153 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bb8e3fde0 con 0x556bb8e49db0 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/2451957153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/2451957153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e54480 0x556bb8e6c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc6add9b640 1 --2- 172.21.15.50:0/2451957153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 0x556bb8e6c100 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009900 tx=0x7fc6a4032f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd5aaddf633314a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc697fff640 1 -- 172.21.15.50:0/2451957153 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4035040 con 0x556bb8e49db0 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc697fff640 1 -- 172.21.15.50:0/2451957153 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6a40318c0 con 0x556bb8e49db0 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.680+0000 7fc697fff640 1 -- 172.21.15.50:0/2451957153 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4004920 con 0x556bb8e49db0 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.681+0000 7fc6b797b740 1 -- 172.21.15.50:0/2451957153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 msgr2=0x556bb8e6c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.681+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2451957153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 0x556bb8e6c100 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009900 tx=0x7fc6a4032f00 comp rx=0 tx=0).stop 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.681+0000 7fc6b797b740 1 -- 172.21.15.50:0/2451957153 shutdown_connections 2024-04-23T18:31:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.681+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2451957153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e54480 0x556bb8e6c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.681+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2451957153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 0x556bb8e6c100 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.681+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2451957153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.681+0000 7fc6b797b740 1 -- 172.21.15.50:0/2451957153 >> 172.21.15.50:0/2451957153 conn(0x556bb8e457b0 msgr2=0x556bb8e50a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.681+0000 7fc6b797b740 1 -- 172.21.15.50:0/2451957153 shutdown_connections 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.681+0000 7fc6b797b740 1 -- 172.21.15.50:0/2451957153 wait complete. 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.681+0000 7fc6b797b740 1 -- start start 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.681+0000 7fc6b797b740 1 Processor -- start 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.681+0000 7fc6b797b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e55420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6b797b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 0x556bb8e55960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6b797b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e54480 0x556bb8e55ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6b797b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01ac250 con 0x556bb8e49db0 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6b797b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01ad6c0 con 0x556bb8e54480 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6b797b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b01ac920 con 0x556bb8e49490 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6b797b740 10 client.0 _ll_register_callbacks cb 0x7fc6a8011b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6add9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 0x556bb8e55960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6add9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 0x556bb8e55960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41488/0 (socket says 172.21.15.50:41488) 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6add9b640 1 -- 172.21.15.50:0/2292115499 learned_addr learned my addr 172.21.15.50:0/2292115499 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6aed9d640 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e54480 0x556bb8e55ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e55420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6add9b640 1 -- 172.21.15.50:0/2292115499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 msgr2=0x556bb8e55420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6add9b640 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e55420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6add9b640 1 -- 172.21.15.50:0/2292115499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e54480 msgr2=0x556bb8e55ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6add9b640 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e54480 0x556bb8e55ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6add9b640 1 -- 172.21.15.50:0/2292115499 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bb8e5ad00 con 0x556bb8e49db0 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e55420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6add9b640 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 0x556bb8e55960 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4031a90 tx=0x7fc6a4032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8bf201f94b3287b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc695ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc695ffb640 1 -- 172.21.15.50:0/2292115499 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4035030 con 0x556bb8e49db0 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc695ffb640 1 -- 172.21.15.50:0/2292115499 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8586+0+0 (secure 0 0 0) 0x7fc6a4045880 con 0x556bb8e49db0 2024-04-23T18:31:13.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6b797b740 1 -- 172.21.15.50:0/2292115499 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6a40318c0 con 0x556bb8e49db0 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc695ffb640 1 -- 172.21.15.50:0/2292115499 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc6a4058050 con 0x556bb8e49db0 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6b797b740 10 client.17091 fetch_fsmap learned FSMap version 48 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.682+0000 7fc6b797b740 1 -- 172.21.15.50:0/2292115499 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x556bb8ecae00 con 0x556bb8e49db0 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.683+0000 7fc695ffb640 1 -- 172.21.15.50:0/2292115499 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6a40318c0 con 0x556bb8e49db0 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.683+0000 7fc695ffb640 1 -- 172.21.15.50:0/2292115499 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7fc6a403ca50 con 0x556bb8e49db0 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.683+0000 7fc6b797b740 10 client.17091 fetch_fsmap finished waiting for FSMap version 48 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.683+0000 7fc6b797b740 10 client.17091 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.683+0000 7fc6b797b740 20 client.17091 populate_metadata read hostname 'smithi050' 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.683+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556bb8ecb600 0x556bb8ecda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.683+0000 7fc6b797b740 4 client.17091 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "d0f64ea1", "release": true, "await": true}] 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.683+0000 7fc6b797b740 1 -- 172.21.15.50:0/2292115499 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "d0f64ea1", "release": true, "await": true}) -- 0x556bb8dbc0f0 con 0x556bb8ecb600 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.683+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556bb8ecb600 0x556bb8ecda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:13.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.683+0000 7fc6ae59c640 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556bb8ecb600 0x556bb8ecda20 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f04becde5050034 server_cookie=978a8e9a3508af26 in_seq=0 out_seq=0 2024-04-23T18:31:13.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.683+0000 7fc695ffb640 10 client.17091 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc695ffb640 1 -- 172.21.15.50:0/2292115499 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x556bb8dbc0f0 con 0x556bb8ecb600 2024-04-23T18:31:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc695ffb640 10 client.17091 handle_command_reply: tid=0 2024-04-23T18:31:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc6b797b740 1 client.17091 shutdown 2024-04-23T18:31:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc6b797b740 1 -- 172.21.15.50:0/2292115499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 msgr2=0x556bb8e55960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 0x556bb8e55960 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4031a90 tx=0x7fc6a4032d10 comp rx=0 tx=0).stop 2024-04-23T18:31:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc6b797b740 1 -- 172.21.15.50:0/2292115499 shutdown_connections 2024-04-23T18:31:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556bb8ecb600 0x556bb8ecda20 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556bb8e54480 0x556bb8e55ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556bb8e49db0 0x556bb8e55960 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc6b797b740 1 --2- 172.21.15.50:0/2292115499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556bb8e49490 0x556bb8e55420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc695ffb640 0 client.17091 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:13.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc6b797b740 1 -- 172.21.15.50:0/2292115499 >> 172.21.15.50:0/2292115499 conn(0x556bb8e457b0 msgr2=0x556bb8e51260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:13.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc6b797b740 1 -- 172.21.15.50:0/2292115499 shutdown_connections 2024-04-23T18:31:13.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc6b797b740 1 -- 172.21.15.50:0/2292115499 wait complete. 2024-04-23T18:31:13.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.688+0000 7fc6b797b740 20 client.17091 trim_cache size 0 max 16384 2024-04-23T18:31:13.684 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:31:13.684 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 48, 2024-04-23T18:31:13.684 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:31:13.684 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 44, 2024-04-23T18:31:13.684 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:31:13.684 INFO:teuthology.orchestra.run.smithi050.stdout: "d0f64ea1": { 2024-04-23T18:31:13.684 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 44, 2024-04-23T18:31:13.684 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 84.0, 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:31:13.685 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:31:13.686 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:31:13.686 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:31:13.686 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:31:13.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.689+0000 7fc6b4b87640 1 -- 172.21.15.50:0/929953885 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc6980a3d20 msgr2=0x7fc6980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.689+0000 7fc6b4b87640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc6980a3d20 0x7fc6980a61e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80149e0 tx=0x7fc6a800cba0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.689+0000 7fc6b4b87640 1 -- 172.21.15.50:0/929953885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b0101090 msgr2=0x7fc6b01986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.689+0000 7fc6b4b87640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b0101090 0x7fc6b01986e0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000cce0 tx=0x7fc6a000a620 comp rx=0 tx=0).stop 2024-04-23T18:31:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.689+0000 7fc6b4b87640 1 -- 172.21.15.50:0/929953885 shutdown_connections 2024-04-23T18:31:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.689+0000 7fc6b4b87640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc6980a3d20 0x7fc6980a61e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.689+0000 7fc6b4b87640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b019cfa0 0x7fc6b019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.689+0000 7fc6b4b87640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6b0106160 0x7fc6b0198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.689+0000 7fc6b4b87640 1 --2- 172.21.15.50:0/929953885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b0101090 0x7fc6b01986e0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.689+0000 7fc6b4b87640 1 -- 172.21.15.50:0/929953885 >> 172.21.15.50:0/929953885 conn(0x7fc6b0077f50 msgr2=0x7fc6b00fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.689+0000 7fc6b4b87640 1 -- 172.21.15.50:0/929953885 shutdown_connections 2024-04-23T18:31:13.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:31:13.689+0000 7fc6b4b87640 1 -- 172.21.15.50:0/929953885 wait complete. 2024-04-23T18:31:13.693 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'd0f64ea1', seconds elapsed: 25.2 2024-04-23T18:31:14.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 742074 5.05 MB/sec execute 588 sec latency 24538.404 ms 2024-04-23T18:31:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:14 smithi177 ceph-mon[32308]: pgmap v400: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 31 KiB/s wr, 13 op/s 2024-04-23T18:31:14.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:14 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4167191097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:31:14.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:14 smithi188 ceph-mon[32694]: pgmap v400: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 31 KiB/s wr, 13 op/s 2024-04-23T18:31:14.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:14 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4167191097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:31:14.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:14 smithi050 ceph-mon[29187]: pgmap v400: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 31 KiB/s wr, 13 op/s 2024-04-23T18:31:14.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:14 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4167191097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:31:15.239 INFO:tasks.workunit.client.0.smithi050.stdout: 1 743513 5.05 MB/sec execute 589 sec latency 14.562 ms 2024-04-23T18:31:16.239 INFO:tasks.workunit.client.0.smithi050.stdout: 1 745616 5.07 MB/sec execute 590 sec latency 16.094 ms 2024-04-23T18:31:16.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:16 smithi177 ceph-mon[32308]: pgmap v401: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 31 KiB/s wr, 13 op/s 2024-04-23T18:31:16.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:16 smithi177 ceph-mon[32308]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T18:31:16.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:16 smithi188 ceph-mon[32694]: pgmap v401: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 31 KiB/s wr, 13 op/s 2024-04-23T18:31:16.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:16 smithi188 ceph-mon[32694]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T18:31:16.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:16 smithi050 ceph-mon[29187]: pgmap v401: 97 pgs: 97 active+clean; 230 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 31 KiB/s wr, 13 op/s 2024-04-23T18:31:16.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:16 smithi050 ceph-mon[29187]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T18:31:17.239 INFO:tasks.workunit.client.0.smithi050.stdout: 1 747101 5.07 MB/sec execute 591 sec latency 16.161 ms 2024-04-23T18:31:18.238 INFO:tasks.workunit.client.0.smithi050.stdout: 1 749218 5.08 MB/sec execute 592 sec latency 13.803 ms 2024-04-23T18:31:18.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:18 smithi177 ceph-mon[32308]: pgmap v403: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 918 KiB/s wr, 98 op/s 2024-04-23T18:31:18.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:18 smithi188 ceph-mon[32694]: pgmap v403: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 918 KiB/s wr, 98 op/s 2024-04-23T18:31:18.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:18 smithi050 ceph-mon[29187]: pgmap v403: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 918 KiB/s wr, 98 op/s 2024-04-23T18:31:19.239 INFO:tasks.workunit.client.0.smithi050.stdout: 1 750629 5.08 MB/sec execute 593 sec latency 13.955 ms 2024-04-23T18:31:20.239 INFO:tasks.workunit.client.0.smithi050.stdout: 1 752581 5.09 MB/sec execute 594 sec latency 19.712 ms 2024-04-23T18:31:20.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:20 smithi177 ceph-mon[32308]: pgmap v404: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 774 KiB/s wr, 83 op/s 2024-04-23T18:31:20.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:20 smithi188 ceph-mon[32694]: pgmap v404: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 774 KiB/s wr, 83 op/s 2024-04-23T18:31:20.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:20 smithi050 ceph-mon[29187]: pgmap v404: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 774 KiB/s wr, 83 op/s 2024-04-23T18:31:21.241 INFO:tasks.workunit.client.0.smithi050.stdout: 1 753976 5.09 MB/sec execute 595 sec latency 17.764 ms 2024-04-23T18:31:22.239 INFO:tasks.workunit.client.0.smithi050.stdout: 1 756125 5.11 MB/sec execute 596 sec latency 18.039 ms 2024-04-23T18:31:22.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:22 smithi050 ceph-mon[29187]: pgmap v405: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 734 KiB/s wr, 79 op/s 2024-04-23T18:31:22.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:22 smithi177 ceph-mon[32308]: pgmap v405: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 734 KiB/s wr, 79 op/s 2024-04-23T18:31:22.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:22 smithi188 ceph-mon[32694]: pgmap v405: 97 pgs: 97 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 734 KiB/s wr, 79 op/s 2024-04-23T18:31:23.239 INFO:tasks.workunit.client.0.smithi050.stdout: 1 757497 5.11 MB/sec execute 597 sec latency 16.663 ms 2024-04-23T18:31:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.319+0000 7fd15a07c640 1 -- 172.21.15.50:0/3700523795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd154109490 msgr2=0x7fd154111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.319+0000 7fd15a07c640 1 --2- 172.21.15.50:0/3700523795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd154109490 0x7fd154111060 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009ca0 tx=0x7fd14c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.319+0000 7fd15a07c640 1 -- 172.21.15.50:0/3700523795 shutdown_connections 2024-04-23T18:31:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.319+0000 7fd15a07c640 1 --2- 172.21.15.50:0/3700523795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd154109490 0x7fd154111060 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.319+0000 7fd15a07c640 1 --2- 172.21.15.50:0/3700523795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd154104d70 0x7fd154108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.319+0000 7fd15a07c640 1 --2- 172.21.15.50:0/3700523795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1541043c0 0x7fd1541047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.319+0000 7fd15a07c640 1 -- 172.21.15.50:0/3700523795 >> 172.21.15.50:0/3700523795 conn(0x7fd1540ffc00 msgr2=0x7fd154102020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.319+0000 7fd15a07c640 1 -- 172.21.15.50:0/3700523795 shutdown_connections 2024-04-23T18:31:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.319+0000 7fd15a07c640 1 -- 172.21.15.50:0/3700523795 wait complete. 2024-04-23T18:31:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd15a07c640 1 Processor -- start 2024-04-23T18:31:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd15a07c640 1 -- start start 2024-04-23T18:31:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd15a07c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1541043c0 0x7fd1541a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd15a07c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd154109490 0x7fd1541a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd15a07c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd15419a6e0 0x7fd15419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd15a07c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd154114400 con 0x7fd15419a6e0 2024-04-23T18:31:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd15a07c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd154114280 con 0x7fd1541043c0 2024-04-23T18:31:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd15a07c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd154114580 con 0x7fd154109490 2024-04-23T18:31:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd153fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd15419a6e0 0x7fd15419ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd152ffd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd154109490 0x7fd1541a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd153fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd15419a6e0 0x7fd15419ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60802/0 (socket says 172.21.15.50:60802) 2024-04-23T18:31:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd153fff640 1 -- 172.21.15.50:0/2361595383 learned_addr learned my addr 172.21.15.50:0/2361595383 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd152ffd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd154109490 0x7fd1541a0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:55044/0 (socket says 172.21.15.50:55044) 2024-04-23T18:31:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.320+0000 7fd1537fe640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1541043c0 0x7fd1541a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.321+0000 7fd153fff640 1 -- 172.21.15.50:0/2361595383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd154109490 msgr2=0x7fd1541a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.321+0000 7fd153fff640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd154109490 0x7fd1541a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.321+0000 7fd153fff640 1 -- 172.21.15.50:0/2361595383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1541043c0 msgr2=0x7fd1541a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.321+0000 7fd153fff640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1541043c0 0x7fd1541a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.321+0000 7fd153fff640 1 -- 172.21.15.50:0/2361595383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15419b2e0 con 0x7fd15419a6e0 2024-04-23T18:31:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.321+0000 7fd1537fe640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1541043c0 0x7fd1541a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:31:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.321+0000 7fd153fff640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd15419a6e0 0x7fd15419ab60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fd14c005cc0 tx=0x7fd14c031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.321+0000 7fd150ff9640 1 -- 172.21.15.50:0/2361595383 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c003b70 con 0x7fd15419a6e0 2024-04-23T18:31:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.321+0000 7fd15a07c640 1 -- 172.21.15.50:0/2361595383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd15419b570 con 0x7fd15419a6e0 2024-04-23T18:31:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.321+0000 7fd150ff9640 1 -- 172.21.15.50:0/2361595383 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd14c0318f0 con 0x7fd15419a6e0 2024-04-23T18:31:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.321+0000 7fd150ff9640 1 -- 172.21.15.50:0/2361595383 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c02f950 con 0x7fd15419a6e0 2024-04-23T18:31:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.321+0000 7fd15a07c640 1 -- 172.21.15.50:0/2361595383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1541a76c0 con 0x7fd15419a6e0 2024-04-23T18:31:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd150ff9640 1 -- 172.21.15.50:0/2361595383 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd14c02fb50 con 0x7fd15419a6e0 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd15cd90740 1 Processor -- start 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd15cd90740 1 -- start start 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd15cd90740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fb620 0x55dd036fba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd15cd90740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 0x55dd037062c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd15cd90740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd03706800 0x55dd03708bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd15cd90740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55dd03660970 con 0x55dd036fbf40 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd15cd90740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dd035fd4a0 con 0x55dd03706800 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd15cd90740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55dd0365c930 con 0x55dd036fb620 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd150ff9640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd13c0a3dc0 0x7fd13c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd1537fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 0x55dd037062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd152ffd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fb620 0x55dd036fba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd1537fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 0x55dd037062c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60808/0 (socket says 172.21.15.50:60808) 2024-04-23T18:31:23.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd1537fe640 1 -- 172.21.15.50:0/71370895 learned_addr learned my addr 172.21.15.50:0/71370895 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.324+0000 7fd152ffd640 1 --2- 172.21.15.50:0/71370895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd03706800 0x55dd03708bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd153fff640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd13c0a3dc0 0x7fd13c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd150ff9640 1 -- 172.21.15.50:0/2361595383 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8542+0+0 (secure 0 0 0) 0x7fd14c0f19f0 con 0x7fd15419a6e0 2024-04-23T18:31:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd152ffd640 1 -- 172.21.15.50:0/71370895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd03706800 msgr2=0x55dd03708bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd152ffd640 1 --2- 172.21.15.50:0/71370895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd03706800 0x55dd03708bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd152ffd640 1 -- 172.21.15.50:0/71370895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 msgr2=0x55dd037062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd152ffd640 1 --2- 172.21.15.50:0/71370895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 0x55dd037062c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd152ffd640 1 -- 172.21.15.50:0/71370895 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd03373210 con 0x55dd036fb620 2024-04-23T18:31:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd1537fe640 1 --2- 172.21.15.50:0/71370895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 0x55dd037062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:31:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd152ffd640 1 --2- 172.21.15.50:0/71370895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fb620 0x55dd036fba00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd15419bee0 tx=0x7fd148004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=867b4be0f77a6787 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd153fff640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd13c0a3dc0 0x7fd13c0a6280 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0ef2f0 tx=0x7fd14c02a8c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd139ffb640 1 -- 172.21.15.50:0/71370895 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148011050 con 0x55dd036fb620 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd139ffb640 1 -- 172.21.15.50:0/71370895 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1480111f0 con 0x55dd036fb620 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.325+0000 7fd139ffb640 1 -- 172.21.15.50:0/71370895 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148004de0 con 0x55dd036fb620 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.326+0000 7fd15cd90740 1 -- 172.21.15.50:0/71370895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fb620 msgr2=0x55dd036fba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.326+0000 7fd15cd90740 1 --2- 172.21.15.50:0/71370895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fb620 0x55dd036fba00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd15419bee0 tx=0x7fd148004730 comp rx=0 tx=0).stop 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.326+0000 7fd15cd90740 1 -- 172.21.15.50:0/71370895 shutdown_connections 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.326+0000 7fd15cd90740 1 --2- 172.21.15.50:0/71370895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd03706800 0x55dd03708bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.326+0000 7fd15cd90740 1 --2- 172.21.15.50:0/71370895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 0x55dd037062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.326+0000 7fd15cd90740 1 --2- 172.21.15.50:0/71370895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fb620 0x55dd036fba00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.326+0000 7fd15cd90740 1 -- 172.21.15.50:0/71370895 >> 172.21.15.50:0/71370895 conn(0x55dd036f7b30 msgr2=0x55dd036f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.326+0000 7fd15cd90740 1 -- 172.21.15.50:0/71370895 shutdown_connections 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.326+0000 7fd15cd90740 1 -- 172.21.15.50:0/71370895 wait complete. 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.326+0000 7fd15cd90740 1 -- start start 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.326+0000 7fd15cd90740 1 Processor -- start 2024-04-23T18:31:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd15cd90740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fb620 0x55dd03706190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd15cd90740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fbf40 0x55dd03703480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd15cd90740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd03706800 0x55dd03703a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd15cd90740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55dd037093d0 con 0x55dd03706800 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd15cd90740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dd035fb0b0 con 0x55dd036fb620 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd15cd90740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55dd03709550 con 0x55dd036fbf40 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd15cd90740 10 client.0 _ll_register_callbacks cb 0x55dd035e8d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd1537fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fbf40 0x55dd03703480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd1537fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fbf40 0x55dd03703480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:55070/0 (socket says 172.21.15.50:55070) 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd1537fe640 1 -- 172.21.15.50:0/2768006989 learned_addr learned my addr 172.21.15.50:0/2768006989 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd152ffd640 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fb620 0x55dd03706190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd152ffd640 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd03706800 0x55dd03703a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd1537fe640 1 -- 172.21.15.50:0/2768006989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fb620 msgr2=0x55dd03706190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd1537fe640 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fb620 0x55dd03706190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd1537fe640 1 -- 172.21.15.50:0/2768006989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd03706800 msgr2=0x55dd03703a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd1537fe640 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd03706800 0x55dd03703a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd1537fe640 1 -- 172.21.15.50:0/2768006989 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd0343f620 con 0x55dd036fbf40 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd1537fe640 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fbf40 0x55dd03703480 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd144012d20 tx=0x7fd14400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e92371c7077b79cd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.327+0000 7fd132ffd640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd132ffd640 1 -- 172.21.15.50:0/2768006989 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144010430 con 0x55dd036fbf40 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd132ffd640 1 -- 172.21.15.50:0/2768006989 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8542+0+0 (secure 0 0 0) 0x7fd1440105d0 con 0x55dd036fbf40 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd15cd90740 1 -- 172.21.15.50:0/2768006989 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1480111f0 con 0x55dd036fbf40 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd132ffd640 1 -- 172.21.15.50:0/2768006989 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fd1440107d0 con 0x55dd036fbf40 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd15cd90740 10 client.26671 fetch_fsmap learned FSMap version 48 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd15cd90740 1 -- 172.21.15.50:0/2768006989 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55dd03715380 con 0x55dd036fbf40 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd132ffd640 1 -- 172.21.15.50:0/2768006989 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1480111f0 con 0x55dd036fbf40 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd132ffd640 1 -- 172.21.15.50:0/2768006989 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7fd14401c6f0 con 0x55dd036fbf40 2024-04-23T18:31:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd15cd90740 10 client.26671 fetch_fsmap finished waiting for FSMap version 48 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd15cd90740 10 client.26671 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd15cd90740 20 client.26671 populate_metadata read hostname 'smithi050' 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd15cd90740 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55dd03716900 0x55dd03718cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd15cd90740 4 client.26671 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd15cd90740 1 -- 172.21.15.50:0/2768006989 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd0366e390 con 0x55dd03716900 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.328+0000 7fd152ffd640 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55dd03716900 0x55dd03718cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.329+0000 7fd152ffd640 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55dd03716900 0x55dd03718cf0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d73a703f1e6157d server_cookie=1549d64c70016f1a in_seq=0 out_seq=0 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.329+0000 7fd132ffd640 10 client.26671 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.329+0000 7fd132ffd640 1 -- 172.21.15.50:0/2768006989 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd0366e390 con 0x55dd03716900 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.329+0000 7fd132ffd640 10 client.26671 handle_command_reply: tid=0 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd15cd90740 1 client.26671 shutdown 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd15cd90740 1 -- 172.21.15.50:0/2768006989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fbf40 msgr2=0x55dd03703480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd15cd90740 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fbf40 0x55dd03703480 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd144012d20 tx=0x7fd14400f790 comp rx=0 tx=0).stop 2024-04-23T18:31:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd15cd90740 1 -- 172.21.15.50:0/2768006989 shutdown_connections 2024-04-23T18:31:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd15cd90740 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55dd03716900 0x55dd03718cf0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd15cd90740 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd03706800 0x55dd03703a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd132ffd640 0 client.26671 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd15cd90740 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd036fbf40 0x55dd03703480 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd15cd90740 1 --2- 172.21.15.50:0/2768006989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fb620 0x55dd03706190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd15cd90740 1 -- 172.21.15.50:0/2768006989 >> 172.21.15.50:0/2768006989 conn(0x55dd036f7b30 msgr2=0x55dd036fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd15cd90740 1 -- 172.21.15.50:0/2768006989 shutdown_connections 2024-04-23T18:31:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd15cd90740 1 -- 172.21.15.50:0/2768006989 wait complete. 2024-04-23T18:31:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.330+0000 7fd15cd90740 20 client.26671 trim_cache size 0 max 16384 2024-04-23T18:31:23.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.345+0000 7fd15cd90740 1 Processor -- start 2024-04-23T18:31:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.345+0000 7fd15cd90740 1 -- start start 2024-04-23T18:31:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.345+0000 7fd15cd90740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fb620 0x55dd03712290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.345+0000 7fd15cd90740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 0x55dd03705cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.345+0000 7fd15cd90740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd03706800 0x55dd0371e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd15cd90740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd1541b2730 con 0x55dd036fbf40 2024-04-23T18:31:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd15cd90740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd1541aff40 con 0x55dd036fb620 2024-04-23T18:31:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd15cd90740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd1541addb0 con 0x55dd03706800 2024-04-23T18:31:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd1537fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 0x55dd03705cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd1537fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 0x55dd03705cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60826/0 (socket says 172.21.15.50:60826) 2024-04-23T18:31:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd1537fe640 1 -- 172.21.15.50:0/2684865997 learned_addr learned my addr 172.21.15.50:0/2684865997 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd152ffd640 1 --2- 172.21.15.50:0/2684865997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fb620 0x55dd03712290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd152ffd640 1 --2- 172.21.15.50:0/2684865997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd03706800 0x55dd0371e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd1537fe640 1 -- 172.21.15.50:0/2684865997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd03706800 msgr2=0x55dd0371e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd1537fe640 1 --2- 172.21.15.50:0/2684865997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd03706800 0x55dd0371e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd1537fe640 1 -- 172.21.15.50:0/2684865997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fb620 msgr2=0x55dd03712290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd1537fe640 1 --2- 172.21.15.50:0/2684865997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fb620 0x55dd03712290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd1537fe640 1 -- 172.21.15.50:0/2684865997 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd03702430 con 0x55dd036fbf40 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd1537fe640 1 --2- 172.21.15.50:0/2684865997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 0x55dd03705cc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd14401cd20 tx=0x7fd144009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7049cca23a51ab server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd1397fa640 1 -- 172.21.15.50:0/2684865997 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144010430 con 0x55dd036fbf40 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd1397fa640 1 -- 172.21.15.50:0/2684865997 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd144010e20 con 0x55dd036fbf40 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.346+0000 7fd1397fa640 1 -- 172.21.15.50:0/2684865997 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14401c840 con 0x55dd036fbf40 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.347+0000 7fd15cd90740 1 -- 172.21.15.50:0/2684865997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 msgr2=0x55dd03705cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.347+0000 7fd15cd90740 1 --2- 172.21.15.50:0/2684865997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 0x55dd03705cc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd14401cd20 tx=0x7fd144009f70 comp rx=0 tx=0).stop 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.347+0000 7fd15cd90740 1 -- 172.21.15.50:0/2684865997 shutdown_connections 2024-04-23T18:31:23.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.347+0000 7fd15cd90740 1 --2- 172.21.15.50:0/2684865997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd03706800 0x55dd0371e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.347+0000 7fd15cd90740 1 --2- 172.21.15.50:0/2684865997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fbf40 0x55dd03705cc0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.347+0000 7fd15cd90740 1 --2- 172.21.15.50:0/2684865997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fb620 0x55dd03712290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.347+0000 7fd15cd90740 1 -- 172.21.15.50:0/2684865997 >> 172.21.15.50:0/2684865997 conn(0x55dd036f7b30 msgr2=0x55dd036feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.347+0000 7fd15cd90740 1 -- 172.21.15.50:0/2684865997 shutdown_connections 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.347+0000 7fd15cd90740 1 -- 172.21.15.50:0/2684865997 wait complete. 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.347+0000 7fd15cd90740 1 -- start start 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd15cd90740 1 Processor -- start 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd15cd90740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fb620 0x55dd0370ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd15cd90740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fbf40 0x55dd036ffd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd15cd90740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd03706800 0x55dd03700270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd15cd90740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd1541ae400 con 0x55dd036fb620 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd15cd90740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd1541af870 con 0x55dd036fbf40 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd15cd90740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd1541aead0 con 0x55dd03706800 2024-04-23T18:31:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd15cd90740 10 client.0 _ll_register_callbacks cb 0x7fd144012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd152ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fb620 0x55dd0370ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd152ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fb620 0x55dd0370ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60830/0 (socket says 172.21.15.50:60830) 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd152ffd640 1 -- 172.21.15.50:0/1340006088 learned_addr learned my addr 172.21.15.50:0/1340006088 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd1537fe640 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fbf40 0x55dd036ffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd152ffd640 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd03706800 0x55dd03700270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd152ffd640 1 -- 172.21.15.50:0/1340006088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd03706800 msgr2=0x55dd03700270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd152ffd640 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd03706800 0x55dd03700270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd152ffd640 1 -- 172.21.15.50:0/1340006088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fbf40 msgr2=0x55dd036ffd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd152ffd640 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fbf40 0x55dd036ffd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd152ffd640 1 -- 172.21.15.50:0/1340006088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd037009b0 con 0x55dd036fb620 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd152ffd640 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fb620 0x55dd0370ea70 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fd148011fd0 tx=0x7fd148009a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e2e7c61232acda server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.348+0000 7fd132ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:31:23.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd132ffd640 1 -- 172.21.15.50:0/1340006088 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148011030 con 0x55dd036fb620 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd132ffd640 1 -- 172.21.15.50:0/1340006088 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8542+0+0 (secure 0 0 0) 0x7fd1480111d0 con 0x55dd036fb620 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd15cd90740 1 -- 172.21.15.50:0/1340006088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd144010e20 con 0x55dd036fb620 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd132ffd640 1 -- 172.21.15.50:0/1340006088 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fd148030670 con 0x55dd036fb620 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd15cd90740 10 client.17109 fetch_fsmap learned FSMap version 48 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd15cd90740 1 -- 172.21.15.50:0/1340006088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55dd0377cdb0 con 0x55dd036fb620 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd132ffd640 1 -- 172.21.15.50:0/1340006088 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd144010e20 con 0x55dd036fb620 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd132ffd640 1 -- 172.21.15.50:0/1340006088 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7fd14802f040 con 0x55dd036fb620 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd15cd90740 10 client.17109 fetch_fsmap finished waiting for FSMap version 48 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd15cd90740 10 client.17109 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd15cd90740 20 client.17109 populate_metadata read hostname 'smithi050' 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd15cd90740 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55dd0377d3f0 0x55dd0377f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd15cd90740 4 client.17109 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd15cd90740 1 -- 172.21.15.50:0/1340006088 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd03668200 con 0x55dd0377d3f0 2024-04-23T18:31:23.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.349+0000 7fd1537fe640 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55dd0377d3f0 0x55dd0377f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:23.348 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.350+0000 7fd132ffd640 10 client.17109 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:23.348 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.350+0000 7fd1537fe640 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55dd0377d3f0 0x55dd0377f810 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ff9c9fc90913c96 server_cookie=783847288cb00ec4 in_seq=0 out_seq=0 2024-04-23T18:31:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.394+0000 7fd132ffd640 1 -- 172.21.15.50:0/1340006088 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd03668200 con 0x55dd0377d3f0 2024-04-23T18:31:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.394+0000 7fd132ffd640 10 client.17109 handle_command_reply: tid=0 2024-04-23T18:31:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.394+0000 7fd15cd90740 1 client.17109 shutdown 2024-04-23T18:31:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.395+0000 7fd15cd90740 1 -- 172.21.15.50:0/1340006088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fb620 msgr2=0x55dd0370ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.395+0000 7fd15cd90740 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fb620 0x55dd0370ea70 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fd148011fd0 tx=0x7fd148009a40 comp rx=0 tx=0).stop 2024-04-23T18:31:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.395+0000 7fd15cd90740 1 -- 172.21.15.50:0/1340006088 shutdown_connections 2024-04-23T18:31:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.395+0000 7fd15cd90740 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55dd0377d3f0 0x55dd0377f810 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.395+0000 7fd15cd90740 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55dd03706800 0x55dd03700270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.395+0000 7fd15cd90740 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd036fbf40 0x55dd036ffd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.395+0000 7fd15cd90740 1 --2- 172.21.15.50:0/1340006088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55dd036fb620 0x55dd0370ea70 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.395+0000 7fd15cd90740 1 -- 172.21.15.50:0/1340006088 >> 172.21.15.50:0/1340006088 conn(0x55dd036f7b30 msgr2=0x55dd036fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.395+0000 7fd132ffd640 0 client.17109 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.395+0000 7fd15cd90740 1 -- 172.21.15.50:0/1340006088 shutdown_connections 2024-04-23T18:31:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.395+0000 7fd15cd90740 1 -- 172.21.15.50:0/1340006088 wait complete. 2024-04-23T18:31:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.395+0000 7fd15cd90740 20 client.17109 trim_cache size 0 max 16384 2024-04-23T18:31:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:31:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:31:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:31:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:31:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.396+0000 7fd15a07c640 1 -- 172.21.15.50:0/2361595383 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd13c0a3dc0 msgr2=0x7fd13c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.396+0000 7fd15a07c640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd13c0a3dc0 0x7fd13c0a6280 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0ef2f0 tx=0x7fd14c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.396+0000 7fd15a07c640 1 -- 172.21.15.50:0/2361595383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd15419a6e0 msgr2=0x7fd15419ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.396+0000 7fd15a07c640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd15419a6e0 0x7fd15419ab60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fd14c005cc0 tx=0x7fd14c031700 comp rx=0 tx=0).stop 2024-04-23T18:31:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.396+0000 7fd15a07c640 1 -- 172.21.15.50:0/2361595383 shutdown_connections 2024-04-23T18:31:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.396+0000 7fd15a07c640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd13c0a3dc0 0x7fd13c0a6280 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.396+0000 7fd15a07c640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd15419a6e0 0x7fd15419ab60 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.396+0000 7fd15a07c640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd154109490 0x7fd1541a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.396+0000 7fd15a07c640 1 --2- 172.21.15.50:0/2361595383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1541043c0 0x7fd1541a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.396+0000 7fd15a07c640 1 -- 172.21.15.50:0/2361595383 >> 172.21.15.50:0/2361595383 conn(0x7fd1540ffc00 msgr2=0x7fd154101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.396+0000 7fd15a07c640 1 -- 172.21.15.50:0/2361595383 shutdown_connections 2024-04-23T18:31:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:23.396+0000 7fd15a07c640 1 -- 172.21.15.50:0/2361595383 wait complete. 2024-04-23T18:31:24.239 INFO:tasks.workunit.client.0.smithi050.stdout: 1 759219 5.12 MB/sec execute 598 sec latency 43.632 ms 2024-04-23T18:31:24.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:24 smithi177 ceph-mon[32308]: pgmap v406: 97 pgs: 97 active+clean; 244 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-04-23T18:31:24.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:24 smithi188 ceph-mon[32694]: pgmap v406: 97 pgs: 97 active+clean; 244 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-04-23T18:31:24.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:24 smithi050 ceph-mon[29187]: pgmap v406: 97 pgs: 97 active+clean; 244 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-04-23T18:31:25.239 INFO:tasks.workunit.client.0.smithi050.stdout: 1 760656 5.12 MB/sec execute 599 sec latency 37.647 ms 2024-04-23T18:31:26.239 INFO:tasks.workunit.client.0.smithi050.stdout: 1 cleanup 600 sec 2024-04-23T18:31:26.451 INFO:tasks.workunit.client.0.smithi050.stdout: 0 cleanup 600 sec 2024-04-23T18:31:26.451 INFO:tasks.workunit.client.0.smithi050.stdout: 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: Operation Count AvgLat MaxLat 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: ---------------------------------------- 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: NTCreateX 98195 2.796 30077.820 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: Close 72132 0.003 0.105 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: Rename 4155 16.178 28429.228 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: Unlink 19834 5.466 30095.216 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: Deltree 2 89.250 131.667 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: Mkdir 1 0.006 0.006 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: Qpathinfo 89000 1.294 8202.806 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: Qfileinfo 15571 0.001 0.021 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: Qfsinfo 16313 0.177 13.014 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: Sfileinfo 8001 0.023 31.771 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: Find 34406 0.168 51.207 2024-04-23T18:31:26.452 INFO:tasks.workunit.client.0.smithi050.stdout: WriteX 48856 0.058 59.745 2024-04-23T18:31:26.453 INFO:tasks.workunit.client.0.smithi050.stdout: ReadX 153657 0.013 7.712 2024-04-23T18:31:26.453 INFO:tasks.workunit.client.0.smithi050.stdout: LockX 318 0.588 1.364 2024-04-23T18:31:26.453 INFO:tasks.workunit.client.0.smithi050.stdout: UnlockX 318 0.543 1.342 2024-04-23T18:31:26.453 INFO:tasks.workunit.client.0.smithi050.stdout: Flush 6882 2.751 100.998 2024-04-23T18:31:26.453 INFO:tasks.workunit.client.0.smithi050.stdout: 2024-04-23T18:31:26.453 INFO:tasks.workunit.client.0.smithi050.stdout:Throughput 5.12169 MB/sec 1 clients 1 procs max_latency=30095.225 ms 2024-04-23T18:31:26.453 INFO:tasks.workunit.client.0.smithi050.stderr:+ dbench 10 2024-04-23T18:31:26.454 INFO:tasks.workunit.client.0.smithi050.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-23T18:31:26.454 INFO:tasks.workunit.client.0.smithi050.stdout: 2024-04-23T18:31:26.454 INFO:tasks.workunit.client.0.smithi050.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-23T18:31:26.454 INFO:tasks.workunit.client.0.smithi050.stdout:8 of 10 processes prepared for launch 0 sec 2024-04-23T18:31:26.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:26 smithi177 ceph-mon[32308]: pgmap v407: 97 pgs: 97 active+clean; 244 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-04-23T18:31:26.553 INFO:tasks.workunit.client.0.smithi050.stdout:10 of 10 processes prepared for launch 0 sec 2024-04-23T18:31:26.553 INFO:tasks.workunit.client.0.smithi050.stdout:releasing clients 2024-04-23T18:31:26.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:26 smithi188 ceph-mon[32694]: pgmap v407: 97 pgs: 97 active+clean; 244 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-04-23T18:31:26.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:26 smithi050 ceph-mon[29187]: pgmap v407: 97 pgs: 97 active+clean; 244 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-04-23T18:31:27.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 361 129.67 MB/sec warmup 1 sec latency 627.928 ms 2024-04-23T18:31:28.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:28 smithi177 ceph-mon[32308]: pgmap v408: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.4 MiB/s wr, 289 op/s 2024-04-23T18:31:28.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 755 99.77 MB/sec warmup 2 sec latency 1627.998 ms 2024-04-23T18:31:28.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:28 smithi188 ceph-mon[32694]: pgmap v408: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.4 MiB/s wr, 289 op/s 2024-04-23T18:31:28.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:28 smithi050 ceph-mon[29187]: pgmap v408: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.4 MiB/s wr, 289 op/s 2024-04-23T18:31:29.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 1173 73.25 MB/sec warmup 3 sec latency 2628.030 ms 2024-04-23T18:31:30.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:30 smithi177 ceph-mon[32308]: pgmap v409: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-04-23T18:31:30.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 1820 61.16 MB/sec warmup 4 sec latency 3628.070 ms 2024-04-23T18:31:30.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:30 smithi188 ceph-mon[32694]: pgmap v409: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-04-23T18:31:30.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:30 smithi050 ceph-mon[29187]: pgmap v409: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-04-23T18:31:31.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:31 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:31:31.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 2386 54.59 MB/sec warmup 5 sec latency 4628.128 ms 2024-04-23T18:31:31.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:31 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:31:31.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:31 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:31:32.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:32 smithi177 ceph-mon[32308]: pgmap v410: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-04-23T18:31:32.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 3055 54.65 MB/sec warmup 6 sec latency 5628.169 ms 2024-04-23T18:31:32.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:32 smithi188 ceph-mon[32694]: pgmap v410: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-04-23T18:31:32.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:32 smithi050 ceph-mon[29187]: pgmap v410: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-04-23T18:31:33.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 3425 51.99 MB/sec warmup 7 sec latency 6628.196 ms 2024-04-23T18:31:34.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:34 smithi177 ceph-mon[32308]: pgmap v411: 97 pgs: 97 active+clean; 255 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 282 op/s 2024-04-23T18:31:34.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 3591 45.83 MB/sec warmup 8 sec latency 7628.232 ms 2024-04-23T18:31:34.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:34 smithi188 ceph-mon[32694]: pgmap v411: 97 pgs: 97 active+clean; 255 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 282 op/s 2024-04-23T18:31:34.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:34 smithi050 ceph-mon[29187]: pgmap v411: 97 pgs: 97 active+clean; 255 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 282 op/s 2024-04-23T18:31:35.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 3755 40.84 MB/sec warmup 9 sec latency 8573.168 ms 2024-04-23T18:31:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.492+0000 7fc87b74f640 1 -- 172.21.15.50:0/1160137798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8741010f0 msgr2=0x7fc8741014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.492+0000 7fc87b74f640 1 --2- 172.21.15.50:0/1160137798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8741010f0 0x7fc8741014d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc870009900 tx=0x7fc87002a410 comp rx=0 tx=0).stop 2024-04-23T18:31:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.492+0000 7fc87b74f640 1 -- 172.21.15.50:0/1160137798 shutdown_connections 2024-04-23T18:31:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.492+0000 7fc87b74f640 1 --2- 172.21.15.50:0/1160137798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc874106170 0x7fc87410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.492+0000 7fc87b74f640 1 --2- 172.21.15.50:0/1160137798 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc874101aa0 0x7fc874105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.492+0000 7fc87b74f640 1 --2- 172.21.15.50:0/1160137798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8741010f0 0x7fc8741014d0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.492+0000 7fc87b74f640 1 -- 172.21.15.50:0/1160137798 >> 172.21.15.50:0/1160137798 conn(0x7fc874077fd0 msgr2=0x7fc8740783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.492+0000 7fc87b74f640 1 -- 172.21.15.50:0/1160137798 shutdown_connections 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.492+0000 7fc87b74f640 1 -- 172.21.15.50:0/1160137798 wait complete. 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.493+0000 7fc87b74f640 1 Processor -- start 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.493+0000 7fc87b74f640 1 -- start start 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.493+0000 7fc87b74f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8741010f0 0x7fc874198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.493+0000 7fc87b74f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc874106170 0x7fc874198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.493+0000 7fc87b74f640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc87419cf80 0x7fc87419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.493+0000 7fc87b74f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741121f0 con 0x7fc874106170 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.493+0000 7fc87b74f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc874112070 con 0x7fc8741010f0 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.493+0000 7fc87b74f640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc874112370 con 0x7fc87419cf80 2024-04-23T18:31:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.493+0000 7fc878cea640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc874106170 0x7fc874198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.493+0000 7fc878cea640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc874106170 0x7fc874198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52872/0 (socket says 172.21.15.50:52872) 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.493+0000 7fc878cea640 1 -- 172.21.15.50:0/150220497 learned_addr learned my addr 172.21.15.50:0/150220497 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.493+0000 7fc8794eb640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8741010f0 0x7fc874198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.494+0000 7fc879cec640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc87419cf80 0x7fc87419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.494+0000 7fc878cea640 1 -- 172.21.15.50:0/150220497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc87419cf80 msgr2=0x7fc87419d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.494+0000 7fc878cea640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc87419cf80 0x7fc87419d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.494+0000 7fc878cea640 1 -- 172.21.15.50:0/150220497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8741010f0 msgr2=0x7fc874198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.494+0000 7fc878cea640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8741010f0 0x7fc874198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.494+0000 7fc878cea640 1 -- 172.21.15.50:0/150220497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87419da70 con 0x7fc874106170 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.494+0000 7fc878cea640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc874106170 0x7fc874198c50 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc86400e9c0 tx=0x7fc86400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.494+0000 7fc86a7fc640 1 -- 172.21.15.50:0/150220497 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86400cd80 con 0x7fc874106170 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.494+0000 7fc87b74f640 1 -- 172.21.15.50:0/150220497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc87419dd60 con 0x7fc874106170 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.494+0000 7fc86a7fc640 1 -- 172.21.15.50:0/150220497 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8640054d0 con 0x7fc874106170 2024-04-23T18:31:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.494+0000 7fc86a7fc640 1 -- 172.21.15.50:0/150220497 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864010430 con 0x7fc874106170 2024-04-23T18:31:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.494+0000 7fc87b74f640 1 -- 172.21.15.50:0/150220497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8741a55a0 con 0x7fc874106170 2024-04-23T18:31:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.497+0000 7fc87e46b740 1 Processor -- start 2024-04-23T18:31:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc87e46b740 1 -- start start 2024-04-23T18:31:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc86a7fc640 1 -- 172.21.15.50:0/150220497 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc864004fd0 con 0x7fc874106170 2024-04-23T18:31:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc87e46b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6620 0x55b4409d6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc87e46b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6f40 0x55b4409e12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc87e46b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409e3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc87e46b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b44093b970 con 0x55b4409d6f40 2024-04-23T18:31:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc87e46b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b4408d84a0 con 0x55b4409d6620 2024-04-23T18:31:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc87e46b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b440937930 con 0x55b4409e1800 2024-04-23T18:31:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc8794eb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6f40 0x55b4409e12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc8794eb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6f40 0x55b4409e12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52884/0 (socket says 172.21.15.50:52884) 2024-04-23T18:31:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc8794eb640 1 -- 172.21.15.50:0/3971235191 learned_addr learned my addr 172.21.15.50:0/3971235191 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc879cec640 1 --2- 172.21.15.50:0/3971235191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6620 0x55b4409d6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc878cea640 1 --2- 172.21.15.50:0/3971235191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409e3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc8794eb640 1 -- 172.21.15.50:0/3971235191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 msgr2=0x55b4409e3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc8794eb640 1 --2- 172.21.15.50:0/3971235191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409e3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc8794eb640 1 -- 172.21.15.50:0/3971235191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6620 msgr2=0x55b4409d6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc8794eb640 1 --2- 172.21.15.50:0/3971235191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6620 0x55b4409d6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.498+0000 7fc8794eb640 1 -- 172.21.15.50:0/3971235191 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b44064e210 con 0x55b4409d6f40 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.499+0000 7fc878cea640 1 --2- 172.21.15.50:0/3971235191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409e3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.499+0000 7fc8794eb640 1 --2- 172.21.15.50:0/3971235191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6f40 0x55b4409e12c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc870009900 tx=0x7fc870032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a8fcf6d44b861a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.499+0000 7fc86a7fc640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc8600a3dc0 0x7fc8600a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.499+0000 7fc879cec640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc8600a3dc0 0x7fc8600a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.499+0000 7fc86a7fc640 1 -- 172.21.15.50:0/150220497 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8542+0+0 (secure 0 0 0) 0x7fc8640d0c10 con 0x7fc874106170 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.499+0000 7fc85f7fe640 1 -- 172.21.15.50:0/3971235191 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc870040070 con 0x55b4409d6f40 2024-04-23T18:31:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.499+0000 7fc85f7fe640 1 -- 172.21.15.50:0/3971235191 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc870004030 con 0x55b4409d6f40 2024-04-23T18:31:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.499+0000 7fc85f7fe640 1 -- 172.21.15.50:0/3971235191 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc870035040 con 0x55b4409d6f40 2024-04-23T18:31:36.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.500+0000 7fc87e46b740 1 -- 172.21.15.50:0/3971235191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6f40 msgr2=0x55b4409e12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:36.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.500+0000 7fc879cec640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc8600a3dc0 0x7fc8600a6280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc86c005fd0 tx=0x7fc86c009500 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:36.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.500+0000 7fc87e46b740 1 --2- 172.21.15.50:0/3971235191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6f40 0x55b4409e12c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc870009900 tx=0x7fc870032b20 comp rx=0 tx=0).stop 2024-04-23T18:31:36.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.500+0000 7fc87e46b740 1 -- 172.21.15.50:0/3971235191 shutdown_connections 2024-04-23T18:31:36.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.500+0000 7fc87e46b740 1 --2- 172.21.15.50:0/3971235191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409e3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.500+0000 7fc87e46b740 1 --2- 172.21.15.50:0/3971235191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6f40 0x55b4409e12c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.500+0000 7fc87e46b740 1 --2- 172.21.15.50:0/3971235191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6620 0x55b4409d6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.500+0000 7fc87e46b740 1 -- 172.21.15.50:0/3971235191 >> 172.21.15.50:0/3971235191 conn(0x55b4409d2b30 msgr2=0x55b4409d2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.500+0000 7fc87e46b740 1 -- 172.21.15.50:0/3971235191 shutdown_connections 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.500+0000 7fc87e46b740 1 -- 172.21.15.50:0/3971235191 wait complete. 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc87e46b740 1 -- start start 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc87e46b740 1 Processor -- start 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc87e46b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 0x55b4409e11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc87e46b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6f40 0x55b4409de4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc87e46b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409deaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc87e46b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b4409e43d0 con 0x55b4409d6620 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc87e46b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b4408d60b0 con 0x55b4409d6f40 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc87e46b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b4409e4550 con 0x55b4409e1800 2024-04-23T18:31:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc87e46b740 10 client.0 _ll_register_callbacks cb 0x55b4408c3d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc8794eb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 0x55b4409e11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc8794eb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 0x55b4409e11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52888/0 (socket says 172.21.15.50:52888) 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc8794eb640 1 -- 172.21.15.50:0/1486035559 learned_addr learned my addr 172.21.15.50:0/1486035559 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc879cec640 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409deaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.501+0000 7fc878cea640 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6f40 0x55b4409de4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc8794eb640 1 -- 172.21.15.50:0/1486035559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 msgr2=0x55b4409deaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc8794eb640 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409deaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc8794eb640 1 -- 172.21.15.50:0/1486035559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6f40 msgr2=0x55b4409de4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc8794eb640 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6f40 0x55b4409de4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc8794eb640 1 -- 172.21.15.50:0/1486035559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b44071a620 con 0x55b4409d6620 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc8794eb640 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 0x55b4409e11f0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc870032ec0 tx=0x7fc870031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59705101d3ecd2eb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc85cff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:31:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc85cff9640 1 -- 172.21.15.50:0/1486035559 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc870040070 con 0x55b4409d6620 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc85cff9640 1 -- 172.21.15.50:0/1486035559 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8542+0+0 (secure 0 0 0) 0x7fc870005230 con 0x55b4409d6620 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc87e46b740 1 -- 172.21.15.50:0/1486035559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc870004030 con 0x55b4409d6620 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc85cff9640 1 -- 172.21.15.50:0/1486035559 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fc870056050 con 0x55b4409d6620 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc87e46b740 10 client.17121 fetch_fsmap learned FSMap version 49 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc87e46b740 1 -- 172.21.15.50:0/1486035559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55b4409f0380 con 0x55b4409d6620 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.502+0000 7fc85cff9640 1 -- 172.21.15.50:0/1486035559 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc870004030 con 0x55b4409d6620 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.503+0000 7fc85cff9640 1 -- 172.21.15.50:0/1486035559 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fc8700039c0 con 0x55b4409d6620 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.503+0000 7fc87e46b740 10 client.17121 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.503+0000 7fc87e46b740 10 client.17121 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.503+0000 7fc87e46b740 20 client.17121 populate_metadata read hostname 'smithi050' 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.503+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b4409f16d0 0x55b4409f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.503+0000 7fc87e46b740 4 client.17121 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.503+0000 7fc87e46b740 1 -- 172.21.15.50:0/1486035559 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b440949390 con 0x55b4409f16d0 2024-04-23T18:31:36.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.503+0000 7fc878cea640 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b4409f16d0 0x55b4409f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.500 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.503+0000 7fc878cea640 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b4409f16d0 0x55b4409f3ac0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7c34b33edf2124e server_cookie=d218ef9a7dd4ba28 in_seq=0 out_seq=0 2024-04-23T18:31:36.500 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.503+0000 7fc85cff9640 10 client.17121 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc85cff9640 1 -- 172.21.15.50:0/1486035559 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b440949390 con 0x55b4409f16d0 2024-04-23T18:31:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc85cff9640 10 client.17121 handle_command_reply: tid=0 2024-04-23T18:31:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc87e46b740 1 client.17121 shutdown 2024-04-23T18:31:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc87e46b740 1 -- 172.21.15.50:0/1486035559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 msgr2=0x55b4409e11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 0x55b4409e11f0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc870032ec0 tx=0x7fc870031400 comp rx=0 tx=0).stop 2024-04-23T18:31:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc87e46b740 1 -- 172.21.15.50:0/1486035559 shutdown_connections 2024-04-23T18:31:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b4409f16d0 0x55b4409f3ac0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409deaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6f40 0x55b4409de4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc85cff9640 0 client.17121 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1486035559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 0x55b4409e11f0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc87e46b740 1 -- 172.21.15.50:0/1486035559 >> 172.21.15.50:0/1486035559 conn(0x55b4409d2b30 msgr2=0x55b4409d88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc87e46b740 1 -- 172.21.15.50:0/1486035559 shutdown_connections 2024-04-23T18:31:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc87e46b740 1 -- 172.21.15.50:0/1486035559 wait complete. 2024-04-23T18:31:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.506+0000 7fc87e46b740 20 client.17121 trim_cache size 0 max 16384 2024-04-23T18:31:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.522+0000 7fc87e46b740 1 Processor -- start 2024-04-23T18:31:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.522+0000 7fc87e46b740 1 -- start start 2024-04-23T18:31:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.522+0000 7fc87e46b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 0x55b4409eceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.522+0000 7fc87e46b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6f40 0x55b4409f6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.522+0000 7fc87e46b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409f7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.522+0000 7fc87e46b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741b0530 con 0x55b4409d6620 2024-04-23T18:31:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.522+0000 7fc87e46b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741add40 con 0x55b4409d6f40 2024-04-23T18:31:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.522+0000 7fc87e46b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741abbb0 con 0x55b4409e1800 2024-04-23T18:31:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.522+0000 7fc8794eb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 0x55b4409eceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.522+0000 7fc8794eb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 0x55b4409eceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52894/0 (socket says 172.21.15.50:52894) 2024-04-23T18:31:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.522+0000 7fc8794eb640 1 -- 172.21.15.50:0/1804446043 learned_addr learned my addr 172.21.15.50:0/1804446043 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.523+0000 7fc878cea640 1 --2- 172.21.15.50:0/1804446043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6f40 0x55b4409f6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.523+0000 7fc879cec640 1 --2- 172.21.15.50:0/1804446043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409f7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.523+0000 7fc8794eb640 1 -- 172.21.15.50:0/1804446043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 msgr2=0x55b4409f7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.523+0000 7fc8794eb640 1 --2- 172.21.15.50:0/1804446043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409f7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.523+0000 7fc8794eb640 1 -- 172.21.15.50:0/1804446043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6f40 msgr2=0x55b4409f6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.523+0000 7fc8794eb640 1 --2- 172.21.15.50:0/1804446043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6f40 0x55b4409f6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.523+0000 7fc8794eb640 1 -- 172.21.15.50:0/1804446043 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4409e4010 con 0x55b4409d6620 2024-04-23T18:31:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.523+0000 7fc8794eb640 1 --2- 172.21.15.50:0/1804446043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 0x55b4409eceb0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fc870000c00 tx=0x7fc87002ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e35746dd608b4e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.523+0000 7fc85effd640 1 -- 172.21.15.50:0/1804446043 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc870040030 con 0x55b4409d6620 2024-04-23T18:31:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.523+0000 7fc85effd640 1 -- 172.21.15.50:0/1804446043 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc87004d070 con 0x55b4409d6620 2024-04-23T18:31:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.523+0000 7fc85effd640 1 -- 172.21.15.50:0/1804446043 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc870035040 con 0x55b4409d6620 2024-04-23T18:31:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.524+0000 7fc87e46b740 1 -- 172.21.15.50:0/1804446043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 msgr2=0x55b4409eceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.524+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1804446043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 0x55b4409eceb0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fc870000c00 tx=0x7fc87002ab10 comp rx=0 tx=0).stop 2024-04-23T18:31:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.524+0000 7fc87e46b740 1 -- 172.21.15.50:0/1804446043 shutdown_connections 2024-04-23T18:31:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.524+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1804446043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409e1800 0x55b4409f7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.524+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1804446043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6f40 0x55b4409f6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.524+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1804446043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409d6620 0x55b4409eceb0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.524+0000 7fc87e46b740 1 -- 172.21.15.50:0/1804446043 >> 172.21.15.50:0/1804446043 conn(0x55b4409d2b30 msgr2=0x55b4409ec7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.524+0000 7fc87e46b740 1 -- 172.21.15.50:0/1804446043 shutdown_connections 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.524+0000 7fc87e46b740 1 -- 172.21.15.50:0/1804446043 wait complete. 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.524+0000 7fc87e46b740 1 -- start start 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.524+0000 7fc87e46b740 1 Processor -- start 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc87e46b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6620 0x55b440a4ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc87e46b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409d6f40 0x55b440a50520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc87e46b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409e1800 0x55b4409f6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc87e46b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741ac200 con 0x55b4409e1800 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc87e46b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741ad670 con 0x55b4409d6620 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc87e46b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc8741ac8d0 con 0x55b4409d6f40 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc87e46b740 10 client.0 _ll_register_callbacks cb 0x7fc8700095e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc879cec640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409e1800 0x55b4409f6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc879cec640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409e1800 0x55b4409f6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52896/0 (socket says 172.21.15.50:52896) 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc879cec640 1 -- 172.21.15.50:0/1677017460 learned_addr learned my addr 172.21.15.50:0/1677017460 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc8794eb640 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6620 0x55b440a4ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc878cea640 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409d6f40 0x55b440a50520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc879cec640 1 -- 172.21.15.50:0/1677017460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409d6f40 msgr2=0x55b440a50520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc879cec640 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409d6f40 0x55b440a50520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc879cec640 1 -- 172.21.15.50:0/1677017460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6620 msgr2=0x55b440a4ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc879cec640 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6620 0x55b440a4ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.525+0000 7fc879cec640 1 -- 172.21.15.50:0/1677017460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4409f6810 con 0x55b4409e1800 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc879cec640 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409e1800 0x55b4409f6120 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fc86c004190 tx=0x7fc86c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732aff7c4b2c4dde server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc85cff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc85cff9640 1 -- 172.21.15.50:0/1677017460 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c004b20 con 0x55b4409e1800 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc85cff9640 1 -- 172.21.15.50:0/1677017460 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8542+0+0 (secure 0 0 0) 0x7fc86c004cc0 con 0x55b4409e1800 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc87e46b740 1 -- 172.21.15.50:0/1677017460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc87004d070 con 0x55b4409e1800 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc85cff9640 1 -- 172.21.15.50:0/1677017460 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fc86c00f990 con 0x55b4409e1800 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc87e46b740 10 client.17127 fetch_fsmap learned FSMap version 49 2024-04-23T18:31:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc87e46b740 1 -- 172.21.15.50:0/1677017460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55b440a577d0 con 0x55b4409e1800 2024-04-23T18:31:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc85cff9640 1 -- 172.21.15.50:0/1677017460 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc87004d070 con 0x55b4409e1800 2024-04-23T18:31:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc85cff9640 1 -- 172.21.15.50:0/1677017460 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fc86c0056e0 con 0x55b4409e1800 2024-04-23T18:31:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc87e46b740 10 client.17127 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T18:31:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc87e46b740 10 client.17127 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:31:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc87e46b740 20 client.17127 populate_metadata read hostname 'smithi050' 2024-04-23T18:31:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b440a57fe0 0x55b440a5a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc87e46b740 4 client.17127 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:31:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.526+0000 7fc87e46b740 1 -- 172.21.15.50:0/1677017460 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b440943200 con 0x55b440a57fe0 2024-04-23T18:31:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.527+0000 7fc8794eb640 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b440a57fe0 0x55b440a5a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.527+0000 7fc85cff9640 10 client.17127 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.527+0000 7fc8794eb640 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b440a57fe0 0x55b440a5a400 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35a7a7d8cd5b4a10 server_cookie=51e6f745dd4da6d8 in_seq=0 out_seq=0 2024-04-23T18:31:36.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:36 smithi177 ceph-mon[32308]: pgmap v412: 97 pgs: 97 active+clean; 255 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-04-23T18:31:36.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 4012 37.84 MB/sec warmup 10 sec latency 100.539 ms 2024-04-23T18:31:36.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:36 smithi188 ceph-mon[32694]: pgmap v412: 97 pgs: 97 active+clean; 255 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-04-23T18:31:36.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:36 smithi050 ceph-mon[29187]: pgmap v412: 97 pgs: 97 active+clean; 255 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-04-23T18:31:36.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc85cff9640 1 -- 172.21.15.50:0/1677017460 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b440943200 con 0x55b440a57fe0 2024-04-23T18:31:36.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc85cff9640 10 client.17127 handle_command_reply: tid=0 2024-04-23T18:31:36.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc87e46b740 1 client.17127 shutdown 2024-04-23T18:31:36.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc87e46b740 1 -- 172.21.15.50:0/1677017460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409e1800 msgr2=0x55b4409f6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:36.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409e1800 0x55b4409f6120 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fc86c004190 tx=0x7fc86c004a40 comp rx=0 tx=0).stop 2024-04-23T18:31:36.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc87e46b740 1 -- 172.21.15.50:0/1677017460 shutdown_connections 2024-04-23T18:31:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b440a57fe0 0x55b440a5a400 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b4409e1800 0x55b4409f6120 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b4409d6f40 0x55b440a50520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc85cff9640 0 client.17127 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc87e46b740 1 --2- 172.21.15.50:0/1677017460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4409d6620 0x55b440a4ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc87e46b740 1 -- 172.21.15.50:0/1677017460 >> 172.21.15.50:0/1677017460 conn(0x55b4409d2b30 msgr2=0x55b4409ed030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc87e46b740 1 -- 172.21.15.50:0/1677017460 shutdown_connections 2024-04-23T18:31:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc87e46b740 1 -- 172.21.15.50:0/1677017460 wait complete. 2024-04-23T18:31:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.957+0000 7fc87e46b740 20 client.17127 trim_cache size 0 max 16384 2024-04-23T18:31:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:31:36.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:31:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:31:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:31:36.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.958+0000 7fc87b74f640 1 -- 172.21.15.50:0/150220497 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc8600a3dc0 msgr2=0x7fc8600a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.958+0000 7fc87b74f640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc8600a3dc0 0x7fc8600a6280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc86c005fd0 tx=0x7fc86c009500 comp rx=0 tx=0).stop 2024-04-23T18:31:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.958+0000 7fc87b74f640 1 -- 172.21.15.50:0/150220497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc874106170 msgr2=0x7fc874198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.958+0000 7fc87b74f640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc874106170 0x7fc874198c50 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc86400e9c0 tx=0x7fc86400ee80 comp rx=0 tx=0).stop 2024-04-23T18:31:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.958+0000 7fc87b74f640 1 -- 172.21.15.50:0/150220497 shutdown_connections 2024-04-23T18:31:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.958+0000 7fc87b74f640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc8600a3dc0 0x7fc8600a6280 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.958+0000 7fc87b74f640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc87419cf80 0x7fc87419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.958+0000 7fc87b74f640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc874106170 0x7fc874198c50 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.958+0000 7fc87b74f640 1 --2- 172.21.15.50:0/150220497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8741010f0 0x7fc874198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.958+0000 7fc87b74f640 1 -- 172.21.15.50:0/150220497 >> 172.21.15.50:0/150220497 conn(0x7fc874077fd0 msgr2=0x7fc8740fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.958+0000 7fc87b74f640 1 -- 172.21.15.50:0/150220497 shutdown_connections 2024-04-23T18:31:36.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:36.958+0000 7fc87b74f640 1 -- 172.21.15.50:0/150220497 wait complete. 2024-04-23T18:31:37.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 4323 36.12 MB/sec warmup 11 sec latency 505.201 ms 2024-04-23T18:31:38.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:38 smithi177 ceph-mon[32308]: pgmap v413: 97 pgs: 97 active+clean; 266 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 263 op/s 2024-04-23T18:31:38.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 4928 35.31 MB/sec warmup 12 sec latency 1505.232 ms 2024-04-23T18:31:38.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:38 smithi188 ceph-mon[32694]: pgmap v413: 97 pgs: 97 active+clean; 266 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 263 op/s 2024-04-23T18:31:38.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:38 smithi050 ceph-mon[29187]: pgmap v413: 97 pgs: 97 active+clean; 266 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 263 op/s 2024-04-23T18:31:39.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 5859 37.20 MB/sec warmup 13 sec latency 2505.264 ms 2024-04-23T18:31:40.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:40 smithi177 ceph-mon[32308]: pgmap v414: 97 pgs: 97 active+clean; 266 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-04-23T18:31:40.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 6717 38.85 MB/sec warmup 14 sec latency 2936.867 ms 2024-04-23T18:31:40.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:40 smithi188 ceph-mon[32694]: pgmap v414: 97 pgs: 97 active+clean; 266 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-04-23T18:31:40.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:40 smithi050 ceph-mon[29187]: pgmap v414: 97 pgs: 97 active+clean; 266 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-04-23T18:31:41.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:41 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:31:41.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 7114 36.82 MB/sec warmup 15 sec latency 540.004 ms 2024-04-23T18:31:41.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:41 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:31:41.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:41 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:31:42.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:42 smithi177 ceph-mon[32308]: pgmap v415: 97 pgs: 97 active+clean; 266 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-04-23T18:31:42.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 8216 37.82 MB/sec warmup 16 sec latency 112.314 ms 2024-04-23T18:31:42.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:42 smithi188 ceph-mon[32694]: pgmap v415: 97 pgs: 97 active+clean; 266 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-04-23T18:31:42.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:42 smithi050 ceph-mon[29187]: pgmap v415: 97 pgs: 97 active+clean; 266 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-04-23T18:31:43.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 9453 40.10 MB/sec warmup 17 sec latency 45.212 ms 2024-04-23T18:31:44.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:44 smithi177 ceph-mon[32308]: pgmap v416: 97 pgs: 97 active+clean; 272 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 506 op/s 2024-04-23T18:31:44.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 10148 40.66 MB/sec warmup 18 sec latency 83.509 ms 2024-04-23T18:31:44.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:44 smithi188 ceph-mon[32694]: pgmap v416: 97 pgs: 97 active+clean; 272 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 506 op/s 2024-04-23T18:31:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:44 smithi050 ceph-mon[29187]: pgmap v416: 97 pgs: 97 active+clean; 272 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 506 op/s 2024-04-23T18:31:45.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 10507 38.74 MB/sec warmup 19 sec latency 121.367 ms 2024-04-23T18:31:46.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:46 smithi177 ceph-mon[32308]: pgmap v417: 97 pgs: 97 active+clean; 272 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 420 op/s 2024-04-23T18:31:46.553 INFO:tasks.workunit.client.0.smithi050.stdout: 10 11545 39.45 MB/sec warmup 20 sec latency 97.996 ms 2024-04-23T18:31:46.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:46 smithi188 ceph-mon[32694]: pgmap v417: 97 pgs: 97 active+clean; 272 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 420 op/s 2024-04-23T18:31:46.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:46 smithi050 ceph-mon[29187]: pgmap v417: 97 pgs: 97 active+clean; 272 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 420 op/s 2024-04-23T18:31:47.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 12783 41.24 MB/sec warmup 21 sec latency 49.699 ms 2024-04-23T18:31:48.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:48 smithi177 ceph-mon[32308]: pgmap v418: 97 pgs: 97 active+clean; 298 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.6 MiB/s wr, 623 op/s 2024-04-23T18:31:48.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 13509 41.65 MB/sec warmup 22 sec latency 28.902 ms 2024-04-23T18:31:48.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:48 smithi188 ceph-mon[32694]: pgmap v418: 97 pgs: 97 active+clean; 298 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.6 MiB/s wr, 623 op/s 2024-04-23T18:31:48.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:48 smithi050 ceph-mon[29187]: pgmap v418: 97 pgs: 97 active+clean; 298 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.6 MiB/s wr, 623 op/s 2024-04-23T18:31:49.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 13918 40.10 MB/sec warmup 23 sec latency 82.186 ms 2024-04-23T18:31:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.073+0000 7fc4a6508640 1 -- 172.21.15.50:0/1190163367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a0106f30 msgr2=0x7fc4a0107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.073+0000 7fc4a6508640 1 --2- 172.21.15.50:0/1190163367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a0106f30 0x7fc4a0107310 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fc490009930 tx=0x7fc49002a420 comp rx=0 tx=0).stop 2024-04-23T18:31:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.073+0000 7fc4a6508640 1 -- 172.21.15.50:0/1190163367 shutdown_connections 2024-04-23T18:31:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.073+0000 7fc4a6508640 1 --2- 172.21.15.50:0/1190163367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4a0100e10 0x7fc4a010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.073+0000 7fc4a6508640 1 --2- 172.21.15.50:0/1190163367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0100450 0x7fc4a01008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.073+0000 7fc4a6508640 1 --2- 172.21.15.50:0/1190163367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a0106f30 0x7fc4a0107310 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.073+0000 7fc4a6508640 1 -- 172.21.15.50:0/1190163367 >> 172.21.15.50:0/1190163367 conn(0x7fc4a00773e0 msgr2=0x7fc4a00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.073+0000 7fc4a6508640 1 -- 172.21.15.50:0/1190163367 shutdown_connections 2024-04-23T18:31:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.073+0000 7fc4a6508640 1 -- 172.21.15.50:0/1190163367 wait complete. 2024-04-23T18:31:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.074+0000 7fc4a6508640 1 Processor -- start 2024-04-23T18:31:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.074+0000 7fc4a6508640 1 -- start start 2024-04-23T18:31:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.074+0000 7fc4a6508640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0100450 0x7fc4a01a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.074+0000 7fc4a6508640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a0100e10 0x7fc4a01a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.074+0000 7fc4a6508640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4a0106f30 0x7fc4a019a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.074+0000 7fc4a6508640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a01122e0 con 0x7fc4a0100e10 2024-04-23T18:31:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.074+0000 7fc4a6508640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a0112160 con 0x7fc4a0100450 2024-04-23T18:31:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.074+0000 7fc4a6508640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a0112460 con 0x7fc4a0106f30 2024-04-23T18:31:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a0100e10 0x7fc4a01a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0100450 0x7fc4a01a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a0100e10 0x7fc4a01a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55214/0 (socket says 172.21.15.50:55214) 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49f7fe640 1 -- 172.21.15.50:0/1832079824 learned_addr learned my addr 172.21.15.50:0/1832079824 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc4a4aa5640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4a0106f30 0x7fc4a019a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49f7fe640 1 -- 172.21.15.50:0/1832079824 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4a0106f30 msgr2=0x7fc4a019a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49f7fe640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4a0106f30 0x7fc4a019a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49f7fe640 1 -- 172.21.15.50:0/1832079824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0100450 msgr2=0x7fc4a01a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49f7fe640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0100450 0x7fc4a01a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49f7fe640 1 -- 172.21.15.50:0/1832079824 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a019af80 con 0x7fc4a0100e10 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49f7fe640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a0100e10 0x7fc4a01a0b80 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc49400e9f0 tx=0x7fc49400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49d7fa640 1 -- 172.21.15.50:0/1832079824 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49400cd80 con 0x7fc4a0100e10 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49d7fa640 1 -- 172.21.15.50:0/1832079824 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4940108e0 con 0x7fc4a0100e10 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc49d7fa640 1 -- 172.21.15.50:0/1832079824 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494018980 con 0x7fc4a0100e10 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.075+0000 7fc4a6508640 1 -- 172.21.15.50:0/1832079824 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a019b270 con 0x7fc4a0100e10 2024-04-23T18:31:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.076+0000 7fc4a6508640 1 -- 172.21.15.50:0/1832079824 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a01a7490 con 0x7fc4a0100e10 2024-04-23T18:31:50.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.078+0000 7fc49d7fa640 1 -- 172.21.15.50:0/1832079824 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc494010430 con 0x7fc4a0100e10 2024-04-23T18:31:50.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.078+0000 7fc4a921e740 1 Processor -- start 2024-04-23T18:31:50.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.078+0000 7fc4a921e740 1 -- start start 2024-04-23T18:31:50.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.078+0000 7fc4a921e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9de9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.078+0000 7fc4a921e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9def10 0x562e3f9e9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.078+0000 7fc4a921e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9e97d0 0x562e3f9ebbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.078+0000 7fc4a921e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562e3f943970 con 0x562e3f9e97d0 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.078+0000 7fc4a921e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562e3f8e04a0 con 0x562e3f9de5f0 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.078+0000 7fc4a921e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562e3f93f930 con 0x562e3f9def10 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc49ffff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9def10 0x562e3f9e9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc4a4aa5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9de9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc4a4aa5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9de9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:47040/0 (socket says 172.21.15.50:47040) 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc4a4aa5640 1 -- 172.21.15.50:0/3847128811 learned_addr learned my addr 172.21.15.50:0/3847128811 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc49f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9e97d0 0x562e3f9ebbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc49d7fa640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4880a3e40 0x7fc4880a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc49d7fa640 1 -- 172.21.15.50:0/1832079824 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8542+0+0 (secure 0 0 0) 0x7fc4940d0ae0 con 0x7fc4a0100e10 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc4a4aa5640 1 -- 172.21.15.50:0/3847128811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9def10 msgr2=0x562e3f9e9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc4a4aa5640 1 --2- 172.21.15.50:0/3847128811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9def10 0x562e3f9e9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc4a4aa5640 1 -- 172.21.15.50:0/3847128811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9e97d0 msgr2=0x562e3f9ebbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:50.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc4a4aa5640 1 --2- 172.21.15.50:0/3847128811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9e97d0 0x562e3f9ebbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc4a4aa5640 1 -- 172.21.15.50:0/3847128811 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e3f656210 con 0x562e3f9de5f0 2024-04-23T18:31:50.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.079+0000 7fc4a4aa5640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4880a3e40 0x7fc4880a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.080+0000 7fc49ffff640 1 --2- 172.21.15.50:0/3847128811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9def10 0x562e3f9e9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.080+0000 7fc4a4aa5640 1 --2- 172.21.15.50:0/3847128811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9de9d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc498011590 tx=0x7fc498011a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=729269f80a8e4774 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.080+0000 7fc4867fc640 1 -- 172.21.15.50:0/3847128811 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498017050 con 0x562e3f9de5f0 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.080+0000 7fc4867fc640 1 -- 172.21.15.50:0/3847128811 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4980046d0 con 0x562e3f9de5f0 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.080+0000 7fc4867fc640 1 -- 172.21.15.50:0/3847128811 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49801bbd0 con 0x562e3f9de5f0 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.081+0000 7fc4a921e740 1 -- 172.21.15.50:0/3847128811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 msgr2=0x562e3f9de9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.081+0000 7fc4a921e740 1 --2- 172.21.15.50:0/3847128811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9de9d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc498011590 tx=0x7fc498011a50 comp rx=0 tx=0).stop 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.081+0000 7fc4a4aa5640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4880a3e40 0x7fc4880a6300 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc498022000 tx=0x7fc498023000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.082+0000 7fc4a921e740 1 -- 172.21.15.50:0/3847128811 shutdown_connections 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.082+0000 7fc4a921e740 1 --2- 172.21.15.50:0/3847128811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9e97d0 0x562e3f9ebbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.082+0000 7fc4a921e740 1 --2- 172.21.15.50:0/3847128811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9def10 0x562e3f9e9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.082+0000 7fc4a921e740 1 --2- 172.21.15.50:0/3847128811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9de9d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.082+0000 7fc4a921e740 1 -- 172.21.15.50:0/3847128811 >> 172.21.15.50:0/3847128811 conn(0x562e3f9dab00 msgr2=0x562e3f9daee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:50.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.082+0000 7fc4a921e740 1 -- 172.21.15.50:0/3847128811 shutdown_connections 2024-04-23T18:31:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.082+0000 7fc4a921e740 1 -- 172.21.15.50:0/3847128811 wait complete. 2024-04-23T18:31:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.083+0000 7fc4a921e740 1 -- start start 2024-04-23T18:31:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.083+0000 7fc4a921e740 1 Processor -- start 2024-04-23T18:31:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.084+0000 7fc4a921e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9de5f0 0x562e3f9e9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.084+0000 7fc4a921e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9def10 0x562e3f9e6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.084+0000 7fc4a921e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9e97d0 0x562e3f9e6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.084+0000 7fc4a921e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562e3f9ec400 con 0x562e3f9de5f0 2024-04-23T18:31:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.084+0000 7fc4a921e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562e3f8de0b0 con 0x562e3f9def10 2024-04-23T18:31:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.084+0000 7fc4a921e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562e3f9ec580 con 0x562e3f9e97d0 2024-04-23T18:31:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.084+0000 7fc4a921e740 10 client.0 _ll_register_callbacks cb 0x562e3f8cbd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.084+0000 7fc49ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9de5f0 0x562e3f9e9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.084+0000 7fc49f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9def10 0x562e3f9e6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.084+0000 7fc49ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9de5f0 0x562e3f9e9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55238/0 (socket says 172.21.15.50:55238) 2024-04-23T18:31:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.084+0000 7fc49ffff640 1 -- 172.21.15.50:0/1599432256 learned_addr learned my addr 172.21.15.50:0/1599432256 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.084+0000 7fc4a4aa5640 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9e97d0 0x562e3f9e6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.085+0000 7fc49f7fe640 1 -- 172.21.15.50:0/1599432256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9e97d0 msgr2=0x562e3f9e6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.085+0000 7fc49f7fe640 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9e97d0 0x562e3f9e6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.085+0000 7fc49f7fe640 1 -- 172.21.15.50:0/1599432256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9de5f0 msgr2=0x562e3f9e9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.085+0000 7fc49f7fe640 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9de5f0 0x562e3f9e9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.085+0000 7fc49f7fe640 1 -- 172.21.15.50:0/1599432256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e3f722620 con 0x562e3f9def10 2024-04-23T18:31:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.085+0000 7fc4a4aa5640 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9e97d0 0x562e3f9e6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:31:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.085+0000 7fc49f7fe640 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9def10 0x562e3f9e6490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc4940ce220 tx=0x7fc494010720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7026f634db42059b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.085+0000 7fc47f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:31:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.085+0000 7fc47f7fe640 1 -- 172.21.15.50:0/1599432256 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49409cb20 con 0x562e3f9def10 2024-04-23T18:31:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.085+0000 7fc4a921e740 1 -- 172.21.15.50:0/1599432256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4980046d0 con 0x562e3f9def10 2024-04-23T18:31:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.086+0000 7fc47f7fe640 1 -- 172.21.15.50:0/1599432256 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8542+0+0 (secure 0 0 0) 0x7fc494010a90 con 0x562e3f9def10 2024-04-23T18:31:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.086+0000 7fc47f7fe640 1 -- 172.21.15.50:0/1599432256 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fc4940a0e20 con 0x562e3f9def10 2024-04-23T18:31:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.086+0000 7fc4a921e740 10 client.26648 fetch_fsmap learned FSMap version 50 2024-04-23T18:31:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.086+0000 7fc4a921e740 1 -- 172.21.15.50:0/1599432256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x562e3f9ef350 con 0x562e3f9def10 2024-04-23T18:31:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.087+0000 7fc47f7fe640 1 -- 172.21.15.50:0/1599432256 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4980046d0 con 0x562e3f9def10 2024-04-23T18:31:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.087+0000 7fc47f7fe640 1 -- 172.21.15.50:0/1599432256 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7fc494098a70 con 0x562e3f9def10 2024-04-23T18:31:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.087+0000 7fc4a921e740 10 client.26648 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:31:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.087+0000 7fc4a921e740 10 client.26648 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:31:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.087+0000 7fc4a921e740 20 client.26648 populate_metadata read hostname 'smithi050' 2024-04-23T18:31:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.087+0000 7fc4a921e740 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562e3f9f08b0 0x562e3f9f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.087+0000 7fc4a921e740 4 client.26648 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:31:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.087+0000 7fc4a921e740 1 -- 172.21.15.50:0/1599432256 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e3f951390 con 0x562e3f9f08b0 2024-04-23T18:31:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.087+0000 7fc49ffff640 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562e3f9f08b0 0x562e3f9f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.088+0000 7fc49ffff640 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562e3f9f08b0 0x562e3f9f2d10 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da7f1686314915e server_cookie=dd46bc4628cdf358 in_seq=0 out_seq=0 2024-04-23T18:31:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.088+0000 7fc47f7fe640 10 client.26648 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.092+0000 7fc47f7fe640 1 -- 172.21.15.50:0/1599432256 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e3f951390 con 0x562e3f9f08b0 2024-04-23T18:31:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.092+0000 7fc47f7fe640 10 client.26648 handle_command_reply: tid=0 2024-04-23T18:31:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.092+0000 7fc4a921e740 1 client.26648 shutdown 2024-04-23T18:31:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.093+0000 7fc4a921e740 1 -- 172.21.15.50:0/1599432256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9def10 msgr2=0x562e3f9e6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.093+0000 7fc4a921e740 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9def10 0x562e3f9e6490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc4940ce220 tx=0x7fc494010720 comp rx=0 tx=0).stop 2024-04-23T18:31:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.093+0000 7fc4a921e740 1 -- 172.21.15.50:0/1599432256 shutdown_connections 2024-04-23T18:31:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.093+0000 7fc4a921e740 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562e3f9f08b0 0x562e3f9f2d10 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.093+0000 7fc4a921e740 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9e97d0 0x562e3f9e6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.093+0000 7fc4a921e740 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9def10 0x562e3f9e6490 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.093+0000 7fc47f7fe640 0 client.26648 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.093+0000 7fc4a921e740 1 --2- 172.21.15.50:0/1599432256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9de5f0 0x562e3f9e9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.093+0000 7fc4a921e740 1 -- 172.21.15.50:0/1599432256 >> 172.21.15.50:0/1599432256 conn(0x562e3f9dab00 msgr2=0x562e3f9e0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.093+0000 7fc4a921e740 1 -- 172.21.15.50:0/1599432256 shutdown_connections 2024-04-23T18:31:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.093+0000 7fc4a921e740 1 -- 172.21.15.50:0/1599432256 wait complete. 2024-04-23T18:31:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.093+0000 7fc4a921e740 20 client.26648 trim_cache size 0 max 16384 2024-04-23T18:31:50.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc4a921e740 1 Processor -- start 2024-04-23T18:31:50.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc4a921e740 1 -- start start 2024-04-23T18:31:50.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc4a921e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9fd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc4a921e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9def10 0x562e3fa14080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc4a921e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9e97d0 0x562e3fa145c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc4a921e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a01ae3b0 con 0x562e3f9e97d0 2024-04-23T18:31:50.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc4a921e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a01abbc0 con 0x562e3f9de5f0 2024-04-23T18:31:50.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc4a921e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a01a9a90 con 0x562e3f9def10 2024-04-23T18:31:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc4a4aa5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9e97d0 0x562e3fa145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc49f7fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9def10 0x562e3fa14080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc49ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9fd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc4a4aa5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9e97d0 0x562e3fa145c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55248/0 (socket says 172.21.15.50:55248) 2024-04-23T18:31:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc4a4aa5640 1 -- 172.21.15.50:0/3367159118 learned_addr learned my addr 172.21.15.50:0/3367159118 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc49f7fe640 1 -- 172.21.15.50:0/3367159118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 msgr2=0x562e3f9fd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc49f7fe640 1 --2- 172.21.15.50:0/3367159118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9fd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc49f7fe640 1 -- 172.21.15.50:0/3367159118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9e97d0 msgr2=0x562e3fa145c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc49f7fe640 1 --2- 172.21.15.50:0/3367159118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9e97d0 0x562e3fa145c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.117+0000 7fc49f7fe640 1 -- 172.21.15.50:0/3367159118 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e3f9e49e0 con 0x562e3f9def10 2024-04-23T18:31:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.118+0000 7fc49f7fe640 1 --2- 172.21.15.50:0/3367159118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9def10 0x562e3fa14080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc4940994f0 tx=0x7fc49409b030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbf725b5cdc0dd33 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.118+0000 7fc485ffb640 1 -- 172.21.15.50:0/3367159118 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49409bd60 con 0x562e3f9def10 2024-04-23T18:31:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.118+0000 7fc485ffb640 1 -- 172.21.15.50:0/3367159118 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4940a56e0 con 0x562e3f9def10 2024-04-23T18:31:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.118+0000 7fc485ffb640 1 -- 172.21.15.50:0/3367159118 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494018d80 con 0x562e3f9def10 2024-04-23T18:31:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.119+0000 7fc4a921e740 1 -- 172.21.15.50:0/3367159118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9def10 msgr2=0x562e3fa14080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.119+0000 7fc4a921e740 1 --2- 172.21.15.50:0/3367159118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9def10 0x562e3fa14080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc4940994f0 tx=0x7fc49409b030 comp rx=0 tx=0).stop 2024-04-23T18:31:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.119+0000 7fc4a921e740 1 -- 172.21.15.50:0/3367159118 shutdown_connections 2024-04-23T18:31:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.119+0000 7fc4a921e740 1 --2- 172.21.15.50:0/3367159118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9e97d0 0x562e3fa145c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.119+0000 7fc4a921e740 1 --2- 172.21.15.50:0/3367159118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9def10 0x562e3fa14080 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.119+0000 7fc4a921e740 1 --2- 172.21.15.50:0/3367159118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9fd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.119+0000 7fc4a921e740 1 -- 172.21.15.50:0/3367159118 >> 172.21.15.50:0/3367159118 conn(0x562e3f9dab00 msgr2=0x562e3f9fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.119+0000 7fc4a921e740 1 -- 172.21.15.50:0/3367159118 shutdown_connections 2024-04-23T18:31:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.119+0000 7fc4a921e740 1 -- 172.21.15.50:0/3367159118 wait complete. 2024-04-23T18:31:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.119+0000 7fc4a921e740 1 -- start start 2024-04-23T18:31:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.119+0000 7fc4a921e740 1 Processor -- start 2024-04-23T18:31:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc4a921e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9f4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc4a921e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9def10 0x562e3f9f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc4a921e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9e97d0 0x562e3fa6f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc4a921e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a01aa0e0 con 0x562e3f9def10 2024-04-23T18:31:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc4a921e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a01ab4f0 con 0x562e3f9de5f0 2024-04-23T18:31:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc4a921e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a01aa750 con 0x562e3f9e97d0 2024-04-23T18:31:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc4a921e740 10 client.0 _ll_register_callbacks cb 0x7fc49409c660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:31:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc49ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc49ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9f4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:47082/0 (socket says 172.21.15.50:47082) 2024-04-23T18:31:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc49ffff640 1 -- 172.21.15.50:0/367370125 learned_addr learned my addr 172.21.15.50:0/367370125 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:31:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc49f7fe640 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9def10 0x562e3f9f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc4a4aa5640 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9e97d0 0x562e3fa6f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc49ffff640 1 -- 172.21.15.50:0/367370125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9e97d0 msgr2=0x562e3fa6f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc49ffff640 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9e97d0 0x562e3fa6f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc49ffff640 1 -- 172.21.15.50:0/367370125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9def10 msgr2=0x562e3f9f4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:31:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc49ffff640 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9def10 0x562e3f9f4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.120+0000 7fc49ffff640 1 -- 172.21.15.50:0/367370125 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e3fa6fe20 con 0x562e3f9de5f0 2024-04-23T18:31:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.121+0000 7fc49ffff640 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9f4730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc490009900 tx=0x7fc490009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff4f694f6fdbf780 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:31:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.121+0000 7fc47f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:31:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.121+0000 7fc47f7fe640 1 -- 172.21.15.50:0/367370125 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc490035040 con 0x562e3f9de5f0 2024-04-23T18:31:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.121+0000 7fc47f7fe640 1 -- 172.21.15.50:0/367370125 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8542+0+0 (secure 0 0 0) 0x7fc490005040 con 0x562e3f9de5f0 2024-04-23T18:31:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.121+0000 7fc4a921e740 1 -- 172.21.15.50:0/367370125 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4940a56e0 con 0x562e3f9de5f0 2024-04-23T18:31:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.121+0000 7fc47f7fe640 1 -- 172.21.15.50:0/367370125 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fc490005240 con 0x562e3f9de5f0 2024-04-23T18:31:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.121+0000 7fc4a921e740 10 client.26654 fetch_fsmap learned FSMap version 50 2024-04-23T18:31:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.121+0000 7fc4a921e740 1 -- 172.21.15.50:0/367370125 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x562e3fa71100 con 0x562e3f9de5f0 2024-04-23T18:31:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.122+0000 7fc47f7fe640 1 -- 172.21.15.50:0/367370125 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc490046740 con 0x562e3f9de5f0 2024-04-23T18:31:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.122+0000 7fc47f7fe640 1 -- 172.21.15.50:0/367370125 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7fc490004a40 con 0x562e3f9de5f0 2024-04-23T18:31:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.122+0000 7fc4a921e740 10 client.26654 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:31:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.122+0000 7fc4a921e740 10 client.26654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:31:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.122+0000 7fc4a921e740 20 client.26654 populate_metadata read hostname 'smithi050' 2024-04-23T18:31:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.122+0000 7fc4a921e740 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562e3fa71880 0x562e3fa73ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:31:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.122+0000 7fc4a921e740 4 client.26654 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:31:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.122+0000 7fc4a921e740 1 -- 172.21.15.50:0/367370125 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fc4940a56e0 con 0x562e3fa71880 2024-04-23T18:31:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.122+0000 7fc49f7fe640 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562e3fa71880 0x562e3fa73ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:31:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.123+0000 7fc47f7fe640 10 client.26654 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.123+0000 7fc49f7fe640 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562e3fa71880 0x562e3fa73ca0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eac7b3f68ed4ebb7 server_cookie=68a369255e0c48eb in_seq=0 out_seq=0 2024-04-23T18:31:50.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc47f7fe640 1 -- 172.21.15.50:0/367370125 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fc4940a56e0 con 0x562e3fa71880 2024-04-23T18:31:50.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc47f7fe640 10 client.26654 handle_command_reply: tid=0 2024-04-23T18:31:50.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc4a921e740 1 client.26654 shutdown 2024-04-23T18:31:50.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc4a921e740 1 -- 172.21.15.50:0/367370125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 msgr2=0x562e3f9f4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:50.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc4a921e740 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9f4730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc490009900 tx=0x7fc490009680 comp rx=0 tx=0).stop 2024-04-23T18:31:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc4a921e740 1 -- 172.21.15.50:0/367370125 shutdown_connections 2024-04-23T18:31:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc4a921e740 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562e3fa71880 0x562e3fa73ca0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc4a921e740 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562e3f9e97d0 0x562e3fa6f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc4a921e740 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562e3f9def10 0x562e3f9f4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc4a921e740 1 --2- 172.21.15.50:0/367370125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e3f9de5f0 0x562e3f9f4730 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc4a921e740 1 -- 172.21.15.50:0/367370125 >> 172.21.15.50:0/367370125 conn(0x562e3f9dab00 msgr2=0x562e3f9e7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc47f7fe640 0 client.26654 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:31:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc4a921e740 1 -- 172.21.15.50:0/367370125 shutdown_connections 2024-04-23T18:31:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc4a921e740 1 -- 172.21.15.50:0/367370125 wait complete. 2024-04-23T18:31:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.376+0000 7fc4a921e740 20 client.26654 trim_cache size 0 max 16384 2024-04-23T18:31:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:31:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:31:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:31:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:31:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.377+0000 7fc4a6508640 1 -- 172.21.15.50:0/1832079824 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4880a3e40 msgr2=0x7fc4880a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.377+0000 7fc4a6508640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4880a3e40 0x7fc4880a6300 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc498022000 tx=0x7fc498023000 comp rx=0 tx=0).stop 2024-04-23T18:31:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.377+0000 7fc4a6508640 1 -- 172.21.15.50:0/1832079824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a0100e10 msgr2=0x7fc4a01a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:31:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.377+0000 7fc4a6508640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a0100e10 0x7fc4a01a0b80 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc49400e9f0 tx=0x7fc49400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.378+0000 7fc4a6508640 1 -- 172.21.15.50:0/1832079824 shutdown_connections 2024-04-23T18:31:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.378+0000 7fc4a6508640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4880a3e40 0x7fc4880a6300 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.378+0000 7fc4a6508640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4a0106f30 0x7fc4a019a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.378+0000 7fc4a6508640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4a0100e10 0x7fc4a01a0b80 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.378+0000 7fc4a6508640 1 --2- 172.21.15.50:0/1832079824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4a0100450 0x7fc4a01a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:31:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.378+0000 7fc4a6508640 1 -- 172.21.15.50:0/1832079824 >> 172.21.15.50:0/1832079824 conn(0x7fc4a00773e0 msgr2=0x7fc4a0108460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:31:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.378+0000 7fc4a6508640 1 -- 172.21.15.50:0/1832079824 shutdown_connections 2024-04-23T18:31:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:31:50.378+0000 7fc4a6508640 1 -- 172.21.15.50:0/1832079824 wait complete. 2024-04-23T18:31:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:50 smithi177 ceph-mon[32308]: pgmap v419: 97 pgs: 97 active+clean; 298 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 550 op/s 2024-04-23T18:31:50.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:50 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:31:50.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 14436 39.54 MB/sec warmup 24 sec latency 188.929 ms 2024-04-23T18:31:50.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:50 smithi188 ceph-mon[32694]: pgmap v419: 97 pgs: 97 active+clean; 298 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 550 op/s 2024-04-23T18:31:50.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:50 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:31:50.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:50 smithi050 ceph-mon[29187]: pgmap v419: 97 pgs: 97 active+clean; 298 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 550 op/s 2024-04-23T18:31:50.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:50 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:31:51.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 15657 40.03 MB/sec warmup 25 sec latency 42.258 ms 2024-04-23T18:31:51.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:51 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:31:51.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:51 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:31:51.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:51 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:31:51.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:51 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:31:51.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:51 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:31:51.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:51 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:31:51.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:51 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:31:51.787 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:51 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:31:51.787 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:51 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:31:52.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 16790 42.02 MB/sec warmup 26 sec latency 51.148 ms 2024-04-23T18:31:52.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:52 smithi188 ceph-mon[32694]: pgmap v420: 97 pgs: 97 active+clean; 298 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 550 op/s 2024-04-23T18:31:52.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:52 smithi050 ceph-mon[29187]: pgmap v420: 97 pgs: 97 active+clean; 298 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 550 op/s 2024-04-23T18:31:52.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:52 smithi177 ceph-mon[32308]: pgmap v420: 97 pgs: 97 active+clean; 298 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 550 op/s 2024-04-23T18:31:53.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 17232 40.83 MB/sec warmup 27 sec latency 66.718 ms 2024-04-23T18:31:54.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 18053 40.98 MB/sec warmup 28 sec latency 120.144 ms 2024-04-23T18:31:54.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:54 smithi188 ceph-mon[32694]: pgmap v421: 97 pgs: 97 active+clean; 305 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 7.6 MiB/s wr, 886 op/s 2024-04-23T18:31:54.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:54 smithi050 ceph-mon[29187]: pgmap v421: 97 pgs: 97 active+clean; 305 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 7.6 MiB/s wr, 886 op/s 2024-04-23T18:31:54.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:54 smithi177 ceph-mon[32308]: pgmap v421: 97 pgs: 97 active+clean; 305 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 7.6 MiB/s wr, 886 op/s 2024-04-23T18:31:55.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 19139 41.15 MB/sec warmup 29 sec latency 67.389 ms 2024-04-23T18:31:56.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 20152 42.47 MB/sec warmup 30 sec latency 40.319 ms 2024-04-23T18:31:56.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:56 smithi188 ceph-mon[32694]: pgmap v422: 97 pgs: 97 active+clean; 305 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.8 MiB/s wr, 537 op/s 2024-04-23T18:31:56.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:56 smithi050 ceph-mon[29187]: pgmap v422: 97 pgs: 97 active+clean; 305 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.8 MiB/s wr, 537 op/s 2024-04-23T18:31:56.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:56 smithi177 ceph-mon[32308]: pgmap v422: 97 pgs: 97 active+clean; 305 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.8 MiB/s wr, 537 op/s 2024-04-23T18:31:57.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 20570 41.50 MB/sec warmup 31 sec latency 77.031 ms 2024-04-23T18:31:58.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 21211 40.96 MB/sec warmup 32 sec latency 90.519 ms 2024-04-23T18:31:58.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:31:58 smithi188 ceph-mon[32694]: pgmap v423: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 8.9 MiB/s wr, 748 op/s 2024-04-23T18:31:58.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:31:58 smithi050 ceph-mon[29187]: pgmap v423: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 8.9 MiB/s wr, 748 op/s 2024-04-23T18:31:58.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:31:58 smithi177 ceph-mon[32308]: pgmap v423: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 8.9 MiB/s wr, 748 op/s 2024-04-23T18:31:59.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 22394 41.17 MB/sec warmup 33 sec latency 96.100 ms 2024-04-23T18:32:00.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 22779 41.01 MB/sec warmup 34 sec latency 367.437 ms 2024-04-23T18:32:00.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:00 smithi188 ceph-mon[32694]: pgmap v424: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 545 op/s 2024-04-23T18:32:00.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:00 smithi050 ceph-mon[29187]: pgmap v424: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 545 op/s 2024-04-23T18:32:00.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:00 smithi177 ceph-mon[32308]: pgmap v424: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 545 op/s 2024-04-23T18:32:01.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 23539 41.60 MB/sec warmup 35 sec latency 231.332 ms 2024-04-23T18:32:02.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 23926 40.88 MB/sec warmup 36 sec latency 103.864 ms 2024-04-23T18:32:02.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:02 smithi188 ceph-mon[32694]: pgmap v425: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 545 op/s 2024-04-23T18:32:02.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:02 smithi050 ceph-mon[29187]: pgmap v425: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 545 op/s 2024-04-23T18:32:02.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:02 smithi177 ceph-mon[32308]: pgmap v425: 97 pgs: 97 active+clean; 328 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 545 op/s 2024-04-23T18:32:03.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.487+0000 7fd1a39af640 1 -- 172.21.15.50:0/786195744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd19c101aa0 msgr2=0x7fd19c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:03.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.487+0000 7fd1a39af640 1 --2- 172.21.15.50:0/786195744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd19c101aa0 0x7fd19c105a90 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009900 tx=0x7fd18c02a410 comp rx=0 tx=0).stop 2024-04-23T18:32:03.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.487+0000 7fd1a39af640 1 -- 172.21.15.50:0/786195744 shutdown_connections 2024-04-23T18:32:03.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.487+0000 7fd1a39af640 1 --2- 172.21.15.50:0/786195744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd19c106170 0x7fd19c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.487+0000 7fd1a39af640 1 --2- 172.21.15.50:0/786195744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd19c101aa0 0x7fd19c105a90 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.487+0000 7fd1a39af640 1 --2- 172.21.15.50:0/786195744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd19c1010f0 0x7fd19c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.487+0000 7fd1a39af640 1 -- 172.21.15.50:0/786195744 >> 172.21.15.50:0/786195744 conn(0x7fd19c077fd0 msgr2=0x7fd19c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.488+0000 7fd1a39af640 1 -- 172.21.15.50:0/786195744 shutdown_connections 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.488+0000 7fd1a39af640 1 -- 172.21.15.50:0/786195744 wait complete. 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.488+0000 7fd1a39af640 1 Processor -- start 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.488+0000 7fd1a39af640 1 -- start start 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.488+0000 7fd1a39af640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd19c1010f0 0x7fd19c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.488+0000 7fd1a39af640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd19c106170 0x7fd19c198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.488+0000 7fd1a39af640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd19c19cf90 0x7fd19c19d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.488+0000 7fd1a39af640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c111090 con 0x7fd19c1010f0 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.488+0000 7fd1a39af640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c110f10 con 0x7fd19c106170 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.488+0000 7fd1a39af640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c111210 con 0x7fd19c19cf90 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a174b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd19c1010f0 0x7fd19c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a0f4a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd19c106170 0x7fd19c198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a174b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd19c1010f0 0x7fd19c1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43056/0 (socket says 172.21.15.50:43056) 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a174b640 1 -- 172.21.15.50:0/2442426988 learned_addr learned my addr 172.21.15.50:0/2442426988 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a1f4c640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd19c19cf90 0x7fd19c19d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a174b640 1 -- 172.21.15.50:0/2442426988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd19c19cf90 msgr2=0x7fd19c19d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a174b640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd19c19cf90 0x7fd19c19d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a174b640 1 -- 172.21.15.50:0/2442426988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd19c106170 msgr2=0x7fd19c198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a174b640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd19c106170 0x7fd19c198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a174b640 1 -- 172.21.15.50:0/2442426988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19c19da80 con 0x7fd19c1010f0 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a0f4a640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd19c106170 0x7fd19c198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a174b640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd19c1010f0 0x7fd19c1986d0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fd19800ea60 tx=0x7fd19800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1927fc640 1 -- 172.21.15.50:0/2442426988 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19800cde0 con 0x7fd19c1010f0 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1a39af640 1 -- 172.21.15.50:0/2442426988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd19c19dd70 con 0x7fd19c1010f0 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1927fc640 1 -- 172.21.15.50:0/2442426988 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1980035f0 con 0x7fd19c1010f0 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.489+0000 7fd1927fc640 1 -- 172.21.15.50:0/2442426988 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd198010690 con 0x7fd19c1010f0 2024-04-23T18:32:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.490+0000 7fd1a39af640 1 -- 172.21.15.50:0/2442426988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd19c1a5580 con 0x7fd19c1010f0 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.492+0000 7fd1927fc640 1 -- 172.21.15.50:0/2442426988 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd1980037a0 con 0x7fd19c1010f0 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a679c740 1 Processor -- start 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a679c740 1 -- start start 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a679c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6f99a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a679c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6f99f40 0x557ec6fa42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a679c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 0x557ec6fa6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a679c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557ec6efe970 con 0x557ec6f99620 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a679c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ec6e9b4a0 con 0x557ec6fa4800 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a679c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557ec6efa930 con 0x557ec6f99f40 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1927fc640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd1880a3e40 0x7fd1880a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a1f4c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6f99a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a0f4a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 0x557ec6fa6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a174b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6f99f40 0x557ec6fa42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a0f4a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 0x557ec6fa6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:51438/0 (socket says 172.21.15.50:51438) 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a0f4a640 1 -- 172.21.15.50:0/349971842 learned_addr learned my addr 172.21.15.50:0/349971842 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a1f4c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6f99a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43066/0 (socket says 172.21.15.50:43066) 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a1f4c640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd1880a3e40 0x7fd1880a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1927fc640 1 -- 172.21.15.50:0/2442426988 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8542+0+0 (secure 0 0 0) 0x7fd1980d0a30 con 0x7fd19c1010f0 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a0f4a640 1 -- 172.21.15.50:0/349971842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6f99f40 msgr2=0x557ec6fa42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a0f4a640 1 --2- 172.21.15.50:0/349971842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6f99f40 0x557ec6fa42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a0f4a640 1 -- 172.21.15.50:0/349971842 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 msgr2=0x557ec6f99a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.493+0000 7fd1a0f4a640 1 --2- 172.21.15.50:0/349971842 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6f99a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.494+0000 7fd1a0f4a640 1 -- 172.21.15.50:0/349971842 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ec6c11210 con 0x557ec6fa4800 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.494+0000 7fd1a0f4a640 1 --2- 172.21.15.50:0/349971842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 0x557ec6fa6bf0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009900 tx=0x7fd18c032f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2845ac914b72c185 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.494+0000 7fd1a1f4c640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd1880a3e40 0x7fd1880a6300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd194005fd0 tx=0x7fd194009500 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.494+0000 7fd1877fe640 1 -- 172.21.15.50:0/349971842 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18c040070 con 0x557ec6fa4800 2024-04-23T18:32:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.494+0000 7fd1877fe640 1 -- 172.21.15.50:0/349971842 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd18c031720 con 0x557ec6fa4800 2024-04-23T18:32:03.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.494+0000 7fd1877fe640 1 -- 172.21.15.50:0/349971842 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18c035040 con 0x557ec6fa4800 2024-04-23T18:32:03.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.495+0000 7fd1a679c740 1 -- 172.21.15.50:0/349971842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 msgr2=0x557ec6fa6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.495+0000 7fd1a679c740 1 --2- 172.21.15.50:0/349971842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 0x557ec6fa6bf0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009900 tx=0x7fd18c032f80 comp rx=0 tx=0).stop 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.495+0000 7fd1a679c740 1 -- 172.21.15.50:0/349971842 shutdown_connections 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.495+0000 7fd1a679c740 1 --2- 172.21.15.50:0/349971842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 0x557ec6fa6bf0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.495+0000 7fd1a679c740 1 --2- 172.21.15.50:0/349971842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6f99f40 0x557ec6fa42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.495+0000 7fd1a679c740 1 --2- 172.21.15.50:0/349971842 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6f99a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.495+0000 7fd1a679c740 1 -- 172.21.15.50:0/349971842 >> 172.21.15.50:0/349971842 conn(0x557ec6f95b30 msgr2=0x557ec6f95f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.495+0000 7fd1a679c740 1 -- 172.21.15.50:0/349971842 shutdown_connections 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.495+0000 7fd1a679c740 1 -- 172.21.15.50:0/349971842 wait complete. 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a679c740 1 -- start start 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a679c740 1 Processor -- start 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a679c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6fa4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a679c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 0x557ec6fa1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a679c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6fa4800 0x557ec6fa1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a679c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557ec6fa73d0 con 0x557ec6f99620 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a679c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ec6e990b0 con 0x557ec6f99f40 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a679c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557ec6fa7550 con 0x557ec6fa4800 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a679c740 10 client.0 _ll_register_callbacks cb 0x557ec6e86d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a174b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6fa4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a1f4c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6fa4800 0x557ec6fa1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a0f4a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 0x557ec6fa1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a174b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6fa4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43078/0 (socket says 172.21.15.50:43078) 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a0f4a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 0x557ec6fa1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:51444/0 (socket says 172.21.15.50:51444) 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a0f4a640 1 -- 172.21.15.50:0/4128298134 learned_addr learned my addr 172.21.15.50:0/4128298134 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.496+0000 7fd1a1f4c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6fa4800 0x557ec6fa1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:51692/0 (socket says 172.21.15.50:51692) 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd1a0f4a640 1 -- 172.21.15.50:0/4128298134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6fa4800 msgr2=0x557ec6fa1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd1a0f4a640 1 --2- 172.21.15.50:0/4128298134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6fa4800 0x557ec6fa1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd1a0f4a640 1 -- 172.21.15.50:0/4128298134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 msgr2=0x557ec6fa4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd1a0f4a640 1 --2- 172.21.15.50:0/4128298134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6fa4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd1a0f4a640 1 -- 172.21.15.50:0/4128298134 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ec6cdd620 con 0x557ec6f99f40 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd1a174b640 1 --2- 172.21.15.50:0/4128298134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6fa4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd1a0f4a640 1 --2- 172.21.15.50:0/4128298134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 0x557ec6fa1410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd18c004620 tx=0x7fd18c02ab10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7a08be2fc010649 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd184ff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd184ff9640 1 -- 172.21.15.50:0/4128298134 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18c04b070 con 0x557ec6f99f40 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd184ff9640 1 -- 172.21.15.50:0/4128298134 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8542+0+0 (secure 0 0 0) 0x7fd18c03c650 con 0x557ec6f99f40 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd1a679c740 1 -- 172.21.15.50:0/4128298134 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd18c031720 con 0x557ec6f99f40 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd184ff9640 1 -- 172.21.15.50:0/4128298134 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fd18c057050 con 0x557ec6f99f40 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd1a679c740 10 client.26666 fetch_fsmap learned FSMap version 50 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.497+0000 7fd1a679c740 1 -- 172.21.15.50:0/4128298134 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x557ec6fb32b0 con 0x557ec6f99f40 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.498+0000 7fd184ff9640 1 -- 172.21.15.50:0/4128298134 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd18c031720 con 0x557ec6f99f40 2024-04-23T18:32:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.498+0000 7fd184ff9640 1 -- 172.21.15.50:0/4128298134 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7fd18c050820 con 0x557ec6f99f40 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.498+0000 7fd1a679c740 10 client.26666 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.498+0000 7fd1a679c740 10 client.26666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.498+0000 7fd1a679c740 20 client.26666 populate_metadata read hostname 'smithi050' 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.498+0000 7fd1a679c740 1 --2- 172.21.15.50:0/4128298134 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557ec6fb4600 0x557ec6fb69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.498+0000 7fd1a679c740 4 client.26666 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.498+0000 7fd1a679c740 1 -- 172.21.15.50:0/4128298134 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ec6f0c390 con 0x557ec6fb4600 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.498+0000 7fd1a174b640 1 --2- 172.21.15.50:0/4128298134 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557ec6fb4600 0x557ec6fb69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.499+0000 7fd1a174b640 1 --2- 172.21.15.50:0/4128298134 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557ec6fb4600 0x557ec6fb69f0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99b747b438ae80e5 server_cookie=eb8e72b87be55697 in_seq=0 out_seq=0 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.499+0000 7fd184ff9640 10 client.26666 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd184ff9640 1 -- 172.21.15.50:0/4128298134 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557ec6f0c390 con 0x557ec6fb4600 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd184ff9640 10 client.26666 handle_command_reply: tid=0 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd1a679c740 1 client.26666 shutdown 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd1a679c740 1 -- 172.21.15.50:0/4128298134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 msgr2=0x557ec6fa1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd1a679c740 1 --2- 172.21.15.50:0/4128298134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 0x557ec6fa1410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd18c004620 tx=0x7fd18c02ab10 comp rx=0 tx=0).stop 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd1a679c740 1 -- 172.21.15.50:0/4128298134 shutdown_connections 2024-04-23T18:32:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd1a679c740 1 --2- 172.21.15.50:0/4128298134 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557ec6fb4600 0x557ec6fb69f0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd184ff9640 0 client.26666 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:03.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd1a679c740 1 --2- 172.21.15.50:0/4128298134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6fa4800 0x557ec6fa1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd1a679c740 1 --2- 172.21.15.50:0/4128298134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 0x557ec6fa1410 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd1a679c740 1 --2- 172.21.15.50:0/4128298134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6fa4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd1a679c740 1 -- 172.21.15.50:0/4128298134 >> 172.21.15.50:0/4128298134 conn(0x557ec6f95b30 msgr2=0x557ec6f9b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:03.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd1a679c740 1 -- 172.21.15.50:0/4128298134 shutdown_connections 2024-04-23T18:32:03.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd1a679c740 1 -- 172.21.15.50:0/4128298134 wait complete. 2024-04-23T18:32:03.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.500+0000 7fd1a679c740 20 client.26666 trim_cache size 0 max 16384 2024-04-23T18:32:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a679c740 1 Processor -- start 2024-04-23T18:32:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a679c740 1 -- start start 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a679c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6fb0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a679c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6f99f40 0x557ec6fbbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a679c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 0x557ec6fbc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a679c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1b04e0 con 0x557ec6f99620 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a679c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1adcf0 con 0x557ec6fa4800 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a679c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1abb60 con 0x557ec6f99f40 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a174b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6fb0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a1f4c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 0x557ec6fbc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a1f4c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 0x557ec6fbc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:51454/0 (socket says 172.21.15.50:51454) 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a1f4c640 1 -- 172.21.15.50:0/2305290683 learned_addr learned my addr 172.21.15.50:0/2305290683 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a0f4a640 1 --2- 172.21.15.50:0/2305290683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6f99f40 0x557ec6fbbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a1f4c640 1 -- 172.21.15.50:0/2305290683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6f99f40 msgr2=0x557ec6fbbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a1f4c640 1 --2- 172.21.15.50:0/2305290683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6f99f40 0x557ec6fbbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a1f4c640 1 -- 172.21.15.50:0/2305290683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 msgr2=0x557ec6fb0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a1f4c640 1 --2- 172.21.15.50:0/2305290683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6fb0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.518+0000 7fd1a1f4c640 1 -- 172.21.15.50:0/2305290683 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ec6fa1c80 con 0x557ec6fa4800 2024-04-23T18:32:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.519+0000 7fd1a1f4c640 1 --2- 172.21.15.50:0/2305290683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 0x557ec6fbc500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd194009de0 tx=0x7fd194008000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4522ac77ba484e8e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.519+0000 7fd186ffd640 1 -- 172.21.15.50:0/2305290683 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194011c70 con 0x557ec6fa4800 2024-04-23T18:32:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.519+0000 7fd186ffd640 1 -- 172.21.15.50:0/2305290683 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd194011e10 con 0x557ec6fa4800 2024-04-23T18:32:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.519+0000 7fd186ffd640 1 -- 172.21.15.50:0/2305290683 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194020780 con 0x557ec6fa4800 2024-04-23T18:32:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.520+0000 7fd1a679c740 1 -- 172.21.15.50:0/2305290683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 msgr2=0x557ec6fbc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.520+0000 7fd1a679c740 1 --2- 172.21.15.50:0/2305290683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 0x557ec6fbc500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd194009de0 tx=0x7fd194008000 comp rx=0 tx=0).stop 2024-04-23T18:32:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.520+0000 7fd1a679c740 1 -- 172.21.15.50:0/2305290683 shutdown_connections 2024-04-23T18:32:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.520+0000 7fd1a679c740 1 --2- 172.21.15.50:0/2305290683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6fa4800 0x557ec6fbc500 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.520+0000 7fd1a679c740 1 --2- 172.21.15.50:0/2305290683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6f99f40 0x557ec6fbbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.520+0000 7fd1a679c740 1 --2- 172.21.15.50:0/2305290683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6fb0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.520+0000 7fd1a679c740 1 -- 172.21.15.50:0/2305290683 >> 172.21.15.50:0/2305290683 conn(0x557ec6f95b30 msgr2=0x557ec6f94420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.520+0000 7fd1a679c740 1 -- 172.21.15.50:0/2305290683 shutdown_connections 2024-04-23T18:32:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.520+0000 7fd1a679c740 1 -- 172.21.15.50:0/2305290683 wait complete. 2024-04-23T18:32:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.520+0000 7fd1a679c740 1 -- start start 2024-04-23T18:32:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.520+0000 7fd1a679c740 1 Processor -- start 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a679c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6facc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a679c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 0x557ec6fa9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a679c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6fa4800 0x557ec6faa5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a679c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1ac1b0 con 0x557ec6f99620 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a679c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1ad620 con 0x557ec6f99f40 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a679c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1ac880 con 0x557ec6fa4800 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a679c740 10 client.0 _ll_register_callbacks cb 0x7fd18c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a174b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6facc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a174b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6facc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43098/0 (socket says 172.21.15.50:43098) 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a174b640 1 -- 172.21.15.50:0/1509570846 learned_addr learned my addr 172.21.15.50:0/1509570846 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a0f4a640 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 0x557ec6fa9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a1f4c640 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6fa4800 0x557ec6faa5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a174b640 1 -- 172.21.15.50:0/1509570846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6fa4800 msgr2=0x557ec6faa5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a174b640 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6fa4800 0x557ec6faa5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a174b640 1 -- 172.21.15.50:0/1509570846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 msgr2=0x557ec6fa9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a174b640 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 0x557ec6fa9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a174b640 1 -- 172.21.15.50:0/1509570846 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ec6faadd0 con 0x557ec6f99620 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a0f4a640 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 0x557ec6fa9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a174b640 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6facc30 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fd198098e20 tx=0x7fd19806c2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42b21702696c05f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd184ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd184ff9640 1 -- 172.21.15.50:0/1509570846 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1980a0d70 con 0x557ec6f99620 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd184ff9640 1 -- 172.21.15.50:0/1509570846 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8542+0+0 (secure 0 0 0) 0x7fd198004000 con 0x557ec6f99620 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.521+0000 7fd1a679c740 1 -- 172.21.15.50:0/1509570846 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd194011e10 con 0x557ec6f99620 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.522+0000 7fd184ff9640 1 -- 172.21.15.50:0/1509570846 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fd19806b1b0 con 0x557ec6f99620 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.522+0000 7fd1a679c740 10 client.17163 fetch_fsmap learned FSMap version 50 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.522+0000 7fd1a679c740 1 -- 172.21.15.50:0/1509570846 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x557ec701a980 con 0x557ec6f99620 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.522+0000 7fd184ff9640 1 -- 172.21.15.50:0/1509570846 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd194011e10 con 0x557ec6f99620 2024-04-23T18:32:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.522+0000 7fd184ff9640 1 -- 172.21.15.50:0/1509570846 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7fd1980bbb80 con 0x557ec6f99620 2024-04-23T18:32:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.522+0000 7fd1a679c740 10 client.17163 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:32:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.522+0000 7fd1a679c740 10 client.17163 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:32:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.522+0000 7fd1a679c740 20 client.17163 populate_metadata read hostname 'smithi050' 2024-04-23T18:32:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.522+0000 7fd1a679c740 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557ec701b0b0 0x557ec701d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.522+0000 7fd1a679c740 4 client.17163 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:32:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.522+0000 7fd1a679c740 1 -- 172.21.15.50:0/1509570846 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557ec6f06200 con 0x557ec701b0b0 2024-04-23T18:32:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.522+0000 7fd1a0f4a640 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557ec701b0b0 0x557ec701d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.523+0000 7fd184ff9640 10 client.17163 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.523+0000 7fd1a0f4a640 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557ec701b0b0 0x557ec701d4d0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b0657f18ed86ade server_cookie=d8a5c4d5f3a7d693 in_seq=0 out_seq=0 2024-04-23T18:32:03.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 24499 40.26 MB/sec warmup 37 sec latency 97.190 ms 2024-04-23T18:32:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.820+0000 7fd184ff9640 1 -- 172.21.15.50:0/1509570846 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557ec6f06200 con 0x557ec701b0b0 2024-04-23T18:32:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.820+0000 7fd184ff9640 10 client.17163 handle_command_reply: tid=0 2024-04-23T18:32:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd1a679c740 1 client.17163 shutdown 2024-04-23T18:32:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd1a679c740 1 -- 172.21.15.50:0/1509570846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 msgr2=0x557ec6facc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd1a679c740 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6facc30 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fd198098e20 tx=0x7fd19806c2a0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd1a679c740 1 -- 172.21.15.50:0/1509570846 shutdown_connections 2024-04-23T18:32:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd1a679c740 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557ec701b0b0 0x557ec701d4d0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd184ff9640 0 client.17163 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd1a679c740 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557ec6fa4800 0x557ec6faa5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd1a679c740 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6f99f40 0x557ec6fa9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd1a679c740 1 --2- 172.21.15.50:0/1509570846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557ec6f99620 0x557ec6facc30 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd1a679c740 1 -- 172.21.15.50:0/1509570846 >> 172.21.15.50:0/1509570846 conn(0x557ec6f95b30 msgr2=0x557ec6f94cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd1a679c740 1 -- 172.21.15.50:0/1509570846 shutdown_connections 2024-04-23T18:32:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd1a679c740 1 -- 172.21.15.50:0/1509570846 wait complete. 2024-04-23T18:32:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.821+0000 7fd1a679c740 20 client.17163 trim_cache size 0 max 16384 2024-04-23T18:32:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:32:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:32:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:32:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:32:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.822+0000 7fd1a39af640 1 -- 172.21.15.50:0/2442426988 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd1880a3e40 msgr2=0x7fd1880a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.822+0000 7fd1a39af640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd1880a3e40 0x7fd1880a6300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd194005fd0 tx=0x7fd194009500 comp rx=0 tx=0).stop 2024-04-23T18:32:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.822+0000 7fd1a39af640 1 -- 172.21.15.50:0/2442426988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd19c1010f0 msgr2=0x7fd19c1986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.822+0000 7fd1a39af640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd19c1010f0 0x7fd19c1986d0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fd19800ea60 tx=0x7fd19800ef20 comp rx=0 tx=0).stop 2024-04-23T18:32:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.822+0000 7fd1a39af640 1 -- 172.21.15.50:0/2442426988 shutdown_connections 2024-04-23T18:32:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.822+0000 7fd1a39af640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd1880a3e40 0x7fd1880a6300 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.822+0000 7fd1a39af640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd19c19cf90 0x7fd19c19d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.822+0000 7fd1a39af640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd19c106170 0x7fd19c198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.822+0000 7fd1a39af640 1 --2- 172.21.15.50:0/2442426988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd19c1010f0 0x7fd19c1986d0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.822+0000 7fd1a39af640 1 -- 172.21.15.50:0/2442426988 >> 172.21.15.50:0/2442426988 conn(0x7fd19c077fd0 msgr2=0x7fd19c069940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.822+0000 7fd1a39af640 1 -- 172.21.15.50:0/2442426988 shutdown_connections 2024-04-23T18:32:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:03.822+0000 7fd1a39af640 1 -- 172.21.15.50:0/2442426988 wait complete. 2024-04-23T18:32:04.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 25075 40.00 MB/sec warmup 38 sec latency 181.307 ms 2024-04-23T18:32:04.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:04 smithi188 ceph-mon[32694]: pgmap v426: 97 pgs: 97 active+clean; 530 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 24 MiB/s wr, 816 op/s 2024-04-23T18:32:04.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:04 smithi050 ceph-mon[29187]: pgmap v426: 97 pgs: 97 active+clean; 530 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 24 MiB/s wr, 816 op/s 2024-04-23T18:32:04.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:04 smithi177 ceph-mon[32308]: pgmap v426: 97 pgs: 97 active+clean; 530 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 24 MiB/s wr, 816 op/s 2024-04-23T18:32:05.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 25792 39.61 MB/sec warmup 39 sec latency 299.380 ms 2024-04-23T18:32:05.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:05 smithi188 ceph-mon[32694]: osdmap e108: 12 total, 12 up, 12 in 2024-04-23T18:32:05.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:05 smithi050 ceph-mon[29187]: osdmap e108: 12 total, 12 up, 12 in 2024-04-23T18:32:05.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:05 smithi177 ceph-mon[32308]: osdmap e108: 12 total, 12 up, 12 in 2024-04-23T18:32:06.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 26734 40.53 MB/sec warmup 40 sec latency 40.502 ms 2024-04-23T18:32:06.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:06 smithi188 ceph-mon[32694]: pgmap v428: 97 pgs: 97 active+clean; 530 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 25 MiB/s wr, 577 op/s 2024-04-23T18:32:06.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:06 smithi050 ceph-mon[29187]: pgmap v428: 97 pgs: 97 active+clean; 530 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 25 MiB/s wr, 577 op/s 2024-04-23T18:32:06.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:06 smithi177 ceph-mon[32308]: pgmap v428: 97 pgs: 97 active+clean; 530 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 25 MiB/s wr, 577 op/s 2024-04-23T18:32:07.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 27233 40.24 MB/sec warmup 41 sec latency 51.742 ms 2024-04-23T18:32:08.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 27717 39.53 MB/sec warmup 42 sec latency 52.534 ms 2024-04-23T18:32:08.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:08 smithi188 ceph-mon[32694]: pgmap v429: 97 pgs: 97 active+clean; 521 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 26 MiB/s wr, 806 op/s 2024-04-23T18:32:08.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:08 smithi050 ceph-mon[29187]: pgmap v429: 97 pgs: 97 active+clean; 521 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 26 MiB/s wr, 806 op/s 2024-04-23T18:32:08.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:08 smithi177 ceph-mon[32308]: pgmap v429: 97 pgs: 97 active+clean; 521 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 26 MiB/s wr, 806 op/s 2024-04-23T18:32:09.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 28654 39.73 MB/sec warmup 43 sec latency 123.019 ms 2024-04-23T18:32:10.554 INFO:tasks.workunit.client.0.smithi050.stdout: 10 29540 39.97 MB/sec warmup 44 sec latency 237.573 ms 2024-04-23T18:32:10.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:10 smithi188 ceph-mon[32694]: pgmap v430: 97 pgs: 97 active+clean; 521 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 26 MiB/s wr, 806 op/s 2024-04-23T18:32:10.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:10 smithi050 ceph-mon[29187]: pgmap v430: 97 pgs: 97 active+clean; 521 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 26 MiB/s wr, 806 op/s 2024-04-23T18:32:10.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:10 smithi177 ceph-mon[32308]: pgmap v430: 97 pgs: 97 active+clean; 521 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 26 MiB/s wr, 806 op/s 2024-04-23T18:32:11.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 30356 40.51 MB/sec warmup 45 sec latency 47.292 ms 2024-04-23T18:32:12.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:12 smithi050 ceph-mon[29187]: pgmap v431: 97 pgs: 97 active+clean; 521 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 26 MiB/s wr, 806 op/s 2024-04-23T18:32:12.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:12 smithi050 ceph-mon[29187]: osdmap e109: 12 total, 12 up, 12 in 2024-04-23T18:32:12.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:12 smithi177 ceph-mon[32308]: pgmap v431: 97 pgs: 97 active+clean; 521 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 26 MiB/s wr, 806 op/s 2024-04-23T18:32:12.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:12 smithi177 ceph-mon[32308]: osdmap e109: 12 total, 12 up, 12 in 2024-04-23T18:32:12.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 30776 39.95 MB/sec warmup 46 sec latency 34.137 ms 2024-04-23T18:32:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:12 smithi188 ceph-mon[32694]: pgmap v431: 97 pgs: 97 active+clean; 521 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 26 MiB/s wr, 806 op/s 2024-04-23T18:32:12.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:12 smithi188 ceph-mon[32694]: osdmap e109: 12 total, 12 up, 12 in 2024-04-23T18:32:13.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 31443 39.69 MB/sec warmup 47 sec latency 63.403 ms 2024-04-23T18:32:14.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:14 smithi050 ceph-mon[29187]: pgmap v433: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 965 op/s 2024-04-23T18:32:14.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:14 smithi177 ceph-mon[32308]: pgmap v433: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 965 op/s 2024-04-23T18:32:14.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 32516 39.75 MB/sec warmup 48 sec latency 92.945 ms 2024-04-23T18:32:14.581 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:14 smithi188 ceph-mon[32694]: pgmap v433: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 965 op/s 2024-04-23T18:32:15.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 33271 40.09 MB/sec warmup 49 sec latency 235.519 ms 2024-04-23T18:32:16.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:16 smithi050 ceph-mon[29187]: pgmap v434: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 811 op/s 2024-04-23T18:32:16.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:16 smithi177 ceph-mon[32308]: pgmap v434: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 811 op/s 2024-04-23T18:32:16.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 33853 40.22 MB/sec warmup 50 sec latency 351.275 ms 2024-04-23T18:32:16.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:16 smithi188 ceph-mon[32694]: pgmap v434: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 811 op/s 2024-04-23T18:32:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.927+0000 7f8656b43640 1 -- 172.21.15.50:0/239705505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8650101aa0 msgr2=0x7f8650105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.927+0000 7f8656b43640 1 --2- 172.21.15.50:0/239705505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8650101aa0 0x7f8650105a90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f863c009860 tx=0x7f863c02a340 comp rx=0 tx=0).stop 2024-04-23T18:32:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.927+0000 7f8656b43640 1 -- 172.21.15.50:0/239705505 shutdown_connections 2024-04-23T18:32:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.927+0000 7f8656b43640 1 --2- 172.21.15.50:0/239705505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8650106170 0x7f865010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.927+0000 7f8656b43640 1 --2- 172.21.15.50:0/239705505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8650101aa0 0x7f8650105a90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.927+0000 7f8656b43640 1 --2- 172.21.15.50:0/239705505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86501010f0 0x7f86501014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.927+0000 7f8656b43640 1 -- 172.21.15.50:0/239705505 >> 172.21.15.50:0/239705505 conn(0x7f8650077fd0 msgr2=0x7f86500783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.928+0000 7f8656b43640 1 -- 172.21.15.50:0/239705505 shutdown_connections 2024-04-23T18:32:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.928+0000 7f8656b43640 1 -- 172.21.15.50:0/239705505 wait complete. 2024-04-23T18:32:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.929+0000 7f8656b43640 1 Processor -- start 2024-04-23T18:32:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.929+0000 7f8656b43640 1 -- start start 2024-04-23T18:32:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.929+0000 7f8656b43640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86501010f0 0x7f8650109a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.929+0000 7f8656b43640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8650106170 0x7f8650109fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.929+0000 7f8656b43640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86501a3550 0x7f86501a3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.929+0000 7f8656b43640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8650110020 con 0x7f86501a3550 2024-04-23T18:32:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.929+0000 7f8656b43640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f865010fea0 con 0x7f8650106170 2024-04-23T18:32:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.929+0000 7f8656b43640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f86501101a0 con 0x7f86501010f0 2024-04-23T18:32:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.929+0000 7f864ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8650106170 0x7f8650109fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.929+0000 7f864ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8650106170 0x7f8650109fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:53590/0 (socket says 172.21.15.50:53590) 2024-04-23T18:32:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.929+0000 7f864ffff640 1 -- 172.21.15.50:0/1629883482 learned_addr learned my addr 172.21.15.50:0/1629883482 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.929+0000 7f86550e0640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86501a3550 0x7f86501a3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.930+0000 7f86550e0640 1 -- 172.21.15.50:0/1629883482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86501010f0 msgr2=0x7f8650109a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.930+0000 7f86548df640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86501010f0 0x7f8650109a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.930+0000 7f86550e0640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86501010f0 0x7f8650109a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.930+0000 7f86550e0640 1 -- 172.21.15.50:0/1629883482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8650106170 msgr2=0x7f8650109fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.930+0000 7f86550e0640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8650106170 0x7f8650109fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.930+0000 7f86550e0640 1 -- 172.21.15.50:0/1629883482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86501a4070 con 0x7f86501a3550 2024-04-23T18:32:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.930+0000 7f86548df640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86501010f0 0x7f8650109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:32:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.930+0000 7f86550e0640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86501a3550 0x7f86501a3a00 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f8648009fd0 tx=0x7f864800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.930+0000 7f864dffb640 1 -- 172.21.15.50:0/1629883482 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864800ce20 con 0x7f86501a3550 2024-04-23T18:32:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.930+0000 7f8656b43640 1 -- 172.21.15.50:0/1629883482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86501a4360 con 0x7f86501a3550 2024-04-23T18:32:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.930+0000 7f864dffb640 1 -- 172.21.15.50:0/1629883482 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8648005020 con 0x7f86501a3550 2024-04-23T18:32:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.930+0000 7f864dffb640 1 -- 172.21.15.50:0/1629883482 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648010690 con 0x7f86501a3550 2024-04-23T18:32:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.931+0000 7f8656b43640 1 -- 172.21.15.50:0/1629883482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86501abab0 con 0x7f86501a3550 2024-04-23T18:32:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.934+0000 7f8659856740 1 Processor -- start 2024-04-23T18:32:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.935+0000 7f864dffb640 1 -- 172.21.15.50:0/1629883482 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f86480054d0 con 0x7f86501a3550 2024-04-23T18:32:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.935+0000 7f8659856740 1 -- start start 2024-04-23T18:32:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.936+0000 7f8659856740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x5628923eca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.936+0000 7f8659856740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923ecf50 0x5628923f72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.936+0000 7f8659856740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923f7810 0x5628923f9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.936+0000 7f8659856740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562892351970 con 0x5628923ec630 2024-04-23T18:32:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.936+0000 7f8659856740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5628922ee4a0 con 0x5628923ecf50 2024-04-23T18:32:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.936+0000 7f8659856740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56289234d930 con 0x5628923f7810 2024-04-23T18:32:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.936+0000 7f864ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x5628923eca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.936+0000 7f864ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x5628923eca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59774/0 (socket says 172.21.15.50:59774) 2024-04-23T18:32:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.936+0000 7f864ffff640 1 -- 172.21.15.50:0/1685870847 learned_addr learned my addr 172.21.15.50:0/1685870847 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.936+0000 7f864dffb640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f86400a3ec0 0x7f86400a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.936+0000 7f864ffff640 1 --2- 172.21.15.50:0/1685870847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923f7810 0x5628923f9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.937+0000 7f864dffb640 1 -- 172.21.15.50:0/1629883482 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7f86480d0e80 con 0x7f86501a3550 2024-04-23T18:32:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.937+0000 7f86550e0640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f86400a3ec0 0x7f86400a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.937+0000 7f86548df640 1 --2- 172.21.15.50:0/1685870847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923ecf50 0x5628923f72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.937+0000 7f864ffff640 1 -- 172.21.15.50:0/1685870847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923f7810 msgr2=0x5628923f9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.937+0000 7f864ffff640 1 --2- 172.21.15.50:0/1685870847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923f7810 0x5628923f9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.937+0000 7f864ffff640 1 -- 172.21.15.50:0/1685870847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923ecf50 msgr2=0x5628923f72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.937+0000 7f864ffff640 1 --2- 172.21.15.50:0/1685870847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923ecf50 0x5628923f72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.937+0000 7f864ffff640 1 -- 172.21.15.50:0/1685870847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562892064210 con 0x5628923ec630 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.937+0000 7f86550e0640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f86400a3ec0 0x7f86400a6380 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f86480a54f0 tx=0x7f86480057c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.938+0000 7f864ffff640 1 --2- 172.21.15.50:0/1685870847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x5628923eca10 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x5628923e9cd0 tx=0x7f863c005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3dfb216d9e25 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.938+0000 7f863affd640 1 -- 172.21.15.50:0/1685870847 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c037040 con 0x5628923ec630 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.938+0000 7f863affd640 1 -- 172.21.15.50:0/1685870847 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f863c030900 con 0x5628923ec630 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.938+0000 7f863affd640 1 -- 172.21.15.50:0/1685870847 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c005480 con 0x5628923ec630 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.939+0000 7f8659856740 1 -- 172.21.15.50:0/1685870847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 msgr2=0x5628923eca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.939+0000 7f8659856740 1 --2- 172.21.15.50:0/1685870847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x5628923eca10 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x5628923e9cd0 tx=0x7f863c005710 comp rx=0 tx=0).stop 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.939+0000 7f8659856740 1 -- 172.21.15.50:0/1685870847 shutdown_connections 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.939+0000 7f8659856740 1 --2- 172.21.15.50:0/1685870847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923f7810 0x5628923f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.939+0000 7f8659856740 1 --2- 172.21.15.50:0/1685870847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923ecf50 0x5628923f72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.939+0000 7f8659856740 1 --2- 172.21.15.50:0/1685870847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x5628923eca10 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.939+0000 7f8659856740 1 -- 172.21.15.50:0/1685870847 >> 172.21.15.50:0/1685870847 conn(0x5628923e8b40 msgr2=0x5628923e8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.939+0000 7f8659856740 1 -- 172.21.15.50:0/1685870847 shutdown_connections 2024-04-23T18:32:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.939+0000 7f8659856740 1 -- 172.21.15.50:0/1685870847 wait complete. 2024-04-23T18:32:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.939+0000 7f8659856740 1 -- start start 2024-04-23T18:32:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.939+0000 7f8659856740 1 Processor -- start 2024-04-23T18:32:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f8659856740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x5628923f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f8659856740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ecf50 0x5628923f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f8659856740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923f7810 0x5628923f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f8659856740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5628923fa3e0 con 0x5628923ec630 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f8659856740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5628922ec0b0 con 0x5628923f7810 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f8659856740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5628923fa560 con 0x5628923ecf50 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f8659856740 10 client.0 _ll_register_callbacks cb 0x5628922d9d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f864ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x5628923f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f864ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x5628923f71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59786/0 (socket says 172.21.15.50:59786) 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f864ffff640 1 -- 172.21.15.50:0/2609171210 learned_addr learned my addr 172.21.15.50:0/2609171210 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f864ffff640 1 -- 172.21.15.50:0/2609171210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ecf50 msgr2=0x5628923f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f864ffff640 1 --2- 172.21.15.50:0/2609171210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ecf50 0x5628923f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f864ffff640 1 -- 172.21.15.50:0/2609171210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923f7810 msgr2=0x5628923f4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f864ffff640 1 --2- 172.21.15.50:0/2609171210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923f7810 0x5628923f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f864ffff640 1 -- 172.21.15.50:0/2609171210 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562892130620 con 0x5628923ec630 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f864ffff640 1 --2- 172.21.15.50:0/2609171210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x5628923f71e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f863c005780 tx=0x7f863c02aac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542fdbd0535298b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f862ffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.940+0000 7f862ffff640 1 -- 172.21.15.50:0/2609171210 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c037030 con 0x5628923ec630 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.941+0000 7f862ffff640 1 -- 172.21.15.50:0/2609171210 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7f863c004230 con 0x5628923ec630 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.941+0000 7f8659856740 1 -- 172.21.15.50:0/2609171210 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f863c030900 con 0x5628923ec630 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.941+0000 7f862ffff640 1 -- 172.21.15.50:0/2609171210 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f863c004430 con 0x5628923ec630 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.941+0000 7f8659856740 10 client.17175 fetch_fsmap learned FSMap version 50 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.941+0000 7f8659856740 1 -- 172.21.15.50:0/2609171210 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x562892406370 con 0x5628923ec630 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.941+0000 7f862ffff640 1 -- 172.21.15.50:0/2609171210 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f863c030900 con 0x5628923ec630 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.942+0000 7f862ffff640 1 -- 172.21.15.50:0/2609171210 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7f863c04a030 con 0x5628923ec630 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.942+0000 7f8659856740 10 client.17175 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.942+0000 7f8659856740 10 client.17175 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.942+0000 7f8659856740 20 client.17175 populate_metadata read hostname 'smithi050' 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.942+0000 7f8659856740 1 --2- 172.21.15.50:0/2609171210 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5628924076c0 0x562892409ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.942+0000 7f8659856740 4 client.17175 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:32:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.942+0000 7f8659856740 1 -- 172.21.15.50:0/2609171210 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56289235f390 con 0x5628924076c0 2024-04-23T18:32:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.942+0000 7f86548df640 1 --2- 172.21.15.50:0/2609171210 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5628924076c0 0x562892409ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.943+0000 7f862ffff640 10 client.17175 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.943+0000 7f86548df640 1 --2- 172.21.15.50:0/2609171210 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5628924076c0 0x562892409ab0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dea10980c26d3724 server_cookie=810736d17d255cf4 in_seq=0 out_seq=0 2024-04-23T18:32:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f862ffff640 1 -- 172.21.15.50:0/2609171210 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56289235f390 con 0x5628924076c0 2024-04-23T18:32:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f862ffff640 10 client.17175 handle_command_reply: tid=0 2024-04-23T18:32:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f8659856740 1 client.17175 shutdown 2024-04-23T18:32:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f8659856740 1 -- 172.21.15.50:0/2609171210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 msgr2=0x5628923f71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f8659856740 1 --2- 172.21.15.50:0/2609171210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x5628923f71e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f863c005780 tx=0x7f863c02aac0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f8659856740 1 -- 172.21.15.50:0/2609171210 shutdown_connections 2024-04-23T18:32:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f8659856740 1 --2- 172.21.15.50:0/2609171210 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5628924076c0 0x562892409ab0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f8659856740 1 --2- 172.21.15.50:0/2609171210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923f7810 0x5628923f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f8659856740 1 --2- 172.21.15.50:0/2609171210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ecf50 0x5628923f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f8659856740 1 --2- 172.21.15.50:0/2609171210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x5628923f71e0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f862ffff640 0 client.17175 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f8659856740 1 -- 172.21.15.50:0/2609171210 >> 172.21.15.50:0/2609171210 conn(0x5628923e8b40 msgr2=0x5628923e7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f8659856740 1 -- 172.21.15.50:0/2609171210 shutdown_connections 2024-04-23T18:32:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f8659856740 1 -- 172.21.15.50:0/2609171210 wait complete. 2024-04-23T18:32:16.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.944+0000 7f8659856740 20 client.17175 trim_cache size 0 max 16384 2024-04-23T18:32:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.967+0000 7f8659856740 1 Processor -- start 2024-04-23T18:32:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.967+0000 7f8659856740 1 -- start start 2024-04-23T18:32:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.968+0000 7f8659856740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ec630 0x5628923f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.968+0000 7f8659856740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923ecf50 0x56289240cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.968+0000 7f8659856740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923f7810 0x56289240d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.968+0000 7f8659856740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f86501b6900 con 0x5628923f7810 2024-04-23T18:32:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.968+0000 7f8659856740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f86501b4110 con 0x5628923ecf50 2024-04-23T18:32:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.968+0000 7f8659856740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f86501b1fa0 con 0x5628923ec630 2024-04-23T18:32:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.968+0000 7f864ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923f7810 0x56289240d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.968+0000 7f864ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923f7810 0x56289240d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59802/0 (socket says 172.21.15.50:59802) 2024-04-23T18:32:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.968+0000 7f864ffff640 1 -- 172.21.15.50:0/1084087216 learned_addr learned my addr 172.21.15.50:0/1084087216 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.968+0000 7f864ffff640 1 --2- 172.21.15.50:0/1084087216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ec630 0x5628923f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.968+0000 7f86548df640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923ecf50 0x56289240cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.969+0000 7f864ffff640 1 -- 172.21.15.50:0/1084087216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923ecf50 msgr2=0x56289240cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.969+0000 7f864ffff640 1 --2- 172.21.15.50:0/1084087216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923ecf50 0x56289240cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.969+0000 7f864ffff640 1 -- 172.21.15.50:0/1084087216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923f7810 msgr2=0x56289240d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.969+0000 7f864ffff640 1 --2- 172.21.15.50:0/1084087216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923f7810 0x56289240d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.969+0000 7f864ffff640 1 -- 172.21.15.50:0/1084087216 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628923f4e50 con 0x5628923ec630 2024-04-23T18:32:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.969+0000 7f864ffff640 1 --2- 172.21.15.50:0/1084087216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ec630 0x5628923f6d10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f863c031000 tx=0x7f863c005d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ae1c738ee3670c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.971+0000 7f863a7fc640 1 -- 172.21.15.50:0/1084087216 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c037070 con 0x5628923ec630 2024-04-23T18:32:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.971+0000 7f863a7fc640 1 -- 172.21.15.50:0/1084087216 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f863c005000 con 0x5628923ec630 2024-04-23T18:32:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.971+0000 7f863a7fc640 1 -- 172.21.15.50:0/1084087216 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c032d60 con 0x5628923ec630 2024-04-23T18:32:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.972+0000 7f8659856740 1 -- 172.21.15.50:0/1084087216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ec630 msgr2=0x5628923f6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.972+0000 7f8659856740 1 --2- 172.21.15.50:0/1084087216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ec630 0x5628923f6d10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f863c031000 tx=0x7f863c005d00 comp rx=0 tx=0).stop 2024-04-23T18:32:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.972+0000 7f8659856740 1 -- 172.21.15.50:0/1084087216 shutdown_connections 2024-04-23T18:32:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.972+0000 7f8659856740 1 --2- 172.21.15.50:0/1084087216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923f7810 0x56289240d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.972+0000 7f8659856740 1 --2- 172.21.15.50:0/1084087216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923ecf50 0x56289240cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.972+0000 7f8659856740 1 --2- 172.21.15.50:0/1084087216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ec630 0x5628923f6d10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.972+0000 7f8659856740 1 -- 172.21.15.50:0/1084087216 >> 172.21.15.50:0/1084087216 conn(0x5628923e8b40 msgr2=0x562892403040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.973+0000 7f8659856740 1 -- 172.21.15.50:0/1084087216 shutdown_connections 2024-04-23T18:32:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.973+0000 7f8659856740 1 -- 172.21.15.50:0/1084087216 wait complete. 2024-04-23T18:32:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.973+0000 7f8659856740 1 -- start start 2024-04-23T18:32:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.973+0000 7f8659856740 1 Processor -- start 2024-04-23T18:32:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.974+0000 7f8659856740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x56289240ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.974+0000 7f8659856740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ecf50 0x56289240a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.974+0000 7f8659856740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923f7810 0x56289240a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.974+0000 7f8659856740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f86501b25f0 con 0x5628923ec630 2024-04-23T18:32:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.974+0000 7f8659856740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f86501b3a40 con 0x5628923f7810 2024-04-23T18:32:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.974+0000 7f8659856740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f86501b2ca0 con 0x5628923ecf50 2024-04-23T18:32:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.974+0000 7f864ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x56289240ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.974+0000 7f864ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x56289240ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59816/0 (socket says 172.21.15.50:59816) 2024-04-23T18:32:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.974+0000 7f8659856740 10 client.0 _ll_register_callbacks cb 0x7f863c005970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.974+0000 7f864ffff640 1 -- 172.21.15.50:0/2116487193 learned_addr learned my addr 172.21.15.50:0/2116487193 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.974+0000 7f86548df640 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ecf50 0x56289240a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.974+0000 7f864ffff640 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923f7810 0x56289240a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.975+0000 7f864ffff640 1 -- 172.21.15.50:0/2116487193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ecf50 msgr2=0x56289240a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.975+0000 7f864ffff640 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ecf50 0x56289240a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.975+0000 7f864ffff640 1 -- 172.21.15.50:0/2116487193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923f7810 msgr2=0x56289240a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.975+0000 7f864ffff640 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923f7810 0x56289240a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.975+0000 7f864ffff640 1 -- 172.21.15.50:0/2116487193 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56289240af00 con 0x5628923ec630 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.975+0000 7f864ffff640 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x56289240ce50 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f863c009830 tx=0x7f863c032940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc6ba9b2cf2a55e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.975+0000 7f862ffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.975+0000 7f862ffff640 1 -- 172.21.15.50:0/2116487193 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c037070 con 0x5628923ec630 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.975+0000 7f862ffff640 1 -- 172.21.15.50:0/2116487193 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7f863c0329e0 con 0x5628923ec630 2024-04-23T18:32:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.975+0000 7f8659856740 1 -- 172.21.15.50:0/2116487193 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f863c005000 con 0x5628923ec630 2024-04-23T18:32:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.976+0000 7f862ffff640 1 -- 172.21.15.50:0/2116487193 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f863c032be0 con 0x5628923ec630 2024-04-23T18:32:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.976+0000 7f8659856740 10 client.17181 fetch_fsmap learned FSMap version 50 2024-04-23T18:32:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.976+0000 7f8659856740 1 -- 172.21.15.50:0/2116487193 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x56289246d8a0 con 0x5628923ec630 2024-04-23T18:32:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.976+0000 7f862ffff640 1 -- 172.21.15.50:0/2116487193 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f863c005000 con 0x5628923ec630 2024-04-23T18:32:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.977+0000 7f862ffff640 1 -- 172.21.15.50:0/2116487193 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7f863c04a070 con 0x5628923ec630 2024-04-23T18:32:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.977+0000 7f8659856740 10 client.17181 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:32:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.977+0000 7f8659856740 10 client.17181 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:32:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.977+0000 7f8659856740 20 client.17181 populate_metadata read hostname 'smithi050' 2024-04-23T18:32:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.977+0000 7f8659856740 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56289246df90 0x5628924703b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.977+0000 7f8659856740 4 client.17181 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:32:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.977+0000 7f8659856740 1 -- 172.21.15.50:0/2116487193 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562892359200 con 0x56289246df90 2024-04-23T18:32:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.977+0000 7f86548df640 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56289246df90 0x5628924703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.977+0000 7f862ffff640 10 client.17181 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:16.977+0000 7f86548df640 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56289246df90 0x5628924703b0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61ceb2d50b445d24 server_cookie=6bb62636d836e7a1 in_seq=0 out_seq=0 2024-04-23T18:32:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.314+0000 7f862ffff640 1 -- 172.21.15.50:0/2116487193 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562892359200 con 0x56289246df90 2024-04-23T18:32:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.314+0000 7f862ffff640 10 client.17181 handle_command_reply: tid=0 2024-04-23T18:32:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.314+0000 7f8659856740 1 client.17181 shutdown 2024-04-23T18:32:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.314+0000 7f8659856740 1 -- 172.21.15.50:0/2116487193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 msgr2=0x56289240ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.314+0000 7f8659856740 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x56289240ce50 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f863c009830 tx=0x7f863c032940 comp rx=0 tx=0).stop 2024-04-23T18:32:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.314+0000 7f8659856740 1 -- 172.21.15.50:0/2116487193 shutdown_connections 2024-04-23T18:32:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.314+0000 7f8659856740 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56289246df90 0x5628924703b0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.314+0000 7f8659856740 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628923f7810 0x56289240a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.314+0000 7f8659856740 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5628923ecf50 0x56289240a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.314+0000 7f862ffff640 0 client.17181 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.314+0000 7f8659856740 1 --2- 172.21.15.50:0/2116487193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5628923ec630 0x56289240ce50 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.314+0000 7f8659856740 1 -- 172.21.15.50:0/2116487193 >> 172.21.15.50:0/2116487193 conn(0x5628923e8b40 msgr2=0x5628923edf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.315+0000 7f8659856740 1 -- 172.21.15.50:0/2116487193 shutdown_connections 2024-04-23T18:32:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.315+0000 7f8659856740 1 -- 172.21.15.50:0/2116487193 wait complete. 2024-04-23T18:32:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.315+0000 7f8659856740 20 client.17181 trim_cache size 0 max 16384 2024-04-23T18:32:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:32:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:32:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:32:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:32:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.316+0000 7f8656b43640 1 -- 172.21.15.50:0/1629883482 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f86400a3ec0 msgr2=0x7f86400a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.316+0000 7f8656b43640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f86400a3ec0 0x7f86400a6380 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f86480a54f0 tx=0x7f86480057c0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.316+0000 7f8656b43640 1 -- 172.21.15.50:0/1629883482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86501a3550 msgr2=0x7f86501a3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.316+0000 7f8656b43640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86501a3550 0x7f86501a3a00 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f8648009fd0 tx=0x7f864800ef10 comp rx=0 tx=0).stop 2024-04-23T18:32:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.316+0000 7f8656b43640 1 -- 172.21.15.50:0/1629883482 shutdown_connections 2024-04-23T18:32:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.316+0000 7f8656b43640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f86400a3ec0 0x7f86400a6380 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.316+0000 7f8656b43640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86501a3550 0x7f86501a3a00 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.316+0000 7f8656b43640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8650106170 0x7f8650109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.316+0000 7f8656b43640 1 --2- 172.21.15.50:0/1629883482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86501010f0 0x7f8650109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.316+0000 7f8656b43640 1 -- 172.21.15.50:0/1629883482 >> 172.21.15.50:0/1629883482 conn(0x7f8650077fd0 msgr2=0x7f86500fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.316+0000 7f8656b43640 1 -- 172.21.15.50:0/1629883482 shutdown_connections 2024-04-23T18:32:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:17.316+0000 7f8656b43640 1 -- 172.21.15.50:0/1629883482 wait complete. 2024-04-23T18:32:17.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 34204 39.71 MB/sec warmup 51 sec latency 180.084 ms 2024-04-23T18:32:18.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:18 smithi050 ceph-mon[29187]: pgmap v435: 97 pgs: 97 active+clean; 599 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 689 op/s 2024-04-23T18:32:18.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:18 smithi177 ceph-mon[32308]: pgmap v435: 97 pgs: 97 active+clean; 599 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 689 op/s 2024-04-23T18:32:18.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 34826 39.46 MB/sec warmup 52 sec latency 49.839 ms 2024-04-23T18:32:18.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:18 smithi188 ceph-mon[32694]: pgmap v435: 97 pgs: 97 active+clean; 599 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 689 op/s 2024-04-23T18:32:19.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 35744 39.45 MB/sec warmup 53 sec latency 307.144 ms 2024-04-23T18:32:20.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:20 smithi050 ceph-mon[29187]: pgmap v436: 97 pgs: 97 active+clean; 599 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 689 op/s 2024-04-23T18:32:20.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:20 smithi177 ceph-mon[32308]: pgmap v436: 97 pgs: 97 active+clean; 599 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 689 op/s 2024-04-23T18:32:20.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 36511 39.61 MB/sec warmup 54 sec latency 1307.177 ms 2024-04-23T18:32:20.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:20 smithi188 ceph-mon[32694]: pgmap v436: 97 pgs: 97 active+clean; 599 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 689 op/s 2024-04-23T18:32:21.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 37053 39.64 MB/sec warmup 55 sec latency 2307.214 ms 2024-04-23T18:32:22.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:22 smithi050 ceph-mon[29187]: pgmap v437: 97 pgs: 97 active+clean; 599 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 689 op/s 2024-04-23T18:32:22.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:22 smithi177 ceph-mon[32308]: pgmap v437: 97 pgs: 97 active+clean; 599 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 689 op/s 2024-04-23T18:32:22.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 37528 39.30 MB/sec warmup 56 sec latency 3307.256 ms 2024-04-23T18:32:22.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:22 smithi188 ceph-mon[32694]: pgmap v437: 97 pgs: 97 active+clean; 599 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 689 op/s 2024-04-23T18:32:23.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 38231 39.13 MB/sec warmup 57 sec latency 4307.288 ms 2024-04-23T18:32:24.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:24 smithi050 ceph-mon[29187]: pgmap v438: 97 pgs: 97 active+clean; 600 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 18 MiB/s wr, 1.01k op/s 2024-04-23T18:32:24.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:24 smithi177 ceph-mon[32308]: pgmap v438: 97 pgs: 97 active+clean; 600 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 18 MiB/s wr, 1.01k op/s 2024-04-23T18:32:24.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 39135 39.29 MB/sec warmup 58 sec latency 4779.958 ms 2024-04-23T18:32:24.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:24 smithi188 ceph-mon[32694]: pgmap v438: 97 pgs: 97 active+clean; 600 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 18 MiB/s wr, 1.01k op/s 2024-04-23T18:32:25.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 39930 39.43 MB/sec warmup 59 sec latency 140.512 ms 2024-04-23T18:32:26.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:26 smithi050 ceph-mon[29187]: pgmap v439: 97 pgs: 97 active+clean; 600 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 12 MiB/s wr, 703 op/s 2024-04-23T18:32:26.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:26 smithi177 ceph-mon[32308]: pgmap v439: 97 pgs: 97 active+clean; 600 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 12 MiB/s wr, 703 op/s 2024-04-23T18:32:26.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 40359 39.28 MB/sec warmup 60 sec latency 288.299 ms 2024-04-23T18:32:26.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:26 smithi188 ceph-mon[32694]: pgmap v439: 97 pgs: 97 active+clean; 600 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 12 MiB/s wr, 703 op/s 2024-04-23T18:32:27.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 40892 39.02 MB/sec warmup 61 sec latency 97.164 ms 2024-04-23T18:32:28.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:28 smithi050 ceph-mon[29187]: pgmap v440: 97 pgs: 97 active+clean; 630 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 18 MiB/s wr, 1.00k op/s 2024-04-23T18:32:28.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:28 smithi177 ceph-mon[32308]: pgmap v440: 97 pgs: 97 active+clean; 630 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 18 MiB/s wr, 1.00k op/s 2024-04-23T18:32:28.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 41512 38.77 MB/sec warmup 62 sec latency 106.924 ms 2024-04-23T18:32:28.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:28 smithi188 ceph-mon[32694]: pgmap v440: 97 pgs: 97 active+clean; 630 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 18 MiB/s wr, 1.00k op/s 2024-04-23T18:32:29.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 42402 38.91 MB/sec warmup 63 sec latency 62.643 ms 2024-04-23T18:32:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.422+0000 7f96c274a640 1 -- 172.21.15.50:0/561047955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96bc100990 msgr2=0x7f96bc100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.422+0000 7f96c274a640 1 --2- 172.21.15.50:0/561047955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96bc100990 0x7f96bc100df0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009930 tx=0x7f96b002a420 comp rx=0 tx=0).stop 2024-04-23T18:32:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.422+0000 7f96c274a640 1 -- 172.21.15.50:0/561047955 shutdown_connections 2024-04-23T18:32:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.422+0000 7f96c274a640 1 --2- 172.21.15.50:0/561047955 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc10ebd0 0x7f96bc110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.422+0000 7f96c274a640 1 --2- 172.21.15.50:0/561047955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96bc100990 0x7f96bc100df0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.422+0000 7f96c274a640 1 --2- 172.21.15.50:0/561047955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f96bc103050 0x7f96bc100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.422+0000 7f96c274a640 1 -- 172.21.15.50:0/561047955 >> 172.21.15.50:0/561047955 conn(0x7f96bc0773e0 msgr2=0x7f96bc0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.422+0000 7f96c274a640 1 -- 172.21.15.50:0/561047955 shutdown_connections 2024-04-23T18:32:30.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.422+0000 7f96c274a640 1 -- 172.21.15.50:0/561047955 wait complete. 2024-04-23T18:32:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.423+0000 7f96c274a640 1 Processor -- start 2024-04-23T18:32:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.423+0000 7f96c274a640 1 -- start start 2024-04-23T18:32:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c274a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96bc100990 0x7f96bc1a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c274a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f96bc103050 0x7f96bc1a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c274a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc10ebd0 0x7f96bc19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c274a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1145d0 con 0x7f96bc10ebd0 2024-04-23T18:32:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c274a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc114450 con 0x7f96bc100990 2024-04-23T18:32:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c274a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc114750 con 0x7f96bc103050 2024-04-23T18:32:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96bbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96bc100990 0x7f96bc1a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c0ce7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc10ebd0 0x7f96bc19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c0ce7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc10ebd0 0x7f96bc19a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58584/0 (socket says 172.21.15.50:58584) 2024-04-23T18:32:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c0ce7640 1 -- 172.21.15.50:0/9001413 learned_addr learned my addr 172.21.15.50:0/9001413 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f96bc103050 0x7f96bc1a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c0ce7640 1 -- 172.21.15.50:0/9001413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f96bc103050 msgr2=0x7f96bc1a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c0ce7640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f96bc103050 0x7f96bc1a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c0ce7640 1 -- 172.21.15.50:0/9001413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96bc100990 msgr2=0x7f96bc1a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c0ce7640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96bc100990 0x7f96bc1a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c0ce7640 1 -- 172.21.15.50:0/9001413 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96bc19b0b0 con 0x7f96bc10ebd0 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96bbfff640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96bc100990 0x7f96bc1a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.424+0000 7f96c0ce7640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc10ebd0 0x7f96bc19a930 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f96b400efd0 tx=0x7f96b400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.425+0000 7f96b97fa640 1 -- 172.21.15.50:0/9001413 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b4003b70 con 0x7f96bc10ebd0 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.425+0000 7f96c274a640 1 -- 172.21.15.50:0/9001413 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96bc19b3a0 con 0x7f96bc10ebd0 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.425+0000 7f96b97fa640 1 -- 172.21.15.50:0/9001413 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96b4003d10 con 0x7f96bc10ebd0 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.425+0000 7f96b97fa640 1 -- 172.21.15.50:0/9001413 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b40108a0 con 0x7f96bc10ebd0 2024-04-23T18:32:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.425+0000 7f96c274a640 1 -- 172.21.15.50:0/9001413 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96bc1a7640 con 0x7f96bc10ebd0 2024-04-23T18:32:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.427+0000 7f96b97fa640 1 -- 172.21.15.50:0/9001413 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f96b4004020 con 0x7f96bc10ebd0 2024-04-23T18:32:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.427+0000 7f96c545b740 1 Processor -- start 2024-04-23T18:32:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.427+0000 7f96c545b740 1 -- start start 2024-04-23T18:32:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96c545b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5ac95620 0x561d5ac95a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96c545b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95f40 0x561d5aca02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96c545b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5aca0800 0x561d5aca2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96c545b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561d5abfa970 con 0x561d5aca0800 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96c545b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561d5ab974a0 con 0x561d5ac95620 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96c545b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561d5abf6930 con 0x561d5ac95f40 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96bbfff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95f40 0x561d5aca02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96bb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5aca0800 0x561d5aca2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96bbfff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95f40 0x561d5aca02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:56772/0 (socket says 172.21.15.50:56772) 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96bbfff640 1 -- 172.21.15.50:0/2550219980 learned_addr learned my addr 172.21.15.50:0/2550219980 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/2550219980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5ac95620 0x561d5ac95a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96b97fa640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f96a40a3dc0 0x7f96a40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96c0ce7640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f96a40a3dc0 0x7f96a40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96bbfff640 1 -- 172.21.15.50:0/2550219980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5ac95620 msgr2=0x561d5ac95a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96bbfff640 1 --2- 172.21.15.50:0/2550219980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5ac95620 0x561d5ac95a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96bbfff640 1 -- 172.21.15.50:0/2550219980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5aca0800 msgr2=0x561d5aca2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96bbfff640 1 --2- 172.21.15.50:0/2550219980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5aca0800 0x561d5aca2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.428+0000 7f96bbfff640 1 -- 172.21.15.50:0/2550219980 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d5a90d210 con 0x561d5ac95f40 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.429+0000 7f96b97fa640 1 -- 172.21.15.50:0/9001413 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7f96b40d0c70 con 0x7f96bc10ebd0 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.429+0000 7f96bbfff640 1 --2- 172.21.15.50:0/2550219980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95f40 0x561d5aca02c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0114f0 tx=0x7f96ac0119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef35c2de2845b5ad server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.429+0000 7f96c0ce7640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f96a40a3dc0 0x7f96a40a6280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f96b40ce9b0 tx=0x7f96b4004560 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.429+0000 7f96a27fc640 1 -- 172.21.15.50:0/2550219980 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac004020 con 0x561d5ac95f40 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.429+0000 7f96a27fc640 1 -- 172.21.15.50:0/2550219980 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96ac0041c0 con 0x561d5ac95f40 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.429+0000 7f96a27fc640 1 -- 172.21.15.50:0/2550219980 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac00f7c0 con 0x561d5ac95f40 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.430+0000 7f96c545b740 1 -- 172.21.15.50:0/2550219980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95f40 msgr2=0x561d5aca02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.430+0000 7f96c545b740 1 --2- 172.21.15.50:0/2550219980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95f40 0x561d5aca02c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0114f0 tx=0x7f96ac0119b0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.430+0000 7f96c545b740 1 -- 172.21.15.50:0/2550219980 shutdown_connections 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.430+0000 7f96c545b740 1 --2- 172.21.15.50:0/2550219980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5aca0800 0x561d5aca2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.430+0000 7f96c545b740 1 --2- 172.21.15.50:0/2550219980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95f40 0x561d5aca02c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.430+0000 7f96c545b740 1 --2- 172.21.15.50:0/2550219980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5ac95620 0x561d5ac95a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.430+0000 7f96c545b740 1 -- 172.21.15.50:0/2550219980 >> 172.21.15.50:0/2550219980 conn(0x561d5ac91b30 msgr2=0x561d5ac91f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.430+0000 7f96c545b740 1 -- 172.21.15.50:0/2550219980 shutdown_connections 2024-04-23T18:32:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.430+0000 7f96c545b740 1 -- 172.21.15.50:0/2550219980 wait complete. 2024-04-23T18:32:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.430+0000 7f96c545b740 1 -- start start 2024-04-23T18:32:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.430+0000 7f96c545b740 1 Processor -- start 2024-04-23T18:32:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96c545b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95620 0x561d5aca0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96c545b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5ac95f40 0x561d5ac9d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96c545b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5aca0800 0x561d5ac9d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96c545b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561d5aca3430 con 0x561d5aca0800 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96c545b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561d5ab950b0 con 0x561d5ac95f40 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96c545b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561d5aca35b0 con 0x561d5ac95620 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96c545b740 10 client.0 _ll_register_callbacks cb 0x561d5ab82d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96bbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5ac95f40 0x561d5ac9d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96bb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5aca0800 0x561d5ac9d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96bbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5ac95f40 0x561d5ac9d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:40614/0 (socket says 172.21.15.50:40614) 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96bbfff640 1 -- 172.21.15.50:0/1173404142 learned_addr learned my addr 172.21.15.50:0/1173404142 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95620 0x561d5aca0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96bbfff640 1 -- 172.21.15.50:0/1173404142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95620 msgr2=0x561d5aca0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96bbfff640 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95620 0x561d5aca0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96bbfff640 1 -- 172.21.15.50:0/1173404142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5aca0800 msgr2=0x561d5ac9d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96bbfff640 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5aca0800 0x561d5ac9d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96bbfff640 1 -- 172.21.15.50:0/1173404142 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d5a9d9620 con 0x561d5ac95f40 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5aca0800 0x561d5ac9d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f96bbfff640 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5ac95f40 0x561d5ac9d420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0056a0 tx=0x7f96ac0056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ab8ebbe5a3d74f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.431+0000 7f969b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f969b7fe640 1 -- 172.21.15.50:0/1173404142 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac028020 con 0x561d5ac95f40 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f969b7fe640 1 -- 172.21.15.50:0/1173404142 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7f96ac016040 con 0x561d5ac95f40 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f96c545b740 1 -- 172.21.15.50:0/1173404142 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96ac0041c0 con 0x561d5ac95f40 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f969b7fe640 1 -- 172.21.15.50:0/1173404142 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f96ac004020 con 0x561d5ac95f40 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f96c545b740 10 client.26699 fetch_fsmap learned FSMap version 50 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f96c545b740 1 -- 172.21.15.50:0/1173404142 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x561d5acaf340 con 0x561d5ac95f40 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f969b7fe640 1 -- 172.21.15.50:0/1173404142 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96ac0041c0 con 0x561d5ac95f40 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f969b7fe640 1 -- 172.21.15.50:0/1173404142 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7f96ac004730 con 0x561d5ac95f40 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f96c545b740 10 client.26699 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:32:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f96c545b740 10 client.26699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f96c545b740 20 client.26699 populate_metadata read hostname 'smithi050' 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f96c545b740 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561d5acb07a0 0x561d5acb2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.432+0000 7f96c545b740 4 client.26699 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.433+0000 7f96c545b740 1 -- 172.21.15.50:0/1173404142 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d5ac08390 con 0x561d5acb07a0 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.433+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561d5acb07a0 0x561d5acb2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.433+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561d5acb07a0 0x561d5acb2b90 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0e08043c28b4667 server_cookie=2392b5465b2be005 in_seq=0 out_seq=0 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.433+0000 7f969b7fe640 10 client.26699 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f969b7fe640 1 -- 172.21.15.50:0/1173404142 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561d5ac08390 con 0x561d5acb07a0 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f969b7fe640 10 client.26699 handle_command_reply: tid=0 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f96c545b740 1 client.26699 shutdown 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f96c545b740 1 -- 172.21.15.50:0/1173404142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5ac95f40 msgr2=0x561d5ac9d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f96c545b740 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5ac95f40 0x561d5ac9d420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0056a0 tx=0x7f96ac0056d0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f96c545b740 1 -- 172.21.15.50:0/1173404142 shutdown_connections 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f96c545b740 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561d5acb07a0 0x561d5acb2b90 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f969b7fe640 0 client.26699 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f96c545b740 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5aca0800 0x561d5ac9d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f96c545b740 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5ac95f40 0x561d5ac9d420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f96c545b740 1 --2- 172.21.15.50:0/1173404142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95620 0x561d5aca0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f96c545b740 1 -- 172.21.15.50:0/1173404142 >> 172.21.15.50:0/1173404142 conn(0x561d5ac91b30 msgr2=0x561d5ac97860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f96c545b740 1 -- 172.21.15.50:0/1173404142 shutdown_connections 2024-04-23T18:32:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f96c545b740 1 -- 172.21.15.50:0/1173404142 wait complete. 2024-04-23T18:32:30.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.434+0000 7f96c545b740 20 client.26699 trim_cache size 0 max 16384 2024-04-23T18:32:30.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:30 smithi050 ceph-mon[29187]: pgmap v441: 97 pgs: 97 active+clean; 630 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 702 op/s 2024-04-23T18:32:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.454+0000 7f96c545b740 1 Processor -- start 2024-04-23T18:32:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.454+0000 7f96c545b740 1 -- start start 2024-04-23T18:32:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96c545b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95620 0x561d5acac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96c545b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95f40 0x561d5ac988e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96c545b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5aca0800 0x561d5ac9fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96c545b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1b2680 con 0x561d5ac95620 2024-04-23T18:32:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96c545b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1afe90 con 0x561d5aca0800 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96c545b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1add00 con 0x561d5ac95f40 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96bb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95620 0x561d5acac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96bb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95620 0x561d5acac2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58610/0 (socket says 172.21.15.50:58610) 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96bb7fe640 1 -- 172.21.15.50:0/1055830245 learned_addr learned my addr 172.21.15.50:0/1055830245 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96bbfff640 1 --2- 172.21.15.50:0/1055830245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95f40 0x561d5ac988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/1055830245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5aca0800 0x561d5ac9fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96bb7fe640 1 -- 172.21.15.50:0/1055830245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95f40 msgr2=0x561d5ac988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/1055830245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95f40 0x561d5ac988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96bb7fe640 1 -- 172.21.15.50:0/1055830245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5aca0800 msgr2=0x561d5ac9fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/1055830245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5aca0800 0x561d5ac9fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96bb7fe640 1 -- 172.21.15.50:0/1055830245 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d5ac95d70 con 0x561d5ac95620 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.455+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/1055830245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95620 0x561d5acac2f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f96b002a940 tx=0x7f96b0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c814ae9c6b57ec76 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.456+0000 7f96a1ffb640 1 -- 172.21.15.50:0/1055830245 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0052070 con 0x561d5ac95620 2024-04-23T18:32:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.456+0000 7f96a1ffb640 1 -- 172.21.15.50:0/1055830245 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96b004d870 con 0x561d5ac95620 2024-04-23T18:32:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.456+0000 7f96a1ffb640 1 -- 172.21.15.50:0/1055830245 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0035040 con 0x561d5ac95620 2024-04-23T18:32:30.453 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.456+0000 7f96c545b740 1 -- 172.21.15.50:0/1055830245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95620 msgr2=0x561d5acac2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:30.453 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.456+0000 7f96c545b740 1 --2- 172.21.15.50:0/1055830245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95620 0x561d5acac2f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f96b002a940 tx=0x7f96b0005dd0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.453 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.456+0000 7f96c545b740 1 -- 172.21.15.50:0/1055830245 shutdown_connections 2024-04-23T18:32:30.453 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.456+0000 7f96c545b740 1 --2- 172.21.15.50:0/1055830245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5aca0800 0x561d5ac9fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.456+0000 7f96c545b740 1 --2- 172.21.15.50:0/1055830245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95f40 0x561d5ac988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.456+0000 7f96c545b740 1 --2- 172.21.15.50:0/1055830245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95620 0x561d5acac2f0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.456+0000 7f96c545b740 1 -- 172.21.15.50:0/1055830245 >> 172.21.15.50:0/1055830245 conn(0x561d5ac91b30 msgr2=0x561d5ac981d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.456+0000 7f96c545b740 1 -- 172.21.15.50:0/1055830245 shutdown_connections 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.456+0000 7f96c545b740 1 -- 172.21.15.50:0/1055830245 wait complete. 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96c545b740 1 -- start start 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96c545b740 1 Processor -- start 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96c545b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95620 0x561d5aca3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96c545b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95f40 0x561d5aca3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96c545b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5aca0800 0x561d5ad14b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96c545b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1ae350 con 0x561d5ac95f40 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96c545b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1af7c0 con 0x561d5aca0800 2024-04-23T18:32:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96c545b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1aea20 con 0x561d5ac95620 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96c545b740 10 client.0 _ll_register_callbacks cb 0x7f96ac009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96bbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95f40 0x561d5aca3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96bb7fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95620 0x561d5aca3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96bbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95f40 0x561d5aca3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58614/0 (socket says 172.21.15.50:58614) 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96bbfff640 1 -- 172.21.15.50:0/3508165115 learned_addr learned my addr 172.21.15.50:0/3508165115 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.457+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/3508165115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5aca0800 0x561d5ad14b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f96bbfff640 1 -- 172.21.15.50:0/3508165115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95620 msgr2=0x561d5aca3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f96bbfff640 1 --2- 172.21.15.50:0/3508165115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95620 0x561d5aca3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f96bbfff640 1 -- 172.21.15.50:0/3508165115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5aca0800 msgr2=0x561d5ad14b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f96bbfff640 1 --2- 172.21.15.50:0/3508165115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5aca0800 0x561d5ad14b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f96bbfff640 1 -- 172.21.15.50:0/3508165115 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d5ad15170 con 0x561d5ac95f40 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f96bbfff640 1 --2- 172.21.15.50:0/3508165115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95f40 0x561d5aca3f50 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x561d5ac9e9f0 tx=0x7f96ac005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997905adcae61744 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f969b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:32:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f969b7fe640 1 -- 172.21.15.50:0/3508165115 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac028070 con 0x561d5ac95f40 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f969b7fe640 1 -- 172.21.15.50:0/3508165115 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7f96ac0035f0 con 0x561d5ac95f40 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f96c545b740 1 -- 172.21.15.50:0/3508165115 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96b004d870 con 0x561d5ac95f40 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f969b7fe640 1 -- 172.21.15.50:0/3508165115 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f96ac0041c0 con 0x561d5ac95f40 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f96c545b740 10 client.17199 fetch_fsmap learned FSMap version 50 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f96c545b740 1 -- 172.21.15.50:0/3508165115 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x561d5ad16480 con 0x561d5ac95f40 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f969b7fe640 1 -- 172.21.15.50:0/3508165115 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96b004d870 con 0x561d5ac95f40 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.458+0000 7f969b7fe640 1 -- 172.21.15.50:0/3508165115 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7f96ac036030 con 0x561d5ac95f40 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.459+0000 7f96c545b740 10 client.17199 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.459+0000 7f96c545b740 10 client.17199 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.459+0000 7f96c545b740 20 client.17199 populate_metadata read hostname 'smithi050' 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.459+0000 7f96c545b740 1 --2- 172.21.15.50:0/3508165115 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561d5acb07a0 0x561d5ad18b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.459+0000 7f96c545b740 4 client.17199 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:32:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.459+0000 7f96c545b740 1 -- 172.21.15.50:0/3508165115 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561d5ac02200 con 0x561d5acb07a0 2024-04-23T18:32:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.459+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/3508165115 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561d5acb07a0 0x561d5ad18b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.459+0000 7f96bb7fe640 1 --2- 172.21.15.50:0/3508165115 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561d5acb07a0 0x561d5ad18b80 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=468b053587db72a server_cookie=11a09084425de789 in_seq=0 out_seq=0 2024-04-23T18:32:30.457 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.459+0000 7f969b7fe640 10 client.17199 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:30.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:30 smithi177 ceph-mon[32308]: pgmap v441: 97 pgs: 97 active+clean; 630 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 702 op/s 2024-04-23T18:32:30.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 43307 39.27 MB/sec warmup 64 sec latency 94.556 ms 2024-04-23T18:32:30.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:30 smithi188 ceph-mon[32694]: pgmap v441: 97 pgs: 97 active+clean; 630 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 702 op/s 2024-04-23T18:32:30.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f969b7fe640 1 -- 172.21.15.50:0/3508165115 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f96b0041040 con 0x561d5acb07a0 2024-04-23T18:32:30.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f969b7fe640 10 client.17199 handle_command_reply: tid=0 2024-04-23T18:32:30.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f96c545b740 1 client.17199 shutdown 2024-04-23T18:32:30.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f96c545b740 1 -- 172.21.15.50:0/3508165115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95f40 msgr2=0x561d5aca3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:30.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f96c545b740 1 --2- 172.21.15.50:0/3508165115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95f40 0x561d5aca3f50 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x561d5ac9e9f0 tx=0x7f96ac005b90 comp rx=0 tx=0).stop 2024-04-23T18:32:30.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f96c545b740 1 -- 172.21.15.50:0/3508165115 shutdown_connections 2024-04-23T18:32:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f96c545b740 1 --2- 172.21.15.50:0/3508165115 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x561d5acb07a0 0x561d5ad18b80 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f96c545b740 1 --2- 172.21.15.50:0/3508165115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561d5aca0800 0x561d5ad14b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f96c545b740 1 --2- 172.21.15.50:0/3508165115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561d5ac95f40 0x561d5aca3f50 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f969b7fe640 0 client.17199 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f96c545b740 1 --2- 172.21.15.50:0/3508165115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561d5ac95620 0x561d5aca3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f96c545b740 1 -- 172.21.15.50:0/3508165115 >> 172.21.15.50:0/3508165115 conn(0x561d5ac91b30 msgr2=0x561d5ac90cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f96c545b740 1 -- 172.21.15.50:0/3508165115 shutdown_connections 2024-04-23T18:32:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f96c545b740 1 -- 172.21.15.50:0/3508165115 wait complete. 2024-04-23T18:32:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.877+0000 7f96c545b740 20 client.17199 trim_cache size 0 max 16384 2024-04-23T18:32:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:32:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:32:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:32:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:32:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.878+0000 7f96c274a640 1 -- 172.21.15.50:0/9001413 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f96a40a3dc0 msgr2=0x7f96a40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.878+0000 7f96c274a640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f96a40a3dc0 0x7f96a40a6280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f96b40ce9b0 tx=0x7f96b4004560 comp rx=0 tx=0).stop 2024-04-23T18:32:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.878+0000 7f96c274a640 1 -- 172.21.15.50:0/9001413 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc10ebd0 msgr2=0x7f96bc19a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.878+0000 7f96c274a640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc10ebd0 0x7f96bc19a930 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f96b400efd0 tx=0x7f96b400c6a0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.878+0000 7f96c274a640 1 -- 172.21.15.50:0/9001413 shutdown_connections 2024-04-23T18:32:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.878+0000 7f96c274a640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f96a40a3dc0 0x7f96a40a6280 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.878+0000 7f96c274a640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc10ebd0 0x7f96bc19a930 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.878+0000 7f96c274a640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f96bc103050 0x7f96bc1a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.878+0000 7f96c274a640 1 --2- 172.21.15.50:0/9001413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96bc100990 0x7f96bc1a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.878+0000 7f96c274a640 1 -- 172.21.15.50:0/9001413 >> 172.21.15.50:0/9001413 conn(0x7f96bc0773e0 msgr2=0x7f96bc110180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.878+0000 7f96c274a640 1 -- 172.21.15.50:0/9001413 shutdown_connections 2024-04-23T18:32:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:30.878+0000 7f96c274a640 1 -- 172.21.15.50:0/9001413 wait complete. 2024-04-23T18:32:31.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 43646 38.91 MB/sec warmup 65 sec latency 292.161 ms 2024-04-23T18:32:32.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:32 smithi050 ceph-mon[29187]: pgmap v442: 97 pgs: 97 active+clean; 630 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 702 op/s 2024-04-23T18:32:32.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:32 smithi177 ceph-mon[32308]: pgmap v442: 97 pgs: 97 active+clean; 630 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 702 op/s 2024-04-23T18:32:32.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 44243 38.92 MB/sec warmup 66 sec latency 44.490 ms 2024-04-23T18:32:32.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:32 smithi188 ceph-mon[32694]: pgmap v442: 97 pgs: 97 active+clean; 630 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 10 MiB/s wr, 702 op/s 2024-04-23T18:32:33.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 44915 38.72 MB/sec warmup 67 sec latency 50.963 ms 2024-04-23T18:32:34.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:34 smithi050 ceph-mon[29187]: pgmap v443: 97 pgs: 97 active+clean; 641 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 15 MiB/s wr, 1.14k op/s 2024-04-23T18:32:34.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:34 smithi177 ceph-mon[32308]: pgmap v443: 97 pgs: 97 active+clean; 641 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 15 MiB/s wr, 1.14k op/s 2024-04-23T18:32:34.555 INFO:tasks.workunit.client.0.smithi050.stdout: 10 45820 38.83 MB/sec warmup 68 sec latency 38.837 ms 2024-04-23T18:32:34.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:34 smithi188 ceph-mon[32694]: pgmap v443: 97 pgs: 97 active+clean; 641 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 15 MiB/s wr, 1.14k op/s 2024-04-23T18:32:35.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 46626 38.98 MB/sec warmup 69 sec latency 77.728 ms 2024-04-23T18:32:36.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:36 smithi050 ceph-mon[29187]: pgmap v444: 97 pgs: 97 active+clean; 641 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 10 MiB/s wr, 734 op/s 2024-04-23T18:32:36.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:36 smithi177 ceph-mon[32308]: pgmap v444: 97 pgs: 97 active+clean; 641 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 10 MiB/s wr, 734 op/s 2024-04-23T18:32:36.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 47170 38.79 MB/sec warmup 70 sec latency 222.948 ms 2024-04-23T18:32:36.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:36 smithi188 ceph-mon[32694]: pgmap v444: 97 pgs: 97 active+clean; 641 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 10 MiB/s wr, 734 op/s 2024-04-23T18:32:37.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 47893 38.79 MB/sec warmup 71 sec latency 87.938 ms 2024-04-23T18:32:38.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:38 smithi050 ceph-mon[29187]: pgmap v445: 97 pgs: 97 active+clean; 676 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 15 MiB/s wr, 1.01k op/s 2024-04-23T18:32:38.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:38 smithi177 ceph-mon[32308]: pgmap v445: 97 pgs: 97 active+clean; 676 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 15 MiB/s wr, 1.01k op/s 2024-04-23T18:32:38.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 48596 38.86 MB/sec warmup 72 sec latency 68.975 ms 2024-04-23T18:32:38.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:38 smithi188 ceph-mon[32694]: pgmap v445: 97 pgs: 97 active+clean; 676 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 15 MiB/s wr, 1.01k op/s 2024-04-23T18:32:39.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 49389 38.85 MB/sec warmup 73 sec latency 52.784 ms 2024-04-23T18:32:40.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:40 smithi050 ceph-mon[29187]: pgmap v446: 97 pgs: 97 active+clean; 676 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.8 MiB/s wr, 717 op/s 2024-04-23T18:32:40.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:40 smithi177 ceph-mon[32308]: pgmap v446: 97 pgs: 97 active+clean; 676 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.8 MiB/s wr, 717 op/s 2024-04-23T18:32:40.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 50170 38.90 MB/sec warmup 74 sec latency 106.862 ms 2024-04-23T18:32:40.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:40 smithi188 ceph-mon[32694]: pgmap v446: 97 pgs: 97 active+clean; 676 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.8 MiB/s wr, 717 op/s 2024-04-23T18:32:41.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 50752 38.85 MB/sec warmup 75 sec latency 292.978 ms 2024-04-23T18:32:42.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:42 smithi050 ceph-mon[29187]: pgmap v447: 97 pgs: 97 active+clean; 676 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.8 MiB/s wr, 717 op/s 2024-04-23T18:32:42.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:42 smithi177 ceph-mon[32308]: pgmap v447: 97 pgs: 97 active+clean; 676 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.8 MiB/s wr, 717 op/s 2024-04-23T18:32:42.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51450 38.84 MB/sec warmup 76 sec latency 76.243 ms 2024-04-23T18:32:42.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:42 smithi188 ceph-mon[32694]: pgmap v447: 97 pgs: 97 active+clean; 676 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.8 MiB/s wr, 717 op/s 2024-04-23T18:32:42.651 INFO:tasks.quiescer.fs.[cephfs]:Total duration (42.9) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:32:42.651 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.45 2024-04-23T18:32:42.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:32:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.780+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1288170612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e81031e0 msgr2=0x7fc4e81035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.780+0000 7fc4ed3d6640 1 --2- 172.21.15.50:0/1288170612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e81031e0 0x7fc4e81035c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009900 tx=0x7fc4d802a510 comp rx=0 tx=0).stop 2024-04-23T18:32:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.780+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1288170612 shutdown_connections 2024-04-23T18:32:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.780+0000 7fc4ed3d6640 1 --2- 172.21.15.50:0/1288170612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4e810c520 0x7fc4e810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.780+0000 7fc4ed3d6640 1 --2- 172.21.15.50:0/1288170612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e8103b00 0x7fc4e810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.780+0000 7fc4ed3d6640 1 --2- 172.21.15.50:0/1288170612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e81031e0 0x7fc4e81035c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.780+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1288170612 >> 172.21.15.50:0/1288170612 conn(0x7fc4e8077f50 msgr2=0x7fc4e8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.780+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1288170612 shutdown_connections 2024-04-23T18:32:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.780+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1288170612 wait complete. 2024-04-23T18:32:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.781+0000 7fc4ed3d6640 1 Processor -- start 2024-04-23T18:32:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.781+0000 7fc4ed3d6640 1 -- start start 2024-04-23T18:32:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4ed3d6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e81031e0 0x7fc4e8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4ed3d6640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e8103b00 0x7fc4e8198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4ed3d6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4e810c520 0x7fc4e819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4ed3d6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e8112190 con 0x7fc4e81031e0 2024-04-23T18:32:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4ed3d6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e8112010 con 0x7fc4e810c520 2024-04-23T18:32:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4ed3d6640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e8112310 con 0x7fc4e8103b00 2024-04-23T18:32:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4e6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e81031e0 0x7fc4e8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4e6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e81031e0 0x7fc4e8198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53516/0 (socket says 172.21.15.50:53516) 2024-04-23T18:32:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1693937685 learned_addr learned my addr 172.21.15.50:0/1693937685 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:42.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4de7fc640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e8103b00 0x7fc4e8198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4e77fe640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4e810c520 0x7fc4e819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1693937685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e8103b00 msgr2=0x7fc4e8198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e8103b00 0x7fc4e8198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1693937685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4e810c520 msgr2=0x7fc4e819cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4e810c520 0x7fc4e819cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.782+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1693937685 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e819d720 con 0x7fc4e81031e0 2024-04-23T18:32:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.783+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e81031e0 0x7fc4e8198730 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8005940 tx=0x7fc4d80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.783+0000 7fc4e4ff9640 1 -- 172.21.15.50:0/1693937685 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d803f030 con 0x7fc4e81031e0 2024-04-23T18:32:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.783+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1693937685 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4e819d9b0 con 0x7fc4e81031e0 2024-04-23T18:32:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.783+0000 7fc4e4ff9640 1 -- 172.21.15.50:0/1693937685 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4d80044c0 con 0x7fc4e81031e0 2024-04-23T18:32:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.783+0000 7fc4e4ff9640 1 -- 172.21.15.50:0/1693937685 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8035070 con 0x7fc4e81031e0 2024-04-23T18:32:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.783+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1693937685 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4e81a52a0 con 0x7fc4e81031e0 2024-04-23T18:32:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.784+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1693937685 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4e8068d70 con 0x7fc4e81031e0 2024-04-23T18:32:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.785+0000 7fc4e4ff9640 1 -- 172.21.15.50:0/1693937685 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc4d8042030 con 0x7fc4e81031e0 2024-04-23T18:32:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.786+0000 7fc4e4ff9640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4d40a3d20 0x7fc4d40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.786+0000 7fc4e4ff9640 1 -- 172.21.15.50:0/1693937685 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7fc4d80f0d60 con 0x7fc4e81031e0 2024-04-23T18:32:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.786+0000 7fc4de7fc640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4d40a3d20 0x7fc4d40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.787+0000 7fc4de7fc640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4d40a3d20 0x7fc4d40a61e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8100930 tx=0x7fc4d0005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:42.790+0000 7fc4e4ff9640 1 -- 172.21.15.50:0/1693937685 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4d80bcda0 con 0x7fc4e81031e0 2024-04-23T18:32:43.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.105+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1693937685 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4e81076a0 con 0x7fc4e81031e0 2024-04-23T18:32:43.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.106+0000 7fc4e4ff9640 1 -- 172.21.15.50:0/1693937685 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+11884 (secure 0 0 0) 0x7fc4d80c10a0 con 0x7fc4e81031e0 2024-04-23T18:32:43.098 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:32:43.099 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":50,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:31:39.945970+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/4033440314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4033440314},{"type":"v1","addr":"172.21.15.177:6837","nonce":4033440314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:32:43.099 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 50 2024-04-23T18:32:43.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.108+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1693937685 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4d40a3d20 msgr2=0x7fc4d40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.108+0000 7fc4ed3d6640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4d40a3d20 0x7fc4d40a61e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8100930 tx=0x7fc4d0005f90 comp rx=0 tx=0).stop 2024-04-23T18:32:43.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.108+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1693937685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e81031e0 msgr2=0x7fc4e8198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.108+0000 7fc4ed3d6640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e81031e0 0x7fc4e8198730 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8005940 tx=0x7fc4d80040c0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.108+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1693937685 shutdown_connections 2024-04-23T18:32:43.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.108+0000 7fc4ed3d6640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4d40a3d20 0x7fc4d40a61e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.108+0000 7fc4ed3d6640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4e810c520 0x7fc4e819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.109+0000 7fc4ed3d6640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e8103b00 0x7fc4e8198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.109+0000 7fc4ed3d6640 1 --2- 172.21.15.50:0/1693937685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e81031e0 0x7fc4e8198730 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.109+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1693937685 >> 172.21.15.50:0/1693937685 conn(0x7fc4e8077f50 msgr2=0x7fc4e80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:43.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.109+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1693937685 shutdown_connections 2024-04-23T18:32:43.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.109+0000 7fc4ed3d6640 1 -- 172.21.15.50:0/1693937685 wait complete. 2024-04-23T18:32:43.115 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 81.45 --await-for 42.0' 2024-04-23T18:32:43.115 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 81.45 --await-for 42.0 2024-04-23T18:32:43.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.252+0000 7fd4fc623640 1 -- 172.21.15.50:0/3719503138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f4101ac0 msgr2=0x7fd4f4105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.252+0000 7fd4fc623640 1 --2- 172.21.15.50:0/3719503138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f4101ac0 0x7fd4f4105ab0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4009900 tx=0x7fd4e402a410 comp rx=0 tx=0).stop 2024-04-23T18:32:43.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.252+0000 7fd4fc623640 1 -- 172.21.15.50:0/3719503138 shutdown_connections 2024-04-23T18:32:43.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.252+0000 7fd4fc623640 1 --2- 172.21.15.50:0/3719503138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4f4106190 0x7fd4f410dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.252+0000 7fd4fc623640 1 --2- 172.21.15.50:0/3719503138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f4101ac0 0x7fd4f4105ab0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.252+0000 7fd4fc623640 1 --2- 172.21.15.50:0/3719503138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f4101110 0x7fd4f41014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.252+0000 7fd4fc623640 1 -- 172.21.15.50:0/3719503138 >> 172.21.15.50:0/3719503138 conn(0x7fd4f4077fb0 msgr2=0x7fd4f40783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:43.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.252+0000 7fd4fc623640 1 -- 172.21.15.50:0/3719503138 shutdown_connections 2024-04-23T18:32:43.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.252+0000 7fd4fc623640 1 -- 172.21.15.50:0/3719503138 wait complete. 2024-04-23T18:32:43.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.252+0000 7fd4fc623640 1 Processor -- start 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4fc623640 1 -- start start 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4fc623640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4f4101110 0x7fd4f4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4fc623640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f4106190 0x7fd4f4198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4fc623640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f419d020 0x7fd4f419d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4fc623640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f4111120 con 0x7fd4f4106190 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4fc623640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f4110fa0 con 0x7fd4f4101110 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4fc623640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f41112a0 con 0x7fd4f419d020 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4f9bbe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f4106190 0x7fd4f4198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4f9bbe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f4106190 0x7fd4f4198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53530/0 (socket says 172.21.15.50:53530) 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4f9bbe640 1 -- 172.21.15.50:0/3039661382 learned_addr learned my addr 172.21.15.50:0/3039661382 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4fa3bf640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4f4101110 0x7fd4f4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4fabc0640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f419d020 0x7fd4f419d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4f9bbe640 1 -- 172.21.15.50:0/3039661382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f419d020 msgr2=0x7fd4f419d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f419d020 0x7fd4f419d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4f9bbe640 1 -- 172.21.15.50:0/3039661382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4f4101110 msgr2=0x7fd4f4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4f4101110 0x7fd4f4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.253+0000 7fd4f9bbe640 1 -- 172.21.15.50:0/3039661382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f419db10 con 0x7fd4f4106190 2024-04-23T18:32:43.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.254+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f4106190 0x7fd4f4198ca0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4005ab0 tx=0x7fd4e4003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:43.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.254+0000 7fd4eb7fe640 1 -- 172.21.15.50:0/3039661382 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e403f070 con 0x7fd4f4106190 2024-04-23T18:32:43.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.254+0000 7fd4eb7fe640 1 -- 172.21.15.50:0/3039661382 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4e4031b50 con 0x7fd4f4106190 2024-04-23T18:32:43.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.254+0000 7fd4fc623640 1 -- 172.21.15.50:0/3039661382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4f419dda0 con 0x7fd4f4106190 2024-04-23T18:32:43.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.254+0000 7fd4eb7fe640 1 -- 172.21.15.50:0/3039661382 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4035030 con 0x7fd4f4106190 2024-04-23T18:32:43.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.254+0000 7fd4fc623640 1 -- 172.21.15.50:0/3039661382 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4f41a55c0 con 0x7fd4f4106190 2024-04-23T18:32:43.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.257+0000 7fd4eb7fe640 1 -- 172.21.15.50:0/3039661382 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd4e4042070 con 0x7fd4f4106190 2024-04-23T18:32:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.258+0000 7fd4ff411740 1 Processor -- start 2024-04-23T18:32:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4ff411740 1 -- start start 2024-04-23T18:32:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4eb7fe640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd4e00a3da0 0x7fd4e00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4ff411740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 0x558693267480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4ff411740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932679c0 0x558693271b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4ff411740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 0x558693274480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4ff411740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5586931cc370 con 0x5586932670a0 2024-04-23T18:32:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4ff411740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5586931692e0 con 0x5586932679c0 2024-04-23T18:32:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4ff411740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5586931c8330 con 0x558693272090 2024-04-23T18:32:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4fabc0640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd4e00a3da0 0x7fd4e00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4fa3bf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 0x558693267480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4eb7fe640 1 -- 172.21.15.50:0/3039661382 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7fd4e40f0ce0 con 0x7fd4f4106190 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4fa3bf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 0x558693267480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53534/0 (socket says 172.21.15.50:53534) 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4fa3bf640 1 -- 172.21.15.50:0/3240057749 learned_addr learned my addr 172.21.15.50:0/3240057749 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/3240057749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932679c0 0x558693271b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4f9bbe640 1 -- 172.21.15.50:0/3240057749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 msgr2=0x558693274480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/3240057749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 0x558693274480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4f9bbe640 1 -- 172.21.15.50:0/3240057749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 msgr2=0x558693267480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/3240057749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 0x558693267480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4f9bbe640 1 -- 172.21.15.50:0/3240057749 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558692faad80 con 0x5586932679c0 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.259+0000 7fd4fa3bf640 1 --2- 172.21.15.50:0/3240057749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 0x558693267480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.260+0000 7fd4fabc0640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd4e00a3da0 0x7fd4e00a6260 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec005fd0 tx=0x7fd4ec009500 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.260+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/3240057749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932679c0 0x558693271b50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40ee420 tx=0x7fd4e402a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7afe9fef59739023 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.260+0000 7fd4e8bfe640 1 -- 172.21.15.50:0/3240057749 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4049070 con 0x5586932679c0 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.260+0000 7fd4e8bfe640 1 -- 172.21.15.50:0/3240057749 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4e40c1020 con 0x5586932679c0 2024-04-23T18:32:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.260+0000 7fd4e8bfe640 1 -- 172.21.15.50:0/3240057749 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e40b9390 con 0x5586932679c0 2024-04-23T18:32:43.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 -- 172.21.15.50:0/3240057749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932679c0 msgr2=0x558693271b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 --2- 172.21.15.50:0/3240057749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932679c0 0x558693271b50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40ee420 tx=0x7fd4e402a410 comp rx=0 tx=0).stop 2024-04-23T18:32:43.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 -- 172.21.15.50:0/3240057749 shutdown_connections 2024-04-23T18:32:43.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 --2- 172.21.15.50:0/3240057749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 0x558693274480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 --2- 172.21.15.50:0/3240057749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932679c0 0x558693271b50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 --2- 172.21.15.50:0/3240057749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 0x558693267480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 -- 172.21.15.50:0/3240057749 >> 172.21.15.50:0/3240057749 conn(0x5586932633c0 msgr2=0x5586932637a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:43.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 -- 172.21.15.50:0/3240057749 shutdown_connections 2024-04-23T18:32:43.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 -- 172.21.15.50:0/3240057749 wait complete. 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 -- start start 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 Processor -- start 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 0x55869326f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932679c0 0x55869326f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 0x55869326fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558693274c40 con 0x5586932670a0 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558693166fb0 con 0x5586932679c0 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.261+0000 7fd4ff411740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558693274dc0 con 0x558693272090 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4ff411740 10 client.0 _ll_register_callbacks cb 0x5586931669a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4fa3bf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 0x55869326f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4fa3bf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 0x55869326f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53540/0 (socket says 172.21.15.50:53540) 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4fa3bf640 1 -- 172.21.15.50:0/2355452689 learned_addr learned my addr 172.21.15.50:0/2355452689 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932679c0 0x55869326f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4fabc0640 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 0x55869326fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4fa3bf640 1 -- 172.21.15.50:0/2355452689 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 msgr2=0x55869326fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4fa3bf640 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 0x55869326fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4fa3bf640 1 -- 172.21.15.50:0/2355452689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932679c0 msgr2=0x55869326f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4fa3bf640 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932679c0 0x55869326f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4fa3bf640 1 -- 172.21.15.50:0/2355452689 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55869300b510 con 0x5586932670a0 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932679c0 0x55869326f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4fa3bf640 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 0x55869326f120 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0012810 tx=0x7fd4f0012cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c347fa1a4b876a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4ddffb640 1 -- 172.21.15.50:0/2355452689 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f000f920 con 0x5586932670a0 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4ddffb640 1 -- 172.21.15.50:0/2355452689 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7fd4f000fac0 con 0x5586932670a0 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.262+0000 7fd4ff411740 1 -- 172.21.15.50:0/2355452689 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4e40c1020 con 0x5586932670a0 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.263+0000 7fd4ddffb640 1 -- 172.21.15.50:0/2355452689 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fd4f000fcc0 con 0x5586932670a0 2024-04-23T18:32:43.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.263+0000 7fd4ff411740 10 client.17217 fetch_fsmap learned FSMap version 50 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.263+0000 7fd4ff411740 1 -- 172.21.15.50:0/2355452689 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x558692e9e670 con 0x5586932670a0 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.263+0000 7fd4ddffb640 1 -- 172.21.15.50:0/2355452689 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4e40c1020 con 0x5586932670a0 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.263+0000 7fd4ddffb640 1 -- 172.21.15.50:0/2355452689 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7fd4f001c420 con 0x5586932670a0 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.263+0000 7fd4ff411740 10 client.17217 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.263+0000 7fd4ff411740 10 client.17217 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.263+0000 7fd4ff411740 20 client.17217 populate_metadata read hostname 'smithi050' 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.263+0000 7fd4ff411740 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5586932822f0 0x558693284770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.263+0000 7fd4ff411740 4 client.17217 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.263+0000 7fd4ff411740 1 -- 172.21.15.50:0/2355452689 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586931d9d90 con 0x5586932822f0 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.263+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5586932822f0 0x558693284770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.264+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5586932822f0 0x558693284770 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdbb5230b337a4f9 server_cookie=8e9894784c16f225 in_seq=0 out_seq=0 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.264+0000 7fd4ddffb640 10 client.17217 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:43.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.265+0000 7fd4ddffb640 1 -- 172.21.15.50:0/2355452689 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5586931d9d90 con 0x5586932822f0 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.265+0000 7fd4ddffb640 10 client.17217 handle_command_reply: tid=0 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.265+0000 7fd4ff411740 1 client.17217 shutdown 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.266+0000 7fd4ff411740 1 -- 172.21.15.50:0/2355452689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 msgr2=0x55869326f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.266+0000 7fd4ff411740 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 0x55869326f120 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0012810 tx=0x7fd4f0012cd0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.266+0000 7fd4ff411740 1 -- 172.21.15.50:0/2355452689 shutdown_connections 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.266+0000 7fd4ff411740 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5586932822f0 0x558693284770 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.266+0000 7fd4ff411740 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 0x55869326fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.266+0000 7fd4ff411740 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932679c0 0x55869326f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.266+0000 7fd4ddffb640 0 client.17217 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.266+0000 7fd4ff411740 1 --2- 172.21.15.50:0/2355452689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932670a0 0x55869326f120 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.266+0000 7fd4ff411740 1 -- 172.21.15.50:0/2355452689 >> 172.21.15.50:0/2355452689 conn(0x5586932633c0 msgr2=0x558693262600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.266+0000 7fd4ff411740 1 -- 172.21.15.50:0/2355452689 shutdown_connections 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.266+0000 7fd4ff411740 1 -- 172.21.15.50:0/2355452689 wait complete. 2024-04-23T18:32:43.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.266+0000 7fd4ff411740 20 client.17217 trim_cache size 0 max 16384 2024-04-23T18:32:43.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4ff411740 1 Processor -- start 2024-04-23T18:32:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4ff411740 1 -- start start 2024-04-23T18:32:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4ff411740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5586932670a0 0x55869327d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4ff411740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 0x55869326f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4ff411740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558693272090 0x55869328a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4ff411740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f41b27a0 con 0x5586932679c0 2024-04-23T18:32:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4ff411740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f41affb0 con 0x558693272090 2024-04-23T18:32:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4ff411740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f41ade20 con 0x5586932670a0 2024-04-23T18:32:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4f9bbe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 0x55869326f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4f9bbe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 0x55869326f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53548/0 (socket says 172.21.15.50:53548) 2024-04-23T18:32:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4f9bbe640 1 -- 172.21.15.50:0/805937054 learned_addr learned my addr 172.21.15.50:0/805937054 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4fabc0640 1 --2- 172.21.15.50:0/805937054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558693272090 0x55869328a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4fa3bf640 1 --2- 172.21.15.50:0/805937054 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5586932670a0 0x55869327d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4f9bbe640 1 -- 172.21.15.50:0/805937054 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5586932670a0 msgr2=0x55869327d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/805937054 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5586932670a0 0x55869327d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4f9bbe640 1 -- 172.21.15.50:0/805937054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558693272090 msgr2=0x55869328a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/805937054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558693272090 0x55869328a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.288+0000 7fd4f9bbe640 1 -- 172.21.15.50:0/805937054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55869325eca0 con 0x5586932679c0 2024-04-23T18:32:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.289+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/805937054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 0x55869326f7f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4005c40 tx=0x7fd4e40bcea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf5c8be0ec8008d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.289+0000 7fd4dffff640 1 -- 172.21.15.50:0/805937054 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4049040 con 0x5586932679c0 2024-04-23T18:32:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.289+0000 7fd4dffff640 1 -- 172.21.15.50:0/805937054 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4e40badc0 con 0x5586932679c0 2024-04-23T18:32:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.289+0000 7fd4dffff640 1 -- 172.21.15.50:0/805937054 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e40dd110 con 0x5586932679c0 2024-04-23T18:32:43.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 -- 172.21.15.50:0/805937054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 msgr2=0x55869326f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 --2- 172.21.15.50:0/805937054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 0x55869326f7f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4005c40 tx=0x7fd4e40bcea0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 -- 172.21.15.50:0/805937054 shutdown_connections 2024-04-23T18:32:43.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 --2- 172.21.15.50:0/805937054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558693272090 0x55869328a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 --2- 172.21.15.50:0/805937054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 0x55869326f7f0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 --2- 172.21.15.50:0/805937054 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5586932670a0 0x55869327d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 -- 172.21.15.50:0/805937054 >> 172.21.15.50:0/805937054 conn(0x5586932633c0 msgr2=0x55869326f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:43.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 -- 172.21.15.50:0/805937054 shutdown_connections 2024-04-23T18:32:43.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 -- 172.21.15.50:0/805937054 wait complete. 2024-04-23T18:32:43.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 -- start start 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 Processor -- start 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932670a0 0x558693275ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 0x558693276400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.290+0000 7fd4ff411740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 0x5586932e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4ff411740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f41ae470 con 0x5586932679c0 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4ff411740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f41af8e0 con 0x5586932670a0 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4ff411740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f41aeb40 con 0x558693272090 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4ff411740 10 client.0 _ll_register_callbacks cb 0x7fd4f0009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4f9bbe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 0x558693276400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4fa3bf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932670a0 0x558693275ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4fa3bf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932670a0 0x558693275ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:60922/0 (socket says 172.21.15.50:60922) 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4fa3bf640 1 -- 172.21.15.50:0/1670981474 learned_addr learned my addr 172.21.15.50:0/1670981474 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4f9bbe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 0x558693276400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53562/0 (socket says 172.21.15.50:53562) 2024-04-23T18:32:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4fabc0640 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 0x5586932e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4fa3bf640 1 -- 172.21.15.50:0/1670981474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 msgr2=0x5586932e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4fa3bf640 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 0x5586932e78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4fa3bf640 1 -- 172.21.15.50:0/1670981474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 msgr2=0x558693276400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4fa3bf640 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 0x558693276400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4fa3bf640 1 -- 172.21.15.50:0/1670981474 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586932e7e30 con 0x5586932670a0 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 0x558693276400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4fa3bf640 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932670a0 0x558693275ec0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55869325e7a0 tx=0x7fd4f000fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4c2ad175edbd5e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.291+0000 7fd4ddffb640 1 -- 172.21.15.50:0/1670981474 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f001c630 con 0x5586932670a0 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ddffb640 1 -- 172.21.15.50:0/1670981474 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7fd4f001cad0 con 0x5586932670a0 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ff411740 1 -- 172.21.15.50:0/1670981474 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5586931d3c00 con 0x5586932670a0 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ddffb640 1 -- 172.21.15.50:0/1670981474 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fd4f0034050 con 0x5586932670a0 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ff411740 10 client.26729 fetch_fsmap learned FSMap version 50 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ff411740 1 -- 172.21.15.50:0/1670981474 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5586932e9040 con 0x5586932670a0 2024-04-23T18:32:43.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ddffb640 1 -- 172.21.15.50:0/1670981474 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5586931d3c00 con 0x5586932670a0 2024-04-23T18:32:43.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ddffb640 1 -- 172.21.15.50:0/1670981474 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7fd4f00037f0 con 0x5586932670a0 2024-04-23T18:32:43.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ff411740 10 client.26729 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:32:43.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ff411740 10 client.26729 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:32:43.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ff411740 20 client.26729 populate_metadata read hostname 'smithi050' 2024-04-23T18:32:43.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ff411740 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5586932e98e0 0x5586932ebd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ff411740 4 client.26729 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.45, "await_for": 42.0}] 2024-04-23T18:32:43.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.292+0000 7fd4ff411740 1 -- 172.21.15.50:0/1670981474 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.45, "await_for": 42.0}) -- 0x55869314fca0 con 0x5586932e98e0 2024-04-23T18:32:43.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.293+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5586932e98e0 0x5586932ebd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.293+0000 7fd4ddffb640 10 client.26729 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:43.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.293+0000 7fd4f9bbe640 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5586932e98e0 0x5586932ebd00 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b58e6ca75cc7e657 server_cookie=337c3b2b7431152f in_seq=0 out_seq=0 2024-04-23T18:32:43.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:43 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1693937685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:32:43.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:43 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1693937685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:32:43.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51989 38.69 MB/sec warmup 77 sec latency 222.950 ms 2024-04-23T18:32:43.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:43 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1693937685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:32:43.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.849+0000 7fd4ddffb640 1 -- 172.21.15.50:0/1670981474 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55869314fca0 con 0x5586932e98e0 2024-04-23T18:32:43.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.849+0000 7fd4ddffb640 10 client.26729 handle_command_reply: tid=0 2024-04-23T18:32:43.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.849+0000 7fd4ff411740 1 client.26729 shutdown 2024-04-23T18:32:43.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4ff411740 1 -- 172.21.15.50:0/1670981474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932670a0 msgr2=0x558693275ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4ff411740 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932670a0 0x558693275ec0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55869325e7a0 tx=0x7fd4f000fc30 comp rx=0 tx=0).stop 2024-04-23T18:32:43.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4ff411740 1 -- 172.21.15.50:0/1670981474 shutdown_connections 2024-04-23T18:32:43.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4ff411740 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5586932e98e0 0x5586932ebd00 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4ff411740 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558693272090 0x5586932e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4ddffb640 0 client.26729 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:43.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4ff411740 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5586932679c0 0x558693276400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4ff411740 1 --2- 172.21.15.50:0/1670981474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586932670a0 0x558693275ec0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4ff411740 1 -- 172.21.15.50:0/1670981474 >> 172.21.15.50:0/1670981474 conn(0x5586932633c0 msgr2=0x55869326f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:43.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4ff411740 1 -- 172.21.15.50:0/1670981474 shutdown_connections 2024-04-23T18:32:43.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4ff411740 1 -- 172.21.15.50:0/1670981474 wait complete. 2024-04-23T18:32:43.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4ff411740 20 client.26729 trim_cache size 0 max 16384 2024-04-23T18:32:43.844 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:32:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 50, 2024-04-23T18:32:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:32:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 46, 2024-04-23T18:32:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:32:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "9c186906": { 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 46, 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 81.5, 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:32:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:32:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:32:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:32:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:32:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:32:43.846 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:32:43.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4fc623640 1 -- 172.21.15.50:0/3039661382 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd4e00a3da0 msgr2=0x7fd4e00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4fc623640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd4e00a3da0 0x7fd4e00a6260 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec005fd0 tx=0x7fd4ec009500 comp rx=0 tx=0).stop 2024-04-23T18:32:43.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4fc623640 1 -- 172.21.15.50:0/3039661382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f4106190 msgr2=0x7fd4f4198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4fc623640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f4106190 0x7fd4f4198ca0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4005ab0 tx=0x7fd4e4003b90 comp rx=0 tx=0).stop 2024-04-23T18:32:43.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4fc623640 1 -- 172.21.15.50:0/3039661382 shutdown_connections 2024-04-23T18:32:43.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4fc623640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd4e00a3da0 0x7fd4e00a6260 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4fc623640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4f419d020 0x7fd4f419d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4fc623640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4f4106190 0x7fd4f4198ca0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.850+0000 7fd4fc623640 1 --2- 172.21.15.50:0/3039661382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4f4101110 0x7fd4f4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.851+0000 7fd4fc623640 1 -- 172.21.15.50:0/3039661382 >> 172.21.15.50:0/3039661382 conn(0x7fd4f4077fb0 msgr2=0x7fd4f406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:43.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.851+0000 7fd4fc623640 1 -- 172.21.15.50:0/3039661382 shutdown_connections 2024-04-23T18:32:43.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:32:43.851+0000 7fd4fc623640 1 -- 172.21.15.50:0/3039661382 wait complete. 2024-04-23T18:32:43.854 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '9c186906', quiesce took 1.2 seconds. Will release after: 21.45 2024-04-23T18:32:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.968+0000 7f1bd7210640 1 -- 172.21.15.50:0/3567487443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd01021c0 msgr2=0x7f1bd01025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.968+0000 7f1bd7210640 1 --2- 172.21.15.50:0/3567487443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd01021c0 0x7f1bd01025a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009900 tx=0x7f1bc402a410 comp rx=0 tx=0).stop 2024-04-23T18:32:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.968+0000 7f1bd7210640 1 -- 172.21.15.50:0/3567487443 shutdown_connections 2024-04-23T18:32:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.968+0000 7f1bd7210640 1 --2- 172.21.15.50:0/3567487443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bd0107290 0x7f1bd010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.968+0000 7f1bd7210640 1 --2- 172.21.15.50:0/3567487443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bd0102b70 0x7f1bd0106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.968+0000 7f1bd7210640 1 --2- 172.21.15.50:0/3567487443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd01021c0 0x7f1bd01025a0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.968+0000 7f1bd7210640 1 -- 172.21.15.50:0/3567487443 >> 172.21.15.50:0/3567487443 conn(0x7f1bd00fda20 msgr2=0x7f1bd00ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.968+0000 7f1bd7210640 1 -- 172.21.15.50:0/3567487443 shutdown_connections 2024-04-23T18:32:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.968+0000 7f1bd7210640 1 -- 172.21.15.50:0/3567487443 wait complete. 2024-04-23T18:32:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bd7210640 1 Processor -- start 2024-04-23T18:32:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bd7210640 1 -- start start 2024-04-23T18:32:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bd7210640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd01021c0 0x7f1bd01986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bd7210640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bd0107290 0x7f1bd0198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bd7210640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bd019cf80 0x7f1bd019d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bd7210640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd0112200 con 0x7f1bd01021c0 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bd7210640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd0112080 con 0x7f1bd0107290 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bd7210640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd0112380 con 0x7f1bd019cf80 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bd4fac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd01021c0 0x7f1bd01986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bd4fac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd01021c0 0x7f1bd01986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53578/0 (socket says 172.21.15.50:53578) 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bd4fac640 1 -- 172.21.15.50:0/1562265094 learned_addr learned my addr 172.21.15.50:0/1562265094 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bd57ad640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bd019cf80 0x7f1bd019d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.969+0000 7f1bcffff640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bd0107290 0x7f1bd0198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.970+0000 7f1bd4fac640 1 -- 172.21.15.50:0/1562265094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bd019cf80 msgr2=0x7f1bd019d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.970+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bd019cf80 0x7f1bd019d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.970+0000 7f1bd4fac640 1 -- 172.21.15.50:0/1562265094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bd0107290 msgr2=0x7f1bd0198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.970+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bd0107290 0x7f1bd0198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.970+0000 7f1bd4fac640 1 -- 172.21.15.50:0/1562265094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd019da70 con 0x7f1bd01021c0 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.970+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd01021c0 0x7f1bd01986c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4005940 tx=0x7f1bc4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.970+0000 7f1bcdffb640 1 -- 172.21.15.50:0/1562265094 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc403f030 con 0x7f1bd01021c0 2024-04-23T18:32:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.970+0000 7f1bcdffb640 1 -- 172.21.15.50:0/1562265094 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bc4003aa0 con 0x7f1bd01021c0 2024-04-23T18:32:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.970+0000 7f1bd7210640 1 -- 172.21.15.50:0/1562265094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd019dd00 con 0x7f1bd01021c0 2024-04-23T18:32:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.970+0000 7f1bcdffb640 1 -- 172.21.15.50:0/1562265094 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4035070 con 0x7f1bd01021c0 2024-04-23T18:32:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.970+0000 7f1bd7210640 1 -- 172.21.15.50:0/1562265094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd01a5520 con 0x7f1bd01021c0 2024-04-23T18:32:43.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bcdffb640 1 -- 172.21.15.50:0/1562265094 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1bc4042030 con 0x7f1bd01021c0 2024-04-23T18:32:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bda001740 1 Processor -- start 2024-04-23T18:32:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bda001740 1 -- start start 2024-04-23T18:32:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bda001740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411d500 0x5631f411d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bda001740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411de20 0x5631f412c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bda001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f412cc10 0x5631f412f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bda001740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5631f4082970 con 0x5631f411d500 2024-04-23T18:32:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bda001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5631f401f4a0 con 0x5631f412cc10 2024-04-23T18:32:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bda001740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5631f407e930 con 0x5631f411de20 2024-04-23T18:32:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bd57ad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411d500 0x5631f411d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bcdffb640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1bc00a3e40 0x7f1bc00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bd57ad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411d500 0x5631f411d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53580/0 (socket says 172.21.15.50:53580) 2024-04-23T18:32:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bd57ad640 1 -- 172.21.15.50:0/937038946 learned_addr learned my addr 172.21.15.50:0/937038946 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bcffff640 1 --2- 172.21.15.50:0/937038946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f412cc10 0x5631f412f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/937038946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411de20 0x5631f412c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.973+0000 7f1bd57ad640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1bc00a3e40 0x7f1bc00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bcffff640 1 -- 172.21.15.50:0/937038946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411de20 msgr2=0x5631f412c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bcdffb640 1 -- 172.21.15.50:0/1562265094 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7f1bc40f0e20 con 0x7f1bd01021c0 2024-04-23T18:32:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bcffff640 1 --2- 172.21.15.50:0/937038946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411de20 0x5631f412c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bcffff640 1 -- 172.21.15.50:0/937038946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411d500 msgr2=0x5631f411d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bcffff640 1 --2- 172.21.15.50:0/937038946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411d500 0x5631f411d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bcffff640 1 -- 172.21.15.50:0/937038946 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631f3d95210 con 0x5631f412cc10 2024-04-23T18:32:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/937038946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411de20 0x5631f412c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:32:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bd57ad640 1 --2- 172.21.15.50:0/937038946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411d500 0x5631f411d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:32:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bcffff640 1 --2- 172.21.15.50:0/937038946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f412cc10 0x5631f412f000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009bf0 tx=0x7f1bbc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c24de9d70931f4f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bd57ad640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1bc00a3e40 0x7f1bc00a6300 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8005f40 tx=0x7f1bc8004ab0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bbaffd640 1 -- 172.21.15.50:0/937038946 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc004050 con 0x5631f412cc10 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bbaffd640 1 -- 172.21.15.50:0/937038946 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bbc0054d0 con 0x5631f412cc10 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.974+0000 7f1bbaffd640 1 -- 172.21.15.50:0/937038946 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc004d00 con 0x5631f412cc10 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.975+0000 7f1bda001740 1 -- 172.21.15.50:0/937038946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f412cc10 msgr2=0x5631f412f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.975+0000 7f1bda001740 1 --2- 172.21.15.50:0/937038946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f412cc10 0x5631f412f000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009bf0 tx=0x7f1bbc011810 comp rx=0 tx=0).stop 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.975+0000 7f1bda001740 1 -- 172.21.15.50:0/937038946 shutdown_connections 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.975+0000 7f1bda001740 1 --2- 172.21.15.50:0/937038946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f412cc10 0x5631f412f000 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.975+0000 7f1bda001740 1 --2- 172.21.15.50:0/937038946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411de20 0x5631f412c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.975+0000 7f1bda001740 1 --2- 172.21.15.50:0/937038946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411d500 0x5631f411d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.975+0000 7f1bda001740 1 -- 172.21.15.50:0/937038946 >> 172.21.15.50:0/937038946 conn(0x5631f4119b30 msgr2=0x5631f4119f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.975+0000 7f1bda001740 1 -- 172.21.15.50:0/937038946 shutdown_connections 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.975+0000 7f1bda001740 1 -- 172.21.15.50:0/937038946 wait complete. 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.975+0000 7f1bda001740 1 -- start start 2024-04-23T18:32:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.975+0000 7f1bda001740 1 Processor -- start 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bda001740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411d500 0x5631f412c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bda001740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 0x5631f4129750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bda001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f412cc10 0x5631f4129c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bda001740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5631f412f840 con 0x5631f411de20 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bda001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5631f401d0b0 con 0x5631f412cc10 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bda001740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5631f412f9c0 con 0x5631f411d500 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bda001740 10 client.0 _ll_register_callbacks cb 0x5631f400ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bcffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 0x5631f4129750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bcffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 0x5631f4129750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53596/0 (socket says 172.21.15.50:53596) 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bcffff640 1 -- 172.21.15.50:0/602065175 learned_addr learned my addr 172.21.15.50:0/602065175 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bd57ad640 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f412cc10 0x5631f4129c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411d500 0x5631f412c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bcffff640 1 -- 172.21.15.50:0/602065175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411d500 msgr2=0x5631f412c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bcffff640 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411d500 0x5631f412c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bcffff640 1 -- 172.21.15.50:0/602065175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f412cc10 msgr2=0x5631f4129c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bcffff640 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f412cc10 0x5631f4129c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bcffff640 1 -- 172.21.15.50:0/602065175 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631f3e61620 con 0x5631f411de20 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1bcffff640 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 0x5631f4129750 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc004740 tx=0x7f1bbc009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43acbf10f5f02e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.976+0000 7f1baffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1baffff640 1 -- 172.21.15.50:0/602065175 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc027070 con 0x5631f411de20 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1baffff640 1 -- 172.21.15.50:0/602065175 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7f1bbc003a00 con 0x5631f411de20 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1bda001740 1 -- 172.21.15.50:0/602065175 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bbc0054d0 con 0x5631f411de20 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1baffff640 1 -- 172.21.15.50:0/602065175 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f1bbc033050 con 0x5631f411de20 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1bda001740 10 client.17235 fetch_fsmap learned FSMap version 50 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1bda001740 1 -- 172.21.15.50:0/602065175 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5631f413b760 con 0x5631f411de20 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1baffff640 1 -- 172.21.15.50:0/602065175 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bbc0054d0 con 0x5631f411de20 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1baffff640 1 -- 172.21.15.50:0/602065175 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7f1bbc0050f0 con 0x5631f411de20 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1bda001740 10 client.17235 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:32:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1bda001740 10 client.17235 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1bda001740 20 client.17235 populate_metadata read hostname 'smithi050' 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1bda001740 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631f413cbd0 0x5631f413efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1bda001740 4 client.17235 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.977+0000 7f1bda001740 1 -- 172.21.15.50:0/602065175 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631f4090390 con 0x5631f413cbd0 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.978+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631f413cbd0 0x5631f413efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.978+0000 7f1baffff640 10 client.17235 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.978+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631f413cbd0 0x5631f413efc0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f90ac8e301ee0c35 server_cookie=bfc69c37115c6073 in_seq=0 out_seq=0 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1baffff640 1 -- 172.21.15.50:0/602065175 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631f4090390 con 0x5631f413cbd0 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1baffff640 10 client.17235 handle_command_reply: tid=0 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1bda001740 1 client.17235 shutdown 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1bda001740 1 -- 172.21.15.50:0/602065175 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 msgr2=0x5631f4129750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1bda001740 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 0x5631f4129750 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc004740 tx=0x7f1bbc009fd0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1bda001740 1 -- 172.21.15.50:0/602065175 shutdown_connections 2024-04-23T18:32:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1bda001740 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631f413cbd0 0x5631f413efc0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1baffff640 0 client.17235 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1bda001740 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f412cc10 0x5631f4129c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1bda001740 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 0x5631f4129750 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1bda001740 1 --2- 172.21.15.50:0/602065175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411d500 0x5631f412c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1bda001740 1 -- 172.21.15.50:0/602065175 >> 172.21.15.50:0/602065175 conn(0x5631f4119b30 msgr2=0x5631f411f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1bda001740 1 -- 172.21.15.50:0/602065175 shutdown_connections 2024-04-23T18:32:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1bda001740 1 -- 172.21.15.50:0/602065175 wait complete. 2024-04-23T18:32:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:43.979+0000 7f1bda001740 20 client.17235 trim_cache size 0 max 16384 2024-04-23T18:32:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bda001740 1 Processor -- start 2024-04-23T18:32:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bda001740 1 -- start start 2024-04-23T18:32:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bda001740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f4119b30 0x5631f41387d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bda001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f411d500 0x5631f4124d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bda001740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411de20 0x5631f412bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bda001740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01ae270 con 0x5631f4119b30 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bda001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01aba80 con 0x5631f411d500 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bda001740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01a9910 con 0x5631f411de20 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bd4fac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f4119b30 0x5631f41387d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bd4fac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f4119b30 0x5631f41387d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53598/0 (socket says 172.21.15.50:53598) 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bd4fac640 1 -- 172.21.15.50:0/2109324681 learned_addr learned my addr 172.21.15.50:0/2109324681 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bcffff640 1 --2- 172.21.15.50:0/2109324681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f411d500 0x5631f4124d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bd57ad640 1 --2- 172.21.15.50:0/2109324681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411de20 0x5631f412bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bd4fac640 1 -- 172.21.15.50:0/2109324681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411de20 msgr2=0x5631f412bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/2109324681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411de20 0x5631f412bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bd4fac640 1 -- 172.21.15.50:0/2109324681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f411d500 msgr2=0x5631f4124d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/2109324681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f411d500 0x5631f4124d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bd4fac640 1 -- 172.21.15.50:0/2109324681 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631f4119960 con 0x5631f4119b30 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.000+0000 7f1bd57ad640 1 --2- 172.21.15.50:0/2109324681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411de20 0x5631f412bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.001+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/2109324681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f4119b30 0x5631f41387d0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f1bc40ee560 tx=0x7f1bc408f8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35cee2e76beac7e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.001+0000 7f1bba7fc640 1 -- 172.21.15.50:0/2109324681 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4049040 con 0x5631f4119b30 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.001+0000 7f1bba7fc640 1 -- 172.21.15.50:0/2109324681 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bc40c59c0 con 0x5631f4119b30 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.001+0000 7f1bba7fc640 1 -- 172.21.15.50:0/2109324681 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc403bcc0 con 0x5631f4119b30 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.001+0000 7f1bda001740 1 -- 172.21.15.50:0/2109324681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f4119b30 msgr2=0x5631f41387d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.001+0000 7f1bda001740 1 --2- 172.21.15.50:0/2109324681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f4119b30 0x5631f41387d0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f1bc40ee560 tx=0x7f1bc408f8f0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 -- 172.21.15.50:0/2109324681 shutdown_connections 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 --2- 172.21.15.50:0/2109324681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f411de20 0x5631f412bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 --2- 172.21.15.50:0/2109324681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f411d500 0x5631f4124d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 --2- 172.21.15.50:0/2109324681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f4119b30 0x5631f41387d0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 -- 172.21.15.50:0/2109324681 >> 172.21.15.50:0/2109324681 conn(0x5631f4129df0 msgr2=0x5631f4124670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 -- 172.21.15.50:0/2109324681 shutdown_connections 2024-04-23T18:32:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 -- 172.21.15.50:0/2109324681 wait complete. 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 -- start start 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 Processor -- start 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f4119b30 0x5631f41307d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f411d500 0x5631f4130d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 0x5631f4131250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01a9f60 con 0x5631f411de20 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01ab3b0 con 0x5631f411d500 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01aa610 con 0x5631f4119b30 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.002+0000 7f1bda001740 10 client.0 _ll_register_callbacks cb 0x7f1bbc0043f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bd57ad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 0x5631f4131250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bd57ad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 0x5631f4131250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53614/0 (socket says 172.21.15.50:53614) 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bd57ad640 1 -- 172.21.15.50:0/3532368928 learned_addr learned my addr 172.21.15.50:0/3532368928 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bcffff640 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f411d500 0x5631f4130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f4119b30 0x5631f41307d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bd57ad640 1 -- 172.21.15.50:0/3532368928 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f4119b30 msgr2=0x5631f41307d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bd57ad640 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f4119b30 0x5631f41307d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bd57ad640 1 -- 172.21.15.50:0/3532368928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f411d500 msgr2=0x5631f4130d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bd57ad640 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f411d500 0x5631f4130d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bd57ad640 1 -- 172.21.15.50:0/3532368928 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631f41a2150 con 0x5631f411de20 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bcffff640 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f411d500 0x5631f4130d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bd57ad640 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 0x5631f4131250 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8014810 tx=0x7f1bc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79752a08108cd67 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1baffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1baffff640 1 -- 172.21.15.50:0/3532368928 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc801a040 con 0x5631f411de20 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1baffff640 1 -- 172.21.15.50:0/3532368928 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8542+0+0 (secure 0 0 0) 0x7f1bc8013a10 con 0x5631f411de20 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bda001740 1 -- 172.21.15.50:0/3532368928 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bc40c59c0 con 0x5631f411de20 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1baffff640 1 -- 172.21.15.50:0/3532368928 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f1bc8013c10 con 0x5631f411de20 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bda001740 10 client.17241 fetch_fsmap learned FSMap version 50 2024-04-23T18:32:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.003+0000 7f1bda001740 1 -- 172.21.15.50:0/3532368928 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5631f41a3430 con 0x5631f411de20 2024-04-23T18:32:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.004+0000 7f1baffff640 1 -- 172.21.15.50:0/3532368928 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bc40c59c0 con 0x5631f411de20 2024-04-23T18:32:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.004+0000 7f1baffff640 1 -- 172.21.15.50:0/3532368928 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7f1bc8020030 con 0x5631f411de20 2024-04-23T18:32:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.004+0000 7f1bda001740 10 client.17241 fetch_fsmap finished waiting for FSMap version 50 2024-04-23T18:32:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.004+0000 7f1bda001740 10 client.17241 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:32:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.004+0000 7f1bda001740 20 client.17241 populate_metadata read hostname 'smithi050' 2024-04-23T18:32:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.004+0000 7f1bda001740 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631f412cc10 0x5631f41a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.004+0000 7f1bda001740 4 client.17241 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:32:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.004+0000 7f1bda001740 1 -- 172.21.15.50:0/3532368928 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5631f408a200 con 0x5631f412cc10 2024-04-23T18:32:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.004+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631f412cc10 0x5631f41a5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.004+0000 7f1baffff640 10 client.17241 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.004+0000 7f1bd4fac640 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631f412cc10 0x5631f41a5c80 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21665f190a4501a server_cookie=78195eac32c966cd in_seq=0 out_seq=0 2024-04-23T18:32:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.245+0000 7f1baffff640 1 -- 172.21.15.50:0/3532368928 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5631f408a200 con 0x5631f412cc10 2024-04-23T18:32:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.245+0000 7f1baffff640 10 client.17241 handle_command_reply: tid=0 2024-04-23T18:32:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.245+0000 7f1bda001740 1 client.17241 shutdown 2024-04-23T18:32:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.246+0000 7f1bda001740 1 -- 172.21.15.50:0/3532368928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 msgr2=0x5631f4131250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.246+0000 7f1bda001740 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 0x5631f4131250 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8014810 tx=0x7f1bc8004a40 comp rx=0 tx=0).stop 2024-04-23T18:32:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.246+0000 7f1bda001740 1 -- 172.21.15.50:0/3532368928 shutdown_connections 2024-04-23T18:32:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.246+0000 7f1bda001740 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5631f412cc10 0x5631f41a5c80 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.246+0000 7f1bda001740 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5631f411de20 0x5631f4131250 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.246+0000 7f1bda001740 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631f411d500 0x5631f4130d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.246+0000 7f1baffff640 0 client.17241 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.246+0000 7f1bda001740 1 --2- 172.21.15.50:0/3532368928 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631f4119b30 0x5631f41307d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.246+0000 7f1bda001740 1 -- 172.21.15.50:0/3532368928 >> 172.21.15.50:0/3532368928 conn(0x5631f4129df0 msgr2=0x5631f4124d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.246+0000 7f1bda001740 1 -- 172.21.15.50:0/3532368928 shutdown_connections 2024-04-23T18:32:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.246+0000 7f1bda001740 1 -- 172.21.15.50:0/3532368928 wait complete. 2024-04-23T18:32:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.246+0000 7f1bda001740 20 client.17241 trim_cache size 0 max 16384 2024-04-23T18:32:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:32:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:32:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:32:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:32:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.247+0000 7f1bd7210640 1 -- 172.21.15.50:0/1562265094 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1bc00a3e40 msgr2=0x7f1bc00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.247+0000 7f1bd7210640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1bc00a3e40 0x7f1bc00a6300 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8005f40 tx=0x7f1bc8004ab0 comp rx=0 tx=0).stop 2024-04-23T18:32:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.247+0000 7f1bd7210640 1 -- 172.21.15.50:0/1562265094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd01021c0 msgr2=0x7f1bd01986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.247+0000 7f1bd7210640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd01021c0 0x7f1bd01986c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4005940 tx=0x7f1bc4004270 comp rx=0 tx=0).stop 2024-04-23T18:32:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.247+0000 7f1bd7210640 1 -- 172.21.15.50:0/1562265094 shutdown_connections 2024-04-23T18:32:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.247+0000 7f1bd7210640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1bc00a3e40 0x7f1bc00a6300 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.247+0000 7f1bd7210640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bd019cf80 0x7f1bd019d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.247+0000 7f1bd7210640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1bd0107290 0x7f1bd0198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.247+0000 7f1bd7210640 1 --2- 172.21.15.50:0/1562265094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bd01021c0 0x7f1bd01986c0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.247+0000 7f1bd7210640 1 -- 172.21.15.50:0/1562265094 >> 172.21.15.50:0/1562265094 conn(0x7f1bd00fda20 msgr2=0x7f1bd00ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.247+0000 7f1bd7210640 1 -- 172.21.15.50:0/1562265094 shutdown_connections 2024-04-23T18:32:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:44.247+0000 7f1bd7210640 1 -- 172.21.15.50:0/1562265094 wait complete. 2024-04-23T18:32:44.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:44 smithi050 ceph-mon[29187]: pgmap v448: 97 pgs: 97 active+clean; 711 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 15 MiB/s wr, 1.00k op/s 2024-04-23T18:32:44.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:44 smithi177 ceph-mon[32308]: pgmap v448: 97 pgs: 97 active+clean; 711 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 15 MiB/s wr, 1.00k op/s 2024-04-23T18:32:44.537 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:32:44 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: ignoring --setuser ceph since I am not root 2024-04-23T18:32:44.537 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:32:44 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: ignoring --setgroup ceph since I am not root 2024-04-23T18:32:44.537 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:32:44 smithi177 ceph-mds[66745]: -- 172.21.15.177:0/2347795199 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f84c83ba40 con 0x55f84c80e800 2024-04-23T18:32:44.537 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:32:44 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: starting mds.h at 2024-04-23T18:32:44.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 38.20 MB/sec warmup 78 sec latency 1210.963 ms 2024-04-23T18:32:44.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:44 smithi188 ceph-mon[32694]: pgmap v448: 97 pgs: 97 active+clean; 711 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 15 MiB/s wr, 1.00k op/s 2024-04-23T18:32:45.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:45 smithi050 ceph-mon[29187]: daemon mds.h restarted 2024-04-23T18:32:45.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:45 smithi177 ceph-mon[32308]: daemon mds.h restarted 2024-04-23T18:32:45.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 37.71 MB/sec warmup 79 sec latency 2211.044 ms 2024-04-23T18:32:45.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:45 smithi188 ceph-mon[32694]: daemon mds.h restarted 2024-04-23T18:32:46.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:46 smithi050 ceph-mon[29187]: pgmap v449: 97 pgs: 97 active+clean; 711 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 11 MiB/s wr, 566 op/s 2024-04-23T18:32:46.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:46 smithi050 ceph-mon[29187]: osdmap e110: 12 total, 12 up, 12 in 2024-04-23T18:32:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:46 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6836/1119979694,v1:172.21.15.177:6837/1119979694] up:boot 2024-04-23T18:32:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:46 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:32:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:46 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:32:46.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:46 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:32:46.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:46 smithi177 ceph-mon[32308]: pgmap v449: 97 pgs: 97 active+clean; 711 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 11 MiB/s wr, 566 op/s 2024-04-23T18:32:46.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:46 smithi177 ceph-mon[32308]: osdmap e110: 12 total, 12 up, 12 in 2024-04-23T18:32:46.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:46 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6836/1119979694,v1:172.21.15.177:6837/1119979694] up:boot 2024-04-23T18:32:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:46 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:32:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:46 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:32:46.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:46 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:32:46.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 37.24 MB/sec warmup 80 sec latency 3211.095 ms 2024-04-23T18:32:46.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:46 smithi188 ceph-mon[32694]: pgmap v449: 97 pgs: 97 active+clean; 711 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 11 MiB/s wr, 566 op/s 2024-04-23T18:32:46.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:46 smithi188 ceph-mon[32694]: osdmap e110: 12 total, 12 up, 12 in 2024-04-23T18:32:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:46 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6836/1119979694,v1:172.21.15.177:6837/1119979694] up:boot 2024-04-23T18:32:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:46 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:32:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:46 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:32:46.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:46 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:32:47.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 36.78 MB/sec warmup 81 sec latency 4211.205 ms 2024-04-23T18:32:48.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:48 smithi050 ceph-mon[29187]: pgmap v451: 97 pgs: 97 active+clean; 699 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 744 op/s 2024-04-23T18:32:48.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:48 smithi177 ceph-mon[32308]: pgmap v451: 97 pgs: 97 active+clean; 699 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 744 op/s 2024-04-23T18:32:48.556 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 36.33 MB/sec warmup 82 sec latency 5211.264 ms 2024-04-23T18:32:48.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:48 smithi188 ceph-mon[32694]: pgmap v451: 97 pgs: 97 active+clean; 699 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 744 op/s 2024-04-23T18:32:49.557 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 35.90 MB/sec warmup 83 sec latency 6211.411 ms 2024-04-23T18:32:50.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:50 smithi050 ceph-mon[29187]: pgmap v452: 97 pgs: 97 active+clean; 699 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 744 op/s 2024-04-23T18:32:50.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:50 smithi177 ceph-mon[32308]: pgmap v452: 97 pgs: 97 active+clean; 699 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 744 op/s 2024-04-23T18:32:50.557 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 35.47 MB/sec warmup 84 sec latency 7211.559 ms 2024-04-23T18:32:50.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:50 smithi188 ceph-mon[32694]: pgmap v452: 97 pgs: 97 active+clean; 699 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 744 op/s 2024-04-23T18:32:51.557 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 35.05 MB/sec warmup 85 sec latency 8211.627 ms 2024-04-23T18:32:52.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:52 smithi050 ceph-mon[29187]: pgmap v453: 97 pgs: 97 active+clean; 699 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 744 op/s 2024-04-23T18:32:52.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:52 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:32:52.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:52 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:32:52.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:52 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:32:52.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:52 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:32:52.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:52 smithi177 ceph-mon[32308]: pgmap v453: 97 pgs: 97 active+clean; 699 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 744 op/s 2024-04-23T18:32:52.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:52 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:32:52.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:52 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:32:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:52 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:32:52.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:52 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:32:52.557 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 34.64 MB/sec warmup 86 sec latency 9211.720 ms 2024-04-23T18:32:52.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:52 smithi188 ceph-mon[32694]: pgmap v453: 97 pgs: 97 active+clean; 699 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 744 op/s 2024-04-23T18:32:52.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:52 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:32:52.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:52 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:32:52.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:52 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:32:52.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:52 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:32:53.557 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 34.25 MB/sec warmup 87 sec latency 10211.862 ms 2024-04-23T18:32:54.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:54 smithi050 ceph-mon[29187]: pgmap v454: 97 pgs: 97 active+clean; 692 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.7 MiB/s wr, 520 op/s 2024-04-23T18:32:54.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:54 smithi177 ceph-mon[32308]: pgmap v454: 97 pgs: 97 active+clean; 692 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.7 MiB/s wr, 520 op/s 2024-04-23T18:32:54.557 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 33.86 MB/sec warmup 88 sec latency 11211.972 ms 2024-04-23T18:32:54.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:54 smithi188 ceph-mon[32694]: pgmap v454: 97 pgs: 97 active+clean; 692 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.7 MiB/s wr, 520 op/s 2024-04-23T18:32:55.557 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 33.48 MB/sec warmup 89 sec latency 12212.110 ms 2024-04-23T18:32:56.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:56 smithi050 ceph-mon[29187]: pgmap v455: 97 pgs: 97 active+clean; 692 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.7 MiB/s wr, 520 op/s 2024-04-23T18:32:56.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:56 smithi177 ceph-mon[32308]: pgmap v455: 97 pgs: 97 active+clean; 692 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.7 MiB/s wr, 520 op/s 2024-04-23T18:32:56.557 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 33.10 MB/sec warmup 90 sec latency 13212.200 ms 2024-04-23T18:32:56.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:56 smithi188 ceph-mon[32694]: pgmap v455: 97 pgs: 97 active+clean; 692 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.7 MiB/s wr, 520 op/s 2024-04-23T18:32:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.350+0000 7f5b5bf20640 1 -- 172.21.15.50:0/3265059743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b54106170 msgr2=0x7f5b5410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.350+0000 7f5b5bf20640 1 --2- 172.21.15.50:0/3265059743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b54106170 0x7f5b5410dd40 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c009ca0 tx=0x7f5b4c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.350+0000 7f5b5bf20640 1 -- 172.21.15.50:0/3265059743 shutdown_connections 2024-04-23T18:32:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.350+0000 7f5b5bf20640 1 --2- 172.21.15.50:0/3265059743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b54106170 0x7f5b5410dd40 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.350+0000 7f5b5bf20640 1 --2- 172.21.15.50:0/3265059743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b54101aa0 0x7f5b54105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.350+0000 7f5b5bf20640 1 --2- 172.21.15.50:0/3265059743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b541010f0 0x7f5b541014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.350+0000 7f5b5bf20640 1 -- 172.21.15.50:0/3265059743 >> 172.21.15.50:0/3265059743 conn(0x7f5b54077fd0 msgr2=0x7f5b540783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.350+0000 7f5b5bf20640 1 -- 172.21.15.50:0/3265059743 shutdown_connections 2024-04-23T18:32:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.350+0000 7f5b5bf20640 1 -- 172.21.15.50:0/3265059743 wait complete. 2024-04-23T18:32:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b5bf20640 1 Processor -- start 2024-04-23T18:32:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b5bf20640 1 -- start start 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b5bf20640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b541010f0 0x7f5b54197210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b5bf20640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b54106170 0x7f5b54197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b5bf20640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b54197c90 0x7f5b541a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b5bf20640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b541110c0 con 0x7f5b54106170 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b5bf20640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5b54110f40 con 0x7f5b541010f0 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b5bf20640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5b54111240 con 0x7f5b54197c90 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b594bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b54106170 0x7f5b54197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b594bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b54106170 0x7f5b54197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51066/0 (socket says 172.21.15.50:51066) 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b594bb640 1 -- 172.21.15.50:0/1158554897 learned_addr learned my addr 172.21.15.50:0/1158554897 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b59cbc640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b541010f0 0x7f5b54197210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b54197c90 0x7f5b541a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b594bb640 1 -- 172.21.15.50:0/1158554897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b54197c90 msgr2=0x7f5b541a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b594bb640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b54197c90 0x7f5b541a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b594bb640 1 -- 172.21.15.50:0/1158554897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b541010f0 msgr2=0x7f5b54197210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b594bb640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b541010f0 0x7f5b54197210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b594bb640 1 -- 172.21.15.50:0/1158554897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b541a6e50 con 0x7f5b54106170 2024-04-23T18:32:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b54197c90 0x7f5b541a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:32:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.351+0000 7f5b59cbc640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b541010f0 0x7f5b54197210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:32:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.352+0000 7f5b594bb640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b54106170 0x7f5b54197750 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400c9a0 tx=0x7f5b4400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.352+0000 7f5b4affd640 1 -- 172.21.15.50:0/1158554897 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44014070 con 0x7f5b54106170 2024-04-23T18:32:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.352+0000 7f5b4affd640 1 -- 172.21.15.50:0/1158554897 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b44004440 con 0x7f5b54106170 2024-04-23T18:32:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.352+0000 7f5b5bf20640 1 -- 172.21.15.50:0/1158554897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b541a7140 con 0x7f5b54106170 2024-04-23T18:32:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.352+0000 7f5b4affd640 1 -- 172.21.15.50:0/1158554897 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44003de0 con 0x7f5b54106170 2024-04-23T18:32:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.352+0000 7f5b5bf20640 1 -- 172.21.15.50:0/1158554897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b541a7690 con 0x7f5b54106170 2024-04-23T18:32:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.355+0000 7f5b4affd640 1 -- 172.21.15.50:0/1158554897 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5b44017070 con 0x7f5b54106170 2024-04-23T18:32:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.355+0000 7f5b5ec3a740 1 Processor -- start 2024-04-23T18:32:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.355+0000 7f5b5ec3a740 1 -- start start 2024-04-23T18:32:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.355+0000 7f5b5ec3a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344b620 0x55830344ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5ec3a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344bf40 0x5583034562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5ec3a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558303456800 0x558303458bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5ec3a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5583033b0970 con 0x55830344b620 2024-04-23T18:32:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5ec3a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55830334d4a0 con 0x558303456800 2024-04-23T18:32:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5ec3a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5583033ac930 con 0x55830344bf40 2024-04-23T18:32:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5a4bd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344b620 0x55830344ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5a4bd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344b620 0x55830344ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51074/0 (socket says 172.21.15.50:51074) 2024-04-23T18:32:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5a4bd640 1 -- 172.21.15.50:0/3844980184 learned_addr learned my addr 172.21.15.50:0/3844980184 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b59cbc640 1 --2- 172.21.15.50:0/3844980184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344bf40 0x5583034562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b4affd640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5b400a3dc0 0x7f5b400a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b594bb640 1 --2- 172.21.15.50:0/3844980184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558303456800 0x558303458bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5a4bd640 1 -- 172.21.15.50:0/3844980184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344bf40 msgr2=0x5583034562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/3844980184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344bf40 0x5583034562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5a4bd640 1 -- 172.21.15.50:0/3844980184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558303456800 msgr2=0x558303458bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/3844980184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558303456800 0x558303458bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5a4bd640 1 -- 172.21.15.50:0/3844980184 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583030c3210 con 0x55830344b620 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b594bb640 1 --2- 172.21.15.50:0/3844980184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558303456800 0x558303458bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b4affd640 1 -- 172.21.15.50:0/1158554897 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8498+0+0 (secure 0 0 0) 0x7f5b440d06b0 con 0x7f5b54106170 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5b400a3dc0 0x7f5b400a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.356+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/3844980184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344b620 0x55830344ba00 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c031b70 tx=0x7f5b4c031ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dfc429e4102bdba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.357+0000 7f5b3ffff640 1 -- 172.21.15.50:0/3844980184 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c004040 con 0x55830344b620 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.357+0000 7f5b3ffff640 1 -- 172.21.15.50:0/3844980184 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b4c036070 con 0x55830344b620 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.357+0000 7f5b3ffff640 1 -- 172.21.15.50:0/3844980184 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c02fbc0 con 0x55830344b620 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.357+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3844980184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344b620 msgr2=0x55830344ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.357+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5b400a3dc0 0x7f5b400a6280 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c04b000 tx=0x7f5b4c048000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.357+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/3844980184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344b620 0x55830344ba00 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c031b70 tx=0x7f5b4c031ba0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.357+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3844980184 shutdown_connections 2024-04-23T18:32:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.357+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/3844980184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558303456800 0x558303458bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.357+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/3844980184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344bf40 0x5583034562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.357+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/3844980184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344b620 0x55830344ba00 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.357+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3844980184 >> 172.21.15.50:0/3844980184 conn(0x558303447b30 msgr2=0x558303447f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.358+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3844980184 shutdown_connections 2024-04-23T18:32:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.358+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3844980184 wait complete. 2024-04-23T18:32:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.358+0000 7f5b5ec3a740 1 -- start start 2024-04-23T18:32:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.358+0000 7f5b5ec3a740 1 Processor -- start 2024-04-23T18:32:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b5ec3a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55830344b620 0x5583034561c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b5ec3a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 0x5583034534b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b5ec3a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558303456800 0x558303453ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b5ec3a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5583034592b0 con 0x55830344bf40 2024-04-23T18:32:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b5ec3a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55830334b0b0 con 0x55830344b620 2024-04-23T18:32:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b5ec3a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558303459430 con 0x558303456800 2024-04-23T18:32:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b5ec3a740 10 client.0 _ll_register_callbacks cb 0x558303338d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b594bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 0x5583034534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b594bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 0x5583034534b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51090/0 (socket says 172.21.15.50:51090) 2024-04-23T18:32:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b594bb640 1 -- 172.21.15.50:0/3168828783 learned_addr learned my addr 172.21.15.50:0/3168828783 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b59cbc640 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55830344b620 0x5583034561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558303456800 0x558303453ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b594bb640 1 -- 172.21.15.50:0/3168828783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558303456800 msgr2=0x558303453ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b594bb640 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558303456800 0x558303453ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b594bb640 1 -- 172.21.15.50:0/3168828783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55830344b620 msgr2=0x5583034561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b594bb640 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55830344b620 0x5583034561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b594bb640 1 -- 172.21.15.50:0/3168828783 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55830318f620 con 0x55830344bf40 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558303456800 0x558303453ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b594bb640 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 0x5583034534b0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5b440cde10 tx=0x7f5b440038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdac785b9574ae58 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b3d7fa640 1 -- 172.21.15.50:0/3168828783 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44098040 con 0x55830344bf40 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b3d7fa640 1 -- 172.21.15.50:0/3168828783 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8498+0+0 (secure 0 0 0) 0x7f5b440a4de0 con 0x55830344bf40 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.359+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3168828783 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b4c036070 con 0x55830344bf40 2024-04-23T18:32:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.360+0000 7f5b3d7fa640 1 -- 172.21.15.50:0/3168828783 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f5b440ce230 con 0x55830344bf40 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.360+0000 7f5b5ec3a740 10 client.17253 fetch_fsmap learned FSMap version 52 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.360+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3168828783 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x5583034652a0 con 0x55830344bf40 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.360+0000 7f5b3d7fa640 1 -- 172.21.15.50:0/3168828783 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b4c036070 con 0x55830344bf40 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.360+0000 7f5b3d7fa640 1 -- 172.21.15.50:0/3168828783 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f5b4409ba50 con 0x55830344bf40 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.360+0000 7f5b5ec3a740 10 client.17253 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.360+0000 7f5b5ec3a740 10 client.17253 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.360+0000 7f5b5ec3a740 20 client.17253 populate_metadata read hostname 'smithi050' 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.360+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x558303466760 0x558303468b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.360+0000 7f5b5ec3a740 4 client.17253 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.360+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3168828783 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583033be390 con 0x558303466760 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.360+0000 7f5b59cbc640 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x558303466760 0x558303468b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.361+0000 7f5b59cbc640 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x558303466760 0x558303468b70 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88b18d535ea6ec65 server_cookie=a0fe5d17b32ca19f in_seq=0 out_seq=0 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.361+0000 7f5b3d7fa640 10 client.17253 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.361+0000 7f5b3d7fa640 1 -- 172.21.15.50:0/3168828783 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5583033be390 con 0x558303466760 2024-04-23T18:32:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.361+0000 7f5b3d7fa640 10 client.17253 handle_command_reply: tid=0 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.361+0000 7f5b5ec3a740 1 client.17253 shutdown 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.362+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3168828783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 msgr2=0x5583034534b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.362+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 0x5583034534b0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5b440cde10 tx=0x7f5b440038b0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.362+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3168828783 shutdown_connections 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.362+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x558303466760 0x558303468b70 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.362+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558303456800 0x558303453ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.362+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 0x5583034534b0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.362+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/3168828783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55830344b620 0x5583034561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.362+0000 7f5b3d7fa640 0 client.17253 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.362+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3168828783 >> 172.21.15.50:0/3168828783 conn(0x558303447b30 msgr2=0x558303446cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.362+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3168828783 shutdown_connections 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.362+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/3168828783 wait complete. 2024-04-23T18:32:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.362+0000 7f5b5ec3a740 20 client.17253 trim_cache size 0 max 16384 2024-04-23T18:32:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.377+0000 7f5b5ec3a740 1 Processor -- start 2024-04-23T18:32:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.377+0000 7f5b5ec3a740 1 -- start start 2024-04-23T18:32:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.377+0000 7f5b5ec3a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344b620 0x558303462170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.377+0000 7f5b5ec3a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 0x558303452610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.377+0000 7f5b5ec3a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558303456800 0x558303446b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.377+0000 7f5b5ec3a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b541b25f0 con 0x55830344bf40 2024-04-23T18:32:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.377+0000 7f5b5ec3a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5b541afe00 con 0x558303456800 2024-04-23T18:32:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.377+0000 7f5b5ec3a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5b541adc50 con 0x55830344b620 2024-04-23T18:32:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.377+0000 7f5b594bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 0x558303452610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.377+0000 7f5b594bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 0x558303452610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51092/0 (socket says 172.21.15.50:51092) 2024-04-23T18:32:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.377+0000 7f5b594bb640 1 -- 172.21.15.50:0/2155121845 learned_addr learned my addr 172.21.15.50:0/2155121845 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.377+0000 7f5b59cbc640 1 --2- 172.21.15.50:0/2155121845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344b620 0x558303462170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.378+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/2155121845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558303456800 0x558303446b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.378+0000 7f5b594bb640 1 -- 172.21.15.50:0/2155121845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344b620 msgr2=0x558303462170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.378+0000 7f5b594bb640 1 --2- 172.21.15.50:0/2155121845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344b620 0x558303462170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.378+0000 7f5b594bb640 1 -- 172.21.15.50:0/2155121845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558303456800 msgr2=0x558303446b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.378+0000 7f5b594bb640 1 --2- 172.21.15.50:0/2155121845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558303456800 0x558303446b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.378+0000 7f5b594bb640 1 -- 172.21.15.50:0/2155121845 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558303451a00 con 0x55830344bf40 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.378+0000 7f5b594bb640 1 --2- 172.21.15.50:0/2155121845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 0x558303452610 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f5b44087170 tx=0x7f5b44007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907ad9c8a0f8fa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.378+0000 7f5b3f7fe640 1 -- 172.21.15.50:0/2155121845 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4409c600 con 0x55830344bf40 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.378+0000 7f5b3f7fe640 1 -- 172.21.15.50:0/2155121845 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b440a09b0 con 0x55830344bf40 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.378+0000 7f5b3f7fe640 1 -- 172.21.15.50:0/2155121845 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b440984a0 con 0x55830344bf40 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.379+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/2155121845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 msgr2=0x558303452610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.379+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/2155121845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 0x558303452610 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f5b44087170 tx=0x7f5b44007ef0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.379+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/2155121845 shutdown_connections 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.379+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/2155121845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558303456800 0x558303446b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.379+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/2155121845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55830344bf40 0x558303452610 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.379+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/2155121845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344b620 0x558303462170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.379+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/2155121845 >> 172.21.15.50:0/2155121845 conn(0x558303447b30 msgr2=0x558303446420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.379+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/2155121845 shutdown_connections 2024-04-23T18:32:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.379+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/2155121845 wait complete. 2024-04-23T18:32:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.379+0000 7f5b5ec3a740 1 -- start start 2024-04-23T18:32:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.379+0000 7f5b5ec3a740 1 Processor -- start 2024-04-23T18:32:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5ec3a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344b620 0x55830345ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5ec3a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55830344bf40 0x558303459f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5ec3a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558303456800 0x55830345a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5ec3a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b541ae2c0 con 0x558303456800 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5ec3a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5b541af730 con 0x55830344bf40 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5ec3a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5b541ae990 con 0x55830344b620 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5ec3a740 10 client.0 _ll_register_callbacks cb 0x7f5b4409c4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5a4bd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558303456800 0x55830345a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5a4bd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558303456800 0x55830345a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51096/0 (socket says 172.21.15.50:51096) 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5a4bd640 1 -- 172.21.15.50:0/1608724923 learned_addr learned my addr 172.21.15.50:0/1608724923 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b594bb640 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55830344bf40 0x558303459f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b59cbc640 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344b620 0x55830345ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5a4bd640 1 -- 172.21.15.50:0/1608724923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344b620 msgr2=0x55830345ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344b620 0x55830345ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5a4bd640 1 -- 172.21.15.50:0/1608724923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55830344bf40 msgr2=0x558303459f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55830344bf40 0x558303459f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5a4bd640 1 -- 172.21.15.50:0/1608724923 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583034cbcb0 con 0x558303456800 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b5a4bd640 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558303456800 0x55830345a670 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c009ca0 tx=0x7f5b4c02fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97fd2fd27edd34f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.380+0000 7f5b3d7fa640 1 -- 172.21.15.50:0/1608724923 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c060060 con 0x558303456800 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b3d7fa640 1 -- 172.21.15.50:0/1608724923 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8498+0+0 (secure 0 0 0) 0x7f5b4c02fe70 con 0x558303456800 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/1608724923 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b440a09b0 con 0x558303456800 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b3d7fa640 1 -- 172.21.15.50:0/1608724923 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f5b4c004250 con 0x558303456800 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b5ec3a740 10 client.17259 fetch_fsmap learned FSMap version 52 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/1608724923 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x5583034cce60 con 0x558303456800 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b3d7fa640 1 -- 172.21.15.50:0/1608724923 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b440a09b0 con 0x558303456800 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b3d7fa640 1 -- 172.21.15.50:0/1608724923 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f5b4c05a030 con 0x558303456800 2024-04-23T18:32:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b5ec3a740 10 client.17259 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:32:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b5ec3a740 10 client.17259 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:32:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b5ec3a740 20 client.17259 populate_metadata read hostname 'smithi050' 2024-04-23T18:32:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5583034cd2a0 0x5583034cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:32:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b5ec3a740 4 client.17259 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:32:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/1608724923 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5583033b8200 con 0x5583034cd2a0 2024-04-23T18:32:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.381+0000 7f5b59cbc640 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5583034cd2a0 0x5583034cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:32:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.382+0000 7f5b59cbc640 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5583034cd2a0 0x5583034cf6d0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fdaf54240ce5c3d server_cookie=7679b13184db4e94 in_seq=0 out_seq=0 2024-04-23T18:32:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.382+0000 7f5b3d7fa640 10 client.17259 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b3d7fa640 1 -- 172.21.15.50:0/1608724923 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5583033b8200 con 0x5583034cd2a0 2024-04-23T18:32:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b3d7fa640 10 client.17259 handle_command_reply: tid=0 2024-04-23T18:32:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5ec3a740 1 client.17259 shutdown 2024-04-23T18:32:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/1608724923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558303456800 msgr2=0x55830345a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558303456800 0x55830345a670 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c009ca0 tx=0x7f5b4c02fc70 comp rx=0 tx=0).stop 2024-04-23T18:32:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/1608724923 shutdown_connections 2024-04-23T18:32:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5583034cd2a0 0x5583034cf6d0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558303456800 0x55830345a670 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55830344bf40 0x558303459f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b3d7fa640 0 client.17259 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:32:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5ec3a740 1 --2- 172.21.15.50:0/1608724923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55830344b620 0x55830345ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/1608724923 >> 172.21.15.50:0/1608724923 conn(0x558303447b30 msgr2=0x558303451d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/1608724923 shutdown_connections 2024-04-23T18:32:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5ec3a740 1 -- 172.21.15.50:0/1608724923 wait complete. 2024-04-23T18:32:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5ec3a740 20 client.17259 trim_cache size 0 max 16384 2024-04-23T18:32:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:32:57.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:32:57.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:32:57.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:32:57.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5bf20640 1 -- 172.21.15.50:0/1158554897 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5b400a3dc0 msgr2=0x7f5b400a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5bf20640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5b400a3dc0 0x7f5b400a6280 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c04b000 tx=0x7f5b4c048000 comp rx=0 tx=0).stop 2024-04-23T18:32:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5bf20640 1 -- 172.21.15.50:0/1158554897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b54106170 msgr2=0x7f5b54197750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:32:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5bf20640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b54106170 0x7f5b54197750 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400c9a0 tx=0x7f5b4400ce60 comp rx=0 tx=0).stop 2024-04-23T18:32:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5bf20640 1 -- 172.21.15.50:0/1158554897 shutdown_connections 2024-04-23T18:32:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5bf20640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5b400a3dc0 0x7f5b400a6280 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5bf20640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b54197c90 0x7f5b541a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5bf20640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b54106170 0x7f5b54197750 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5bf20640 1 --2- 172.21.15.50:0/1158554897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b541010f0 0x7f5b54197210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:32:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5bf20640 1 -- 172.21.15.50:0/1158554897 >> 172.21.15.50:0/1158554897 conn(0x7f5b54077fd0 msgr2=0x7f5b540fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:32:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5bf20640 1 -- 172.21.15.50:0/1158554897 shutdown_connections 2024-04-23T18:32:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:32:57.487+0000 7f5b5bf20640 1 -- 172.21.15.50:0/1158554897 wait complete. 2024-04-23T18:32:57.557 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 32.74 MB/sec warmup 91 sec latency 14212.272 ms 2024-04-23T18:32:58.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:32:58 smithi050 ceph-mon[29187]: pgmap v456: 97 pgs: 97 active+clean; 693 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 9.3 MiB/s wr, 539 op/s 2024-04-23T18:32:58.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:32:58 smithi177 ceph-mon[32308]: pgmap v456: 97 pgs: 97 active+clean; 693 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 9.3 MiB/s wr, 539 op/s 2024-04-23T18:32:58.557 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 32.38 MB/sec warmup 92 sec latency 15212.401 ms 2024-04-23T18:32:58.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:32:58 smithi188 ceph-mon[32694]: pgmap v456: 97 pgs: 97 active+clean; 693 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 9.3 MiB/s wr, 539 op/s 2024-04-23T18:32:59.558 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 32.04 MB/sec warmup 93 sec latency 16212.509 ms 2024-04-23T18:33:00.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:00 smithi050 ceph-mon[29187]: pgmap v457: 97 pgs: 97 active+clean; 693 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 5.2 MiB/s wr, 184 op/s 2024-04-23T18:33:00.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:00 smithi177 ceph-mon[32308]: pgmap v457: 97 pgs: 97 active+clean; 693 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 5.2 MiB/s wr, 184 op/s 2024-04-23T18:33:00.558 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 31.70 MB/sec warmup 94 sec latency 17212.558 ms 2024-04-23T18:33:00.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:00 smithi188 ceph-mon[32694]: pgmap v457: 97 pgs: 97 active+clean; 693 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 5.2 MiB/s wr, 184 op/s 2024-04-23T18:33:01.558 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 31.36 MB/sec warmup 95 sec latency 18212.644 ms 2024-04-23T18:33:02.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:02 smithi177 ceph-mon[32308]: pgmap v458: 97 pgs: 97 active+clean; 693 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 5.2 MiB/s wr, 184 op/s 2024-04-23T18:33:02.558 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 31.04 MB/sec warmup 96 sec latency 19212.753 ms 2024-04-23T18:33:02.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:02 smithi188 ceph-mon[32694]: pgmap v458: 97 pgs: 97 active+clean; 693 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 5.2 MiB/s wr, 184 op/s 2024-04-23T18:33:02.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:02 smithi050 ceph-mon[29187]: pgmap v458: 97 pgs: 97 active+clean; 693 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 5.2 MiB/s wr, 184 op/s 2024-04-23T18:33:03.558 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 30.72 MB/sec warmup 97 sec latency 20212.840 ms 2024-04-23T18:33:04.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:04 smithi177 ceph-mon[32308]: pgmap v459: 97 pgs: 97 active+clean; 690 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 7.7 MiB/s wr, 315 op/s 2024-04-23T18:33:04.558 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 30.40 MB/sec warmup 98 sec latency 21212.881 ms 2024-04-23T18:33:04.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:04 smithi188 ceph-mon[32694]: pgmap v459: 97 pgs: 97 active+clean; 690 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 7.7 MiB/s wr, 315 op/s 2024-04-23T18:33:04.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:04 smithi050 ceph-mon[29187]: pgmap v459: 97 pgs: 97 active+clean; 690 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 7.7 MiB/s wr, 315 op/s 2024-04-23T18:33:05.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:33:05.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.415+0000 7f5e84143640 1 -- 172.21.15.50:0/583646313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e7c102bf0 msgr2=0x7f5e7c106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.415+0000 7f5e84143640 1 --2- 172.21.15.50:0/583646313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e7c102bf0 0x7f5e7c106ba0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009900 tx=0x7f5e6c02a410 comp rx=0 tx=0).stop 2024-04-23T18:33:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.415+0000 7f5e84143640 1 -- 172.21.15.50:0/583646313 shutdown_connections 2024-04-23T18:33:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.415+0000 7f5e84143640 1 --2- 172.21.15.50:0/583646313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c107280 0x7f5e7c10ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.415+0000 7f5e84143640 1 --2- 172.21.15.50:0/583646313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e7c102bf0 0x7f5e7c106ba0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.415+0000 7f5e84143640 1 --2- 172.21.15.50:0/583646313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c102240 0x7f5e7c102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.415+0000 7f5e84143640 1 -- 172.21.15.50:0/583646313 >> 172.21.15.50:0/583646313 conn(0x7f5e7c0790e0 msgr2=0x7f5e7c0794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.415+0000 7f5e84143640 1 -- 172.21.15.50:0/583646313 shutdown_connections 2024-04-23T18:33:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.415+0000 7f5e84143640 1 -- 172.21.15.50:0/583646313 wait complete. 2024-04-23T18:33:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e84143640 1 Processor -- start 2024-04-23T18:33:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e84143640 1 -- start start 2024-04-23T18:33:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e84143640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c102240 0x7f5e7c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e84143640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c107280 0x7f5e7c198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e84143640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e7c19d020 0x7f5e7c19d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e84143640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c1121c0 con 0x7f5e7c19d020 2024-04-23T18:33:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e84143640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c112040 con 0x7f5e7c102240 2024-04-23T18:33:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e84143640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c112340 con 0x7f5e7c107280 2024-04-23T18:33:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e826e0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e7c19d020 0x7f5e7c19d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e81edf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c102240 0x7f5e7c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e826e0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e7c19d020 0x7f5e7c19d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50752/0 (socket says 172.21.15.50:50752) 2024-04-23T18:33:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e826e0640 1 -- 172.21.15.50:0/248017705 learned_addr learned my addr 172.21.15.50:0/248017705 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.416+0000 7f5e816de640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c107280 0x7f5e7c198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.417+0000 7f5e826e0640 1 -- 172.21.15.50:0/248017705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c107280 msgr2=0x7f5e7c198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.417+0000 7f5e826e0640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c107280 0x7f5e7c198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.417+0000 7f5e826e0640 1 -- 172.21.15.50:0/248017705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c102240 msgr2=0x7f5e7c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.417+0000 7f5e826e0640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c102240 0x7f5e7c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.417+0000 7f5e826e0640 1 -- 172.21.15.50:0/248017705 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e7c19db10 con 0x7f5e7c19d020 2024-04-23T18:33:05.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.417+0000 7f5e816de640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c107280 0x7f5e7c198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:33:05.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.417+0000 7f5e826e0640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e7c19d020 0x7f5e7c19d4d0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400eb10 tx=0x7f5e7400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.417+0000 7f5e72ffd640 1 -- 172.21.15.50:0/248017705 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74004b20 con 0x7f5e7c19d020 2024-04-23T18:33:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.417+0000 7f5e72ffd640 1 -- 172.21.15.50:0/248017705 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e74004cc0 con 0x7f5e7c19d020 2024-04-23T18:33:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.417+0000 7f5e84143640 1 -- 172.21.15.50:0/248017705 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e7c19de00 con 0x7f5e7c19d020 2024-04-23T18:33:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.417+0000 7f5e72ffd640 1 -- 172.21.15.50:0/248017705 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74010670 con 0x7f5e7c19d020 2024-04-23T18:33:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.417+0000 7f5e84143640 1 -- 172.21.15.50:0/248017705 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e7c1a5610 con 0x7f5e7c19d020 2024-04-23T18:33:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.418+0000 7f5e84143640 1 -- 172.21.15.50:0/248017705 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e7c0691c0 con 0x7f5e7c19d020 2024-04-23T18:33:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.419+0000 7f5e72ffd640 1 -- 172.21.15.50:0/248017705 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5e74005020 con 0x7f5e7c19d020 2024-04-23T18:33:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.420+0000 7f5e72ffd640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5e680a3da0 0x7f5e680a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.420+0000 7f5e81edf640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5e680a3da0 0x7f5e680a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.420+0000 7f5e72ffd640 1 -- 172.21.15.50:0/248017705 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8586+0+0 (secure 0 0 0) 0x7f5e740d0d80 con 0x7f5e7c19d020 2024-04-23T18:33:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.420+0000 7f5e81edf640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5e680a3da0 0x7f5e680a6260 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009bd0 tx=0x7f5e78005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:05.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.425+0000 7f5e72ffd640 1 -- 172.21.15.50:0/248017705 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5e7409d380 con 0x7f5e7c19d020 2024-04-23T18:33:05.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:05 smithi177 ceph-mon[32308]: osdmap e111: 12 total, 12 up, 12 in 2024-04-23T18:33:05.558 INFO:tasks.workunit.client.0.smithi050.stdout: 10 51993 30.09 MB/sec warmup 99 sec latency 22212.939 ms 2024-04-23T18:33:05.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:05 smithi188 ceph-mon[32694]: osdmap e111: 12 total, 12 up, 12 in 2024-04-23T18:33:05.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.651+0000 7f5e84143640 1 -- 172.21.15.50:0/248017705 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e7c102bf0 con 0x7f5e7c19d020 2024-04-23T18:33:05.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.652+0000 7f5e72ffd640 1 -- 172.21.15.50:0/248017705 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+11884 (secure 0 0 0) 0x7f5e7409cc80 con 0x7f5e7c19d020 2024-04-23T18:33:05.645 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:33:05.645 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":52,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:32:45.187722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26753":{"gid":26753,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1119979694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1119979694},{"type":"v1","addr":"172.21.15.177:6837","nonce":1119979694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:33:05.645 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 52 2024-04-23T18:33:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.654+0000 7f5e84143640 1 -- 172.21.15.50:0/248017705 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5e680a3da0 msgr2=0x7f5e680a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.654+0000 7f5e84143640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5e680a3da0 0x7f5e680a6260 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009bd0 tx=0x7f5e78005f90 comp rx=0 tx=0).stop 2024-04-23T18:33:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.654+0000 7f5e84143640 1 -- 172.21.15.50:0/248017705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e7c19d020 msgr2=0x7f5e7c19d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.654+0000 7f5e84143640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e7c19d020 0x7f5e7c19d4d0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400eb10 tx=0x7f5e7400c2d0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.654+0000 7f5e84143640 1 -- 172.21.15.50:0/248017705 shutdown_connections 2024-04-23T18:33:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.654+0000 7f5e84143640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f5e680a3da0 0x7f5e680a6260 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.654+0000 7f5e84143640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e7c19d020 0x7f5e7c19d4d0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.654+0000 7f5e84143640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e7c107280 0x7f5e7c198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.654+0000 7f5e84143640 1 --2- 172.21.15.50:0/248017705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c102240 0x7f5e7c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.654+0000 7f5e84143640 1 -- 172.21.15.50:0/248017705 >> 172.21.15.50:0/248017705 conn(0x7f5e7c0790e0 msgr2=0x7f5e7c1002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:05.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.654+0000 7f5e84143640 1 -- 172.21.15.50:0/248017705 shutdown_connections 2024-04-23T18:33:05.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.654+0000 7f5e84143640 1 -- 172.21.15.50:0/248017705 wait complete. 2024-04-23T18:33:05.656 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:05 smithi050 ceph-mon[29187]: osdmap e111: 12 total, 12 up, 12 in 2024-04-23T18:33:05.658 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id 9c186906 --release --await' 2024-04-23T18:33:05.658 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id 9c186906 --release --await 2024-04-23T18:33:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.774+0000 7fc420776640 1 -- 172.21.15.50:0/718886700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc41806a840 msgr2=0x7fc41806ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.774+0000 7fc420776640 1 --2- 172.21.15.50:0/718886700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc41806a840 0x7fc41806ac20 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc414009900 tx=0x7fc41402a410 comp rx=0 tx=0).stop 2024-04-23T18:33:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.774+0000 7fc420776640 1 -- 172.21.15.50:0/718886700 shutdown_connections 2024-04-23T18:33:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.774+0000 7fc420776640 1 --2- 172.21.15.50:0/718886700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc418107270 0x7fc41810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.774+0000 7fc420776640 1 --2- 172.21.15.50:0/718886700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc41806b1f0 0x7fc418106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.774+0000 7fc420776640 1 --2- 172.21.15.50:0/718886700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc41806a840 0x7fc41806ac20 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.774+0000 7fc420776640 1 -- 172.21.15.50:0/718886700 >> 172.21.15.50:0/718886700 conn(0x7fc418077eb0 msgr2=0x7fc4180782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.774+0000 7fc420776640 1 -- 172.21.15.50:0/718886700 shutdown_connections 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.774+0000 7fc420776640 1 -- 172.21.15.50:0/718886700 wait complete. 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.774+0000 7fc420776640 1 Processor -- start 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.774+0000 7fc420776640 1 -- start start 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc420776640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc41806a840 0x7fc418198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc420776640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc418107270 0x7fc418198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc420776640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc41819d010 0x7fc41819d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc420776640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181121d0 con 0x7fc418107270 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc420776640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc418112050 con 0x7fc41806a840 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc420776640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc418112350 con 0x7fc41819d010 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc41dd11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc418107270 0x7fc418198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc41dd11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc418107270 0x7fc418198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50766/0 (socket says 172.21.15.50:50766) 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc41dd11640 1 -- 172.21.15.50:0/1711968782 learned_addr learned my addr 172.21.15.50:0/1711968782 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc41ed13640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc41819d010 0x7fc41819d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc41e512640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc41806a840 0x7fc418198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc41dd11640 1 -- 172.21.15.50:0/1711968782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc41819d010 msgr2=0x7fc41819d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc41dd11640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc41819d010 0x7fc41819d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc41dd11640 1 -- 172.21.15.50:0/1711968782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc41806a840 msgr2=0x7fc418198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc41dd11640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc41806a840 0x7fc418198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc41dd11640 1 -- 172.21.15.50:0/1711968782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc41819db00 con 0x7fc418107270 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc41dd11640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc418107270 0x7fc418198c90 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fc40800d740 tx=0x7fc40800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.775+0000 7fc40f7fe640 1 -- 172.21.15.50:0/1711968782 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc408017070 con 0x7fc418107270 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.776+0000 7fc40f7fe640 1 -- 172.21.15.50:0/1711968782 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc40800de50 con 0x7fc418107270 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.776+0000 7fc420776640 1 -- 172.21.15.50:0/1711968782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc41819ddf0 con 0x7fc418107270 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.776+0000 7fc40f7fe640 1 -- 172.21.15.50:0/1711968782 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc408004ce0 con 0x7fc418107270 2024-04-23T18:33:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.776+0000 7fc420776640 1 -- 172.21.15.50:0/1711968782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4181a55e0 con 0x7fc418107270 2024-04-23T18:33:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.778+0000 7fc40f7fe640 1 -- 172.21.15.50:0/1711968782 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc408004e80 con 0x7fc418107270 2024-04-23T18:33:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.778+0000 7fc42356f740 1 Processor -- start 2024-04-23T18:33:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.778+0000 7fc42356f740 1 -- start start 2024-04-23T18:33:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.778+0000 7fc42356f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b77870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.778+0000 7fc42356f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b81f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.778+0000 7fc42356f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4b84870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.778+0000 7fc42356f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556db4adc6d0 con 0x556db4b77db0 2024-04-23T18:33:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.778+0000 7fc42356f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556db4a791a0 con 0x556db4b77490 2024-04-23T18:33:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.778+0000 7fc42356f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556db4ad8690 con 0x556db4b82480 2024-04-23T18:33:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41e512640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b81f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc40f7fe640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4040a3da0 0x7fc4040a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41e512640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b81f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50782/0 (socket says 172.21.15.50:50782) 2024-04-23T18:33:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41e512640 1 -- 172.21.15.50:0/1746807613 learned_addr learned my addr 172.21.15.50:0/1746807613 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41dd11640 1 --2- 172.21.15.50:0/1746807613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4b84870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41ed13640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4040a3da0 0x7fc4040a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41ed13640 1 --2- 172.21.15.50:0/1746807613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b77870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc40f7fe640 1 -- 172.21.15.50:0/1711968782 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8586+0+0 (secure 0 0 0) 0x7fc4080d00c0 con 0x7fc418107270 2024-04-23T18:33:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41e512640 1 -- 172.21.15.50:0/1746807613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 msgr2=0x556db4b84870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41e512640 1 --2- 172.21.15.50:0/1746807613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4b84870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41e512640 1 -- 172.21.15.50:0/1746807613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 msgr2=0x556db4b77870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41e512640 1 --2- 172.21.15.50:0/1746807613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b77870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41e512640 1 -- 172.21.15.50:0/1746807613 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556db4a835b0 con 0x556db4b77db0 2024-04-23T18:33:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41ed13640 1 --2- 172.21.15.50:0/1746807613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b77870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:33:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41e512640 1 --2- 172.21.15.50:0/1746807613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b81f40 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fc414009900 tx=0x7fc414032e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63904f8fa072303 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc41ed13640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4040a3da0 0x7fc4040a6260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc41000a780 tx=0x7fc410005bd0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc40cff9640 1 -- 172.21.15.50:0/1746807613 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc414040070 con 0x556db4b77db0 2024-04-23T18:33:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc40cff9640 1 -- 172.21.15.50:0/1746807613 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4140316a0 con 0x556db4b77db0 2024-04-23T18:33:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.779+0000 7fc40cff9640 1 -- 172.21.15.50:0/1746807613 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc414035040 con 0x556db4b77db0 2024-04-23T18:33:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.780+0000 7fc42356f740 1 -- 172.21.15.50:0/1746807613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 msgr2=0x556db4b81f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.780+0000 7fc42356f740 1 --2- 172.21.15.50:0/1746807613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b81f40 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fc414009900 tx=0x7fc414032e60 comp rx=0 tx=0).stop 2024-04-23T18:33:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.780+0000 7fc42356f740 1 -- 172.21.15.50:0/1746807613 shutdown_connections 2024-04-23T18:33:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.780+0000 7fc42356f740 1 --2- 172.21.15.50:0/1746807613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4b84870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.780+0000 7fc42356f740 1 --2- 172.21.15.50:0/1746807613 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b81f40 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.780+0000 7fc42356f740 1 --2- 172.21.15.50:0/1746807613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b77870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.780+0000 7fc42356f740 1 -- 172.21.15.50:0/1746807613 >> 172.21.15.50:0/1746807613 conn(0x556db4b737b0 msgr2=0x556db4b73b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.780+0000 7fc42356f740 1 -- 172.21.15.50:0/1746807613 shutdown_connections 2024-04-23T18:33:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.780+0000 7fc42356f740 1 -- 172.21.15.50:0/1746807613 wait complete. 2024-04-23T18:33:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.780+0000 7fc42356f740 1 -- start start 2024-04-23T18:33:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc42356f740 1 Processor -- start 2024-04-23T18:33:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc42356f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b81f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc42356f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b7f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc42356f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4b7f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc42356f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556db4b852b0 con 0x556db4b77db0 2024-04-23T18:33:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc42356f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556db4a76b20 con 0x556db4b77490 2024-04-23T18:33:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc42356f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556db4b85430 con 0x556db4b82480 2024-04-23T18:33:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc42356f740 10 client.0 _ll_register_callbacks cb 0x556db4a76530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc41dd11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b7f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc41dd11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b7f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50796/0 (socket says 172.21.15.50:50796) 2024-04-23T18:33:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc41dd11640 1 -- 172.21.15.50:0/3602508385 learned_addr learned my addr 172.21.15.50:0/3602508385 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc41ed13640 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4b7f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc41e512640 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b81f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc41dd11640 1 -- 172.21.15.50:0/3602508385 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 msgr2=0x556db4b7f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc41dd11640 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4b7f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc41dd11640 1 -- 172.21.15.50:0/3602508385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 msgr2=0x556db4b81f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc41dd11640 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b81f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc41dd11640 1 -- 172.21.15.50:0/3602508385 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556db47ee920 con 0x556db4b77db0 2024-04-23T18:33:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc41e512640 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b81f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:33:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc41dd11640 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b7f200 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fc408098530 tx=0x7fc40809c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44aaf2865385f9c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.781+0000 7fc401ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc401ffb640 1 -- 172.21.15.50:0/3602508385 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc408020070 con 0x556db4b77db0 2024-04-23T18:33:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc401ffb640 1 -- 172.21.15.50:0/3602508385 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8586+0+0 (secure 0 0 0) 0x7fc4080a0080 con 0x556db4b77db0 2024-04-23T18:33:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc42356f740 1 -- 172.21.15.50:0/3602508385 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4140316a0 con 0x556db4b77db0 2024-04-23T18:33:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc401ffb640 1 -- 172.21.15.50:0/3602508385 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fc4080cda50 con 0x556db4b77db0 2024-04-23T18:33:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc42356f740 10 client.17277 fetch_fsmap learned FSMap version 52 2024-04-23T18:33:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc42356f740 1 -- 172.21.15.50:0/3602508385 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x556db48bad30 con 0x556db4b77db0 2024-04-23T18:33:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc401ffb640 1 -- 172.21.15.50:0/3602508385 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4140316a0 con 0x556db4b77db0 2024-04-23T18:33:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc401ffb640 1 -- 172.21.15.50:0/3602508385 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7fc4080d2200 con 0x556db4b77db0 2024-04-23T18:33:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc42356f740 10 client.17277 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:33:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc42356f740 10 client.17277 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:33:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc42356f740 20 client.17277 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc42356f740 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556db4b925f0 0x556db4b949e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc42356f740 4 client.17277 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:33:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc42356f740 1 -- 172.21.15.50:0/3602508385 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556db4a9a500 con 0x556db4b925f0 2024-04-23T18:33:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.782+0000 7fc41e512640 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556db4b925f0 0x556db4b949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.783+0000 7fc41e512640 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556db4b925f0 0x556db4b949e0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38bb0b239cc52040 server_cookie=35e76bf20122c48b in_seq=0 out_seq=0 2024-04-23T18:33:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.783+0000 7fc401ffb640 10 client.17277 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.783+0000 7fc401ffb640 1 -- 172.21.15.50:0/3602508385 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556db4a9a500 con 0x556db4b925f0 2024-04-23T18:33:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.783+0000 7fc401ffb640 10 client.17277 handle_command_reply: tid=0 2024-04-23T18:33:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc42356f740 1 client.17277 shutdown 2024-04-23T18:33:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc42356f740 1 -- 172.21.15.50:0/3602508385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 msgr2=0x556db4b7f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc42356f740 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b7f200 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fc408098530 tx=0x7fc40809c210 comp rx=0 tx=0).stop 2024-04-23T18:33:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc42356f740 1 -- 172.21.15.50:0/3602508385 shutdown_connections 2024-04-23T18:33:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc42356f740 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556db4b925f0 0x556db4b949e0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc42356f740 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4b7f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc42356f740 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b7f200 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc42356f740 1 --2- 172.21.15.50:0/3602508385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b81f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc401ffb640 0 client.17277 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc42356f740 1 -- 172.21.15.50:0/3602508385 >> 172.21.15.50:0/3602508385 conn(0x556db4b737b0 msgr2=0x556db4b795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc42356f740 1 -- 172.21.15.50:0/3602508385 shutdown_connections 2024-04-23T18:33:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc42356f740 1 -- 172.21.15.50:0/3602508385 wait complete. 2024-04-23T18:33:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.784+0000 7fc42356f740 20 client.17277 trim_cache size 0 max 16384 2024-04-23T18:33:05.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.801+0000 7fc42356f740 1 Processor -- start 2024-04-23T18:33:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.801+0000 7fc42356f740 1 -- start start 2024-04-23T18:33:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.801+0000 7fc42356f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b77490 0x556db4b73150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.801+0000 7fc42356f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77db0 0x556db4b9a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.801+0000 7fc42356f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b82480 0x556db4b9a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.801+0000 7fc42356f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181b05f0 con 0x556db4b82480 2024-04-23T18:33:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.801+0000 7fc42356f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181ade00 con 0x556db4b77db0 2024-04-23T18:33:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.801+0000 7fc42356f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181abc70 con 0x556db4b77490 2024-04-23T18:33:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.801+0000 7fc41ed13640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b82480 0x556db4b9a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.801+0000 7fc41ed13640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b82480 0x556db4b9a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50802/0 (socket says 172.21.15.50:50802) 2024-04-23T18:33:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.801+0000 7fc41ed13640 1 -- 172.21.15.50:0/911825580 learned_addr learned my addr 172.21.15.50:0/911825580 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.802+0000 7fc41e512640 1 --2- 172.21.15.50:0/911825580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b77490 0x556db4b73150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.802+0000 7fc41dd11640 1 --2- 172.21.15.50:0/911825580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77db0 0x556db4b9a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.802+0000 7fc41ed13640 1 -- 172.21.15.50:0/911825580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b77490 msgr2=0x556db4b73150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.802+0000 7fc41ed13640 1 --2- 172.21.15.50:0/911825580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b77490 0x556db4b73150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.802+0000 7fc41ed13640 1 -- 172.21.15.50:0/911825580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77db0 msgr2=0x556db4b9a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.802+0000 7fc41ed13640 1 --2- 172.21.15.50:0/911825580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77db0 0x556db4b9a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.802+0000 7fc41ed13640 1 -- 172.21.15.50:0/911825580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556db4b77be0 con 0x556db4b82480 2024-04-23T18:33:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.802+0000 7fc41dd11640 1 --2- 172.21.15.50:0/911825580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77db0 0x556db4b9a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:33:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.802+0000 7fc41ed13640 1 --2- 172.21.15.50:0/911825580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b82480 0x556db4b9a6e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fc410013840 tx=0x7fc410014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3617c5b24d0a37e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.802+0000 7fc403fff640 1 -- 172.21.15.50:0/911825580 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41001d820 con 0x556db4b82480 2024-04-23T18:33:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.802+0000 7fc403fff640 1 -- 172.21.15.50:0/911825580 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc410016070 con 0x556db4b82480 2024-04-23T18:33:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.802+0000 7fc403fff640 1 -- 172.21.15.50:0/911825580 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc410019c50 con 0x556db4b82480 2024-04-23T18:33:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 -- 172.21.15.50:0/911825580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b82480 msgr2=0x556db4b9a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 --2- 172.21.15.50:0/911825580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b82480 0x556db4b9a6e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fc410013840 tx=0x7fc410014070 comp rx=0 tx=0).stop 2024-04-23T18:33:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 -- 172.21.15.50:0/911825580 shutdown_connections 2024-04-23T18:33:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 --2- 172.21.15.50:0/911825580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b82480 0x556db4b9a6e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 --2- 172.21.15.50:0/911825580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77db0 0x556db4b9a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 --2- 172.21.15.50:0/911825580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b77490 0x556db4b73150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 -- 172.21.15.50:0/911825580 >> 172.21.15.50:0/911825580 conn(0x556db4b737b0 msgr2=0x556db4b8d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 -- 172.21.15.50:0/911825580 shutdown_connections 2024-04-23T18:33:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 -- 172.21.15.50:0/911825580 wait complete. 2024-04-23T18:33:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 -- start start 2024-04-23T18:33:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 Processor -- start 2024-04-23T18:33:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b85640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b85b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4bf8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181ac2c0 con 0x556db4b77db0 2024-04-23T18:33:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181ad730 con 0x556db4b77490 2024-04-23T18:33:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181ac990 con 0x556db4b82480 2024-04-23T18:33:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc42356f740 10 client.0 _ll_register_callbacks cb 0x7fc40809ba70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc41dd11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b85b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc41dd11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b85b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50812/0 (socket says 172.21.15.50:50812) 2024-04-23T18:33:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.803+0000 7fc41dd11640 1 -- 172.21.15.50:0/871809048 learned_addr learned my addr 172.21.15.50:0/871809048 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc41ed13640 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4bf8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc41e512640 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b85640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc41dd11640 1 -- 172.21.15.50:0/871809048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 msgr2=0x556db4bf8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc41dd11640 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4bf8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc41dd11640 1 -- 172.21.15.50:0/871809048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 msgr2=0x556db4b85640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc41dd11640 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b85640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc41dd11640 1 -- 172.21.15.50:0/871809048 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556db4bf8f50 con 0x556db4b77db0 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc41ed13640 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4bf8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc41dd11640 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b85b80 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fc408097e20 tx=0x7fc408009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164e5caf0c14b01e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc401ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc401ffb640 1 -- 172.21.15.50:0/871809048 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc408020070 con 0x556db4b77db0 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc401ffb640 1 -- 172.21.15.50:0/871809048 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8586+0+0 (secure 0 0 0) 0x7fc40809b730 con 0x556db4b77db0 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc42356f740 1 -- 172.21.15.50:0/871809048 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc410016070 con 0x556db4b77db0 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc401ffb640 1 -- 172.21.15.50:0/871809048 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fc408098160 con 0x556db4b77db0 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc42356f740 10 client.17283 fetch_fsmap learned FSMap version 52 2024-04-23T18:33:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc42356f740 1 -- 172.21.15.50:0/871809048 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x556db4bfa0d0 con 0x556db4b77db0 2024-04-23T18:33:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc401ffb640 1 -- 172.21.15.50:0/871809048 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc410016070 con 0x556db4b77db0 2024-04-23T18:33:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.804+0000 7fc401ffb640 1 -- 172.21.15.50:0/871809048 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7fc40809b0b0 con 0x556db4b77db0 2024-04-23T18:33:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.805+0000 7fc42356f740 10 client.17283 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:33:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.805+0000 7fc42356f740 10 client.17283 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:33:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.805+0000 7fc42356f740 20 client.17283 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.805+0000 7fc42356f740 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556db4bfa760 0x556db4bfcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.805+0000 7fc42356f740 4 client.17283 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "9c186906", "release": true, "await": true}] 2024-04-23T18:33:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.805+0000 7fc42356f740 1 -- 172.21.15.50:0/871809048 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "9c186906", "release": true, "await": true}) -- 0x556db4aea0f0 con 0x556db4bfa760 2024-04-23T18:33:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.805+0000 7fc41e512640 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556db4bfa760 0x556db4bfcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.805+0000 7fc401ffb640 10 client.17283 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.805+0000 7fc41e512640 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556db4bfa760 0x556db4bfcb80 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc5781de58c3c7e9 server_cookie=a121af935097abf6 in_seq=0 out_seq=0 2024-04-23T18:33:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc401ffb640 1 -- 172.21.15.50:0/871809048 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x556db4aea0f0 con 0x556db4bfa760 2024-04-23T18:33:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc401ffb640 10 client.17283 handle_command_reply: tid=0 2024-04-23T18:33:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc42356f740 1 client.17283 shutdown 2024-04-23T18:33:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc42356f740 1 -- 172.21.15.50:0/871809048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 msgr2=0x556db4b85b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc42356f740 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b85b80 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fc408097e20 tx=0x7fc408009e60 comp rx=0 tx=0).stop 2024-04-23T18:33:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc42356f740 1 -- 172.21.15.50:0/871809048 shutdown_connections 2024-04-23T18:33:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc42356f740 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556db4bfa760 0x556db4bfcb80 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc42356f740 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556db4b82480 0x556db4bf8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc42356f740 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556db4b77db0 0x556db4b85b80 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc401ffb640 0 client.17283 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc42356f740 1 --2- 172.21.15.50:0/871809048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db4b77490 0x556db4b85640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc42356f740 1 -- 172.21.15.50:0/871809048 >> 172.21.15.50:0/871809048 conn(0x556db4b737b0 msgr2=0x556db4b8d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.844+0000 7fc42356f740 1 -- 172.21.15.50:0/871809048 shutdown_connections 2024-04-23T18:33:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.845+0000 7fc42356f740 1 -- 172.21.15.50:0/871809048 wait complete. 2024-04-23T18:33:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.845+0000 7fc42356f740 20 client.17283 trim_cache size 0 max 16384 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 52, 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 48, 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: "9c186906": { 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 48, 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:33:05.839 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 81.5, 2024-04-23T18:33:05.840 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:33:05.840 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:33:05.840 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:33:05.840 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:33:05.840 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:33:05.840 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:33:05.840 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:05.840 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:05.840 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:05.840 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:05.840 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:05.840 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:33:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.845+0000 7fc420776640 1 -- 172.21.15.50:0/1711968782 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4040a3da0 msgr2=0x7fc4040a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.845+0000 7fc420776640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4040a3da0 0x7fc4040a6260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc41000a780 tx=0x7fc410005bd0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.845+0000 7fc420776640 1 -- 172.21.15.50:0/1711968782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc418107270 msgr2=0x7fc418198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.845+0000 7fc420776640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc418107270 0x7fc418198c90 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fc40800d740 tx=0x7fc40800dc00 comp rx=0 tx=0).stop 2024-04-23T18:33:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.845+0000 7fc420776640 1 -- 172.21.15.50:0/1711968782 shutdown_connections 2024-04-23T18:33:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.845+0000 7fc420776640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc4040a3da0 0x7fc4040a6260 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.845+0000 7fc420776640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc41819d010 0x7fc41819d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.845+0000 7fc420776640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc418107270 0x7fc418198c90 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.845+0000 7fc420776640 1 --2- 172.21.15.50:0/1711968782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc41806a840 0x7fc418198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.845+0000 7fc420776640 1 -- 172.21.15.50:0/1711968782 >> 172.21.15.50:0/1711968782 conn(0x7fc418077eb0 msgr2=0x7fc418108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.846+0000 7fc420776640 1 -- 172.21.15.50:0/1711968782 shutdown_connections 2024-04-23T18:33:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.846+0000 7fc420776640 1 -- 172.21.15.50:0/1711968782 wait complete. 2024-04-23T18:33:05.851 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '9c186906', seconds elapsed: 23.2 2024-04-23T18:33:05.852 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.45 2024-04-23T18:33:05.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:33:05.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.979+0000 7fb730c69640 1 -- 172.21.15.50:0/94293552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb72c103b00 msgr2=0x7fb72c10bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.979+0000 7fb730c69640 1 --2- 172.21.15.50:0/94293552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb72c103b00 0x7fb72c10bfe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb714009860 tx=0x7fb71402a460 comp rx=0 tx=0).stop 2024-04-23T18:33:05.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.979+0000 7fb730c69640 1 -- 172.21.15.50:0/94293552 shutdown_connections 2024-04-23T18:33:05.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.979+0000 7fb730c69640 1 --2- 172.21.15.50:0/94293552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb72c10c520 0x7fb72c10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.979+0000 7fb730c69640 1 --2- 172.21.15.50:0/94293552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb72c103b00 0x7fb72c10bfe0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.979+0000 7fb730c69640 1 --2- 172.21.15.50:0/94293552 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb72c1031e0 0x7fb72c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.979+0000 7fb730c69640 1 -- 172.21.15.50:0/94293552 >> 172.21.15.50:0/94293552 conn(0x7fb72c077f50 msgr2=0x7fb72c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:05.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.979+0000 7fb730c69640 1 -- 172.21.15.50:0/94293552 shutdown_connections 2024-04-23T18:33:05.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.979+0000 7fb730c69640 1 -- 172.21.15.50:0/94293552 wait complete. 2024-04-23T18:33:05.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.980+0000 7fb730c69640 1 Processor -- start 2024-04-23T18:33:05.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.980+0000 7fb730c69640 1 -- start start 2024-04-23T18:33:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.980+0000 7fb730c69640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb72c1031e0 0x7fb72c198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.980+0000 7fb730c69640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb72c103b00 0x7fb72c198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.980+0000 7fb730c69640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb72c10c520 0x7fb72c19cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.980+0000 7fb730c69640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb72c1121e0 con 0x7fb72c103b00 2024-04-23T18:33:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.980+0000 7fb730c69640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb72c112060 con 0x7fb72c10c520 2024-04-23T18:33:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.980+0000 7fb730c69640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb72c112360 con 0x7fb72c1031e0 2024-04-23T18:33:05.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.980+0000 7fb723fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb72c103b00 0x7fb72c198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb723fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb72c103b00 0x7fb72c198cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50824/0 (socket says 172.21.15.50:50824) 2024-04-23T18:33:05.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb723fff640 1 -- 172.21.15.50:0/2638534731 learned_addr learned my addr 172.21.15.50:0/2638534731 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:05.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb72ad9d640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb72c10c520 0x7fb72c19cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb72a59c640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb72c1031e0 0x7fb72c198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb723fff640 1 -- 172.21.15.50:0/2638534731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb72c1031e0 msgr2=0x7fb72c198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb723fff640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb72c1031e0 0x7fb72c198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb723fff640 1 -- 172.21.15.50:0/2638534731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb72c10c520 msgr2=0x7fb72c19cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:05.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb723fff640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb72c10c520 0x7fb72c19cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:05.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb723fff640 1 -- 172.21.15.50:0/2638534731 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb72c19d720 con 0x7fb72c103b00 2024-04-23T18:33:05.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb72ad9d640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb72c10c520 0x7fb72c19cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:33:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb723fff640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb72c103b00 0x7fb72c198cc0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb714005940 tx=0x7fb714030fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb7237fe640 1 -- 172.21.15.50:0/2638534731 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb714040070 con 0x7fb72c103b00 2024-04-23T18:33:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb730c69640 1 -- 172.21.15.50:0/2638534731 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb72c19d9b0 con 0x7fb72c103b00 2024-04-23T18:33:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb7237fe640 1 -- 172.21.15.50:0/2638534731 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb714005570 con 0x7fb72c103b00 2024-04-23T18:33:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb7237fe640 1 -- 172.21.15.50:0/2638534731 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7140046c0 con 0x7fb72c103b00 2024-04-23T18:33:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.981+0000 7fb730c69640 1 -- 172.21.15.50:0/2638534731 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb72c1a52a0 con 0x7fb72c103b00 2024-04-23T18:33:05.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.982+0000 7fb730c69640 1 -- 172.21.15.50:0/2638534731 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb72c068d70 con 0x7fb72c103b00 2024-04-23T18:33:05.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.984+0000 7fb7237fe640 1 -- 172.21.15.50:0/2638534731 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb714043070 con 0x7fb72c103b00 2024-04-23T18:33:05.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.984+0000 7fb7237fe640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb7180a3d20 0x7fb7180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:05.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.985+0000 7fb72a59c640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb7180a3d20 0x7fb7180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:05.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.985+0000 7fb7237fe640 1 -- 172.21.15.50:0/2638534731 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8586+0+0 (secure 0 0 0) 0x7fb7140f1720 con 0x7fb72c103b00 2024-04-23T18:33:05.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.985+0000 7fb72a59c640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb7180a3d20 0x7fb7180a61e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb71c009530 tx=0x7fb71c009260 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:05.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:05.988+0000 7fb7237fe640 1 -- 172.21.15.50:0/2638534731 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7140bd760 con 0x7fb72c103b00 2024-04-23T18:33:06.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.338+0000 7fb730c69640 1 -- 172.21.15.50:0/2638534731 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb72c1076d0 con 0x7fb72c103b00 2024-04-23T18:33:06.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.339+0000 7fb7237fe640 1 -- 172.21.15.50:0/2638534731 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+11884 (secure 0 0 0) 0x7fb7140c1a60 con 0x7fb72c103b00 2024-04-23T18:33:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:33:06.332 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":52,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:32:45.187722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26753":{"gid":26753,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1119979694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1119979694},{"type":"v1","addr":"172.21.15.177:6837","nonce":1119979694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:33:06.332 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 52 2024-04-23T18:33:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.342+0000 7fb730c69640 1 -- 172.21.15.50:0/2638534731 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb7180a3d20 msgr2=0x7fb7180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.342+0000 7fb730c69640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb7180a3d20 0x7fb7180a61e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb71c009530 tx=0x7fb71c009260 comp rx=0 tx=0).stop 2024-04-23T18:33:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.342+0000 7fb730c69640 1 -- 172.21.15.50:0/2638534731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb72c103b00 msgr2=0x7fb72c198cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.342+0000 7fb730c69640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb72c103b00 0x7fb72c198cc0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb714005940 tx=0x7fb714030fa0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.342+0000 7fb730c69640 1 -- 172.21.15.50:0/2638534731 shutdown_connections 2024-04-23T18:33:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.342+0000 7fb730c69640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fb7180a3d20 0x7fb7180a61e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.342+0000 7fb730c69640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb72c10c520 0x7fb72c19cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.342+0000 7fb730c69640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb72c103b00 0x7fb72c198cc0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.342+0000 7fb730c69640 1 --2- 172.21.15.50:0/2638534731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb72c1031e0 0x7fb72c198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.342+0000 7fb730c69640 1 -- 172.21.15.50:0/2638534731 >> 172.21.15.50:0/2638534731 conn(0x7fb72c077f50 msgr2=0x7fb72c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:06.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.342+0000 7fb730c69640 1 -- 172.21.15.50:0/2638534731 shutdown_connections 2024-04-23T18:33:06.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.342+0000 7fb730c69640 1 -- 172.21.15.50:0/2638534731 wait complete. 2024-04-23T18:33:06.349 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 81.45 --await-for 42.0' 2024-04-23T18:33:06.349 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 81.45 --await-for 42.0 2024-04-23T18:33:06.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:06 smithi177 ceph-mon[32308]: pgmap v461: 97 pgs: 97 active+clean; 690 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 6.1 MiB/s wr, 259 op/s 2024-04-23T18:33:06.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:06 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/248017705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:06.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.545+0000 7f98bcb20640 1 -- 172.21.15.50:0/903256907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98b81106b0 msgr2=0x7f98b810e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:06.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.545+0000 7f98bcb20640 1 --2- 172.21.15.50:0/903256907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98b81106b0 0x7f98b810e290 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009860 tx=0x7f98a002a360 comp rx=0 tx=0).stop 2024-04-23T18:33:06.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.549+0000 7f98bcb20640 1 -- 172.21.15.50:0/903256907 shutdown_connections 2024-04-23T18:33:06.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.549+0000 7f98bcb20640 1 --2- 172.21.15.50:0/903256907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98b810eeb0 0x7f98b811d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.549+0000 7f98bcb20640 1 --2- 172.21.15.50:0/903256907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98b8110b20 0x7f98b810e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.549+0000 7f98bcb20640 1 --2- 172.21.15.50:0/903256907 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98b81106b0 0x7f98b810e290 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.549+0000 7f98bcb20640 1 -- 172.21.15.50:0/903256907 >> 172.21.15.50:0/903256907 conn(0x7f98b806d070 msgr2=0x7f98b806d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:06.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.550+0000 7f98bcb20640 1 -- 172.21.15.50:0/903256907 shutdown_connections 2024-04-23T18:33:06.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.550+0000 7f98bcb20640 1 -- 172.21.15.50:0/903256907 wait complete. 2024-04-23T18:33:06.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.551+0000 7f98bcb20640 1 Processor -- start 2024-04-23T18:33:06.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.552+0000 7f98bcb20640 1 -- start start 2024-04-23T18:33:06.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.552+0000 7f98bcb20640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98b81106b0 0x7f98b8118de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.552+0000 7f98bcb20640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98b8119320 0x7f98b8111e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.552+0000 7f98bcb20640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98b8112500 0x7f98b8116900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.552+0000 7f98bcb20640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f98b8120fd0 con 0x7f98b8112500 2024-04-23T18:33:06.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.552+0000 7f98bcb20640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f98b8120e50 con 0x7f98b81106b0 2024-04-23T18:33:06.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.552+0000 7f98bcb20640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f98b8121150 con 0x7f98b8119320 2024-04-23T18:33:06.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.553+0000 7f98b659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98b81106b0 0x7f98b8118de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.553+0000 7f98b659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98b81106b0 0x7f98b8118de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:34818/0 (socket says 172.21.15.50:34818) 2024-04-23T18:33:06.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.553+0000 7f98b659c640 1 -- 172.21.15.50:0/3565144245 learned_addr learned my addr 172.21.15.50:0/3565144245 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:06.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.553+0000 7f98b6d9d640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98b8112500 0x7f98b8116900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.553+0000 7f98affff640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98b8119320 0x7f98b8111e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.554+0000 7f98b659c640 1 -- 172.21.15.50:0/3565144245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98b8119320 msgr2=0x7f98b8111e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.554+0000 7f98b659c640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98b8119320 0x7f98b8111e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.554+0000 7f98b659c640 1 -- 172.21.15.50:0/3565144245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98b8112500 msgr2=0x7f98b8116900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.554+0000 7f98b659c640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98b8112500 0x7f98b8116900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.554+0000 7f98b659c640 1 -- 172.21.15.50:0/3565144245 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b8116e40 con 0x7f98b81106b0 2024-04-23T18:33:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.554+0000 7f98b659c640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98b81106b0 0x7f98b8118de0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f98a0035040 tx=0x7f98a0033e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.555+0000 7f98af7fe640 1 -- 172.21.15.50:0/3565144245 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a003f030 con 0x7f98b81106b0 2024-04-23T18:33:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.555+0000 7f98bcb20640 1 -- 172.21.15.50:0/3565144245 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b81170d0 con 0x7f98b81106b0 2024-04-23T18:33:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.555+0000 7f98bcb20640 1 -- 172.21.15.50:0/3565144245 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b81175c0 con 0x7f98b81106b0 2024-04-23T18:33:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.555+0000 7f98af7fe640 1 -- 172.21.15.50:0/3565144245 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a0005040 con 0x7f98b81106b0 2024-04-23T18:33:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.555+0000 7f98af7fe640 1 -- 172.21.15.50:0/3565144245 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a0037040 con 0x7f98b81106b0 2024-04-23T18:33:06.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.559+0000 7f98af7fe640 1 -- 172.21.15.50:0/3565144245 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f98a0042070 con 0x7f98b81106b0 2024-04-23T18:33:06.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.565+0000 7f98af7fe640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f98a40a3f20 0x7f98a40a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.565+0000 7f98af7fe640 1 -- 172.21.15.50:0/3565144245 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8586+0+0 (secure 0 0 0) 0x7f98a00efb60 con 0x7f98b81106b0 2024-04-23T18:33:06.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.565+0000 7f98bf839740 1 Processor -- start 2024-04-23T18:33:06.559 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52363 29.94 MB/sec warmup 100 sec latency 22513.764 ms 2024-04-23T18:33:06.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.567+0000 7f98bf839740 1 -- start start 2024-04-23T18:33:06.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.568+0000 7f98affff640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f98a40a3f20 0x7f98a40a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.568+0000 7f98bf839740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb40a0 0x559c60cb4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.568+0000 7f98bf839740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 0x559c60cbeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.568+0000 7f98bf839740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cbf090 0x559c60cc1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.568+0000 7f98bf839740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559c60c19370 con 0x559c60cbf090 2024-04-23T18:33:06.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.568+0000 7f98bf839740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559c60bb62e0 con 0x559c60cb40a0 2024-04-23T18:33:06.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.568+0000 7f98bf839740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559c60c15330 con 0x559c60cb49c0 2024-04-23T18:33:06.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.568+0000 7f98b6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cbf090 0x559c60cc1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.568+0000 7f98b6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cbf090 0x559c60cc1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50846/0 (socket says 172.21.15.50:50846) 2024-04-23T18:33:06.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.568+0000 7f98b6d9d640 1 -- 172.21.15.50:0/3291714373 learned_addr learned my addr 172.21.15.50:0/3291714373 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:06.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.569+0000 7f98affff640 1 --2- 172.21.15.50:0/3291714373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 0x559c60cbeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.569+0000 7f98b659c640 1 --2- 172.21.15.50:0/3291714373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb40a0 0x559c60cb4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.569+0000 7f98affff640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f98a40a3f20 0x7f98a40a63e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f98b8113620 tx=0x7f98a8011810 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:06.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.569+0000 7f98affff640 1 -- 172.21.15.50:0/3291714373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb40a0 msgr2=0x559c60cb4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:06.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.569+0000 7f98affff640 1 --2- 172.21.15.50:0/3291714373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb40a0 0x559c60cb4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.569+0000 7f98affff640 1 -- 172.21.15.50:0/3291714373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cbf090 msgr2=0x559c60cc1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:06.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.569+0000 7f98affff640 1 --2- 172.21.15.50:0/3291714373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cbf090 0x559c60cc1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.569+0000 7f98affff640 1 -- 172.21.15.50:0/3291714373 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c609f7d80 con 0x559c60cb49c0 2024-04-23T18:33:06.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.570+0000 7f98affff640 1 --2- 172.21.15.50:0/3291714373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 0x559c60cbeb50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f98a8005fd0 tx=0x7f98a801e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b02d0919f8c1219 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:06.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.571+0000 7f98acff9640 1 -- 172.21.15.50:0/3291714373 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a8004320 con 0x559c60cb49c0 2024-04-23T18:33:06.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.571+0000 7f98acff9640 1 -- 172.21.15.50:0/3291714373 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a80044c0 con 0x559c60cb49c0 2024-04-23T18:33:06.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.572+0000 7f98bf839740 1 -- 172.21.15.50:0/3291714373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 msgr2=0x559c60cbeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:06.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.572+0000 7f98bf839740 1 --2- 172.21.15.50:0/3291714373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 0x559c60cbeb50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f98a8005fd0 tx=0x7f98a801e000 comp rx=0 tx=0).stop 2024-04-23T18:33:06.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.574+0000 7f98bf839740 1 -- 172.21.15.50:0/3291714373 shutdown_connections 2024-04-23T18:33:06.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.574+0000 7f98bf839740 1 --2- 172.21.15.50:0/3291714373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cbf090 0x559c60cc1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.574+0000 7f98bf839740 1 --2- 172.21.15.50:0/3291714373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 0x559c60cbeb50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.574+0000 7f98bf839740 1 --2- 172.21.15.50:0/3291714373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb40a0 0x559c60cb4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.574+0000 7f98bf839740 1 -- 172.21.15.50:0/3291714373 >> 172.21.15.50:0/3291714373 conn(0x559c60cb03c0 msgr2=0x559c60cb07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:06.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.577+0000 7f98bf839740 1 -- 172.21.15.50:0/3291714373 shutdown_connections 2024-04-23T18:33:06.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.577+0000 7f98bf839740 1 -- 172.21.15.50:0/3291714373 wait complete. 2024-04-23T18:33:06.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.577+0000 7f98bf839740 1 -- start start 2024-04-23T18:33:06.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.577+0000 7f98bf839740 1 Processor -- start 2024-04-23T18:33:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.578+0000 7f98bf839740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb40a0 0x559c60cbbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.578+0000 7f98bf839740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 0x559c60cbc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.578+0000 7f98bf839740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cbf090 0x559c60cbca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.578+0000 7f98bf839740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559c60cc1e70 con 0x559c60cbf090 2024-04-23T18:33:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.578+0000 7f98bf839740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559c60bb3fb0 con 0x559c60cb40a0 2024-04-23T18:33:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.578+0000 7f98bf839740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559c60cc1ff0 con 0x559c60cb49c0 2024-04-23T18:33:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.578+0000 7f98bf839740 10 client.0 _ll_register_callbacks cb 0x559c60bb39a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:06.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.578+0000 7f98affff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 0x559c60cbc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.579+0000 7f98affff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 0x559c60cbc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:41360/0 (socket says 172.21.15.50:41360) 2024-04-23T18:33:06.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.579+0000 7f98affff640 1 -- 172.21.15.50:0/204976302 learned_addr learned my addr 172.21.15.50:0/204976302 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:06.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.579+0000 7f98b659c640 1 --2- 172.21.15.50:0/204976302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb40a0 0x559c60cbbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.579+0000 7f98affff640 1 -- 172.21.15.50:0/204976302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb40a0 msgr2=0x559c60cbbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:06.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.579+0000 7f98affff640 1 --2- 172.21.15.50:0/204976302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb40a0 0x559c60cbbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.579+0000 7f98affff640 1 -- 172.21.15.50:0/204976302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cbf090 msgr2=0x559c60cbca40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.579+0000 7f98affff640 1 --2- 172.21.15.50:0/204976302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cbf090 0x559c60cbca40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.579+0000 7f98affff640 1 -- 172.21.15.50:0/204976302 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c60a58510 con 0x559c60cb49c0 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.579+0000 7f98affff640 1 --2- 172.21.15.50:0/204976302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 0x559c60cbc500 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009a30 tx=0x7f98a8005680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95b2302f742ca153 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.579+0000 7f989dffb640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.580+0000 7f989dffb640 1 -- 172.21.15.50:0/204976302 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a801f8c0 con 0x559c60cb49c0 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.580+0000 7f98bf839740 1 -- 172.21.15.50:0/204976302 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559c60c26d90 con 0x559c60cb49c0 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.580+0000 7f989dffb640 1 -- 172.21.15.50:0/204976302 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8586+0+0 (secure 0 0 0) 0x7f98a801fa60 con 0x559c60cb49c0 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.580+0000 7f989dffb640 1 -- 172.21.15.50:0/204976302 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f98a8015020 con 0x559c60cb49c0 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.580+0000 7f98bf839740 10 client.26851 fetch_fsmap learned FSMap version 52 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.580+0000 7f98bf839740 1 -- 172.21.15.50:0/204976302 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x559c608eb670 con 0x559c60cb49c0 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.580+0000 7f989dffb640 1 -- 172.21.15.50:0/204976302 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x559c60c26d90 con 0x559c60cb49c0 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.581+0000 7f989dffb640 1 -- 172.21.15.50:0/204976302 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f98a8028410 con 0x559c60cb49c0 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.581+0000 7f98bf839740 10 client.26851 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:33:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.581+0000 7f98bf839740 10 client.26851 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:33:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.581+0000 7f98bf839740 20 client.26851 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.581+0000 7f98bf839740 1 --2- 172.21.15.50:0/204976302 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559c60ccf420 0x559c60cd1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.581+0000 7f98bf839740 4 client.26851 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:33:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.581+0000 7f98bf839740 1 -- 172.21.15.50:0/204976302 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c60c20c00 con 0x559c60ccf420 2024-04-23T18:33:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.581+0000 7f98b659c640 1 --2- 172.21.15.50:0/204976302 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559c60ccf420 0x559c60cd1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.583+0000 7f98b659c640 1 --2- 172.21.15.50:0/204976302 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559c60ccf420 0x559c60cd1810 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5977f178bd3e1a59 server_cookie=9638c66bf9f504da in_seq=0 out_seq=0 2024-04-23T18:33:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.583+0000 7f989dffb640 10 client.26851 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:06.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.585+0000 7f989dffb640 1 -- 172.21.15.50:0/204976302 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559c60c20c00 con 0x559c60ccf420 2024-04-23T18:33:06.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.585+0000 7f989dffb640 10 client.26851 handle_command_reply: tid=0 2024-04-23T18:33:06.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.585+0000 7f98bf839740 1 client.26851 shutdown 2024-04-23T18:33:06.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.585+0000 7f98bf839740 1 -- 172.21.15.50:0/204976302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 msgr2=0x559c60cbc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:06.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.585+0000 7f98bf839740 1 --2- 172.21.15.50:0/204976302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 0x559c60cbc500 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009a30 tx=0x7f98a8005680 comp rx=0 tx=0).stop 2024-04-23T18:33:06.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.586+0000 7f98bf839740 1 -- 172.21.15.50:0/204976302 shutdown_connections 2024-04-23T18:33:06.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.586+0000 7f98bf839740 1 --2- 172.21.15.50:0/204976302 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559c60ccf420 0x559c60cd1810 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.586+0000 7f98bf839740 1 --2- 172.21.15.50:0/204976302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cbf090 0x559c60cbca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.586+0000 7f98bf839740 1 --2- 172.21.15.50:0/204976302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cb49c0 0x559c60cbc500 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.586+0000 7f98bf839740 1 --2- 172.21.15.50:0/204976302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb40a0 0x559c60cbbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.586+0000 7f98bf839740 1 -- 172.21.15.50:0/204976302 >> 172.21.15.50:0/204976302 conn(0x559c60cb03c0 msgr2=0x559c60cb61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:06.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.586+0000 7f989dffb640 0 client.26851 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:06.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:06 smithi188 ceph-mon[32694]: pgmap v461: 97 pgs: 97 active+clean; 690 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 6.1 MiB/s wr, 259 op/s 2024-04-23T18:33:06.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:06 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/248017705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:06.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.588+0000 7f98bf839740 1 -- 172.21.15.50:0/204976302 shutdown_connections 2024-04-23T18:33:06.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.588+0000 7f98bf839740 1 -- 172.21.15.50:0/204976302 wait complete. 2024-04-23T18:33:06.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.588+0000 7f98bf839740 20 client.26851 trim_cache size 0 max 16384 2024-04-23T18:33:06.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.622+0000 7f98bf839740 1 Processor -- start 2024-04-23T18:33:06.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.622+0000 7f98bf839740 1 -- start start 2024-04-23T18:33:06.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.622+0000 7f98bf839740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 0x559c60ccae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.622+0000 7f98bf839740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb49c0 0x559c60cd6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98bf839740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cbf090 0x559c60cd74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98bf839740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f98980111f0 con 0x559c60cb40a0 2024-04-23T18:33:06.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98bf839740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f989800ea00 con 0x559c60cb49c0 2024-04-23T18:33:06.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98bf839740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f989800c870 con 0x559c60cbf090 2024-04-23T18:33:06.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98b659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 0x559c60ccae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98b659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 0x559c60ccae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50848/0 (socket says 172.21.15.50:50848) 2024-04-23T18:33:06.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98b659c640 1 -- 172.21.15.50:0/437755500 learned_addr learned my addr 172.21.15.50:0/437755500 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:06.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98affff640 1 --2- 172.21.15.50:0/437755500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb49c0 0x559c60cd6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98b6d9d640 1 --2- 172.21.15.50:0/437755500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cbf090 0x559c60cd74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98b659c640 1 -- 172.21.15.50:0/437755500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cbf090 msgr2=0x559c60cd74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:06.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98b659c640 1 --2- 172.21.15.50:0/437755500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cbf090 0x559c60cd74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98b659c640 1 -- 172.21.15.50:0/437755500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb49c0 msgr2=0x559c60cd6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:06.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98b659c640 1 --2- 172.21.15.50:0/437755500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb49c0 0x559c60cd6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98b659c640 1 -- 172.21.15.50:0/437755500 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c60cb5560 con 0x559c60cb40a0 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f98b659c640 1 --2- 172.21.15.50:0/437755500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 0x559c60ccae10 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f98a0005b60 tx=0x7f98a0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4298d6b3784ef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f989ffff640 1 -- 172.21.15.50:0/437755500 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a00d4070 con 0x559c60cb40a0 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f989ffff640 1 -- 172.21.15.50:0/437755500 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a00bff20 con 0x559c60cb40a0 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.623+0000 7f989ffff640 1 -- 172.21.15.50:0/437755500 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a00ed490 con 0x559c60cb40a0 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.624+0000 7f98bf839740 1 -- 172.21.15.50:0/437755500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 msgr2=0x559c60ccae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.624+0000 7f98bf839740 1 --2- 172.21.15.50:0/437755500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 0x559c60ccae10 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f98a0005b60 tx=0x7f98a0005f90 comp rx=0 tx=0).stop 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.624+0000 7f98bf839740 1 -- 172.21.15.50:0/437755500 shutdown_connections 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.624+0000 7f98bf839740 1 --2- 172.21.15.50:0/437755500 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cbf090 0x559c60cd74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.624+0000 7f98bf839740 1 --2- 172.21.15.50:0/437755500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb49c0 0x559c60cd6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.624+0000 7f98bf839740 1 --2- 172.21.15.50:0/437755500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 0x559c60ccae10 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.624+0000 7f98bf839740 1 -- 172.21.15.50:0/437755500 >> 172.21.15.50:0/437755500 conn(0x559c60cb03c0 msgr2=0x559c60cbe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98bf839740 1 -- 172.21.15.50:0/437755500 shutdown_connections 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98bf839740 1 -- 172.21.15.50:0/437755500 wait complete. 2024-04-23T18:33:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98bf839740 1 -- start start 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98bf839740 1 Processor -- start 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98bf839740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 0x559c60cb8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98bf839740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb49c0 0x559c60cb9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98bf839740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cbf090 0x559c60cb9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98bf839740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f989800cec0 con 0x559c60cb40a0 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98bf839740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f989800e330 con 0x559c60cb49c0 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98bf839740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f989800d590 con 0x559c60cbf090 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98bf839740 10 client.0 _ll_register_callbacks cb 0x7f98a8004270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98b659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 0x559c60cb8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98b659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 0x559c60cb8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50850/0 (socket says 172.21.15.50:50850) 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98b659c640 1 -- 172.21.15.50:0/1856590986 learned_addr learned my addr 172.21.15.50:0/1856590986 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.625+0000 7f98affff640 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb49c0 0x559c60cb9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f98b659c640 1 -- 172.21.15.50:0/1856590986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cbf090 msgr2=0x559c60cb9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:06.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f98b659c640 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cbf090 0x559c60cb9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f98b659c640 1 -- 172.21.15.50:0/1856590986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb49c0 msgr2=0x559c60cb9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f98b659c640 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb49c0 0x559c60cb9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f98b659c640 1 -- 172.21.15.50:0/1856590986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c60d347f0 con 0x559c60cb40a0 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f98affff640 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb49c0 0x559c60cb9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f98b659c640 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 0x559c60cb8d10 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009830 tx=0x7f98a00bbcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55089f3cfdfb9c74 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f989dffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f989dffb640 1 -- 172.21.15.50:0/1856590986 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a00d4040 con 0x559c60cb40a0 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f989dffb640 1 -- 172.21.15.50:0/1856590986 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8586+0+0 (secure 0 0 0) 0x7f98a00b81e0 con 0x559c60cb40a0 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f98bf839740 1 -- 172.21.15.50:0/1856590986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98a00bff20 con 0x559c60cb40a0 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f989dffb640 1 -- 172.21.15.50:0/1856590986 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f98a00ed490 con 0x559c60cb40a0 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f98bf839740 10 client.17304 fetch_fsmap learned FSMap version 52 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.626+0000 7f98bf839740 1 -- 172.21.15.50:0/1856590986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x559c60d35ab0 con 0x559c60cb40a0 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.627+0000 7f989dffb640 1 -- 172.21.15.50:0/1856590986 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a00bff20 con 0x559c60cb40a0 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.627+0000 7f989dffb640 1 -- 172.21.15.50:0/1856590986 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f98a00d6070 con 0x559c60cb40a0 2024-04-23T18:33:06.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.627+0000 7f98bf839740 10 client.17304 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:33:06.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.627+0000 7f98bf839740 10 client.17304 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:33:06.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.627+0000 7f98bf839740 20 client.17304 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:06.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.627+0000 7f98bf839740 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559c60d36170 0x559c60d38590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:06.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.627+0000 7f98bf839740 4 client.17304 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.45, "await_for": 42.0}] 2024-04-23T18:33:06.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.627+0000 7f98bf839740 1 -- 172.21.15.50:0/1856590986 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.45, "await_for": 42.0}) -- 0x559c60b9cca0 con 0x559c60d36170 2024-04-23T18:33:06.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.627+0000 7f98affff640 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559c60d36170 0x559c60d38590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:06.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.627+0000 7f989dffb640 10 client.17304 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:06.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:06.628+0000 7f98affff640 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559c60d36170 0x559c60d38590 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf3589cacb001ab7 server_cookie=880e7ba03d74f35a in_seq=0 out_seq=0 2024-04-23T18:33:06.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:06 smithi050 ceph-mon[29187]: pgmap v461: 97 pgs: 97 active+clean; 690 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 6.1 MiB/s wr, 259 op/s 2024-04-23T18:33:06.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:06 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/248017705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:07.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.234+0000 7f989dffb640 1 -- 172.21.15.50:0/1856590986 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x559c60b9cca0 con 0x559c60d36170 2024-04-23T18:33:07.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f989dffb640 10 client.17304 handle_command_reply: tid=0 2024-04-23T18:33:07.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f98bf839740 1 client.17304 shutdown 2024-04-23T18:33:07.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f98bf839740 1 -- 172.21.15.50:0/1856590986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 msgr2=0x559c60cb8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:07.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f98bf839740 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 0x559c60cb8d10 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009830 tx=0x7f98a00bbcf0 comp rx=0 tx=0).stop 2024-04-23T18:33:07.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f98bf839740 1 -- 172.21.15.50:0/1856590986 shutdown_connections 2024-04-23T18:33:07.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f98bf839740 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559c60d36170 0x559c60d38590 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f98bf839740 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559c60cbf090 0x559c60cb9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f98bf839740 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c60cb49c0 0x559c60cb9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f989dffb640 0 client.17304 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f98bf839740 1 --2- 172.21.15.50:0/1856590986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559c60cb40a0 0x559c60cb8d10 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f98bf839740 1 -- 172.21.15.50:0/1856590986 >> 172.21.15.50:0/1856590986 conn(0x559c60cb03c0 msgr2=0x559c60caed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f98bf839740 1 -- 172.21.15.50:0/1856590986 shutdown_connections 2024-04-23T18:33:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f98bf839740 1 -- 172.21.15.50:0/1856590986 wait complete. 2024-04-23T18:33:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.235+0000 7f98bf839740 20 client.17304 trim_cache size 0 max 16384 2024-04-23T18:33:07.230 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:33:07.230 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 52, 2024-04-23T18:33:07.230 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:33:07.230 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 50, 2024-04-23T18:33:07.230 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:33:07.230 INFO:teuthology.orchestra.run.smithi050.stdout: "b751727a": { 2024-04-23T18:33:07.230 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 50, 2024-04-23T18:33:07.230 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:33:07.230 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:33:07.230 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 81.5, 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:07.231 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:07.232 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:33:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.236+0000 7f98bcb20640 1 -- 172.21.15.50:0/3565144245 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f98a40a3f20 msgr2=0x7f98a40a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.236+0000 7f98bcb20640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f98a40a3f20 0x7f98a40a63e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f98b8113620 tx=0x7f98a8011810 comp rx=0 tx=0).stop 2024-04-23T18:33:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.236+0000 7f98bcb20640 1 -- 172.21.15.50:0/3565144245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98b81106b0 msgr2=0x7f98b8118de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.236+0000 7f98bcb20640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98b81106b0 0x7f98b8118de0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f98a0035040 tx=0x7f98a0033e00 comp rx=0 tx=0).stop 2024-04-23T18:33:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.236+0000 7f98bcb20640 1 -- 172.21.15.50:0/3565144245 shutdown_connections 2024-04-23T18:33:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.236+0000 7f98bcb20640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f98a40a3f20 0x7f98a40a63e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.236+0000 7f98bcb20640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98b8112500 0x7f98b8116900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.236+0000 7f98bcb20640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f98b8119320 0x7f98b8111e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.236+0000 7f98bcb20640 1 --2- 172.21.15.50:0/3565144245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98b81106b0 0x7f98b8118de0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.236+0000 7f98bcb20640 1 -- 172.21.15.50:0/3565144245 >> 172.21.15.50:0/3565144245 conn(0x7f98b806d070 msgr2=0x7f98b811a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.236+0000 7f98bcb20640 1 -- 172.21.15.50:0/3565144245 shutdown_connections 2024-04-23T18:33:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:07.236+0000 7f98bcb20640 1 -- 172.21.15.50:0/3565144245 wait complete. 2024-04-23T18:33:07.245 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b751727a', quiesce took 1.4 seconds. Will release after: 21.45 2024-04-23T18:33:07.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:07 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2638534731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:07.560 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 29.65 MB/sec warmup 101 sec latency 963.574 ms 2024-04-23T18:33:07.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:07 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2638534731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:07.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:07 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2638534731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:08.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:08 smithi177 ceph-mon[32308]: pgmap v462: 97 pgs: 97 active+clean; 690 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 310 op/s 2024-04-23T18:33:08.560 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 29.36 MB/sec warmup 102 sec latency 1963.727 ms 2024-04-23T18:33:08.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:08 smithi188 ceph-mon[32694]: pgmap v462: 97 pgs: 97 active+clean; 690 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 310 op/s 2024-04-23T18:33:08.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:08 smithi050 ceph-mon[29187]: pgmap v462: 97 pgs: 97 active+clean; 690 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 310 op/s 2024-04-23T18:33:09.560 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 29.07 MB/sec warmup 103 sec latency 2963.835 ms 2024-04-23T18:33:10.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:10 smithi177 ceph-mon[32308]: pgmap v463: 97 pgs: 97 active+clean; 690 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 310 op/s 2024-04-23T18:33:10.560 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 28.79 MB/sec warmup 104 sec latency 3963.897 ms 2024-04-23T18:33:10.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:10 smithi188 ceph-mon[32694]: pgmap v463: 97 pgs: 97 active+clean; 690 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 310 op/s 2024-04-23T18:33:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.586+0000 7f2a88f4a640 1 -- 172.21.15.50:0/2294038712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a841061b0 msgr2=0x7f2a8410dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.586+0000 7f2a88f4a640 1 --2- 172.21.15.50:0/2294038712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a841061b0 0x7f2a8410dd80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009ca0 tx=0x7f2a7c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.586+0000 7f2a88f4a640 1 -- 172.21.15.50:0/2294038712 shutdown_connections 2024-04-23T18:33:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.586+0000 7f2a88f4a640 1 --2- 172.21.15.50:0/2294038712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a841061b0 0x7f2a8410dd80 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.586+0000 7f2a88f4a640 1 --2- 172.21.15.50:0/2294038712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a84101a90 0x7f2a84105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.586+0000 7f2a88f4a640 1 --2- 172.21.15.50:0/2294038712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a841010e0 0x7f2a841014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.586+0000 7f2a88f4a640 1 -- 172.21.15.50:0/2294038712 >> 172.21.15.50:0/2294038712 conn(0x7f2a840771d0 msgr2=0x7f2a840775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.586+0000 7f2a88f4a640 1 -- 172.21.15.50:0/2294038712 shutdown_connections 2024-04-23T18:33:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.586+0000 7f2a88f4a640 1 -- 172.21.15.50:0/2294038712 wait complete. 2024-04-23T18:33:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a88f4a640 1 Processor -- start 2024-04-23T18:33:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a88f4a640 1 -- start start 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a88f4a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a841010e0 0x7f2a84102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a88f4a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a841061b0 0x7f2a841030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a88f4a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a841035f0 0x7f2a841a6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a88f4a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84111100 con 0x7f2a841061b0 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a88f4a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84110f80 con 0x7f2a841010e0 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a88f4a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84111280 con 0x7f2a841035f0 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a81d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a841061b0 0x7f2a841030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a81d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a841061b0 0x7f2a841030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50862/0 (socket says 172.21.15.50:50862) 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a8259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a841010e0 0x7f2a84102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a82d9d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a841035f0 0x7f2a841a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.587+0000 7f2a81d9b640 1 -- 172.21.15.50:0/1807801293 learned_addr learned my addr 172.21.15.50:0/1807801293 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.588+0000 7f2a8259c640 1 -- 172.21.15.50:0/1807801293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a841035f0 msgr2=0x7f2a841a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.588+0000 7f2a8259c640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a841035f0 0x7f2a841a6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.588+0000 7f2a8259c640 1 -- 172.21.15.50:0/1807801293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a841061b0 msgr2=0x7f2a841030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.588+0000 7f2a8259c640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a841061b0 0x7f2a841030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.588+0000 7f2a8259c640 1 -- 172.21.15.50:0/1807801293 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a841a6f00 con 0x7f2a841010e0 2024-04-23T18:33:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.588+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a841061b0 0x7f2a841030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:33:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.588+0000 7f2a8259c640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a841010e0 0x7f2a84102b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2a74005fd0 tx=0x7f2a7400dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.588+0000 7f2a737fe640 1 -- 172.21.15.50:0/1807801293 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74009c30 con 0x7f2a841010e0 2024-04-23T18:33:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.588+0000 7f2a88f4a640 1 -- 172.21.15.50:0/1807801293 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a841a71f0 con 0x7f2a841010e0 2024-04-23T18:33:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.588+0000 7f2a737fe640 1 -- 172.21.15.50:0/1807801293 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a74009dd0 con 0x7f2a841010e0 2024-04-23T18:33:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.588+0000 7f2a737fe640 1 -- 172.21.15.50:0/1807801293 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74003e20 con 0x7f2a841010e0 2024-04-23T18:33:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.588+0000 7f2a88f4a640 1 -- 172.21.15.50:0/1807801293 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a841a7790 con 0x7f2a841010e0 2024-04-23T18:33:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.591+0000 7f2a8bc5c740 1 Processor -- start 2024-04-23T18:33:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.591+0000 7f2a737fe640 1 -- 172.21.15.50:0/1807801293 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2a74004000 con 0x7f2a841010e0 2024-04-23T18:33:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.591+0000 7f2a8bc5c740 1 -- start start 2024-04-23T18:33:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.591+0000 7f2a8bc5c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2c2620 0x564a4b2c2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.591+0000 7f2a8bc5c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2f40 0x564a4b2cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.591+0000 7f2a8bc5c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2cd800 0x564a4b2cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.591+0000 7f2a8bc5c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564a4b227970 con 0x564a4b2cd800 2024-04-23T18:33:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.591+0000 7f2a8bc5c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564a4b1c44a0 con 0x564a4b2c2f40 2024-04-23T18:33:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.591+0000 7f2a8bc5c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564a4b223930 con 0x564a4b2c2620 2024-04-23T18:33:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a81d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2cd800 0x564a4b2cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a8259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2f40 0x564a4b2cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a81d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2cd800 0x564a4b2cfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50866/0 (socket says 172.21.15.50:50866) 2024-04-23T18:33:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a82d9d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2c2620 0x564a4b2c2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a81d9b640 1 -- 172.21.15.50:0/3940006595 learned_addr learned my addr 172.21.15.50:0/3940006595 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a737fe640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2a6c0a3df0 0x7f2a6c0a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a81d9b640 1 -- 172.21.15.50:0/3940006595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2c2620 msgr2=0x564a4b2c2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3940006595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2c2620 0x564a4b2c2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a81d9b640 1 -- 172.21.15.50:0/3940006595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2f40 msgr2=0x564a4b2cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3940006595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2f40 0x564a4b2cd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a81d9b640 1 -- 172.21.15.50:0/3940006595 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a4af3a210 con 0x564a4b2cd800 2024-04-23T18:33:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a82d9d640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2a6c0a3df0 0x7f2a6c0a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a737fe640 1 -- 172.21.15.50:0/1807801293 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8586+0+0 (secure 0 0 0) 0x7f2a740d03c0 con 0x7f2a841010e0 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3940006595 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2cd800 0x564a4b2cfbf0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2a78011510 tx=0x7f2a780119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ce276fc1c66fd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a70ff9640 1 -- 172.21.15.50:0/3940006595 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7801e070 con 0x564a4b2cd800 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.592+0000 7f2a70ff9640 1 -- 172.21.15.50:0/3940006595 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a7800fa70 con 0x564a4b2cd800 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.593+0000 7f2a70ff9640 1 -- 172.21.15.50:0/3940006595 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a78019b10 con 0x564a4b2cd800 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.593+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3940006595 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2cd800 msgr2=0x564a4b2cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.593+0000 7f2a82d9d640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2a6c0a3df0 0x7f2a6c0a62b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009430 tx=0x7f2a7c0095c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.593+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3940006595 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2cd800 0x564a4b2cfbf0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2a78011510 tx=0x7f2a780119d0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.593+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3940006595 shutdown_connections 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.593+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3940006595 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2cd800 0x564a4b2cfbf0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.593+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3940006595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2f40 0x564a4b2cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.593+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3940006595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2c2620 0x564a4b2c2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.593+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3940006595 >> 172.21.15.50:0/3940006595 conn(0x564a4b2beb30 msgr2=0x564a4b2bef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.593+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3940006595 shutdown_connections 2024-04-23T18:33:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.593+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3940006595 wait complete. 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a8bc5c740 1 -- start start 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a8bc5c740 1 Processor -- start 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a8bc5c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2620 0x564a4b2cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a8bc5c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2f40 0x564a4b2ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a8bc5c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2cd800 0x564a4b2caa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a8bc5c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564a4b2d0430 con 0x564a4b2c2620 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a8bc5c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564a4b1c20b0 con 0x564a4b2c2f40 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a8bc5c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564a4b2d05b0 con 0x564a4b2cd800 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a8259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2620 0x564a4b2cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a8bc5c740 10 client.0 _ll_register_callbacks cb 0x564a4b1afd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a8259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2620 0x564a4b2cd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50874/0 (socket says 172.21.15.50:50874) 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a8259c640 1 -- 172.21.15.50:0/3528550790 learned_addr learned my addr 172.21.15.50:0/3528550790 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a82d9d640 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2cd800 0x564a4b2caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.594+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2f40 0x564a4b2ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a8259c640 1 -- 172.21.15.50:0/3528550790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2cd800 msgr2=0x564a4b2caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a8259c640 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2cd800 0x564a4b2caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a8259c640 1 -- 172.21.15.50:0/3528550790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2f40 msgr2=0x564a4b2ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a8259c640 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2f40 0x564a4b2ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a8259c640 1 -- 172.21.15.50:0/3528550790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a4b006620 con 0x564a4b2c2620 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2f40 0x564a4b2ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a8259c640 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2620 0x564a4b2cd1b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009610 tx=0x7f2a74005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3a9c2b92dbab023 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a69ffb640 1 -- 172.21.15.50:0/3528550790 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74020070 con 0x564a4b2c2620 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a69ffb640 1 -- 172.21.15.50:0/3528550790 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8586+0+0 (secure 0 0 0) 0x7f2a740989e0 con 0x564a4b2c2620 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3528550790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a7800fa70 con 0x564a4b2c2620 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a69ffb640 1 -- 172.21.15.50:0/3528550790 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f2a7409bcd0 con 0x564a4b2c2620 2024-04-23T18:33:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a8bc5c740 10 client.17316 fetch_fsmap learned FSMap version 52 2024-04-23T18:33:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3528550790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x564a4b2dc3a0 con 0x564a4b2c2620 2024-04-23T18:33:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a69ffb640 1 -- 172.21.15.50:0/3528550790 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a7800fa70 con 0x564a4b2c2620 2024-04-23T18:33:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.595+0000 7f2a69ffb640 1 -- 172.21.15.50:0/3528550790 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f2a740cdcf0 con 0x564a4b2c2620 2024-04-23T18:33:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.596+0000 7f2a8bc5c740 10 client.17316 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:33:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.596+0000 7f2a8bc5c740 10 client.17316 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:33:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.596+0000 7f2a8bc5c740 20 client.17316 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.596+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564a4b2dd810 0x564a4b2dfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.596+0000 7f2a8bc5c740 4 client.17316 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:33:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.596+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3528550790 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a4b235390 con 0x564a4b2dd810 2024-04-23T18:33:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.596+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564a4b2dd810 0x564a4b2dfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.596+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564a4b2dd810 0x564a4b2dfc40 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dba4d5eed72295e0 server_cookie=280740c0c83be580 in_seq=0 out_seq=0 2024-04-23T18:33:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.596+0000 7f2a69ffb640 10 client.17316 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a69ffb640 1 -- 172.21.15.50:0/3528550790 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a4b235390 con 0x564a4b2dd810 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a69ffb640 10 client.17316 handle_command_reply: tid=0 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a8bc5c740 1 client.17316 shutdown 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3528550790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2620 msgr2=0x564a4b2cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2620 0x564a4b2cd1b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009610 tx=0x7f2a74005dc0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3528550790 shutdown_connections 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564a4b2dd810 0x564a4b2dfc40 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2cd800 0x564a4b2caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2f40 0x564a4b2ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3528550790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2620 0x564a4b2cd1b0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a69ffb640 0 client.17316 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3528550790 >> 172.21.15.50:0/3528550790 conn(0x564a4b2beb30 msgr2=0x564a4b2c4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3528550790 shutdown_connections 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.597+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3528550790 wait complete. 2024-04-23T18:33:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.598+0000 7f2a8bc5c740 20 client.17316 trim_cache size 0 max 16384 2024-04-23T18:33:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.612+0000 7f2a8bc5c740 1 Processor -- start 2024-04-23T18:33:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a8bc5c740 1 -- start start 2024-04-23T18:33:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a8bc5c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2620 0x564a4b2d92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a8bc5c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 0x564a4b2ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a8bc5c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2cd800 0x564a4b2ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a8bc5c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2a841b2870 con 0x564a4b2c2f40 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a8bc5c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a841b0080 con 0x564a4b2c2620 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a8bc5c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2a841adef0 con 0x564a4b2cd800 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a81d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 0x564a4b2ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a81d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 0x564a4b2ca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50884/0 (socket says 172.21.15.50:50884) 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a81d9b640 1 -- 172.21.15.50:0/3366691679 learned_addr learned my addr 172.21.15.50:0/3366691679 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a8259c640 1 --2- 172.21.15.50:0/3366691679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2620 0x564a4b2d92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a82d9d640 1 --2- 172.21.15.50:0/3366691679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2cd800 0x564a4b2ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a81d9b640 1 -- 172.21.15.50:0/3366691679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2cd800 msgr2=0x564a4b2ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3366691679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2cd800 0x564a4b2ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a81d9b640 1 -- 172.21.15.50:0/3366691679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2620 msgr2=0x564a4b2d92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3366691679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2620 0x564a4b2d92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a81d9b640 1 -- 172.21.15.50:0/3366691679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a4b2d9e20 con 0x564a4b2c2f40 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a82d9d640 1 --2- 172.21.15.50:0/3366691679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2cd800 0x564a4b2ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a8259c640 1 --2- 172.21.15.50:0/3366691679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2620 0x564a4b2d92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3366691679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 0x564a4b2ca3a0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f2a78011ed0 tx=0x7f2a78000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac8c090f89340a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.613+0000 7f2a6bfff640 1 -- 172.21.15.50:0/3366691679 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7801e040 con 0x564a4b2c2f40 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.614+0000 7f2a6bfff640 1 -- 172.21.15.50:0/3366691679 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a78029e50 con 0x564a4b2c2f40 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.614+0000 7f2a6bfff640 1 -- 172.21.15.50:0/3366691679 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7800f820 con 0x564a4b2c2f40 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.614+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3366691679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 msgr2=0x564a4b2ca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.614+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3366691679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 0x564a4b2ca3a0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f2a78011ed0 tx=0x7f2a78000bc0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.614+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3366691679 shutdown_connections 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.614+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3366691679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2cd800 0x564a4b2ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.614+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3366691679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 0x564a4b2ca3a0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.614+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3366691679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2c2620 0x564a4b2d92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.614+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3366691679 >> 172.21.15.50:0/3366691679 conn(0x564a4b2beb30 msgr2=0x564a4b2c5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.614+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3366691679 shutdown_connections 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.614+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3366691679 wait complete. 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.614+0000 7f2a8bc5c740 1 -- start start 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a8bc5c740 1 Processor -- start 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a8bc5c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2c2620 0x564a4b2d1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a8bc5c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 0x564a4b2d1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a8bc5c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2cd800 0x564a4b2d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a8bc5c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2a841ae540 con 0x564a4b2c2f40 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a8bc5c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a841af9b0 con 0x564a4b2cd800 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a8bc5c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2a841aec10 con 0x564a4b2c2620 2024-04-23T18:33:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a8bc5c740 10 client.0 _ll_register_callbacks cb 0x7f2a7400bef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a81d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 0x564a4b2d1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a81d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 0x564a4b2d1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50900/0 (socket says 172.21.15.50:50900) 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a81d9b640 1 -- 172.21.15.50:0/3382966250 learned_addr learned my addr 172.21.15.50:0/3382966250 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a82d9d640 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2cd800 0x564a4b2d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a8259c640 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2c2620 0x564a4b2d1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a81d9b640 1 -- 172.21.15.50:0/3382966250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2c2620 msgr2=0x564a4b2d1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2c2620 0x564a4b2d1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a81d9b640 1 -- 172.21.15.50:0/3382966250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2cd800 msgr2=0x564a4b2d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2cd800 0x564a4b2d1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a81d9b640 1 -- 172.21.15.50:0/3382966250 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a4b342e90 con 0x564a4b2c2f40 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a8259c640 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2c2620 0x564a4b2d1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a81d9b640 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 0x564a4b2d1890 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f2a7801c040 tx=0x7f2a780093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51490757de13d1c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.615+0000 7f2a69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a69ffb640 1 -- 172.21.15.50:0/3382966250 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a78036060 con 0x564a4b2c2f40 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a69ffb640 1 -- 172.21.15.50:0/3382966250 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8586+0+0 (secure 0 0 0) 0x7f2a78005410 con 0x564a4b2c2f40 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3382966250 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a78029e50 con 0x564a4b2c2f40 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a69ffb640 1 -- 172.21.15.50:0/3382966250 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f2a7801e070 con 0x564a4b2c2f40 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a8bc5c740 10 client.17322 fetch_fsmap learned FSMap version 52 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3382966250 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x564a4b344110 con 0x564a4b2c2f40 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a69ffb640 1 -- 172.21.15.50:0/3382966250 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a78029e50 con 0x564a4b2c2f40 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a69ffb640 1 -- 172.21.15.50:0/3382966250 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f2a78019790 con 0x564a4b2c2f40 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a8bc5c740 10 client.17322 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a8bc5c740 10 client.17322 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a8bc5c740 20 client.17322 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564a4b344660 0x564a4b346a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a8bc5c740 4 client.17322 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:33:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.616+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3382966250 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564a4b22f200 con 0x564a4b344660 2024-04-23T18:33:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.617+0000 7f2a8259c640 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564a4b344660 0x564a4b346a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.617+0000 7f2a69ffb640 10 client.17322 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.617+0000 7f2a8259c640 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564a4b344660 0x564a4b346a80 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bf193a8326653b6 server_cookie=463947009be51646 in_seq=0 out_seq=0 2024-04-23T18:33:10.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:10 smithi050 ceph-mon[29187]: pgmap v463: 97 pgs: 97 active+clean; 690 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 310 op/s 2024-04-23T18:33:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.713+0000 7f2a69ffb640 1 -- 172.21.15.50:0/3382966250 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564a4b22f200 con 0x564a4b344660 2024-04-23T18:33:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.713+0000 7f2a69ffb640 10 client.17322 handle_command_reply: tid=0 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.713+0000 7f2a8bc5c740 1 client.17322 shutdown 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3382966250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 msgr2=0x564a4b2d1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 0x564a4b2d1890 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f2a7801c040 tx=0x7f2a780093f0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3382966250 shutdown_connections 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564a4b344660 0x564a4b346a80 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a4b2cd800 0x564a4b2d1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564a4b2c2f40 0x564a4b2d1890 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a69ffb640 0 client.17322 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a8bc5c740 1 --2- 172.21.15.50:0/3382966250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564a4b2c2620 0x564a4b2d1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3382966250 >> 172.21.15.50:0/3382966250 conn(0x564a4b2beb30 msgr2=0x564a4b2d82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3382966250 shutdown_connections 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a8bc5c740 1 -- 172.21.15.50:0/3382966250 wait complete. 2024-04-23T18:33:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a8bc5c740 20 client.17322 trim_cache size 0 max 16384 2024-04-23T18:33:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:33:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:33:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:33:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:33:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a88f4a640 1 -- 172.21.15.50:0/1807801293 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2a6c0a3df0 msgr2=0x7f2a6c0a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a88f4a640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2a6c0a3df0 0x7f2a6c0a62b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009430 tx=0x7f2a7c0095c0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a88f4a640 1 -- 172.21.15.50:0/1807801293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a841010e0 msgr2=0x7f2a84102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.714+0000 7f2a88f4a640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a841010e0 0x7f2a84102b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2a74005fd0 tx=0x7f2a7400dde0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.715+0000 7f2a88f4a640 1 -- 172.21.15.50:0/1807801293 shutdown_connections 2024-04-23T18:33:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.715+0000 7f2a88f4a640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f2a6c0a3df0 0x7f2a6c0a62b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.715+0000 7f2a88f4a640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a841035f0 0x7f2a841a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.715+0000 7f2a88f4a640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a841061b0 0x7f2a841030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.715+0000 7f2a88f4a640 1 --2- 172.21.15.50:0/1807801293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a841010e0 0x7f2a84102b70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.715+0000 7f2a88f4a640 1 -- 172.21.15.50:0/1807801293 >> 172.21.15.50:0/1807801293 conn(0x7f2a840771d0 msgr2=0x7f2a8406aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.715+0000 7f2a88f4a640 1 -- 172.21.15.50:0/1807801293 shutdown_connections 2024-04-23T18:33:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:10.715+0000 7f2a88f4a640 1 -- 172.21.15.50:0/1807801293 wait complete. 2024-04-23T18:33:11.560 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 28.52 MB/sec warmup 105 sec latency 4963.958 ms 2024-04-23T18:33:12.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:12 smithi050 ceph-mon[29187]: pgmap v464: 97 pgs: 97 active+clean; 690 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 310 op/s 2024-04-23T18:33:12.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:12 smithi050 ceph-mon[29187]: osdmap e112: 12 total, 12 up, 12 in 2024-04-23T18:33:12.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:12 smithi177 ceph-mon[32308]: pgmap v464: 97 pgs: 97 active+clean; 690 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 310 op/s 2024-04-23T18:33:12.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:12 smithi177 ceph-mon[32308]: osdmap e112: 12 total, 12 up, 12 in 2024-04-23T18:33:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 28.25 MB/sec warmup 106 sec latency 5964.077 ms 2024-04-23T18:33:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:12 smithi188 ceph-mon[32694]: pgmap v464: 97 pgs: 97 active+clean; 690 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 310 op/s 2024-04-23T18:33:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:12 smithi188 ceph-mon[32694]: osdmap e112: 12 total, 12 up, 12 in 2024-04-23T18:33:13.560 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 27.99 MB/sec warmup 107 sec latency 6964.227 ms 2024-04-23T18:33:14.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:14 smithi050 ceph-mon[29187]: pgmap v466: 97 pgs: 97 active+clean; 687 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 8.0 MiB/s wr, 380 op/s 2024-04-23T18:33:14.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:14 smithi177 ceph-mon[32308]: pgmap v466: 97 pgs: 97 active+clean; 687 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 8.0 MiB/s wr, 380 op/s 2024-04-23T18:33:14.560 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 27.73 MB/sec warmup 108 sec latency 7964.353 ms 2024-04-23T18:33:14.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:14 smithi188 ceph-mon[32694]: pgmap v466: 97 pgs: 97 active+clean; 687 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 8.0 MiB/s wr, 380 op/s 2024-04-23T18:33:15.560 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 27.47 MB/sec warmup 109 sec latency 8964.435 ms 2024-04-23T18:33:16.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:16 smithi050 ceph-mon[29187]: pgmap v467: 97 pgs: 97 active+clean; 687 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.8 MiB/s wr, 324 op/s 2024-04-23T18:33:16.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:16 smithi177 ceph-mon[32308]: pgmap v467: 97 pgs: 97 active+clean; 687 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.8 MiB/s wr, 324 op/s 2024-04-23T18:33:16.560 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 27.22 MB/sec warmup 110 sec latency 9964.543 ms 2024-04-23T18:33:16.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:16 smithi188 ceph-mon[32694]: pgmap v467: 97 pgs: 97 active+clean; 687 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.8 MiB/s wr, 324 op/s 2024-04-23T18:33:17.561 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 26.98 MB/sec warmup 111 sec latency 10964.604 ms 2024-04-23T18:33:18.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:18 smithi050 ceph-mon[29187]: pgmap v468: 97 pgs: 97 active+clean; 671 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.1 MiB/s wr, 259 op/s 2024-04-23T18:33:18.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:18 smithi177 ceph-mon[32308]: pgmap v468: 97 pgs: 97 active+clean; 671 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.1 MiB/s wr, 259 op/s 2024-04-23T18:33:18.561 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 26.74 MB/sec warmup 112 sec latency 11964.735 ms 2024-04-23T18:33:18.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:18 smithi188 ceph-mon[32694]: pgmap v468: 97 pgs: 97 active+clean; 671 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.1 MiB/s wr, 259 op/s 2024-04-23T18:33:19.561 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 26.50 MB/sec warmup 113 sec latency 12964.822 ms 2024-04-23T18:33:20.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:20 smithi050 ceph-mon[29187]: pgmap v469: 97 pgs: 97 active+clean; 671 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.1 MiB/s wr, 259 op/s 2024-04-23T18:33:20.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:20 smithi177 ceph-mon[32308]: pgmap v469: 97 pgs: 97 active+clean; 671 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.1 MiB/s wr, 259 op/s 2024-04-23T18:33:20.561 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 26.27 MB/sec warmup 114 sec latency 13964.964 ms 2024-04-23T18:33:20.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:20 smithi188 ceph-mon[32694]: pgmap v469: 97 pgs: 97 active+clean; 671 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.1 MiB/s wr, 259 op/s 2024-04-23T18:33:21.561 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 26.04 MB/sec warmup 115 sec latency 14965.050 ms 2024-04-23T18:33:22.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:22 smithi050 ceph-mon[29187]: pgmap v470: 97 pgs: 97 active+clean; 671 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.1 MiB/s wr, 259 op/s 2024-04-23T18:33:22.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:22 smithi177 ceph-mon[32308]: pgmap v470: 97 pgs: 97 active+clean; 671 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.1 MiB/s wr, 259 op/s 2024-04-23T18:33:22.561 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 25.82 MB/sec warmup 116 sec latency 15965.111 ms 2024-04-23T18:33:22.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:22 smithi188 ceph-mon[32694]: pgmap v470: 97 pgs: 97 active+clean; 671 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.1 MiB/s wr, 259 op/s 2024-04-23T18:33:23.561 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 25.59 MB/sec warmup 117 sec latency 16965.244 ms 2024-04-23T18:33:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.814+0000 7f1b58e2a640 1 -- 172.21.15.50:0/2680635404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b541043a0 msgr2=0x7f1b54104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.814+0000 7f1b58e2a640 1 --2- 172.21.15.50:0/2680635404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b541043a0 0x7f1b54104780 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009900 tx=0x7f1b4402a410 comp rx=0 tx=0).stop 2024-04-23T18:33:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.814+0000 7f1b58e2a640 1 -- 172.21.15.50:0/2680635404 shutdown_connections 2024-04-23T18:33:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.814+0000 7f1b58e2a640 1 --2- 172.21.15.50:0/2680635404 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b54109470 0x7f1b54111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.814+0000 7f1b58e2a640 1 --2- 172.21.15.50:0/2680635404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b54104d50 0x7f1b54108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.814+0000 7f1b58e2a640 1 --2- 172.21.15.50:0/2680635404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b541043a0 0x7f1b54104780 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.814+0000 7f1b58e2a640 1 -- 172.21.15.50:0/2680635404 >> 172.21.15.50:0/2680635404 conn(0x7f1b540ffc40 msgr2=0x7f1b54102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.814+0000 7f1b58e2a640 1 -- 172.21.15.50:0/2680635404 shutdown_connections 2024-04-23T18:33:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.814+0000 7f1b58e2a640 1 -- 172.21.15.50:0/2680635404 wait complete. 2024-04-23T18:33:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.815+0000 7f1b58e2a640 1 Processor -- start 2024-04-23T18:33:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.815+0000 7f1b58e2a640 1 -- start start 2024-04-23T18:33:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.815+0000 7f1b58e2a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b541043a0 0x7f1b541a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.815+0000 7f1b58e2a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b54109470 0x7f1b541a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.815+0000 7f1b58e2a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b5419a6c0 0x7f1b5419ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.815+0000 7f1b58e2a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1b541143e0 con 0x7f1b54109470 2024-04-23T18:33:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.815+0000 7f1b58e2a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1b54114260 con 0x7f1b5419a6c0 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.815+0000 7f1b58e2a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1b54114560 con 0x7f1b541043a0 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b51d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b54109470 0x7f1b541a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b51d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b54109470 0x7f1b541a0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38808/0 (socket says 172.21.15.50:38808) 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b51d9b640 1 -- 172.21.15.50:0/1077458549 learned_addr learned my addr 172.21.15.50:0/1077458549 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b5259c640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b541043a0 0x7f1b541a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b52d9d640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b5419a6c0 0x7f1b5419ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b51d9b640 1 -- 172.21.15.50:0/1077458549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b541043a0 msgr2=0x7f1b541a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b51d9b640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b541043a0 0x7f1b541a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b51d9b640 1 -- 172.21.15.50:0/1077458549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b5419a6c0 msgr2=0x7f1b5419ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b51d9b640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b5419a6c0 0x7f1b5419ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b51d9b640 1 -- 172.21.15.50:0/1077458549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5419b2c0 con 0x7f1b54109470 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b5259c640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b541043a0 0x7f1b541a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b52d9d640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b5419a6c0 0x7f1b5419ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:33:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b51d9b640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b54109470 0x7f1b541a0b40 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800e9c0 tx=0x7f1b4800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:23.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b437fe640 1 -- 172.21.15.50:0/1077458549 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4800cd80 con 0x7f1b54109470 2024-04-23T18:33:23.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b437fe640 1 -- 172.21.15.50:0/1077458549 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b480054d0 con 0x7f1b54109470 2024-04-23T18:33:23.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b58e2a640 1 -- 172.21.15.50:0/1077458549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b541a73a0 con 0x7f1b54109470 2024-04-23T18:33:23.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.816+0000 7f1b437fe640 1 -- 172.21.15.50:0/1077458549 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48010430 con 0x7f1b54109470 2024-04-23T18:33:23.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.817+0000 7f1b58e2a640 1 -- 172.21.15.50:0/1077458549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b541a77a0 con 0x7f1b54109470 2024-04-23T18:33:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.819+0000 7f1b437fe640 1 -- 172.21.15.50:0/1077458549 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1b48004fd0 con 0x7f1b54109470 2024-04-23T18:33:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.819+0000 7f1b5bc1e740 1 Processor -- start 2024-04-23T18:33:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.819+0000 7f1b5bc1e740 1 -- start start 2024-04-23T18:33:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5bc1e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2620 0x562da13a2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5bc1e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2f40 0x562da13ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5bc1e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5bc1e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562da1307970 con 0x562da13a2f40 2024-04-23T18:33:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5bc1e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562da12a44a0 con 0x562da13ad800 2024-04-23T18:33:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5bc1e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562da1303930 con 0x562da13a2620 2024-04-23T18:33:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2f40 0x562da13ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2f40 0x562da13ad2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38810/0 (socket says 172.21.15.50:38810) 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5259c640 1 -- 172.21.15.50:0/1940733370 learned_addr learned my addr 172.21.15.50:0/1940733370 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b52d9d640 1 --2- 172.21.15.50:0/1940733370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2620 0x562da13a2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b437fe640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1b3c0a3dc0 0x7f1b3c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b51d9b640 1 --2- 172.21.15.50:0/1940733370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b52d9d640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1b3c0a3dc0 0x7f1b3c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5259c640 1 -- 172.21.15.50:0/1940733370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2620 msgr2=0x562da13a2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5259c640 1 --2- 172.21.15.50:0/1940733370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2620 0x562da13a2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5259c640 1 -- 172.21.15.50:0/1940733370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 msgr2=0x562da13afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5259c640 1 --2- 172.21.15.50:0/1940733370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13afbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b5259c640 1 -- 172.21.15.50:0/1940733370 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562da101a210 con 0x562da13a2f40 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.820+0000 7f1b437fe640 1 -- 172.21.15.50:0/1077458549 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f1b480d0d20 con 0x7f1b54109470 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.821+0000 7f1b5259c640 1 --2- 172.21.15.50:0/1940733370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2f40 0x562da13ad2c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009900 tx=0x7f1b44032c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413bb0bce6b2d57f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.821+0000 7f1b52d9d640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1b3c0a3dc0 0x7f1b3c0a6280 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c005fd0 tx=0x7f1b4c004ab0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:23.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.821+0000 7f1b40ff9640 1 -- 172.21.15.50:0/1940733370 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4403f070 con 0x562da13a2f40 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.821+0000 7f1b40ff9640 1 -- 172.21.15.50:0/1940733370 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b44032e10 con 0x562da13a2f40 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.821+0000 7f1b40ff9640 1 -- 172.21.15.50:0/1940733370 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44035070 con 0x562da13a2f40 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.821+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/1940733370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2f40 msgr2=0x562da13ad2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.821+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/1940733370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2f40 0x562da13ad2c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009900 tx=0x7f1b44032c20 comp rx=0 tx=0).stop 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/1940733370 shutdown_connections 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/1940733370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/1940733370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2f40 0x562da13ad2c0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/1940733370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2620 0x562da13a2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/1940733370 >> 172.21.15.50:0/1940733370 conn(0x562da139eb30 msgr2=0x562da139ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/1940733370 shutdown_connections 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/1940733370 wait complete. 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 -- start start 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 Processor -- start 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2620 0x562da13ad150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13a2f40 0x562da13aa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13ad800 0x562da13aa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562da13b03d0 con 0x562da13ad800 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562da12a20b0 con 0x562da13a2f40 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562da13b0550 con 0x562da13a2620 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b5bc1e740 10 client.0 _ll_register_callbacks cb 0x562da128fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.822+0000 7f1b52d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13ad800 0x562da13aa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b52d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13ad800 0x562da13aa9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38824/0 (socket says 172.21.15.50:38824) 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b52d9d640 1 -- 172.21.15.50:0/3817043558 learned_addr learned my addr 172.21.15.50:0/3817043558 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b5259c640 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2620 0x562da13ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b51d9b640 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13a2f40 0x562da13aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b52d9d640 1 -- 172.21.15.50:0/3817043558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2620 msgr2=0x562da13ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b52d9d640 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2620 0x562da13ad150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b52d9d640 1 -- 172.21.15.50:0/3817043558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13a2f40 msgr2=0x562da13aa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b52d9d640 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13a2f40 0x562da13aa440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b52d9d640 1 -- 172.21.15.50:0/3817043558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562da10e6620 con 0x562da13ad800 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b52d9d640 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13ad800 0x562da13aa9a0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c012bc0 tx=0x7f1b4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63506c12dcc750f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b39ffb640 1 -- 172.21.15.50:0/3817043558 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c01a070 con 0x562da13ad800 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b39ffb640 1 -- 172.21.15.50:0/3817043558 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f1b4c029400 con 0x562da13ad800 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3817043558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b44032e10 con 0x562da13ad800 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b39ffb640 1 -- 172.21.15.50:0/3817043558 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f1b4c029600 con 0x562da13ad800 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b5bc1e740 10 client.17334 fetch_fsmap learned FSMap version 52 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.823+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3817043558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x562da13bc360 con 0x562da13ad800 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.824+0000 7f1b39ffb640 1 -- 172.21.15.50:0/3817043558 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b44032e10 con 0x562da13ad800 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.824+0000 7f1b39ffb640 1 -- 172.21.15.50:0/3817043558 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f1b4c00f040 con 0x562da13ad800 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.824+0000 7f1b5bc1e740 10 client.17334 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.824+0000 7f1b5bc1e740 10 client.17334 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.824+0000 7f1b5bc1e740 20 client.17334 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.824+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562da13bd7d0 0x562da13bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.824+0000 7f1b5bc1e740 4 client.17334 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:33:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.824+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3817043558 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562da1315390 con 0x562da13bd7d0 2024-04-23T18:33:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.824+0000 7f1b5259c640 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562da13bd7d0 0x562da13bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.825+0000 7f1b39ffb640 10 client.17334 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.825+0000 7f1b5259c640 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562da13bd7d0 0x562da13bfc20 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93e3dac4d8d26f30 server_cookie=aa1c19130cf6b42c in_seq=0 out_seq=0 2024-04-23T18:33:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.825+0000 7f1b39ffb640 1 -- 172.21.15.50:0/3817043558 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562da1315390 con 0x562da13bd7d0 2024-04-23T18:33:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.825+0000 7f1b39ffb640 10 client.17334 handle_command_reply: tid=0 2024-04-23T18:33:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b5bc1e740 1 client.17334 shutdown 2024-04-23T18:33:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3817043558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13ad800 msgr2=0x562da13aa9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13ad800 0x562da13aa9a0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c012bc0 tx=0x7f1b4c004a40 comp rx=0 tx=0).stop 2024-04-23T18:33:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3817043558 shutdown_connections 2024-04-23T18:33:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562da13bd7d0 0x562da13bfc20 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13ad800 0x562da13aa9a0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13a2f40 0x562da13aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b39ffb640 0 client.17334 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/3817043558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2620 0x562da13ad150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3817043558 >> 172.21.15.50:0/3817043558 conn(0x562da139eb30 msgr2=0x562da13a4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3817043558 shutdown_connections 2024-04-23T18:33:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3817043558 wait complete. 2024-04-23T18:33:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.826+0000 7f1b5bc1e740 20 client.17334 trim_cache size 0 max 16384 2024-04-23T18:33:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5bc1e740 1 Processor -- start 2024-04-23T18:33:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5bc1e740 1 -- start start 2024-04-23T18:33:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5bc1e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 0x562da13b9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5bc1e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2f40 0x562da139db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5bc1e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5bc1e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1b541b2720 con 0x562da13a2620 2024-04-23T18:33:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5bc1e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1b541aff30 con 0x562da13ad800 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5bc1e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1b541adda0 con 0x562da13a2f40 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 0x562da13b9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 0x562da13b9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38840/0 (socket says 172.21.15.50:38840) 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5259c640 1 -- 172.21.15.50:0/2047199001 learned_addr learned my addr 172.21.15.50:0/2047199001 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b51d9b640 1 --2- 172.21.15.50:0/2047199001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2f40 0x562da139db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5259c640 1 -- 172.21.15.50:0/2047199001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2f40 msgr2=0x562da139db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5259c640 1 --2- 172.21.15.50:0/2047199001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2f40 0x562da139db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.841+0000 7f1b5259c640 1 -- 172.21.15.50:0/2047199001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 msgr2=0x562da13c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.842+0000 7f1b52d9d640 1 --2- 172.21.15.50:0/2047199001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.842+0000 7f1b5259c640 1 --2- 172.21.15.50:0/2047199001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13c5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.842+0000 7f1b5259c640 1 -- 172.21.15.50:0/2047199001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562da13aff60 con 0x562da13a2620 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.842+0000 7f1b51d9b640 1 --2- 172.21.15.50:0/2047199001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2f40 0x562da139db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.842+0000 7f1b52d9d640 1 --2- 172.21.15.50:0/2047199001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:33:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.842+0000 7f1b5259c640 1 --2- 172.21.15.50:0/2047199001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 0x562da13b9290 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f1b440057c0 tx=0x7f1b440051e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1666cef515d20f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.842+0000 7f1b3bfff640 1 -- 172.21.15.50:0/2047199001 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44051070 con 0x562da13a2620 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.842+0000 7f1b3bfff640 1 -- 172.21.15.50:0/2047199001 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b4404cbc0 con 0x562da13a2620 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.842+0000 7f1b3bfff640 1 -- 172.21.15.50:0/2047199001 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4403f040 con 0x562da13a2620 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/2047199001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 msgr2=0x562da13b9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/2047199001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 0x562da13b9290 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f1b440057c0 tx=0x7f1b440051e0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/2047199001 shutdown_connections 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/2047199001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/2047199001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2f40 0x562da139db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/2047199001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 0x562da13b9290 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/2047199001 >> 172.21.15.50:0/2047199001 conn(0x562da139eb30 msgr2=0x562da139d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/2047199001 shutdown_connections 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/2047199001 wait complete. 2024-04-23T18:33:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 -- start start 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 Processor -- start 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 0x562da13b3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2f40 0x562da13b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1b541ae3f0 con 0x562da13a2620 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1b541af860 con 0x562da13ad800 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1b541aeac0 con 0x562da13a2f40 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5bc1e740 10 client.0 _ll_register_callbacks cb 0x7f1b4c014ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 0x562da13b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 0x562da13b3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38842/0 (socket says 172.21.15.50:38842) 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b5259c640 1 -- 172.21.15.50:0/3036274130 learned_addr learned my addr 172.21.15.50:0/3036274130 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.843+0000 7f1b51d9b640 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2f40 0x562da13b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b52d9d640 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b5259c640 1 -- 172.21.15.50:0/3036274130 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2f40 msgr2=0x562da13b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b5259c640 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2f40 0x562da13b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b5259c640 1 -- 172.21.15.50:0/3036274130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 msgr2=0x562da13b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b5259c640 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13b3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b5259c640 1 -- 172.21.15.50:0/3036274130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562da1422de0 con 0x562da13a2620 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b5259c640 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 0x562da13b3210 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009900 tx=0x7f1b4402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af038f9f6988e351 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b39ffb640 1 -- 172.21.15.50:0/3036274130 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44059060 con 0x562da13a2620 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b39ffb640 1 -- 172.21.15.50:0/3036274130 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f1b44032e20 con 0x562da13a2620 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3036274130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b4404cbc0 con 0x562da13a2620 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b39ffb640 1 -- 172.21.15.50:0/3036274130 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f1b44051070 con 0x562da13a2620 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b5bc1e740 10 client.17340 fetch_fsmap learned FSMap version 52 2024-04-23T18:33:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3036274130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x562da1424040 con 0x562da13a2620 2024-04-23T18:33:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.844+0000 7f1b39ffb640 1 -- 172.21.15.50:0/3036274130 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b4404cbc0 con 0x562da13a2620 2024-04-23T18:33:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.845+0000 7f1b39ffb640 1 -- 172.21.15.50:0/3036274130 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f1b440046c0 con 0x562da13a2620 2024-04-23T18:33:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.845+0000 7f1b5bc1e740 10 client.17340 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:33:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.845+0000 7f1b5bc1e740 10 client.17340 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:33:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.845+0000 7f1b5bc1e740 20 client.17340 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.845+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562da14242e0 0x562da1426700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.845+0000 7f1b5bc1e740 4 client.17340 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:33:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.845+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3036274130 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562da130f200 con 0x562da14242e0 2024-04-23T18:33:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.845+0000 7f1b51d9b640 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562da14242e0 0x562da1426700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.845+0000 7f1b39ffb640 10 client.17340 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.845+0000 7f1b51d9b640 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562da14242e0 0x562da1426700 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56168017660e6c32 server_cookie=22a6e5326b877d4b in_seq=0 out_seq=0 2024-04-23T18:33:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b39ffb640 1 -- 172.21.15.50:0/3036274130 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562da130f200 con 0x562da14242e0 2024-04-23T18:33:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b39ffb640 10 client.17340 handle_command_reply: tid=0 2024-04-23T18:33:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b5bc1e740 1 client.17340 shutdown 2024-04-23T18:33:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3036274130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 msgr2=0x562da13b3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 0x562da13b3210 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009900 tx=0x7f1b4402a410 comp rx=0 tx=0).stop 2024-04-23T18:33:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3036274130 shutdown_connections 2024-04-23T18:33:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562da14242e0 0x562da1426700 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562da13ad800 0x562da13b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562da13a2f40 0x562da13b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b39ffb640 0 client.17340 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b5bc1e740 1 --2- 172.21.15.50:0/3036274130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562da13a2620 0x562da13b3210 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3036274130 >> 172.21.15.50:0/3036274130 conn(0x562da139eb30 msgr2=0x562da139dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3036274130 shutdown_connections 2024-04-23T18:33:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b5bc1e740 1 -- 172.21.15.50:0/3036274130 wait complete. 2024-04-23T18:33:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.919+0000 7f1b5bc1e740 20 client.17340 trim_cache size 0 max 16384 2024-04-23T18:33:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:33:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:33:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:33:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:33:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.920+0000 7f1b58e2a640 1 -- 172.21.15.50:0/1077458549 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1b3c0a3dc0 msgr2=0x7f1b3c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.920+0000 7f1b58e2a640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1b3c0a3dc0 0x7f1b3c0a6280 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c005fd0 tx=0x7f1b4c004ab0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.920+0000 7f1b58e2a640 1 -- 172.21.15.50:0/1077458549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b54109470 msgr2=0x7f1b541a0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.920+0000 7f1b58e2a640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b54109470 0x7f1b541a0b40 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800e9c0 tx=0x7f1b4800ee80 comp rx=0 tx=0).stop 2024-04-23T18:33:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.920+0000 7f1b58e2a640 1 -- 172.21.15.50:0/1077458549 shutdown_connections 2024-04-23T18:33:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.920+0000 7f1b58e2a640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1b3c0a3dc0 0x7f1b3c0a6280 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.920+0000 7f1b58e2a640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b5419a6c0 0x7f1b5419ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.920+0000 7f1b58e2a640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b54109470 0x7f1b541a0b40 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.920+0000 7f1b58e2a640 1 --2- 172.21.15.50:0/1077458549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1b541043a0 0x7f1b541a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.920+0000 7f1b58e2a640 1 -- 172.21.15.50:0/1077458549 >> 172.21.15.50:0/1077458549 conn(0x7f1b540ffc40 msgr2=0x7f1b54101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.920+0000 7f1b58e2a640 1 -- 172.21.15.50:0/1077458549 shutdown_connections 2024-04-23T18:33:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:23.920+0000 7f1b58e2a640 1 -- 172.21.15.50:0/1077458549 wait complete. 2024-04-23T18:33:24.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:24 smithi050 ceph-mon[29187]: pgmap v471: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 4.7 MiB/s wr, 246 op/s 2024-04-23T18:33:24.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:24 smithi177 ceph-mon[32308]: pgmap v471: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 4.7 MiB/s wr, 246 op/s 2024-04-23T18:33:24.561 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 25.38 MB/sec warmup 118 sec latency 17965.351 ms 2024-04-23T18:33:24.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:24 smithi188 ceph-mon[32694]: pgmap v471: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 4.7 MiB/s wr, 246 op/s 2024-04-23T18:33:25.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 25.16 MB/sec warmup 119 sec latency 18965.443 ms 2024-04-23T18:33:26.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:26 smithi050 ceph-mon[29187]: pgmap v472: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 1.7 MiB/s wr, 96 op/s 2024-04-23T18:33:26.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:26 smithi177 ceph-mon[32308]: pgmap v472: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 1.7 MiB/s wr, 96 op/s 2024-04-23T18:33:26.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:26 smithi188 ceph-mon[32694]: pgmap v472: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 1.7 MiB/s wr, 96 op/s 2024-04-23T18:33:27.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 0.00 MB/sec execute 1 sec latency 20965.628 ms 2024-04-23T18:33:28.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:28 smithi050 ceph-mon[29187]: pgmap v473: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 1.7 MiB/s wr, 119 op/s 2024-04-23T18:33:28.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:28 smithi177 ceph-mon[32308]: pgmap v473: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 1.7 MiB/s wr, 119 op/s 2024-04-23T18:33:28.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52379 0.00 MB/sec execute 2 sec latency 21965.749 ms 2024-04-23T18:33:28.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:28 smithi188 ceph-mon[32694]: pgmap v473: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 1.7 MiB/s wr, 119 op/s 2024-04-23T18:33:28.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:33:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.819+0000 7f9c18869640 1 -- 172.21.15.50:0/2104893597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c14103b00 msgr2=0x7f9c1410bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.819+0000 7f9c18869640 1 --2- 172.21.15.50:0/2104893597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c14103b00 0x7f9c1410bfe0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009900 tx=0x7f9c0802a480 comp rx=0 tx=0).stop 2024-04-23T18:33:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.820+0000 7f9c18869640 1 -- 172.21.15.50:0/2104893597 shutdown_connections 2024-04-23T18:33:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.820+0000 7f9c18869640 1 --2- 172.21.15.50:0/2104893597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c1410c5b0 0x7f9c1410e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.820+0000 7f9c18869640 1 --2- 172.21.15.50:0/2104893597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c14103b00 0x7f9c1410bfe0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.820+0000 7f9c18869640 1 --2- 172.21.15.50:0/2104893597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c141031e0 0x7f9c141035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.820+0000 7f9c18869640 1 -- 172.21.15.50:0/2104893597 >> 172.21.15.50:0/2104893597 conn(0x7f9c140771d0 msgr2=0x7f9c140775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.820+0000 7f9c18869640 1 -- 172.21.15.50:0/2104893597 shutdown_connections 2024-04-23T18:33:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.820+0000 7f9c18869640 1 -- 172.21.15.50:0/2104893597 wait complete. 2024-04-23T18:33:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.820+0000 7f9c18869640 1 Processor -- start 2024-04-23T18:33:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c18869640 1 -- start start 2024-04-23T18:33:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c18869640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c141031e0 0x7f9c1419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c18869640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c14103b00 0x7f9c1419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c18869640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c1410c5b0 0x7f9c141985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c18869640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9c141121f0 con 0x7f9c141031e0 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c18869640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9c14112070 con 0x7f9c1410c5b0 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c18869640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9c14112370 con 0x7f9c14103b00 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c1259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c141031e0 0x7f9c1419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c1259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c141031e0 0x7f9c1419e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38870/0 (socket says 172.21.15.50:38870) 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c1259c640 1 -- 172.21.15.50:0/2225301545 learned_addr learned my addr 172.21.15.50:0/2225301545 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c12d9d640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c1410c5b0 0x7f9c141985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c11d9b640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c14103b00 0x7f9c1419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c1259c640 1 -- 172.21.15.50:0/2225301545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c14103b00 msgr2=0x7f9c1419e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c1259c640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c14103b00 0x7f9c1419e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c1259c640 1 -- 172.21.15.50:0/2225301545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c1410c5b0 msgr2=0x7f9c141985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c1259c640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c1410c5b0 0x7f9c141985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c1259c640 1 -- 172.21.15.50:0/2225301545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c14198d70 con 0x7f9c141031e0 2024-04-23T18:33:28.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.821+0000 7f9c12d9d640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c1410c5b0 0x7f9c141985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:33:28.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.822+0000 7f9c1259c640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c141031e0 0x7f9c1419e3f0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400d770 tx=0x7f9c0400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:28.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.822+0000 7f9c037fe640 1 -- 172.21.15.50:0/2225301545 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c04017070 con 0x7f9c141031e0 2024-04-23T18:33:28.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.822+0000 7f9c037fe640 1 -- 172.21.15.50:0/2225301545 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c040054d0 con 0x7f9c141031e0 2024-04-23T18:33:28.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.822+0000 7f9c18869640 1 -- 172.21.15.50:0/2225301545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c14199060 con 0x7f9c141031e0 2024-04-23T18:33:28.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.822+0000 7f9c037fe640 1 -- 172.21.15.50:0/2225301545 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c04004c30 con 0x7f9c141031e0 2024-04-23T18:33:28.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.822+0000 7f9c18869640 1 -- 172.21.15.50:0/2225301545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c141a52b0 con 0x7f9c141031e0 2024-04-23T18:33:28.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.823+0000 7f9c18869640 1 -- 172.21.15.50:0/2225301545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c140691c0 con 0x7f9c141031e0 2024-04-23T18:33:28.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.824+0000 7f9c037fe640 1 -- 172.21.15.50:0/2225301545 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9c04004e10 con 0x7f9c141031e0 2024-04-23T18:33:28.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.825+0000 7f9c037fe640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f9bfc0a3d20 0x7f9bfc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:28.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.825+0000 7f9c11d9b640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f9bfc0a3d20 0x7f9bfc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:28.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.825+0000 7f9c037fe640 1 -- 172.21.15.50:0/2225301545 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f9c040d0340 con 0x7f9c141031e0 2024-04-23T18:33:28.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.825+0000 7f9c11d9b640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f9bfc0a3d20 0x7f9bfc0a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9c08031440 tx=0x7f9c080095f0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:28.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:28.829+0000 7f9c037fe640 1 -- 172.21.15.50:0/2225301545 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c0409c970 con 0x7f9c141031e0 2024-04-23T18:33:29.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.055+0000 7f9c18869640 1 -- 172.21.15.50:0/2225301545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c14199ad0 con 0x7f9c141031e0 2024-04-23T18:33:29.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.056+0000 7f9c037fe640 1 -- 172.21.15.50:0/2225301545 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+11884 (secure 0 0 0) 0x7f9c0409c270 con 0x7f9c141031e0 2024-04-23T18:33:29.048 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:33:29.049 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":52,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:32:45.187722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26753":{"gid":26753,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1119979694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1119979694},{"type":"v1","addr":"172.21.15.177:6837","nonce":1119979694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:33:29.049 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 52 2024-04-23T18:33:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.058+0000 7f9c18869640 1 -- 172.21.15.50:0/2225301545 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f9bfc0a3d20 msgr2=0x7f9bfc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.058+0000 7f9c18869640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f9bfc0a3d20 0x7f9bfc0a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9c08031440 tx=0x7f9c080095f0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.058+0000 7f9c18869640 1 -- 172.21.15.50:0/2225301545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c141031e0 msgr2=0x7f9c1419e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.058+0000 7f9c18869640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c141031e0 0x7f9c1419e3f0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400d770 tx=0x7f9c0400dc30 comp rx=0 tx=0).stop 2024-04-23T18:33:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.058+0000 7f9c18869640 1 -- 172.21.15.50:0/2225301545 shutdown_connections 2024-04-23T18:33:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.058+0000 7f9c18869640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f9bfc0a3d20 0x7f9bfc0a61e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.058+0000 7f9c18869640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c1410c5b0 0x7f9c141985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.058+0000 7f9c18869640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c14103b00 0x7f9c1419e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.058+0000 7f9c18869640 1 --2- 172.21.15.50:0/2225301545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c141031e0 0x7f9c1419e3f0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.058+0000 7f9c18869640 1 -- 172.21.15.50:0/2225301545 >> 172.21.15.50:0/2225301545 conn(0x7f9c140771d0 msgr2=0x7f9c140fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.058+0000 7f9c18869640 1 -- 172.21.15.50:0/2225301545 shutdown_connections 2024-04-23T18:33:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.058+0000 7f9c18869640 1 -- 172.21.15.50:0/2225301545 wait complete. 2024-04-23T18:33:29.061 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id b751727a --release --await' 2024-04-23T18:33:29.062 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id b751727a --release --await 2024-04-23T18:33:29.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.179+0000 7f6b1d987640 1 -- 172.21.15.50:0/3114923405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b18101ac0 msgr2=0x7f6b18105b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:29.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.179+0000 7f6b1d987640 1 --2- 172.21.15.50:0/3114923405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b18101ac0 0x7f6b18105b00 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f6b00009900 tx=0x7f6b0002a410 comp rx=0 tx=0).stop 2024-04-23T18:33:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.180+0000 7f6b1d987640 1 -- 172.21.15.50:0/3114923405 shutdown_connections 2024-04-23T18:33:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.180+0000 7f6b1d987640 1 --2- 172.21.15.50:0/3114923405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b181061e0 0x7f6b1810ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.180+0000 7f6b1d987640 1 --2- 172.21.15.50:0/3114923405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b18101ac0 0x7f6b18105b00 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.180+0000 7f6b1d987640 1 --2- 172.21.15.50:0/3114923405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b18101110 0x7f6b181014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.180+0000 7f6b1d987640 1 -- 172.21.15.50:0/3114923405 >> 172.21.15.50:0/3114923405 conn(0x7f6b18077f80 msgr2=0x7f6b18078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.180+0000 7f6b1d987640 1 -- 172.21.15.50:0/3114923405 shutdown_connections 2024-04-23T18:33:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.180+0000 7f6b1d987640 1 -- 172.21.15.50:0/3114923405 wait complete. 2024-04-23T18:33:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.180+0000 7f6b1d987640 1 Processor -- start 2024-04-23T18:33:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.180+0000 7f6b1d987640 1 -- start start 2024-04-23T18:33:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b1d987640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b18101110 0x7f6b18102ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b1d987640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b181061e0 0x7f6b181030e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b1d987640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b18103620 0x7f6b181a6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b1d987640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6b18111170 con 0x7f6b181061e0 2024-04-23T18:33:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b1d987640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6b18110ff0 con 0x7f6b18101110 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b1d987640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6b181112f0 con 0x7f6b18103620 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b0e7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b181061e0 0x7f6b181030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b0e7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b181061e0 0x7f6b181030e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38886/0 (socket says 172.21.15.50:38886) 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b0e7fc640 1 -- 172.21.15.50:0/358249811 learned_addr learned my addr 172.21.15.50:0/358249811 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b177fe640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b18103620 0x7f6b181a6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b16ffd640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b18101110 0x7f6b18102ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b0e7fc640 1 -- 172.21.15.50:0/358249811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b18103620 msgr2=0x7f6b181a6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b0e7fc640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b18103620 0x7f6b181a6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b0e7fc640 1 -- 172.21.15.50:0/358249811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b18101110 msgr2=0x7f6b18102ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b0e7fc640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b18101110 0x7f6b18102ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b0e7fc640 1 -- 172.21.15.50:0/358249811 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b181a7170 con 0x7f6b181061e0 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b0e7fc640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b181061e0 0x7f6b181030e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f6b00005ab0 tx=0x7f6b00003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b1c985640 1 -- 172.21.15.50:0/358249811 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0003f070 con 0x7f6b181061e0 2024-04-23T18:33:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b1d987640 1 -- 172.21.15.50:0/358249811 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b181a7400 con 0x7f6b181061e0 2024-04-23T18:33:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.181+0000 7f6b1c985640 1 -- 172.21.15.50:0/358249811 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b00031b50 con 0x7f6b181061e0 2024-04-23T18:33:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.182+0000 7f6b1c985640 1 -- 172.21.15.50:0/358249811 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b00035030 con 0x7f6b181061e0 2024-04-23T18:33:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.182+0000 7f6b1d987640 1 -- 172.21.15.50:0/358249811 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b181a78f0 con 0x7f6b181061e0 2024-04-23T18:33:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b1c985640 1 -- 172.21.15.50:0/358249811 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6b00042070 con 0x7f6b181061e0 2024-04-23T18:33:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b2077b740 1 Processor -- start 2024-04-23T18:33:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b2077b740 1 -- start start 2024-04-23T18:33:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b2077b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931490 0x5635b1931870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b2077b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 0x5635b193bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b2077b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b193e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b2077b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5635b18966d0 con 0x5635b1931db0 2024-04-23T18:33:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b2077b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5635b18331a0 con 0x5635b1931490 2024-04-23T18:33:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b2077b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5635b1892690 con 0x5635b193c480 2024-04-23T18:33:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b16ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 0x5635b193bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b16ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 0x5635b193bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38888/0 (socket says 172.21.15.50:38888) 2024-04-23T18:33:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b16ffd640 1 -- 172.21.15.50:0/456249950 learned_addr learned my addr 172.21.15.50:0/456249950 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b1c985640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6b040a3d50 0x7f6b040a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b0e7fc640 1 --2- 172.21.15.50:0/456249950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b193e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.184+0000 7f6b177fe640 1 --2- 172.21.15.50:0/456249950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931490 0x5635b1931870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.185+0000 7f6b177fe640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6b040a3d50 0x7f6b040a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.185+0000 7f6b16ffd640 1 -- 172.21.15.50:0/456249950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 msgr2=0x5635b193e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.185+0000 7f6b16ffd640 1 --2- 172.21.15.50:0/456249950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b193e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.185+0000 7f6b16ffd640 1 -- 172.21.15.50:0/456249950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931490 msgr2=0x5635b1931870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.185+0000 7f6b16ffd640 1 --2- 172.21.15.50:0/456249950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931490 0x5635b1931870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.185+0000 7f6b1c985640 1 -- 172.21.15.50:0/358249811 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f6b000f0d40 con 0x7f6b181061e0 2024-04-23T18:33:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.185+0000 7f6b16ffd640 1 -- 172.21.15.50:0/456249950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635b183d5b0 con 0x5635b1931db0 2024-04-23T18:33:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.185+0000 7f6b16ffd640 1 --2- 172.21.15.50:0/456249950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 0x5635b193bf40 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f6b08009d20 tx=0x7f6b08011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93aa332cc79b50e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.185+0000 7f6b177fe640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6b040a3d50 0x7f6b040a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6b10005fd0 tx=0x7f6b10009500 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.185+0000 7f6b0dffb640 1 -- 172.21.15.50:0/456249950 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0801e070 con 0x5635b1931db0 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.185+0000 7f6b0dffb640 1 -- 172.21.15.50:0/456249950 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b08019ae0 con 0x5635b1931db0 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.185+0000 7f6b0dffb640 1 -- 172.21.15.50:0/456249950 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0800eac0 con 0x5635b1931db0 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.186+0000 7f6b2077b740 1 -- 172.21.15.50:0/456249950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 msgr2=0x5635b193bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.186+0000 7f6b2077b740 1 --2- 172.21.15.50:0/456249950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 0x5635b193bf40 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f6b08009d20 tx=0x7f6b08011810 comp rx=0 tx=0).stop 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.186+0000 7f6b2077b740 1 -- 172.21.15.50:0/456249950 shutdown_connections 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.186+0000 7f6b2077b740 1 --2- 172.21.15.50:0/456249950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b193e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.186+0000 7f6b2077b740 1 --2- 172.21.15.50:0/456249950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 0x5635b193bf40 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.186+0000 7f6b2077b740 1 --2- 172.21.15.50:0/456249950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931490 0x5635b1931870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.186+0000 7f6b2077b740 1 -- 172.21.15.50:0/456249950 >> 172.21.15.50:0/456249950 conn(0x5635b192d7b0 msgr2=0x5635b192db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.186+0000 7f6b2077b740 1 -- 172.21.15.50:0/456249950 shutdown_connections 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.186+0000 7f6b2077b740 1 -- 172.21.15.50:0/456249950 wait complete. 2024-04-23T18:33:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.186+0000 7f6b2077b740 1 -- start start 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.186+0000 7f6b2077b740 1 Processor -- start 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b2077b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931490 0x5635b193bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b2077b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931db0 0x5635b19391d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b2077b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b19398f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b2077b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5635b193f310 con 0x5635b1931490 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b2077b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5635b1830b20 con 0x5635b1931db0 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b2077b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5635b193f490 con 0x5635b193c480 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b2077b740 10 client.0 _ll_register_callbacks cb 0x5635b1830530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b16ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931490 0x5635b193bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b16ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931490 0x5635b193bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38902/0 (socket says 172.21.15.50:38902) 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b16ffd640 1 -- 172.21.15.50:0/1440467406 learned_addr learned my addr 172.21.15.50:0/1440467406 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b0e7fc640 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931db0 0x5635b19391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b177fe640 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b19398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b16ffd640 1 -- 172.21.15.50:0/1440467406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 msgr2=0x5635b19398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b16ffd640 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b19398f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b16ffd640 1 -- 172.21.15.50:0/1440467406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931db0 msgr2=0x5635b19391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b16ffd640 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931db0 0x5635b19391d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b16ffd640 1 -- 172.21.15.50:0/1440467406 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635b15a8920 con 0x5635b1931490 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b16ffd640 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931490 0x5635b193bee0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6b0801c040 tx=0x7f6b08004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ada97f15ef453d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6af6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6af6ffd640 1 -- 172.21.15.50:0/1440467406 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0801e050 con 0x5635b1931490 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6af6ffd640 1 -- 172.21.15.50:0/1440467406 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f6b08019630 con 0x5635b1931490 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.187+0000 7f6b2077b740 1 -- 172.21.15.50:0/1440467406 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b08019ae0 con 0x5635b1931490 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.188+0000 7f6af6ffd640 1 -- 172.21.15.50:0/1440467406 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f6b08032050 con 0x5635b1931490 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.188+0000 7f6b2077b740 10 client.17358 fetch_fsmap learned FSMap version 52 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.188+0000 7f6b2077b740 1 -- 172.21.15.50:0/1440467406 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x5635b1674d30 con 0x5635b1931490 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.188+0000 7f6af6ffd640 1 -- 172.21.15.50:0/1440467406 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b08019ae0 con 0x5635b1931490 2024-04-23T18:33:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.188+0000 7f6af6ffd640 1 -- 172.21.15.50:0/1440467406 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f6b0802d030 con 0x5635b1931490 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.188+0000 7f6b2077b740 10 client.17358 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.188+0000 7f6b2077b740 10 client.17358 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.188+0000 7f6b2077b740 20 client.17358 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.188+0000 7f6b2077b740 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5635b194c610 0x5635b194ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.188+0000 7f6b2077b740 4 client.17358 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.188+0000 7f6b2077b740 1 -- 172.21.15.50:0/1440467406 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635b1854500 con 0x5635b194c610 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.188+0000 7f6b0e7fc640 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5635b194c610 0x5635b194ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.189+0000 7f6b0e7fc640 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5635b194c610 0x5635b194ea00 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c906b53c0ab17609 server_cookie=e3d0f833c96dd534 in_seq=0 out_seq=0 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.189+0000 7f6af6ffd640 10 client.17358 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.189+0000 7f6af6ffd640 1 -- 172.21.15.50:0/1440467406 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5635b1854500 con 0x5635b194c610 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.189+0000 7f6af6ffd640 10 client.17358 handle_command_reply: tid=0 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.189+0000 7f6b2077b740 1 client.17358 shutdown 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.190+0000 7f6b2077b740 1 -- 172.21.15.50:0/1440467406 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931490 msgr2=0x5635b193bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.190+0000 7f6b2077b740 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931490 0x5635b193bee0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6b0801c040 tx=0x7f6b08004270 comp rx=0 tx=0).stop 2024-04-23T18:33:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.190+0000 7f6b2077b740 1 -- 172.21.15.50:0/1440467406 shutdown_connections 2024-04-23T18:33:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.190+0000 7f6b2077b740 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5635b194c610 0x5635b194ea00 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.190+0000 7f6b2077b740 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b19398f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.190+0000 7f6b2077b740 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931db0 0x5635b19391d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.190+0000 7f6af6ffd640 0 client.17358 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.190+0000 7f6b2077b740 1 --2- 172.21.15.50:0/1440467406 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931490 0x5635b193bee0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.190+0000 7f6b2077b740 1 -- 172.21.15.50:0/1440467406 >> 172.21.15.50:0/1440467406 conn(0x5635b192d7b0 msgr2=0x5635b192c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.190+0000 7f6b2077b740 1 -- 172.21.15.50:0/1440467406 shutdown_connections 2024-04-23T18:33:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.190+0000 7f6b2077b740 1 -- 172.21.15.50:0/1440467406 wait complete. 2024-04-23T18:33:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.190+0000 7f6b2077b740 20 client.17358 trim_cache size 0 max 16384 2024-04-23T18:33:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.207+0000 7f6b2077b740 1 Processor -- start 2024-04-23T18:33:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.207+0000 7f6b2077b740 1 -- start start 2024-04-23T18:33:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.207+0000 7f6b2077b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931490 0x5635b1938330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.207+0000 7f6b2077b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 0x5635b193ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.207+0000 7f6b2077b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b192c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.207+0000 7f6b2077b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6b181b2810 con 0x5635b1931db0 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.207+0000 7f6b2077b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6b181b0020 con 0x5635b1931490 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.207+0000 7f6b2077b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6b181adeb0 con 0x5635b193c480 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.207+0000 7f6b0e7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 0x5635b193ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.207+0000 7f6b0e7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 0x5635b193ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38914/0 (socket says 172.21.15.50:38914) 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.207+0000 7f6b0e7fc640 1 -- 172.21.15.50:0/3959099205 learned_addr learned my addr 172.21.15.50:0/3959099205 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b177fe640 1 --2- 172.21.15.50:0/3959099205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b192c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b0e7fc640 1 -- 172.21.15.50:0/3959099205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 msgr2=0x5635b192c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b0e7fc640 1 --2- 172.21.15.50:0/3959099205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b192c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b0e7fc640 1 -- 172.21.15.50:0/3959099205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931490 msgr2=0x5635b1938330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b0e7fc640 1 --2- 172.21.15.50:0/3959099205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931490 0x5635b1938330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b0e7fc640 1 -- 172.21.15.50:0/3959099205 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635b1939c60 con 0x5635b1931db0 2024-04-23T18:33:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b177fe640 1 --2- 172.21.15.50:0/3959099205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b192c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b0e7fc640 1 --2- 172.21.15.50:0/3959099205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 0x5635b193ba10 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6b00031440 tx=0x7f6b00031f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21305833554e81 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b0d7fa640 1 -- 172.21.15.50:0/3959099205 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b000d5070 con 0x5635b1931db0 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b0d7fa640 1 -- 172.21.15.50:0/3959099205 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b000c5940 con 0x5635b1931db0 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b0d7fa640 1 -- 172.21.15.50:0/3959099205 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b00049040 con 0x5635b1931db0 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b2077b740 1 -- 172.21.15.50:0/3959099205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 msgr2=0x5635b193ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b2077b740 1 --2- 172.21.15.50:0/3959099205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 0x5635b193ba10 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6b00031440 tx=0x7f6b00031f50 comp rx=0 tx=0).stop 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b2077b740 1 -- 172.21.15.50:0/3959099205 shutdown_connections 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b2077b740 1 --2- 172.21.15.50:0/3959099205 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b193c480 0x5635b192c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b2077b740 1 --2- 172.21.15.50:0/3959099205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b1931db0 0x5635b193ba10 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b2077b740 1 --2- 172.21.15.50:0/3959099205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931490 0x5635b1938330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.208+0000 7f6b2077b740 1 -- 172.21.15.50:0/3959099205 >> 172.21.15.50:0/3959099205 conn(0x5635b192d7b0 msgr2=0x5635b192c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b2077b740 1 -- 172.21.15.50:0/3959099205 shutdown_connections 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b2077b740 1 -- 172.21.15.50:0/3959099205 wait complete. 2024-04-23T18:33:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b2077b740 1 -- start start 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b2077b740 1 Processor -- start 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b2077b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b1931490 0x5635b193d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b2077b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931db0 0x5635b193dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b2077b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b193c480 0x5635b193e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b2077b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6b181ae500 con 0x5635b193c480 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b2077b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6b181af950 con 0x5635b1931db0 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b2077b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6b181aebb0 con 0x5635b1931490 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b2077b740 10 client.0 _ll_register_callbacks cb 0x7f6b000bcbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b193c480 0x5635b193e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b193c480 0x5635b193e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38920/0 (socket says 172.21.15.50:38920) 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.209+0000 7f6b177fe640 1 -- 172.21.15.50:0/2328873782 learned_addr learned my addr 172.21.15.50:0/2328873782 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6b16ffd640 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b1931490 0x5635b193d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6b0e7fc640 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931db0 0x5635b193dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6b177fe640 1 -- 172.21.15.50:0/2328873782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b1931490 msgr2=0x5635b193d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6b177fe640 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b1931490 0x5635b193d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6b177fe640 1 -- 172.21.15.50:0/2328873782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931db0 msgr2=0x5635b193dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6b177fe640 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931db0 0x5635b193dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6b177fe640 1 -- 172.21.15.50:0/2328873782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635b1931be0 con 0x5635b193c480 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6b177fe640 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b193c480 0x5635b193e0f0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000fb90 tx=0x7f6b10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0271392ea53a9f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6af6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6af6ffd640 1 -- 172.21.15.50:0/2328873782 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1000fc80 con 0x5635b193c480 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6af6ffd640 1 -- 172.21.15.50:0/2328873782 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f6b1001a070 con 0x5635b193c480 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6b2077b740 1 -- 172.21.15.50:0/2328873782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b000c5940 con 0x5635b193c480 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6af6ffd640 1 -- 172.21.15.50:0/2328873782 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f6b1000fe20 con 0x5635b193c480 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6b2077b740 10 client.17364 fetch_fsmap learned FSMap version 52 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6b2077b740 1 -- 172.21.15.50:0/2328873782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x5635b19b2d70 con 0x5635b193c480 2024-04-23T18:33:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.210+0000 7f6af6ffd640 1 -- 172.21.15.50:0/2328873782 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b000c5940 con 0x5635b193c480 2024-04-23T18:33:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.211+0000 7f6af6ffd640 1 -- 172.21.15.50:0/2328873782 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f6b1002c930 con 0x5635b193c480 2024-04-23T18:33:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.211+0000 7f6b2077b740 10 client.17364 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:33:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.211+0000 7f6b2077b740 10 client.17364 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:33:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.211+0000 7f6b2077b740 20 client.17364 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.211+0000 7f6b2077b740 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5635b19b3580 0x5635b19b59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.211+0000 7f6b2077b740 4 client.17364 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b751727a", "release": true, "await": true}] 2024-04-23T18:33:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.211+0000 7f6b2077b740 1 -- 172.21.15.50:0/2328873782 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b751727a", "release": true, "await": true}) -- 0x5635b18a40f0 con 0x5635b19b3580 2024-04-23T18:33:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.211+0000 7f6b16ffd640 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5635b19b3580 0x5635b19b59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.211+0000 7f6b16ffd640 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5635b19b3580 0x5635b19b59a0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2879327fb80c5e1 server_cookie=dbd5be19ec13b380 in_seq=0 out_seq=0 2024-04-23T18:33:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.211+0000 7f6af6ffd640 10 client.17364 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.244+0000 7f6af6ffd640 1 -- 172.21.15.50:0/2328873782 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5635b18a40f0 con 0x5635b19b3580 2024-04-23T18:33:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.244+0000 7f6af6ffd640 10 client.17364 handle_command_reply: tid=0 2024-04-23T18:33:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.244+0000 7f6b2077b740 1 client.17364 shutdown 2024-04-23T18:33:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b2077b740 1 -- 172.21.15.50:0/2328873782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b193c480 msgr2=0x5635b193e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b2077b740 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b193c480 0x5635b193e0f0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000fb90 tx=0x7f6b10004a40 comp rx=0 tx=0).stop 2024-04-23T18:33:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b2077b740 1 -- 172.21.15.50:0/2328873782 shutdown_connections 2024-04-23T18:33:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b2077b740 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5635b19b3580 0x5635b19b59a0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b2077b740 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5635b193c480 0x5635b193e0f0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6af6ffd640 0 client.17364 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b2077b740 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635b1931db0 0x5635b193dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b2077b740 1 --2- 172.21.15.50:0/2328873782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5635b1931490 0x5635b193d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b2077b740 1 -- 172.21.15.50:0/2328873782 >> 172.21.15.50:0/2328873782 conn(0x5635b192d7b0 msgr2=0x5635b19329c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:29.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b2077b740 1 -- 172.21.15.50:0/2328873782 shutdown_connections 2024-04-23T18:33:29.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b2077b740 1 -- 172.21.15.50:0/2328873782 wait complete. 2024-04-23T18:33:29.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b2077b740 20 client.17364 trim_cache size 0 max 16384 2024-04-23T18:33:29.239 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:33:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 52, 2024-04-23T18:33:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:33:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 52, 2024-04-23T18:33:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:33:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "b751727a": { 2024-04-23T18:33:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 52, 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 81.5, 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:29.241 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:29.241 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:29.241 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:33:29.241 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:33:29.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b1d987640 1 -- 172.21.15.50:0/358249811 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6b040a3d50 msgr2=0x7f6b040a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b1d987640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6b040a3d50 0x7f6b040a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6b10005fd0 tx=0x7f6b10009500 comp rx=0 tx=0).stop 2024-04-23T18:33:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b1d987640 1 -- 172.21.15.50:0/358249811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b181061e0 msgr2=0x7f6b181030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.245+0000 7f6b1d987640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b181061e0 0x7f6b181030e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f6b00005ab0 tx=0x7f6b00003b90 comp rx=0 tx=0).stop 2024-04-23T18:33:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.246+0000 7f6b1d987640 1 -- 172.21.15.50:0/358249811 shutdown_connections 2024-04-23T18:33:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.246+0000 7f6b1d987640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6b040a3d50 0x7f6b040a6210 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.246+0000 7f6b1d987640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b18103620 0x7f6b181a6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.246+0000 7f6b1d987640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b181061e0 0x7f6b181030e0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.246+0000 7f6b1d987640 1 --2- 172.21.15.50:0/358249811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b18101110 0x7f6b18102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.246+0000 7f6b1d987640 1 -- 172.21.15.50:0/358249811 >> 172.21.15.50:0/358249811 conn(0x7f6b18077f80 msgr2=0x7f6b1806a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.246+0000 7f6b1d987640 1 -- 172.21.15.50:0/358249811 shutdown_connections 2024-04-23T18:33:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:33:29.246+0000 7f6b1d987640 1 -- 172.21.15.50:0/358249811 wait complete. 2024-04-23T18:33:29.249 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b751727a', seconds elapsed: 23.4 2024-04-23T18:33:29.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:29 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2225301545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:29.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:29 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2225301545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:29.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 52541 1.23 MB/sec execute 3 sec latency 22654.167 ms 2024-04-23T18:33:29.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:29 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2225301545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:33:30.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:30 smithi050 ceph-mon[29187]: pgmap v474: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 361 KiB/s wr, 45 op/s 2024-04-23T18:33:30.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:30 smithi177 ceph-mon[32308]: pgmap v474: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 361 KiB/s wr, 45 op/s 2024-04-23T18:33:30.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 53335 13.03 MB/sec execute 4 sec latency 46.419 ms 2024-04-23T18:33:30.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:30 smithi188 ceph-mon[32694]: pgmap v474: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 361 KiB/s wr, 45 op/s 2024-04-23T18:33:31.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 53976 17.65 MB/sec execute 5 sec latency 91.020 ms 2024-04-23T18:33:32.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:32 smithi050 ceph-mon[29187]: pgmap v475: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 361 KiB/s wr, 45 op/s 2024-04-23T18:33:32.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:32 smithi177 ceph-mon[32308]: pgmap v475: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 361 KiB/s wr, 45 op/s 2024-04-23T18:33:32.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 54606 21.33 MB/sec execute 6 sec latency 62.579 ms 2024-04-23T18:33:32.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:32 smithi188 ceph-mon[32694]: pgmap v475: 97 pgs: 97 active+clean; 672 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 361 KiB/s wr, 45 op/s 2024-04-23T18:33:33.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 55307 23.26 MB/sec execute 7 sec latency 85.775 ms 2024-04-23T18:33:34.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:34 smithi050 ceph-mon[29187]: pgmap v476: 97 pgs: 97 active+clean; 686 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 157 op/s 2024-04-23T18:33:34.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:34 smithi177 ceph-mon[32308]: pgmap v476: 97 pgs: 97 active+clean; 686 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 157 op/s 2024-04-23T18:33:34.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 55931 24.48 MB/sec execute 8 sec latency 54.479 ms 2024-04-23T18:33:34.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:34 smithi188 ceph-mon[32694]: pgmap v476: 97 pgs: 97 active+clean; 686 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 157 op/s 2024-04-23T18:33:35.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 56538 25.08 MB/sec execute 9 sec latency 65.996 ms 2024-04-23T18:33:36.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:36 smithi050 ceph-mon[29187]: pgmap v477: 97 pgs: 97 active+clean; 686 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-04-23T18:33:36.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:36 smithi177 ceph-mon[32308]: pgmap v477: 97 pgs: 97 active+clean; 686 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-04-23T18:33:36.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 57215 26.40 MB/sec execute 10 sec latency 46.991 ms 2024-04-23T18:33:36.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:36 smithi188 ceph-mon[32694]: pgmap v477: 97 pgs: 97 active+clean; 686 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-04-23T18:33:37.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.026+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1101098853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36e81010f0 msgr2=0x7f36e81014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:37.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.026+0000 7f36ed3a1640 1 --2- 172.21.15.50:0/1101098853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36e81010f0 0x7f36e81014d0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009900 tx=0x7f36d802a410 comp rx=0 tx=0).stop 2024-04-23T18:33:37.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.026+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1101098853 shutdown_connections 2024-04-23T18:33:37.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.026+0000 7f36ed3a1640 1 --2- 172.21.15.50:0/1101098853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e8106170 0x7f36e810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.026+0000 7f36ed3a1640 1 --2- 172.21.15.50:0/1101098853 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36e8101aa0 0x7f36e8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.026+0000 7f36ed3a1640 1 --2- 172.21.15.50:0/1101098853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36e81010f0 0x7f36e81014d0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.026+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1101098853 >> 172.21.15.50:0/1101098853 conn(0x7f36e8077fd0 msgr2=0x7f36e80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.026+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1101098853 shutdown_connections 2024-04-23T18:33:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.026+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1101098853 wait complete. 2024-04-23T18:33:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.027+0000 7f36ed3a1640 1 Processor -- start 2024-04-23T18:33:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.027+0000 7f36ed3a1640 1 -- start start 2024-04-23T18:33:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.027+0000 7f36ed3a1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36e81010f0 0x7f36e810e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.027+0000 7f36ed3a1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e8106170 0x7f36e810ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.027+0000 7f36ed3a1640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36e810f150 0x7f36e81a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.027+0000 7f36ed3a1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81121f0 con 0x7f36e81010f0 2024-04-23T18:33:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.027+0000 7f36ed3a1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f36e8112070 con 0x7f36e8106170 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.027+0000 7f36ed3a1640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f36e8112370 con 0x7f36e810f150 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e67fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e8106170 0x7f36e810ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36e81010f0 0x7f36e810e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e77fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36e810f150 0x7f36e81a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36e81010f0 0x7f36e810e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42370/0 (socket says 172.21.15.50:42370) 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e6ffd640 1 -- 172.21.15.50:0/1791089855 learned_addr learned my addr 172.21.15.50:0/1791089855 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e67fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e8106170 0x7f36e810ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:39164/0 (socket says 172.21.15.50:39164) 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e67fc640 1 -- 172.21.15.50:0/1791089855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36e810f150 msgr2=0x7f36e81a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e67fc640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36e810f150 0x7f36e81a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e67fc640 1 -- 172.21.15.50:0/1791089855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36e81010f0 msgr2=0x7f36e810e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e67fc640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36e81010f0 0x7f36e810e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e67fc640 1 -- 172.21.15.50:0/1791089855 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e81a6f80 con 0x7f36e8106170 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e6ffd640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36e81010f0 0x7f36e810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36e67fc640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e8106170 0x7f36e810ec10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00e9c0 tx=0x7f36dc00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36cffff640 1 -- 172.21.15.50:0/1791089855 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc00ccc0 con 0x7f36e8106170 2024-04-23T18:33:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1791089855 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e81a7270 con 0x7f36e8106170 2024-04-23T18:33:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36cffff640 1 -- 172.21.15.50:0/1791089855 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36dc005020 con 0x7f36e8106170 2024-04-23T18:33:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.028+0000 7f36cffff640 1 -- 172.21.15.50:0/1791089855 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc00ce60 con 0x7f36e8106170 2024-04-23T18:33:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.029+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1791089855 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e81a77c0 con 0x7f36e8106170 2024-04-23T18:33:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36cffff640 1 -- 172.21.15.50:0/1791089855 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f36dc0051d0 con 0x7f36e8106170 2024-04-23T18:33:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36f018b740 1 Processor -- start 2024-04-23T18:33:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36f018b740 1 -- start start 2024-04-23T18:33:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36f018b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0620 0x55d8b69a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36f018b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 0x55d8b69ab2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36f018b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69ab800 0x55d8b69adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36f018b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d8b6905970 con 0x55d8b69a0f40 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36f018b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d8b68a24a0 con 0x55d8b69a0620 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36f018b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d8b6901930 con 0x55d8b69ab800 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36cffff640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f36d00a3df0 0x7f36d00a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36e6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 0x55d8b69ab2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36e6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 0x55d8b69ab2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42376/0 (socket says 172.21.15.50:42376) 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36e67fc640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69ab800 0x55d8b69adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.031+0000 7f36e6ffd640 1 -- 172.21.15.50:0/123800530 learned_addr learned my addr 172.21.15.50:0/123800530 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36e77fe640 1 --2- 172.21.15.50:0/123800530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0620 0x55d8b69a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36e77fe640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f36d00a3df0 0x7f36d00a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36e6ffd640 1 -- 172.21.15.50:0/123800530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69ab800 msgr2=0x55d8b69adbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36e6ffd640 1 --2- 172.21.15.50:0/123800530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69ab800 0x55d8b69adbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36e6ffd640 1 -- 172.21.15.50:0/123800530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0620 msgr2=0x55d8b69a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36e6ffd640 1 --2- 172.21.15.50:0/123800530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0620 0x55d8b69a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36cffff640 1 -- 172.21.15.50:0/1791089855 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f36dc0d0700 con 0x7f36e8106170 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36e6ffd640 1 -- 172.21.15.50:0/123800530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8b6618210 con 0x55d8b69a0f40 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36e6ffd640 1 --2- 172.21.15.50:0/123800530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 0x55d8b69ab2c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f36d8000c00 tx=0x7f36d8032ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b3f6b9294955b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36e77fe640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f36d00a3df0 0x7f36d00a62b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f36e0005fd0 tx=0x7f36e0009500 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36cd7fa640 1 -- 172.21.15.50:0/123800530 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8035040 con 0x55d8b69a0f40 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36cd7fa640 1 -- 172.21.15.50:0/123800530 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36d80317a0 con 0x55d8b69a0f40 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.032+0000 7f36cd7fa640 1 -- 172.21.15.50:0/123800530 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d80047a0 con 0x55d8b69a0f40 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.033+0000 7f36f018b740 1 -- 172.21.15.50:0/123800530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 msgr2=0x55d8b69ab2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.033+0000 7f36f018b740 1 --2- 172.21.15.50:0/123800530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 0x55d8b69ab2c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f36d8000c00 tx=0x7f36d8032ed0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.033+0000 7f36f018b740 1 -- 172.21.15.50:0/123800530 shutdown_connections 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.033+0000 7f36f018b740 1 --2- 172.21.15.50:0/123800530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69ab800 0x55d8b69adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.033+0000 7f36f018b740 1 --2- 172.21.15.50:0/123800530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 0x55d8b69ab2c0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.033+0000 7f36f018b740 1 --2- 172.21.15.50:0/123800530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0620 0x55d8b69a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.033+0000 7f36f018b740 1 -- 172.21.15.50:0/123800530 >> 172.21.15.50:0/123800530 conn(0x55d8b699cb30 msgr2=0x55d8b699cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.033+0000 7f36f018b740 1 -- 172.21.15.50:0/123800530 shutdown_connections 2024-04-23T18:33:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.033+0000 7f36f018b740 1 -- 172.21.15.50:0/123800530 wait complete. 2024-04-23T18:33:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.033+0000 7f36f018b740 1 -- start start 2024-04-23T18:33:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.033+0000 7f36f018b740 1 Processor -- start 2024-04-23T18:33:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36f018b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0620 0x55d8b69ab160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36f018b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0f40 0x55d8b69a8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36f018b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69ab800 0x55d8b69a89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36f018b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55d8b69ae3d0 con 0x55d8b69a0620 2024-04-23T18:33:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36f018b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d8b68a00b0 con 0x55d8b69a0f40 2024-04-23T18:33:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36f018b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55d8b69ae550 con 0x55d8b69ab800 2024-04-23T18:33:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36f018b740 10 client.0 _ll_register_callbacks cb 0x55d8b688dd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36e6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0620 0x55d8b69ab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36e6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0620 0x55d8b69ab160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42386/0 (socket says 172.21.15.50:42386) 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36e6ffd640 1 -- 172.21.15.50:0/1150399885 learned_addr learned my addr 172.21.15.50:0/1150399885 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36e77fe640 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69ab800 0x55d8b69a89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36e67fc640 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0f40 0x55d8b69a8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36e6ffd640 1 -- 172.21.15.50:0/1150399885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69ab800 msgr2=0x55d8b69a89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36e6ffd640 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69ab800 0x55d8b69a89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36e6ffd640 1 -- 172.21.15.50:0/1150399885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0f40 msgr2=0x55d8b69a8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36e6ffd640 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0f40 0x55d8b69a8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36e6ffd640 1 -- 172.21.15.50:0/1150399885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8b66e4620 con 0x55d8b69a0620 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36e6ffd640 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0620 0x55d8b69ab160 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f36d8004620 tx=0x7f36d8005360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110c978a5f3000ad server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36caff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36caff5640 1 -- 172.21.15.50:0/1150399885 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d804e070 con 0x55d8b69a0620 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.034+0000 7f36caff5640 1 -- 172.21.15.50:0/1150399885 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f36d8046660 con 0x55d8b69a0620 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36f018b740 1 -- 172.21.15.50:0/1150399885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36d80317a0 con 0x55d8b69a0620 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36caff5640 1 -- 172.21.15.50:0/1150399885 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f36d805a050 con 0x55d8b69a0620 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36f018b740 10 client.17376 fetch_fsmap learned FSMap version 52 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36f018b740 1 -- 172.21.15.50:0/1150399885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x55d8b69ba300 con 0x55d8b69a0620 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36caff5640 1 -- 172.21.15.50:0/1150399885 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36d80317a0 con 0x55d8b69a0620 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36caff5640 1 -- 172.21.15.50:0/1150399885 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f36d803c930 con 0x55d8b69a0620 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36f018b740 10 client.17376 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36f018b740 10 client.17376 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36f018b740 20 client.17376 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36f018b740 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d8b69bb650 0x55d8b69bda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36f018b740 4 client.17376 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36f018b740 1 -- 172.21.15.50:0/1150399885 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8b6913390 con 0x55d8b69bb650 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.035+0000 7f36e67fc640 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d8b69bb650 0x55d8b69bda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.036+0000 7f36e67fc640 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d8b69bb650 0x55d8b69bda40 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5db3c90195d5874b server_cookie=a22fbf4c50b3e385 in_seq=0 out_seq=0 2024-04-23T18:33:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.036+0000 7f36caff5640 10 client.17376 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.036+0000 7f36caff5640 1 -- 172.21.15.50:0/1150399885 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d8b6913390 con 0x55d8b69bb650 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.036+0000 7f36caff5640 10 client.17376 handle_command_reply: tid=0 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.036+0000 7f36f018b740 1 client.17376 shutdown 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.037+0000 7f36f018b740 1 -- 172.21.15.50:0/1150399885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0620 msgr2=0x55d8b69ab160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.037+0000 7f36f018b740 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0620 0x55d8b69ab160 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f36d8004620 tx=0x7f36d8005360 comp rx=0 tx=0).stop 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.037+0000 7f36f018b740 1 -- 172.21.15.50:0/1150399885 shutdown_connections 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.037+0000 7f36f018b740 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d8b69bb650 0x55d8b69bda40 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.037+0000 7f36f018b740 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69ab800 0x55d8b69a89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.037+0000 7f36f018b740 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0f40 0x55d8b69a8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.037+0000 7f36caff5640 0 client.17376 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.037+0000 7f36f018b740 1 --2- 172.21.15.50:0/1150399885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0620 0x55d8b69ab160 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.037+0000 7f36f018b740 1 -- 172.21.15.50:0/1150399885 >> 172.21.15.50:0/1150399885 conn(0x55d8b699cb30 msgr2=0x55d8b69a28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.037+0000 7f36f018b740 1 -- 172.21.15.50:0/1150399885 shutdown_connections 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.037+0000 7f36f018b740 1 -- 172.21.15.50:0/1150399885 wait complete. 2024-04-23T18:33:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.037+0000 7f36f018b740 20 client.17376 trim_cache size 0 max 16384 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36f018b740 1 Processor -- start 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36f018b740 1 -- start start 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36f018b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69a0620 0x55d8b69b7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36f018b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0f40 0x55d8b69a3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36f018b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69ab800 0x55d8b69aac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36f018b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81b2670 con 0x55d8b69ab800 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36f018b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81afe80 con 0x55d8b69a0f40 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36f018b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81adcf0 con 0x55d8b69a0620 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36e77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69ab800 0x55d8b69aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36e77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69ab800 0x55d8b69aac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42398/0 (socket says 172.21.15.50:42398) 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36e77fe640 1 -- 172.21.15.50:0/3155334067 learned_addr learned my addr 172.21.15.50:0/3155334067 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36e67fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0f40 0x55d8b69a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.055+0000 7f36e6ffd640 1 --2- 172.21.15.50:0/3155334067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69a0620 0x55d8b69b7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.056+0000 7f36e67fc640 1 -- 172.21.15.50:0/3155334067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69a0620 msgr2=0x55d8b69b7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.056+0000 7f36e67fc640 1 --2- 172.21.15.50:0/3155334067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69a0620 0x55d8b69b7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.056+0000 7f36e67fc640 1 -- 172.21.15.50:0/3155334067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69ab800 msgr2=0x55d8b69aac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.056+0000 7f36e67fc640 1 --2- 172.21.15.50:0/3155334067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69ab800 0x55d8b69aac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.056+0000 7f36e67fc640 1 -- 172.21.15.50:0/3155334067 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8b69a0d30 con 0x55d8b69a0f40 2024-04-23T18:33:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.056+0000 7f36e77fe640 1 --2- 172.21.15.50:0/3155334067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69ab800 0x55d8b69aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:33:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.056+0000 7f36e67fc640 1 --2- 172.21.15.50:0/3155334067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0f40 0x55d8b69a3970 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0cde40 tx=0x7f36dc0c77a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=483ecbac84617611 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.056+0000 7f36ccff9640 1 -- 172.21.15.50:0/3155334067 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc01c040 con 0x55d8b69a0f40 2024-04-23T18:33:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.056+0000 7f36ccff9640 1 -- 172.21.15.50:0/3155334067 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36dc0a5300 con 0x55d8b69a0f40 2024-04-23T18:33:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.056+0000 7f36ccff9640 1 -- 172.21.15.50:0/3155334067 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc085bd0 con 0x55d8b69a0f40 2024-04-23T18:33:37.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.057+0000 7f36f018b740 1 -- 172.21.15.50:0/3155334067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0f40 msgr2=0x55d8b69a3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.057+0000 7f36f018b740 1 --2- 172.21.15.50:0/3155334067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0f40 0x55d8b69a3970 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0cde40 tx=0x7f36dc0c77a0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.057+0000 7f36f018b740 1 -- 172.21.15.50:0/3155334067 shutdown_connections 2024-04-23T18:33:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.057+0000 7f36f018b740 1 --2- 172.21.15.50:0/3155334067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69ab800 0x55d8b69aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.057+0000 7f36f018b740 1 --2- 172.21.15.50:0/3155334067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69a0f40 0x55d8b69a3970 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.057+0000 7f36f018b740 1 --2- 172.21.15.50:0/3155334067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69a0620 0x55d8b69b7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.057+0000 7f36f018b740 1 -- 172.21.15.50:0/3155334067 >> 172.21.15.50:0/3155334067 conn(0x55d8b699cb30 msgr2=0x55d8b69a3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.057+0000 7f36f018b740 1 -- 172.21.15.50:0/3155334067 shutdown_connections 2024-04-23T18:33:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.057+0000 7f36f018b740 1 -- 172.21.15.50:0/3155334067 wait complete. 2024-04-23T18:33:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.057+0000 7f36f018b740 1 -- start start 2024-04-23T18:33:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.057+0000 7f36f018b740 1 Processor -- start 2024-04-23T18:33:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36f018b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69a0620 0x55d8b69b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36f018b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 0x55d8b69aef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36f018b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69ab800 0x55d8b69af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36f018b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81ae340 con 0x55d8b69a0f40 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36f018b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81af7b0 con 0x55d8b69ab800 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36f018b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81aea10 con 0x55d8b69a0620 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36f018b740 10 client.0 _ll_register_callbacks cb 0x7f36dc0982f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36e67fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 0x55d8b69aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36e6ffd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69a0620 0x55d8b69b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36e67fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 0x55d8b69aef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42406/0 (socket says 172.21.15.50:42406) 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36e67fc640 1 -- 172.21.15.50:0/512850030 learned_addr learned my addr 172.21.15.50:0/512850030 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36e77fe640 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69ab800 0x55d8b69af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36e67fc640 1 -- 172.21.15.50:0/512850030 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69a0620 msgr2=0x55d8b69b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36e67fc640 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69a0620 0x55d8b69b3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36e67fc640 1 -- 172.21.15.50:0/512850030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69ab800 msgr2=0x55d8b69af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36e67fc640 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69ab800 0x55d8b69af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36e67fc640 1 -- 172.21.15.50:0/512850030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8b69afce0 con 0x55d8b69a0f40 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36e77fe640 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69ab800 0x55d8b69af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36e67fc640 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 0x55d8b69aef70 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f36dc09c620 tx=0x7f36dc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=566a544ce3e1247f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36caff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36caff5640 1 -- 172.21.15.50:0/512850030 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc01c070 con 0x55d8b69a0f40 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36caff5640 1 -- 172.21.15.50:0/512850030 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f36dc0054f0 con 0x55d8b69a0f40 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.058+0000 7f36f018b740 1 -- 172.21.15.50:0/512850030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36dc0a5300 con 0x55d8b69a0f40 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.059+0000 7f36caff5640 1 -- 172.21.15.50:0/512850030 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f36dc09c690 con 0x55d8b69a0f40 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.059+0000 7f36f018b740 10 client.17382 fetch_fsmap learned FSMap version 52 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.059+0000 7f36f018b740 1 -- 172.21.15.50:0/512850030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x55d8b6a218f0 con 0x55d8b69a0f40 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.059+0000 7f36caff5640 1 -- 172.21.15.50:0/512850030 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36dc0a5300 con 0x55d8b69a0f40 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.059+0000 7f36caff5640 1 -- 172.21.15.50:0/512850030 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 52) ==== 8362+0+0 (secure 0 0 0) 0x7f36dc01f030 con 0x55d8b69a0f40 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.059+0000 7f36f018b740 10 client.17382 fetch_fsmap finished waiting for FSMap version 52 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.059+0000 7f36f018b740 10 client.17382 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.059+0000 7f36f018b740 20 client.17382 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.059+0000 7f36f018b740 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d8b6a21fe0 0x55d8b6a24400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.059+0000 7f36f018b740 4 client.17382 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:33:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.059+0000 7f36f018b740 1 -- 172.21.15.50:0/512850030 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d8b690d200 con 0x55d8b6a21fe0 2024-04-23T18:33:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.059+0000 7f36e6ffd640 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d8b6a21fe0 0x55d8b6a24400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.060+0000 7f36e6ffd640 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d8b6a21fe0 0x55d8b6a24400 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dca4085eb7b67b46 server_cookie=bbf7592de828332 in_seq=0 out_seq=0 2024-04-23T18:33:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.060+0000 7f36caff5640 10 client.17382 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36caff5640 1 -- 172.21.15.50:0/512850030 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d8b690d200 con 0x55d8b6a21fe0 2024-04-23T18:33:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36caff5640 10 client.17382 handle_command_reply: tid=0 2024-04-23T18:33:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36f018b740 1 client.17382 shutdown 2024-04-23T18:33:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36f018b740 1 -- 172.21.15.50:0/512850030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 msgr2=0x55d8b69aef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36f018b740 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 0x55d8b69aef70 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f36dc09c620 tx=0x7f36dc009f90 comp rx=0 tx=0).stop 2024-04-23T18:33:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36f018b740 1 -- 172.21.15.50:0/512850030 shutdown_connections 2024-04-23T18:33:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36f018b740 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55d8b6a21fe0 0x55d8b6a24400 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36f018b740 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8b69ab800 0x55d8b69af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36caff5640 0 client.17382 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36f018b740 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55d8b69a0f40 0x55d8b69aef70 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36f018b740 1 --2- 172.21.15.50:0/512850030 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d8b69a0620 0x55d8b69b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36f018b740 1 -- 172.21.15.50:0/512850030 >> 172.21.15.50:0/512850030 conn(0x55d8b699cb30 msgr2=0x55d8b699bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.251+0000 7f36f018b740 1 -- 172.21.15.50:0/512850030 shutdown_connections 2024-04-23T18:33:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.252+0000 7f36f018b740 1 -- 172.21.15.50:0/512850030 wait complete. 2024-04-23T18:33:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.252+0000 7f36f018b740 20 client.17382 trim_cache size 0 max 16384 2024-04-23T18:33:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:33:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:33:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:33:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:33:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.252+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1791089855 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f36d00a3df0 msgr2=0x7f36d00a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.252+0000 7f36ed3a1640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f36d00a3df0 0x7f36d00a62b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f36e0005fd0 tx=0x7f36e0009500 comp rx=0 tx=0).stop 2024-04-23T18:33:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.252+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1791089855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e8106170 msgr2=0x7f36e810ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.252+0000 7f36ed3a1640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e8106170 0x7f36e810ec10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00e9c0 tx=0x7f36dc00edc0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.252+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1791089855 shutdown_connections 2024-04-23T18:33:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.252+0000 7f36ed3a1640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f36d00a3df0 0x7f36d00a62b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.252+0000 7f36ed3a1640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36e810f150 0x7f36e81a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.252+0000 7f36ed3a1640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36e8106170 0x7f36e810ec10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.252+0000 7f36ed3a1640 1 --2- 172.21.15.50:0/1791089855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36e81010f0 0x7f36e810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.252+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1791089855 >> 172.21.15.50:0/1791089855 conn(0x7f36e8077fd0 msgr2=0x7f36e80ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.253+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1791089855 shutdown_connections 2024-04-23T18:33:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:37.253+0000 7f36ed3a1640 1 -- 172.21.15.50:0/1791089855 wait complete. 2024-04-23T18:33:37.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 57705 26.30 MB/sec execute 11 sec latency 137.003 ms 2024-04-23T18:33:38.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:38 smithi050 ceph-mon[29187]: pgmap v478: 97 pgs: 97 active+clean; 715 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.0 MiB/s wr, 343 op/s 2024-04-23T18:33:38.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:38 smithi177 ceph-mon[32308]: pgmap v478: 97 pgs: 97 active+clean; 715 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.0 MiB/s wr, 343 op/s 2024-04-23T18:33:38.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 58320 26.22 MB/sec execute 12 sec latency 107.482 ms 2024-04-23T18:33:38.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:38 smithi188 ceph-mon[32694]: pgmap v478: 97 pgs: 97 active+clean; 715 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.0 MiB/s wr, 343 op/s 2024-04-23T18:33:39.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 59079 27.69 MB/sec execute 13 sec latency 60.288 ms 2024-04-23T18:33:40.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:40 smithi050 ceph-mon[29187]: pgmap v479: 97 pgs: 97 active+clean; 715 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.0 MiB/s wr, 319 op/s 2024-04-23T18:33:40.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:40 smithi177 ceph-mon[32308]: pgmap v479: 97 pgs: 97 active+clean; 715 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.0 MiB/s wr, 319 op/s 2024-04-23T18:33:40.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 59770 28.67 MB/sec execute 14 sec latency 61.431 ms 2024-04-23T18:33:40.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:40 smithi188 ceph-mon[32694]: pgmap v479: 97 pgs: 97 active+clean; 715 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.0 MiB/s wr, 319 op/s 2024-04-23T18:33:41.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 60452 29.82 MB/sec execute 15 sec latency 40.844 ms 2024-04-23T18:33:42.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:42 smithi050 ceph-mon[29187]: pgmap v480: 97 pgs: 97 active+clean; 715 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.0 MiB/s wr, 319 op/s 2024-04-23T18:33:42.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:42 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:33:42.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:42 smithi177 ceph-mon[32308]: pgmap v480: 97 pgs: 97 active+clean; 715 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.0 MiB/s wr, 319 op/s 2024-04-23T18:33:42.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:42 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:33:42.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 60955 29.05 MB/sec execute 16 sec latency 110.868 ms 2024-04-23T18:33:42.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:42 smithi188 ceph-mon[32694]: pgmap v480: 97 pgs: 97 active+clean; 715 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.0 MiB/s wr, 319 op/s 2024-04-23T18:33:42.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:42 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:33:43.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 61639 29.32 MB/sec execute 17 sec latency 65.885 ms 2024-04-23T18:33:44.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:44 smithi050 ceph-mon[29187]: pgmap v481: 97 pgs: 97 active+clean; 734 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 6.6 MiB/s wr, 580 op/s 2024-04-23T18:33:44.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:44 smithi177 ceph-mon[32308]: pgmap v481: 97 pgs: 97 active+clean; 734 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 6.6 MiB/s wr, 580 op/s 2024-04-23T18:33:44.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 62323 29.84 MB/sec execute 18 sec latency 265.005 ms 2024-04-23T18:33:44.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:44 smithi188 ceph-mon[32694]: pgmap v481: 97 pgs: 97 active+clean; 734 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 6.6 MiB/s wr, 580 op/s 2024-04-23T18:33:45.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 62975 30.07 MB/sec execute 19 sec latency 851.147 ms 2024-04-23T18:33:46.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:46 smithi050 ceph-mon[29187]: pgmap v482: 97 pgs: 97 active+clean; 734 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.4 MiB/s wr, 469 op/s 2024-04-23T18:33:46.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:46 smithi177 ceph-mon[32308]: pgmap v482: 97 pgs: 97 active+clean; 734 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.4 MiB/s wr, 469 op/s 2024-04-23T18:33:46.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 63757 30.91 MB/sec execute 20 sec latency 42.595 ms 2024-04-23T18:33:46.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:46 smithi188 ceph-mon[32694]: pgmap v482: 97 pgs: 97 active+clean; 734 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.4 MiB/s wr, 469 op/s 2024-04-23T18:33:47.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 64362 31.16 MB/sec execute 21 sec latency 77.254 ms 2024-04-23T18:33:48.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:48 smithi050 ceph-mon[29187]: pgmap v483: 97 pgs: 97 active+clean; 763 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 8.2 MiB/s wr, 673 op/s 2024-04-23T18:33:48.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:48 smithi177 ceph-mon[32308]: pgmap v483: 97 pgs: 97 active+clean; 763 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 8.2 MiB/s wr, 673 op/s 2024-04-23T18:33:48.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 64910 30.96 MB/sec execute 22 sec latency 54.781 ms 2024-04-23T18:33:48.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:48 smithi188 ceph-mon[32694]: pgmap v483: 97 pgs: 97 active+clean; 763 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 8.2 MiB/s wr, 673 op/s 2024-04-23T18:33:49.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 65542 30.62 MB/sec execute 23 sec latency 113.500 ms 2024-04-23T18:33:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.365+0000 7f628e2e1640 1 -- 172.21.15.50:0/2388367148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62881021b0 msgr2=0x7f6288102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.365+0000 7f628e2e1640 1 --2- 172.21.15.50:0/2388367148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62881021b0 0x7f6288102590 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f6278009900 tx=0x7f627802a3f0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.365+0000 7f628e2e1640 1 -- 172.21.15.50:0/2388367148 shutdown_connections 2024-04-23T18:33:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.365+0000 7f628e2e1640 1 --2- 172.21.15.50:0/2388367148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6288107280 0x7f628810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.365+0000 7f628e2e1640 1 --2- 172.21.15.50:0/2388367148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6288102b60 0x7f6288106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.365+0000 7f628e2e1640 1 --2- 172.21.15.50:0/2388367148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62881021b0 0x7f6288102590 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.365+0000 7f628e2e1640 1 -- 172.21.15.50:0/2388367148 >> 172.21.15.50:0/2388367148 conn(0x7f62880fda10 msgr2=0x7f62880ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.365+0000 7f628e2e1640 1 -- 172.21.15.50:0/2388367148 shutdown_connections 2024-04-23T18:33:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.365+0000 7f628e2e1640 1 -- 172.21.15.50:0/2388367148 wait complete. 2024-04-23T18:33:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.366+0000 7f628e2e1640 1 Processor -- start 2024-04-23T18:33:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.366+0000 7f628e2e1640 1 -- start start 2024-04-23T18:33:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.366+0000 7f628e2e1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62881021b0 0x7f6288198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.366+0000 7f628e2e1640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6288107280 0x7f6288198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.366+0000 7f628e2e1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f628819cfa0 0x7f628819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.366+0000 7f628e2e1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f62881121f0 con 0x7f62881021b0 2024-04-23T18:33:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.366+0000 7f628e2e1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6288112070 con 0x7f628819cfa0 2024-04-23T18:33:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.366+0000 7f628e2e1640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6288112370 con 0x7f6288107280 2024-04-23T18:33:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.366+0000 7f6287fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62881021b0 0x7f6288198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f628c87e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f628819cfa0 0x7f628819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f62877fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6288107280 0x7f6288198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f628c87e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f628819cfa0 0x7f628819d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:44086/0 (socket says 172.21.15.50:44086) 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f62877fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6288107280 0x7f6288198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:59036/0 (socket says 172.21.15.50:59036) 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f6287fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62881021b0 0x7f6288198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51214/0 (socket says 172.21.15.50:51214) 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f628c87e640 1 -- 172.21.15.50:0/2229665987 learned_addr learned my addr 172.21.15.50:0/2229665987 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f6287fff640 1 -- 172.21.15.50:0/2229665987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6288107280 msgr2=0x7f6288198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f6287fff640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6288107280 0x7f6288198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f6287fff640 1 -- 172.21.15.50:0/2229665987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f628819cfa0 msgr2=0x7f628819d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f6287fff640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f628819cfa0 0x7f628819d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f6287fff640 1 -- 172.21.15.50:0/2229665987 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628819da90 con 0x7f62881021b0 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f628c87e640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f628819cfa0 0x7f628819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f6287fff640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62881021b0 0x7f6288198730 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f627802a910 tx=0x7f6278031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f62857fa640 1 -- 172.21.15.50:0/2229665987 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627803f030 con 0x7f62881021b0 2024-04-23T18:33:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f628e2e1640 1 -- 172.21.15.50:0/2229665987 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f628819dd20 con 0x7f62881021b0 2024-04-23T18:33:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f62857fa640 1 -- 172.21.15.50:0/2229665987 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6278004490 con 0x7f62881021b0 2024-04-23T18:33:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f62857fa640 1 -- 172.21.15.50:0/2229665987 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6278035070 con 0x7f62881021b0 2024-04-23T18:33:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.367+0000 7f628e2e1640 1 -- 172.21.15.50:0/2229665987 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62881a5540 con 0x7f62881021b0 2024-04-23T18:33:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f62857fa640 1 -- 172.21.15.50:0/2229665987 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6278042030 con 0x7f62881021b0 2024-04-23T18:33:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f6290ff4740 1 Processor -- start 2024-04-23T18:33:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f6290ff4740 1 -- start start 2024-04-23T18:33:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f6290ff4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fb620 0x55b44b8fba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f6290ff4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 0x55b44b9062c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f6290ff4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b906800 0x55b44b908bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f6290ff4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b44b860970 con 0x55b44b8fbf40 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f6290ff4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b44b7fd4a0 con 0x55b44b8fb620 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f6290ff4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b44b85c930 con 0x55b44b906800 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f62857fa640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f62700a3dc0 0x7f62700a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f6287fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 0x55b44b9062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f628c87e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fb620 0x55b44b8fba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f628c87e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fb620 0x55b44b8fba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:44098/0 (socket says 172.21.15.50:44098) 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f628c87e640 1 -- 172.21.15.50:0/237642541 learned_addr learned my addr 172.21.15.50:0/237642541 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f62877fe640 1 --2- 172.21.15.50:0/237642541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b906800 0x55b44b908bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.370+0000 7f6287fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 0x55b44b9062c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51230/0 (socket says 172.21.15.50:51230) 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f628c87e640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f62700a3dc0 0x7f62700a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f62857fa640 1 -- 172.21.15.50:0/2229665987 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f62780f0a20 con 0x7f62881021b0 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f628c87e640 1 -- 172.21.15.50:0/237642541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b906800 msgr2=0x55b44b908bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f628c87e640 1 --2- 172.21.15.50:0/237642541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b906800 0x55b44b908bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f628c87e640 1 -- 172.21.15.50:0/237642541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 msgr2=0x55b44b9062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f628c87e640 1 --2- 172.21.15.50:0/237642541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 0x55b44b9062c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f628c87e640 1 -- 172.21.15.50:0/237642541 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b44b573210 con 0x55b44b8fb620 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f628c87e640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f62700a3dc0 0x7f62700a6280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f628000e720 tx=0x7f6280018000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f628c87e640 1 --2- 172.21.15.50:0/237642541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fb620 0x55b44b8fba00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f62800116e0 tx=0x7f6280013f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f9c87061c597896 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f626e7fc640 1 -- 172.21.15.50:0/237642541 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6280025070 con 0x55b44b8fb620 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f626e7fc640 1 -- 172.21.15.50:0/237642541 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6280013690 con 0x55b44b8fb620 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.371+0000 7f626e7fc640 1 -- 172.21.15.50:0/237642541 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628000cd40 con 0x55b44b8fb620 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.372+0000 7f6290ff4740 1 -- 172.21.15.50:0/237642541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fb620 msgr2=0x55b44b8fba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.372+0000 7f6290ff4740 1 --2- 172.21.15.50:0/237642541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fb620 0x55b44b8fba00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f62800116e0 tx=0x7f6280013f90 comp rx=0 tx=0).stop 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.372+0000 7f6290ff4740 1 -- 172.21.15.50:0/237642541 shutdown_connections 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.372+0000 7f6290ff4740 1 --2- 172.21.15.50:0/237642541 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b906800 0x55b44b908bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.372+0000 7f6290ff4740 1 --2- 172.21.15.50:0/237642541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 0x55b44b9062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.372+0000 7f6290ff4740 1 --2- 172.21.15.50:0/237642541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fb620 0x55b44b8fba00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.372+0000 7f6290ff4740 1 -- 172.21.15.50:0/237642541 >> 172.21.15.50:0/237642541 conn(0x55b44b8f7b30 msgr2=0x55b44b8f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.372+0000 7f6290ff4740 1 -- 172.21.15.50:0/237642541 shutdown_connections 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.372+0000 7f6290ff4740 1 -- 172.21.15.50:0/237642541 wait complete. 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.372+0000 7f6290ff4740 1 -- start start 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6290ff4740 1 Processor -- start 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6290ff4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fb620 0x55b44b906150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6290ff4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fbf40 0x55b44b903440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6290ff4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b906800 0x55b44b9039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6290ff4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b44b9093d0 con 0x55b44b8fb620 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6290ff4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b44b7fb0b0 con 0x55b44b8fbf40 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6290ff4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b44b909550 con 0x55b44b906800 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6290ff4740 10 client.0 _ll_register_callbacks cb 0x55b44b7e8d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6287fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fb620 0x55b44b906150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6287fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fb620 0x55b44b906150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51234/0 (socket says 172.21.15.50:51234) 2024-04-23T18:33:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6287fff640 1 -- 172.21.15.50:0/437008999 learned_addr learned my addr 172.21.15.50:0/437008999 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f62877fe640 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fbf40 0x55b44b903440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f628c87e640 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b906800 0x55b44b9039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6287fff640 1 -- 172.21.15.50:0/437008999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b906800 msgr2=0x55b44b9039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6287fff640 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b906800 0x55b44b9039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6287fff640 1 -- 172.21.15.50:0/437008999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fbf40 msgr2=0x55b44b903440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6287fff640 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fbf40 0x55b44b903440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f6287fff640 1 -- 172.21.15.50:0/437008999 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b44b63f620 con 0x55b44b8fb620 2024-04-23T18:33:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.373+0000 7f62877fe640 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fbf40 0x55b44b903440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:33:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f6287fff640 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fb620 0x55b44b906150 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f62780ee1c0 tx=0x7f6278009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d295ecbb99b1b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f62677fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f62677fe640 1 -- 172.21.15.50:0/437008999 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6278049040 con 0x55b44b8fb620 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f62677fe640 1 -- 172.21.15.50:0/437008999 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f62780ee3e0 con 0x55b44b8fb620 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f6290ff4740 1 -- 172.21.15.50:0/437008999 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6280013690 con 0x55b44b8fb620 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f62677fe640 1 -- 172.21.15.50:0/437008999 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f62780c0d60 con 0x55b44b8fb620 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f6290ff4740 10 client.17394 fetch_fsmap learned FSMap version 53 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f6290ff4740 1 -- 172.21.15.50:0/437008999 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55b44b915340 con 0x55b44b8fb620 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f62677fe640 1 -- 172.21.15.50:0/437008999 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6280013690 con 0x55b44b8fb620 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f62677fe640 1 -- 172.21.15.50:0/437008999 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 53) ==== 8358+0+0 (secure 0 0 0) 0x7f62780bdce0 con 0x55b44b8fb620 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f6290ff4740 10 client.17394 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f6290ff4740 10 client.17394 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f6290ff4740 20 client.17394 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f6290ff4740 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b44b9167d0 0x55b44b918c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f6290ff4740 4 client.17394 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.374+0000 7f6290ff4740 1 -- 172.21.15.50:0/437008999 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b44b86e390 con 0x55b44b9167d0 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.375+0000 7f62877fe640 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b44b9167d0 0x55b44b918c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.375+0000 7f62677fe640 10 client.17394 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.375+0000 7f62877fe640 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b44b9167d0 0x55b44b918c50 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eadd33ce9577683d server_cookie=fd9ab7c5fac74781 in_seq=0 out_seq=0 2024-04-23T18:33:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f62677fe640 1 -- 172.21.15.50:0/437008999 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b44b86e390 con 0x55b44b9167d0 2024-04-23T18:33:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f62677fe640 10 client.17394 handle_command_reply: tid=0 2024-04-23T18:33:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f6290ff4740 1 client.17394 shutdown 2024-04-23T18:33:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f6290ff4740 1 -- 172.21.15.50:0/437008999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fb620 msgr2=0x55b44b906150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f6290ff4740 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fb620 0x55b44b906150 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f62780ee1c0 tx=0x7f6278009670 comp rx=0 tx=0).stop 2024-04-23T18:33:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f6290ff4740 1 -- 172.21.15.50:0/437008999 shutdown_connections 2024-04-23T18:33:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f6290ff4740 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b44b9167d0 0x55b44b918c50 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f6290ff4740 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b906800 0x55b44b9039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f6290ff4740 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fbf40 0x55b44b903440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f62677fe640 0 client.17394 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f6290ff4740 1 --2- 172.21.15.50:0/437008999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fb620 0x55b44b906150 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f6290ff4740 1 -- 172.21.15.50:0/437008999 >> 172.21.15.50:0/437008999 conn(0x55b44b8f7b30 msgr2=0x55b44b8fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f6290ff4740 1 -- 172.21.15.50:0/437008999 shutdown_connections 2024-04-23T18:33:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f6290ff4740 1 -- 172.21.15.50:0/437008999 wait complete. 2024-04-23T18:33:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.376+0000 7f6290ff4740 20 client.17394 trim_cache size 0 max 16384 2024-04-23T18:33:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f6290ff4740 1 Processor -- start 2024-04-23T18:33:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f6290ff4740 1 -- start start 2024-04-23T18:33:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f6290ff4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fbf40 0x55b44b912290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f6290ff4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b906800 0x55b44b8fe850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f6290ff4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b903590 0x55b44b905c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f6290ff4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f62881b0530 con 0x55b44b906800 2024-04-23T18:33:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f6290ff4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f62881add70 con 0x55b44b8fbf40 2024-04-23T18:33:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f6290ff4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f62881abbe0 con 0x55b44b903590 2024-04-23T18:33:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f62877fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b906800 0x55b44b8fe850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f62877fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b906800 0x55b44b8fe850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51246/0 (socket says 172.21.15.50:51246) 2024-04-23T18:33:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f62877fe640 1 -- 172.21.15.50:0/482556732 learned_addr learned my addr 172.21.15.50:0/482556732 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f6287fff640 1 --2- 172.21.15.50:0/482556732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fbf40 0x55b44b912290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f628c87e640 1 --2- 172.21.15.50:0/482556732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b903590 0x55b44b905c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.393+0000 7f62877fe640 1 -- 172.21.15.50:0/482556732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b903590 msgr2=0x55b44b905c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.394+0000 7f62877fe640 1 --2- 172.21.15.50:0/482556732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b903590 0x55b44b905c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.394+0000 7f62877fe640 1 -- 172.21.15.50:0/482556732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fbf40 msgr2=0x55b44b912290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.394+0000 7f62877fe640 1 --2- 172.21.15.50:0/482556732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fbf40 0x55b44b912290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.394+0000 7f62877fe640 1 -- 172.21.15.50:0/482556732 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b44b9023f0 con 0x55b44b906800 2024-04-23T18:33:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.394+0000 7f62877fe640 1 --2- 172.21.15.50:0/482556732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b906800 0x55b44b8fe850 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f62880fdec0 tx=0x7f627c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b557fcbf71c7d50f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.394+0000 7f626dffb640 1 -- 172.21.15.50:0/482556732 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c028070 con 0x55b44b906800 2024-04-23T18:33:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.394+0000 7f626dffb640 1 -- 172.21.15.50:0/482556732 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f627c00f790 con 0x55b44b906800 2024-04-23T18:33:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.394+0000 7f626dffb640 1 -- 172.21.15.50:0/482556732 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c022410 con 0x55b44b906800 2024-04-23T18:33:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.395+0000 7f6290ff4740 1 -- 172.21.15.50:0/482556732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b906800 msgr2=0x55b44b8fe850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.395+0000 7f6290ff4740 1 --2- 172.21.15.50:0/482556732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b906800 0x55b44b8fe850 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f62880fdec0 tx=0x7f627c005f90 comp rx=0 tx=0).stop 2024-04-23T18:33:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.395+0000 7f6290ff4740 1 -- 172.21.15.50:0/482556732 shutdown_connections 2024-04-23T18:33:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.395+0000 7f6290ff4740 1 --2- 172.21.15.50:0/482556732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b903590 0x55b44b905c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.395+0000 7f6290ff4740 1 --2- 172.21.15.50:0/482556732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b906800 0x55b44b8fe850 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.395+0000 7f6290ff4740 1 --2- 172.21.15.50:0/482556732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b8fbf40 0x55b44b912290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.395+0000 7f6290ff4740 1 -- 172.21.15.50:0/482556732 >> 172.21.15.50:0/482556732 conn(0x55b44b8f7b30 msgr2=0x55b44b8fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.395+0000 7f6290ff4740 1 -- 172.21.15.50:0/482556732 shutdown_connections 2024-04-23T18:33:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.395+0000 7f6290ff4740 1 -- 172.21.15.50:0/482556732 wait complete. 2024-04-23T18:33:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.395+0000 7f6290ff4740 1 -- start start 2024-04-23T18:33:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6290ff4740 1 Processor -- start 2024-04-23T18:33:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6290ff4740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 0x55b44b90a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6290ff4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b906800 0x55b44b90a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6290ff4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b90ac80 0x55b44b97bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6290ff4740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f62881ac230 con 0x55b44b8fbf40 2024-04-23T18:33:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6290ff4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f62881ad6a0 con 0x55b44b906800 2024-04-23T18:33:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6290ff4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f62881ac900 con 0x55b44b90ac80 2024-04-23T18:33:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6290ff4740 10 client.0 _ll_register_callbacks cb 0x7f62780b85e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:33:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6287fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 0x55b44b90a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6287fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 0x55b44b90a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51252/0 (socket says 172.21.15.50:51252) 2024-04-23T18:33:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6287fff640 1 -- 172.21.15.50:0/2538788864 learned_addr learned my addr 172.21.15.50:0/2538788864 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:33:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f628c87e640 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b90ac80 0x55b44b97bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f62877fe640 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b906800 0x55b44b90a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6287fff640 1 -- 172.21.15.50:0/2538788864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b90ac80 msgr2=0x55b44b97bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6287fff640 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b90ac80 0x55b44b97bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6287fff640 1 -- 172.21.15.50:0/2538788864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b906800 msgr2=0x55b44b90a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6287fff640 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b906800 0x55b44b90a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.396+0000 7f6287fff640 1 -- 172.21.15.50:0/2538788864 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b44b903d10 con 0x55b44b8fbf40 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.397+0000 7f6287fff640 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 0x55b44b90a200 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f62780ee760 tx=0x7f627803b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ed4bf11c76a7e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.397+0000 7f62677fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.397+0000 7f62677fe640 1 -- 172.21.15.50:0/2538788864 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6278049050 con 0x55b44b8fbf40 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.397+0000 7f62677fe640 1 -- 172.21.15.50:0/2538788864 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f62780ee3e0 con 0x55b44b8fbf40 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.397+0000 7f6290ff4740 1 -- 172.21.15.50:0/2538788864 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f627c00f790 con 0x55b44b8fbf40 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.397+0000 7f62677fe640 1 -- 172.21.15.50:0/2538788864 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f62780c5190 con 0x55b44b8fbf40 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.397+0000 7f6290ff4740 10 client.17400 fetch_fsmap learned FSMap version 53 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.397+0000 7f6290ff4740 1 -- 172.21.15.50:0/2538788864 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55b44b97d2d0 con 0x55b44b8fbf40 2024-04-23T18:33:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.398+0000 7f62677fe640 1 -- 172.21.15.50:0/2538788864 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f627c00f790 con 0x55b44b8fbf40 2024-04-23T18:33:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.398+0000 7f62677fe640 1 -- 172.21.15.50:0/2538788864 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 53) ==== 8358+0+0 (secure 0 0 0) 0x7f62780bdce0 con 0x55b44b8fbf40 2024-04-23T18:33:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.398+0000 7f6290ff4740 10 client.17400 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:33:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.398+0000 7f6290ff4740 10 client.17400 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:33:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.398+0000 7f6290ff4740 20 client.17400 populate_metadata read hostname 'smithi050' 2024-04-23T18:33:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.398+0000 7f6290ff4740 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b44b97d8c0 0x55b44b97fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:33:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.398+0000 7f6290ff4740 4 client.17400 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:33:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.398+0000 7f6290ff4740 1 -- 172.21.15.50:0/2538788864 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b44b868200 con 0x55b44b97d8c0 2024-04-23T18:33:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.398+0000 7f62877fe640 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b44b97d8c0 0x55b44b97fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:33:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.398+0000 7f62677fe640 10 client.17400 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.398+0000 7f62877fe640 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b44b97d8c0 0x55b44b97fce0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d045e9157a7ec6d7 server_cookie=1dbaf90145781f15 in_seq=0 out_seq=0 2024-04-23T18:33:50.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:50 smithi177 ceph-mon[32308]: pgmap v484: 97 pgs: 97 active+clean; 763 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.5 MiB/s wr, 464 op/s 2024-04-23T18:33:50.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 66232 31.03 MB/sec execute 24 sec latency 113.031 ms 2024-04-23T18:33:50.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:50 smithi188 ceph-mon[32694]: pgmap v484: 97 pgs: 97 active+clean; 763 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.5 MiB/s wr, 464 op/s 2024-04-23T18:33:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f62677fe640 1 -- 172.21.15.50:0/2538788864 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b44b868200 con 0x55b44b97d8c0 2024-04-23T18:33:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f62677fe640 10 client.17400 handle_command_reply: tid=0 2024-04-23T18:33:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f6290ff4740 1 client.17400 shutdown 2024-04-23T18:33:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f6290ff4740 1 -- 172.21.15.50:0/2538788864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 msgr2=0x55b44b90a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f6290ff4740 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 0x55b44b90a200 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f62780ee760 tx=0x7f627803b4d0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f6290ff4740 1 -- 172.21.15.50:0/2538788864 shutdown_connections 2024-04-23T18:33:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f6290ff4740 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b44b97d8c0 0x55b44b97fce0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f6290ff4740 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b44b90ac80 0x55b44b97bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f6290ff4740 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b44b906800 0x55b44b90a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f62677fe640 0 client.17400 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:33:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f6290ff4740 1 --2- 172.21.15.50:0/2538788864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b44b8fbf40 0x55b44b90a200 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f6290ff4740 1 -- 172.21.15.50:0/2538788864 >> 172.21.15.50:0/2538788864 conn(0x55b44b8f7b30 msgr2=0x55b44b8fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f6290ff4740 1 -- 172.21.15.50:0/2538788864 shutdown_connections 2024-04-23T18:33:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f6290ff4740 1 -- 172.21.15.50:0/2538788864 wait complete. 2024-04-23T18:33:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.588+0000 7f6290ff4740 20 client.17400 trim_cache size 0 max 16384 2024-04-23T18:33:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:33:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:33:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:33:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:33:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.589+0000 7f628e2e1640 1 -- 172.21.15.50:0/2229665987 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f62700a3dc0 msgr2=0x7f62700a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.589+0000 7f628e2e1640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f62700a3dc0 0x7f62700a6280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f628000e720 tx=0x7f6280018000 comp rx=0 tx=0).stop 2024-04-23T18:33:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.589+0000 7f628e2e1640 1 -- 172.21.15.50:0/2229665987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62881021b0 msgr2=0x7f6288198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:33:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.589+0000 7f628e2e1640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62881021b0 0x7f6288198730 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f627802a910 tx=0x7f6278031a20 comp rx=0 tx=0).stop 2024-04-23T18:33:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.589+0000 7f628e2e1640 1 -- 172.21.15.50:0/2229665987 shutdown_connections 2024-04-23T18:33:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.589+0000 7f628e2e1640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f62700a3dc0 0x7f62700a6280 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.589+0000 7f628e2e1640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f628819cfa0 0x7f628819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.589+0000 7f628e2e1640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6288107280 0x7f6288198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.589+0000 7f628e2e1640 1 --2- 172.21.15.50:0/2229665987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62881021b0 0x7f6288198730 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:33:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.589+0000 7f628e2e1640 1 -- 172.21.15.50:0/2229665987 >> 172.21.15.50:0/2229665987 conn(0x7f62880fda10 msgr2=0x7f62880fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:33:50.585 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.589+0000 7f628e2e1640 1 -- 172.21.15.50:0/2229665987 shutdown_connections 2024-04-23T18:33:50.585 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:33:50.589+0000 7f628e2e1640 1 -- 172.21.15.50:0/2229665987 wait complete. 2024-04-23T18:33:50.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:50 smithi050 ceph-mon[29187]: pgmap v484: 97 pgs: 97 active+clean; 763 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.5 MiB/s wr, 464 op/s 2024-04-23T18:33:51.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 67085 31.95 MB/sec execute 25 sec latency 109.530 ms 2024-04-23T18:33:52.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:52 smithi050 ceph-mon[29187]: pgmap v485: 97 pgs: 97 active+clean; 763 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.5 MiB/s wr, 464 op/s 2024-04-23T18:33:52.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:52 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:33:52.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:52 smithi177 ceph-mon[32308]: pgmap v485: 97 pgs: 97 active+clean; 763 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.5 MiB/s wr, 464 op/s 2024-04-23T18:33:52.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:52 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:33:52.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 67611 32.18 MB/sec execute 26 sec latency 130.248 ms 2024-04-23T18:33:52.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:52 smithi188 ceph-mon[32694]: pgmap v485: 97 pgs: 97 active+clean; 763 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.5 MiB/s wr, 464 op/s 2024-04-23T18:33:52.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:52 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:33:53.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:53 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:33:53.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:53 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:33:53.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:53 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:33:53.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 67982 31.46 MB/sec execute 27 sec latency 48.693 ms 2024-04-23T18:33:53.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:53 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:33:53.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:53 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:33:53.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:53 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:33:53.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:53 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:33:53.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:53 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:33:53.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:53 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:33:54.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:54 smithi177 ceph-mon[32308]: pgmap v486: 97 pgs: 97 active+clean; 780 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.9 MiB/s wr, 733 op/s 2024-04-23T18:33:54.562 INFO:tasks.workunit.client.0.smithi050.stdout: 10 68550 31.24 MB/sec execute 28 sec latency 75.723 ms 2024-04-23T18:33:54.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:54 smithi188 ceph-mon[32694]: pgmap v486: 97 pgs: 97 active+clean; 780 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.9 MiB/s wr, 733 op/s 2024-04-23T18:33:54.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:54 smithi050 ceph-mon[29187]: pgmap v486: 97 pgs: 97 active+clean; 780 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.9 MiB/s wr, 733 op/s 2024-04-23T18:33:55.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 69451 31.41 MB/sec execute 29 sec latency 63.337 ms 2024-04-23T18:33:56.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:56 smithi177 ceph-mon[32308]: pgmap v487: 97 pgs: 97 active+clean; 780 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 5.2 MiB/s wr, 472 op/s 2024-04-23T18:33:56.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 70203 32.16 MB/sec execute 30 sec latency 407.223 ms 2024-04-23T18:33:56.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:56 smithi188 ceph-mon[32694]: pgmap v487: 97 pgs: 97 active+clean; 780 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 5.2 MiB/s wr, 472 op/s 2024-04-23T18:33:56.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:56 smithi050 ceph-mon[29187]: pgmap v487: 97 pgs: 97 active+clean; 780 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 5.2 MiB/s wr, 472 op/s 2024-04-23T18:33:57.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 70944 32.37 MB/sec execute 31 sec latency 162.440 ms 2024-04-23T18:33:58.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:33:58 smithi177 ceph-mon[32308]: pgmap v488: 97 pgs: 97 active+clean; 806 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.8 MiB/s wr, 666 op/s 2024-04-23T18:33:58.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 71629 32.53 MB/sec execute 32 sec latency 51.987 ms 2024-04-23T18:33:58.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:33:58 smithi188 ceph-mon[32694]: pgmap v488: 97 pgs: 97 active+clean; 806 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.8 MiB/s wr, 666 op/s 2024-04-23T18:33:58.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:33:58 smithi050 ceph-mon[29187]: pgmap v488: 97 pgs: 97 active+clean; 806 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.8 MiB/s wr, 666 op/s 2024-04-23T18:33:59.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 72276 32.70 MB/sec execute 33 sec latency 58.667 ms 2024-04-23T18:34:00.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:00 smithi177 ceph-mon[32308]: pgmap v489: 97 pgs: 97 active+clean; 806 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.0 MiB/s wr, 462 op/s 2024-04-23T18:34:00.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 72770 32.47 MB/sec execute 34 sec latency 223.287 ms 2024-04-23T18:34:00.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:00 smithi188 ceph-mon[32694]: pgmap v489: 97 pgs: 97 active+clean; 806 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.0 MiB/s wr, 462 op/s 2024-04-23T18:34:00.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:00 smithi050 ceph-mon[29187]: pgmap v489: 97 pgs: 97 active+clean; 806 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.0 MiB/s wr, 462 op/s 2024-04-23T18:34:01.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 73205 31.90 MB/sec execute 35 sec latency 88.045 ms 2024-04-23T18:34:02.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:02 smithi177 ceph-mon[32308]: pgmap v490: 97 pgs: 97 active+clean; 806 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.0 MiB/s wr, 462 op/s 2024-04-23T18:34:02.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 73588 31.73 MB/sec execute 36 sec latency 155.719 ms 2024-04-23T18:34:02.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:02 smithi188 ceph-mon[32694]: pgmap v490: 97 pgs: 97 active+clean; 806 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.0 MiB/s wr, 462 op/s 2024-04-23T18:34:02.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:02 smithi050 ceph-mon[29187]: pgmap v490: 97 pgs: 97 active+clean; 806 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.0 MiB/s wr, 462 op/s 2024-04-23T18:34:03.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 74141 31.83 MB/sec execute 37 sec latency 242.735 ms 2024-04-23T18:34:03.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.697+0000 7f1da28fa640 1 -- 172.21.15.50:0/3410498544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9c1072a0 msgr2=0x7f1d9c10ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.697+0000 7f1da28fa640 1 --2- 172.21.15.50:0/3410498544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9c1072a0 0x7f1d9c10ee70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009ca0 tx=0x7f1d9402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.697+0000 7f1da28fa640 1 -- 172.21.15.50:0/3410498544 shutdown_connections 2024-04-23T18:34:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.697+0000 7f1da28fa640 1 --2- 172.21.15.50:0/3410498544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9c1072a0 0x7f1d9c10ee70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.697+0000 7f1da28fa640 1 --2- 172.21.15.50:0/3410498544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d9c102b80 0x7f1d9c106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.697+0000 7f1da28fa640 1 --2- 172.21.15.50:0/3410498544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d9c1021d0 0x7f1d9c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.697+0000 7f1da28fa640 1 -- 172.21.15.50:0/3410498544 >> 172.21.15.50:0/3410498544 conn(0x7f1d9c0fda30 msgr2=0x7f1d9c0ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.697+0000 7f1da28fa640 1 -- 172.21.15.50:0/3410498544 shutdown_connections 2024-04-23T18:34:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.697+0000 7f1da28fa640 1 -- 172.21.15.50:0/3410498544 wait complete. 2024-04-23T18:34:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.698+0000 7f1da28fa640 1 Processor -- start 2024-04-23T18:34:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.698+0000 7f1da28fa640 1 -- start start 2024-04-23T18:34:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1da28fa640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d9c1021d0 0x7f1d9c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1da28fa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9c1072a0 0x7f1d9c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1da28fa640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d9c074cd0 0x7f1d9c1a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1da28fa640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c112260 con 0x7f1d9c1021d0 2024-04-23T18:34:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1da28fa640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c1120e0 con 0x7f1d9c1072a0 2024-04-23T18:34:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1da28fa640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c1123e0 con 0x7f1d9c074cd0 2024-04-23T18:34:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d9c1021d0 0x7f1d9c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1d9b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9c1072a0 0x7f1d9c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d9c1021d0 0x7f1d9c074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54886/0 (socket says 172.21.15.50:54886) 2024-04-23T18:34:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1d9bfff640 1 -- 172.21.15.50:0/2496214426 learned_addr learned my addr 172.21.15.50:0/2496214426 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1da0e97640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d9c074cd0 0x7f1d9c1a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1d9b7fe640 1 -- 172.21.15.50:0/2496214426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d9c074cd0 msgr2=0x7f1d9c1a6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1d9b7fe640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d9c074cd0 0x7f1d9c1a6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1d9b7fe640 1 -- 172.21.15.50:0/2496214426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d9c1021d0 msgr2=0x7f1d9c074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1d9b7fe640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d9c1021d0 0x7f1d9c074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1d9b7fe640 1 -- 172.21.15.50:0/2496214426 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d9c1a6f70 con 0x7f1d9c1072a0 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1d9bfff640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d9c1021d0 0x7f1d9c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1d9b7fe640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9c1072a0 0x7f1d9c074790 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1d9000e9c0 tx=0x7f1d9000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.699+0000 7f1d997fa640 1 -- 172.21.15.50:0/2496214426 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9000cd80 con 0x7f1d9c1072a0 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.700+0000 7f1d997fa640 1 -- 172.21.15.50:0/2496214426 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d900035f0 con 0x7f1d9c1072a0 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.700+0000 7f1da28fa640 1 -- 172.21.15.50:0/2496214426 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d9c1a7260 con 0x7f1d9c1072a0 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.700+0000 7f1d997fa640 1 -- 172.21.15.50:0/2496214426 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d90010690 con 0x7f1d9c1072a0 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.700+0000 7f1da28fa640 1 -- 172.21.15.50:0/2496214426 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d9c1a7800 con 0x7f1d9c1072a0 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.702+0000 7f1d997fa640 1 -- 172.21.15.50:0/2496214426 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1d900037a0 con 0x7f1d9c1072a0 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.702+0000 7f1da5613740 1 Processor -- start 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.702+0000 7f1da5613740 1 -- start start 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1da5613740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688ef4620 0x55b688ef4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1da5613740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 0x55b688eff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1da5613740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688eff800 0x55b688f01bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1da5613740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b688e59970 con 0x55b688ef4f40 2024-04-23T18:34:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1da5613740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b688df64a0 con 0x55b688eff800 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1da5613740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b688e55930 con 0x55b688ef4620 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1d997fa640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1d840a3e70 0x7f1d840a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1da0e97640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688ef4620 0x55b688ef4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 0x55b688eff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1d9b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688eff800 0x55b688f01bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1da0e97640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688ef4620 0x55b688ef4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:55708/0 (socket says 172.21.15.50:55708) 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1da0e97640 1 -- 172.21.15.50:0/3226217154 learned_addr learned my addr 172.21.15.50:0/3226217154 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1da0e97640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1d840a3e70 0x7f1d840a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1d997fa640 1 -- 172.21.15.50:0/2496214426 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f1d900d09a0 con 0x7f1d9c1072a0 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1d9bfff640 1 -- 172.21.15.50:0/3226217154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688ef4620 msgr2=0x55b688ef4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1d9bfff640 1 --2- 172.21.15.50:0/3226217154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688ef4620 0x55b688ef4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1d9bfff640 1 -- 172.21.15.50:0/3226217154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688eff800 msgr2=0x55b688f01bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1d9bfff640 1 --2- 172.21.15.50:0/3226217154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688eff800 0x55b688f01bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1d9bfff640 1 -- 172.21.15.50:0/3226217154 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b688b6c210 con 0x55b688ef4f40 2024-04-23T18:34:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1da0e97640 1 --2- 172.21.15.50:0/3226217154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688ef4620 0x55b688ef4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1d9bfff640 1 --2- 172.21.15.50:0/3226217154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 0x55b688eff2c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c011510 tx=0x7f1d8c0119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8655d2f0c32c53 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.703+0000 7f1da0e97640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1d840a3e70 0x7f1d840a6330 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d94005fe0 tx=0x7f1d94005d80 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.704+0000 7f1d827fc640 1 -- 172.21.15.50:0/3226217154 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c01e070 con 0x55b688ef4f40 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.704+0000 7f1d827fc640 1 -- 172.21.15.50:0/3226217154 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d8c00fa70 con 0x55b688ef4f40 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.704+0000 7f1d827fc640 1 -- 172.21.15.50:0/3226217154 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c019b60 con 0x55b688ef4f40 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.704+0000 7f1da5613740 1 -- 172.21.15.50:0/3226217154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 msgr2=0x55b688eff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.704+0000 7f1da5613740 1 --2- 172.21.15.50:0/3226217154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 0x55b688eff2c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c011510 tx=0x7f1d8c0119d0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.704+0000 7f1da5613740 1 -- 172.21.15.50:0/3226217154 shutdown_connections 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.704+0000 7f1da5613740 1 --2- 172.21.15.50:0/3226217154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688eff800 0x55b688f01bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.704+0000 7f1da5613740 1 --2- 172.21.15.50:0/3226217154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 0x55b688eff2c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.704+0000 7f1da5613740 1 --2- 172.21.15.50:0/3226217154 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688ef4620 0x55b688ef4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.704+0000 7f1da5613740 1 -- 172.21.15.50:0/3226217154 >> 172.21.15.50:0/3226217154 conn(0x55b688ef0b30 msgr2=0x55b688ef0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.704+0000 7f1da5613740 1 -- 172.21.15.50:0/3226217154 shutdown_connections 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.704+0000 7f1da5613740 1 -- 172.21.15.50:0/3226217154 wait complete. 2024-04-23T18:34:03.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.705+0000 7f1da5613740 1 -- start start 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.705+0000 7f1da5613740 1 Processor -- start 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.705+0000 7f1da5613740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 0x55b688eff1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.705+0000 7f1da5613740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4f40 0x55b688efc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.705+0000 7f1da5613740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 0x55b688efca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.705+0000 7f1da5613740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b688f02430 con 0x55b688ef4620 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.705+0000 7f1da5613740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b688df40b0 con 0x55b688ef4f40 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.705+0000 7f1da5613740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b688f025b0 con 0x55b688eff800 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1da5613740 10 client.0 _ll_register_callbacks cb 0x55b688de1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 0x55b688eff1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d9b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4f40 0x55b688efc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 0x55b688eff1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54904/0 (socket says 172.21.15.50:54904) 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d9bfff640 1 -- 172.21.15.50:0/3367020927 learned_addr learned my addr 172.21.15.50:0/3367020927 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1da0e97640 1 --2- 172.21.15.50:0/3367020927 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 0x55b688efca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d9bfff640 1 -- 172.21.15.50:0/3367020927 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 msgr2=0x55b688efca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d9bfff640 1 --2- 172.21.15.50:0/3367020927 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 0x55b688efca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d9bfff640 1 -- 172.21.15.50:0/3367020927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4f40 msgr2=0x55b688efc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d9bfff640 1 --2- 172.21.15.50:0/3367020927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4f40 0x55b688efc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d9bfff640 1 -- 172.21.15.50:0/3367020927 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b688c38620 con 0x55b688ef4620 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d9bfff640 1 --2- 172.21.15.50:0/3367020927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 0x55b688eff1b0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c01c040 tx=0x7f1d8c019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b992727b953733 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d7b7fe640 1 -- 172.21.15.50:0/3367020927 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c01e040 con 0x55b688ef4620 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1d7b7fe640 1 -- 172.21.15.50:0/3367020927 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f1d8c00f680 con 0x55b688ef4620 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.706+0000 7f1da5613740 1 -- 172.21.15.50:0/3367020927 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d8c00fa70 con 0x55b688ef4620 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.707+0000 7f1d7b7fe640 1 -- 172.21.15.50:0/3367020927 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f1d8c032050 con 0x55b688ef4620 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.707+0000 7f1da5613740 10 client.17412 fetch_fsmap learned FSMap version 53 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.707+0000 7f1da5613740 1 -- 172.21.15.50:0/3367020927 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55b688f0e3a0 con 0x55b688ef4620 2024-04-23T18:34:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.707+0000 7f1d7b7fe640 1 -- 172.21.15.50:0/3367020927 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d8c00fa70 con 0x55b688ef4620 2024-04-23T18:34:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.707+0000 7f1d7b7fe640 1 -- 172.21.15.50:0/3367020927 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 53) ==== 8358+0+0 (secure 0 0 0) 0x7f1d8c021030 con 0x55b688ef4620 2024-04-23T18:34:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.707+0000 7f1da5613740 10 client.17412 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:34:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.707+0000 7f1da5613740 10 client.17412 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:34:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.707+0000 7f1da5613740 20 client.17412 populate_metadata read hostname 'smithi050' 2024-04-23T18:34:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.707+0000 7f1da5613740 1 --2- 172.21.15.50:0/3367020927 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b688f0f7a0 0x55b688f11b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.707+0000 7f1da5613740 4 client.17412 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:34:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.707+0000 7f1da5613740 1 -- 172.21.15.50:0/3367020927 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b688e67390 con 0x55b688f0f7a0 2024-04-23T18:34:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.707+0000 7f1d9b7fe640 1 --2- 172.21.15.50:0/3367020927 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b688f0f7a0 0x55b688f11b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.708+0000 7f1d7b7fe640 10 client.17412 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.708+0000 7f1d9b7fe640 1 --2- 172.21.15.50:0/3367020927 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b688f0f7a0 0x55b688f11b90 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=259c6e836bc5e664 server_cookie=a9af4ee4420e3526 in_seq=0 out_seq=0 2024-04-23T18:34:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.709+0000 7f1d7b7fe640 1 -- 172.21.15.50:0/3367020927 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b688e67390 con 0x55b688f0f7a0 2024-04-23T18:34:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.709+0000 7f1d7b7fe640 10 client.17412 handle_command_reply: tid=0 2024-04-23T18:34:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.709+0000 7f1da5613740 1 client.17412 shutdown 2024-04-23T18:34:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.710+0000 7f1da5613740 1 -- 172.21.15.50:0/3367020927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 msgr2=0x55b688eff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.710+0000 7f1da5613740 1 --2- 172.21.15.50:0/3367020927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 0x55b688eff1b0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c01c040 tx=0x7f1d8c019640 comp rx=0 tx=0).stop 2024-04-23T18:34:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.710+0000 7f1da5613740 1 -- 172.21.15.50:0/3367020927 shutdown_connections 2024-04-23T18:34:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.710+0000 7f1da5613740 1 --2- 172.21.15.50:0/3367020927 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b688f0f7a0 0x55b688f11b90 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.710+0000 7f1da5613740 1 --2- 172.21.15.50:0/3367020927 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 0x55b688efca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.710+0000 7f1da5613740 1 --2- 172.21.15.50:0/3367020927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4f40 0x55b688efc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.710+0000 7f1da5613740 1 --2- 172.21.15.50:0/3367020927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 0x55b688eff1b0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.710+0000 7f1d7b7fe640 0 client.17412 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.710+0000 7f1da5613740 1 -- 172.21.15.50:0/3367020927 >> 172.21.15.50:0/3367020927 conn(0x55b688ef0b30 msgr2=0x55b688ef6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.710+0000 7f1da5613740 1 -- 172.21.15.50:0/3367020927 shutdown_connections 2024-04-23T18:34:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.710+0000 7f1da5613740 1 -- 172.21.15.50:0/3367020927 wait complete. 2024-04-23T18:34:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.710+0000 7f1da5613740 20 client.17412 trim_cache size 0 max 16384 2024-04-23T18:34:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1da5613740 1 Processor -- start 2024-04-23T18:34:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1da5613740 1 -- start start 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1da5613740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4620 0x55b688f0b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1da5613740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 0x55b688efc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1da5613740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 0x55b688f17320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1da5613740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c1b26d0 con 0x55b688ef4f40 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1da5613740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c1aff10 con 0x55b688ef4620 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1da5613740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c1add80 con 0x55b688eff800 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1d9b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 0x55b688efc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4620 0x55b688f0b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1d9b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 0x55b688efc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54908/0 (socket says 172.21.15.50:54908) 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4620 0x55b688f0b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:46950/0 (socket says 172.21.15.50:46950) 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1d9b7fe640 1 -- 172.21.15.50:0/2844585177 learned_addr learned my addr 172.21.15.50:0/2844585177 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1da0e97640 1 --2- 172.21.15.50:0/2844585177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 0x55b688f17320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1d9b7fe640 1 -- 172.21.15.50:0/2844585177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 msgr2=0x55b688f17320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1d9b7fe640 1 --2- 172.21.15.50:0/2844585177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 0x55b688f17320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1d9b7fe640 1 -- 172.21.15.50:0/2844585177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4620 msgr2=0x55b688f0b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1d9b7fe640 1 --2- 172.21.15.50:0/2844585177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4620 0x55b688f0b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1d9b7fe640 1 -- 172.21.15.50:0/2844585177 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b688ef4d70 con 0x55b688ef4f40 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.730+0000 7f1d9bfff640 1 --2- 172.21.15.50:0/2844585177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4620 0x55b688f0b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.731+0000 7f1d9b7fe640 1 --2- 172.21.15.50:0/2844585177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 0x55b688efc440 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f1d9001f000 tx=0x7f1d9006f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5aa6849b6a2caed server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.731+0000 7f1d81ffb640 1 -- 172.21.15.50:0/2844585177 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d900a0ce0 con 0x55b688ef4f40 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.731+0000 7f1d81ffb640 1 -- 172.21.15.50:0/2844585177 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d900a21f0 con 0x55b688ef4f40 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.731+0000 7f1d81ffb640 1 -- 172.21.15.50:0/2844585177 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d900ce360 con 0x55b688ef4f40 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.732+0000 7f1da5613740 1 -- 172.21.15.50:0/2844585177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 msgr2=0x55b688efc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.732+0000 7f1da5613740 1 --2- 172.21.15.50:0/2844585177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 0x55b688efc440 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f1d9001f000 tx=0x7f1d9006f420 comp rx=0 tx=0).stop 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.732+0000 7f1da5613740 1 -- 172.21.15.50:0/2844585177 shutdown_connections 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.732+0000 7f1da5613740 1 --2- 172.21.15.50:0/2844585177 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 0x55b688f17320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.732+0000 7f1da5613740 1 --2- 172.21.15.50:0/2844585177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4f40 0x55b688efc440 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.732+0000 7f1da5613740 1 --2- 172.21.15.50:0/2844585177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4620 0x55b688f0b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.732+0000 7f1da5613740 1 -- 172.21.15.50:0/2844585177 >> 172.21.15.50:0/2844585177 conn(0x55b688ef0b30 msgr2=0x55b688efbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:03.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.732+0000 7f1da5613740 1 -- 172.21.15.50:0/2844585177 shutdown_connections 2024-04-23T18:34:03.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.732+0000 7f1da5613740 1 -- 172.21.15.50:0/2844585177 wait complete. 2024-04-23T18:34:03.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.732+0000 7f1da5613740 1 -- start start 2024-04-23T18:34:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1da5613740 1 Processor -- start 2024-04-23T18:34:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1da5613740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 0x55b688f07d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1da5613740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4f40 0x55b688f050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1da5613740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 0x55b688f057e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1da5613740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c1ae3d0 con 0x55b688ef4620 2024-04-23T18:34:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1da5613740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c1af840 con 0x55b688ef4f40 2024-04-23T18:34:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1da5613740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c1aeaa0 con 0x55b688eff800 2024-04-23T18:34:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1da5613740 10 client.0 _ll_register_callbacks cb 0x7f1d8c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:34:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 0x55b688f07d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 0x55b688f07d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54910/0 (socket says 172.21.15.50:54910) 2024-04-23T18:34:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1d9bfff640 1 -- 172.21.15.50:0/4008583338 learned_addr learned my addr 172.21.15.50:0/4008583338 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1da0e97640 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 0x55b688f057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1d9b7fe640 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4f40 0x55b688f050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1d9bfff640 1 -- 172.21.15.50:0/4008583338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 msgr2=0x55b688f057e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1d9bfff640 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 0x55b688f057e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1d9bfff640 1 -- 172.21.15.50:0/4008583338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4f40 msgr2=0x55b688f050a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1d9bfff640 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4f40 0x55b688f050a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.733+0000 7f1d9bfff640 1 -- 172.21.15.50:0/4008583338 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b688f74ba0 con 0x55b688ef4620 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.734+0000 7f1d9bfff640 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 0x55b688f07d10 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c011510 tx=0x7f1d8c004930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458ab30093b24d93 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.734+0000 7f1d7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.734+0000 7f1d7b7fe640 1 -- 172.21.15.50:0/4008583338 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c03b060 con 0x55b688ef4620 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.734+0000 7f1d7b7fe640 1 -- 172.21.15.50:0/4008583338 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8542+0+0 (secure 0 0 0) 0x7f1d8c00f680 con 0x55b688ef4620 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.734+0000 7f1da5613740 1 -- 172.21.15.50:0/4008583338 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d900a21f0 con 0x55b688ef4620 2024-04-23T18:34:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.734+0000 7f1d7b7fe640 1 -- 172.21.15.50:0/4008583338 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f1d8c01e040 con 0x55b688ef4620 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.734+0000 7f1da5613740 10 client.17418 fetch_fsmap learned FSMap version 53 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.734+0000 7f1da5613740 1 -- 172.21.15.50:0/4008583338 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55b688f75cf0 con 0x55b688ef4620 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.734+0000 7f1d7b7fe640 1 -- 172.21.15.50:0/4008583338 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d900a21f0 con 0x55b688ef4620 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.734+0000 7f1d7b7fe640 1 -- 172.21.15.50:0/4008583338 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 53) ==== 8358+0+0 (secure 0 0 0) 0x7f1d8c021030 con 0x55b688ef4620 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.735+0000 7f1da5613740 10 client.17418 fetch_fsmap finished waiting for FSMap version 53 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.735+0000 7f1da5613740 10 client.17418 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.735+0000 7f1da5613740 20 client.17418 populate_metadata read hostname 'smithi050' 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.735+0000 7f1da5613740 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b688f76500 0x55b688f78920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.735+0000 7f1da5613740 4 client.17418 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.735+0000 7f1da5613740 1 -- 172.21.15.50:0/4008583338 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b688e61200 con 0x55b688f76500 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.735+0000 7f1d9b7fe640 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b688f76500 0x55b688f78920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.735+0000 7f1d9b7fe640 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b688f76500 0x55b688f78920 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab8eea3b392ea338 server_cookie=6acbb9cd1ef993c in_seq=0 out_seq=0 2024-04-23T18:34:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.735+0000 7f1d7b7fe640 10 client.17418 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.941+0000 7f1d7b7fe640 1 -- 172.21.15.50:0/4008583338 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b688e61200 con 0x55b688f76500 2024-04-23T18:34:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.941+0000 7f1d7b7fe640 10 client.17418 handle_command_reply: tid=0 2024-04-23T18:34:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.941+0000 7f1da5613740 1 client.17418 shutdown 2024-04-23T18:34:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.941+0000 7f1da5613740 1 -- 172.21.15.50:0/4008583338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 msgr2=0x55b688f07d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.941+0000 7f1da5613740 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 0x55b688f07d10 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c011510 tx=0x7f1d8c004930 comp rx=0 tx=0).stop 2024-04-23T18:34:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.941+0000 7f1da5613740 1 -- 172.21.15.50:0/4008583338 shutdown_connections 2024-04-23T18:34:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.941+0000 7f1da5613740 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b688f76500 0x55b688f78920 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1d7b7fe640 0 client.17418 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.941+0000 7f1da5613740 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b688eff800 0x55b688f057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da5613740 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b688ef4f40 0x55b688f050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da5613740 1 --2- 172.21.15.50:0/4008583338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b688ef4620 0x55b688f07d10 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da5613740 1 -- 172.21.15.50:0/4008583338 >> 172.21.15.50:0/4008583338 conn(0x55b688ef0b30 msgr2=0x55b688f0a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da5613740 1 -- 172.21.15.50:0/4008583338 shutdown_connections 2024-04-23T18:34:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da5613740 1 -- 172.21.15.50:0/4008583338 wait complete. 2024-04-23T18:34:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da5613740 20 client.17418 trim_cache size 0 max 16384 2024-04-23T18:34:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:34:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:34:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:34:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:34:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da28fa640 1 -- 172.21.15.50:0/2496214426 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1d840a3e70 msgr2=0x7f1d840a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da28fa640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1d840a3e70 0x7f1d840a6330 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d94005fe0 tx=0x7f1d94005d80 comp rx=0 tx=0).stop 2024-04-23T18:34:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da28fa640 1 -- 172.21.15.50:0/2496214426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9c1072a0 msgr2=0x7f1d9c074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da28fa640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9c1072a0 0x7f1d9c074790 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1d9000e9c0 tx=0x7f1d9000ee80 comp rx=0 tx=0).stop 2024-04-23T18:34:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da28fa640 1 -- 172.21.15.50:0/2496214426 shutdown_connections 2024-04-23T18:34:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da28fa640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f1d840a3e70 0x7f1d840a6330 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da28fa640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d9c074cd0 0x7f1d9c1a6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da28fa640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9c1072a0 0x7f1d9c074790 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da28fa640 1 --2- 172.21.15.50:0/2496214426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d9c1021d0 0x7f1d9c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.942+0000 7f1da28fa640 1 -- 172.21.15.50:0/2496214426 >> 172.21.15.50:0/2496214426 conn(0x7f1d9c0fda30 msgr2=0x7f1d9c0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.943+0000 7f1da28fa640 1 -- 172.21.15.50:0/2496214426 shutdown_connections 2024-04-23T18:34:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:03.943+0000 7f1da28fa640 1 -- 172.21.15.50:0/2496214426 wait complete. 2024-04-23T18:34:04.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:04 smithi177 ceph-mon[32308]: pgmap v491: 97 pgs: 97 active+clean; 822 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.9 MiB/s wr, 755 op/s 2024-04-23T18:34:04.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 74591 31.46 MB/sec execute 38 sec latency 202.313 ms 2024-04-23T18:34:04.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:04 smithi188 ceph-mon[32694]: pgmap v491: 97 pgs: 97 active+clean; 822 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.9 MiB/s wr, 755 op/s 2024-04-23T18:34:04.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:04 smithi050 ceph-mon[29187]: pgmap v491: 97 pgs: 97 active+clean; 822 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.9 MiB/s wr, 755 op/s 2024-04-23T18:34:05.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:05 smithi177 ceph-mon[32308]: osdmap e113: 12 total, 12 up, 12 in 2024-04-23T18:34:05.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 74969 31.13 MB/sec execute 39 sec latency 144.790 ms 2024-04-23T18:34:05.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:05 smithi188 ceph-mon[32694]: osdmap e113: 12 total, 12 up, 12 in 2024-04-23T18:34:05.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:05 smithi050 ceph-mon[29187]: osdmap e113: 12 total, 12 up, 12 in 2024-04-23T18:34:06.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:06 smithi177 ceph-mon[32308]: pgmap v493: 97 pgs: 97 active+clean; 822 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 585 op/s 2024-04-23T18:34:06.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 75608 31.45 MB/sec execute 40 sec latency 80.269 ms 2024-04-23T18:34:06.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:06 smithi188 ceph-mon[32694]: pgmap v493: 97 pgs: 97 active+clean; 822 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 585 op/s 2024-04-23T18:34:06.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:06 smithi050 ceph-mon[29187]: pgmap v493: 97 pgs: 97 active+clean; 822 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 585 op/s 2024-04-23T18:34:07.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:07 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:34:07.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 76225 31.39 MB/sec execute 41 sec latency 67.587 ms 2024-04-23T18:34:07.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:07 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:34:07.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:07 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:34:08.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:08 smithi177 ceph-mon[32308]: pgmap v494: 97 pgs: 97 active+clean; 836 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 824 op/s 2024-04-23T18:34:08.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 76691 31.18 MB/sec execute 42 sec latency 226.124 ms 2024-04-23T18:34:08.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:08 smithi188 ceph-mon[32694]: pgmap v494: 97 pgs: 97 active+clean; 836 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 824 op/s 2024-04-23T18:34:08.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:08 smithi050 ceph-mon[29187]: pgmap v494: 97 pgs: 97 active+clean; 836 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 824 op/s 2024-04-23T18:34:09.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 77380 31.42 MB/sec execute 43 sec latency 68.338 ms 2024-04-23T18:34:10.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:10 smithi177 ceph-mon[32308]: pgmap v495: 97 pgs: 97 active+clean; 836 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 824 op/s 2024-04-23T18:34:10.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 78061 31.59 MB/sec execute 44 sec latency 44.654 ms 2024-04-23T18:34:10.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:10 smithi188 ceph-mon[32694]: pgmap v495: 97 pgs: 97 active+clean; 836 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 824 op/s 2024-04-23T18:34:10.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:10 smithi050 ceph-mon[29187]: pgmap v495: 97 pgs: 97 active+clean; 836 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 824 op/s 2024-04-23T18:34:11.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 78622 31.49 MB/sec execute 45 sec latency 57.273 ms 2024-04-23T18:34:12.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:12 smithi050 ceph-mon[29187]: pgmap v496: 97 pgs: 97 active+clean; 836 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 824 op/s 2024-04-23T18:34:12.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:12 smithi050 ceph-mon[29187]: osdmap e114: 12 total, 12 up, 12 in 2024-04-23T18:34:12.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:12 smithi177 ceph-mon[32308]: pgmap v496: 97 pgs: 97 active+clean; 836 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 824 op/s 2024-04-23T18:34:12.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:12 smithi177 ceph-mon[32308]: osdmap e114: 12 total, 12 up, 12 in 2024-04-23T18:34:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 79301 31.60 MB/sec execute 46 sec latency 72.522 ms 2024-04-23T18:34:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:12 smithi188 ceph-mon[32694]: pgmap v496: 97 pgs: 97 active+clean; 836 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 824 op/s 2024-04-23T18:34:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:12 smithi188 ceph-mon[32694]: osdmap e114: 12 total, 12 up, 12 in 2024-04-23T18:34:13.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 79897 31.45 MB/sec execute 47 sec latency 254.922 ms 2024-04-23T18:34:14.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:14 smithi050 ceph-mon[29187]: pgmap v498: 97 pgs: 97 active+clean; 863 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 930 op/s 2024-04-23T18:34:14.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:14 smithi177 ceph-mon[32308]: pgmap v498: 97 pgs: 97 active+clean; 863 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 930 op/s 2024-04-23T18:34:14.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 80500 31.51 MB/sec execute 48 sec latency 192.745 ms 2024-04-23T18:34:14.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:14 smithi188 ceph-mon[32694]: pgmap v498: 97 pgs: 97 active+clean; 863 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 930 op/s 2024-04-23T18:34:15.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 81009 31.57 MB/sec execute 49 sec latency 142.793 ms 2024-04-23T18:34:16.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:16 smithi050 ceph-mon[29187]: pgmap v499: 97 pgs: 97 active+clean; 863 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 792 op/s 2024-04-23T18:34:16.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:16 smithi177 ceph-mon[32308]: pgmap v499: 97 pgs: 97 active+clean; 863 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 792 op/s 2024-04-23T18:34:16.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 81521 31.50 MB/sec execute 50 sec latency 132.794 ms 2024-04-23T18:34:16.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:16 smithi188 ceph-mon[32694]: pgmap v499: 97 pgs: 97 active+clean; 863 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 792 op/s 2024-04-23T18:34:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.047+0000 7f60bc37d640 1 -- 172.21.15.50:0/445204902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b41031e0 msgr2=0x7f60b41035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.047+0000 7f60bc37d640 1 --2- 172.21.15.50:0/445204902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b41031e0 0x7f60b41035c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009900 tx=0x7f60b002a510 comp rx=0 tx=0).stop 2024-04-23T18:34:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.047+0000 7f60bc37d640 1 -- 172.21.15.50:0/445204902 shutdown_connections 2024-04-23T18:34:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.047+0000 7f60bc37d640 1 --2- 172.21.15.50:0/445204902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60b410c520 0x7f60b410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.047+0000 7f60bc37d640 1 --2- 172.21.15.50:0/445204902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60b4103b00 0x7f60b410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.047+0000 7f60bc37d640 1 --2- 172.21.15.50:0/445204902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b41031e0 0x7f60b41035c0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.047+0000 7f60bc37d640 1 -- 172.21.15.50:0/445204902 >> 172.21.15.50:0/445204902 conn(0x7f60b4077fd0 msgr2=0x7f60b40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.047+0000 7f60bc37d640 1 -- 172.21.15.50:0/445204902 shutdown_connections 2024-04-23T18:34:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.047+0000 7f60bc37d640 1 -- 172.21.15.50:0/445204902 wait complete. 2024-04-23T18:34:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.048+0000 7f60bc37d640 1 Processor -- start 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.048+0000 7f60bc37d640 1 -- start start 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.048+0000 7f60bc37d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60b41031e0 0x7f60b41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60bc37d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4103b00 0x7f60b4198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60bc37d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60b410c520 0x7f60b419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60bc37d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f60b4112200 con 0x7f60b4103b00 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60bc37d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f60b4112080 con 0x7f60b41031e0 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60bc37d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f60b4112380 con 0x7f60b410c520 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60b9918640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4103b00 0x7f60b4198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60ba119640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60b41031e0 0x7f60b41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60ba119640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60b41031e0 0x7f60b41986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:49920/0 (socket says 172.21.15.50:49920) 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60b9918640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4103b00 0x7f60b4198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59258/0 (socket says 172.21.15.50:59258) 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60ba119640 1 -- 172.21.15.50:0/1852129435 learned_addr learned my addr 172.21.15.50:0/1852129435 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60ba91a640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60b410c520 0x7f60b419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60ba119640 1 -- 172.21.15.50:0/1852129435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60b410c520 msgr2=0x7f60b419cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60ba119640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60b410c520 0x7f60b419cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60ba119640 1 -- 172.21.15.50:0/1852129435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4103b00 msgr2=0x7f60b4198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60ba119640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4103b00 0x7f60b4198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60ba119640 1 -- 172.21.15.50:0/1852129435 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b419d6d0 con 0x7f60b41031e0 2024-04-23T18:34:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60ba91a640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60b410c520 0x7f60b419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:34:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60ba119640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60b41031e0 0x7f60b41986e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f60b0005ab0 tx=0x7f60b0003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.049+0000 7f60ab7fe640 1 -- 172.21.15.50:0/1852129435 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b003f070 con 0x7f60b41031e0 2024-04-23T18:34:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.050+0000 7f60ab7fe640 1 -- 172.21.15.50:0/1852129435 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b0031b60 con 0x7f60b41031e0 2024-04-23T18:34:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.050+0000 7f60ab7fe640 1 -- 172.21.15.50:0/1852129435 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b0035030 con 0x7f60b41031e0 2024-04-23T18:34:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.050+0000 7f60bc37d640 1 -- 172.21.15.50:0/1852129435 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b419d960 con 0x7f60b41031e0 2024-04-23T18:34:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.050+0000 7f60bc37d640 1 -- 172.21.15.50:0/1852129435 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b41a5250 con 0x7f60b41031e0 2024-04-23T18:34:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.053+0000 7f60ab7fe640 1 -- 172.21.15.50:0/1852129435 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f60b0042070 con 0x7f60b41031e0 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.053+0000 7f60bf175740 1 Processor -- start 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.053+0000 7f60bf175740 1 -- start start 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.053+0000 7f60bf175740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be1065a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.053+0000 7f60bf175740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 0x557be10702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.053+0000 7f60bf175740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be1072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.053+0000 7f60bf175740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557be0fca970 con 0x557be1065f40 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.053+0000 7f60bf175740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557be0f674a0 con 0x557be1070800 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.053+0000 7f60bf175740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557be0fc6930 con 0x557be1065620 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ab7fe640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f60a00a3e40 0x7f60a00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ba119640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 0x557be10702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ba119640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 0x557be10702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59268/0 (socket says 172.21.15.50:59268) 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ba119640 1 -- 172.21.15.50:0/2030791256 learned_addr learned my addr 172.21.15.50:0/2030791256 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ba91a640 1 --2- 172.21.15.50:0/2030791256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be1065a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60b9918640 1 --2- 172.21.15.50:0/2030791256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be1072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ba91a640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f60a00a3e40 0x7f60a00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ab7fe640 1 -- 172.21.15.50:0/1852129435 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f60b00f0f00 con 0x7f60b41031e0 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ba119640 1 -- 172.21.15.50:0/2030791256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 msgr2=0x557be1065a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ba119640 1 --2- 172.21.15.50:0/2030791256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be1065a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ba119640 1 -- 172.21.15.50:0/2030791256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 msgr2=0x557be1072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ba119640 1 --2- 172.21.15.50:0/2030791256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be1072bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ba119640 1 -- 172.21.15.50:0/2030791256 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557be0cdd210 con 0x557be1065f40 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60b9918640 1 --2- 172.21.15.50:0/2030791256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be1072bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ba91a640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f60a00a3e40 0x7f60a00a6300 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009fd0 tx=0x7f60ac009500 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60ba119640 1 --2- 172.21.15.50:0/2030791256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 0x557be10702c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f60b0005ec0 tx=0x7f60b00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c0353c8cd12b4e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60a8bfe640 1 -- 172.21.15.50:0/2030791256 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b0049070 con 0x557be1065f40 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60a8bfe640 1 -- 172.21.15.50:0/2030791256 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b00bce90 con 0x557be1065f40 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.054+0000 7f60a8bfe640 1 -- 172.21.15.50:0/2030791256 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b00b9500 con 0x557be1065f40 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.055+0000 7f60bf175740 1 -- 172.21.15.50:0/2030791256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 msgr2=0x557be10702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.055+0000 7f60bf175740 1 --2- 172.21.15.50:0/2030791256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 0x557be10702c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f60b0005ec0 tx=0x7f60b00045e0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.055+0000 7f60bf175740 1 -- 172.21.15.50:0/2030791256 shutdown_connections 2024-04-23T18:34:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.055+0000 7f60bf175740 1 --2- 172.21.15.50:0/2030791256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be1072bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.055+0000 7f60bf175740 1 --2- 172.21.15.50:0/2030791256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 0x557be10702c0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.055+0000 7f60bf175740 1 --2- 172.21.15.50:0/2030791256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be1065a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.055+0000 7f60bf175740 1 -- 172.21.15.50:0/2030791256 >> 172.21.15.50:0/2030791256 conn(0x557be1061b30 msgr2=0x557be1061f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.055+0000 7f60bf175740 1 -- 172.21.15.50:0/2030791256 shutdown_connections 2024-04-23T18:34:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.055+0000 7f60bf175740 1 -- 172.21.15.50:0/2030791256 wait complete. 2024-04-23T18:34:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60bf175740 1 -- start start 2024-04-23T18:34:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60bf175740 1 Processor -- start 2024-04-23T18:34:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60bf175740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065620 0x557be1070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60bf175740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065f40 0x557be106d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60bf175740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be106d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60bf175740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x557be10731f0 con 0x557be1065620 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60bf175740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557be0f650b0 con 0x557be1070800 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60bf175740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x557be1073370 con 0x557be1065f40 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60bf175740 10 client.0 _ll_register_callbacks cb 0x557be0f52d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60ba119640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065620 0x557be1070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60ba119640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065620 0x557be1070150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59282/0 (socket says 172.21.15.50:59282) 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60ba119640 1 -- 172.21.15.50:0/4271511763 learned_addr learned my addr 172.21.15.50:0/4271511763 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60b9918640 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065f40 0x557be106d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60ba91a640 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be106d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60ba119640 1 -- 172.21.15.50:0/4271511763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065f40 msgr2=0x557be106d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60ba119640 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065f40 0x557be106d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60ba119640 1 -- 172.21.15.50:0/4271511763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 msgr2=0x557be106d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60ba119640 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be106d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.056+0000 7f60ba119640 1 -- 172.21.15.50:0/4271511763 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557be0da9620 con 0x557be1065620 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f60ba119640 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065620 0x557be1070150 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f60b00ee6f0 tx=0x7f60b00c1380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155dc50728ede097 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f609dffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f609dffb640 1 -- 172.21.15.50:0/4271511763 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b0049070 con 0x557be1065620 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f609dffb640 1 -- 172.21.15.50:0/4271511763 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f60b003b460 con 0x557be1065620 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f60bf175740 1 -- 172.21.15.50:0/4271511763 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60b00bce90 con 0x557be1065620 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f609dffb640 1 -- 172.21.15.50:0/4271511763 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f60b00bb190 con 0x557be1065620 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f60bf175740 10 client.17430 fetch_fsmap learned FSMap version 54 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f60bf175740 1 -- 172.21.15.50:0/4271511763 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x557be10731f0 con 0x557be1065620 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f609dffb640 1 -- 172.21.15.50:0/4271511763 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b00bce90 con 0x557be1065620 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f609dffb640 1 -- 172.21.15.50:0/4271511763 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f60b00bd0f0 con 0x557be1065620 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f60bf175740 10 client.17430 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f60bf175740 10 client.17430 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f60bf175740 20 client.17430 populate_metadata read hostname 'smithi050' 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f60bf175740 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557be10802e0 0x557be10826d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f60bf175740 4 client.17430 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.057+0000 7f60bf175740 1 -- 172.21.15.50:0/4271511763 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557be0fd8390 con 0x557be10802e0 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.058+0000 7f60b9918640 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557be10802e0 0x557be10826d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.058+0000 7f60b9918640 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557be10802e0 0x557be10826d0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75b6bb5e88b545ce server_cookie=78bea4bef56b204f in_seq=0 out_seq=0 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.058+0000 7f609dffb640 10 client.17430 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.058+0000 7f609dffb640 1 -- 172.21.15.50:0/4271511763 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557be0fd8390 con 0x557be10802e0 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.058+0000 7f609dffb640 10 client.17430 handle_command_reply: tid=0 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.058+0000 7f60bf175740 1 client.17430 shutdown 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.059+0000 7f60bf175740 1 -- 172.21.15.50:0/4271511763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065620 msgr2=0x557be1070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.059+0000 7f60bf175740 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065620 0x557be1070150 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f60b00ee6f0 tx=0x7f60b00c1380 comp rx=0 tx=0).stop 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.059+0000 7f60bf175740 1 -- 172.21.15.50:0/4271511763 shutdown_connections 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.059+0000 7f60bf175740 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557be10802e0 0x557be10826d0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.059+0000 7f60bf175740 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be106d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.059+0000 7f60bf175740 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065f40 0x557be106d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.059+0000 7f60bf175740 1 --2- 172.21.15.50:0/4271511763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065620 0x557be1070150 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.059+0000 7f609dffb640 0 client.17430 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.059+0000 7f60bf175740 1 -- 172.21.15.50:0/4271511763 >> 172.21.15.50:0/4271511763 conn(0x557be1061b30 msgr2=0x557be1067860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.059+0000 7f60bf175740 1 -- 172.21.15.50:0/4271511763 shutdown_connections 2024-04-23T18:34:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.059+0000 7f60bf175740 1 -- 172.21.15.50:0/4271511763 wait complete. 2024-04-23T18:34:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.059+0000 7f60bf175740 20 client.17430 trim_cache size 0 max 16384 2024-04-23T18:34:17.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60bf175740 1 Processor -- start 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60bf175740 1 -- start start 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60bf175740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be1060b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60bf175740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 0x557be106fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60bf175740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be1087fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60bf175740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41b0560 con 0x557be1065f40 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60bf175740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41addd0 con 0x557be1070800 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60bf175740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41abc40 con 0x557be1065620 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60b9918640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 0x557be106fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60b9918640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 0x557be106fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59298/0 (socket says 172.21.15.50:59298) 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60b9918640 1 -- 172.21.15.50:0/1804592317 learned_addr learned my addr 172.21.15.50:0/1804592317 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60ba119640 1 --2- 172.21.15.50:0/1804592317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be1060b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60ba91a640 1 --2- 172.21.15.50:0/1804592317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be1087fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60b9918640 1 -- 172.21.15.50:0/1804592317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 msgr2=0x557be1060b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60b9918640 1 --2- 172.21.15.50:0/1804592317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be1060b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60b9918640 1 -- 172.21.15.50:0/1804592317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 msgr2=0x557be1087fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60b9918640 1 --2- 172.21.15.50:0/1804592317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be1087fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60b9918640 1 -- 172.21.15.50:0/1804592317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557be106dd10 con 0x557be1065f40 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60ba119640 1 --2- 172.21.15.50:0/1804592317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be1060b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f60b9918640 1 --2- 172.21.15.50:0/1804592317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 0x557be106fc80 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009540 tx=0x7f60a4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb92bb39ce3b9a5c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.074+0000 7f609ffff640 1 -- 172.21.15.50:0/1804592317 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a40203d0 con 0x557be1065f40 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.075+0000 7f609ffff640 1 -- 172.21.15.50:0/1804592317 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60a4026070 con 0x557be1065f40 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.075+0000 7f609ffff640 1 -- 172.21.15.50:0/1804592317 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a401d380 con 0x557be1065f40 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.075+0000 7f60bf175740 1 -- 172.21.15.50:0/1804592317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 msgr2=0x557be106fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.075+0000 7f60bf175740 1 --2- 172.21.15.50:0/1804592317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 0x557be106fc80 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009540 tx=0x7f60a4005f90 comp rx=0 tx=0).stop 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.075+0000 7f60bf175740 1 -- 172.21.15.50:0/1804592317 shutdown_connections 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.075+0000 7f60bf175740 1 --2- 172.21.15.50:0/1804592317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1070800 0x557be1087fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.075+0000 7f60bf175740 1 --2- 172.21.15.50:0/1804592317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1065f40 0x557be106fc80 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.075+0000 7f60bf175740 1 --2- 172.21.15.50:0/1804592317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be1060b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.075+0000 7f60bf175740 1 -- 172.21.15.50:0/1804592317 >> 172.21.15.50:0/1804592317 conn(0x557be1061b30 msgr2=0x557be1060420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.075+0000 7f60bf175740 1 -- 172.21.15.50:0/1804592317 shutdown_connections 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.075+0000 7f60bf175740 1 -- 172.21.15.50:0/1804592317 wait complete. 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.075+0000 7f60bf175740 1 -- start start 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60bf175740 1 Processor -- start 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60bf175740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be10de7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60bf175740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1065f40 0x557be10ded30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60bf175740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1070800 0x557be10df270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60bf175740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41ac290 con 0x557be1070800 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60bf175740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41ad700 con 0x557be1065f40 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60bf175740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41ac960 con 0x557be1065620 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60bf175740 10 client.0 _ll_register_callbacks cb 0x7f60b00bcd10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60ba91a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1070800 0x557be10df270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60ba91a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1070800 0x557be10df270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59308/0 (socket says 172.21.15.50:59308) 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60ba91a640 1 -- 172.21.15.50:0/2988882725 learned_addr learned my addr 172.21.15.50:0/2988882725 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60b9918640 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1065f40 0x557be10ded30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60ba119640 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be10de7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60ba91a640 1 -- 172.21.15.50:0/2988882725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 msgr2=0x557be10de7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60ba91a640 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be10de7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60ba91a640 1 -- 172.21.15.50:0/2988882725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1065f40 msgr2=0x557be10ded30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60ba91a640 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1065f40 0x557be10ded30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60ba91a640 1 -- 172.21.15.50:0/2988882725 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557be10e9ac0 con 0x557be1070800 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60b9918640 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1065f40 0x557be10ded30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f60ba91a640 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1070800 0x557be10df270 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f60ac013000 tx=0x7f60ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97296289df65807d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.076+0000 7f609dffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f609dffb640 1 -- 172.21.15.50:0/2988882725 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac004b20 con 0x557be1070800 2024-04-23T18:34:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f609dffb640 1 -- 172.21.15.50:0/2988882725 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f60ac01b070 con 0x557be1070800 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f60bf175740 1 -- 172.21.15.50:0/2988882725 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60a4026070 con 0x557be1070800 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f609dffb640 1 -- 172.21.15.50:0/2988882725 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f60ac004cc0 con 0x557be1070800 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f60bf175740 10 client.17436 fetch_fsmap learned FSMap version 54 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f60bf175740 1 -- 172.21.15.50:0/2988882725 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x557be10eadd0 con 0x557be1070800 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f609dffb640 1 -- 172.21.15.50:0/2988882725 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60a4026070 con 0x557be1070800 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f609dffb640 1 -- 172.21.15.50:0/2988882725 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f60ac02b840 con 0x557be1070800 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f60bf175740 10 client.17436 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f60bf175740 10 client.17436 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f60bf175740 20 client.17436 populate_metadata read hostname 'smithi050' 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f60bf175740 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557be10eb610 0x557be10eda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f60bf175740 4 client.17436 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f60bf175740 1 -- 172.21.15.50:0/2988882725 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557be0fd2200 con 0x557be10eb610 2024-04-23T18:34:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.077+0000 7f60ba119640 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557be10eb610 0x557be10eda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:17.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.078+0000 7f60ba119640 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557be10eb610 0x557be10eda30 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=155e8a0a8df79e1b server_cookie=7f3aa5ffc9ea98f in_seq=0 out_seq=0 2024-04-23T18:34:17.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.078+0000 7f609dffb640 10 client.17436 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.346+0000 7f609dffb640 1 -- 172.21.15.50:0/2988882725 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557be0fd2200 con 0x557be10eb610 2024-04-23T18:34:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.346+0000 7f609dffb640 10 client.17436 handle_command_reply: tid=0 2024-04-23T18:34:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.346+0000 7f60bf175740 1 client.17436 shutdown 2024-04-23T18:34:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bf175740 1 -- 172.21.15.50:0/2988882725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1070800 msgr2=0x557be10df270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bf175740 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1070800 0x557be10df270 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f60ac013000 tx=0x7f60ac004a40 comp rx=0 tx=0).stop 2024-04-23T18:34:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bf175740 1 -- 172.21.15.50:0/2988882725 shutdown_connections 2024-04-23T18:34:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bf175740 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557be10eb610 0x557be10eda30 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bf175740 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x557be1070800 0x557be10df270 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bf175740 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557be1065f40 0x557be10ded30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bf175740 1 --2- 172.21.15.50:0/2988882725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557be1065620 0x557be10de7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f609dffb640 0 client.17436 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bf175740 1 -- 172.21.15.50:0/2988882725 >> 172.21.15.50:0/2988882725 conn(0x557be1061b30 msgr2=0x557be1060cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bf175740 1 -- 172.21.15.50:0/2988882725 shutdown_connections 2024-04-23T18:34:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bf175740 1 -- 172.21.15.50:0/2988882725 wait complete. 2024-04-23T18:34:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bf175740 20 client.17436 trim_cache size 0 max 16384 2024-04-23T18:34:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:34:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:34:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:34:17.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:34:17.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bc37d640 1 -- 172.21.15.50:0/1852129435 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f60a00a3e40 msgr2=0x7f60a00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:17.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bc37d640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f60a00a3e40 0x7f60a00a6300 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009fd0 tx=0x7f60ac009500 comp rx=0 tx=0).stop 2024-04-23T18:34:17.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bc37d640 1 -- 172.21.15.50:0/1852129435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60b41031e0 msgr2=0x7f60b41986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:17.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.347+0000 7f60bc37d640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60b41031e0 0x7f60b41986e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f60b0005ab0 tx=0x7f60b0003c10 comp rx=0 tx=0).stop 2024-04-23T18:34:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.348+0000 7f60bc37d640 1 -- 172.21.15.50:0/1852129435 shutdown_connections 2024-04-23T18:34:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.348+0000 7f60bc37d640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f60a00a3e40 0x7f60a00a6300 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.348+0000 7f60bc37d640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60b410c520 0x7f60b419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.348+0000 7f60bc37d640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4103b00 0x7f60b4198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.348+0000 7f60bc37d640 1 --2- 172.21.15.50:0/1852129435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60b41031e0 0x7f60b41986e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.348+0000 7f60bc37d640 1 -- 172.21.15.50:0/1852129435 >> 172.21.15.50:0/1852129435 conn(0x7f60b4077fd0 msgr2=0x7f60b40fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.348+0000 7f60bc37d640 1 -- 172.21.15.50:0/1852129435 shutdown_connections 2024-04-23T18:34:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:17.348+0000 7f60bc37d640 1 -- 172.21.15.50:0/1852129435 wait complete. 2024-04-23T18:34:17.564 INFO:tasks.workunit.client.0.smithi050.stdout: 10 81974 31.35 MB/sec execute 51 sec latency 214.763 ms 2024-04-23T18:34:18.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:18 smithi050 ceph-mon[29187]: pgmap v500: 97 pgs: 97 active+clean; 900 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 664 op/s 2024-04-23T18:34:18.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:18 smithi177 ceph-mon[32308]: pgmap v500: 97 pgs: 97 active+clean; 900 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 664 op/s 2024-04-23T18:34:18.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 82394 31.10 MB/sec execute 52 sec latency 304.695 ms 2024-04-23T18:34:18.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:18 smithi188 ceph-mon[32694]: pgmap v500: 97 pgs: 97 active+clean; 900 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 664 op/s 2024-04-23T18:34:19.564 INFO:tasks.workunit.client.0.smithi050.stdout: 10 83090 31.26 MB/sec execute 53 sec latency 66.734 ms 2024-04-23T18:34:20.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:20 smithi050 ceph-mon[29187]: pgmap v501: 97 pgs: 97 active+clean; 900 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 664 op/s 2024-04-23T18:34:20.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:20 smithi177 ceph-mon[32308]: pgmap v501: 97 pgs: 97 active+clean; 900 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 664 op/s 2024-04-23T18:34:20.563 INFO:tasks.workunit.client.0.smithi050.stdout: 10 83849 31.41 MB/sec execute 54 sec latency 42.270 ms 2024-04-23T18:34:20.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:20 smithi188 ceph-mon[32694]: pgmap v501: 97 pgs: 97 active+clean; 900 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 664 op/s 2024-04-23T18:34:21.564 INFO:tasks.workunit.client.0.smithi050.stdout: 10 84508 31.62 MB/sec execute 55 sec latency 59.833 ms 2024-04-23T18:34:22.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:22 smithi050 ceph-mon[29187]: pgmap v502: 97 pgs: 97 active+clean; 900 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 664 op/s 2024-04-23T18:34:22.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:22 smithi177 ceph-mon[32308]: pgmap v502: 97 pgs: 97 active+clean; 900 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 664 op/s 2024-04-23T18:34:22.564 INFO:tasks.workunit.client.0.smithi050.stdout: 10 85142 31.63 MB/sec execute 56 sec latency 95.506 ms 2024-04-23T18:34:22.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:22 smithi188 ceph-mon[32694]: pgmap v502: 97 pgs: 97 active+clean; 900 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 664 op/s 2024-04-23T18:34:23.564 INFO:tasks.workunit.client.0.smithi050.stdout: 10 85614 31.43 MB/sec execute 57 sec latency 290.873 ms 2024-04-23T18:34:24.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:24 smithi050 ceph-mon[29187]: pgmap v503: 97 pgs: 97 active+clean; 914 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 17 MiB/s wr, 943 op/s 2024-04-23T18:34:24.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:24 smithi177 ceph-mon[32308]: pgmap v503: 97 pgs: 97 active+clean; 914 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 17 MiB/s wr, 943 op/s 2024-04-23T18:34:24.564 INFO:tasks.workunit.client.0.smithi050.stdout: 10 86164 31.36 MB/sec execute 58 sec latency 84.171 ms 2024-04-23T18:34:24.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:24 smithi188 ceph-mon[32694]: pgmap v503: 97 pgs: 97 active+clean; 914 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 17 MiB/s wr, 943 op/s 2024-04-23T18:34:25.564 INFO:tasks.workunit.client.0.smithi050.stdout: 10 86867 31.53 MB/sec execute 59 sec latency 55.564 ms 2024-04-23T18:34:26.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:26 smithi050 ceph-mon[29187]: pgmap v504: 97 pgs: 97 active+clean; 914 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 644 op/s 2024-04-23T18:34:26.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:26 smithi177 ceph-mon[32308]: pgmap v504: 97 pgs: 97 active+clean; 914 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 644 op/s 2024-04-23T18:34:26.564 INFO:tasks.workunit.client.0.smithi050.stdout: 10 87662 31.62 MB/sec execute 60 sec latency 180.186 ms 2024-04-23T18:34:26.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:26 smithi188 ceph-mon[32694]: pgmap v504: 97 pgs: 97 active+clean; 914 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 644 op/s 2024-04-23T18:34:27.564 INFO:tasks.workunit.client.0.smithi050.stdout: 10 88401 31.96 MB/sec execute 61 sec latency 139.850 ms 2024-04-23T18:34:28.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:28 smithi050 ceph-mon[29187]: pgmap v505: 97 pgs: 97 active+clean; 919 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 17 MiB/s wr, 908 op/s 2024-04-23T18:34:28.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:28 smithi177 ceph-mon[32308]: pgmap v505: 97 pgs: 97 active+clean; 919 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 17 MiB/s wr, 908 op/s 2024-04-23T18:34:28.564 INFO:tasks.workunit.client.0.smithi050.stdout: 10 88836 31.92 MB/sec execute 62 sec latency 214.485 ms 2024-04-23T18:34:28.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:28 smithi188 ceph-mon[32694]: pgmap v505: 97 pgs: 97 active+clean; 919 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 17 MiB/s wr, 908 op/s 2024-04-23T18:34:29.564 INFO:tasks.workunit.client.0.smithi050.stdout: 10 89232 31.61 MB/sec execute 63 sec latency 205.423 ms 2024-04-23T18:34:30.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:30 smithi050 ceph-mon[29187]: pgmap v506: 97 pgs: 97 active+clean; 919 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 621 op/s 2024-04-23T18:34:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.452+0000 7f3810b5d640 1 -- 172.21.15.50:0/513240925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380c1010f0 msgr2=0x7f380c1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.452+0000 7f3810b5d640 1 --2- 172.21.15.50:0/513240925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380c1010f0 0x7f380c1014d0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f37fc009900 tx=0x7f37fc02a410 comp rx=0 tx=0).stop 2024-04-23T18:34:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.453+0000 7f3810b5d640 1 -- 172.21.15.50:0/513240925 shutdown_connections 2024-04-23T18:34:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.453+0000 7f3810b5d640 1 --2- 172.21.15.50:0/513240925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f380c106170 0x7f380c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.453+0000 7f3810b5d640 1 --2- 172.21.15.50:0/513240925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f380c101aa0 0x7f380c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.453+0000 7f3810b5d640 1 --2- 172.21.15.50:0/513240925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380c1010f0 0x7f380c1014d0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.453+0000 7f3810b5d640 1 -- 172.21.15.50:0/513240925 >> 172.21.15.50:0/513240925 conn(0x7f380c077fd0 msgr2=0x7f380c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.453+0000 7f3810b5d640 1 -- 172.21.15.50:0/513240925 shutdown_connections 2024-04-23T18:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.453+0000 7f3810b5d640 1 -- 172.21.15.50:0/513240925 wait complete. 2024-04-23T18:34:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.453+0000 7f3810b5d640 1 Processor -- start 2024-04-23T18:34:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f3810b5d640 1 -- start start 2024-04-23T18:34:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f3810b5d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f380c1010f0 0x7f380c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f3810b5d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f380c106170 0x7f380c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f3810b5d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380c19cf50 0x7f380c19d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f3810b5d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f380c1110b0 con 0x7f380c19cf50 2024-04-23T18:34:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f3810b5d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f380c110f30 con 0x7f380c1010f0 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f3810b5d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f380c111230 con 0x7f380c106170 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f380ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380c19cf50 0x7f380c19d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f380a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f380c1010f0 0x7f380c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f380ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380c19cf50 0x7f380c19d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56380/0 (socket says 172.21.15.50:56380) 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f380ad9d640 1 -- 172.21.15.50:0/3742412383 learned_addr learned my addr 172.21.15.50:0/3742412383 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f3809d9b640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f380c106170 0x7f380c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f380a59c640 1 -- 172.21.15.50:0/3742412383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f380c106170 msgr2=0x7f380c198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f380a59c640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f380c106170 0x7f380c198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f380a59c640 1 -- 172.21.15.50:0/3742412383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380c19cf50 msgr2=0x7f380c19d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f380a59c640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380c19cf50 0x7f380c19d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f380a59c640 1 -- 172.21.15.50:0/3742412383 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f380c19da40 con 0x7f380c1010f0 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.454+0000 7f380ad9d640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380c19cf50 0x7f380c19d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.455+0000 7f380a59c640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f380c1010f0 0x7f380c198690 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f37fc02a930 tx=0x7f37fc031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.455+0000 7f38037fe640 1 -- 172.21.15.50:0/3742412383 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc03f030 con 0x7f380c1010f0 2024-04-23T18:34:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.455+0000 7f3810b5d640 1 -- 172.21.15.50:0/3742412383 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f380c19dcd0 con 0x7f380c1010f0 2024-04-23T18:34:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.455+0000 7f38037fe640 1 -- 172.21.15.50:0/3742412383 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37fc004490 con 0x7f380c1010f0 2024-04-23T18:34:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.455+0000 7f38037fe640 1 -- 172.21.15.50:0/3742412383 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc035070 con 0x7f380c1010f0 2024-04-23T18:34:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.455+0000 7f3810b5d640 1 -- 172.21.15.50:0/3742412383 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f380c1a54f0 con 0x7f380c1010f0 2024-04-23T18:34:30.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.459+0000 7f38037fe640 1 -- 172.21.15.50:0/3742412383 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f37fc042030 con 0x7f380c1010f0 2024-04-23T18:34:30.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.459+0000 7f3813876740 1 Processor -- start 2024-04-23T18:34:30.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.459+0000 7f3813876740 1 -- start start 2024-04-23T18:34:30.453 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.459+0000 7f3813876740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9520 0x562708aa9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.453 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.459+0000 7f3813876740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708aa9e40 0x562708ab86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.453 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.459+0000 7f3813876740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708ab8c10 0x562708abb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.459+0000 7f3813876740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562708a0e970 con 0x562708ab8c10 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.459+0000 7f3813876740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5627089ab4a0 con 0x562708aa9e40 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.459+0000 7f3813876740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562708a0a930 con 0x562708aa9520 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f380ad9d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9520 0x562708aa9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f380a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708aa9e40 0x562708ab86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f380ad9d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9520 0x562708aa9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:46908/0 (socket says 172.21.15.50:46908) 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f380ad9d640 1 -- 172.21.15.50:0/3189951443 learned_addr learned my addr 172.21.15.50:0/3189951443 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f3809d9b640 1 --2- 172.21.15.50:0/3189951443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708ab8c10 0x562708abb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f38037fe640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f37f80a3dc0 0x7f37f80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f380ad9d640 1 -- 172.21.15.50:0/3189951443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708aa9e40 msgr2=0x562708ab86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f380ad9d640 1 --2- 172.21.15.50:0/3189951443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708aa9e40 0x562708ab86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f380ad9d640 1 -- 172.21.15.50:0/3189951443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708ab8c10 msgr2=0x562708abb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f380ad9d640 1 --2- 172.21.15.50:0/3189951443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708ab8c10 0x562708abb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f380ad9d640 1 -- 172.21.15.50:0/3189951443 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562708721210 con 0x562708aa9520 2024-04-23T18:34:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f38037fe640 1 -- 172.21.15.50:0/3742412383 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f37fc0f0b30 con 0x7f380c1010f0 2024-04-23T18:34:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.460+0000 7f380ad9d640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f37f80a3dc0 0x7f37f80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.461+0000 7f380ad9d640 1 --2- 172.21.15.50:0/3189951443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9520 0x562708aa9900 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f38040119d0 tx=0x7f3804011e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b29a82eb31aadd52 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.461+0000 7f3800ff9640 1 -- 172.21.15.50:0/3189951443 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3804017070 con 0x562708aa9520 2024-04-23T18:34:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.461+0000 7f3800ff9640 1 -- 172.21.15.50:0/3189951443 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3804026430 con 0x562708aa9520 2024-04-23T18:34:30.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.462+0000 7f3813876740 1 -- 172.21.15.50:0/3189951443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9520 msgr2=0x562708aa9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:30.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.462+0000 7f3813876740 1 --2- 172.21.15.50:0/3189951443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9520 0x562708aa9900 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f38040119d0 tx=0x7f3804011e90 comp rx=0 tx=0).stop 2024-04-23T18:34:30.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.462+0000 7f380ad9d640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f37f80a3dc0 0x7f37f80a6280 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3804021000 tx=0x7f3804022000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:30.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.462+0000 7f3813876740 1 -- 172.21.15.50:0/3189951443 shutdown_connections 2024-04-23T18:34:30.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.462+0000 7f3813876740 1 --2- 172.21.15.50:0/3189951443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708ab8c10 0x562708abb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.462+0000 7f3813876740 1 --2- 172.21.15.50:0/3189951443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708aa9e40 0x562708ab86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.462+0000 7f3813876740 1 --2- 172.21.15.50:0/3189951443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9520 0x562708aa9900 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.462+0000 7f3813876740 1 -- 172.21.15.50:0/3189951443 >> 172.21.15.50:0/3189951443 conn(0x562708aa5b30 msgr2=0x562708aa5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.462+0000 7f3813876740 1 -- 172.21.15.50:0/3189951443 shutdown_connections 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.463+0000 7f3813876740 1 -- 172.21.15.50:0/3189951443 wait complete. 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.463+0000 7f3813876740 1 -- start start 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.463+0000 7f3813876740 1 Processor -- start 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.463+0000 7f3813876740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 0x562708ab83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.463+0000 7f3813876740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708aa9e40 0x562708ab56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.463+0000 7f3813876740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708ab8c10 0x562708ab5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.463+0000 7f3813876740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x562708abb840 con 0x562708aa9520 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.463+0000 7f3813876740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5627089a90b0 con 0x562708aa9e40 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.463+0000 7f3813876740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x562708abb9c0 con 0x562708ab8c10 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.463+0000 7f3813876740 10 client.0 _ll_register_callbacks cb 0x562708996d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:34:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f380a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 0x562708ab83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f3809d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708aa9e40 0x562708ab56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f3809d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708aa9e40 0x562708ab56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:41822/0 (socket says 172.21.15.50:41822) 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f3809d9b640 1 -- 172.21.15.50:0/1944832453 learned_addr learned my addr 172.21.15.50:0/1944832453 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f380ad9d640 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708ab8c10 0x562708ab5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f3809d9b640 1 -- 172.21.15.50:0/1944832453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708ab8c10 msgr2=0x562708ab5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f3809d9b640 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708ab8c10 0x562708ab5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f3809d9b640 1 -- 172.21.15.50:0/1944832453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 msgr2=0x562708ab83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f3809d9b640 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 0x562708ab83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f3809d9b640 1 -- 172.21.15.50:0/1944832453 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627087ed620 con 0x562708aa9e40 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f380a59c640 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 0x562708ab83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f380ad9d640 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708ab8c10 0x562708ab5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f3809d9b640 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708aa9e40 0x562708ab56c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f37f4012700 tx=0x7f37f4012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e355cdfd1f39bd75 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f37f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:34:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f37f1ffb640 1 -- 172.21.15.50:0/1944832453 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4004320 con 0x562708aa9e40 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f37f1ffb640 1 -- 172.21.15.50:0/1944832453 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f37f40119a0 con 0x562708aa9e40 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.464+0000 7f3813876740 1 -- 172.21.15.50:0/1944832453 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3804026430 con 0x562708aa9e40 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.465+0000 7f37f1ffb640 1 -- 172.21.15.50:0/1944832453 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f37f40044c0 con 0x562708aa9e40 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.465+0000 7f3813876740 10 client.26945 fetch_fsmap learned FSMap version 54 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.465+0000 7f3813876740 1 -- 172.21.15.50:0/1944832453 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x562708ac7730 con 0x562708aa9e40 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.465+0000 7f37f1ffb640 1 -- 172.21.15.50:0/1944832453 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3804026430 con 0x562708aa9e40 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.465+0000 7f37f1ffb640 1 -- 172.21.15.50:0/1944832453 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f37f400f440 con 0x562708aa9e40 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.465+0000 7f3813876740 10 client.26945 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.465+0000 7f3813876740 10 client.26945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.465+0000 7f3813876740 20 client.26945 populate_metadata read hostname 'smithi050' 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.465+0000 7f3813876740 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562708ac8c40 0x562708acb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.465+0000 7f3813876740 4 client.26945 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:34:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.465+0000 7f3813876740 1 -- 172.21.15.50:0/1944832453 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562708a1c390 con 0x562708ac8c40 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.465+0000 7f380a59c640 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562708ac8c40 0x562708acb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.466+0000 7f37f1ffb640 10 client.26945 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.466+0000 7f380a59c640 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562708ac8c40 0x562708acb090 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25c2671064efa4a7 server_cookie=f501eaa7cda59a04 in_seq=0 out_seq=0 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f37f1ffb640 1 -- 172.21.15.50:0/1944832453 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562708a1c390 con 0x562708ac8c40 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f37f1ffb640 10 client.26945 handle_command_reply: tid=0 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f3813876740 1 client.26945 shutdown 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f3813876740 1 -- 172.21.15.50:0/1944832453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708aa9e40 msgr2=0x562708ab56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f3813876740 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708aa9e40 0x562708ab56c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f37f4012700 tx=0x7f37f4012bc0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f3813876740 1 -- 172.21.15.50:0/1944832453 shutdown_connections 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f3813876740 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562708ac8c40 0x562708acb090 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f3813876740 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708ab8c10 0x562708ab5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f3813876740 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708aa9e40 0x562708ab56c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f3813876740 1 --2- 172.21.15.50:0/1944832453 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 0x562708ab83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f37f1ffb640 0 client.26945 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f3813876740 1 -- 172.21.15.50:0/1944832453 >> 172.21.15.50:0/1944832453 conn(0x562708aa5b30 msgr2=0x562708aab9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f3813876740 1 -- 172.21.15.50:0/1944832453 shutdown_connections 2024-04-23T18:34:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f3813876740 1 -- 172.21.15.50:0/1944832453 wait complete. 2024-04-23T18:34:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.467+0000 7f3813876740 20 client.26945 trim_cache size 0 max 16384 2024-04-23T18:34:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3813876740 1 Processor -- start 2024-04-23T18:34:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3813876740 1 -- start start 2024-04-23T18:34:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3813876740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9520 0x562708ac47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3813876740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9e40 0x562708ad0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3813876740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708ab8c10 0x562708ad0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3813876740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f380c1ae290 con 0x562708aa9e40 2024-04-23T18:34:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3813876740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f380c1abb00 con 0x562708ab8c10 2024-04-23T18:34:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3813876740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f380c1a9950 con 0x562708aa9520 2024-04-23T18:34:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3809d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9e40 0x562708ad0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3809d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9e40 0x562708ad0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56390/0 (socket says 172.21.15.50:56390) 2024-04-23T18:34:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3809d9b640 1 -- 172.21.15.50:0/1521209985 learned_addr learned my addr 172.21.15.50:0/1521209985 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f380a59c640 1 --2- 172.21.15.50:0/1521209985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9520 0x562708ac47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f380ad9d640 1 --2- 172.21.15.50:0/1521209985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708ab8c10 0x562708ad0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3809d9b640 1 -- 172.21.15.50:0/1521209985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9520 msgr2=0x562708ac47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3809d9b640 1 --2- 172.21.15.50:0/1521209985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9520 0x562708ac47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3809d9b640 1 -- 172.21.15.50:0/1521209985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708ab8c10 msgr2=0x562708ad0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3809d9b640 1 --2- 172.21.15.50:0/1521209985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708ab8c10 0x562708ad0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f3809d9b640 1 -- 172.21.15.50:0/1521209985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562708aa5960 con 0x562708aa9e40 2024-04-23T18:34:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.491+0000 7f380ad9d640 1 --2- 172.21.15.50:0/1521209985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708ab8c10 0x562708ad0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:34:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.492+0000 7f3809d9b640 1 --2- 172.21.15.50:0/1521209985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9e40 0x562708ad0960 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f37f400f860 tx=0x7f37f400fea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784d04982f9f213a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.492+0000 7f37f3fff640 1 -- 172.21.15.50:0/1521209985 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4032070 con 0x562708aa9e40 2024-04-23T18:34:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.492+0000 7f37f3fff640 1 -- 172.21.15.50:0/1521209985 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37f4011e50 con 0x562708aa9e40 2024-04-23T18:34:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.492+0000 7f37f3fff640 1 -- 172.21.15.50:0/1521209985 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4029040 con 0x562708aa9e40 2024-04-23T18:34:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 -- 172.21.15.50:0/1521209985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9e40 msgr2=0x562708ad0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 --2- 172.21.15.50:0/1521209985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9e40 0x562708ad0960 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f37f400f860 tx=0x7f37f400fea0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 -- 172.21.15.50:0/1521209985 shutdown_connections 2024-04-23T18:34:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 --2- 172.21.15.50:0/1521209985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708ab8c10 0x562708ad0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 --2- 172.21.15.50:0/1521209985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9e40 0x562708ad0960 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 --2- 172.21.15.50:0/1521209985 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9520 0x562708ac47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 -- 172.21.15.50:0/1521209985 >> 172.21.15.50:0/1521209985 conn(0x562708aa5b30 msgr2=0x562708ab5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 -- 172.21.15.50:0/1521209985 shutdown_connections 2024-04-23T18:34:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 -- 172.21.15.50:0/1521209985 wait complete. 2024-04-23T18:34:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 -- start start 2024-04-23T18:34:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 Processor -- start 2024-04-23T18:34:30.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 0x562708ac11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9e40 0x562708ab2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708ab8c10 0x562708ab3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f380c1a9fc0 con 0x562708aa9520 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f380c1ab430 con 0x562708ab8c10 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.493+0000 7f3813876740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f380c1aa690 con 0x562708aa9e40 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f3813876740 10 client.0 _ll_register_callbacks cb 0x7f37f4009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f380a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 0x562708ac11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f380a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 0x562708ac11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56392/0 (socket says 172.21.15.50:56392) 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f380a59c640 1 -- 172.21.15.50:0/566403443 learned_addr learned my addr 172.21.15.50:0/566403443 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f380ad9d640 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708ab8c10 0x562708ab3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f3809d9b640 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9e40 0x562708ab2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f380a59c640 1 -- 172.21.15.50:0/566403443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9e40 msgr2=0x562708ab2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f380a59c640 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9e40 0x562708ab2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f380a59c640 1 -- 172.21.15.50:0/566403443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708ab8c10 msgr2=0x562708ab3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f380a59c640 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708ab8c10 0x562708ab3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f380a59c640 1 -- 172.21.15.50:0/566403443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562708ab1b70 con 0x562708aa9520 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f380a59c640 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 0x562708ac11a0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0bc760 tx=0x7f37fc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b93ecfaab095caf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f37f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f37f1ffb640 1 -- 172.21.15.50:0/566403443 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc049040 con 0x562708aa9520 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f37f1ffb640 1 -- 172.21.15.50:0/566403443 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f37fc003d30 con 0x562708aa9520 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f3813876740 1 -- 172.21.15.50:0/566403443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37f4011e50 con 0x562708aa9520 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f37f1ffb640 1 -- 172.21.15.50:0/566403443 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f37fc0bc3c0 con 0x562708aa9520 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f3813876740 10 client.17454 fetch_fsmap learned FSMap version 54 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.494+0000 7f3813876740 1 -- 172.21.15.50:0/566403443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x562708b2e700 con 0x562708aa9520 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.495+0000 7f37f1ffb640 1 -- 172.21.15.50:0/566403443 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37f4011e50 con 0x562708aa9520 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.495+0000 7f37f1ffb640 1 -- 172.21.15.50:0/566403443 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f37fc0b90d0 con 0x562708aa9520 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.495+0000 7f3813876740 10 client.17454 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.495+0000 7f3813876740 10 client.17454 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:34:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.495+0000 7f3813876740 20 client.17454 populate_metadata read hostname 'smithi050' 2024-04-23T18:34:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.495+0000 7f3813876740 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562708b2ee30 0x562708b31250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.495+0000 7f3813876740 4 client.17454 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:34:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.495+0000 7f3813876740 1 -- 172.21.15.50:0/566403443 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562708a16200 con 0x562708b2ee30 2024-04-23T18:34:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.495+0000 7f3809d9b640 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562708b2ee30 0x562708b31250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.496+0000 7f3809d9b640 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562708b2ee30 0x562708b31250 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc10e14e002fdf3d server_cookie=2bb58baf88f5df5d in_seq=0 out_seq=0 2024-04-23T18:34:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.496+0000 7f37f1ffb640 10 client.17454 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:30.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:30 smithi177 ceph-mon[32308]: pgmap v506: 97 pgs: 97 active+clean; 919 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 621 op/s 2024-04-23T18:34:30.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 89790 31.41 MB/sec execute 64 sec latency 68.933 ms 2024-04-23T18:34:30.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:30 smithi188 ceph-mon[32694]: pgmap v506: 97 pgs: 97 active+clean; 919 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 621 op/s 2024-04-23T18:34:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f37f1ffb640 1 -- 172.21.15.50:0/566403443 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562708a16200 con 0x562708b2ee30 2024-04-23T18:34:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f37f1ffb640 10 client.17454 handle_command_reply: tid=0 2024-04-23T18:34:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f3813876740 1 client.17454 shutdown 2024-04-23T18:34:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f3813876740 1 -- 172.21.15.50:0/566403443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 msgr2=0x562708ac11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f3813876740 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 0x562708ac11a0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0bc760 tx=0x7f37fc000bc0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f3813876740 1 -- 172.21.15.50:0/566403443 shutdown_connections 2024-04-23T18:34:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f3813876740 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x562708b2ee30 0x562708b31250 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f3813876740 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562708ab8c10 0x562708ab3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f3813876740 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562708aa9e40 0x562708ab2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f3813876740 1 --2- 172.21.15.50:0/566403443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x562708aa9520 0x562708ac11a0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f3813876740 1 -- 172.21.15.50:0/566403443 >> 172.21.15.50:0/566403443 conn(0x562708aa5b30 msgr2=0x562708ab0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f37f1ffb640 0 client.17454 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f3813876740 1 -- 172.21.15.50:0/566403443 shutdown_connections 2024-04-23T18:34:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f3813876740 1 -- 172.21.15.50:0/566403443 wait complete. 2024-04-23T18:34:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.737+0000 7f3813876740 20 client.17454 trim_cache size 0 max 16384 2024-04-23T18:34:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:34:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:34:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:34:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:34:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.738+0000 7f3810b5d640 1 -- 172.21.15.50:0/3742412383 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f37f80a3dc0 msgr2=0x7f37f80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.738+0000 7f3810b5d640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f37f80a3dc0 0x7f37f80a6280 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3804021000 tx=0x7f3804022000 comp rx=0 tx=0).stop 2024-04-23T18:34:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.738+0000 7f3810b5d640 1 -- 172.21.15.50:0/3742412383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f380c1010f0 msgr2=0x7f380c198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.738+0000 7f3810b5d640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f380c1010f0 0x7f380c198690 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f37fc02a930 tx=0x7f37fc031a20 comp rx=0 tx=0).stop 2024-04-23T18:34:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.738+0000 7f3810b5d640 1 -- 172.21.15.50:0/3742412383 shutdown_connections 2024-04-23T18:34:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.738+0000 7f3810b5d640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f37f80a3dc0 0x7f37f80a6280 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.738+0000 7f3810b5d640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f380c19cf50 0x7f380c19d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.738+0000 7f3810b5d640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f380c106170 0x7f380c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.738+0000 7f3810b5d640 1 --2- 172.21.15.50:0/3742412383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f380c1010f0 0x7f380c198690 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.738+0000 7f3810b5d640 1 -- 172.21.15.50:0/3742412383 >> 172.21.15.50:0/3742412383 conn(0x7f380c077fd0 msgr2=0x7f380c104db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.738+0000 7f3810b5d640 1 -- 172.21.15.50:0/3742412383 shutdown_connections 2024-04-23T18:34:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:30.738+0000 7f3810b5d640 1 -- 172.21.15.50:0/3742412383 wait complete. 2024-04-23T18:34:31.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 90466 31.54 MB/sec execute 65 sec latency 143.659 ms 2024-04-23T18:34:32.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:32 smithi050 ceph-mon[29187]: pgmap v507: 97 pgs: 97 active+clean; 919 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 621 op/s 2024-04-23T18:34:32.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:32 smithi177 ceph-mon[32308]: pgmap v507: 97 pgs: 97 active+clean; 919 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 621 op/s 2024-04-23T18:34:32.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 91372 31.80 MB/sec execute 66 sec latency 57.963 ms 2024-04-23T18:34:32.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:32 smithi188 ceph-mon[32694]: pgmap v507: 97 pgs: 97 active+clean; 919 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 621 op/s 2024-04-23T18:34:33.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 91965 31.99 MB/sec execute 67 sec latency 155.991 ms 2024-04-23T18:34:34.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:34 smithi050 ceph-mon[29187]: pgmap v508: 97 pgs: 97 active+clean; 931 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 15 MiB/s wr, 968 op/s 2024-04-23T18:34:34.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:34 smithi177 ceph-mon[32308]: pgmap v508: 97 pgs: 97 active+clean; 931 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 15 MiB/s wr, 968 op/s 2024-04-23T18:34:34.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 92443 31.92 MB/sec execute 68 sec latency 152.873 ms 2024-04-23T18:34:34.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:34 smithi188 ceph-mon[32694]: pgmap v508: 97 pgs: 97 active+clean; 931 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 15 MiB/s wr, 968 op/s 2024-04-23T18:34:35.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 93002 31.74 MB/sec execute 69 sec latency 112.757 ms 2024-04-23T18:34:36.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:36 smithi050 ceph-mon[29187]: pgmap v509: 97 pgs: 97 active+clean; 931 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.7 MiB/s wr, 610 op/s 2024-04-23T18:34:36.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:36 smithi177 ceph-mon[32308]: pgmap v509: 97 pgs: 97 active+clean; 931 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.7 MiB/s wr, 610 op/s 2024-04-23T18:34:36.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 93749 31.88 MB/sec execute 70 sec latency 93.521 ms 2024-04-23T18:34:36.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:36 smithi188 ceph-mon[32694]: pgmap v509: 97 pgs: 97 active+clean; 931 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.7 MiB/s wr, 610 op/s 2024-04-23T18:34:37.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 94441 31.92 MB/sec execute 71 sec latency 57.479 ms 2024-04-23T18:34:38.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:38 smithi050 ceph-mon[29187]: pgmap v510: 97 pgs: 97 active+clean; 958 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 849 op/s 2024-04-23T18:34:38.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:38 smithi177 ceph-mon[32308]: pgmap v510: 97 pgs: 97 active+clean; 958 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 849 op/s 2024-04-23T18:34:38.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 95285 32.22 MB/sec execute 72 sec latency 53.202 ms 2024-04-23T18:34:38.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:38 smithi188 ceph-mon[32694]: pgmap v510: 97 pgs: 97 active+clean; 958 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 849 op/s 2024-04-23T18:34:39.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 95700 32.02 MB/sec execute 73 sec latency 236.947 ms 2024-04-23T18:34:40.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:40 smithi050 ceph-mon[29187]: pgmap v511: 97 pgs: 97 active+clean; 958 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 585 op/s 2024-04-23T18:34:40.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:40 smithi177 ceph-mon[32308]: pgmap v511: 97 pgs: 97 active+clean; 958 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 585 op/s 2024-04-23T18:34:40.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 96257 32.01 MB/sec execute 74 sec latency 76.036 ms 2024-04-23T18:34:40.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:40 smithi188 ceph-mon[32694]: pgmap v511: 97 pgs: 97 active+clean; 958 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 585 op/s 2024-04-23T18:34:41.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 96804 31.90 MB/sec execute 75 sec latency 42.793 ms 2024-04-23T18:34:42.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:42 smithi050 ceph-mon[29187]: pgmap v512: 97 pgs: 97 active+clean; 958 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 585 op/s 2024-04-23T18:34:42.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:42 smithi177 ceph-mon[32308]: pgmap v512: 97 pgs: 97 active+clean; 958 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 585 op/s 2024-04-23T18:34:42.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 97508 31.98 MB/sec execute 76 sec latency 48.833 ms 2024-04-23T18:34:42.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:42 smithi188 ceph-mon[32694]: pgmap v512: 97 pgs: 97 active+clean; 958 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 585 op/s 2024-04-23T18:34:43.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 98260 32.11 MB/sec execute 77 sec latency 85.968 ms 2024-04-23T18:34:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.889+0000 7f8f941d6640 1 -- 172.21.15.50:0/4002874360 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c1003e0 msgr2=0x7f8f8c1007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.889+0000 7f8f941d6640 1 --2- 172.21.15.50:0/4002874360 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c1003e0 0x7f8f8c1007c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009900 tx=0x7f8f7c02a3f0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.891+0000 7f8f941d6640 1 -- 172.21.15.50:0/4002874360 shutdown_connections 2024-04-23T18:34:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.891+0000 7f8f941d6640 1 --2- 172.21.15.50:0/4002874360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f8c1054b0 0x7f8f8c10d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.891+0000 7f8f941d6640 1 --2- 172.21.15.50:0/4002874360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f8c100d90 0x7f8f8c104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.891+0000 7f8f941d6640 1 --2- 172.21.15.50:0/4002874360 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c1003e0 0x7f8f8c1007c0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.891+0000 7f8f941d6640 1 -- 172.21.15.50:0/4002874360 >> 172.21.15.50:0/4002874360 conn(0x7f8f8c076f50 msgr2=0x7f8f8c0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.891+0000 7f8f941d6640 1 -- 172.21.15.50:0/4002874360 shutdown_connections 2024-04-23T18:34:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.891+0000 7f8f941d6640 1 -- 172.21.15.50:0/4002874360 wait complete. 2024-04-23T18:34:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.892+0000 7f8f941d6640 1 Processor -- start 2024-04-23T18:34:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.893+0000 7f8f941d6640 1 -- start start 2024-04-23T18:34:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.893+0000 7f8f941d6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c1003e0 0x7f8f8c196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.893+0000 7f8f941d6640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f8c1054b0 0x7f8f8c196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.893+0000 7f8f941d6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f8c196da0 0x7f8f8c1aaec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.893+0000 7f8f941d6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8c110000 con 0x7f8f8c1003e0 2024-04-23T18:34:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.893+0000 7f8f941d6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8c10fe80 con 0x7f8f8c196da0 2024-04-23T18:34:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.893+0000 7f8f941d6640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8c110180 con 0x7f8f8c1054b0 2024-04-23T18:34:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.893+0000 7f8f92773640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f8c196da0 0x7f8f8c1aaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.893+0000 7f8f92773640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f8c196da0 0x7f8f8c1aaec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:52004/0 (socket says 172.21.15.50:52004) 2024-04-23T18:34:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.893+0000 7f8f92773640 1 -- 172.21.15.50:0/868339900 learned_addr learned my addr 172.21.15.50:0/868339900 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:43.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.894+0000 7f8f91771640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f8c1054b0 0x7f8f8c196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.894+0000 7f8f92773640 1 -- 172.21.15.50:0/868339900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f8c1054b0 msgr2=0x7f8f8c196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:43.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.894+0000 7f8f92773640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f8c1054b0 0x7f8f8c196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.894+0000 7f8f92773640 1 -- 172.21.15.50:0/868339900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c1003e0 msgr2=0x7f8f8c196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:43.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.894+0000 7f8f92773640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c1003e0 0x7f8f8c196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.894+0000 7f8f92773640 1 -- 172.21.15.50:0/868339900 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8c1ab400 con 0x7f8f8c196da0 2024-04-23T18:34:43.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.894+0000 7f8f91771640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f8c1054b0 0x7f8f8c196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:34:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.894+0000 7f8f92773640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f8c196da0 0x7f8f8c1aaec0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8f84007ba0 tx=0x7f8f8400d0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.895+0000 7f8f82ffd640 1 -- 172.21.15.50:0/868339900 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84017030 con 0x7f8f8c196da0 2024-04-23T18:34:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.895+0000 7f8f82ffd640 1 -- 172.21.15.50:0/868339900 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f8400daa0 con 0x7f8f8c196da0 2024-04-23T18:34:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.895+0000 7f8f941d6640 1 -- 172.21.15.50:0/868339900 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f8c1ab6f0 con 0x7f8f8c196da0 2024-04-23T18:34:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.895+0000 7f8f941d6640 1 -- 172.21.15.50:0/868339900 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f8c1abc10 con 0x7f8f8c196da0 2024-04-23T18:34:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.895+0000 7f8f82ffd640 1 -- 172.21.15.50:0/868339900 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84005840 con 0x7f8f8c196da0 2024-04-23T18:34:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.898+0000 7f8f82ffd640 1 -- 172.21.15.50:0/868339900 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8f8401a070 con 0x7f8f8c196da0 2024-04-23T18:34:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.899+0000 7f8f96fc5740 1 Processor -- start 2024-04-23T18:34:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.899+0000 7f8f96fc5740 1 -- start start 2024-04-23T18:34:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.899+0000 7f8f82ffd640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f8f780a3f60 0x7f8f780a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.899+0000 7f8f96fc5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 0x55b7df958a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.899+0000 7f8f91771640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f8f780a3f60 0x7f8f780a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.899+0000 7f8f82ffd640 1 -- 172.21.15.50:0/868339900 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f8f840d0450 con 0x7f8f8c196da0 2024-04-23T18:34:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f96fc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958f40 0x55b7df9632c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f96fc5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df963800 0x55b7df965bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f91f72640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 0x55b7df958a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f91f72640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 0x55b7df958a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:38476/0 (socket says 172.21.15.50:38476) 2024-04-23T18:34:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f91f72640 1 -- 172.21.15.50:0/1251051454 learned_addr learned my addr 172.21.15.50:0/1251051454 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1251051454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b7df8bd970 con 0x55b7df963800 2024-04-23T18:34:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1251051454 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b7df85a4a0 con 0x55b7df958f40 2024-04-23T18:34:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f91771640 1 --2- 172.21.15.50:0/1251051454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958f40 0x55b7df9632c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1251051454 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b7df8b9930 con 0x55b7df958620 2024-04-23T18:34:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f92773640 1 --2- 172.21.15.50:0/1251051454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df963800 0x55b7df965bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f91771640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f8f780a3f60 0x7f8f780a6420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8f880058c0 tx=0x7f8f88005850 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f91771640 1 -- 172.21.15.50:0/1251051454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 msgr2=0x55b7df958a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f91771640 1 --2- 172.21.15.50:0/1251051454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 0x55b7df958a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f91771640 1 -- 172.21.15.50:0/1251051454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df963800 msgr2=0x55b7df965bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f91771640 1 --2- 172.21.15.50:0/1251051454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df963800 0x55b7df965bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.900+0000 7f8f91771640 1 -- 172.21.15.50:0/1251051454 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7df5d0210 con 0x55b7df958f40 2024-04-23T18:34:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.901+0000 7f8f91771640 1 --2- 172.21.15.50:0/1251051454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958f40 0x55b7df9632c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c197b70 tx=0x7f8f88017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13e1c913c929b1d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.901+0000 7f8f77fff640 1 -- 172.21.15.50:0/1251051454 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8801d630 con 0x55b7df958f40 2024-04-23T18:34:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.901+0000 7f8f77fff640 1 -- 172.21.15.50:0/1251051454 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f88017070 con 0x55b7df958f40 2024-04-23T18:34:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.901+0000 7f8f77fff640 1 -- 172.21.15.50:0/1251051454 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f88025aa0 con 0x55b7df958f40 2024-04-23T18:34:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.902+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1251051454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958f40 msgr2=0x55b7df9632c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.902+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/1251051454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958f40 0x55b7df9632c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c197b70 tx=0x7f8f88017000 comp rx=0 tx=0).stop 2024-04-23T18:34:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.902+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1251051454 shutdown_connections 2024-04-23T18:34:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.902+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/1251051454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df963800 0x55b7df965bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.902+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/1251051454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958f40 0x55b7df9632c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.902+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/1251051454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 0x55b7df958a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.902+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1251051454 >> 172.21.15.50:0/1251051454 conn(0x55b7df954b30 msgr2=0x55b7df954f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.903+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1251051454 shutdown_connections 2024-04-23T18:34:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.903+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1251051454 wait complete. 2024-04-23T18:34:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.903+0000 7f8f96fc5740 1 -- start start 2024-04-23T18:34:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.903+0000 7f8f96fc5740 1 Processor -- start 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.904+0000 7f8f96fc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958620 0x55b7df9630b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.904+0000 7f8f96fc5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 0x55b7df9603a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.904+0000 7f8f96fc5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df963800 0x55b7df9608e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.904+0000 7f8f91f72640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958620 0x55b7df9630b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.904+0000 7f8f91771640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 0x55b7df9603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.904+0000 7f8f91f72640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958620 0x55b7df9630b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:52012/0 (socket says 172.21.15.50:52012) 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.904+0000 7f8f91771640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 0x55b7df9603a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58302/0 (socket says 172.21.15.50:58302) 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.904+0000 7f8f91f72640 1 -- 172.21.15.50:0/867068605 learned_addr learned my addr 172.21.15.50:0/867068605 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.904+0000 7f8f92773640 1 --2- 172.21.15.50:0/867068605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df963800 0x55b7df9608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.905+0000 7f8f96fc5740 1 -- 172.21.15.50:0/867068605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b7df966430 con 0x55b7df958f40 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.905+0000 7f8f96fc5740 1 -- 172.21.15.50:0/867068605 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b7df8580b0 con 0x55b7df958620 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.905+0000 7f8f96fc5740 1 -- 172.21.15.50:0/867068605 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b7df9665b0 con 0x55b7df963800 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.905+0000 7f8f96fc5740 10 client.0 _ll_register_callbacks cb 0x55b7df845d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:34:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.905+0000 7f8f91f72640 1 -- 172.21.15.50:0/867068605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df963800 msgr2=0x55b7df9608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.905+0000 7f8f91f72640 1 --2- 172.21.15.50:0/867068605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df963800 0x55b7df9608e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.905+0000 7f8f91f72640 1 -- 172.21.15.50:0/867068605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 msgr2=0x55b7df9603a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.905+0000 7f8f91f72640 1 --2- 172.21.15.50:0/867068605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 0x55b7df9603a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.905+0000 7f8f91f72640 1 -- 172.21.15.50:0/867068605 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7df69c620 con 0x55b7df958620 2024-04-23T18:34:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.905+0000 7f8f91f72640 1 --2- 172.21.15.50:0/867068605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958620 0x55b7df9630b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c0099f0 tx=0x7f8f7c0042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e1e527b78fad3cf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.905+0000 7f8f757fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:34:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.906+0000 7f8f757fa640 1 -- 172.21.15.50:0/867068605 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c035040 con 0x55b7df958620 2024-04-23T18:34:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.906+0000 7f8f757fa640 1 -- 172.21.15.50:0/867068605 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f8f7c005250 con 0x55b7df958620 2024-04-23T18:34:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.906+0000 7f8f96fc5740 1 -- 172.21.15.50:0/867068605 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f88017070 con 0x55b7df958620 2024-04-23T18:34:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.906+0000 7f8f757fa640 1 -- 172.21.15.50:0/867068605 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f8f7c005450 con 0x55b7df958620 2024-04-23T18:34:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.906+0000 7f8f96fc5740 10 client.26963 fetch_fsmap learned FSMap version 54 2024-04-23T18:34:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.906+0000 7f8f96fc5740 1 -- 172.21.15.50:0/867068605 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55b7df9722a0 con 0x55b7df958620 2024-04-23T18:34:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.906+0000 7f8f757fa640 1 -- 172.21.15.50:0/867068605 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f7c0055f0 con 0x55b7df958620 2024-04-23T18:34:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.906+0000 7f8f757fa640 1 -- 172.21.15.50:0/867068605 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f8f7c003a10 con 0x55b7df958620 2024-04-23T18:34:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.907+0000 7f8f96fc5740 10 client.26963 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:34:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.907+0000 7f8f96fc5740 10 client.26963 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:34:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.907+0000 7f8f96fc5740 20 client.26963 populate_metadata read hostname 'smithi050' 2024-04-23T18:34:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.907+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/867068605 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b7df973820 0x55b7df975ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.907+0000 7f8f96fc5740 4 client.26963 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.907+0000 7f8f96fc5740 1 -- 172.21.15.50:0/867068605 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8f88017070 con 0x55b7df973820 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.907+0000 7f8f91771640 1 --2- 172.21.15.50:0/867068605 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b7df973820 0x55b7df975ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.908+0000 7f8f91771640 1 --2- 172.21.15.50:0/867068605 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b7df973820 0x55b7df975ca0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9ed0168863eb434 server_cookie=b2f05fef3a960e78 in_seq=0 out_seq=0 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.908+0000 7f8f757fa640 10 client.26963 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f757fa640 1 -- 172.21.15.50:0/867068605 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f8f88017070 con 0x55b7df973820 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f757fa640 10 client.26963 handle_command_reply: tid=0 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f96fc5740 1 client.26963 shutdown 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f96fc5740 1 -- 172.21.15.50:0/867068605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958620 msgr2=0x55b7df9630b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/867068605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958620 0x55b7df9630b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c0099f0 tx=0x7f8f7c0042e0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f96fc5740 1 -- 172.21.15.50:0/867068605 shutdown_connections 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/867068605 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b7df973820 0x55b7df975ca0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/867068605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df963800 0x55b7df9608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f757fa640 0 client.26963 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/867068605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 0x55b7df9603a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/867068605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df958620 0x55b7df9630b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f96fc5740 1 -- 172.21.15.50:0/867068605 >> 172.21.15.50:0/867068605 conn(0x55b7df954b30 msgr2=0x55b7df95a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.909+0000 7f8f96fc5740 1 -- 172.21.15.50:0/867068605 shutdown_connections 2024-04-23T18:34:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.910+0000 7f8f96fc5740 1 -- 172.21.15.50:0/867068605 wait complete. 2024-04-23T18:34:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.910+0000 7f8f96fc5740 20 client.26963 trim_cache size 0 max 16384 2024-04-23T18:34:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f96fc5740 1 Processor -- start 2024-04-23T18:34:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f96fc5740 1 -- start start 2024-04-23T18:34:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f96fc5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958620 0x55b7df96f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f96fc5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958f40 0x55b7df95baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f96fc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df963800 0x55b7df95a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f96fc5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8f700110f0 con 0x55b7df958620 2024-04-23T18:34:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f96fc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7000e960 con 0x55b7df963800 2024-04-23T18:34:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f91f72640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958620 0x55b7df96f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f96fc5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7000c7d0 con 0x55b7df958f40 2024-04-23T18:34:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f91f72640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958620 0x55b7df96f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58314/0 (socket says 172.21.15.50:58314) 2024-04-23T18:34:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f91f72640 1 -- 172.21.15.50:0/2914104708 learned_addr learned my addr 172.21.15.50:0/2914104708 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f91771640 1 --2- 172.21.15.50:0/2914104708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958f40 0x55b7df95baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.929+0000 7f8f92773640 1 --2- 172.21.15.50:0/2914104708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df963800 0x55b7df95a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.930+0000 7f8f91f72640 1 -- 172.21.15.50:0/2914104708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958f40 msgr2=0x55b7df95baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.930+0000 7f8f91f72640 1 --2- 172.21.15.50:0/2914104708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958f40 0x55b7df95baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.930+0000 7f8f91f72640 1 -- 172.21.15.50:0/2914104708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df963800 msgr2=0x55b7df95a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.930+0000 7f8f91f72640 1 --2- 172.21.15.50:0/2914104708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df963800 0x55b7df95a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.930+0000 7f8f91f72640 1 -- 172.21.15.50:0/2914104708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7df9599f0 con 0x55b7df958620 2024-04-23T18:34:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.930+0000 7f8f91f72640 1 --2- 172.21.15.50:0/2914104708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958620 0x55b7df96f2f0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009a20 tx=0x7f8f7c0052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90293adec510531 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.930+0000 7f8f777fe640 1 -- 172.21.15.50:0/2914104708 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c051070 con 0x55b7df958620 2024-04-23T18:34:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.930+0000 7f8f777fe640 1 -- 172.21.15.50:0/2914104708 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f7c0316a0 con 0x55b7df958620 2024-04-23T18:34:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.930+0000 7f8f777fe640 1 -- 172.21.15.50:0/2914104708 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c035030 con 0x55b7df958620 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.931+0000 7f8f96fc5740 1 -- 172.21.15.50:0/2914104708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958620 msgr2=0x55b7df96f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.931+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/2914104708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958620 0x55b7df96f2f0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009a20 tx=0x7f8f7c0052d0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.931+0000 7f8f96fc5740 1 -- 172.21.15.50:0/2914104708 shutdown_connections 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.931+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/2914104708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df963800 0x55b7df95a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.931+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/2914104708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958f40 0x55b7df95baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.931+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/2914104708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958620 0x55b7df96f2f0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.931+0000 7f8f96fc5740 1 -- 172.21.15.50:0/2914104708 >> 172.21.15.50:0/2914104708 conn(0x55b7df954b30 msgr2=0x55b7df959fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.931+0000 7f8f96fc5740 1 -- 172.21.15.50:0/2914104708 shutdown_connections 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.931+0000 7f8f96fc5740 1 -- 172.21.15.50:0/2914104708 wait complete. 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.932+0000 7f8f96fc5740 1 -- start start 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.932+0000 7f8f96fc5740 1 Processor -- start 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.933+0000 7f8f96fc5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 0x55b7df96b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.933+0000 7f8f96fc5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 0x55b7df966de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.933+0000 7f8f91f72640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 0x55b7df96b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.933+0000 7f8f91f72640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 0x55b7df96b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:38488/0 (socket says 172.21.15.50:38488) 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.933+0000 7f8f91771640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 0x55b7df966de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.933+0000 7f8f91771640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 0x55b7df966de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58322/0 (socket says 172.21.15.50:58322) 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.933+0000 7f8f96fc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df963800 0x55b7df967320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.933+0000 7f8f91f72640 1 -- 172.21.15.50:0/1521960468 learned_addr learned my addr 172.21.15.50:0/1521960468 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.933+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1521960468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7000ce20 con 0x55b7df958f40 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.933+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1521960468 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7000e290 con 0x55b7df963800 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.933+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1521960468 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7000d4f0 con 0x55b7df958620 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f92773640 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df963800 0x55b7df967320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f96fc5740 10 client.0 _ll_register_callbacks cb 0x55b7df845d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f91771640 1 -- 172.21.15.50:0/1521960468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 msgr2=0x55b7df96b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f91771640 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 0x55b7df96b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f91771640 1 -- 172.21.15.50:0/1521960468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df963800 msgr2=0x55b7df967320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f91771640 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df963800 0x55b7df967320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f91771640 1 -- 172.21.15.50:0/1521960468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7df967860 con 0x55b7df958f40 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f92773640 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df963800 0x55b7df967320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f91f72640 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 0x55b7df96b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f91771640 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 0x55b7df966de0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f8f880255e0 tx=0x7f8f88025640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6adf028802c4092e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f757fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f757fa640 1 -- 172.21.15.50:0/1521960468 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f88039060 con 0x55b7df958f40 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1521960468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f7c0316a0 con 0x55b7df958f40 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f757fa640 1 -- 172.21.15.50:0/1521960468 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f8f88005450 con 0x55b7df958f40 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f757fa640 1 -- 172.21.15.50:0/1521960468 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f8f88025770 con 0x55b7df958f40 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f96fc5740 10 client.17469 fetch_fsmap learned FSMap version 54 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.934+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1521960468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55b7df9d9aa0 con 0x55b7df958f40 2024-04-23T18:34:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.935+0000 7f8f757fa640 1 -- 172.21.15.50:0/1521960468 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f7c0316a0 con 0x55b7df958f40 2024-04-23T18:34:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.935+0000 7f8f757fa640 1 -- 172.21.15.50:0/1521960468 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f8f88034470 con 0x55b7df958f40 2024-04-23T18:34:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.935+0000 7f8f96fc5740 10 client.17469 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:34:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.935+0000 7f8f96fc5740 10 client.17469 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:34:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.935+0000 7f8f96fc5740 20 client.17469 populate_metadata read hostname 'smithi050' 2024-04-23T18:34:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.935+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b7df973820 0x55b7df9dbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.935+0000 7f8f96fc5740 4 client.17469 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:34:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.935+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1521960468 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b7df8cb390 con 0x55b7df973820 2024-04-23T18:34:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.935+0000 7f8f91f72640 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b7df973820 0x55b7df9dbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.935+0000 7f8f757fa640 10 client.17469 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:43.936+0000 7f8f91f72640 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b7df973820 0x55b7df9dbfd0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49d60524dc80be08 server_cookie=dc7e7ca3f65b6611 in_seq=0 out_seq=0 2024-04-23T18:34:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f757fa640 1 -- 172.21.15.50:0/1521960468 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b7df8cb390 con 0x55b7df973820 2024-04-23T18:34:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f757fa640 10 client.17469 handle_command_reply: tid=0 2024-04-23T18:34:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f96fc5740 1 client.17469 shutdown 2024-04-23T18:34:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1521960468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 msgr2=0x55b7df966de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 0x55b7df966de0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f8f880255e0 tx=0x7f8f88025640 comp rx=0 tx=0).stop 2024-04-23T18:34:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1521960468 shutdown_connections 2024-04-23T18:34:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b7df973820 0x55b7df9dbfd0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7df963800 0x55b7df967320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b7df958f40 0x55b7df966de0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f757fa640 0 client.17469 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f96fc5740 1 --2- 172.21.15.50:0/1521960468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b7df958620 0x55b7df96b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1521960468 >> 172.21.15.50:0/1521960468 conn(0x55b7df954b30 msgr2=0x55b7df95ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1521960468 shutdown_connections 2024-04-23T18:34:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.279+0000 7f8f96fc5740 1 -- 172.21.15.50:0/1521960468 wait complete. 2024-04-23T18:34:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.280+0000 7f8f96fc5740 20 client.17469 trim_cache size 0 max 16384 2024-04-23T18:34:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:34:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:34:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:34:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:34:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.280+0000 7f8f941d6640 1 -- 172.21.15.50:0/868339900 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f8f780a3f60 msgr2=0x7f8f780a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.280+0000 7f8f941d6640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f8f780a3f60 0x7f8f780a6420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8f880058c0 tx=0x7f8f88005850 comp rx=0 tx=0).stop 2024-04-23T18:34:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.280+0000 7f8f941d6640 1 -- 172.21.15.50:0/868339900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f8c196da0 msgr2=0x7f8f8c1aaec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.280+0000 7f8f941d6640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f8c196da0 0x7f8f8c1aaec0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8f84007ba0 tx=0x7f8f8400d0c0 comp rx=0 tx=0).stop 2024-04-23T18:34:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.280+0000 7f8f941d6640 1 -- 172.21.15.50:0/868339900 shutdown_connections 2024-04-23T18:34:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.280+0000 7f8f941d6640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f8f780a3f60 0x7f8f780a6420 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:44.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.281+0000 7f8f941d6640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f8c196da0 0x7f8f8c1aaec0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:44.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.281+0000 7f8f941d6640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f8c1054b0 0x7f8f8c196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:44.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.281+0000 7f8f941d6640 1 --2- 172.21.15.50:0/868339900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c1003e0 0x7f8f8c196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:44.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.281+0000 7f8f941d6640 1 -- 172.21.15.50:0/868339900 >> 172.21.15.50:0/868339900 conn(0x7f8f8c076f50 msgr2=0x7f8f8c103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:44.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.281+0000 7f8f941d6640 1 -- 172.21.15.50:0/868339900 shutdown_connections 2024-04-23T18:34:44.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:44.281+0000 7f8f941d6640 1 -- 172.21.15.50:0/868339900 wait complete. 2024-04-23T18:34:44.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:44 smithi177 ceph-mon[32308]: pgmap v513: 97 pgs: 97 active+clean; 984 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 842 op/s 2024-04-23T18:34:44.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 98588 31.93 MB/sec execute 78 sec latency 322.169 ms 2024-04-23T18:34:44.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:44 smithi188 ceph-mon[32694]: pgmap v513: 97 pgs: 97 active+clean; 984 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 842 op/s 2024-04-23T18:34:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:44 smithi050 ceph-mon[29187]: pgmap v513: 97 pgs: 97 active+clean; 984 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 842 op/s 2024-04-23T18:34:45.567 INFO:tasks.workunit.client.0.smithi050.stdout: 10 99299 32.11 MB/sec execute 79 sec latency 107.761 ms 2024-04-23T18:34:46.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:46 smithi177 ceph-mon[32308]: pgmap v514: 97 pgs: 97 active+clean; 984 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 496 op/s 2024-04-23T18:34:46.566 INFO:tasks.workunit.client.0.smithi050.stdout: 10 99814 32.05 MB/sec execute 80 sec latency 79.971 ms 2024-04-23T18:34:46.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:46 smithi188 ceph-mon[32694]: pgmap v514: 97 pgs: 97 active+clean; 984 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 496 op/s 2024-04-23T18:34:46.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:46 smithi050 ceph-mon[29187]: pgmap v514: 97 pgs: 97 active+clean; 984 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 496 op/s 2024-04-23T18:34:47.567 INFO:tasks.workunit.client.0.smithi050.stdout: 10 100465 31.97 MB/sec execute 81 sec latency 41.252 ms 2024-04-23T18:34:48.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:48 smithi177 ceph-mon[32308]: pgmap v515: 97 pgs: 97 active+clean; 993 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 14 MiB/s wr, 869 op/s 2024-04-23T18:34:48.567 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101239 32.08 MB/sec execute 82 sec latency 72.377 ms 2024-04-23T18:34:48.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:48 smithi188 ceph-mon[32694]: pgmap v515: 97 pgs: 97 active+clean; 993 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 14 MiB/s wr, 869 op/s 2024-04-23T18:34:48.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:48 smithi050 ceph-mon[29187]: pgmap v515: 97 pgs: 97 active+clean; 993 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 14 MiB/s wr, 869 op/s 2024-04-23T18:34:48.921 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.8 2024-04-23T18:34:48.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:34:49.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.053+0000 7feea2d5d640 1 -- 172.21.15.50:0/1858985043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9c1010a0 msgr2=0x7fee9c101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:49.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.053+0000 7feea2d5d640 1 --2- 172.21.15.50:0/1858985043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9c1010a0 0x7fee9c101480 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fee90009900 tx=0x7fee9002a410 comp rx=0 tx=0).stop 2024-04-23T18:34:49.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.053+0000 7feea2d5d640 1 -- 172.21.15.50:0/1858985043 shutdown_connections 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.053+0000 7feea2d5d640 1 --2- 172.21.15.50:0/1858985043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee9c106170 0x7fee9c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.053+0000 7feea2d5d640 1 --2- 172.21.15.50:0/1858985043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee9c101a50 0x7fee9c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.053+0000 7feea2d5d640 1 --2- 172.21.15.50:0/1858985043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9c1010a0 0x7fee9c101480 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.053+0000 7feea2d5d640 1 -- 172.21.15.50:0/1858985043 >> 172.21.15.50:0/1858985043 conn(0x7fee9c077f50 msgr2=0x7fee9c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.053+0000 7feea2d5d640 1 -- 172.21.15.50:0/1858985043 shutdown_connections 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.053+0000 7feea2d5d640 1 -- 172.21.15.50:0/1858985043 wait complete. 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.053+0000 7feea2d5d640 1 Processor -- start 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.053+0000 7feea2d5d640 1 -- start start 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea2d5d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9c1010a0 0x7fee9c199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea2d5d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee9c106170 0x7fee9c199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea2d5d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee9c19a060 0x7fee9c1a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea2d5d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c1110d0 con 0x7fee9c1010a0 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea2d5d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c110f50 con 0x7fee9c19a060 2024-04-23T18:34:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea2d5d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c111250 con 0x7fee9c106170 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea0af9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9c1010a0 0x7fee9c199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea0af9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9c1010a0 0x7fee9c199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58342/0 (socket says 172.21.15.50:58342) 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea0af9640 1 -- 172.21.15.50:0/325023582 learned_addr learned my addr 172.21.15.50:0/325023582 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea12fa640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee9c19a060 0x7fee9c1a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7fee9bfff640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee9c106170 0x7fee9c199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea0af9640 1 -- 172.21.15.50:0/325023582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee9c106170 msgr2=0x7fee9c199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea0af9640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee9c106170 0x7fee9c199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea0af9640 1 -- 172.21.15.50:0/325023582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee9c19a060 msgr2=0x7fee9c1a4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea0af9640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee9c19a060 0x7fee9c1a4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea0af9640 1 -- 172.21.15.50:0/325023582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9c1a4de0 con 0x7fee9c1010a0 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea0af9640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9c1010a0 0x7fee9c199590 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fee90005940 tx=0x7fee900040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7fee99ffb640 1 -- 172.21.15.50:0/325023582 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9003f030 con 0x7fee9c1010a0 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7fee99ffb640 1 -- 172.21.15.50:0/325023582 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee900044c0 con 0x7fee9c1010a0 2024-04-23T18:34:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7feea2d5d640 1 -- 172.21.15.50:0/325023582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee9c1a5070 con 0x7fee9c1010a0 2024-04-23T18:34:49.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.054+0000 7fee99ffb640 1 -- 172.21.15.50:0/325023582 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee90035070 con 0x7fee9c1010a0 2024-04-23T18:34:49.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.055+0000 7feea2d5d640 1 -- 172.21.15.50:0/325023582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee9c1a5560 con 0x7fee9c1010a0 2024-04-23T18:34:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.056+0000 7feea2d5d640 1 -- 172.21.15.50:0/325023582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee9c068d70 con 0x7fee9c1010a0 2024-04-23T18:34:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.057+0000 7fee99ffb640 1 -- 172.21.15.50:0/325023582 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fee90042030 con 0x7fee9c1010a0 2024-04-23T18:34:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.057+0000 7fee99ffb640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee8c0a3d20 0x7fee8c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.058+0000 7fee9bfff640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee8c0a3d20 0x7fee8c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.058+0000 7fee99ffb640 1 -- 172.21.15.50:0/325023582 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7fee900f0b50 con 0x7fee9c1010a0 2024-04-23T18:34:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.058+0000 7fee9bfff640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee8c0a3d20 0x7fee8c0a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fee9c069c30 tx=0x7fee88005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:49.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.062+0000 7fee99ffb640 1 -- 172.21.15.50:0/325023582 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee900bcb90 con 0x7fee9c1010a0 2024-04-23T18:34:49.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.353+0000 7feea2d5d640 1 -- 172.21.15.50:0/325023582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee9c1093a0 con 0x7fee9c1010a0 2024-04-23T18:34:49.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.354+0000 7fee99ffb640 1 -- 172.21.15.50:0/325023582 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+11881 (secure 0 0 0) 0x7fee900c0e90 con 0x7fee9c1010a0 2024-04-23T18:34:49.346 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:34:49.347 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":54,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:34:05.578646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26753":{"gid":26753,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1119979694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1119979694},{"type":"v1","addr":"172.21.15.177:6837","nonce":1119979694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:34:49.347 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 54 2024-04-23T18:34:49.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.356+0000 7feea2d5d640 1 -- 172.21.15.50:0/325023582 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee8c0a3d20 msgr2=0x7fee8c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:49.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.356+0000 7feea2d5d640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee8c0a3d20 0x7fee8c0a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fee9c069c30 tx=0x7fee88005f90 comp rx=0 tx=0).stop 2024-04-23T18:34:49.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.356+0000 7feea2d5d640 1 -- 172.21.15.50:0/325023582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9c1010a0 msgr2=0x7fee9c199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:49.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.356+0000 7feea2d5d640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9c1010a0 0x7fee9c199590 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fee90005940 tx=0x7fee900040c0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.356+0000 7feea2d5d640 1 -- 172.21.15.50:0/325023582 shutdown_connections 2024-04-23T18:34:49.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.356+0000 7feea2d5d640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fee8c0a3d20 0x7fee8c0a61e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.356+0000 7feea2d5d640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee9c19a060 0x7fee9c1a4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.356+0000 7feea2d5d640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee9c106170 0x7fee9c199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.356+0000 7feea2d5d640 1 --2- 172.21.15.50:0/325023582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9c1010a0 0x7fee9c199590 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.356+0000 7feea2d5d640 1 -- 172.21.15.50:0/325023582 >> 172.21.15.50:0/325023582 conn(0x7fee9c077f50 msgr2=0x7fee9c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:49.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.356+0000 7feea2d5d640 1 -- 172.21.15.50:0/325023582 shutdown_connections 2024-04-23T18:34:49.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.356+0000 7feea2d5d640 1 -- 172.21.15.50:0/325023582 wait complete. 2024-04-23T18:34:49.363 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 79.8 --await-for 42.0' 2024-04-23T18:34:49.363 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 79.8 --await-for 42.0 2024-04-23T18:34:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.538+0000 7f81985cd640 1 -- 172.21.15.50:0/172131482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8190101080 msgr2=0x7f8190106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.538+0000 7f81985cd640 1 --2- 172.21.15.50:0/172131482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8190101080 0x7f8190106bb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8180009900 tx=0x7f818002a3f0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.539+0000 7f81985cd640 1 -- 172.21.15.50:0/172131482 shutdown_connections 2024-04-23T18:34:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.539+0000 7f81985cd640 1 --2- 172.21.15.50:0/172131482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81901072e0 0x7f819010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.539+0000 7f81985cd640 1 --2- 172.21.15.50:0/172131482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8190101080 0x7f8190106bb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.539+0000 7f81985cd640 1 --2- 172.21.15.50:0/172131482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81901006d0 0x7f8190100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.539+0000 7f81985cd640 1 -- 172.21.15.50:0/172131482 >> 172.21.15.50:0/172131482 conn(0x7f81900773f0 msgr2=0x7f81900fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:49.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.539+0000 7f81985cd640 1 -- 172.21.15.50:0/172131482 shutdown_connections 2024-04-23T18:34:49.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.539+0000 7f81985cd640 1 -- 172.21.15.50:0/172131482 wait complete. 2024-04-23T18:34:49.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.540+0000 7f81985cd640 1 Processor -- start 2024-04-23T18:34:49.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.540+0000 7f81985cd640 1 -- start start 2024-04-23T18:34:49.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.540+0000 7f81985cd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81901006d0 0x7f819019f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.540+0000 7f81985cd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81901072e0 0x7f819019f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.540+0000 7f81985cd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f819010e230 0x7f819010e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.540+0000 7f81985cd640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8190112190 con 0x7f81901006d0 2024-04-23T18:34:49.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.541+0000 7f81985cd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8190112010 con 0x7f81901072e0 2024-04-23T18:34:49.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.541+0000 7f81985cd640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8190112310 con 0x7f819010e230 2024-04-23T18:34:49.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.541+0000 7f8196369640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81901006d0 0x7f819019f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.541+0000 7f8196369640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81901006d0 0x7f819019f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58352/0 (socket says 172.21.15.50:58352) 2024-04-23T18:34:49.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.541+0000 7f8196369640 1 -- 172.21.15.50:0/2348594149 learned_addr learned my addr 172.21.15.50:0/2348594149 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:49.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.541+0000 7f8196b6a640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f819010e230 0x7f819010e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.541+0000 7f8195b68640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81901072e0 0x7f819019f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.542+0000 7f8196b6a640 1 -- 172.21.15.50:0/2348594149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81901072e0 msgr2=0x7f819019f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:49.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.542+0000 7f8196b6a640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81901072e0 0x7f819019f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.542+0000 7f8196b6a640 1 -- 172.21.15.50:0/2348594149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81901006d0 msgr2=0x7f819019f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:49.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.542+0000 7f8196b6a640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81901006d0 0x7f819019f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.542+0000 7f8196b6a640 1 -- 172.21.15.50:0/2348594149 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819019aeb0 con 0x7f819010e230 2024-04-23T18:34:49.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.542+0000 7f8196b6a640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f819010e230 0x7f819010e6e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8188009fd0 tx=0x7f818800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:49.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.542+0000 7f81877fe640 1 -- 172.21.15.50:0/2348594149 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818800ce20 con 0x7f819010e230 2024-04-23T18:34:49.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.543+0000 7f81985cd640 1 -- 172.21.15.50:0/2348594149 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f819019b1a0 con 0x7f819010e230 2024-04-23T18:34:49.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.544+0000 7f81985cd640 1 -- 172.21.15.50:0/2348594149 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f819019b6c0 con 0x7f819010e230 2024-04-23T18:34:49.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.545+0000 7f81877fe640 1 -- 172.21.15.50:0/2348594149 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8188005020 con 0x7f819010e230 2024-04-23T18:34:49.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.545+0000 7f81877fe640 1 -- 172.21.15.50:0/2348594149 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188010690 con 0x7f819010e230 2024-04-23T18:34:49.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.548+0000 7f81877fe640 1 -- 172.21.15.50:0/2348594149 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8188004a40 con 0x7f819010e230 2024-04-23T18:34:49.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.550+0000 7f819b3be740 1 Processor -- start 2024-04-23T18:34:49.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.551+0000 7f81877fe640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f817c0a3e20 0x7f817c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.552+0000 7f81877fe640 1 -- 172.21.15.50:0/2348594149 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f81880d0bc0 con 0x7f819010e230 2024-04-23T18:34:49.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.553+0000 7f819b3be740 1 -- start start 2024-04-23T18:34:49.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.553+0000 7f819b3be740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 0x55a5474d9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.553+0000 7f819b3be740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474d99c0 0x55a5474e3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.553+0000 7f819b3be740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 0x55a5474e6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.553+0000 7f819b3be740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a54743e370 con 0x55a5474d90a0 2024-04-23T18:34:49.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.553+0000 7f819b3be740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a5473db2e0 con 0x55a5474d99c0 2024-04-23T18:34:49.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.553+0000 7f819b3be740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a54743a330 con 0x55a5474e4090 2024-04-23T18:34:49.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.553+0000 7f8196b6a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 0x55a5474e6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.554+0000 7f8196b6a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 0x55a5474e6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:38528/0 (socket says 172.21.15.50:38528) 2024-04-23T18:34:49.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.554+0000 7f8196b6a640 1 -- 172.21.15.50:0/1406692304 learned_addr learned my addr 172.21.15.50:0/1406692304 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:49.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.554+0000 7f8196b6a640 1 -- 172.21.15.50:0/1406692304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474d99c0 msgr2=0x55a5474e3b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T18:34:49.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.554+0000 7f8196b6a640 1 --2- 172.21.15.50:0/1406692304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474d99c0 0x55a5474e3b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.554+0000 7f8196b6a640 1 -- 172.21.15.50:0/1406692304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 msgr2=0x55a5474d9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:49.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.554+0000 7f8196b6a640 1 --2- 172.21.15.50:0/1406692304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 0x55a5474d9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.554+0000 7f8196b6a640 1 -- 172.21.15.50:0/1406692304 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a54721cd80 con 0x55a5474e4090 2024-04-23T18:34:49.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.554+0000 7f8195b68640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f817c0a3e20 0x7f817c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.555+0000 7f8195b68640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f817c0a3e20 0x7f817c0a62e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8180009230 tx=0x7f8180031400 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:49.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.556+0000 7f8196b6a640 1 --2- 172.21.15.50:0/1406692304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 0x55a5474e6480 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f81880994c0 tx=0x7f81880a0f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e576914980a25941 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:49.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.556+0000 7f8184bfe640 1 -- 172.21.15.50:0/1406692304 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81880b6070 con 0x55a5474e4090 2024-04-23T18:34:49.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.557+0000 7f8184bfe640 1 -- 172.21.15.50:0/1406692304 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8188004d30 con 0x55a5474e4090 2024-04-23T18:34:49.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.557+0000 7f8184bfe640 1 -- 172.21.15.50:0/1406692304 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81880a5530 con 0x55a5474e4090 2024-04-23T18:34:49.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.557+0000 7f819b3be740 1 -- 172.21.15.50:0/1406692304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 msgr2=0x55a5474e6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:49.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.558+0000 7f819b3be740 1 --2- 172.21.15.50:0/1406692304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 0x55a5474e6480 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f81880994c0 tx=0x7f81880a0f60 comp rx=0 tx=0).stop 2024-04-23T18:34:49.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.558+0000 7f819b3be740 1 -- 172.21.15.50:0/1406692304 shutdown_connections 2024-04-23T18:34:49.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.558+0000 7f819b3be740 1 --2- 172.21.15.50:0/1406692304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 0x55a5474e6480 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.558+0000 7f819b3be740 1 --2- 172.21.15.50:0/1406692304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474d99c0 0x55a5474e3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.558+0000 7f819b3be740 1 --2- 172.21.15.50:0/1406692304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 0x55a5474d9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.558+0000 7f819b3be740 1 -- 172.21.15.50:0/1406692304 >> 172.21.15.50:0/1406692304 conn(0x55a5474d53c0 msgr2=0x55a5474d57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:49.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.558+0000 7f819b3be740 1 -- 172.21.15.50:0/1406692304 shutdown_connections 2024-04-23T18:34:49.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.558+0000 7f819b3be740 1 -- 172.21.15.50:0/1406692304 wait complete. 2024-04-23T18:34:49.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.559+0000 7f819b3be740 1 -- start start 2024-04-23T18:34:49.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.559+0000 7f819b3be740 1 Processor -- start 2024-04-23T18:34:49.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.559+0000 7f819b3be740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 0x55a5474e10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.560+0000 7f819b3be740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474d99c0 0x55a5474e1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.560+0000 7f819b3be740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 0x55a5474e1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.560+0000 7f8195b68640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474d99c0 0x55a5474e1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.560+0000 7f8195b68640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474d99c0 0x55a5474e1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:38540/0 (socket says 172.21.15.50:38540) 2024-04-23T18:34:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.560+0000 7f8196b6a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 0x55a5474e1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.560+0000 7f8196b6a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 0x55a5474e1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:52048/0 (socket says 172.21.15.50:52048) 2024-04-23T18:34:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.560+0000 7f8196b6a640 1 -- 172.21.15.50:0/3898041489 learned_addr learned my addr 172.21.15.50:0/3898041489 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.561+0000 7f819b3be740 1 -- 172.21.15.50:0/3898041489 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a5474e6e80 con 0x55a5474d90a0 2024-04-23T18:34:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.561+0000 7f819b3be740 1 -- 172.21.15.50:0/3898041489 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a5473d8fb0 con 0x55a5474e4090 2024-04-23T18:34:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.561+0000 7f819b3be740 1 -- 172.21.15.50:0/3898041489 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a5474e7000 con 0x55a5474d99c0 2024-04-23T18:34:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.561+0000 7f819b3be740 10 client.0 _ll_register_callbacks cb 0x55a5473d89a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:34:49.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.562+0000 7f8196369640 1 --2- 172.21.15.50:0/3898041489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 0x55a5474e10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.563+0000 7f8196b6a640 1 -- 172.21.15.50:0/3898041489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474d99c0 msgr2=0x55a5474e1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:49.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.563+0000 7f8196b6a640 1 --2- 172.21.15.50:0/3898041489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474d99c0 0x55a5474e1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.563+0000 7f8196b6a640 1 -- 172.21.15.50:0/3898041489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 msgr2=0x55a5474e10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:49.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.563+0000 7f8196b6a640 1 --2- 172.21.15.50:0/3898041489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 0x55a5474e10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.563+0000 7f8196b6a640 1 -- 172.21.15.50:0/3898041489 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a54727d510 con 0x55a5474e4090 2024-04-23T18:34:49.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.564+0000 7f8196b6a640 1 --2- 172.21.15.50:0/3898041489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 0x55a5474e1b70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f818800c760 tx=0x7f818809c170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a7705414e19cf4c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:49.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.564+0000 7f8179ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:34:49.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.565+0000 7f8179ffb640 1 -- 172.21.15.50:0/3898041489 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81880b6040 con 0x55a5474e4090 2024-04-23T18:34:49.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.565+0000 7f8179ffb640 1 -- 172.21.15.50:0/3898041489 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f8188098a50 con 0x55a5474e4090 2024-04-23T18:34:49.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.566+0000 7f819b3be740 1 -- 172.21.15.50:0/3898041489 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8188004d30 con 0x55a5474e4090 2024-04-23T18:34:49.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.567+0000 7f8179ffb640 1 -- 172.21.15.50:0/3898041489 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f81880992f0 con 0x55a5474e4090 2024-04-23T18:34:49.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.570+0000 7f819b3be740 10 client.26984 fetch_fsmap learned FSMap version 54 2024-04-23T18:34:49.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.570+0000 7f819b3be740 1 -- 172.21.15.50:0/3898041489 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55a547110670 con 0x55a5474e4090 2024-04-23T18:34:49.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.571+0000 7f8179ffb640 1 -- 172.21.15.50:0/3898041489 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81880a2030 con 0x55a5474e4090 2024-04-23T18:34:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.571+0000 7f8179ffb640 1 -- 172.21.15.50:0/3898041489 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f81880870b0 con 0x55a5474e4090 2024-04-23T18:34:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.572+0000 7f819b3be740 10 client.26984 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:34:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.572+0000 7f819b3be740 10 client.26984 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:34:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.572+0000 7f819b3be740 20 client.26984 populate_metadata read hostname 'smithi050' 2024-04-23T18:34:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.572+0000 7f819b3be740 1 --2- 172.21.15.50:0/3898041489 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a5474f41d0 0x55a5474f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.572+0000 7f819b3be740 4 client.26984 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:34:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.572+0000 7f819b3be740 1 -- 172.21.15.50:0/3898041489 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8188004d30 con 0x55a5474f41d0 2024-04-23T18:34:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.572+0000 7f8196369640 1 --2- 172.21.15.50:0/3898041489 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a5474f41d0 0x55a5474f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.573+0000 7f8179ffb640 10 client.26984 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.573+0000 7f8196369640 1 --2- 172.21.15.50:0/3898041489 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a5474f41d0 0x55a5474f65c0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fc872c842ae29b6 server_cookie=86637b52c0df32bd in_seq=0 out_seq=0 2024-04-23T18:34:49.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.574+0000 7f8179ffb640 1 -- 172.21.15.50:0/3898041489 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f8188004d30 con 0x55a5474f41d0 2024-04-23T18:34:49.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.574+0000 7f8179ffb640 10 client.26984 handle_command_reply: tid=0 2024-04-23T18:34:49.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.574+0000 7f819b3be740 1 client.26984 shutdown 2024-04-23T18:34:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.575+0000 7f819b3be740 1 -- 172.21.15.50:0/3898041489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 msgr2=0x55a5474e1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.575+0000 7f819b3be740 1 --2- 172.21.15.50:0/3898041489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 0x55a5474e1b70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f818800c760 tx=0x7f818809c170 comp rx=0 tx=0).stop 2024-04-23T18:34:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.575+0000 7f819b3be740 1 -- 172.21.15.50:0/3898041489 shutdown_connections 2024-04-23T18:34:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.575+0000 7f819b3be740 1 --2- 172.21.15.50:0/3898041489 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a5474f41d0 0x55a5474f65c0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.575+0000 7f819b3be740 1 --2- 172.21.15.50:0/3898041489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 0x55a5474e1b70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.575+0000 7f819b3be740 1 --2- 172.21.15.50:0/3898041489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474d99c0 0x55a5474e1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.575+0000 7f819b3be740 1 --2- 172.21.15.50:0/3898041489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 0x55a5474e10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.575+0000 7f819b3be740 1 -- 172.21.15.50:0/3898041489 >> 172.21.15.50:0/3898041489 conn(0x55a5474d53c0 msgr2=0x55a5474d4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.575+0000 7f8179ffb640 0 client.26984 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:49.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101818 32.10 MB/sec execute 83 sec latency 178.324 ms 2024-04-23T18:34:49.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.575+0000 7f819b3be740 1 -- 172.21.15.50:0/3898041489 shutdown_connections 2024-04-23T18:34:49.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.575+0000 7f819b3be740 1 -- 172.21.15.50:0/3898041489 wait complete. 2024-04-23T18:34:49.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.575+0000 7f819b3be740 20 client.26984 trim_cache size 0 max 16384 2024-04-23T18:34:49.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.610+0000 7f819b3be740 1 Processor -- start 2024-04-23T18:34:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.610+0000 7f819b3be740 1 -- start start 2024-04-23T18:34:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.610+0000 7f819b3be740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474d90a0 0x55a5474db840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.610+0000 7f819b3be740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d99c0 0x55a5474d4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.610+0000 7f819b3be740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 0x55a5474d49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.610+0000 7f819b3be740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f81740111a0 con 0x55a5474d99c0 2024-04-23T18:34:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.610+0000 7f819b3be740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f817400ea10 con 0x55a5474e4090 2024-04-23T18:34:49.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.610+0000 7f819b3be740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f817400c880 con 0x55a5474d90a0 2024-04-23T18:34:49.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.611+0000 7f8196b6a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 0x55a5474d49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.611+0000 7f8196b6a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 0x55a5474d49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:52062/0 (socket says 172.21.15.50:52062) 2024-04-23T18:34:49.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.611+0000 7f8196b6a640 1 -- 172.21.15.50:0/2751956629 learned_addr learned my addr 172.21.15.50:0/2751956629 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:49.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.611+0000 7f8196369640 1 --2- 172.21.15.50:0/2751956629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474d90a0 0x55a5474db840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.612+0000 7f8196b6a640 1 -- 172.21.15.50:0/2751956629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474d90a0 msgr2=0x55a5474db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:49.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.612+0000 7f8196b6a640 1 --2- 172.21.15.50:0/2751956629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474d90a0 0x55a5474db840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.612+0000 7f8196b6a640 1 -- 172.21.15.50:0/2751956629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d99c0 msgr2=0x55a5474d4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:49.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.612+0000 7f8196b6a640 1 --2- 172.21.15.50:0/2751956629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d99c0 0x55a5474d4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.612+0000 7f8196b6a640 1 -- 172.21.15.50:0/2751956629 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5474d6550 con 0x55a5474e4090 2024-04-23T18:34:49.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.612+0000 7f8196b6a640 1 --2- 172.21.15.50:0/2751956629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 0x55a5474d49c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f818809c7f0 tx=0x7f8188009590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=523db77e5abe2106 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:49.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.612+0000 7f817bfff640 1 -- 172.21.15.50:0/2751956629 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818809c260 con 0x55a5474e4090 2024-04-23T18:34:49.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.612+0000 7f817bfff640 1 -- 172.21.15.50:0/2751956629 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8188004d30 con 0x55a5474e4090 2024-04-23T18:34:49.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.613+0000 7f817bfff640 1 -- 172.21.15.50:0/2751956629 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81880870b0 con 0x55a5474e4090 2024-04-23T18:34:49.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.614+0000 7f819b3be740 1 -- 172.21.15.50:0/2751956629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 msgr2=0x55a5474d49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.614+0000 7f819b3be740 1 --2- 172.21.15.50:0/2751956629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 0x55a5474d49c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f818809c7f0 tx=0x7f8188009590 comp rx=0 tx=0).stop 2024-04-23T18:34:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.614+0000 7f819b3be740 1 -- 172.21.15.50:0/2751956629 shutdown_connections 2024-04-23T18:34:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.614+0000 7f819b3be740 1 --2- 172.21.15.50:0/2751956629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474e4090 0x55a5474d49c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.614+0000 7f819b3be740 1 --2- 172.21.15.50:0/2751956629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d99c0 0x55a5474d4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.614+0000 7f819b3be740 1 --2- 172.21.15.50:0/2751956629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474d90a0 0x55a5474db840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.614+0000 7f819b3be740 1 -- 172.21.15.50:0/2751956629 >> 172.21.15.50:0/2751956629 conn(0x55a5474d53c0 msgr2=0x55a5474d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.614+0000 7f819b3be740 1 -- 172.21.15.50:0/2751956629 shutdown_connections 2024-04-23T18:34:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.614+0000 7f819b3be740 1 -- 172.21.15.50:0/2751956629 wait complete. 2024-04-23T18:34:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.614+0000 7f819b3be740 1 -- start start 2024-04-23T18:34:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.614+0000 7f819b3be740 1 Processor -- start 2024-04-23T18:34:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.615+0000 7f819b3be740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 0x55a5474f8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.615+0000 7f819b3be740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474d99c0 0x55a5474f9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.615+0000 7f819b3be740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 0x55a5474f7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.615+0000 7f819b3be740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f817400ced0 con 0x55a5474d90a0 2024-04-23T18:34:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.615+0000 7f819b3be740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f817400e340 con 0x55a5474d99c0 2024-04-23T18:34:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.615+0000 7f819b3be740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f817400d5a0 con 0x55a5474e4090 2024-04-23T18:34:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.615+0000 7f819b3be740 10 client.0 _ll_register_callbacks cb 0x7f81880a58a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:34:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.615+0000 7f8196b6a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 0x55a5474f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.615+0000 7f8196b6a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 0x55a5474f7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:38562/0 (socket says 172.21.15.50:38562) 2024-04-23T18:34:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.615+0000 7f8196b6a640 1 -- 172.21.15.50:0/2266640120 learned_addr learned my addr 172.21.15.50:0/2266640120 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.615+0000 7f8195b68640 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474d99c0 0x55a5474f9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.615+0000 7f8196369640 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 0x55a5474f8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.616+0000 7f8196b6a640 1 -- 172.21.15.50:0/2266640120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474d99c0 msgr2=0x55a5474f9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.616+0000 7f8196b6a640 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474d99c0 0x55a5474f9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.616+0000 7f8196b6a640 1 -- 172.21.15.50:0/2266640120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 msgr2=0x55a5474f8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.616+0000 7f8196b6a640 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 0x55a5474f8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.616+0000 7f8196b6a640 1 -- 172.21.15.50:0/2266640120 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5474f7b00 con 0x55a5474e4090 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.616+0000 7f8196369640 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 0x55a5474f8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.616+0000 7f8196b6a640 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 0x55a5474f7410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f818809c5c0 tx=0x7f8188005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b7fb0fb0a7b607a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.616+0000 7f8179ffb640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.616+0000 7f8179ffb640 1 -- 172.21.15.50:0/2266640120 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81880870b0 con 0x55a5474e4090 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.616+0000 7f8179ffb640 1 -- 172.21.15.50:0/2266640120 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f8188087250 con 0x55a5474e4090 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.616+0000 7f819b3be740 1 -- 172.21.15.50:0/2266640120 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8188004d30 con 0x55a5474e4090 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.617+0000 7f8179ffb640 1 -- 172.21.15.50:0/2266640120 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f81880ceb30 con 0x55a5474e4090 2024-04-23T18:34:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.617+0000 7f819b3be740 10 client.27034 fetch_fsmap learned FSMap version 54 2024-04-23T18:34:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.617+0000 7f819b3be740 1 -- 172.21.15.50:0/2266640120 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55a5474f7b00 con 0x55a5474e4090 2024-04-23T18:34:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.617+0000 7f8179ffb640 1 -- 172.21.15.50:0/2266640120 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8188004d30 con 0x55a5474e4090 2024-04-23T18:34:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.618+0000 7f8179ffb640 1 -- 172.21.15.50:0/2266640120 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f81880a2030 con 0x55a5474e4090 2024-04-23T18:34:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.618+0000 7f819b3be740 10 client.27034 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:34:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.618+0000 7f819b3be740 10 client.27034 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:34:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.618+0000 7f819b3be740 20 client.27034 populate_metadata read hostname 'smithi050' 2024-04-23T18:34:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.618+0000 7f819b3be740 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a54755b180 0x55a54755d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.618+0000 7f819b3be740 4 client.27034 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.8, "await_for": 42.0}] 2024-04-23T18:34:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.618+0000 7f819b3be740 1 -- 172.21.15.50:0/2266640120 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.8, "await_for": 42.0}) -- 0x55a54744bd90 con 0x55a54755b180 2024-04-23T18:34:49.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.618+0000 7f8196369640 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a54755b180 0x55a54755d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:49.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.618+0000 7f8179ffb640 10 client.27034 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:49.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:49.618+0000 7f8196369640 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a54755b180 0x55a54755d570 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba0463d7045ead25 server_cookie=5136c818823bba7c in_seq=0 out_seq=0 2024-04-23T18:34:50.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f8179ffb640 1 -- 172.21.15.50:0/2266640120 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55a54744bd90 con 0x55a54755b180 2024-04-23T18:34:50.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f8179ffb640 10 client.27034 handle_command_reply: tid=0 2024-04-23T18:34:50.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f819b3be740 1 client.27034 shutdown 2024-04-23T18:34:50.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f819b3be740 1 -- 172.21.15.50:0/2266640120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 msgr2=0x55a5474f7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:50.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f819b3be740 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 0x55a5474f7410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f818809c5c0 tx=0x7f8188005e80 comp rx=0 tx=0).stop 2024-04-23T18:34:50.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f819b3be740 1 -- 172.21.15.50:0/2266640120 shutdown_connections 2024-04-23T18:34:50.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f819b3be740 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a54755b180 0x55a54755d570 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f819b3be740 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a5474e4090 0x55a5474f7410 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f8179ffb640 0 client.27034 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f819b3be740 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5474d99c0 0x55a5474f9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f819b3be740 1 --2- 172.21.15.50:0/2266640120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a5474d90a0 0x55a5474f8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f819b3be740 1 -- 172.21.15.50:0/2266640120 >> 172.21.15.50:0/2266640120 conn(0x55a5474d53c0 msgr2=0x55a5474d4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f819b3be740 1 -- 172.21.15.50:0/2266640120 shutdown_connections 2024-04-23T18:34:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f819b3be740 1 -- 172.21.15.50:0/2266640120 wait complete. 2024-04-23T18:34:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.221+0000 7f819b3be740 20 client.27034 trim_cache size 0 max 16384 2024-04-23T18:34:50.215 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:34:50.215 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 54, 2024-04-23T18:34:50.215 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 54, 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "8c92f231": { 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 54, 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 79.8, 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:34:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:34:50.217 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:34:50.217 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:34:50.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:34:50.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:34:50.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:34:50.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:34:50.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:34:50.217 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:34:50.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.222+0000 7f81985cd640 1 -- 172.21.15.50:0/2348594149 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f817c0a3e20 msgr2=0x7f817c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:50.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.222+0000 7f81985cd640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f817c0a3e20 0x7f817c0a62e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8180009230 tx=0x7f8180031400 comp rx=0 tx=0).stop 2024-04-23T18:34:50.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.222+0000 7f81985cd640 1 -- 172.21.15.50:0/2348594149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f819010e230 msgr2=0x7f819010e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.222+0000 7f81985cd640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f819010e230 0x7f819010e6e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8188009fd0 tx=0x7f818800ef10 comp rx=0 tx=0).stop 2024-04-23T18:34:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.222+0000 7f81985cd640 1 -- 172.21.15.50:0/2348594149 shutdown_connections 2024-04-23T18:34:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.222+0000 7f81985cd640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f817c0a3e20 0x7f817c0a62e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.222+0000 7f81985cd640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f819010e230 0x7f819010e6e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.222+0000 7f81985cd640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81901072e0 0x7f819019f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.222+0000 7f81985cd640 1 --2- 172.21.15.50:0/2348594149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81901006d0 0x7f819019f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.222+0000 7f81985cd640 1 -- 172.21.15.50:0/2348594149 >> 172.21.15.50:0/2348594149 conn(0x7f81900773f0 msgr2=0x7f81901027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.222+0000 7f81985cd640 1 -- 172.21.15.50:0/2348594149 shutdown_connections 2024-04-23T18:34:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:34:50.222+0000 7f81985cd640 1 -- 172.21.15.50:0/2348594149 wait complete. 2024-04-23T18:34:50.225 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '8c92f231', quiesce took 1.3 seconds. Will release after: 19.8 2024-04-23T18:34:50.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:50 smithi177 ceph-mon[32308]: pgmap v516: 97 pgs: 97 active+clean; 993 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 10 MiB/s wr, 630 op/s 2024-04-23T18:34:50.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:50 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/325023582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:34:50.567 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 31.82 MB/sec execute 84 sec latency 850.469 ms 2024-04-23T18:34:50.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:50 smithi188 ceph-mon[32694]: pgmap v516: 97 pgs: 97 active+clean; 993 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 10 MiB/s wr, 630 op/s 2024-04-23T18:34:50.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:50 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/325023582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:34:50.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:50 smithi050 ceph-mon[29187]: pgmap v516: 97 pgs: 97 active+clean; 993 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 10 MiB/s wr, 630 op/s 2024-04-23T18:34:50.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:50 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/325023582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:34:51.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 31.45 MB/sec execute 85 sec latency 1850.517 ms 2024-04-23T18:34:52.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:52 smithi050 ceph-mon[29187]: pgmap v517: 97 pgs: 97 active+clean; 993 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 10 MiB/s wr, 630 op/s 2024-04-23T18:34:52.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:52 smithi177 ceph-mon[32308]: pgmap v517: 97 pgs: 97 active+clean; 993 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 10 MiB/s wr, 630 op/s 2024-04-23T18:34:52.567 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 31.08 MB/sec execute 86 sec latency 2850.645 ms 2024-04-23T18:34:52.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:52 smithi188 ceph-mon[32694]: pgmap v517: 97 pgs: 97 active+clean; 993 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 10 MiB/s wr, 630 op/s 2024-04-23T18:34:53.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:53 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:34:53.567 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 30.72 MB/sec execute 87 sec latency 3850.696 ms 2024-04-23T18:34:53.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:53 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:34:53.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:53 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:34:54.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:54 smithi177 ceph-mon[32308]: pgmap v518: 97 pgs: 97 active+clean; 1009 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 14 MiB/s wr, 851 op/s 2024-04-23T18:34:54.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 30.37 MB/sec execute 88 sec latency 4850.768 ms 2024-04-23T18:34:54.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:54 smithi188 ceph-mon[32694]: pgmap v518: 97 pgs: 97 active+clean; 1009 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 14 MiB/s wr, 851 op/s 2024-04-23T18:34:54.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:54 smithi050 ceph-mon[29187]: pgmap v518: 97 pgs: 97 active+clean; 1009 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 14 MiB/s wr, 851 op/s 2024-04-23T18:34:55.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 30.03 MB/sec execute 89 sec latency 5850.823 ms 2024-04-23T18:34:56.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:56 smithi177 ceph-mon[32308]: pgmap v519: 97 pgs: 97 active+clean; 1009 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 593 op/s 2024-04-23T18:34:56.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 29.70 MB/sec execute 90 sec latency 6850.854 ms 2024-04-23T18:34:56.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:56 smithi188 ceph-mon[32694]: pgmap v519: 97 pgs: 97 active+clean; 1009 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 593 op/s 2024-04-23T18:34:56.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:56 smithi050 ceph-mon[29187]: pgmap v519: 97 pgs: 97 active+clean; 1009 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 593 op/s 2024-04-23T18:34:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.381+0000 7f56f9220640 1 -- 172.21.15.50:0/3175148871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56f41021c0 msgr2=0x7f56f41025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.381+0000 7f56f9220640 1 --2- 172.21.15.50:0/3175148871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56f41021c0 0x7f56f41025a0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009900 tx=0x7f56e402a410 comp rx=0 tx=0).stop 2024-04-23T18:34:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.381+0000 7f56f9220640 1 -- 172.21.15.50:0/3175148871 shutdown_connections 2024-04-23T18:34:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.381+0000 7f56f9220640 1 --2- 172.21.15.50:0/3175148871 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56f4107290 0x7f56f410ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.381+0000 7f56f9220640 1 --2- 172.21.15.50:0/3175148871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56f4102b70 0x7f56f4106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.381+0000 7f56f9220640 1 --2- 172.21.15.50:0/3175148871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56f41021c0 0x7f56f41025a0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.381+0000 7f56f9220640 1 -- 172.21.15.50:0/3175148871 >> 172.21.15.50:0/3175148871 conn(0x7f56f40fda20 msgr2=0x7f56f40ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.381+0000 7f56f9220640 1 -- 172.21.15.50:0/3175148871 shutdown_connections 2024-04-23T18:34:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.381+0000 7f56f9220640 1 -- 172.21.15.50:0/3175148871 wait complete. 2024-04-23T18:34:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.382+0000 7f56f9220640 1 Processor -- start 2024-04-23T18:34:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f9220640 1 -- start start 2024-04-23T18:34:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f9220640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56f41021c0 0x7f56f41986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f9220640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56f4107290 0x7f56f4198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f9220640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56f419cf40 0x7f56f419d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f9220640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f56f4112250 con 0x7f56f41021c0 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f9220640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f56f41120d0 con 0x7f56f4107290 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f9220640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f56f41123d0 con 0x7f56f419cf40 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56f41021c0 0x7f56f41986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56f41021c0 0x7f56f41986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34368/0 (socket says 172.21.15.50:34368) 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f2d9d640 1 -- 172.21.15.50:0/3006291863 learned_addr learned my addr 172.21.15.50:0/3006291863 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f359e640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56f419cf40 0x7f56f419d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f259c640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56f4107290 0x7f56f4198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f2d9d640 1 -- 172.21.15.50:0/3006291863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56f419cf40 msgr2=0x7f56f419d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56f419cf40 0x7f56f419d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f2d9d640 1 -- 172.21.15.50:0/3006291863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56f4107290 msgr2=0x7f56f4198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56f4107290 0x7f56f4198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f2d9d640 1 -- 172.21.15.50:0/3006291863 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f419da30 con 0x7f56f41021c0 2024-04-23T18:34:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f359e640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56f419cf40 0x7f56f419d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f259c640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56f4107290 0x7f56f4198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:34:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.383+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56f41021c0 0x7f56f41986d0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f56e4005ab0 tx=0x7f56e4003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.384+0000 7f56dbfff640 1 -- 172.21.15.50:0/3006291863 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e403f070 con 0x7f56f41021c0 2024-04-23T18:34:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.384+0000 7f56dbfff640 1 -- 172.21.15.50:0/3006291863 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56e4031d60 con 0x7f56f41021c0 2024-04-23T18:34:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.384+0000 7f56f9220640 1 -- 172.21.15.50:0/3006291863 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f419dcc0 con 0x7f56f41021c0 2024-04-23T18:34:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.384+0000 7f56dbfff640 1 -- 172.21.15.50:0/3006291863 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4035030 con 0x7f56f41021c0 2024-04-23T18:34:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.384+0000 7f56f9220640 1 -- 172.21.15.50:0/3006291863 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f41a5480 con 0x7f56f41021c0 2024-04-23T18:34:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.386+0000 7f56dbfff640 1 -- 172.21.15.50:0/3006291863 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f56e4042070 con 0x7f56f41021c0 2024-04-23T18:34:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56fc010740 1 Processor -- start 2024-04-23T18:34:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56fc010740 1 -- start start 2024-04-23T18:34:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56dbfff640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f56dc0a3dc0 0x7f56dc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56fc010740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f90394900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f359e640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f56dc0a3dc0 0x7f56dc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56fc010740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f90394e40 0x563f903a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56fc010740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f903a3c10 0x563f903a6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56fc010740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563f902f9970 con 0x563f90394e40 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56fc010740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563f902964a0 con 0x563f903a3c10 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56fc010740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563f902f5930 con 0x563f90394520 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56dbfff640 1 -- 172.21.15.50:0/3006291863 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f56e40f1010 con 0x7f56f41021c0 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f90394e40 0x563f903a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f90394e40 0x563f903a36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34380/0 (socket says 172.21.15.50:34380) 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f259c640 1 -- 172.21.15.50:0/1827945915 learned_addr learned my addr 172.21.15.50:0/1827945915 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/1827945915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f90394900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f359e640 1 --2- 172.21.15.50:0/1827945915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f903a3c10 0x563f903a6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f2d9d640 1 -- 172.21.15.50:0/1827945915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f903a3c10 msgr2=0x563f903a6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/1827945915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f903a3c10 0x563f903a6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f2d9d640 1 -- 172.21.15.50:0/1827945915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f90394e40 msgr2=0x563f903a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/1827945915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f90394e40 0x563f903a36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f2d9d640 1 -- 172.21.15.50:0/1827945915 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f9000c210 con 0x563f90394520 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f259c640 1 --2- 172.21.15.50:0/1827945915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f90394e40 0x563f903a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.387+0000 7f56f359e640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f56dc0a3dc0 0x7f56dc0a6280 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0099e0 tx=0x7f56ec00fe70 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.388+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/1827945915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f90394900 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f56e40ee750 tx=0x7f56e402a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89ede53d1b14d3a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.388+0000 7f56d97fa640 1 -- 172.21.15.50:0/1827945915 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4049040 con 0x563f90394520 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.388+0000 7f56d97fa640 1 -- 172.21.15.50:0/1827945915 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56e40bd6a0 con 0x563f90394520 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.388+0000 7f56d97fa640 1 -- 172.21.15.50:0/1827945915 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e403b460 con 0x563f90394520 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.388+0000 7f56fc010740 1 -- 172.21.15.50:0/1827945915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 msgr2=0x563f90394900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.388+0000 7f56fc010740 1 --2- 172.21.15.50:0/1827945915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f90394900 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f56e40ee750 tx=0x7f56e402a410 comp rx=0 tx=0).stop 2024-04-23T18:34:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 -- 172.21.15.50:0/1827945915 shutdown_connections 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 --2- 172.21.15.50:0/1827945915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f903a3c10 0x563f903a6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 --2- 172.21.15.50:0/1827945915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f90394e40 0x563f903a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 --2- 172.21.15.50:0/1827945915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f90394900 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 -- 172.21.15.50:0/1827945915 >> 172.21.15.50:0/1827945915 conn(0x563f90390b30 msgr2=0x563f90390f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 -- 172.21.15.50:0/1827945915 shutdown_connections 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 -- 172.21.15.50:0/1827945915 wait complete. 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 -- start start 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 Processor -- start 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903a3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 0x563f903a0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f903a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563f903a6600 con 0x563f903a3c10 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563f902940b0 con 0x563f90394e40 2024-04-23T18:34:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563f903a6780 con 0x563f90394520 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56fc010740 10 client.0 _ll_register_callbacks cb 0x563f90281d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56f359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f903a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56f2d9d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903a3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56f359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f903a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34392/0 (socket says 172.21.15.50:34392) 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56f259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 0x563f903a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.389+0000 7f56f359e640 1 -- 172.21.15.50:0/3896971092 learned_addr learned my addr 172.21.15.50:0/3896971092 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56f359e640 1 -- 172.21.15.50:0/3896971092 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 msgr2=0x563f903a3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56f359e640 1 --2- 172.21.15.50:0/3896971092 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903a3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56f359e640 1 -- 172.21.15.50:0/3896971092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 msgr2=0x563f903a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56f359e640 1 --2- 172.21.15.50:0/3896971092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 0x563f903a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56f359e640 1 -- 172.21.15.50:0/3896971092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f900d8620 con 0x563f903a3c10 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56f359e640 1 --2- 172.21.15.50:0/3896971092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f903a0db0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f56ec003c80 tx=0x7f56ec004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f402cb13f6bcf75 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:34:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56d6ff5640 1 -- 172.21.15.50:0/3896971092 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec020840 con 0x563f903a3c10 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56d6ff5640 1 -- 172.21.15.50:0/3896971092 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f56ec0209e0 con 0x563f903a3c10 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56fc010740 1 -- 172.21.15.50:0/3896971092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56e40bd6a0 con 0x563f903a3c10 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56d6ff5640 1 -- 172.21.15.50:0/3896971092 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f56ec020be0 con 0x563f903a3c10 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56fc010740 10 client.17499 fetch_fsmap learned FSMap version 54 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.390+0000 7f56fc010740 1 -- 172.21.15.50:0/3896971092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x563f903b24f0 con 0x563f903a3c10 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.391+0000 7f56d6ff5640 1 -- 172.21.15.50:0/3896971092 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56e40bd6a0 con 0x563f903a3c10 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.391+0000 7f56d6ff5640 1 -- 172.21.15.50:0/3896971092 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f56ec01f070 con 0x563f903a3c10 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.391+0000 7f56fc010740 10 client.17499 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.391+0000 7f56fc010740 10 client.17499 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.391+0000 7f56fc010740 20 client.17499 populate_metadata read hostname 'smithi050' 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.391+0000 7f56fc010740 1 --2- 172.21.15.50:0/3896971092 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563f903b3a20 0x563f903b5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.391+0000 7f56fc010740 4 client.17499 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:34:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.391+0000 7f56fc010740 1 -- 172.21.15.50:0/3896971092 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f90307390 con 0x563f903b3a20 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.391+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/3896971092 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563f903b3a20 0x563f903b5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.392+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/3896971092 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563f903b3a20 0x563f903b5ea0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cde0def3151a2ee9 server_cookie=5e308c8511f2564e in_seq=0 out_seq=0 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.392+0000 7f56d6ff5640 10 client.17499 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.392+0000 7f56d6ff5640 1 -- 172.21.15.50:0/3896971092 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f90307390 con 0x563f903b3a20 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.392+0000 7f56d6ff5640 10 client.17499 handle_command_reply: tid=0 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.392+0000 7f56fc010740 1 client.17499 shutdown 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.392+0000 7f56fc010740 1 -- 172.21.15.50:0/3896971092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 msgr2=0x563f903a0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.392+0000 7f56fc010740 1 --2- 172.21.15.50:0/3896971092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f903a0db0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f56ec003c80 tx=0x7f56ec004ab0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.393+0000 7f56fc010740 1 -- 172.21.15.50:0/3896971092 shutdown_connections 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.393+0000 7f56fc010740 1 --2- 172.21.15.50:0/3896971092 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563f903b3a20 0x563f903b5ea0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.393+0000 7f56fc010740 1 --2- 172.21.15.50:0/3896971092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f903a0db0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.393+0000 7f56fc010740 1 --2- 172.21.15.50:0/3896971092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 0x563f903a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.393+0000 7f56fc010740 1 --2- 172.21.15.50:0/3896971092 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903a3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.393+0000 7f56d6ff5640 0 client.17499 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.393+0000 7f56fc010740 1 -- 172.21.15.50:0/3896971092 >> 172.21.15.50:0/3896971092 conn(0x563f90390b30 msgr2=0x563f903969a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.393+0000 7f56fc010740 1 -- 172.21.15.50:0/3896971092 shutdown_connections 2024-04-23T18:34:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.393+0000 7f56fc010740 1 -- 172.21.15.50:0/3896971092 wait complete. 2024-04-23T18:34:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.393+0000 7f56fc010740 20 client.17499 trim_cache size 0 max 16384 2024-04-23T18:34:57.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56fc010740 1 Processor -- start 2024-04-23T18:34:57.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56fc010740 1 -- start start 2024-04-23T18:34:57.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56fc010740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56fc010740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 0x563f9039f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56fc010740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f9039bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56fc010740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f56f41ae160 con 0x563f903a3c10 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56fc010740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f56f41ab9d0 con 0x563f90394e40 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56fc010740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f56f41a9840 con 0x563f90394520 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56f359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f9039bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56f359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f9039bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34396/0 (socket says 172.21.15.50:34396) 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56f359e640 1 -- 172.21.15.50:0/3927437063 learned_addr learned my addr 172.21.15.50:0/3927437063 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/3927437063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56f259c640 1 --2- 172.21.15.50:0/3927437063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 0x563f9039f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56f359e640 1 -- 172.21.15.50:0/3927437063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 msgr2=0x563f903af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56f359e640 1 --2- 172.21.15.50:0/3927437063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56f359e640 1 -- 172.21.15.50:0/3927437063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 msgr2=0x563f9039f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56f359e640 1 --2- 172.21.15.50:0/3927437063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 0x563f9039f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56f359e640 1 -- 172.21.15.50:0/3927437063 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f90390960 con 0x563f903a3c10 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.408+0000 7f56f359e640 1 --2- 172.21.15.50:0/3927437063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f9039bd80 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f56ec006010 tx=0x7f56ec009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b1ec815976fab8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56d8ff9640 1 -- 172.21.15.50:0/3927437063 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec020640 con 0x563f903a3c10 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56d8ff9640 1 -- 172.21.15.50:0/3927437063 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56ec0207e0 con 0x563f903a3c10 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56d8ff9640 1 -- 172.21.15.50:0/3927437063 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec02b6c0 con 0x563f903a3c10 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56fc010740 1 -- 172.21.15.50:0/3927437063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 msgr2=0x563f9039bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56fc010740 1 --2- 172.21.15.50:0/3927437063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f9039bd80 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f56ec006010 tx=0x7f56ec009fd0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56fc010740 1 -- 172.21.15.50:0/3927437063 shutdown_connections 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56fc010740 1 --2- 172.21.15.50:0/3927437063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f9039bd80 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56fc010740 1 --2- 172.21.15.50:0/3927437063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 0x563f9039f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56fc010740 1 --2- 172.21.15.50:0/3927437063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56fc010740 1 -- 172.21.15.50:0/3927437063 >> 172.21.15.50:0/3927437063 conn(0x563f90390b30 msgr2=0x563f9039b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56fc010740 1 -- 172.21.15.50:0/3927437063 shutdown_connections 2024-04-23T18:34:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56fc010740 1 -- 172.21.15.50:0/3927437063 wait complete. 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.409+0000 7f56fc010740 1 -- start start 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56fc010740 1 Processor -- start 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56fc010740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903ab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56fc010740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 0x563f903abe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56fc010740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f9039d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56fc010740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f56f41a9e90 con 0x563f903a3c10 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56fc010740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f56f41ab300 con 0x563f90394e40 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56fc010740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f56f41aa560 con 0x563f90394520 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56fc010740 10 client.0 _ll_register_callbacks cb 0x7f56ec003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56f359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f9039d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56f2d9d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56f359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f9039d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34412/0 (socket says 172.21.15.50:34412) 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56f359e640 1 -- 172.21.15.50:0/2789892336 learned_addr learned my addr 172.21.15.50:0/2789892336 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56f259c640 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 0x563f903abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56f359e640 1 -- 172.21.15.50:0/2789892336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 msgr2=0x563f903ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56f359e640 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903ab900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56f359e640 1 -- 172.21.15.50:0/2789892336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 msgr2=0x563f903abe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56f359e640 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 0x563f903abe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56f359e640 1 -- 172.21.15.50:0/2789892336 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f90395380 con 0x563f903a3c10 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.410+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56f359e640 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f9039d0f0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f56ec012390 tx=0x7f56ec0123c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4c46eb0a59d2f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56d6ff5640 1 -- 172.21.15.50:0/2789892336 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec041020 con 0x563f903a3c10 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56d6ff5640 1 -- 172.21.15.50:0/2789892336 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8542+0+0 (secure 0 0 0) 0x7f56ec004a90 con 0x563f903a3c10 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56fc010740 1 -- 172.21.15.50:0/2789892336 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56ec0207e0 con 0x563f903a3c10 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56d6ff5640 1 -- 172.21.15.50:0/2789892336 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f56ec04c050 con 0x563f903a3c10 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56fc010740 10 client.17505 fetch_fsmap learned FSMap version 54 2024-04-23T18:34:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56fc010740 1 -- 172.21.15.50:0/2789892336 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x563f90419a30 con 0x563f903a3c10 2024-04-23T18:34:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56d6ff5640 1 -- 172.21.15.50:0/2789892336 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56ec0207e0 con 0x563f903a3c10 2024-04-23T18:34:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56d6ff5640 1 -- 172.21.15.50:0/2789892336 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f56ec02b930 con 0x563f903a3c10 2024-04-23T18:34:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56fc010740 10 client.17505 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:34:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56fc010740 10 client.17505 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:34:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56fc010740 20 client.17505 populate_metadata read hostname 'smithi050' 2024-04-23T18:34:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56fc010740 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563f9041a120 0x563f9041c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:34:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56fc010740 4 client.17505 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:34:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.411+0000 7f56fc010740 1 -- 172.21.15.50:0/2789892336 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563f90301200 con 0x563f9041a120 2024-04-23T18:34:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.412+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563f9041a120 0x563f9041c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:34:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.412+0000 7f56f2d9d640 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563f9041a120 0x563f9041c510 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=722c10ae0d4799a7 server_cookie=c19a3ad562b357a5 in_seq=0 out_seq=0 2024-04-23T18:34:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.412+0000 7f56d6ff5640 10 client.17505 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56d6ff5640 1 -- 172.21.15.50:0/2789892336 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563f90301200 con 0x563f9041a120 2024-04-23T18:34:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56d6ff5640 10 client.17505 handle_command_reply: tid=0 2024-04-23T18:34:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56fc010740 1 client.17505 shutdown 2024-04-23T18:34:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56fc010740 1 -- 172.21.15.50:0/2789892336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 msgr2=0x563f9039d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56fc010740 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f9039d0f0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f56ec012390 tx=0x7f56ec0123c0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56fc010740 1 -- 172.21.15.50:0/2789892336 shutdown_connections 2024-04-23T18:34:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56fc010740 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563f9041a120 0x563f9041c510 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56d6ff5640 0 client.17505 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:34:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56fc010740 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563f903a3c10 0x563f9039d0f0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56fc010740 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563f90394e40 0x563f903abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56fc010740 1 --2- 172.21.15.50:0/2789892336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563f90394520 0x563f903ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56fc010740 1 -- 172.21.15.50:0/2789892336 >> 172.21.15.50:0/2789892336 conn(0x563f90390b30 msgr2=0x563f9039bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56fc010740 1 -- 172.21.15.50:0/2789892336 shutdown_connections 2024-04-23T18:34:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56fc010740 1 -- 172.21.15.50:0/2789892336 wait complete. 2024-04-23T18:34:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.551+0000 7f56fc010740 20 client.17505 trim_cache size 0 max 16384 2024-04-23T18:34:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:34:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:34:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:34:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:34:57.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.552+0000 7f56f9220640 1 -- 172.21.15.50:0/3006291863 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f56dc0a3dc0 msgr2=0x7f56dc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:57.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.552+0000 7f56f9220640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f56dc0a3dc0 0x7f56dc0a6280 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0099e0 tx=0x7f56ec00fe70 comp rx=0 tx=0).stop 2024-04-23T18:34:57.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.552+0000 7f56f9220640 1 -- 172.21.15.50:0/3006291863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56f41021c0 msgr2=0x7f56f41986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:34:57.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.552+0000 7f56f9220640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56f41021c0 0x7f56f41986d0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f56e4005ab0 tx=0x7f56e4003c80 comp rx=0 tx=0).stop 2024-04-23T18:34:57.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.552+0000 7f56f9220640 1 -- 172.21.15.50:0/3006291863 shutdown_connections 2024-04-23T18:34:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.552+0000 7f56f9220640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f56dc0a3dc0 0x7f56dc0a6280 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.552+0000 7f56f9220640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f56f419cf40 0x7f56f419d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.552+0000 7f56f9220640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56f4107290 0x7f56f4198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.552+0000 7f56f9220640 1 --2- 172.21.15.50:0/3006291863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56f41021c0 0x7f56f41986d0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:34:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.552+0000 7f56f9220640 1 -- 172.21.15.50:0/3006291863 >> 172.21.15.50:0/3006291863 conn(0x7f56f40fda20 msgr2=0x7f56f40fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:34:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.552+0000 7f56f9220640 1 -- 172.21.15.50:0/3006291863 shutdown_connections 2024-04-23T18:34:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:34:57.552+0000 7f56f9220640 1 -- 172.21.15.50:0/3006291863 wait complete. 2024-04-23T18:34:57.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 29.37 MB/sec execute 91 sec latency 7850.884 ms 2024-04-23T18:34:58.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:58 smithi177 ceph-mon[32308]: pgmap v520: 97 pgs: 97 active+clean; 1007 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 681 op/s 2024-04-23T18:34:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:58 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:34:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:58 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:34:58.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:34:58 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:34:58.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 29.05 MB/sec execute 92 sec latency 8850.988 ms 2024-04-23T18:34:58.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:58 smithi188 ceph-mon[32694]: pgmap v520: 97 pgs: 97 active+clean; 1007 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 681 op/s 2024-04-23T18:34:58.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:58 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:34:58.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:58 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:34:58.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:34:58 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:34:58.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:58 smithi050 ceph-mon[29187]: pgmap v520: 97 pgs: 97 active+clean; 1007 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 681 op/s 2024-04-23T18:34:58.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:58 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:34:58.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:58 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:34:58.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:34:58 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:34:59.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 28.74 MB/sec execute 93 sec latency 9851.043 ms 2024-04-23T18:35:00.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:00 smithi177 ceph-mon[32308]: pgmap v521: 97 pgs: 97 active+clean; 1007 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.1 MiB/s wr, 308 op/s 2024-04-23T18:35:00.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 28.44 MB/sec execute 94 sec latency 10851.122 ms 2024-04-23T18:35:00.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:00 smithi188 ceph-mon[32694]: pgmap v521: 97 pgs: 97 active+clean; 1007 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.1 MiB/s wr, 308 op/s 2024-04-23T18:35:00.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:00 smithi050 ceph-mon[29187]: pgmap v521: 97 pgs: 97 active+clean; 1007 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.1 MiB/s wr, 308 op/s 2024-04-23T18:35:01.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 28.14 MB/sec execute 95 sec latency 11851.169 ms 2024-04-23T18:35:01.693 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:35:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:35:01.204+0000 7f0edafb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:35:01.693 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:35:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:35:01.204+0000 7f0edafb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:35:01.693 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:35:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:35:01.204+0000 7f0edafb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:35:01.693 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:35:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:35:01.204+0000 7f0edafb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:35:01.693 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:35:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x[29419]: 2024-04-23T18:35:01.204+0000 7f0edafb5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T18:35:02.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:02 smithi177 ceph-mon[32308]: pgmap v522: 97 pgs: 97 active+clean; 1007 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.1 MiB/s wr, 308 op/s 2024-04-23T18:35:02.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 27.84 MB/sec execute 96 sec latency 12851.239 ms 2024-04-23T18:35:02.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:02 smithi188 ceph-mon[32694]: pgmap v522: 97 pgs: 97 active+clean; 1007 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.1 MiB/s wr, 308 op/s 2024-04-23T18:35:02.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:02 smithi050 ceph-mon[29187]: pgmap v522: 97 pgs: 97 active+clean; 1007 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.1 MiB/s wr, 308 op/s 2024-04-23T18:35:03.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 27.56 MB/sec execute 97 sec latency 13851.365 ms 2024-04-23T18:35:04.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:04 smithi177 ceph-mon[32308]: pgmap v523: 97 pgs: 97 active+clean; 1002 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 8.7 MiB/s wr, 466 op/s 2024-04-23T18:35:04.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:04 smithi177 ceph-mon[32308]: mgrmap e27: x(active, since 16m), standbys: y, z 2024-04-23T18:35:04.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 27.27 MB/sec execute 98 sec latency 14851.423 ms 2024-04-23T18:35:04.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:04 smithi188 ceph-mon[32694]: pgmap v523: 97 pgs: 97 active+clean; 1002 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 8.7 MiB/s wr, 466 op/s 2024-04-23T18:35:04.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:04 smithi188 ceph-mon[32694]: mgrmap e27: x(active, since 16m), standbys: y, z 2024-04-23T18:35:04.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:04 smithi050 ceph-mon[29187]: pgmap v523: 97 pgs: 97 active+clean; 1002 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 8.7 MiB/s wr, 466 op/s 2024-04-23T18:35:04.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:04 smithi050 ceph-mon[29187]: mgrmap e27: x(active, since 16m), standbys: y, z 2024-04-23T18:35:05.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:05 smithi177 ceph-mon[32308]: osdmap e115: 12 total, 12 up, 12 in 2024-04-23T18:35:05.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 27.00 MB/sec execute 99 sec latency 15851.468 ms 2024-04-23T18:35:05.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:05 smithi188 ceph-mon[32694]: osdmap e115: 12 total, 12 up, 12 in 2024-04-23T18:35:05.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:05 smithi050 ceph-mon[29187]: osdmap e115: 12 total, 12 up, 12 in 2024-04-23T18:35:06.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:06 smithi177 ceph-mon[32308]: pgmap v525: 97 pgs: 97 active+clean; 1002 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 5.9 MiB/s wr, 294 op/s 2024-04-23T18:35:06.568 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 26.73 MB/sec execute 100 sec latency 16851.506 ms 2024-04-23T18:35:06.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:06 smithi188 ceph-mon[32694]: pgmap v525: 97 pgs: 97 active+clean; 1002 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 5.9 MiB/s wr, 294 op/s 2024-04-23T18:35:06.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:06 smithi050 ceph-mon[29187]: pgmap v525: 97 pgs: 97 active+clean; 1002 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 5.9 MiB/s wr, 294 op/s 2024-04-23T18:35:07.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 26.46 MB/sec execute 101 sec latency 17851.596 ms 2024-04-23T18:35:08.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:08 smithi177 ceph-mon[32308]: pgmap v526: 97 pgs: 97 active+clean; 998 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 6.3 MiB/s wr, 295 op/s 2024-04-23T18:35:08.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 26.20 MB/sec execute 102 sec latency 18851.725 ms 2024-04-23T18:35:08.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:08 smithi188 ceph-mon[32694]: pgmap v526: 97 pgs: 97 active+clean; 998 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 6.3 MiB/s wr, 295 op/s 2024-04-23T18:35:08.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:08 smithi050 ceph-mon[29187]: pgmap v526: 97 pgs: 97 active+clean; 998 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 6.3 MiB/s wr, 295 op/s 2024-04-23T18:35:09.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 25.95 MB/sec execute 103 sec latency 19851.829 ms 2024-04-23T18:35:10.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:35:10.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.150+0000 7f34de129640 1 -- 172.21.15.50:0/1997464511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8103b50 msgr2=0x7f34d810c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.150+0000 7f34de129640 1 --2- 172.21.15.50:0/1997464511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8103b50 0x7f34d810c030 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f34cc009900 tx=0x7f34cc02a480 comp rx=0 tx=0).stop 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.150+0000 7f34de129640 1 -- 172.21.15.50:0/1997464511 shutdown_connections 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.150+0000 7f34de129640 1 --2- 172.21.15.50:0/1997464511 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d810c570 0x7f34d810e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.150+0000 7f34de129640 1 --2- 172.21.15.50:0/1997464511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8103b50 0x7f34d810c030 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.150+0000 7f34de129640 1 --2- 172.21.15.50:0/1997464511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34d8103230 0x7f34d8103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.150+0000 7f34de129640 1 -- 172.21.15.50:0/1997464511 >> 172.21.15.50:0/1997464511 conn(0x7f34d8077fb0 msgr2=0x7f34d80783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.150+0000 7f34de129640 1 -- 172.21.15.50:0/1997464511 shutdown_connections 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.150+0000 7f34de129640 1 -- 172.21.15.50:0/1997464511 wait complete. 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.150+0000 7f34de129640 1 Processor -- start 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34de129640 1 -- start start 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34de129640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34d8103230 0x7f34d8198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34de129640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d8103b50 0x7f34d8198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34de129640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d810c570 0x7f34d819d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34de129640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f34d81121d0 con 0x7f34d810c570 2024-04-23T18:35:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34de129640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f34d8112050 con 0x7f34d8103230 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34de129640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f34d8112350 con 0x7f34d8103b50 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34d7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d810c570 0x7f34d819d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34d7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d810c570 0x7f34d819d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37240/0 (socket says 172.21.15.50:37240) 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34d7fff640 1 -- 172.21.15.50:0/2449169727 learned_addr learned my addr 172.21.15.50:0/2449169727 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34d6ffd640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d8103b50 0x7f34d8198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34d77fe640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34d8103230 0x7f34d8198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34d7fff640 1 -- 172.21.15.50:0/2449169727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d8103b50 msgr2=0x7f34d8198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34d7fff640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d8103b50 0x7f34d8198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34d7fff640 1 -- 172.21.15.50:0/2449169727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34d8103230 msgr2=0x7f34d8198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34d7fff640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34d8103230 0x7f34d8198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.151+0000 7f34d7fff640 1 -- 172.21.15.50:0/2449169727 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d819d740 con 0x7f34d810c570 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.152+0000 7f34d7fff640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d810c570 0x7f34d819d010 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f34d000ea60 tx=0x7f34d000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.152+0000 7f34d4ff9640 1 -- 172.21.15.50:0/2449169727 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d000ce40 con 0x7f34d810c570 2024-04-23T18:35:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.152+0000 7f34d4ff9640 1 -- 172.21.15.50:0/2449169727 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34d0005020 con 0x7f34d810c570 2024-04-23T18:35:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.152+0000 7f34de129640 1 -- 172.21.15.50:0/2449169727 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d819da30 con 0x7f34d810c570 2024-04-23T18:35:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.152+0000 7f34d4ff9640 1 -- 172.21.15.50:0/2449169727 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d0010670 con 0x7f34d810c570 2024-04-23T18:35:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.152+0000 7f34de129640 1 -- 172.21.15.50:0/2449169727 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d81a52c0 con 0x7f34d810c570 2024-04-23T18:35:10.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.153+0000 7f34de129640 1 -- 172.21.15.50:0/2449169727 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d80691c0 con 0x7f34d810c570 2024-04-23T18:35:10.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.154+0000 7f34d4ff9640 1 -- 172.21.15.50:0/2449169727 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f34d00054b0 con 0x7f34d810c570 2024-04-23T18:35:10.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.155+0000 7f34d4ff9640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f34c00a3d20 0x7f34c00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.155+0000 7f34d77fe640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f34c00a3d20 0x7f34c00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.155+0000 7f34d4ff9640 1 -- 172.21.15.50:0/2449169727 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8586+0+0 (secure 0 0 0) 0x7f34d00d0c60 con 0x7f34d810c570 2024-04-23T18:35:10.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.155+0000 7f34d77fe640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f34c00a3d20 0x7f34c00a61e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f34c80095f0 tx=0x7f34c8005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.159+0000 7f34d4ff9640 1 -- 172.21.15.50:0/2449169727 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34d009d2c0 con 0x7f34d810c570 2024-04-23T18:35:10.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.423+0000 7f34de129640 1 -- 172.21.15.50:0/2449169727 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34d802d5c0 con 0x7f34d810c570 2024-04-23T18:35:10.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.424+0000 7f34d4ff9640 1 -- 172.21.15.50:0/2449169727 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+11881 (secure 0 0 0) 0x7f34d009cbc0 con 0x7f34d810c570 2024-04-23T18:35:10.417 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:35:10.417 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":54,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:34:05.578646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26753":{"gid":26753,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/1119979694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":1119979694},{"type":"v1","addr":"172.21.15.177:6837","nonce":1119979694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:35:10.417 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 54 2024-04-23T18:35:10.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.426+0000 7f34de129640 1 -- 172.21.15.50:0/2449169727 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f34c00a3d20 msgr2=0x7f34c00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.426+0000 7f34de129640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f34c00a3d20 0x7f34c00a61e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f34c80095f0 tx=0x7f34c8005f90 comp rx=0 tx=0).stop 2024-04-23T18:35:10.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.426+0000 7f34de129640 1 -- 172.21.15.50:0/2449169727 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d810c570 msgr2=0x7f34d819d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.426+0000 7f34de129640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d810c570 0x7f34d819d010 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f34d000ea60 tx=0x7f34d000ef20 comp rx=0 tx=0).stop 2024-04-23T18:35:10.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.426+0000 7f34de129640 1 -- 172.21.15.50:0/2449169727 shutdown_connections 2024-04-23T18:35:10.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.426+0000 7f34de129640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f34c00a3d20 0x7f34c00a61e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.426+0000 7f34de129640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d810c570 0x7f34d819d010 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.426+0000 7f34de129640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34d8103b50 0x7f34d8198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.426+0000 7f34de129640 1 --2- 172.21.15.50:0/2449169727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34d8103230 0x7f34d8198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.426+0000 7f34de129640 1 -- 172.21.15.50:0/2449169727 >> 172.21.15.50:0/2449169727 conn(0x7f34d8077fb0 msgr2=0x7f34d80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.427+0000 7f34de129640 1 -- 172.21.15.50:0/2449169727 shutdown_connections 2024-04-23T18:35:10.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.427+0000 7f34de129640 1 -- 172.21.15.50:0/2449169727 wait complete. 2024-04-23T18:35:10.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:10 smithi050 ceph-mon[29187]: pgmap v527: 97 pgs: 97 active+clean; 998 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 6.3 MiB/s wr, 295 op/s 2024-04-23T18:35:10.430 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id 8c92f231 --release --await' 2024-04-23T18:35:10.430 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id 8c92f231 --release --await 2024-04-23T18:35:10.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:10 smithi177 ceph-mon[32308]: pgmap v527: 97 pgs: 97 active+clean; 998 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 6.3 MiB/s wr, 295 op/s 2024-04-23T18:35:10.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 101893 25.70 MB/sec execute 104 sec latency 20851.880 ms 2024-04-23T18:35:10.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:10 smithi188 ceph-mon[32694]: pgmap v527: 97 pgs: 97 active+clean; 998 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 6.3 MiB/s wr, 295 op/s 2024-04-23T18:35:10.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.598+0000 7efc55d07640 1 -- 172.21.15.50:0/553524038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc50100490 msgr2=0x7efc50100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.598+0000 7efc55d07640 1 --2- 172.21.15.50:0/553524038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc50100490 0x7efc50100870 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efc40009900 tx=0x7efc4002a410 comp rx=0 tx=0).stop 2024-04-23T18:35:10.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.598+0000 7efc55d07640 1 -- 172.21.15.50:0/553524038 shutdown_connections 2024-04-23T18:35:10.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.598+0000 7efc55d07640 1 --2- 172.21.15.50:0/553524038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc50105510 0x7efc5010d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.598+0000 7efc55d07640 1 --2- 172.21.15.50:0/553524038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc50100e40 0x7efc50104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.598+0000 7efc55d07640 1 --2- 172.21.15.50:0/553524038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc50100490 0x7efc50100870 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.598+0000 7efc55d07640 1 -- 172.21.15.50:0/553524038 >> 172.21.15.50:0/553524038 conn(0x7efc50076fb0 msgr2=0x7efc500fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.598+0000 7efc55d07640 1 -- 172.21.15.50:0/553524038 shutdown_connections 2024-04-23T18:35:10.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.598+0000 7efc55d07640 1 -- 172.21.15.50:0/553524038 wait complete. 2024-04-23T18:35:10.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.599+0000 7efc55d07640 1 Processor -- start 2024-04-23T18:35:10.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.599+0000 7efc55d07640 1 -- start start 2024-04-23T18:35:10.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.599+0000 7efc55d07640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc50100490 0x7efc50107100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.599+0000 7efc55d07640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc50105510 0x7efc50107640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.599+0000 7efc55d07640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc50107b80 0x7efc501aaff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.600+0000 7efc4f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc50100490 0x7efc50107100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.600+0000 7efc4f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc50100490 0x7efc50107100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:59726/0 (socket says 172.21.15.50:59726) 2024-04-23T18:35:10.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.600+0000 7efc4f7fe640 1 -- 172.21.15.50:0/2834851283 learned_addr learned my addr 172.21.15.50:0/2834851283 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:10.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.600+0000 7efc55d07640 1 -- 172.21.15.50:0/2834851283 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efc501111f0 con 0x7efc50107b80 2024-04-23T18:35:10.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.600+0000 7efc55d07640 1 -- 172.21.15.50:0/2834851283 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efc50111070 con 0x7efc50100490 2024-04-23T18:35:10.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.600+0000 7efc55d07640 1 -- 172.21.15.50:0/2834851283 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efc50111370 con 0x7efc50105510 2024-04-23T18:35:10.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.600+0000 7efc4effd640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc50105510 0x7efc50107640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.600+0000 7efc4ffff640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc50107b80 0x7efc501aaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.601+0000 7efc4ffff640 1 -- 172.21.15.50:0/2834851283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc50105510 msgr2=0x7efc50107640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.601+0000 7efc4ffff640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc50105510 0x7efc50107640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.601+0000 7efc4ffff640 1 -- 172.21.15.50:0/2834851283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc50100490 msgr2=0x7efc50107100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.601+0000 7efc4ffff640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc50100490 0x7efc50107100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.601+0000 7efc4ffff640 1 -- 172.21.15.50:0/2834851283 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc501ab530 con 0x7efc50107b80 2024-04-23T18:35:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.601+0000 7efc4ffff640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc50107b80 0x7efc501aaff0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7efc4400b410 tx=0x7efc4400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.601+0000 7efc4cff9640 1 -- 172.21.15.50:0/2834851283 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc44015020 con 0x7efc50107b80 2024-04-23T18:35:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.601+0000 7efc4cff9640 1 -- 172.21.15.50:0/2834851283 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc44011ce0 con 0x7efc50107b80 2024-04-23T18:35:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.601+0000 7efc4cff9640 1 -- 172.21.15.50:0/2834851283 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc44010d60 con 0x7efc50107b80 2024-04-23T18:35:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.602+0000 7efc55d07640 1 -- 172.21.15.50:0/2834851283 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc501ab820 con 0x7efc50107b80 2024-04-23T18:35:10.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.602+0000 7efc55d07640 1 -- 172.21.15.50:0/2834851283 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc501abc90 con 0x7efc50107b80 2024-04-23T18:35:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.605+0000 7efc58a1b740 1 Processor -- start 2024-04-23T18:35:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.605+0000 7efc4cff9640 1 -- 172.21.15.50:0/2834851283 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7efc44018020 con 0x7efc50107b80 2024-04-23T18:35:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.606+0000 7efc58a1b740 1 -- start start 2024-04-23T18:35:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.607+0000 7efc58a1b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 0x55a884c66870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.607+0000 7efc58a1b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66db0 0x55a884c70f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.607+0000 7efc58a1b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c71480 0x55a884c73870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.607+0000 7efc58a1b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a884bcb6d0 con 0x55a884c71480 2024-04-23T18:35:10.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.607+0000 7efc58a1b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a884b681a0 con 0x55a884c66db0 2024-04-23T18:35:10.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.607+0000 7efc58a1b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a884bc7690 con 0x55a884c66490 2024-04-23T18:35:10.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.607+0000 7efc4cff9640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efc380a3de0 0x7efc380a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.607+0000 7efc4effd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 0x55a884c66870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.607+0000 7efc4effd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 0x55a884c66870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:59114/0 (socket says 172.21.15.50:59114) 2024-04-23T18:35:10.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.607+0000 7efc4effd640 1 -- 172.21.15.50:0/3058692810 learned_addr learned my addr 172.21.15.50:0/3058692810 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:10.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.607+0000 7efc4effd640 1 --2- 172.21.15.50:0/3058692810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c71480 0x55a884c73870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.608+0000 7efc4cff9640 1 -- 172.21.15.50:0/2834851283 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8586+0+0 (secure 0 0 0) 0x7efc440d0e90 con 0x7efc50107b80 2024-04-23T18:35:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.608+0000 7efc4effd640 1 -- 172.21.15.50:0/3058692810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66db0 msgr2=0x55a884c70f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T18:35:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.608+0000 7efc4effd640 1 --2- 172.21.15.50:0/3058692810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66db0 0x55a884c70f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.608+0000 7efc4effd640 1 -- 172.21.15.50:0/3058692810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c71480 msgr2=0x55a884c73870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.608+0000 7efc4effd640 1 --2- 172.21.15.50:0/3058692810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c71480 0x55a884c73870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.608+0000 7efc4effd640 1 -- 172.21.15.50:0/3058692810 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a884b725b0 con 0x55a884c66490 2024-04-23T18:35:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.608+0000 7efc4ffff640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efc380a3de0 0x7efc380a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.608+0000 7efc4effd640 1 --2- 172.21.15.50:0/3058692810 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 0x55a884c66870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efc48005fd0 tx=0x7efc48009d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c858550f1d12a154 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.609+0000 7efc4ffff640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efc380a3de0 0x7efc380a62a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efc44099300 tx=0x7efc440047d0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.609+0000 7efc35ffb640 1 -- 172.21.15.50:0/3058692810 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4801f070 con 0x55a884c66490 2024-04-23T18:35:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.609+0000 7efc35ffb640 1 -- 172.21.15.50:0/3058692810 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc4800bbb0 con 0x55a884c66490 2024-04-23T18:35:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.610+0000 7efc35ffb640 1 -- 172.21.15.50:0/3058692810 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48004490 con 0x55a884c66490 2024-04-23T18:35:10.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.610+0000 7efc58a1b740 1 -- 172.21.15.50:0/3058692810 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 msgr2=0x55a884c66870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.610+0000 7efc58a1b740 1 --2- 172.21.15.50:0/3058692810 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 0x55a884c66870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efc48005fd0 tx=0x7efc48009d90 comp rx=0 tx=0).stop 2024-04-23T18:35:10.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.610+0000 7efc58a1b740 1 -- 172.21.15.50:0/3058692810 shutdown_connections 2024-04-23T18:35:10.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.610+0000 7efc58a1b740 1 --2- 172.21.15.50:0/3058692810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c71480 0x55a884c73870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.610+0000 7efc58a1b740 1 --2- 172.21.15.50:0/3058692810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66db0 0x55a884c70f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.610+0000 7efc58a1b740 1 --2- 172.21.15.50:0/3058692810 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 0x55a884c66870 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.610+0000 7efc58a1b740 1 -- 172.21.15.50:0/3058692810 >> 172.21.15.50:0/3058692810 conn(0x55a884c627b0 msgr2=0x55a884c62b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.610+0000 7efc58a1b740 1 -- 172.21.15.50:0/3058692810 shutdown_connections 2024-04-23T18:35:10.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.610+0000 7efc58a1b740 1 -- 172.21.15.50:0/3058692810 wait complete. 2024-04-23T18:35:10.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.611+0000 7efc58a1b740 1 -- start start 2024-04-23T18:35:10.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.611+0000 7efc58a1b740 1 Processor -- start 2024-04-23T18:35:10.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.612+0000 7efc58a1b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 0x55a884c70e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.612+0000 7efc58a1b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66db0 0x55a884c6e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.612+0000 7efc58a1b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c71480 0x55a884c6e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.612+0000 7efc4effd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 0x55a884c70e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.612+0000 7efc4effd640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 0x55a884c70e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:59124/0 (socket says 172.21.15.50:59124) 2024-04-23T18:35:10.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.612+0000 7efc4f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66db0 0x55a884c6e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.612+0000 7efc4f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66db0 0x55a884c6e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:59748/0 (socket says 172.21.15.50:59748) 2024-04-23T18:35:10.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.612+0000 7efc4effd640 1 -- 172.21.15.50:0/1373591377 learned_addr learned my addr 172.21.15.50:0/1373591377 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:10.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.612+0000 7efc58a1b740 1 -- 172.21.15.50:0/1373591377 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a884c85370 con 0x55a884c71480 2024-04-23T18:35:10.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.612+0000 7efc58a1b740 1 -- 172.21.15.50:0/1373591377 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a884b65b20 con 0x55a884c66db0 2024-04-23T18:35:10.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.612+0000 7efc58a1b740 1 -- 172.21.15.50:0/1373591377 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a884c854f0 con 0x55a884c66490 2024-04-23T18:35:10.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.613+0000 7efc58a1b740 10 client.0 _ll_register_callbacks cb 0x55a884b65530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:35:10.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.613+0000 7efc4effd640 1 -- 172.21.15.50:0/1373591377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66db0 msgr2=0x55a884c6e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.613+0000 7efc4effd640 1 --2- 172.21.15.50:0/1373591377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66db0 0x55a884c6e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.613+0000 7efc4effd640 1 -- 172.21.15.50:0/1373591377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c71480 msgr2=0x55a884c6e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.613+0000 7efc4effd640 1 --2- 172.21.15.50:0/1373591377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c71480 0x55a884c6e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.613+0000 7efc4effd640 1 -- 172.21.15.50:0/1373591377 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8848dd920 con 0x55a884c66490 2024-04-23T18:35:10.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.613+0000 7efc4effd640 1 --2- 172.21.15.50:0/1373591377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 0x55a884c70e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efc48005c40 tx=0x7efc4800bb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97cb8ccb89e53c92 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.613+0000 7efc2effd640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:35:10.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.613+0000 7efc2effd640 1 -- 172.21.15.50:0/1373591377 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4801f040 con 0x55a884c66490 2024-04-23T18:35:10.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.613+0000 7efc58a1b740 1 -- 172.21.15.50:0/1373591377 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc4800bbb0 con 0x55a884c66490 2024-04-23T18:35:10.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.613+0000 7efc2effd640 1 -- 172.21.15.50:0/1373591377 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8586+0+0 (secure 0 0 0) 0x7efc480040d0 con 0x55a884c66490 2024-04-23T18:35:10.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.614+0000 7efc2effd640 1 -- 172.21.15.50:0/1373591377 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7efc480042d0 con 0x55a884c66490 2024-04-23T18:35:10.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.614+0000 7efc58a1b740 10 client.27076 fetch_fsmap learned FSMap version 54 2024-04-23T18:35:10.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.614+0000 7efc58a1b740 1 -- 172.21.15.50:0/1373591377 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55a8849a9d30 con 0x55a884c66490 2024-04-23T18:35:10.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.614+0000 7efc2effd640 1 -- 172.21.15.50:0/1373591377 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc4800bbb0 con 0x55a884c66490 2024-04-23T18:35:10.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.614+0000 7efc2effd640 1 -- 172.21.15.50:0/1373591377 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7efc4802e030 con 0x55a884c66490 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.614+0000 7efc58a1b740 10 client.27076 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.614+0000 7efc58a1b740 10 client.27076 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.614+0000 7efc58a1b740 20 client.27076 populate_metadata read hostname 'smithi050' 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.614+0000 7efc58a1b740 1 --2- 172.21.15.50:0/1373591377 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a884c92390 0x55a884c94780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.614+0000 7efc58a1b740 4 client.27076 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.614+0000 7efc58a1b740 1 -- 172.21.15.50:0/1373591377 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a884b89500 con 0x55a884c92390 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.614+0000 7efc4f7fe640 1 --2- 172.21.15.50:0/1373591377 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a884c92390 0x55a884c94780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.615+0000 7efc4f7fe640 1 --2- 172.21.15.50:0/1373591377 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a884c92390 0x55a884c94780 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e56e4445bed30a10 server_cookie=2c8a3e8e1df97008 in_seq=0 out_seq=0 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.615+0000 7efc2effd640 10 client.27076 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.615+0000 7efc2effd640 1 -- 172.21.15.50:0/1373591377 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a884b89500 con 0x55a884c92390 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.615+0000 7efc2effd640 10 client.27076 handle_command_reply: tid=0 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.615+0000 7efc58a1b740 1 client.27076 shutdown 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.616+0000 7efc58a1b740 1 -- 172.21.15.50:0/1373591377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 msgr2=0x55a884c70e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.616+0000 7efc58a1b740 1 --2- 172.21.15.50:0/1373591377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 0x55a884c70e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efc48005c40 tx=0x7efc4800bb30 comp rx=0 tx=0).stop 2024-04-23T18:35:10.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.616+0000 7efc58a1b740 1 -- 172.21.15.50:0/1373591377 shutdown_connections 2024-04-23T18:35:10.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.616+0000 7efc58a1b740 1 --2- 172.21.15.50:0/1373591377 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a884c92390 0x55a884c94780 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.616+0000 7efc58a1b740 1 --2- 172.21.15.50:0/1373591377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c71480 0x55a884c6e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.616+0000 7efc58a1b740 1 --2- 172.21.15.50:0/1373591377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66db0 0x55a884c6e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.616+0000 7efc58a1b740 1 --2- 172.21.15.50:0/1373591377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c66490 0x55a884c70e30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.616+0000 7efc2effd640 0 client.27076 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:10.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.616+0000 7efc58a1b740 1 -- 172.21.15.50:0/1373591377 >> 172.21.15.50:0/1373591377 conn(0x55a884c627b0 msgr2=0x55a884c684b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.616+0000 7efc58a1b740 1 -- 172.21.15.50:0/1373591377 shutdown_connections 2024-04-23T18:35:10.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.616+0000 7efc58a1b740 1 -- 172.21.15.50:0/1373591377 wait complete. 2024-04-23T18:35:10.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.616+0000 7efc58a1b740 20 client.27076 trim_cache size 0 max 16384 2024-04-23T18:35:10.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.633+0000 7efc58a1b740 1 Processor -- start 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.633+0000 7efc58a1b740 1 -- start start 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.633+0000 7efc58a1b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66490 0x55a884c8e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.633+0000 7efc58a1b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 0x55a884c6d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.633+0000 7efc58a1b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c71480 0x55a884c70960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.633+0000 7efc58a1b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c011180 con 0x55a884c66db0 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.633+0000 7efc58a1b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c00e9f0 con 0x55a884c66490 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.633+0000 7efc58a1b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c00c860 con 0x55a884c71480 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc4f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 0x55a884c6d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc4f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 0x55a884c6d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37272/0 (socket says 172.21.15.50:37272) 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc4f7fe640 1 -- 172.21.15.50:0/3684198008 learned_addr learned my addr 172.21.15.50:0/3684198008 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc4effd640 1 --2- 172.21.15.50:0/3684198008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c71480 0x55a884c70960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc4effd640 1 --2- 172.21.15.50:0/3684198008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66490 0x55a884c8e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc4f7fe640 1 -- 172.21.15.50:0/3684198008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c71480 msgr2=0x55a884c70960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc4f7fe640 1 --2- 172.21.15.50:0/3684198008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c71480 0x55a884c70960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc4f7fe640 1 -- 172.21.15.50:0/3684198008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66490 msgr2=0x55a884c8e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc4f7fe640 1 --2- 172.21.15.50:0/3684198008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66490 0x55a884c8e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc4f7fe640 1 -- 172.21.15.50:0/3684198008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a884c94d10 con 0x55a884c66db0 2024-04-23T18:35:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc4effd640 1 --2- 172.21.15.50:0/3684198008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c71480 0x55a884c70960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc4f7fe640 1 --2- 172.21.15.50:0/3684198008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 0x55a884c6d0c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7efc400096c0 tx=0x7efc400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644ba0d7312c3245 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc357fa640 1 -- 172.21.15.50:0/3684198008 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc40049020 con 0x55a884c66db0 2024-04-23T18:35:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc357fa640 1 -- 172.21.15.50:0/3684198008 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc40045780 con 0x55a884c66db0 2024-04-23T18:35:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.634+0000 7efc357fa640 1 -- 172.21.15.50:0/3684198008 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc40035040 con 0x55a884c66db0 2024-04-23T18:35:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.635+0000 7efc58a1b740 1 -- 172.21.15.50:0/3684198008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 msgr2=0x55a884c6d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.635+0000 7efc58a1b740 1 --2- 172.21.15.50:0/3684198008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 0x55a884c6d0c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7efc400096c0 tx=0x7efc400046c0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.635+0000 7efc58a1b740 1 -- 172.21.15.50:0/3684198008 shutdown_connections 2024-04-23T18:35:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.635+0000 7efc58a1b740 1 --2- 172.21.15.50:0/3684198008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c71480 0x55a884c70960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.635+0000 7efc58a1b740 1 --2- 172.21.15.50:0/3684198008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 0x55a884c6d0c0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.635+0000 7efc58a1b740 1 --2- 172.21.15.50:0/3684198008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66490 0x55a884c8e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.635+0000 7efc58a1b740 1 -- 172.21.15.50:0/3684198008 >> 172.21.15.50:0/3684198008 conn(0x55a884c627b0 msgr2=0x55a884c68470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.635+0000 7efc58a1b740 1 -- 172.21.15.50:0/3684198008 shutdown_connections 2024-04-23T18:35:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.635+0000 7efc58a1b740 1 -- 172.21.15.50:0/3684198008 wait complete. 2024-04-23T18:35:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.635+0000 7efc58a1b740 1 -- start start 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc58a1b740 1 Processor -- start 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc58a1b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66490 0x55a884c8a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc58a1b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 0x55a884c8ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc58a1b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c71480 0x55a884c6a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc58a1b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c00ceb0 con 0x55a884c66db0 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc58a1b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c00e320 con 0x55a884c66490 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc58a1b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c00d580 con 0x55a884c71480 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc58a1b740 10 client.0 _ll_register_callbacks cb 0x7efc48017780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc4f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 0x55a884c8ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc4f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 0x55a884c8ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37286/0 (socket says 172.21.15.50:37286) 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc4f7fe640 1 -- 172.21.15.50:0/1284890235 learned_addr learned my addr 172.21.15.50:0/1284890235 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc4effd640 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66490 0x55a884c8a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc4effd640 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c71480 0x55a884c6a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc4f7fe640 1 -- 172.21.15.50:0/1284890235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c71480 msgr2=0x55a884c6a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc4f7fe640 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c71480 0x55a884c6a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc4f7fe640 1 -- 172.21.15.50:0/1284890235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66490 msgr2=0x55a884c8a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc4f7fe640 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66490 0x55a884c8a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc4f7fe640 1 -- 172.21.15.50:0/1284890235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a884c6aae0 con 0x55a884c66db0 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.636+0000 7efc4effd640 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66490 0x55a884c8a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc4f7fe640 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 0x55a884c8ad60 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7efc40005ab0 tx=0x7efc400044a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271dcb138d88598f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc2effd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc2effd640 1 -- 172.21.15.50:0/1284890235 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc40052070 con 0x55a884c66db0 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc58a1b740 1 -- 172.21.15.50:0/1284890235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc40045780 con 0x55a884c66db0 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc2effd640 1 -- 172.21.15.50:0/1284890235 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8586+0+0 (secure 0 0 0) 0x7efc400454e0 con 0x55a884c66db0 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc2effd640 1 -- 172.21.15.50:0/1284890235 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7efc40035070 con 0x55a884c66db0 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc58a1b740 10 client.17529 fetch_fsmap learned FSMap version 54 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc58a1b740 1 -- 172.21.15.50:0/1284890235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55a884c6be20 con 0x55a884c66db0 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc2effd640 1 -- 172.21.15.50:0/1284890235 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc40045780 con 0x55a884c66db0 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc2effd640 1 -- 172.21.15.50:0/1284890235 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7efc40004b10 con 0x55a884c66db0 2024-04-23T18:35:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc58a1b740 10 client.17529 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:35:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc58a1b740 10 client.17529 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:35:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc58a1b740 20 client.17529 populate_metadata read hostname 'smithi050' 2024-04-23T18:35:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc58a1b740 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a884cf9a20 0x55a884cfbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc58a1b740 4 client.17529 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "8c92f231", "release": true, "await": true}] 2024-04-23T18:35:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.637+0000 7efc58a1b740 1 -- 172.21.15.50:0/1284890235 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "8c92f231", "release": true, "await": true}) -- 0x55a884bd90f0 con 0x55a884cf9a20 2024-04-23T18:35:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.638+0000 7efc4effd640 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a884cf9a20 0x55a884cfbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.638+0000 7efc4effd640 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a884cf9a20 0x55a884cfbe40 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=721c4dbaf7f6e659 server_cookie=1d17b3ec89ec675b in_seq=0 out_seq=0 2024-04-23T18:35:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.638+0000 7efc2effd640 10 client.17529 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc2effd640 1 -- 172.21.15.50:0/1284890235 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55a884bd90f0 con 0x55a884cf9a20 2024-04-23T18:35:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc2effd640 10 client.17529 handle_command_reply: tid=0 2024-04-23T18:35:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc58a1b740 1 client.17529 shutdown 2024-04-23T18:35:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc58a1b740 1 -- 172.21.15.50:0/1284890235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 msgr2=0x55a884c8ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc58a1b740 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 0x55a884c8ad60 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7efc40005ab0 tx=0x7efc400044a0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc58a1b740 1 -- 172.21.15.50:0/1284890235 shutdown_connections 2024-04-23T18:35:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc58a1b740 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a884cf9a20 0x55a884cfbe40 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc58a1b740 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a884c71480 0x55a884c6a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc58a1b740 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a884c66db0 0x55a884c8ad60 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc58a1b740 1 --2- 172.21.15.50:0/1284890235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a884c66490 0x55a884c8a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc58a1b740 1 -- 172.21.15.50:0/1284890235 >> 172.21.15.50:0/1284890235 conn(0x55a884c627b0 msgr2=0x55a884c8d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc2effd640 0 client.17529 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc58a1b740 1 -- 172.21.15.50:0/1284890235 shutdown_connections 2024-04-23T18:35:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc58a1b740 1 -- 172.21.15.50:0/1284890235 wait complete. 2024-04-23T18:35:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.660+0000 7efc58a1b740 20 client.17529 trim_cache size 0 max 16384 2024-04-23T18:35:10.654 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:35:10.654 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 54, 2024-04-23T18:35:10.654 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 56, 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "8c92f231": { 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 56, 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 79.8, 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:35:10.655 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:35:10.656 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:35:10.656 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:35:10.656 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:35:10.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:35:10.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:35:10.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:35:10.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:35:10.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:35:10.656 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:35:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.661+0000 7efc55d07640 1 -- 172.21.15.50:0/2834851283 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efc380a3de0 msgr2=0x7efc380a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.661+0000 7efc55d07640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efc380a3de0 0x7efc380a62a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efc44099300 tx=0x7efc440047d0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.661+0000 7efc55d07640 1 -- 172.21.15.50:0/2834851283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc50107b80 msgr2=0x7efc501aaff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.661+0000 7efc55d07640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc50107b80 0x7efc501aaff0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7efc4400b410 tx=0x7efc4400b8d0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.661+0000 7efc55d07640 1 -- 172.21.15.50:0/2834851283 shutdown_connections 2024-04-23T18:35:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.661+0000 7efc55d07640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efc380a3de0 0x7efc380a62a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.661+0000 7efc55d07640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc50107b80 0x7efc501aaff0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.661+0000 7efc55d07640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc50105510 0x7efc50107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.661+0000 7efc55d07640 1 --2- 172.21.15.50:0/2834851283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc50100490 0x7efc50107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.661+0000 7efc55d07640 1 -- 172.21.15.50:0/2834851283 >> 172.21.15.50:0/2834851283 conn(0x7efc50076fb0 msgr2=0x7efc5010e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.661+0000 7efc55d07640 1 -- 172.21.15.50:0/2834851283 shutdown_connections 2024-04-23T18:35:10.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:10.661+0000 7efc55d07640 1 -- 172.21.15.50:0/2834851283 wait complete. 2024-04-23T18:35:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.665+0000 7fd1a6368640 1 -- 172.21.15.50:0/665437206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1a01010f0 msgr2=0x7fd1a01014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.665+0000 7fd1a6368640 1 --2- 172.21.15.50:0/665437206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1a01010f0 0x7fd1a01014d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd190009860 tx=0x7fd19002a340 comp rx=0 tx=0).stop 2024-04-23T18:35:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.665+0000 7fd1a6368640 1 -- 172.21.15.50:0/665437206 shutdown_connections 2024-04-23T18:35:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.665+0000 7fd1a6368640 1 --2- 172.21.15.50:0/665437206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1a0106170 0x7fd1a010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.665+0000 7fd1a6368640 1 --2- 172.21.15.50:0/665437206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1a0101aa0 0x7fd1a0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.665+0000 7fd1a6368640 1 --2- 172.21.15.50:0/665437206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1a01010f0 0x7fd1a01014d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.665+0000 7fd1a6368640 1 -- 172.21.15.50:0/665437206 >> 172.21.15.50:0/665437206 conn(0x7fd1a0077fd0 msgr2=0x7fd1a00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.665+0000 7fd1a6368640 1 -- 172.21.15.50:0/665437206 shutdown_connections 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.665+0000 7fd1a6368640 1 -- 172.21.15.50:0/665437206 wait complete. 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.666+0000 7fd1a6368640 1 Processor -- start 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.666+0000 7fd1a6368640 1 -- start start 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.666+0000 7fd1a6368640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1a01010f0 0x7fd1a0198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd1a6368640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1a0106170 0x7fd1a0198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd1a6368640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1a019cfc0 0x7fd1a019d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd1a6368640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01110b0 con 0x7fd1a0106170 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd1a6368640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a0110f30 con 0x7fd1a01010f0 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd1a6368640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a0111230 con 0x7fd1a019cfc0 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1a01010f0 0x7fd1a0198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1a0106170 0x7fd1a0198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1a01010f0 0x7fd1a0198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:59792/0 (socket says 172.21.15.50:59792) 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd1a4905640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1a019cfc0 0x7fd1a019d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19ffff640 1 -- 172.21.15.50:0/4046158720 learned_addr learned my addr 172.21.15.50:0/4046158720 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19ffff640 1 -- 172.21.15.50:0/4046158720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1a019cfc0 msgr2=0x7fd1a019d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19ffff640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1a019cfc0 0x7fd1a019d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19ffff640 1 -- 172.21.15.50:0/4046158720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1a0106170 msgr2=0x7fd1a0198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19ffff640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1a0106170 0x7fd1a0198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19ffff640 1 -- 172.21.15.50:0/4046158720 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a019dab0 con 0x7fd1a01010f0 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19f7fe640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1a0106170 0x7fd1a0198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19ffff640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1a01010f0 0x7fd1a0198750 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd190035040 tx=0x7fd190032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19d7fa640 1 -- 172.21.15.50:0/4046158720 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd190037040 con 0x7fd1a01010f0 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.667+0000 7fd19d7fa640 1 -- 172.21.15.50:0/4046158720 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd190041070 con 0x7fd1a01010f0 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.668+0000 7fd19d7fa640 1 -- 172.21.15.50:0/4046158720 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd190003850 con 0x7fd1a01010f0 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.668+0000 7fd1a6368640 1 -- 172.21.15.50:0/4046158720 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a019dd40 con 0x7fd1a01010f0 2024-04-23T18:35:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.668+0000 7fd1a6368640 1 -- 172.21.15.50:0/4046158720 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1a01a54a0 con 0x7fd1a01010f0 2024-04-23T18:35:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.670+0000 7fd19d7fa640 1 -- 172.21.15.50:0/4046158720 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd1900039f0 con 0x7fd1a01010f0 2024-04-23T18:35:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.670+0000 7fd1a8d22740 1 Processor -- start 2024-04-23T18:35:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.670+0000 7fd1a8d22740 1 -- start start 2024-04-23T18:35:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a8d22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fa540 0x55f9df5fa920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd19d7fa640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd1880a3dc0 0x7fd1880a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a8d22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fae60 0x55f9df6096d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a8d22740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df60c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a8d22740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f9df55f970 con 0x55f9df609c10 2024-04-23T18:35:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a8d22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f9df4fc4a0 con 0x55f9df5fa540 2024-04-23T18:35:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a8d22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f9df55b930 con 0x55f9df5fae60 2024-04-23T18:35:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd19ffff640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd1880a3dc0 0x7fd1880a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a4905640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fa540 0x55f9df5fa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd19f7fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fae60 0x55f9df6096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a4905640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fa540 0x55f9df5fa920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:59804/0 (socket says 172.21.15.50:59804) 2024-04-23T18:35:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd19f7fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fae60 0x55f9df6096d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:59166/0 (socket says 172.21.15.50:59166) 2024-04-23T18:35:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a4905640 1 -- 172.21.15.50:0/1246577363 learned_addr learned my addr 172.21.15.50:0/1246577363 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a4905640 1 --2- 172.21.15.50:0/1246577363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df60c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd19d7fa640 1 -- 172.21.15.50:0/4046158720 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8586+0+0 (secure 0 0 0) 0x7fd1900f04e0 con 0x7fd1a01010f0 2024-04-23T18:35:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a4905640 1 -- 172.21.15.50:0/1246577363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fae60 msgr2=0x55f9df6096d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a4905640 1 --2- 172.21.15.50:0/1246577363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fae60 0x55f9df6096d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a4905640 1 -- 172.21.15.50:0/1246577363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 msgr2=0x55f9df60c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a4905640 1 --2- 172.21.15.50:0/1246577363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df60c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a4905640 1 -- 172.21.15.50:0/1246577363 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9df272210 con 0x55f9df5fa540 2024-04-23T18:35:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd19ffff640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd1880a3dc0 0x7fd1880a6280 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd1900b81d0 tx=0x7fd1900b8810 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.671+0000 7fd1a4905640 1 --2- 172.21.15.50:0/1246577363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fa540 0x55f9df5fa920 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd198009fd0 tx=0x7fd198010040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0dbc5f2d5d5f7f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.672+0000 7fd1867fc640 1 -- 172.21.15.50:0/1246577363 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19800cc00 con 0x55f9df5fa540 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.672+0000 7fd1867fc640 1 -- 172.21.15.50:0/1246577363 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd198003e20 con 0x55f9df5fa540 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.672+0000 7fd1867fc640 1 -- 172.21.15.50:0/1246577363 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1980047c0 con 0x55f9df5fa540 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.672+0000 7fd1a8d22740 1 -- 172.21.15.50:0/1246577363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fa540 msgr2=0x55f9df5fa920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.672+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/1246577363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fa540 0x55f9df5fa920 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd198009fd0 tx=0x7fd198010040 comp rx=0 tx=0).stop 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 -- 172.21.15.50:0/1246577363 shutdown_connections 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/1246577363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df60c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/1246577363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fae60 0x55f9df6096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/1246577363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fa540 0x55f9df5fa920 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 -- 172.21.15.50:0/1246577363 >> 172.21.15.50:0/1246577363 conn(0x55f9df5f6b30 msgr2=0x55f9df5f6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 -- 172.21.15.50:0/1246577363 shutdown_connections 2024-04-23T18:35:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 -- 172.21.15.50:0/1246577363 wait complete. 2024-04-23T18:35:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 -- start start 2024-04-23T18:35:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 Processor -- start 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df609580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fae60 0x55f9df606870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df606dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f9df60c7e0 con 0x55f9df609c10 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f9df4fa0b0 con 0x55f9df5fae60 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.673+0000 7fd1a8d22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f9df60c960 con 0x55f9df5fa540 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd1a8d22740 10 client.0 _ll_register_callbacks cb 0x55f9df4e7d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd1a4905640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df606dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd19f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fae60 0x55f9df606870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd1a4905640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df606dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37312/0 (socket says 172.21.15.50:37312) 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd19f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fae60 0x55f9df606870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:59806/0 (socket says 172.21.15.50:59806) 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd1a4905640 1 -- 172.21.15.50:0/3600406788 learned_addr learned my addr 172.21.15.50:0/3600406788 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd1a4905640 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df609580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd19f7fe640 1 -- 172.21.15.50:0/3600406788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 msgr2=0x55f9df609580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd19f7fe640 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df609580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd19f7fe640 1 -- 172.21.15.50:0/3600406788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 msgr2=0x55f9df606dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd19f7fe640 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df606dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd19f7fe640 1 -- 172.21.15.50:0/3600406788 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9df33e620 con 0x55f9df5fae60 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd1a4905640 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df606dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd19f7fe640 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fae60 0x55f9df606870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0105cc0 tx=0x7fd19400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d668e3fba70a0c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd17f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd17f7fe640 1 -- 172.21.15.50:0/3600406788 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194012110 con 0x55f9df5fae60 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd17f7fe640 1 -- 172.21.15.50:0/3600406788 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8586+0+0 (secure 0 0 0) 0x7fd194004050 con 0x55f9df5fae60 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd1a8d22740 1 -- 172.21.15.50:0/3600406788 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd198003e20 con 0x55f9df5fae60 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd17f7fe640 1 -- 172.21.15.50:0/3600406788 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fd1940122b0 con 0x55f9df5fae60 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd1a8d22740 10 client.27047 fetch_fsmap learned FSMap version 54 2024-04-23T18:35:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.674+0000 7fd1a8d22740 1 -- 172.21.15.50:0/3600406788 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55f9df6187b0 con 0x55f9df5fae60 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.675+0000 7fd17f7fe640 1 -- 172.21.15.50:0/3600406788 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd198003e20 con 0x55f9df5fae60 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.675+0000 7fd17f7fe640 1 -- 172.21.15.50:0/3600406788 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7fd194010430 con 0x55f9df5fae60 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.675+0000 7fd1a8d22740 10 client.27047 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.675+0000 7fd1a8d22740 10 client.27047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.675+0000 7fd1a8d22740 20 client.27047 populate_metadata read hostname 'smithi050' 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.675+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f9df619cc0 0x55f9df61c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.675+0000 7fd1a8d22740 4 client.27047 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.675+0000 7fd1a8d22740 1 -- 172.21.15.50:0/3600406788 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9df56d390 con 0x55f9df619cc0 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.675+0000 7fd1a4905640 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f9df619cc0 0x55f9df61c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.675+0000 7fd1a4905640 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f9df619cc0 0x55f9df61c0d0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb25cf3d11fd7f80 server_cookie=6b84adb4f93f8bda in_seq=0 out_seq=0 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.675+0000 7fd17f7fe640 10 client.27047 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd17f7fe640 1 -- 172.21.15.50:0/3600406788 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f9df56d390 con 0x55f9df619cc0 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd17f7fe640 10 client.27047 handle_command_reply: tid=0 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd1a8d22740 1 client.27047 shutdown 2024-04-23T18:35:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd1a8d22740 1 -- 172.21.15.50:0/3600406788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fae60 msgr2=0x55f9df606870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fae60 0x55f9df606870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0105cc0 tx=0x7fd19400f790 comp rx=0 tx=0).stop 2024-04-23T18:35:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd1a8d22740 1 -- 172.21.15.50:0/3600406788 shutdown_connections 2024-04-23T18:35:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f9df619cc0 0x55f9df61c0d0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df606dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fae60 0x55f9df606870 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd17f7fe640 0 client.27047 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/3600406788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df609580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd1a8d22740 1 -- 172.21.15.50:0/3600406788 >> 172.21.15.50:0/3600406788 conn(0x55f9df5f6b30 msgr2=0x55f9df5fc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd1a8d22740 1 -- 172.21.15.50:0/3600406788 shutdown_connections 2024-04-23T18:35:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.676+0000 7fd1a8d22740 1 -- 172.21.15.50:0/3600406788 wait complete. 2024-04-23T18:35:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.677+0000 7fd1a8d22740 20 client.27047 trim_cache size 0 max 16384 2024-04-23T18:35:10.676 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '8c92f231', seconds elapsed: 21.8 2024-04-23T18:35:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.692+0000 7fd1a8d22740 1 Processor -- start 2024-04-23T18:35:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.692+0000 7fd1a8d22740 1 -- start start 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.692+0000 7fd1a8d22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df61c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.692+0000 7fd1a8d22740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df5fae60 0x55f9df6156a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.692+0000 7fd1a8d22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df609c10 0x55f9df5f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.692+0000 7fd1a8d22740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01ae390 con 0x55f9df5fae60 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.692+0000 7fd1a8d22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01abc00 con 0x55f9df609c10 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.692+0000 7fd1a8d22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01a9a70 con 0x55f9df5fa540 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.692+0000 7fd19f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df5fae60 0x55f9df6156a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.692+0000 7fd19f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df5fae60 0x55f9df6156a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37318/0 (socket says 172.21.15.50:37318) 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.692+0000 7fd19f7fe640 1 -- 172.21.15.50:0/1746137695 learned_addr learned my addr 172.21.15.50:0/1746137695 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.693+0000 7fd1a4905640 1 --2- 172.21.15.50:0/1746137695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df609c10 0x55f9df5f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.693+0000 7fd1a4905640 1 --2- 172.21.15.50:0/1746137695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df61c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.693+0000 7fd19f7fe640 1 -- 172.21.15.50:0/1746137695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 msgr2=0x55f9df61c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.693+0000 7fd19f7fe640 1 --2- 172.21.15.50:0/1746137695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df61c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.693+0000 7fd19f7fe640 1 -- 172.21.15.50:0/1746137695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df609c10 msgr2=0x55f9df5f6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.693+0000 7fd19f7fe640 1 --2- 172.21.15.50:0/1746137695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df609c10 0x55f9df5f6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.693+0000 7fd19f7fe640 1 -- 172.21.15.50:0/1746137695 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9df605820 con 0x55f9df5fae60 2024-04-23T18:35:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.693+0000 7fd1a4905640 1 --2- 172.21.15.50:0/1746137695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df61c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:35:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.693+0000 7fd19f7fe640 1 --2- 172.21.15.50:0/1746137695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df5fae60 0x55f9df6156a0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fd194010a40 tx=0x7fd194012830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3a35a2d2484580 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.693+0000 7fd185ffb640 1 -- 172.21.15.50:0/1746137695 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194010b80 con 0x55f9df5fae60 2024-04-23T18:35:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.693+0000 7fd185ffb640 1 -- 172.21.15.50:0/1746137695 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd194005590 con 0x55f9df5fae60 2024-04-23T18:35:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.693+0000 7fd185ffb640 1 -- 172.21.15.50:0/1746137695 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1940039e0 con 0x55f9df5fae60 2024-04-23T18:35:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.694+0000 7fd1a8d22740 1 -- 172.21.15.50:0/1746137695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df5fae60 msgr2=0x55f9df6156a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.694+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/1746137695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df5fae60 0x55f9df6156a0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fd194010a40 tx=0x7fd194012830 comp rx=0 tx=0).stop 2024-04-23T18:35:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.694+0000 7fd1a8d22740 1 -- 172.21.15.50:0/1746137695 shutdown_connections 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.694+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/1746137695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df609c10 0x55f9df5f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.694+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/1746137695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df5fae60 0x55f9df6156a0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.694+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/1746137695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df61c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.694+0000 7fd1a8d22740 1 -- 172.21.15.50:0/1746137695 >> 172.21.15.50:0/1746137695 conn(0x55f9df5f6b30 msgr2=0x55f9df601f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.694+0000 7fd1a8d22740 1 -- 172.21.15.50:0/1746137695 shutdown_connections 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.694+0000 7fd1a8d22740 1 -- 172.21.15.50:0/1746137695 wait complete. 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.694+0000 7fd1a8d22740 1 -- start start 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a8d22740 1 Processor -- start 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a8d22740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df611e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a8d22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fae60 0x55f9df60d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a8d22740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df60d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a8d22740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01aa0c0 con 0x55f9df609c10 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a8d22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01ab530 con 0x55f9df5fae60 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a8d22740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01aa790 con 0x55f9df5fa540 2024-04-23T18:35:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a8d22740 10 client.0 _ll_register_callbacks cb 0x7fd194012910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a4905640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df60d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a4905640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df60d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37324/0 (socket says 172.21.15.50:37324) 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a4905640 1 -- 172.21.15.50:0/201295949 learned_addr learned my addr 172.21.15.50:0/201295949 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd19f7fe640 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fae60 0x55f9df60d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a4905640 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df611e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a4905640 1 -- 172.21.15.50:0/201295949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 msgr2=0x55f9df611e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a4905640 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df611e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a4905640 1 -- 172.21.15.50:0/201295949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fae60 msgr2=0x55f9df60d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a4905640 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fae60 0x55f9df60d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a4905640 1 -- 172.21.15.50:0/201295949 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9df60dd20 con 0x55f9df609c10 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd1a4905640 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df60d6b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fd198009dc0 tx=0x7fd1940128a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e8c619cca88416e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.695+0000 7fd17f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd17f7fe640 1 -- 172.21.15.50:0/201295949 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1980095c0 con 0x55f9df609c10 2024-04-23T18:35:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd17f7fe640 1 -- 172.21.15.50:0/201295949 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8586+0+0 (secure 0 0 0) 0x7fd198003c80 con 0x55f9df609c10 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd1a8d22740 1 -- 172.21.15.50:0/201295949 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd194005590 con 0x55f9df609c10 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd17f7fe640 1 -- 172.21.15.50:0/201295949 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fd19803d080 con 0x55f9df609c10 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd1a8d22740 10 client.17547 fetch_fsmap learned FSMap version 54 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd1a8d22740 1 -- 172.21.15.50:0/201295949 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55f9df6800a0 con 0x55f9df609c10 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd17f7fe640 1 -- 172.21.15.50:0/201295949 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd194005590 con 0x55f9df609c10 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd17f7fe640 1 -- 172.21.15.50:0/201295949 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7fd19800b040 con 0x55f9df609c10 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd1a8d22740 10 client.17547 fetch_fsmap finished waiting for FSMap version 54 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd1a8d22740 10 client.17547 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd1a8d22740 20 client.17547 populate_metadata read hostname 'smithi050' 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f9df619cc0 0x55f9df6826a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd1a8d22740 4 client.17547 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.696+0000 7fd1a8d22740 1 -- 172.21.15.50:0/201295949 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f9df567200 con 0x55f9df619cc0 2024-04-23T18:35:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.697+0000 7fd19f7fe640 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f9df619cc0 0x55f9df6826a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.697+0000 7fd17f7fe640 10 client.17547 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.697+0000 7fd19f7fe640 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f9df619cc0 0x55f9df6826a0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5c4f133d25e1cac server_cookie=1798d5b6078a4277 in_seq=0 out_seq=0 2024-04-23T18:35:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd17f7fe640 1 -- 172.21.15.50:0/201295949 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f9df567200 con 0x55f9df619cc0 2024-04-23T18:35:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd17f7fe640 10 client.17547 handle_command_reply: tid=0 2024-04-23T18:35:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd1a8d22740 1 client.17547 shutdown 2024-04-23T18:35:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd1a8d22740 1 -- 172.21.15.50:0/201295949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 msgr2=0x55f9df60d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df60d6b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fd198009dc0 tx=0x7fd1940128a0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd1a8d22740 1 -- 172.21.15.50:0/201295949 shutdown_connections 2024-04-23T18:35:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f9df619cc0 0x55f9df6826a0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f9df609c10 0x55f9df60d6b0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd17f7fe640 0 client.17547 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f9df5fae60 0x55f9df60d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd1a8d22740 1 --2- 172.21.15.50:0/201295949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f9df5fa540 0x55f9df611e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd1a8d22740 1 -- 172.21.15.50:0/201295949 >> 172.21.15.50:0/201295949 conn(0x55f9df5f6b30 msgr2=0x55f9df601f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd1a8d22740 1 -- 172.21.15.50:0/201295949 shutdown_connections 2024-04-23T18:35:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd1a8d22740 1 -- 172.21.15.50:0/201295949 wait complete. 2024-04-23T18:35:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.773+0000 7fd1a8d22740 20 client.17547 trim_cache size 0 max 16384 2024-04-23T18:35:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:35:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:35:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:35:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:35:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.774+0000 7fd1a6368640 1 -- 172.21.15.50:0/4046158720 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd1880a3dc0 msgr2=0x7fd1880a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.774+0000 7fd1a6368640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd1880a3dc0 0x7fd1880a6280 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd1900b81d0 tx=0x7fd1900b8810 comp rx=0 tx=0).stop 2024-04-23T18:35:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.774+0000 7fd1a6368640 1 -- 172.21.15.50:0/4046158720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1a01010f0 msgr2=0x7fd1a0198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.774+0000 7fd1a6368640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1a01010f0 0x7fd1a0198750 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd190035040 tx=0x7fd190032d50 comp rx=0 tx=0).stop 2024-04-23T18:35:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.774+0000 7fd1a6368640 1 -- 172.21.15.50:0/4046158720 shutdown_connections 2024-04-23T18:35:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.774+0000 7fd1a6368640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd1880a3dc0 0x7fd1880a6280 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.774+0000 7fd1a6368640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1a019cfc0 0x7fd1a019d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.774+0000 7fd1a6368640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1a0106170 0x7fd1a0198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.774+0000 7fd1a6368640 1 --2- 172.21.15.50:0/4046158720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1a01010f0 0x7fd1a0198750 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.774+0000 7fd1a6368640 1 -- 172.21.15.50:0/4046158720 >> 172.21.15.50:0/4046158720 conn(0x7fd1a0077fd0 msgr2=0x7fd1a00699e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.774+0000 7fd1a6368640 1 -- 172.21.15.50:0/4046158720 shutdown_connections 2024-04-23T18:35:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:10.774+0000 7fd1a6368640 1 -- 172.21.15.50:0/4046158720 wait complete. 2024-04-23T18:35:11.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:11 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2449169727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:35:11.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:11 smithi177 ceph-mon[32308]: osdmap e116: 12 total, 12 up, 12 in 2024-04-23T18:35:11.537 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:35:11 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: ignoring --setuser ceph since I am not root 2024-04-23T18:35:11.537 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:35:11 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: ignoring --setgroup ceph since I am not root 2024-04-23T18:35:11.537 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:35:11 smithi177 ceph-mds[66745]: -- 172.21.15.177:0/2596955549 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5650ae4a1a40 con 0x5650ae474c00 2024-04-23T18:35:11.537 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:35:11 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: starting mds.h at 2024-04-23T18:35:11.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 102403 25.69 MB/sec execute 105 sec latency 20976.036 ms 2024-04-23T18:35:11.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:11 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2449169727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:35:11.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:11 smithi188 ceph-mon[32694]: osdmap e116: 12 total, 12 up, 12 in 2024-04-23T18:35:11.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:11 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2449169727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:35:11.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:11 smithi050 ceph-mon[29187]: osdmap e116: 12 total, 12 up, 12 in 2024-04-23T18:35:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 102966 25.78 MB/sec execute 106 sec latency 89.923 ms 2024-04-23T18:35:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:12 smithi188 ceph-mon[32694]: pgmap v528: 97 pgs: 97 active+clean; 998 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 6.3 MiB/s wr, 295 op/s 2024-04-23T18:35:12.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:12 smithi188 ceph-mon[32694]: daemon mds.h restarted 2024-04-23T18:35:12.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:12 smithi050 ceph-mon[29187]: pgmap v528: 97 pgs: 97 active+clean; 998 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 6.3 MiB/s wr, 295 op/s 2024-04-23T18:35:12.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:12 smithi050 ceph-mon[29187]: daemon mds.h restarted 2024-04-23T18:35:12.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:12 smithi177 ceph-mon[32308]: pgmap v528: 97 pgs: 97 active+clean; 998 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 6.3 MiB/s wr, 295 op/s 2024-04-23T18:35:12.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:12 smithi177 ceph-mon[32308]: daemon mds.h restarted 2024-04-23T18:35:13.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 103542 25.81 MB/sec execute 107 sec latency 100.016 ms 2024-04-23T18:35:13.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:13 smithi188 ceph-mon[32694]: osdmap e117: 12 total, 12 up, 12 in 2024-04-23T18:35:13.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:13 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6836/3902222090,v1:172.21.15.177:6837/3902222090] up:boot 2024-04-23T18:35:13.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:13 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:35:13.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:13 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:35:13.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:13 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:35:13.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:13 smithi050 ceph-mon[29187]: osdmap e117: 12 total, 12 up, 12 in 2024-04-23T18:35:13.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:13 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6836/3902222090,v1:172.21.15.177:6837/3902222090] up:boot 2024-04-23T18:35:13.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:13 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:35:13.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:13 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:35:13.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:13 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:35:13.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:13 smithi177 ceph-mon[32308]: osdmap e117: 12 total, 12 up, 12 in 2024-04-23T18:35:13.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:13 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6836/3902222090,v1:172.21.15.177:6837/3902222090] up:boot 2024-04-23T18:35:13.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:13 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:35:13.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:13 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:35:13.787 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:13 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:35:14.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 104112 25.87 MB/sec execute 108 sec latency 111.448 ms 2024-04-23T18:35:14.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:14 smithi188 ceph-mon[32694]: pgmap v531: 97 pgs: 97 active+clean; 978 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 8.6 MiB/s wr, 374 op/s 2024-04-23T18:35:14.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:14 smithi050 ceph-mon[29187]: pgmap v531: 97 pgs: 97 active+clean; 978 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 8.6 MiB/s wr, 374 op/s 2024-04-23T18:35:14.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:14 smithi177 ceph-mon[32308]: pgmap v531: 97 pgs: 97 active+clean; 978 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 8.6 MiB/s wr, 374 op/s 2024-04-23T18:35:15.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 104687 25.88 MB/sec execute 109 sec latency 235.515 ms 2024-04-23T18:35:16.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 105317 25.99 MB/sec execute 110 sec latency 120.808 ms 2024-04-23T18:35:16.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:16 smithi188 ceph-mon[32694]: pgmap v532: 97 pgs: 97 active+clean; 978 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 4.6 MiB/s wr, 242 op/s 2024-04-23T18:35:16.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:16 smithi050 ceph-mon[29187]: pgmap v532: 97 pgs: 97 active+clean; 978 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 4.6 MiB/s wr, 242 op/s 2024-04-23T18:35:16.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:16 smithi177 ceph-mon[32308]: pgmap v532: 97 pgs: 97 active+clean; 978 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 4.6 MiB/s wr, 242 op/s 2024-04-23T18:35:17.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 105991 26.07 MB/sec execute 111 sec latency 50.110 ms 2024-04-23T18:35:18.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 106639 26.16 MB/sec execute 112 sec latency 859.233 ms 2024-04-23T18:35:18.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:18 smithi188 ceph-mon[32694]: pgmap v533: 97 pgs: 97 active+clean; 1010 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 13 MiB/s wr, 722 op/s 2024-04-23T18:35:18.587 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:18 smithi177 ceph-mon[32308]: pgmap v533: 97 pgs: 97 active+clean; 1010 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 13 MiB/s wr, 722 op/s 2024-04-23T18:35:18.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:18 smithi050 ceph-mon[29187]: pgmap v533: 97 pgs: 97 active+clean; 1010 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 13 MiB/s wr, 722 op/s 2024-04-23T18:35:19.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 107173 26.14 MB/sec execute 113 sec latency 1859.273 ms 2024-04-23T18:35:20.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 107500 26.14 MB/sec execute 114 sec latency 2859.318 ms 2024-04-23T18:35:20.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:20 smithi188 ceph-mon[32694]: pgmap v534: 97 pgs: 97 active+clean; 1010 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 13 MiB/s wr, 722 op/s 2024-04-23T18:35:20.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:20 smithi050 ceph-mon[29187]: pgmap v534: 97 pgs: 97 active+clean; 1010 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 13 MiB/s wr, 722 op/s 2024-04-23T18:35:20.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:20 smithi177 ceph-mon[32308]: pgmap v534: 97 pgs: 97 active+clean; 1010 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 13 MiB/s wr, 722 op/s 2024-04-23T18:35:21.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 107943 26.08 MB/sec execute 115 sec latency 3859.387 ms 2024-04-23T18:35:22.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 108291 26.02 MB/sec execute 116 sec latency 4859.416 ms 2024-04-23T18:35:22.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:22 smithi188 ceph-mon[32694]: pgmap v535: 97 pgs: 97 active+clean; 1010 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 602 op/s 2024-04-23T18:35:22.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:22 smithi050 ceph-mon[29187]: pgmap v535: 97 pgs: 97 active+clean; 1010 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 602 op/s 2024-04-23T18:35:22.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:22 smithi177 ceph-mon[32308]: pgmap v535: 97 pgs: 97 active+clean; 1010 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 602 op/s 2024-04-23T18:35:23.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 108624 25.96 MB/sec execute 117 sec latency 5859.448 ms 2024-04-23T18:35:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.877+0000 7fd51e507640 1 -- 172.21.15.50:0/850788345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd518105de0 msgr2=0x7fd5181061c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.877+0000 7fd51e507640 1 --2- 172.21.15.50:0/850788345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd518105de0 0x7fd5181061c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd508009900 tx=0x7fd50802a3f0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.877+0000 7fd51e507640 1 -- 172.21.15.50:0/850788345 shutdown_connections 2024-04-23T18:35:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.877+0000 7fd51e507640 1 --2- 172.21.15.50:0/850788345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd518101a60 0x7fd51810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.877+0000 7fd51e507640 1 --2- 172.21.15.50:0/850788345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5181010a0 0x7fd518101520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.877+0000 7fd51e507640 1 --2- 172.21.15.50:0/850788345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd518105de0 0x7fd5181061c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.877+0000 7fd51e507640 1 -- 172.21.15.50:0/850788345 >> 172.21.15.50:0/850788345 conn(0x7fd518077d00 msgr2=0x7fd518078110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.877+0000 7fd51e507640 1 -- 172.21.15.50:0/850788345 shutdown_connections 2024-04-23T18:35:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.877+0000 7fd51e507640 1 -- 172.21.15.50:0/850788345 wait complete. 2024-04-23T18:35:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.878+0000 7fd51e507640 1 Processor -- start 2024-04-23T18:35:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.878+0000 7fd51e507640 1 -- start start 2024-04-23T18:35:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd51e507640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5181010a0 0x7fd518199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd51e507640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd518101a60 0x7fd518199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd51e507640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd518105de0 0x7fd5181a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd51e507640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181111c0 con 0x7fd518101a60 2024-04-23T18:35:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd51e507640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd518111040 con 0x7fd518105de0 2024-04-23T18:35:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd51e507640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd518111340 con 0x7fd5181010a0 2024-04-23T18:35:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd51caa4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd518105de0 0x7fd5181a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd5177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd518101a60 0x7fd518199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd517fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5181010a0 0x7fd518199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd517fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5181010a0 0x7fd518199560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:58794/0 (socket says 172.21.15.50:58794) 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd51caa4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd518105de0 0x7fd5181a45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:37672/0 (socket says 172.21.15.50:37672) 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd517fff640 1 -- 172.21.15.50:0/2645368164 learned_addr learned my addr 172.21.15.50:0/2645368164 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd517fff640 1 -- 172.21.15.50:0/2645368164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd518105de0 msgr2=0x7fd5181a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd517fff640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd518105de0 0x7fd5181a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd517fff640 1 -- 172.21.15.50:0/2645368164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd518101a60 msgr2=0x7fd518199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd517fff640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd518101a60 0x7fd518199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd517fff640 1 -- 172.21.15.50:0/2645368164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5181a4b20 con 0x7fd5181010a0 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.879+0000 7fd5177fe640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd518101a60 0x7fd518199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.880+0000 7fd517fff640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5181010a0 0x7fd518199560 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd508005ab0 tx=0x7fd508003c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.880+0000 7fd5157fa640 1 -- 172.21.15.50:0/2645368164 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50803f070 con 0x7fd5181010a0 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.880+0000 7fd5157fa640 1 -- 172.21.15.50:0/2645368164 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd508031d60 con 0x7fd5181010a0 2024-04-23T18:35:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.880+0000 7fd5157fa640 1 -- 172.21.15.50:0/2645368164 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508035030 con 0x7fd5181010a0 2024-04-23T18:35:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.880+0000 7fd51e507640 1 -- 172.21.15.50:0/2645368164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5181a4cf0 con 0x7fd5181010a0 2024-04-23T18:35:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.880+0000 7fd51e507640 1 -- 172.21.15.50:0/2645368164 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5181a50f0 con 0x7fd5181010a0 2024-04-23T18:35:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.883+0000 7fd5157fa640 1 -- 172.21.15.50:0/2645368164 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd508042070 con 0x7fd5181010a0 2024-04-23T18:35:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.883+0000 7fd52121b740 1 Processor -- start 2024-04-23T18:35:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.883+0000 7fd52121b740 1 -- start start 2024-04-23T18:35:23.882 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd52121b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af0188aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.882 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd52121b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188af40 0x55af018952c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.882 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd52121b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af01895800 0x55af01897bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.882 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd52121b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55af017ef970 con 0x55af0188af40 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd52121b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55af0178c4a0 con 0x55af01895800 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd52121b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55af017eb930 con 0x55af0188a620 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd51caa4640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af0188aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd5157fa640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd5000a3dc0 0x7fd5000a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd51caa4640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af0188aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:58804/0 (socket says 172.21.15.50:58804) 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd517fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188af40 0x55af018952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd51caa4640 1 -- 172.21.15.50:0/2700865903 learned_addr learned my addr 172.21.15.50:0/2700865903 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd5177fe640 1 --2- 172.21.15.50:0/2700865903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af01895800 0x55af01897bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd51caa4640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd5000a3dc0 0x7fd5000a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd51caa4640 1 -- 172.21.15.50:0/2700865903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af01895800 msgr2=0x55af01897bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd51caa4640 1 --2- 172.21.15.50:0/2700865903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af01895800 0x55af01897bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd51caa4640 1 -- 172.21.15.50:0/2700865903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188af40 msgr2=0x55af018952c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd5157fa640 1 -- 172.21.15.50:0/2645368164 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7fd5080f0fc0 con 0x7fd5181010a0 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd51caa4640 1 --2- 172.21.15.50:0/2700865903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188af40 0x55af018952c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd51caa4640 1 -- 172.21.15.50:0/2700865903 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af01502210 con 0x55af0188a620 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.884+0000 7fd517fff640 1 --2- 172.21.15.50:0/2700865903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188af40 0x55af018952c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.885+0000 7fd51caa4640 1 --2- 172.21.15.50:0/2700865903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af0188aa00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd510005c40 tx=0x7fd510013f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a26948d952b8cd82 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.885+0000 7fd51caa4640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd5000a3dc0 0x7fd5000a6280 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd51001d000 tx=0x7fd51001f000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.885+0000 7fd4fe7fc640 1 -- 172.21.15.50:0/2700865903 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51000fde0 con 0x55af0188a620 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.885+0000 7fd4fe7fc640 1 -- 172.21.15.50:0/2700865903 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd510022020 con 0x55af0188a620 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.885+0000 7fd4fe7fc640 1 -- 172.21.15.50:0/2700865903 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd510010690 con 0x55af0188a620 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.886+0000 7fd52121b740 1 -- 172.21.15.50:0/2700865903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 msgr2=0x55af0188aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.886+0000 7fd52121b740 1 --2- 172.21.15.50:0/2700865903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af0188aa00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd510005c40 tx=0x7fd510013f50 comp rx=0 tx=0).stop 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.886+0000 7fd52121b740 1 -- 172.21.15.50:0/2700865903 shutdown_connections 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.886+0000 7fd52121b740 1 --2- 172.21.15.50:0/2700865903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af01895800 0x55af01897bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.886+0000 7fd52121b740 1 --2- 172.21.15.50:0/2700865903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188af40 0x55af018952c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.886+0000 7fd52121b740 1 --2- 172.21.15.50:0/2700865903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af0188aa00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.886+0000 7fd52121b740 1 -- 172.21.15.50:0/2700865903 >> 172.21.15.50:0/2700865903 conn(0x55af01886b30 msgr2=0x55af01886f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.886+0000 7fd52121b740 1 -- 172.21.15.50:0/2700865903 shutdown_connections 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.886+0000 7fd52121b740 1 -- 172.21.15.50:0/2700865903 wait complete. 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.886+0000 7fd52121b740 1 -- start start 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd52121b740 1 Processor -- start 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd52121b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af018951b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd52121b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af018924a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd52121b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 0x55af01892a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd52121b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55af018983d0 con 0x55af01895800 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd52121b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55af0178a0b0 con 0x55af0188af40 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd52121b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55af01898550 con 0x55af0188a620 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd517fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af018951b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd51caa4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 0x55af01892a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd517fff640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af018951b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:58806/0 (socket says 172.21.15.50:58806) 2024-04-23T18:35:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd517fff640 1 -- 172.21.15.50:0/3907288671 learned_addr learned my addr 172.21.15.50:0/3907288671 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd5177fe640 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af018924a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd52121b740 10 client.0 _ll_register_callbacks cb 0x55af01777d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:35:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd51caa4640 1 -- 172.21.15.50:0/3907288671 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 msgr2=0x55af018951b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd51caa4640 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af018951b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd51caa4640 1 -- 172.21.15.50:0/3907288671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 msgr2=0x55af018924a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd51caa4640 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af018924a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd51caa4640 1 -- 172.21.15.50:0/3907288671 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af015ce620 con 0x55af01895800 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd517fff640 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af018951b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.887+0000 7fd5177fe640 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af018924a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd51caa4640 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 0x55af01892a40 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fd510000f80 tx=0x7fd510012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f86a4210303a3a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd4f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd4f77fe640 1 -- 172.21.15.50:0/3907288671 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd510034060 con 0x55af01895800 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd4f77fe640 1 -- 172.21.15.50:0/3907288671 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7fd510004de0 con 0x55af01895800 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd52121b740 1 -- 172.21.15.50:0/3907288671 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd510022020 con 0x55af01895800 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd4f77fe640 1 -- 172.21.15.50:0/3907288671 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fd51000fde0 con 0x55af01895800 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd52121b740 10 client.17565 fetch_fsmap learned FSMap version 56 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd52121b740 1 -- 172.21.15.50:0/3907288671 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55af018a4300 con 0x55af01895800 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd4f77fe640 1 -- 172.21.15.50:0/3907288671 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd510022020 con 0x55af01895800 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd4f77fe640 1 -- 172.21.15.50:0/3907288671 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 56) ==== 8354+0+0 (secure 0 0 0) 0x7fd510010430 con 0x55af01895800 2024-04-23T18:35:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd52121b740 10 client.17565 fetch_fsmap finished waiting for FSMap version 56 2024-04-23T18:35:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd52121b740 10 client.17565 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:35:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd52121b740 20 client.17565 populate_metadata read hostname 'smithi050' 2024-04-23T18:35:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.888+0000 7fd52121b740 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55af018a5760 0x55af018a7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.889+0000 7fd52121b740 4 client.17565 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:35:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.889+0000 7fd52121b740 1 -- 172.21.15.50:0/3907288671 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af017fd390 con 0x55af018a5760 2024-04-23T18:35:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.889+0000 7fd517fff640 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55af018a5760 0x55af018a7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.889+0000 7fd517fff640 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55af018a5760 0x55af018a7b50 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9202b182159f36e4 server_cookie=f76dc252b02e74ee in_seq=0 out_seq=0 2024-04-23T18:35:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.889+0000 7fd4f77fe640 10 client.17565 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.891+0000 7fd4f77fe640 1 -- 172.21.15.50:0/3907288671 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55af017fd390 con 0x55af018a5760 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.891+0000 7fd4f77fe640 10 client.17565 handle_command_reply: tid=0 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.891+0000 7fd52121b740 1 client.17565 shutdown 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.891+0000 7fd52121b740 1 -- 172.21.15.50:0/3907288671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 msgr2=0x55af01892a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.891+0000 7fd52121b740 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 0x55af01892a40 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fd510000f80 tx=0x7fd510012c00 comp rx=0 tx=0).stop 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.892+0000 7fd52121b740 1 -- 172.21.15.50:0/3907288671 shutdown_connections 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.892+0000 7fd52121b740 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55af018a5760 0x55af018a7b50 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.892+0000 7fd52121b740 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 0x55af01892a40 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.892+0000 7fd52121b740 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af018924a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.892+0000 7fd52121b740 1 --2- 172.21.15.50:0/3907288671 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af018951b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.892+0000 7fd52121b740 1 -- 172.21.15.50:0/3907288671 >> 172.21.15.50:0/3907288671 conn(0x55af01886b30 msgr2=0x55af0188c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.892+0000 7fd4f77fe640 0 client.17565 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.892+0000 7fd52121b740 1 -- 172.21.15.50:0/3907288671 shutdown_connections 2024-04-23T18:35:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.892+0000 7fd52121b740 1 -- 172.21.15.50:0/3907288671 wait complete. 2024-04-23T18:35:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.892+0000 7fd52121b740 20 client.17565 trim_cache size 0 max 16384 2024-04-23T18:35:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.908+0000 7fd52121b740 1 Processor -- start 2024-04-23T18:35:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.908+0000 7fd52121b740 1 -- start start 2024-04-23T18:35:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.908+0000 7fd52121b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af018a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.908+0000 7fd52121b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af01885b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.908+0000 7fd52121b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 0x55af01894ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.908+0000 7fd52121b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181b0360 con 0x55af01895800 2024-04-23T18:35:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.908+0000 7fd52121b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181adbd0 con 0x55af0188af40 2024-04-23T18:35:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.908+0000 7fd52121b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181aba20 con 0x55af0188a620 2024-04-23T18:35:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd51caa4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 0x55af01894ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd5177fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af01885b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd51caa4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 0x55af01894ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38242/0 (socket says 172.21.15.50:38242) 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd5177fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af01885b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:37706/0 (socket says 172.21.15.50:37706) 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd51caa4640 1 -- 172.21.15.50:0/3492509038 learned_addr learned my addr 172.21.15.50:0/3492509038 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd517fff640 1 --2- 172.21.15.50:0/3492509038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af018a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd51caa4640 1 -- 172.21.15.50:0/3492509038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 msgr2=0x55af018a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd51caa4640 1 --2- 172.21.15.50:0/3492509038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af018a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd51caa4640 1 -- 172.21.15.50:0/3492509038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 msgr2=0x55af01885b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd51caa4640 1 --2- 172.21.15.50:0/3492509038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af01885b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd51caa4640 1 -- 172.21.15.50:0/3492509038 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af018874b0 con 0x55af01895800 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd5177fe640 1 --2- 172.21.15.50:0/3492509038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af01885b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd51caa4640 1 --2- 172.21.15.50:0/3492509038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 0x55af01894ce0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fd510010850 tx=0x7fd5100011c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf4a057dac37e1d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd4fdffb640 1 -- 172.21.15.50:0/3492509038 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd510034040 con 0x55af01895800 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd4fdffb640 1 -- 172.21.15.50:0/3492509038 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd510022070 con 0x55af01895800 2024-04-23T18:35:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.909+0000 7fd4fdffb640 1 -- 172.21.15.50:0/3492509038 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5100105d0 con 0x55af01895800 2024-04-23T18:35:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.910+0000 7fd52121b740 1 -- 172.21.15.50:0/3492509038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 msgr2=0x55af01894ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.910+0000 7fd52121b740 1 --2- 172.21.15.50:0/3492509038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 0x55af01894ce0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fd510010850 tx=0x7fd5100011c0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.910+0000 7fd52121b740 1 -- 172.21.15.50:0/3492509038 shutdown_connections 2024-04-23T18:35:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.910+0000 7fd52121b740 1 --2- 172.21.15.50:0/3492509038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af01895800 0x55af01894ce0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.910+0000 7fd52121b740 1 --2- 172.21.15.50:0/3492509038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af01885b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.910+0000 7fd52121b740 1 --2- 172.21.15.50:0/3492509038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af0188a620 0x55af018a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.910+0000 7fd52121b740 1 -- 172.21.15.50:0/3492509038 >> 172.21.15.50:0/3492509038 conn(0x55af01886b30 msgr2=0x55af01885420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.910+0000 7fd52121b740 1 -- 172.21.15.50:0/3492509038 shutdown_connections 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.910+0000 7fd52121b740 1 -- 172.21.15.50:0/3492509038 wait complete. 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.910+0000 7fd52121b740 1 -- start start 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd52121b740 1 Processor -- start 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd52121b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188a620 0x55af01898ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd52121b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af01899000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd52121b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af01895800 0x55af01903bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd52121b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181ac090 con 0x55af0188a620 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd52121b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181ad500 con 0x55af0188af40 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd52121b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181ac760 con 0x55af01895800 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd52121b740 10 client.0 _ll_register_callbacks cb 0x7fd5100138c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd517fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188a620 0x55af01898ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd517fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188a620 0x55af01898ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38246/0 (socket says 172.21.15.50:38246) 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd517fff640 1 -- 172.21.15.50:0/1798239224 learned_addr learned my addr 172.21.15.50:0/1798239224 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd5177fe640 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af01899000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd51caa4640 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af01895800 0x55af01903bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd517fff640 1 -- 172.21.15.50:0/1798239224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af01895800 msgr2=0x55af01903bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd517fff640 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af01895800 0x55af01903bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd517fff640 1 -- 172.21.15.50:0/1798239224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 msgr2=0x55af01899000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd517fff640 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af01899000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.911+0000 7fd517fff640 1 -- 172.21.15.50:0/1798239224 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af01904260 con 0x55af0188a620 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd517fff640 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188a620 0x55af01898ac0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fd5080bcf50 tx=0x7fd5080dd6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f879bfd820c596 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd4f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd4f77fe640 1 -- 172.21.15.50:0/1798239224 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5080d5070 con 0x55af0188a620 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd4f77fe640 1 -- 172.21.15.50:0/1798239224 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7fd5080ee940 con 0x55af0188a620 2024-04-23T18:35:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd52121b740 1 -- 172.21.15.50:0/1798239224 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd510022070 con 0x55af0188a620 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd4f77fe640 1 -- 172.21.15.50:0/1798239224 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fd508049040 con 0x55af0188a620 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd52121b740 10 client.17571 fetch_fsmap learned FSMap version 56 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd52121b740 1 -- 172.21.15.50:0/1798239224 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55af0190b610 con 0x55af0188a620 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd4f77fe640 1 -- 172.21.15.50:0/1798239224 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd510022070 con 0x55af0188a620 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd4f77fe640 1 -- 172.21.15.50:0/1798239224 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 56) ==== 8354+0+0 (secure 0 0 0) 0x7fd5080d6260 con 0x55af0188a620 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd52121b740 10 client.17571 fetch_fsmap finished waiting for FSMap version 56 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd52121b740 10 client.17571 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd52121b740 20 client.17571 populate_metadata read hostname 'smithi050' 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd52121b740 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55af0190bd90 0x55af0190e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd52121b740 4 client.17571 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.912+0000 7fd52121b740 1 -- 172.21.15.50:0/1798239224 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55af017f7200 con 0x55af0190bd90 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.913+0000 7fd5177fe640 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55af0190bd90 0x55af0190e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.913+0000 7fd4f77fe640 10 client.17571 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:23.913+0000 7fd5177fe640 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55af0190bd90 0x55af0190e1b0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3bed2e41b176e10 server_cookie=d2d611cc7d00335e in_seq=0 out_seq=0 2024-04-23T18:35:24.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 108978 25.88 MB/sec execute 118 sec latency 6413.569 ms 2024-04-23T18:35:24.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:24 smithi188 ceph-mon[32694]: pgmap v536: 97 pgs: 97 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 664 op/s 2024-04-23T18:35:24.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:24 smithi050 ceph-mon[29187]: pgmap v536: 97 pgs: 97 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 664 op/s 2024-04-23T18:35:24.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:24 smithi177 ceph-mon[32308]: pgmap v536: 97 pgs: 97 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 664 op/s 2024-04-23T18:35:25.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 109464 25.87 MB/sec execute 119 sec latency 5793.812 ms 2024-04-23T18:35:26.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 109979 25.88 MB/sec execute 120 sec latency 6529.627 ms 2024-04-23T18:35:26.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:26 smithi188 ceph-mon[32694]: pgmap v537: 97 pgs: 97 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 10 MiB/s wr, 580 op/s 2024-04-23T18:35:26.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:26 smithi050 ceph-mon[29187]: pgmap v537: 97 pgs: 97 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 10 MiB/s wr, 580 op/s 2024-04-23T18:35:26.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:26 smithi177 ceph-mon[32308]: pgmap v537: 97 pgs: 97 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 10 MiB/s wr, 580 op/s 2024-04-23T18:35:27.569 INFO:tasks.workunit.client.0.smithi050.stdout: 10 110543 25.88 MB/sec execute 121 sec latency 7529.665 ms 2024-04-23T18:35:28.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 111208 26.05 MB/sec execute 122 sec latency 8529.737 ms 2024-04-23T18:35:28.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:28 smithi188 ceph-mon[32694]: pgmap v538: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 13 MiB/s wr, 815 op/s 2024-04-23T18:35:28.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:28 smithi050 ceph-mon[29187]: pgmap v538: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 13 MiB/s wr, 815 op/s 2024-04-23T18:35:28.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:28 smithi177 ceph-mon[32308]: pgmap v538: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 13 MiB/s wr, 815 op/s 2024-04-23T18:35:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd4f77fe640 1 -- 172.21.15.50:0/1798239224 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55af017f7200 con 0x55af0190bd90 2024-04-23T18:35:29.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd4f77fe640 10 client.17571 handle_command_reply: tid=0 2024-04-23T18:35:29.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd52121b740 1 client.17571 shutdown 2024-04-23T18:35:29.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd52121b740 1 -- 172.21.15.50:0/1798239224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188a620 msgr2=0x55af01898ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:29.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd52121b740 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188a620 0x55af01898ac0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fd5080bcf50 tx=0x7fd5080dd6c0 comp rx=0 tx=0).stop 2024-04-23T18:35:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd52121b740 1 -- 172.21.15.50:0/1798239224 shutdown_connections 2024-04-23T18:35:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd52121b740 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55af0190bd90 0x55af0190e1b0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd52121b740 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af01895800 0x55af01903bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd52121b740 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af0188af40 0x55af01899000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd4f77fe640 0 client.17571 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd52121b740 1 --2- 172.21.15.50:0/1798239224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55af0188a620 0x55af01898ac0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd52121b740 1 -- 172.21.15.50:0/1798239224 >> 172.21.15.50:0/1798239224 conn(0x55af01886b30 msgr2=0x55af01885cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd52121b740 1 -- 172.21.15.50:0/1798239224 shutdown_connections 2024-04-23T18:35:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd52121b740 1 -- 172.21.15.50:0/1798239224 wait complete. 2024-04-23T18:35:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.945+0000 7fd52121b740 20 client.17571 trim_cache size 0 max 16384 2024-04-23T18:35:29.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:35:29.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:35:29.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:35:29.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:35:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.946+0000 7fd51e507640 1 -- 172.21.15.50:0/2645368164 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd5000a3dc0 msgr2=0x7fd5000a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.946+0000 7fd51e507640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd5000a3dc0 0x7fd5000a6280 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd51001d000 tx=0x7fd51001f000 comp rx=0 tx=0).stop 2024-04-23T18:35:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.946+0000 7fd51e507640 1 -- 172.21.15.50:0/2645368164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5181010a0 msgr2=0x7fd518199560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.946+0000 7fd51e507640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5181010a0 0x7fd518199560 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd508005ab0 tx=0x7fd508003c80 comp rx=0 tx=0).stop 2024-04-23T18:35:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.946+0000 7fd51e507640 1 -- 172.21.15.50:0/2645368164 shutdown_connections 2024-04-23T18:35:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.946+0000 7fd51e507640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd5000a3dc0 0x7fd5000a6280 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.946+0000 7fd51e507640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd518105de0 0x7fd5181a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.946+0000 7fd51e507640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd518101a60 0x7fd518199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.946+0000 7fd51e507640 1 --2- 172.21.15.50:0/2645368164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5181010a0 0x7fd518199560 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.946+0000 7fd51e507640 1 -- 172.21.15.50:0/2645368164 >> 172.21.15.50:0/2645368164 conn(0x7fd518077d00 msgr2=0x7fd5180fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:29.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.946+0000 7fd51e507640 1 -- 172.21.15.50:0/2645368164 shutdown_connections 2024-04-23T18:35:29.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:28.946+0000 7fd51e507640 1 -- 172.21.15.50:0/2645368164 wait complete. 2024-04-23T18:35:29.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 111629 26.00 MB/sec execute 123 sec latency 9529.777 ms 2024-04-23T18:35:30.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 112104 25.96 MB/sec execute 124 sec latency 10085.506 ms 2024-04-23T18:35:30.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:30 smithi188 ceph-mon[32694]: pgmap v539: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.8 MiB/s wr, 494 op/s 2024-04-23T18:35:30.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:30 smithi050 ceph-mon[29187]: pgmap v539: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.8 MiB/s wr, 494 op/s 2024-04-23T18:35:30.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:30 smithi177 ceph-mon[32308]: pgmap v539: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.8 MiB/s wr, 494 op/s 2024-04-23T18:35:31.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 112724 26.04 MB/sec execute 125 sec latency 94.419 ms 2024-04-23T18:35:32.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 113327 26.07 MB/sec execute 126 sec latency 104.053 ms 2024-04-23T18:35:32.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:32 smithi188 ceph-mon[32694]: pgmap v540: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.8 MiB/s wr, 494 op/s 2024-04-23T18:35:32.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:32 smithi050 ceph-mon[29187]: pgmap v540: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.8 MiB/s wr, 494 op/s 2024-04-23T18:35:32.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:32 smithi177 ceph-mon[32308]: pgmap v540: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.8 MiB/s wr, 494 op/s 2024-04-23T18:35:33.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 113948 26.16 MB/sec execute 127 sec latency 104.111 ms 2024-04-23T18:35:34.571 INFO:tasks.workunit.client.0.smithi050.stdout: 10 114543 26.17 MB/sec execute 128 sec latency 90.904 ms 2024-04-23T18:35:34.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:34 smithi188 ceph-mon[32694]: pgmap v541: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 10 MiB/s wr, 676 op/s 2024-04-23T18:35:34.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:34 smithi050 ceph-mon[29187]: pgmap v541: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 10 MiB/s wr, 676 op/s 2024-04-23T18:35:34.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:34 smithi177 ceph-mon[32308]: pgmap v541: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 10 MiB/s wr, 676 op/s 2024-04-23T18:35:35.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 115136 26.24 MB/sec execute 129 sec latency 81.561 ms 2024-04-23T18:35:36.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 115635 26.24 MB/sec execute 130 sec latency 114.988 ms 2024-04-23T18:35:36.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:36 smithi188 ceph-mon[32694]: pgmap v542: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.6 MiB/s wr, 416 op/s 2024-04-23T18:35:36.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:36 smithi050 ceph-mon[29187]: pgmap v542: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.6 MiB/s wr, 416 op/s 2024-04-23T18:35:36.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:36 smithi177 ceph-mon[32308]: pgmap v542: 97 pgs: 97 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.6 MiB/s wr, 416 op/s 2024-04-23T18:35:37.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 116285 26.30 MB/sec execute 131 sec latency 83.971 ms 2024-04-23T18:35:38.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 116888 26.35 MB/sec execute 132 sec latency 112.762 ms 2024-04-23T18:35:38.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:38 smithi188 ceph-mon[32694]: pgmap v543: 97 pgs: 97 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.3 MiB/s wr, 600 op/s 2024-04-23T18:35:38.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:38 smithi050 ceph-mon[29187]: pgmap v543: 97 pgs: 97 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.3 MiB/s wr, 600 op/s 2024-04-23T18:35:38.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:38 smithi177 ceph-mon[32308]: pgmap v543: 97 pgs: 97 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.3 MiB/s wr, 600 op/s 2024-04-23T18:35:39.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 117443 26.40 MB/sec execute 133 sec latency 79.265 ms 2024-04-23T18:35:40.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 117891 26.38 MB/sec execute 134 sec latency 271.106 ms 2024-04-23T18:35:40.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:40 smithi188 ceph-mon[32694]: pgmap v544: 97 pgs: 97 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 5.4 MiB/s wr, 365 op/s 2024-04-23T18:35:40.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:40 smithi050 ceph-mon[29187]: pgmap v544: 97 pgs: 97 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 5.4 MiB/s wr, 365 op/s 2024-04-23T18:35:40.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:40 smithi177 ceph-mon[32308]: pgmap v544: 97 pgs: 97 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 5.4 MiB/s wr, 365 op/s 2024-04-23T18:35:41.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 118410 26.40 MB/sec execute 135 sec latency 161.630 ms 2024-04-23T18:35:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.051+0000 7f2284672640 1 -- 172.21.15.50:0/3261100309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f227c106170 msgr2=0x7f227c10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.051+0000 7f2284672640 1 --2- 172.21.15.50:0/3261100309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f227c106170 0x7f227c10dd40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2274009ca0 tx=0x7f227402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.051+0000 7f2284672640 1 -- 172.21.15.50:0/3261100309 shutdown_connections 2024-04-23T18:35:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.051+0000 7f2284672640 1 --2- 172.21.15.50:0/3261100309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f227c106170 0x7f227c10dd40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.051+0000 7f2284672640 1 --2- 172.21.15.50:0/3261100309 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f227c101aa0 0x7f227c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.051+0000 7f2284672640 1 --2- 172.21.15.50:0/3261100309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f227c1010f0 0x7f227c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.051+0000 7f2284672640 1 -- 172.21.15.50:0/3261100309 >> 172.21.15.50:0/3261100309 conn(0x7f227c077fd0 msgr2=0x7f227c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.051+0000 7f2284672640 1 -- 172.21.15.50:0/3261100309 shutdown_connections 2024-04-23T18:35:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.051+0000 7f2284672640 1 -- 172.21.15.50:0/3261100309 wait complete. 2024-04-23T18:35:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.052+0000 7f2284672640 1 Processor -- start 2024-04-23T18:35:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.052+0000 7f2284672640 1 -- start start 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.052+0000 7f2284672640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f227c1010f0 0x7f227c1a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.052+0000 7f2284672640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f227c106170 0x7f227c1a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.052+0000 7f2284672640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f227c19a690 0x7f227c19ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.052+0000 7f2284672640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f227c1110e0 con 0x7f227c106170 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.052+0000 7f2284672640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f227c110f60 con 0x7f227c19a690 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.052+0000 7f2284672640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f227c111260 con 0x7f227c1010f0 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.053+0000 7f2281c0d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f227c106170 0x7f227c1a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.053+0000 7f228240e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f227c1010f0 0x7f227c1a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.053+0000 7f2282c0f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f227c19a690 0x7f227c19ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.053+0000 7f2281c0d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f227c106170 0x7f227c1a0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35958/0 (socket says 172.21.15.50:35958) 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.053+0000 7f2281c0d640 1 -- 172.21.15.50:0/769201982 learned_addr learned my addr 172.21.15.50:0/769201982 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.053+0000 7f2281c0d640 1 -- 172.21.15.50:0/769201982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f227c1010f0 msgr2=0x7f227c1a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.053+0000 7f2281c0d640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f227c1010f0 0x7f227c1a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.053+0000 7f2281c0d640 1 -- 172.21.15.50:0/769201982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f227c19a690 msgr2=0x7f227c19ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.053+0000 7f2281c0d640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f227c19a690 0x7f227c19ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.053+0000 7f2281c0d640 1 -- 172.21.15.50:0/769201982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227c19b290 con 0x7f227c106170 2024-04-23T18:35:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.053+0000 7f2282c0f640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f227c19a690 0x7f227c19ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.054+0000 7f2281c0d640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f227c106170 0x7f227c1a0b10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f226c00e9a0 tx=0x7f226c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.054+0000 7f22737fe640 1 -- 172.21.15.50:0/769201982 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c00cd80 con 0x7f227c106170 2024-04-23T18:35:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.054+0000 7f2284672640 1 -- 172.21.15.50:0/769201982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f227c1a7370 con 0x7f227c106170 2024-04-23T18:35:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.054+0000 7f22737fe640 1 -- 172.21.15.50:0/769201982 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f226c0035f0 con 0x7f227c106170 2024-04-23T18:35:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.054+0000 7f22737fe640 1 -- 172.21.15.50:0/769201982 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c010690 con 0x7f227c106170 2024-04-23T18:35:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.054+0000 7f2284672640 1 -- 172.21.15.50:0/769201982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f227c1a7770 con 0x7f227c106170 2024-04-23T18:35:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.057+0000 7f22737fe640 1 -- 172.21.15.50:0/769201982 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f226c004000 con 0x7f227c106170 2024-04-23T18:35:42.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.057+0000 7f2287384740 1 Processor -- start 2024-04-23T18:35:42.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f2287384740 1 -- start start 2024-04-23T18:35:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f2287384740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc130a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f2287384740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130f40 0x5601dc13b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f2287384740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc13b800 0x5601dc13dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f2287384740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5601dc095970 con 0x5601dc130620 2024-04-23T18:35:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f2287384740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5601dc0324a0 con 0x5601dc130f40 2024-04-23T18:35:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f2287384740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5601dc091930 con 0x5601dc13b800 2024-04-23T18:35:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f2282c0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc130a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f2282c0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc130a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35970/0 (socket says 172.21.15.50:35970) 2024-04-23T18:35:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f2282c0f640 1 -- 172.21.15.50:0/176782439 learned_addr learned my addr 172.21.15.50:0/176782439 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f228240e640 1 --2- 172.21.15.50:0/176782439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130f40 0x5601dc13b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f2281c0d640 1 --2- 172.21.15.50:0/176782439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc13b800 0x5601dc13dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.058+0000 7f22737fe640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22680a3f20 0x7f22680a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.059+0000 7f228240e640 1 -- 172.21.15.50:0/176782439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc13b800 msgr2=0x5601dc13dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.059+0000 7f228240e640 1 --2- 172.21.15.50:0/176782439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc13b800 0x5601dc13dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.059+0000 7f228240e640 1 -- 172.21.15.50:0/176782439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 msgr2=0x5601dc130a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.059+0000 7f22737fe640 1 -- 172.21.15.50:0/769201982 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7f226c0d08d0 con 0x7f227c106170 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.059+0000 7f228240e640 1 --2- 172.21.15.50:0/176782439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc130a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.059+0000 7f228240e640 1 -- 172.21.15.50:0/176782439 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601dbda8210 con 0x5601dc130f40 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.059+0000 7f2282c0f640 1 --2- 172.21.15.50:0/176782439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc130a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.059+0000 7f2282c0f640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22680a3f20 0x7f22680a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.059+0000 7f228240e640 1 --2- 172.21.15.50:0/176782439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130f40 0x5601dc13b2c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2278009c00 tx=0x7f2278011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=958f7def28ee190a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.059+0000 7f2270ff9640 1 -- 172.21.15.50:0/176782439 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278004230 con 0x5601dc130f40 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.059+0000 7f2270ff9640 1 -- 172.21.15.50:0/176782439 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22780043d0 con 0x5601dc130f40 2024-04-23T18:35:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.060+0000 7f2270ff9640 1 -- 172.21.15.50:0/176782439 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278004c10 con 0x5601dc130f40 2024-04-23T18:35:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.060+0000 7f2287384740 1 -- 172.21.15.50:0/176782439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130f40 msgr2=0x5601dc13b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.060+0000 7f2282c0f640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22680a3f20 0x7f22680a63e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2274035000 tx=0x7f22740095c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.060+0000 7f2287384740 1 --2- 172.21.15.50:0/176782439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130f40 0x5601dc13b2c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2278009c00 tx=0x7f2278011810 comp rx=0 tx=0).stop 2024-04-23T18:35:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.061+0000 7f2287384740 1 -- 172.21.15.50:0/176782439 shutdown_connections 2024-04-23T18:35:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.061+0000 7f2287384740 1 --2- 172.21.15.50:0/176782439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc13b800 0x5601dc13dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.061+0000 7f2287384740 1 --2- 172.21.15.50:0/176782439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130f40 0x5601dc13b2c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.061+0000 7f2287384740 1 --2- 172.21.15.50:0/176782439 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc130a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.061+0000 7f2287384740 1 -- 172.21.15.50:0/176782439 >> 172.21.15.50:0/176782439 conn(0x5601dc12cb30 msgr2=0x5601dc12cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.061+0000 7f2287384740 1 -- 172.21.15.50:0/176782439 shutdown_connections 2024-04-23T18:35:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.061+0000 7f2287384740 1 -- 172.21.15.50:0/176782439 wait complete. 2024-04-23T18:35:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.061+0000 7f2287384740 1 -- start start 2024-04-23T18:35:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.061+0000 7f2287384740 1 Processor -- start 2024-04-23T18:35:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f2287384740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130620 0x5601dc13b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f2287384740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc130f40 0x5601dc1383c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f2287384740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc13b800 0x5601dc138900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f2287384740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5601dc13e430 con 0x5601dc13b800 2024-04-23T18:35:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f2287384740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5601dc0300b0 con 0x5601dc130620 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f2287384740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5601dc13e5b0 con 0x5601dc130f40 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f2287384740 10 client.0 _ll_register_callbacks cb 0x5601dc01dd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f2282c0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc13b800 0x5601dc138900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f228240e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130620 0x5601dc13b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f228240e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130620 0x5601dc13b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:53310/0 (socket says 172.21.15.50:53310) 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f2281c0d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc130f40 0x5601dc1383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f228240e640 1 -- 172.21.15.50:0/971547156 learned_addr learned my addr 172.21.15.50:0/971547156 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f228240e640 1 -- 172.21.15.50:0/971547156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc130f40 msgr2=0x5601dc1383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f228240e640 1 --2- 172.21.15.50:0/971547156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc130f40 0x5601dc1383c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f228240e640 1 -- 172.21.15.50:0/971547156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc13b800 msgr2=0x5601dc138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f228240e640 1 --2- 172.21.15.50:0/971547156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc13b800 0x5601dc138900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f228240e640 1 -- 172.21.15.50:0/971547156 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601dbe74620 con 0x5601dc130620 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f2282c0f640 1 --2- 172.21.15.50:0/971547156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc13b800 0x5601dc138900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:42.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f228240e640 1 --2- 172.21.15.50:0/971547156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130620 0x5601dc13b0d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f22780048f0 tx=0x7f2278019a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3c94c585a43cea1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.062+0000 7f2265ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2265ffb640 1 -- 172.21.15.50:0/971547156 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278028020 con 0x5601dc130620 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2265ffb640 1 -- 172.21.15.50:0/971547156 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7f2278024620 con 0x5601dc130620 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2287384740 1 -- 172.21.15.50:0/971547156 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22780043d0 con 0x5601dc130620 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2265ffb640 1 -- 172.21.15.50:0/971547156 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f2278033050 con 0x5601dc130620 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2287384740 10 client.27089 fetch_fsmap learned FSMap version 56 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2287384740 1 -- 172.21.15.50:0/971547156 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x5601dc14a2d0 con 0x5601dc130620 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2265ffb640 1 -- 172.21.15.50:0/971547156 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22780043d0 con 0x5601dc130620 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2265ffb640 1 -- 172.21.15.50:0/971547156 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 56) ==== 8354+0+0 (secure 0 0 0) 0x7f2278018070 con 0x5601dc130620 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2287384740 10 client.27089 fetch_fsmap finished waiting for FSMap version 56 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2287384740 10 client.27089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2287384740 20 client.27089 populate_metadata read hostname 'smithi050' 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2287384740 1 --2- 172.21.15.50:0/971547156 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5601dc14b6d0 0x5601dc14dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2287384740 4 client.27089 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.063+0000 7f2287384740 1 -- 172.21.15.50:0/971547156 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601dc0a3390 con 0x5601dc14b6d0 2024-04-23T18:35:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.064+0000 7f2281c0d640 1 --2- 172.21.15.50:0/971547156 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5601dc14b6d0 0x5601dc14dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.064+0000 7f2281c0d640 1 --2- 172.21.15.50:0/971547156 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5601dc14b6d0 0x5601dc14dac0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc089afe41cfd722 server_cookie=777f74b2c3911790 in_seq=0 out_seq=0 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.064+0000 7f2265ffb640 10 client.27089 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2265ffb640 1 -- 172.21.15.50:0/971547156 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5601dc0a3390 con 0x5601dc14b6d0 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2265ffb640 10 client.27089 handle_command_reply: tid=0 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2287384740 1 client.27089 shutdown 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2287384740 1 -- 172.21.15.50:0/971547156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130620 msgr2=0x5601dc13b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2287384740 1 --2- 172.21.15.50:0/971547156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130620 0x5601dc13b0d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f22780048f0 tx=0x7f2278019a50 comp rx=0 tx=0).stop 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2287384740 1 -- 172.21.15.50:0/971547156 shutdown_connections 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2287384740 1 --2- 172.21.15.50:0/971547156 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5601dc14b6d0 0x5601dc14dac0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2287384740 1 --2- 172.21.15.50:0/971547156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc13b800 0x5601dc138900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2287384740 1 --2- 172.21.15.50:0/971547156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc130f40 0x5601dc1383c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2287384740 1 --2- 172.21.15.50:0/971547156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130620 0x5601dc13b0d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2265ffb640 0 client.27089 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2287384740 1 -- 172.21.15.50:0/971547156 >> 172.21.15.50:0/971547156 conn(0x5601dc12cb30 msgr2=0x5601dc1328b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2287384740 1 -- 172.21.15.50:0/971547156 shutdown_connections 2024-04-23T18:35:42.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2287384740 1 -- 172.21.15.50:0/971547156 wait complete. 2024-04-23T18:35:42.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.066+0000 7f2287384740 20 client.27089 trim_cache size 0 max 16384 2024-04-23T18:35:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.084+0000 7f2287384740 1 Processor -- start 2024-04-23T18:35:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.084+0000 7f2287384740 1 -- start start 2024-04-23T18:35:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.084+0000 7f2287384740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc13ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.084+0000 7f2287384740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc130f40 0x5601dc1532f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.084+0000 7f2287384740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc13b800 0x5601dc153830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.084+0000 7f2287384740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f227c1b26b0 con 0x5601dc130620 2024-04-23T18:35:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.084+0000 7f2287384740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f227c1aff20 con 0x5601dc13b800 2024-04-23T18:35:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.084+0000 7f2287384740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f227c1add70 con 0x5601dc130f40 2024-04-23T18:35:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.084+0000 7f228240e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc13ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.084+0000 7f228240e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc13ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35994/0 (socket says 172.21.15.50:35994) 2024-04-23T18:35:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.084+0000 7f228240e640 1 -- 172.21.15.50:0/699730855 learned_addr learned my addr 172.21.15.50:0/699730855 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.085+0000 7f2282c0f640 1 --2- 172.21.15.50:0/699730855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc13b800 0x5601dc153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.085+0000 7f2281c0d640 1 --2- 172.21.15.50:0/699730855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc130f40 0x5601dc1532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.085+0000 7f228240e640 1 -- 172.21.15.50:0/699730855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc130f40 msgr2=0x5601dc1532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.085+0000 7f228240e640 1 --2- 172.21.15.50:0/699730855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc130f40 0x5601dc1532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.085+0000 7f228240e640 1 -- 172.21.15.50:0/699730855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc13b800 msgr2=0x5601dc153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.085+0000 7f228240e640 1 --2- 172.21.15.50:0/699730855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc13b800 0x5601dc153830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.085+0000 7f228240e640 1 -- 172.21.15.50:0/699730855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601dc13e310 con 0x5601dc130620 2024-04-23T18:35:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.085+0000 7f228240e640 1 --2- 172.21.15.50:0/699730855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc13ac00 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f2278009800 tx=0x7f2278011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae55180deff5200 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.085+0000 7f2267fff640 1 -- 172.21.15.50:0/699730855 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278028040 con 0x5601dc130620 2024-04-23T18:35:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.085+0000 7f2267fff640 1 -- 172.21.15.50:0/699730855 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f227801f040 con 0x5601dc130620 2024-04-23T18:35:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.085+0000 7f2267fff640 1 -- 172.21.15.50:0/699730855 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278018b50 con 0x5601dc130620 2024-04-23T18:35:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.086+0000 7f2287384740 1 -- 172.21.15.50:0/699730855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 msgr2=0x5601dc13ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.086+0000 7f2287384740 1 --2- 172.21.15.50:0/699730855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc13ac00 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f2278009800 tx=0x7f2278011810 comp rx=0 tx=0).stop 2024-04-23T18:35:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.086+0000 7f2287384740 1 -- 172.21.15.50:0/699730855 shutdown_connections 2024-04-23T18:35:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.086+0000 7f2287384740 1 --2- 172.21.15.50:0/699730855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc13b800 0x5601dc153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.086+0000 7f2287384740 1 --2- 172.21.15.50:0/699730855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc130f40 0x5601dc1532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.086+0000 7f2287384740 1 --2- 172.21.15.50:0/699730855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc13ac00 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.086+0000 7f2287384740 1 -- 172.21.15.50:0/699730855 >> 172.21.15.50:0/699730855 conn(0x5601dc12cb30 msgr2=0x5601dc1384e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.086+0000 7f2287384740 1 -- 172.21.15.50:0/699730855 shutdown_connections 2024-04-23T18:35:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.086+0000 7f2287384740 1 -- 172.21.15.50:0/699730855 wait complete. 2024-04-23T18:35:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.086+0000 7f2287384740 1 -- start start 2024-04-23T18:35:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f2287384740 1 Processor -- start 2024-04-23T18:35:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f2287384740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc140630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f2287384740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130f40 0x5601dc140b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f2287384740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc13b800 0x5601dc1afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f2287384740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f227c1ae3e0 con 0x5601dc130620 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f2287384740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f227c1af850 con 0x5601dc130f40 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f2287384740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f227c1aeab0 con 0x5601dc13b800 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f2287384740 10 client.0 _ll_register_callbacks cb 0x7f2278009a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f228240e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc140630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f2282c0f640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc13b800 0x5601dc1afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f228240e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc140630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36010/0 (socket says 172.21.15.50:36010) 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f228240e640 1 -- 172.21.15.50:0/3488439165 learned_addr learned my addr 172.21.15.50:0/3488439165 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f2281c0d640 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130f40 0x5601dc140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f228240e640 1 -- 172.21.15.50:0/3488439165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc13b800 msgr2=0x5601dc1afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f228240e640 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc13b800 0x5601dc1afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f228240e640 1 -- 172.21.15.50:0/3488439165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130f40 msgr2=0x5601dc140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f228240e640 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130f40 0x5601dc140b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f228240e640 1 -- 172.21.15.50:0/3488439165 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601dc1b00a0 con 0x5601dc130620 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f2282c0f640 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc13b800 0x5601dc1afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.087+0000 7f2281c0d640 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130f40 0x5601dc140b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f228240e640 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc140630 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f2278018b00 tx=0x7f2278018b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7881b2cb33fd952e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2265ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2265ffb640 1 -- 172.21.15.50:0/3488439165 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278028030 con 0x5601dc130620 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2265ffb640 1 -- 172.21.15.50:0/3488439165 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7f2278005310 con 0x5601dc130620 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2287384740 1 -- 172.21.15.50:0/3488439165 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f227801f040 con 0x5601dc130620 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2265ffb640 1 -- 172.21.15.50:0/3488439165 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f2278048050 con 0x5601dc130620 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2287384740 10 client.17589 fetch_fsmap learned FSMap version 56 2024-04-23T18:35:42.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2287384740 1 -- 172.21.15.50:0/3488439165 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x5601dc1b1210 con 0x5601dc130620 2024-04-23T18:35:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2265ffb640 1 -- 172.21.15.50:0/3488439165 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f227801f040 con 0x5601dc130620 2024-04-23T18:35:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2265ffb640 1 -- 172.21.15.50:0/3488439165 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 56) ==== 8354+0+0 (secure 0 0 0) 0x7f2278018040 con 0x5601dc130620 2024-04-23T18:35:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2287384740 10 client.17589 fetch_fsmap finished waiting for FSMap version 56 2024-04-23T18:35:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2287384740 10 client.17589 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:35:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2287384740 20 client.17589 populate_metadata read hostname 'smithi050' 2024-04-23T18:35:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.088+0000 7f2287384740 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5601dc1b1960 0x5601dc1b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.089+0000 7f2287384740 4 client.17589 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:35:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.089+0000 7f2287384740 1 -- 172.21.15.50:0/3488439165 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5601dc09d200 con 0x5601dc1b1960 2024-04-23T18:35:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.089+0000 7f2281c0d640 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5601dc1b1960 0x5601dc1b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.089+0000 7f2265ffb640 10 client.17589 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.089+0000 7f2281c0d640 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5601dc1b1960 0x5601dc1b3d80 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84d08d0d9c79022e server_cookie=457d70f6c71218a1 in_seq=0 out_seq=0 2024-04-23T18:35:42.368 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.375+0000 7f2265ffb640 1 -- 172.21.15.50:0/3488439165 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5601dc09d200 con 0x5601dc1b1960 2024-04-23T18:35:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.375+0000 7f2265ffb640 10 client.17589 handle_command_reply: tid=0 2024-04-23T18:35:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.375+0000 7f2287384740 1 client.17589 shutdown 2024-04-23T18:35:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.375+0000 7f2287384740 1 -- 172.21.15.50:0/3488439165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 msgr2=0x5601dc140630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.375+0000 7f2287384740 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc140630 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f2278018b00 tx=0x7f2278018b30 comp rx=0 tx=0).stop 2024-04-23T18:35:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.375+0000 7f2287384740 1 -- 172.21.15.50:0/3488439165 shutdown_connections 2024-04-23T18:35:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.375+0000 7f2287384740 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5601dc1b1960 0x5601dc1b3d80 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.375+0000 7f2265ffb640 0 client.17589 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.375+0000 7f2287384740 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5601dc13b800 0x5601dc1afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.375+0000 7f2287384740 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601dc130f40 0x5601dc140b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.375+0000 7f2287384740 1 --2- 172.21.15.50:0/3488439165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5601dc130620 0x5601dc140630 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.376+0000 7f2287384740 1 -- 172.21.15.50:0/3488439165 >> 172.21.15.50:0/3488439165 conn(0x5601dc12cb30 msgr2=0x5601dc12b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.376+0000 7f2287384740 1 -- 172.21.15.50:0/3488439165 shutdown_connections 2024-04-23T18:35:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.376+0000 7f2287384740 1 -- 172.21.15.50:0/3488439165 wait complete. 2024-04-23T18:35:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.376+0000 7f2287384740 20 client.17589 trim_cache size 0 max 16384 2024-04-23T18:35:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:35:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:35:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:35:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:35:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.377+0000 7f2284672640 1 -- 172.21.15.50:0/769201982 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22680a3f20 msgr2=0x7f22680a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.377+0000 7f2284672640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22680a3f20 0x7f22680a63e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2274035000 tx=0x7f22740095c0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.377+0000 7f2284672640 1 -- 172.21.15.50:0/769201982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f227c106170 msgr2=0x7f227c1a0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.377+0000 7f2284672640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f227c106170 0x7f227c1a0b10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f226c00e9a0 tx=0x7f226c00ee60 comp rx=0 tx=0).stop 2024-04-23T18:35:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.377+0000 7f2284672640 1 -- 172.21.15.50:0/769201982 shutdown_connections 2024-04-23T18:35:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.377+0000 7f2284672640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22680a3f20 0x7f22680a63e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.377+0000 7f2284672640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f227c19a690 0x7f227c19ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.377+0000 7f2284672640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f227c106170 0x7f227c1a0b10 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.377+0000 7f2284672640 1 --2- 172.21.15.50:0/769201982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f227c1010f0 0x7f227c1a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.377+0000 7f2284672640 1 -- 172.21.15.50:0/769201982 >> 172.21.15.50:0/769201982 conn(0x7f227c077fd0 msgr2=0x7f227c10cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.377+0000 7f2284672640 1 -- 172.21.15.50:0/769201982 shutdown_connections 2024-04-23T18:35:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:42.377+0000 7f2284672640 1 -- 172.21.15.50:0/769201982 wait complete. 2024-04-23T18:35:42.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 118835 26.34 MB/sec execute 136 sec latency 209.583 ms 2024-04-23T18:35:42.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:42 smithi188 ceph-mon[32694]: pgmap v545: 97 pgs: 97 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 5.4 MiB/s wr, 365 op/s 2024-04-23T18:35:42.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:42 smithi050 ceph-mon[29187]: pgmap v545: 97 pgs: 97 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 5.4 MiB/s wr, 365 op/s 2024-04-23T18:35:42.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:42 smithi177 ceph-mon[32308]: pgmap v545: 97 pgs: 97 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 5.4 MiB/s wr, 365 op/s 2024-04-23T18:35:43.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 119521 26.42 MB/sec execute 137 sec latency 100.734 ms 2024-04-23T18:35:44.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 120128 26.46 MB/sec execute 138 sec latency 74.998 ms 2024-04-23T18:35:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:44 smithi050 ceph-mon[29187]: pgmap v546: 97 pgs: 97 active+clean; 1.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.8 MiB/s wr, 533 op/s 2024-04-23T18:35:44.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:44 smithi177 ceph-mon[32308]: pgmap v546: 97 pgs: 97 active+clean; 1.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.8 MiB/s wr, 533 op/s 2024-04-23T18:35:44.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:44 smithi188 ceph-mon[32694]: pgmap v546: 97 pgs: 97 active+clean; 1.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.8 MiB/s wr, 533 op/s 2024-04-23T18:35:45.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 120629 26.49 MB/sec execute 139 sec latency 246.851 ms 2024-04-23T18:35:46.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121111 26.51 MB/sec execute 140 sec latency 119.941 ms 2024-04-23T18:35:46.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:46 smithi050 ceph-mon[29187]: pgmap v547: 97 pgs: 97 active+clean; 1.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.2 MiB/s wr, 352 op/s 2024-04-23T18:35:46.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:46 smithi177 ceph-mon[32308]: pgmap v547: 97 pgs: 97 active+clean; 1.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.2 MiB/s wr, 352 op/s 2024-04-23T18:35:46.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:46 smithi188 ceph-mon[32694]: pgmap v547: 97 pgs: 97 active+clean; 1.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.2 MiB/s wr, 352 op/s 2024-04-23T18:35:47.449 INFO:tasks.quiescer.fs.[cephfs]:Total duration (39.4) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:35:47.449 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.7 2024-04-23T18:35:47.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:35:47.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.576+0000 7f06849a1640 1 -- 172.21.15.50:0/2947413957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0680106670 msgr2=0x7f0680106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:47.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.576+0000 7f06849a1640 1 --2- 172.21.15.50:0/2947413957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0680106670 0x7f0680106a50 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f0670009900 tx=0x7f067002a410 comp rx=0 tx=0).stop 2024-04-23T18:35:47.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.576+0000 7f06849a1640 1 -- 172.21.15.50:0/2947413957 shutdown_connections 2024-04-23T18:35:47.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.576+0000 7f06849a1640 1 --2- 172.21.15.50:0/2947413957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0680101240 0x7f068010d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:47.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.576+0000 7f06849a1640 1 --2- 172.21.15.50:0/2947413957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0680100880 0x7f0680100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:47.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.576+0000 7f06849a1640 1 --2- 172.21.15.50:0/2947413957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0680106670 0x7f0680106a50 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:47.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.576+0000 7f06849a1640 1 -- 172.21.15.50:0/2947413957 >> 172.21.15.50:0/2947413957 conn(0x7f06800761d0 msgr2=0x7f06800765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:47.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.576+0000 7f06849a1640 1 -- 172.21.15.50:0/2947413957 shutdown_connections 2024-04-23T18:35:47.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.576+0000 7f06849a1640 1 -- 172.21.15.50:0/2947413957 wait complete. 2024-04-23T18:35:47.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.577+0000 7f06849a1640 1 Processor -- start 2024-04-23T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.577+0000 7f06849a1640 1 -- start start 2024-04-23T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.577+0000 7f06849a1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0680100880 0x7f068010eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.577+0000 7f06849a1640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0680101240 0x7f068010f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.577+0000 7f06849a1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0680106670 0x7f068010f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.577+0000 7f06849a1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0680112220 con 0x7f0680100880 2024-04-23T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.577+0000 7f06849a1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f06801120a0 con 0x7f0680106670 2024-04-23T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.577+0000 7f06849a1640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f06801123a0 con 0x7f0680101240 2024-04-23T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.578+0000 7f067e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0680100880 0x7f068010eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.578+0000 7f067ed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0680106670 0x7f068010f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.578+0000 7f067e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0680100880 0x7f068010eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36026/0 (socket says 172.21.15.50:36026) 2024-04-23T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.578+0000 7f067e59c640 1 -- 172.21.15.50:0/2335368888 learned_addr learned my addr 172.21.15.50:0/2335368888 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.578+0000 7f0677fff640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0680101240 0x7f068010f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.578+0000 7f067ed9d640 1 -- 172.21.15.50:0/2335368888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0680101240 msgr2=0x7f068010f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.578+0000 7f067ed9d640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0680101240 0x7f068010f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.578+0000 7f067ed9d640 1 -- 172.21.15.50:0/2335368888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0680100880 msgr2=0x7f068010eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.578+0000 7f067ed9d640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0680100880 0x7f068010eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.578+0000 7f067ed9d640 1 -- 172.21.15.50:0/2335368888 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06801ab190 con 0x7f0680106670 2024-04-23T18:35:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.578+0000 7f067ed9d640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0680106670 0x7f068010f5d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0678009fd0 tx=0x7f067800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.579+0000 7f06777fe640 1 -- 172.21.15.50:0/2335368888 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067800ce20 con 0x7f0680106670 2024-04-23T18:35:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.579+0000 7f06777fe640 1 -- 172.21.15.50:0/2335368888 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0678005020 con 0x7f0680106670 2024-04-23T18:35:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.579+0000 7f06777fe640 1 -- 172.21.15.50:0/2335368888 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0678010690 con 0x7f0680106670 2024-04-23T18:35:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.579+0000 7f06849a1640 1 -- 172.21.15.50:0/2335368888 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06801ab3c0 con 0x7f0680106670 2024-04-23T18:35:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.579+0000 7f06849a1640 1 -- 172.21.15.50:0/2335368888 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06801ab8c0 con 0x7f0680106670 2024-04-23T18:35:47.573 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121597 26.49 MB/sec execute 141 sec latency 121.820 ms 2024-04-23T18:35:47.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.580+0000 7f06849a1640 1 -- 172.21.15.50:0/2335368888 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0660005290 con 0x7f0680106670 2024-04-23T18:35:47.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.581+0000 7f06777fe640 1 -- 172.21.15.50:0/2335368888 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0678004a40 con 0x7f0680106670 2024-04-23T18:35:47.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.582+0000 7f06777fe640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f066c0a3d20 0x7f066c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:47.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.582+0000 7f067e59c640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f066c0a3d20 0x7f066c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:47.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.582+0000 7f06777fe640 1 -- 172.21.15.50:0/2335368888 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7f06780d0d10 con 0x7f0680106670 2024-04-23T18:35:47.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.582+0000 7f067e59c640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f066c0a3d20 0x7f066c0a61e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0670031440 tx=0x7f06700058d0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:47.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.590+0000 7f06777fe640 1 -- 172.21.15.50:0/2335368888 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f067809d3c0 con 0x7f0680106670 2024-04-23T18:35:47.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.866+0000 7f06849a1640 1 -- 172.21.15.50:0/2335368888 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0660005850 con 0x7f0680106670 2024-04-23T18:35:47.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.867+0000 7f06777fe640 1 -- 172.21.15.50:0/2335368888 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11881 (secure 0 0 0) 0x7f067809ccc0 con 0x7f0680106670 2024-04-23T18:35:47.859 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:35:47.859 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:35:12.295117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27059":{"gid":27059,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/3902222090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3902222090},{"type":"v1","addr":"172.21.15.177:6837","nonce":3902222090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:35:47.859 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 56 2024-04-23T18:35:47.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.869+0000 7f06849a1640 1 -- 172.21.15.50:0/2335368888 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f066c0a3d20 msgr2=0x7f066c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:47.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.869+0000 7f06849a1640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f066c0a3d20 0x7f066c0a61e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0670031440 tx=0x7f06700058d0 comp rx=0 tx=0).stop 2024-04-23T18:35:47.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.869+0000 7f06849a1640 1 -- 172.21.15.50:0/2335368888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0680106670 msgr2=0x7f068010f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:47.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.869+0000 7f06849a1640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0680106670 0x7f068010f5d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0678009fd0 tx=0x7f067800ef10 comp rx=0 tx=0).stop 2024-04-23T18:35:47.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.869+0000 7f06849a1640 1 -- 172.21.15.50:0/2335368888 shutdown_connections 2024-04-23T18:35:47.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.869+0000 7f06849a1640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f066c0a3d20 0x7f066c0a61e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:47.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.869+0000 7f06849a1640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0680106670 0x7f068010f5d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:47.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.869+0000 7f06849a1640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0680101240 0x7f068010f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:47.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.869+0000 7f06849a1640 1 --2- 172.21.15.50:0/2335368888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0680100880 0x7f068010eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:47.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.869+0000 7f06849a1640 1 -- 172.21.15.50:0/2335368888 >> 172.21.15.50:0/2335368888 conn(0x7f06800761d0 msgr2=0x7f06800ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:47.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.869+0000 7f06849a1640 1 -- 172.21.15.50:0/2335368888 shutdown_connections 2024-04-23T18:35:47.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:47.870+0000 7f06849a1640 1 -- 172.21.15.50:0/2335368888 wait complete. 2024-04-23T18:35:47.875 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 79.7 --await-for 42.0' 2024-04-23T18:35:47.875 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 79.7 --await-for 42.0 2024-04-23T18:35:48.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.011+0000 7ff95b291640 1 -- 172.21.15.50:0/3575015101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff954106170 msgr2=0x7ff95410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:48.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.011+0000 7ff95b291640 1 --2- 172.21.15.50:0/3575015101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff954106170 0x7ff95410dd40 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff94c009ca0 tx=0x7ff94c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.011+0000 7ff95b291640 1 -- 172.21.15.50:0/3575015101 shutdown_connections 2024-04-23T18:35:48.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.011+0000 7ff95b291640 1 --2- 172.21.15.50:0/3575015101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff954106170 0x7ff95410dd40 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.011+0000 7ff95b291640 1 --2- 172.21.15.50:0/3575015101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff954101a50 0x7ff954105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.011+0000 7ff95b291640 1 --2- 172.21.15.50:0/3575015101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9541010a0 0x7ff954101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.011+0000 7ff95b291640 1 -- 172.21.15.50:0/3575015101 >> 172.21.15.50:0/3575015101 conn(0x7ff954077f50 msgr2=0x7ff954078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:48.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.011+0000 7ff95b291640 1 -- 172.21.15.50:0/3575015101 shutdown_connections 2024-04-23T18:35:48.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.011+0000 7ff95b291640 1 -- 172.21.15.50:0/3575015101 wait complete. 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95b291640 1 Processor -- start 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95b291640 1 -- start start 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95b291640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9541010a0 0x7ff954102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95b291640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff954106170 0x7ff954103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95b291640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9541035b0 0x7ff9541a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95b291640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541110e0 con 0x7ff9541035b0 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95b291640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff954110f60 con 0x7ff954106170 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95b291640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff954111260 con 0x7ff9541010a0 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95982e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9541035b0 0x7ff9541a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95982e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9541035b0 0x7ff9541a6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36048/0 (socket says 172.21.15.50:36048) 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95982e640 1 -- 172.21.15.50:0/4244283014 learned_addr learned my addr 172.21.15.50:0/4244283014 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95882c640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff954106170 0x7ff954103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.012+0000 7ff95902d640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9541010a0 0x7ff954102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.013+0000 7ff95982e640 1 -- 172.21.15.50:0/4244283014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9541010a0 msgr2=0x7ff954102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.013+0000 7ff95982e640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9541010a0 0x7ff954102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.013+0000 7ff95982e640 1 -- 172.21.15.50:0/4244283014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff954106170 msgr2=0x7ff954103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.013+0000 7ff95982e640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff954106170 0x7ff954103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.013+0000 7ff95982e640 1 -- 172.21.15.50:0/4244283014 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9541a7080 con 0x7ff9541035b0 2024-04-23T18:35:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.013+0000 7ff95882c640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff954106170 0x7ff954103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.013+0000 7ff95982e640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9541035b0 0x7ff9541a6980 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff94c005cc0 tx=0x7ff94c031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.013+0000 7ff94a7fc640 1 -- 172.21.15.50:0/4244283014 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c003b70 con 0x7ff9541035b0 2024-04-23T18:35:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.013+0000 7ff95b291640 1 -- 172.21.15.50:0/4244283014 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9541a7310 con 0x7ff9541035b0 2024-04-23T18:35:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.013+0000 7ff94a7fc640 1 -- 172.21.15.50:0/4244283014 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff94c0318f0 con 0x7ff9541035b0 2024-04-23T18:35:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.013+0000 7ff94a7fc640 1 -- 172.21.15.50:0/4244283014 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c02f950 con 0x7ff9541035b0 2024-04-23T18:35:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.013+0000 7ff95b291640 1 -- 172.21.15.50:0/4244283014 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9541a78b0 con 0x7ff9541035b0 2024-04-23T18:35:48.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.016+0000 7ff94a7fc640 1 -- 172.21.15.50:0/4244283014 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff94c02fb10 con 0x7ff9541035b0 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95dfaa740 1 Processor -- start 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95dfaa740 1 -- start start 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff94a7fc640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff9400a3d50 0x7ff9400a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95882c640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff9400a3d50 0x7ff9400a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95dfaa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 0x55bf538d6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95dfaa740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 0x55bf538e0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95dfaa740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538e1090 0x55bf538e3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff94a7fc640 1 -- 172.21.15.50:0/4244283014 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7ff94c0f1970 con 0x7ff9541035b0 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95dfaa740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bf5383b370 con 0x55bf538d69c0 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95dfaa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bf537d82e0 con 0x55bf538d60a0 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95dfaa740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bf53837330 con 0x55bf538e1090 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95882c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 0x55bf538e0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95882c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 0x55bf538e0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36056/0 (socket says 172.21.15.50:36056) 2024-04-23T18:35:48.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95882c640 1 -- 172.21.15.50:0/3469555365 learned_addr learned my addr 172.21.15.50:0/3469555365 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:48.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95982e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538e1090 0x55bf538e3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.017+0000 7ff95902d640 1 --2- 172.21.15.50:0/3469555365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 0x55bf538d6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.018+0000 7ff95902d640 1 -- 172.21.15.50:0/3469555365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538e1090 msgr2=0x55bf538e3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.018+0000 7ff95902d640 1 --2- 172.21.15.50:0/3469555365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538e1090 0x55bf538e3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.018+0000 7ff95902d640 1 -- 172.21.15.50:0/3469555365 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 msgr2=0x55bf538e0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.018+0000 7ff95902d640 1 --2- 172.21.15.50:0/3469555365 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 0x55bf538e0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.018+0000 7ff95882c640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff9400a3d50 0x7ff9400a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff944013840 tx=0x7ff944013c40 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.018+0000 7ff95902d640 1 -- 172.21.15.50:0/3469555365 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf53619d80 con 0x55bf538d60a0 2024-04-23T18:35:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.018+0000 7ff95982e640 1 --2- 172.21.15.50:0/3469555365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538e1090 0x55bf538e3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.018+0000 7ff95902d640 1 --2- 172.21.15.50:0/3469555365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 0x55bf538d6480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff950009c00 tx=0x7ff950011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d4338b7bb78f6e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.018+0000 7ff93f7fe640 1 -- 172.21.15.50:0/3469555365 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950004050 con 0x55bf538d60a0 2024-04-23T18:35:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.018+0000 7ff93f7fe640 1 -- 172.21.15.50:0/3469555365 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9500054d0 con 0x55bf538d60a0 2024-04-23T18:35:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.018+0000 7ff93f7fe640 1 -- 172.21.15.50:0/3469555365 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950004ca0 con 0x55bf538d60a0 2024-04-23T18:35:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.019+0000 7ff95dfaa740 1 -- 172.21.15.50:0/3469555365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 msgr2=0x55bf538d6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.019+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/3469555365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 0x55bf538d6480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff950009c00 tx=0x7ff950011810 comp rx=0 tx=0).stop 2024-04-23T18:35:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.019+0000 7ff95dfaa740 1 -- 172.21.15.50:0/3469555365 shutdown_connections 2024-04-23T18:35:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.019+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/3469555365 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538e1090 0x55bf538e3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.019+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/3469555365 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 0x55bf538e0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.019+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/3469555365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 0x55bf538d6480 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.019+0000 7ff95dfaa740 1 -- 172.21.15.50:0/3469555365 >> 172.21.15.50:0/3469555365 conn(0x55bf538d23c0 msgr2=0x55bf538d27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.019+0000 7ff95dfaa740 1 -- 172.21.15.50:0/3469555365 shutdown_connections 2024-04-23T18:35:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.019+0000 7ff95dfaa740 1 -- 172.21.15.50:0/3469555365 wait complete. 2024-04-23T18:35:48.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95dfaa740 1 -- start start 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95dfaa740 1 Processor -- start 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95dfaa740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538d60a0 0x55bf538ddfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95dfaa740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 0x55bf538de4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95dfaa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538e1090 0x55bf538dea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95dfaa740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bf538e3e80 con 0x55bf538d69c0 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95dfaa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bf537d5fb0 con 0x55bf538e1090 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95dfaa740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bf538e4000 con 0x55bf538d60a0 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95882c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 0x55bf538de4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95dfaa740 10 client.0 _ll_register_callbacks cb 0x55bf537d59a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95902d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538d60a0 0x55bf538ddfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95882c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 0x55bf538de4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36060/0 (socket says 172.21.15.50:36060) 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95902d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538d60a0 0x55bf538ddfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:33750/0 (socket says 172.21.15.50:33750) 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.020+0000 7ff95882c640 1 -- 172.21.15.50:0/284912166 learned_addr learned my addr 172.21.15.50:0/284912166 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff95982e640 1 --2- 172.21.15.50:0/284912166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538e1090 0x55bf538dea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff95882c640 1 -- 172.21.15.50:0/284912166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538d60a0 msgr2=0x55bf538ddfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff95882c640 1 --2- 172.21.15.50:0/284912166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538d60a0 0x55bf538ddfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff95882c640 1 -- 172.21.15.50:0/284912166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538e1090 msgr2=0x55bf538dea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff95882c640 1 --2- 172.21.15.50:0/284912166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538e1090 0x55bf538dea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff95882c640 1 -- 172.21.15.50:0/284912166 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf5367a510 con 0x55bf538d69c0 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff95882c640 1 --2- 172.21.15.50:0/284912166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 0x55bf538de4e0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff94400fc70 tx=0x7ff944004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4ba6fe38544adc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff93cff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff93cff9640 1 -- 172.21.15.50:0/284912166 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff944004250 con 0x55bf538d69c0 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff93cff9640 1 -- 172.21.15.50:0/284912166 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7ff9440043f0 con 0x55bf538d69c0 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff95dfaa740 1 -- 172.21.15.50:0/284912166 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9500054d0 con 0x55bf538d69c0 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff93cff9640 1 -- 172.21.15.50:0/284912166 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7ff944005020 con 0x55bf538d69c0 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff95dfaa740 10 client.17607 fetch_fsmap learned FSMap version 56 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.021+0000 7ff95dfaa740 1 -- 172.21.15.50:0/284912166 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55bf5350d670 con 0x55bf538d69c0 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.022+0000 7ff93cff9640 1 -- 172.21.15.50:0/284912166 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9500054d0 con 0x55bf538d69c0 2024-04-23T18:35:48.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.022+0000 7ff93cff9640 1 -- 172.21.15.50:0/284912166 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 56) ==== 8354+0+0 (secure 0 0 0) 0x7ff94401f070 con 0x55bf538d69c0 2024-04-23T18:35:48.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.022+0000 7ff95dfaa740 10 client.17607 fetch_fsmap finished waiting for FSMap version 56 2024-04-23T18:35:48.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.022+0000 7ff95dfaa740 10 client.17607 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:35:48.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.022+0000 7ff95dfaa740 20 client.17607 populate_metadata read hostname 'smithi050' 2024-04-23T18:35:48.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.022+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/284912166 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf538f1350 0x55bf538f3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.022+0000 7ff95dfaa740 4 client.17607 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:35:48.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.022+0000 7ff95dfaa740 1 -- 172.21.15.50:0/284912166 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf53848d90 con 0x55bf538f1350 2024-04-23T18:35:48.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.022+0000 7ff95902d640 1 --2- 172.21.15.50:0/284912166 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf538f1350 0x55bf538f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.023+0000 7ff95902d640 1 --2- 172.21.15.50:0/284912166 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf538f1350 0x55bf538f3810 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67de478bc4dec39c server_cookie=5561048a535f8db6 in_seq=0 out_seq=0 2024-04-23T18:35:48.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.023+0000 7ff93cff9640 10 client.17607 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:48.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff93cff9640 1 -- 172.21.15.50:0/284912166 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf53848d90 con 0x55bf538f1350 2024-04-23T18:35:48.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff93cff9640 10 client.17607 handle_command_reply: tid=0 2024-04-23T18:35:48.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff95dfaa740 1 client.17607 shutdown 2024-04-23T18:35:48.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff95dfaa740 1 -- 172.21.15.50:0/284912166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 msgr2=0x55bf538de4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:48.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/284912166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 0x55bf538de4e0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff94400fc70 tx=0x7ff944004730 comp rx=0 tx=0).stop 2024-04-23T18:35:48.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff95dfaa740 1 -- 172.21.15.50:0/284912166 shutdown_connections 2024-04-23T18:35:48.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/284912166 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf538f1350 0x55bf538f3810 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/284912166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538e1090 0x55bf538dea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff93cff9640 0 client.17607 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:48.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/284912166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d69c0 0x55bf538de4e0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/284912166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538d60a0 0x55bf538ddfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff95dfaa740 1 -- 172.21.15.50:0/284912166 >> 172.21.15.50:0/284912166 conn(0x55bf538d23c0 msgr2=0x55bf538d1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:48.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff95dfaa740 1 -- 172.21.15.50:0/284912166 shutdown_connections 2024-04-23T18:35:48.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff95dfaa740 1 -- 172.21.15.50:0/284912166 wait complete. 2024-04-23T18:35:48.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.024+0000 7ff95dfaa740 20 client.17607 trim_cache size 0 max 16384 2024-04-23T18:35:48.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95dfaa740 1 Processor -- start 2024-04-23T18:35:48.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95dfaa740 1 -- start start 2024-04-23T18:35:48.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95dfaa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 0x55bf538ecd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95dfaa740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538d69c0 0x55bf538d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95dfaa740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538e1090 0x55bf538f9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95dfaa740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541b2630 con 0x55bf538e1090 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95dfaa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541afea0 con 0x55bf538d60a0 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95dfaa740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541adcf0 con 0x55bf538d69c0 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95982e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538e1090 0x55bf538f9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95902d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 0x55bf538ecd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95982e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538e1090 0x55bf538f9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36070/0 (socket says 172.21.15.50:36070) 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95982e640 1 -- 172.21.15.50:0/2645934950 learned_addr learned my addr 172.21.15.50:0/2645934950 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95902d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 0x55bf538ecd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:53400/0 (socket says 172.21.15.50:53400) 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.044+0000 7ff95882c640 1 --2- 172.21.15.50:0/2645934950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538d69c0 0x55bf538d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.045+0000 7ff95902d640 1 -- 172.21.15.50:0/2645934950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538d69c0 msgr2=0x55bf538d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.045+0000 7ff95902d640 1 --2- 172.21.15.50:0/2645934950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538d69c0 0x55bf538d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.045+0000 7ff95902d640 1 -- 172.21.15.50:0/2645934950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538e1090 msgr2=0x55bf538f9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:48.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.045+0000 7ff95902d640 1 --2- 172.21.15.50:0/2645934950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538e1090 0x55bf538f9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.045+0000 7ff95902d640 1 -- 172.21.15.50:0/2645934950 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf538dc4f0 con 0x55bf538d60a0 2024-04-23T18:35:48.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.045+0000 7ff95982e640 1 --2- 172.21.15.50:0/2645934950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538e1090 0x55bf538f9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:48.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.045+0000 7ff95902d640 1 --2- 172.21.15.50:0/2645934950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 0x55bf538ecd40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff950012fd0 tx=0x7ff9500048e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aa90289f128195c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:48.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.045+0000 7ff93effd640 1 -- 172.21.15.50:0/2645934950 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95002edd0 con 0x55bf538d60a0 2024-04-23T18:35:48.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.045+0000 7ff93effd640 1 -- 172.21.15.50:0/2645934950 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff95002e780 con 0x55bf538d60a0 2024-04-23T18:35:48.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.045+0000 7ff93effd640 1 -- 172.21.15.50:0/2645934950 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950004710 con 0x55bf538d60a0 2024-04-23T18:35:48.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.046+0000 7ff95dfaa740 1 -- 172.21.15.50:0/2645934950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 msgr2=0x55bf538ecd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.046+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/2645934950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 0x55bf538ecd40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff950012fd0 tx=0x7ff9500048e0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.046+0000 7ff95dfaa740 1 -- 172.21.15.50:0/2645934950 shutdown_connections 2024-04-23T18:35:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.046+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/2645934950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538e1090 0x55bf538f9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.046+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/2645934950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538d69c0 0x55bf538d1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.046+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/2645934950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d60a0 0x55bf538ecd40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.046+0000 7ff95dfaa740 1 -- 172.21.15.50:0/2645934950 >> 172.21.15.50:0/2645934950 conn(0x55bf538d23c0 msgr2=0x55bf538de7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.046+0000 7ff95dfaa740 1 -- 172.21.15.50:0/2645934950 shutdown_connections 2024-04-23T18:35:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.046+0000 7ff95dfaa740 1 -- 172.21.15.50:0/2645934950 wait complete. 2024-04-23T18:35:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.046+0000 7ff95dfaa740 1 -- start start 2024-04-23T18:35:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.046+0000 7ff95dfaa740 1 Processor -- start 2024-04-23T18:35:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95dfaa740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d60a0 0x55bf538e9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95dfaa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d69c0 0x55bf538e9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95dfaa740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538e1090 0x55bf538e6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95dfaa740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541ae360 con 0x55bf538d60a0 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95dfaa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541af7d0 con 0x55bf538d69c0 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95dfaa740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541aea30 con 0x55bf538e1090 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95dfaa740 10 client.0 _ll_register_callbacks cb 0x7ff9440134f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95902d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d60a0 0x55bf538e9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95882c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d69c0 0x55bf538e9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95902d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d60a0 0x55bf538e9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36076/0 (socket says 172.21.15.50:36076) 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95982e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538e1090 0x55bf538e6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95902d640 1 -- 172.21.15.50:0/1768183549 learned_addr learned my addr 172.21.15.50:0/1768183549 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95882c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d69c0 0x55bf538e9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:53402/0 (socket says 172.21.15.50:53402) 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95902d640 1 -- 172.21.15.50:0/1768183549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538e1090 msgr2=0x55bf538e6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95902d640 1 --2- 172.21.15.50:0/1768183549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538e1090 0x55bf538e6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95902d640 1 -- 172.21.15.50:0/1768183549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d69c0 msgr2=0x55bf538e9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:48.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95902d640 1 --2- 172.21.15.50:0/1768183549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d69c0 0x55bf538e9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95902d640 1 -- 172.21.15.50:0/1768183549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf538e6910 con 0x55bf538d60a0 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95882c640 1 --2- 172.21.15.50:0/1768183549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d69c0 0x55bf538e9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff95902d640 1 --2- 172.21.15.50:0/1768183549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d60a0 0x55bf538e9250 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff950009350 tx=0x7ff950009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=904e77515dbf91a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff93cff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.047+0000 7ff93cff9640 1 -- 172.21.15.50:0/1768183549 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95003b020 con 0x55bf538d60a0 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff95dfaa740 1 -- 172.21.15.50:0/1768183549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff95002e780 con 0x55bf538d60a0 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff93cff9640 1 -- 172.21.15.50:0/1768183549 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7ff95002edd0 con 0x55bf538d60a0 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff93cff9640 1 -- 172.21.15.50:0/1768183549 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7ff950046050 con 0x55bf538d60a0 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff95dfaa740 10 client.17613 fetch_fsmap learned FSMap version 56 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff95dfaa740 1 -- 172.21.15.50:0/1768183549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x55bf539583b0 con 0x55bf538d60a0 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff93cff9640 1 -- 172.21.15.50:0/1768183549 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff95002e780 con 0x55bf538d60a0 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff93cff9640 1 -- 172.21.15.50:0/1768183549 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 56) ==== 8354+0+0 (secure 0 0 0) 0x7ff9500035f0 con 0x55bf538d60a0 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff95dfaa740 10 client.17613 fetch_fsmap finished waiting for FSMap version 56 2024-04-23T18:35:48.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff95dfaa740 10 client.17613 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:35:48.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff95dfaa740 20 client.17613 populate_metadata read hostname 'smithi050' 2024-04-23T18:35:48.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/1768183549 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf539589f0 0x55bf5395ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:48.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff95dfaa740 4 client.17613 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.7, "await_for": 42.0}] 2024-04-23T18:35:48.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff95dfaa740 1 -- 172.21.15.50:0/1768183549 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.7, "await_for": 42.0}) -- 0x55bf53842c00 con 0x55bf539589f0 2024-04-23T18:35:48.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.048+0000 7ff95882c640 1 --2- 172.21.15.50:0/1768183549 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf539589f0 0x55bf5395ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:48.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.049+0000 7ff93cff9640 10 client.17613 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:48.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.049+0000 7ff95882c640 1 --2- 172.21.15.50:0/1768183549 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf539589f0 0x55bf5395ade0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63990214164c8430 server_cookie=13e5f2ce6bb83141 in_seq=0 out_seq=0 2024-04-23T18:35:48.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121886 26.43 MB/sec execute 142 sec latency 475.097 ms 2024-04-23T18:35:48.587 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:48 smithi177 ceph-mon[32308]: pgmap v548: 97 pgs: 97 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 10 MiB/s wr, 690 op/s 2024-04-23T18:35:48.587 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:48 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2335368888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:35:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff93cff9640 1 -- 172.21.15.50:0/1768183549 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55bf53842c00 con 0x55bf539589f0 2024-04-23T18:35:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff93cff9640 10 client.17613 handle_command_reply: tid=0 2024-04-23T18:35:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff95dfaa740 1 client.17613 shutdown 2024-04-23T18:35:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff95dfaa740 1 -- 172.21.15.50:0/1768183549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d60a0 msgr2=0x55bf538e9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/1768183549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d60a0 0x55bf538e9250 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff950009350 tx=0x7ff950009eb0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff95dfaa740 1 -- 172.21.15.50:0/1768183549 shutdown_connections 2024-04-23T18:35:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/1768183549 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bf539589f0 0x55bf5395ade0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/1768183549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bf538e1090 0x55bf538e6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/1768183549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf538d69c0 0x55bf538e9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff93cff9640 0 client.17613 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff95dfaa740 1 --2- 172.21.15.50:0/1768183549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bf538d60a0 0x55bf538e9250 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff95dfaa740 1 -- 172.21.15.50:0/1768183549 >> 172.21.15.50:0/1768183549 conn(0x55bf538d23c0 msgr2=0x55bf538d77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff95dfaa740 1 -- 172.21.15.50:0/1768183549 shutdown_connections 2024-04-23T18:35:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff95dfaa740 1 -- 172.21.15.50:0/1768183549 wait complete. 2024-04-23T18:35:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.625+0000 7ff95dfaa740 20 client.17613 trim_cache size 0 max 16384 2024-04-23T18:35:48.619 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 56, 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 58, 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "7fa60ad4": { 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 58, 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 79.7, 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:35:48.620 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:35:48.621 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:35:48.621 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:35:48.621 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:35:48.621 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:35:48.621 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:35:48.621 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:35:48.621 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:35:48.621 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:35:48.621 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:35:48.621 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:35:48.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.626+0000 7ff95b291640 1 -- 172.21.15.50:0/4244283014 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff9400a3d50 msgr2=0x7ff9400a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:48.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.626+0000 7ff95b291640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff9400a3d50 0x7ff9400a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff944013840 tx=0x7ff944013c40 comp rx=0 tx=0).stop 2024-04-23T18:35:48.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.626+0000 7ff95b291640 1 -- 172.21.15.50:0/4244283014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9541035b0 msgr2=0x7ff9541a6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:48.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.626+0000 7ff95b291640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9541035b0 0x7ff9541a6980 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff94c005cc0 tx=0x7ff94c031700 comp rx=0 tx=0).stop 2024-04-23T18:35:48.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.626+0000 7ff95b291640 1 -- 172.21.15.50:0/4244283014 shutdown_connections 2024-04-23T18:35:48.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.626+0000 7ff95b291640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff9400a3d50 0x7ff9400a6210 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.626+0000 7ff95b291640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9541035b0 0x7ff9541a6980 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.626+0000 7ff95b291640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff954106170 0x7ff954103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.626+0000 7ff95b291640 1 --2- 172.21.15.50:0/4244283014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9541010a0 0x7ff954102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.626+0000 7ff95b291640 1 -- 172.21.15.50:0/4244283014 >> 172.21.15.50:0/4244283014 conn(0x7ff954077f50 msgr2=0x7ff954104db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.626+0000 7ff95b291640 1 -- 172.21.15.50:0/4244283014 shutdown_connections 2024-04-23T18:35:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:35:48.626+0000 7ff95b291640 1 -- 172.21.15.50:0/4244283014 wait complete. 2024-04-23T18:35:48.630 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:48 smithi188 ceph-mon[32694]: pgmap v548: 97 pgs: 97 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 10 MiB/s wr, 690 op/s 2024-04-23T18:35:48.630 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:48 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2335368888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:35:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:48 smithi050 ceph-mon[29187]: pgmap v548: 97 pgs: 97 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 10 MiB/s wr, 690 op/s 2024-04-23T18:35:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:48 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2335368888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:35:48.634 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '7fa60ad4', quiesce took 1.2 seconds. Will release after: 19.7 2024-04-23T18:35:49.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 26.24 MB/sec execute 143 sec latency 1475.233 ms 2024-04-23T18:35:50.570 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 26.06 MB/sec execute 144 sec latency 2475.353 ms 2024-04-23T18:35:50.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:50 smithi050 ceph-mon[29187]: pgmap v549: 97 pgs: 97 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 7.3 MiB/s wr, 506 op/s 2024-04-23T18:35:50.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:50 smithi177 ceph-mon[32308]: pgmap v549: 97 pgs: 97 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 7.3 MiB/s wr, 506 op/s 2024-04-23T18:35:50.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:50 smithi188 ceph-mon[32694]: pgmap v549: 97 pgs: 97 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 7.3 MiB/s wr, 506 op/s 2024-04-23T18:35:51.571 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 25.88 MB/sec execute 145 sec latency 3475.406 ms 2024-04-23T18:35:52.571 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 25.70 MB/sec execute 146 sec latency 4475.521 ms 2024-04-23T18:35:52.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:52 smithi050 ceph-mon[29187]: pgmap v550: 97 pgs: 97 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 7.3 MiB/s wr, 506 op/s 2024-04-23T18:35:52.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:52 smithi177 ceph-mon[32308]: pgmap v550: 97 pgs: 97 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 7.3 MiB/s wr, 506 op/s 2024-04-23T18:35:52.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:52 smithi188 ceph-mon[32694]: pgmap v550: 97 pgs: 97 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 7.3 MiB/s wr, 506 op/s 2024-04-23T18:35:53.571 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 25.53 MB/sec execute 147 sec latency 5475.661 ms 2024-04-23T18:35:54.571 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 25.36 MB/sec execute 148 sec latency 6475.759 ms 2024-04-23T18:35:54.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:54 smithi050 ceph-mon[29187]: pgmap v551: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 659 op/s 2024-04-23T18:35:54.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:54 smithi177 ceph-mon[32308]: pgmap v551: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 659 op/s 2024-04-23T18:35:54.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:54 smithi188 ceph-mon[32694]: pgmap v551: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 659 op/s 2024-04-23T18:35:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.531+0000 7fca28315640 1 -- 172.21.15.50:0/875784563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca20106170 msgr2=0x7fca2010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.531+0000 7fca28315640 1 --2- 172.21.15.50:0/875784563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca20106170 0x7fca2010dd40 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fca10009ca0 tx=0x7fca1002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.531+0000 7fca28315640 1 -- 172.21.15.50:0/875784563 shutdown_connections 2024-04-23T18:35:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.531+0000 7fca28315640 1 --2- 172.21.15.50:0/875784563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca20106170 0x7fca2010dd40 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.531+0000 7fca28315640 1 --2- 172.21.15.50:0/875784563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca20101aa0 0x7fca20105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.531+0000 7fca28315640 1 --2- 172.21.15.50:0/875784563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca201010f0 0x7fca201014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.531+0000 7fca28315640 1 -- 172.21.15.50:0/875784563 >> 172.21.15.50:0/875784563 conn(0x7fca20077fd0 msgr2=0x7fca200783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.531+0000 7fca28315640 1 -- 172.21.15.50:0/875784563 shutdown_connections 2024-04-23T18:35:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.531+0000 7fca28315640 1 -- 172.21.15.50:0/875784563 wait complete. 2024-04-23T18:35:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca28315640 1 Processor -- start 2024-04-23T18:35:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca28315640 1 -- start start 2024-04-23T18:35:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca28315640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca201010f0 0x7fca201986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca28315640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca20106170 0x7fca20198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca28315640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca2019cf20 0x7fca2019d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca28315640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fca20111130 con 0x7fca2019cf20 2024-04-23T18:35:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca28315640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca20110fb0 con 0x7fca201010f0 2024-04-23T18:35:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca28315640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fca201112b0 con 0x7fca20106170 2024-04-23T18:35:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca268b2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca2019cf20 0x7fca2019d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca258b0640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca20106170 0x7fca20198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca268b2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca2019cf20 0x7fca2019d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39748/0 (socket says 172.21.15.50:39748) 2024-04-23T18:35:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca268b2640 1 -- 172.21.15.50:0/163600003 learned_addr learned my addr 172.21.15.50:0/163600003 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca260b1640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca201010f0 0x7fca201986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca268b2640 1 -- 172.21.15.50:0/163600003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca20106170 msgr2=0x7fca20198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca268b2640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca20106170 0x7fca20198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca268b2640 1 -- 172.21.15.50:0/163600003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca201010f0 msgr2=0x7fca201986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:55.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca268b2640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca201010f0 0x7fca201986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.532+0000 7fca268b2640 1 -- 172.21.15.50:0/163600003 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2019da10 con 0x7fca2019cf20 2024-04-23T18:35:55.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.533+0000 7fca268b2640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca2019cf20 0x7fca2019d3d0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fca10009430 tx=0x7fca100317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:55.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.533+0000 7fca1f7fe640 1 -- 172.21.15.50:0/163600003 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10003b70 con 0x7fca2019cf20 2024-04-23T18:35:55.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.533+0000 7fca1f7fe640 1 -- 172.21.15.50:0/163600003 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca10003d10 con 0x7fca2019cf20 2024-04-23T18:35:55.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.533+0000 7fca28315640 1 -- 172.21.15.50:0/163600003 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca2019dca0 con 0x7fca2019cf20 2024-04-23T18:35:55.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.533+0000 7fca1f7fe640 1 -- 172.21.15.50:0/163600003 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1002fa30 con 0x7fca2019cf20 2024-04-23T18:35:55.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.533+0000 7fca28315640 1 -- 172.21.15.50:0/163600003 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca201a54c0 con 0x7fca2019cf20 2024-04-23T18:35:55.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.536+0000 7fca2b020740 1 Processor -- start 2024-04-23T18:35:55.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.536+0000 7fca1f7fe640 1 -- 172.21.15.50:0/163600003 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fca1002fbd0 con 0x7fca2019cf20 2024-04-23T18:35:55.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.536+0000 7fca2b020740 1 -- start start 2024-04-23T18:35:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca2b020740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50c0500 0x5595e50c08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca2b020740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0e20 0x5595e50cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca2b020740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50cfc10 0x5595e50d2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca2b020740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5595e5025970 con 0x5595e50c0e20 2024-04-23T18:35:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca2b020740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5595e4fc24a0 con 0x5595e50c0500 2024-04-23T18:35:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca2b020740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5595e5021930 con 0x5595e50cfc10 2024-04-23T18:35:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca260b1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0e20 0x5595e50cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca260b1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0e20 0x5595e50cf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39754/0 (socket says 172.21.15.50:39754) 2024-04-23T18:35:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca260b1640 1 -- 172.21.15.50:0/3725229679 learned_addr learned my addr 172.21.15.50:0/3725229679 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca258b0640 1 --2- 172.21.15.50:0/3725229679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50cfc10 0x5595e50d2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca258b0640 1 --2- 172.21.15.50:0/3725229679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50c0500 0x5595e50c08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca1f7fe640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca080a3dc0 0x7fca080a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca260b1640 1 -- 172.21.15.50:0/3725229679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50cfc10 msgr2=0x5595e50d2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca260b1640 1 --2- 172.21.15.50:0/3725229679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50cfc10 0x5595e50d2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca260b1640 1 -- 172.21.15.50:0/3725229679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50c0500 msgr2=0x5595e50c08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca260b1640 1 --2- 172.21.15.50:0/3725229679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50c0500 0x5595e50c08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca260b1640 1 -- 172.21.15.50:0/3725229679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595e4d38210 con 0x5595e50c0e20 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca258b0640 1 --2- 172.21.15.50:0/3725229679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50c0500 0x5595e50c08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca268b2640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca080a3dc0 0x7fca080a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca1f7fe640 1 -- 172.21.15.50:0/163600003 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7fca100f1b60 con 0x7fca2019cf20 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.537+0000 7fca260b1640 1 --2- 172.21.15.50:0/3725229679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0e20 0x5595e50cf6d0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fca1800bd40 tx=0x7fca180107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caadce4306cd48c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.538+0000 7fca1cbfe640 1 -- 172.21.15.50:0/3725229679 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca18004230 con 0x5595e50c0e20 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.538+0000 7fca268b2640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca080a3dc0 0x7fca080a6280 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fca100bdbb0 tx=0x7fca1002a8c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.538+0000 7fca1cbfe640 1 -- 172.21.15.50:0/3725229679 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca1800be30 con 0x5595e50c0e20 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.538+0000 7fca1cbfe640 1 -- 172.21.15.50:0/3725229679 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1801a850 con 0x5595e50c0e20 2024-04-23T18:35:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.538+0000 7fca2b020740 1 -- 172.21.15.50:0/3725229679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0e20 msgr2=0x5595e50cf6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.538+0000 7fca2b020740 1 --2- 172.21.15.50:0/3725229679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0e20 0x5595e50cf6d0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fca1800bd40 tx=0x7fca180107a0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 -- 172.21.15.50:0/3725229679 shutdown_connections 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 --2- 172.21.15.50:0/3725229679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50cfc10 0x5595e50d2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 --2- 172.21.15.50:0/3725229679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0e20 0x5595e50cf6d0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 --2- 172.21.15.50:0/3725229679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50c0500 0x5595e50c08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 -- 172.21.15.50:0/3725229679 >> 172.21.15.50:0/3725229679 conn(0x5595e50bcb30 msgr2=0x5595e50bcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 -- 172.21.15.50:0/3725229679 shutdown_connections 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 -- 172.21.15.50:0/3725229679 wait complete. 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 -- start start 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 Processor -- start 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50cf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 0x5595e50cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50cfc10 0x5595e50ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5595e50d2840 con 0x5595e50c0500 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5595e4fc00b0 con 0x5595e50cfc10 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5595e50d29c0 con 0x5595e50c0e20 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca258b0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca2b020740 10 client.0 _ll_register_callbacks cb 0x5595e4fadd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca258b0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50cf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39768/0 (socket says 172.21.15.50:39768) 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca258b0640 1 -- 172.21.15.50:0/3927476833 learned_addr learned my addr 172.21.15.50:0/3927476833 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.539+0000 7fca260b1640 1 --2- 172.21.15.50:0/3927476833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 0x5595e50cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca258b0640 1 -- 172.21.15.50:0/3927476833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 msgr2=0x5595e50cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca258b0640 1 --2- 172.21.15.50:0/3927476833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 0x5595e50cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca258b0640 1 -- 172.21.15.50:0/3927476833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50cfc10 msgr2=0x5595e50ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca258b0640 1 --2- 172.21.15.50:0/3927476833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50cfc10 0x5595e50ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca258b0640 1 -- 172.21.15.50:0/3927476833 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595e4e04620 con 0x5595e50c0500 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca258b0640 1 --2- 172.21.15.50:0/3927476833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50cf540 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fca14004620 tx=0x7fca14010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a383728f87ba20d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca05ffb640 1 -- 172.21.15.50:0/3927476833 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1400faa0 con 0x5595e50c0500 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca05ffb640 1 -- 172.21.15.50:0/3927476833 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7fca1400fc40 con 0x5595e50c0500 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca2b020740 1 -- 172.21.15.50:0/3927476833 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca1800be30 con 0x5595e50c0500 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca05ffb640 1 -- 172.21.15.50:0/3927476833 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fca1400fe40 con 0x5595e50c0500 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca2b020740 10 client.17625 fetch_fsmap learned FSMap version 56 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca2b020740 1 -- 172.21.15.50:0/3927476833 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x5595e50de860 con 0x5595e50c0500 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.540+0000 7fca05ffb640 1 -- 172.21.15.50:0/3927476833 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca1800be30 con 0x5595e50c0500 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.541+0000 7fca05ffb640 1 -- 172.21.15.50:0/3927476833 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 56) ==== 8354+0+0 (secure 0 0 0) 0x7fca140255c0 con 0x5595e50c0500 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.541+0000 7fca2b020740 10 client.17625 fetch_fsmap finished waiting for FSMap version 56 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.541+0000 7fca2b020740 10 client.17625 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:35:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.541+0000 7fca2b020740 20 client.17625 populate_metadata read hostname 'smithi050' 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.541+0000 7fca2b020740 1 --2- 172.21.15.50:0/3927476833 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595e50dfd90 0x5595e50e21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.541+0000 7fca2b020740 4 client.17625 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.541+0000 7fca2b020740 1 -- 172.21.15.50:0/3927476833 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595e5033390 con 0x5595e50dfd90 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.541+0000 7fca260b1640 1 --2- 172.21.15.50:0/3927476833 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595e50dfd90 0x5595e50e21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.541+0000 7fca05ffb640 10 client.17625 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.541+0000 7fca260b1640 1 --2- 172.21.15.50:0/3927476833 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595e50dfd90 0x5595e50e21e0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4b2a4c7a5f9854d server_cookie=1e0fe93154ee1d4c in_seq=0 out_seq=0 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca05ffb640 1 -- 172.21.15.50:0/3927476833 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5595e5033390 con 0x5595e50dfd90 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca05ffb640 10 client.17625 handle_command_reply: tid=0 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca2b020740 1 client.17625 shutdown 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca2b020740 1 -- 172.21.15.50:0/3927476833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 msgr2=0x5595e50cf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca2b020740 1 --2- 172.21.15.50:0/3927476833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50cf540 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fca14004620 tx=0x7fca14010d00 comp rx=0 tx=0).stop 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca2b020740 1 -- 172.21.15.50:0/3927476833 shutdown_connections 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca2b020740 1 --2- 172.21.15.50:0/3927476833 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595e50dfd90 0x5595e50e21e0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca2b020740 1 --2- 172.21.15.50:0/3927476833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50cfc10 0x5595e50ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca2b020740 1 --2- 172.21.15.50:0/3927476833 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 0x5595e50cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca05ffb640 0 client.17625 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca2b020740 1 --2- 172.21.15.50:0/3927476833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50cf540 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca2b020740 1 -- 172.21.15.50:0/3927476833 >> 172.21.15.50:0/3927476833 conn(0x5595e50bcb30 msgr2=0x5595e50c2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.542+0000 7fca2b020740 1 -- 172.21.15.50:0/3927476833 shutdown_connections 2024-04-23T18:35:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.543+0000 7fca2b020740 1 -- 172.21.15.50:0/3927476833 wait complete. 2024-04-23T18:35:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.543+0000 7fca2b020740 20 client.17625 trim_cache size 0 max 16384 2024-04-23T18:35:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca2b020740 1 Processor -- start 2024-04-23T18:35:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca2b020740 1 -- start start 2024-04-23T18:35:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca2b020740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50bcb30 0x5595e50db770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca2b020740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca2b020740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 0x5595e50cf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca2b020740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fca201ae290 con 0x5595e50c0500 2024-04-23T18:35:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca2b020740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca201abb00 con 0x5595e50bcb30 2024-04-23T18:35:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca2b020740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fca201a9970 con 0x5595e50c0e20 2024-04-23T18:35:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca260b1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca260b1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50c7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39772/0 (socket says 172.21.15.50:39772) 2024-04-23T18:35:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca260b1640 1 -- 172.21.15.50:0/3271168335 learned_addr learned my addr 172.21.15.50:0/3271168335 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca258b0640 1 --2- 172.21.15.50:0/3271168335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50bcb30 0x5595e50db770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca258b0640 1 --2- 172.21.15.50:0/3271168335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 0x5595e50cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca260b1640 1 -- 172.21.15.50:0/3271168335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 msgr2=0x5595e50cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca260b1640 1 --2- 172.21.15.50:0/3271168335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 0x5595e50cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca260b1640 1 -- 172.21.15.50:0/3271168335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50bcb30 msgr2=0x5595e50db770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca260b1640 1 --2- 172.21.15.50:0/3271168335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50bcb30 0x5595e50db770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca260b1640 1 -- 172.21.15.50:0/3271168335 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595e50c1870 con 0x5595e50c0500 2024-04-23T18:35:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.558+0000 7fca258b0640 1 --2- 172.21.15.50:0/3271168335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50bcb30 0x5595e50db770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:35:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca260b1640 1 --2- 172.21.15.50:0/3271168335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50c7d80 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fca18000c00 tx=0x7fca1801a470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808999d1917efa73 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca07fff640 1 -- 172.21.15.50:0/3271168335 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca18030070 con 0x5595e50c0500 2024-04-23T18:35:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca07fff640 1 -- 172.21.15.50:0/3271168335 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca18005040 con 0x5595e50c0500 2024-04-23T18:35:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca07fff640 1 -- 172.21.15.50:0/3271168335 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1802a410 con 0x5595e50c0500 2024-04-23T18:35:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca2b020740 1 -- 172.21.15.50:0/3271168335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 msgr2=0x5595e50c7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca2b020740 1 --2- 172.21.15.50:0/3271168335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50c7d80 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fca18000c00 tx=0x7fca1801a470 comp rx=0 tx=0).stop 2024-04-23T18:35:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca2b020740 1 -- 172.21.15.50:0/3271168335 shutdown_connections 2024-04-23T18:35:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca2b020740 1 --2- 172.21.15.50:0/3271168335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 0x5595e50cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca2b020740 1 --2- 172.21.15.50:0/3271168335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50c7d80 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca2b020740 1 --2- 172.21.15.50:0/3271168335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50bcb30 0x5595e50db770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca2b020740 1 -- 172.21.15.50:0/3271168335 >> 172.21.15.50:0/3271168335 conn(0x5595e50cced0 msgr2=0x5595e50c7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca2b020740 1 -- 172.21.15.50:0/3271168335 shutdown_connections 2024-04-23T18:35:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.559+0000 7fca2b020740 1 -- 172.21.15.50:0/3271168335 wait complete. 2024-04-23T18:35:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca2b020740 1 -- start start 2024-04-23T18:35:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca2b020740 1 Processor -- start 2024-04-23T18:35:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca2b020740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50bcb30 0x5595e50d7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca2b020740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50d8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca2b020740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 0x5595e50d3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca2b020740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fca201a9fc0 con 0x5595e50c0500 2024-04-23T18:35:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca2b020740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca201ab430 con 0x5595e50bcb30 2024-04-23T18:35:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca2b020740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fca201aa690 con 0x5595e50c0e20 2024-04-23T18:35:55.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca2b020740 10 client.0 _ll_register_callbacks cb 0x7fca14010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:35:55.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca260b1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50d8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca260b1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50d8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39784/0 (socket says 172.21.15.50:39784) 2024-04-23T18:35:55.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca260b1640 1 -- 172.21.15.50:0/3198406256 learned_addr learned my addr 172.21.15.50:0/3198406256 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:35:55.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca258b0640 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50bcb30 0x5595e50d7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca258b0640 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 0x5595e50d3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca260b1640 1 -- 172.21.15.50:0/3198406256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 msgr2=0x5595e50d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:55.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca260b1640 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 0x5595e50d3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca260b1640 1 -- 172.21.15.50:0/3198406256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50bcb30 msgr2=0x5595e50d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca260b1640 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50bcb30 0x5595e50d7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca260b1640 1 -- 172.21.15.50:0/3198406256 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595e50d3aa0 con 0x5595e50c0500 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca260b1640 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50d8180 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fca18010ca0 tx=0x7fca180107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba28c2e2abbc2b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.560+0000 7fca05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca05ffb640 1 -- 172.21.15.50:0/3198406256 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca18030050 con 0x5595e50c0500 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca05ffb640 1 -- 172.21.15.50:0/3198406256 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8498+0+0 (secure 0 0 0) 0x7fca180036f0 con 0x5595e50c0500 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca2b020740 1 -- 172.21.15.50:0/3198406256 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca18005040 con 0x5595e50c0500 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca05ffb640 1 -- 172.21.15.50:0/3198406256 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7fca18041050 con 0x5595e50c0500 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca2b020740 10 client.17631 fetch_fsmap learned FSMap version 56 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca2b020740 1 -- 172.21.15.50:0/3198406256 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x5595e5146000 con 0x5595e50c0500 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca05ffb640 1 -- 172.21.15.50:0/3198406256 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca18005040 con 0x5595e50c0500 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca05ffb640 1 -- 172.21.15.50:0/3198406256 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 56) ==== 8354+0+0 (secure 0 0 0) 0x7fca1802a460 con 0x5595e50c0500 2024-04-23T18:35:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca2b020740 10 client.17631 fetch_fsmap finished waiting for FSMap version 56 2024-04-23T18:35:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca2b020740 10 client.17631 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:35:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca2b020740 20 client.17631 populate_metadata read hostname 'smithi050' 2024-04-23T18:35:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca2b020740 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595e50cfc10 0x5595e5148640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:35:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca2b020740 4 client.17631 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:35:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca2b020740 1 -- 172.21.15.50:0/3198406256 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5595e502d200 con 0x5595e50cfc10 2024-04-23T18:35:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.561+0000 7fca258b0640 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595e50cfc10 0x5595e5148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:35:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.562+0000 7fca258b0640 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595e50cfc10 0x5595e5148640 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4d902801305a3d1 server_cookie=de5d4a7084128a99 in_seq=0 out_seq=0 2024-04-23T18:35:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.562+0000 7fca05ffb640 10 client.17631 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:55.571 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 25.19 MB/sec execute 149 sec latency 7475.822 ms 2024-04-23T18:35:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.709+0000 7fca05ffb640 1 -- 172.21.15.50:0/3198406256 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5595e502d200 con 0x5595e50cfc10 2024-04-23T18:35:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca05ffb640 10 client.17631 handle_command_reply: tid=0 2024-04-23T18:35:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca2b020740 1 client.17631 shutdown 2024-04-23T18:35:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca2b020740 1 -- 172.21.15.50:0/3198406256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 msgr2=0x5595e50d8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca2b020740 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50d8180 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fca18010ca0 tx=0x7fca180107a0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca2b020740 1 -- 172.21.15.50:0/3198406256 shutdown_connections 2024-04-23T18:35:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca2b020740 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5595e50cfc10 0x5595e5148640 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca2b020740 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5595e50c0e20 0x5595e50d3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca2b020740 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5595e50c0500 0x5595e50d8180 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca05ffb640 0 client.17631 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:35:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca2b020740 1 --2- 172.21.15.50:0/3198406256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595e50bcb30 0x5595e50d7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca2b020740 1 -- 172.21.15.50:0/3198406256 >> 172.21.15.50:0/3198406256 conn(0x5595e50cced0 msgr2=0x5595e50bb420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca2b020740 1 -- 172.21.15.50:0/3198406256 shutdown_connections 2024-04-23T18:35:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca2b020740 1 -- 172.21.15.50:0/3198406256 wait complete. 2024-04-23T18:35:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.710+0000 7fca2b020740 20 client.17631 trim_cache size 0 max 16384 2024-04-23T18:35:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:35:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:35:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:35:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:35:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.711+0000 7fca28315640 1 -- 172.21.15.50:0/163600003 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca080a3dc0 msgr2=0x7fca080a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.711+0000 7fca28315640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca080a3dc0 0x7fca080a6280 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fca100bdbb0 tx=0x7fca1002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.711+0000 7fca28315640 1 -- 172.21.15.50:0/163600003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca2019cf20 msgr2=0x7fca2019d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:35:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.711+0000 7fca28315640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca2019cf20 0x7fca2019d3d0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fca10009430 tx=0x7fca100317c0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.711+0000 7fca28315640 1 -- 172.21.15.50:0/163600003 shutdown_connections 2024-04-23T18:35:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.711+0000 7fca28315640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fca080a3dc0 0x7fca080a6280 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.711+0000 7fca28315640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca2019cf20 0x7fca2019d3d0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.711+0000 7fca28315640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca20106170 0x7fca20198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.711+0000 7fca28315640 1 --2- 172.21.15.50:0/163600003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca201010f0 0x7fca201986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:35:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.711+0000 7fca28315640 1 -- 172.21.15.50:0/163600003 >> 172.21.15.50:0/163600003 conn(0x7fca20077fd0 msgr2=0x7fca20069880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:35:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.711+0000 7fca28315640 1 -- 172.21.15.50:0/163600003 shutdown_connections 2024-04-23T18:35:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:35:55.711+0000 7fca28315640 1 -- 172.21.15.50:0/163600003 wait complete. 2024-04-23T18:35:56.571 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 25.02 MB/sec execute 150 sec latency 8475.879 ms 2024-04-23T18:35:56.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:56 smithi050 ceph-mon[29187]: pgmap v552: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.3 MiB/s wr, 491 op/s 2024-04-23T18:35:56.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:56 smithi177 ceph-mon[32308]: pgmap v552: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.3 MiB/s wr, 491 op/s 2024-04-23T18:35:56.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:56 smithi188 ceph-mon[32694]: pgmap v552: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.3 MiB/s wr, 491 op/s 2024-04-23T18:35:57.571 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 24.85 MB/sec execute 151 sec latency 9476.033 ms 2024-04-23T18:35:58.571 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 24.69 MB/sec execute 152 sec latency 10476.085 ms 2024-04-23T18:35:58.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:58 smithi050 ceph-mon[29187]: pgmap v553: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 9.6 MiB/s wr, 602 op/s 2024-04-23T18:35:58.704 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:58 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:35:58.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:58 smithi177 ceph-mon[32308]: pgmap v553: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 9.6 MiB/s wr, 602 op/s 2024-04-23T18:35:58.796 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:58 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:35:58.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:58 smithi188 ceph-mon[32694]: pgmap v553: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 9.6 MiB/s wr, 602 op/s 2024-04-23T18:35:58.837 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:58 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:35:59.571 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 24.53 MB/sec execute 153 sec latency 11476.207 ms 2024-04-23T18:35:59.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:59 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:35:59.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:59 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:35:59.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:35:59 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:35:59.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:59 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:35:59.795 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:59 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:35:59.803 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:35:59 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:35:59.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:59 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:35:59.837 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:59 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:35:59.847 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:35:59 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:36:00.571 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 24.37 MB/sec execute 154 sec latency 12476.263 ms 2024-04-23T18:36:00.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:00 smithi050 ceph-mon[29187]: pgmap v554: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 264 op/s 2024-04-23T18:36:00.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:00 smithi177 ceph-mon[32308]: pgmap v554: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 264 op/s 2024-04-23T18:36:00.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:00 smithi188 ceph-mon[32694]: pgmap v554: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 264 op/s 2024-04-23T18:36:01.571 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 24.21 MB/sec execute 155 sec latency 13476.306 ms 2024-04-23T18:36:02.572 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 24.06 MB/sec execute 156 sec latency 14476.359 ms 2024-04-23T18:36:02.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:02 smithi050 ceph-mon[29187]: pgmap v555: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 264 op/s 2024-04-23T18:36:02.790 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:02 smithi177 ceph-mon[32308]: pgmap v555: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 264 op/s 2024-04-23T18:36:02.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:02 smithi188 ceph-mon[32694]: pgmap v555: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 264 op/s 2024-04-23T18:36:03.572 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 23.90 MB/sec execute 157 sec latency 15476.499 ms 2024-04-23T18:36:04.572 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 23.75 MB/sec execute 158 sec latency 16476.541 ms 2024-04-23T18:36:04.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:04 smithi050 ceph-mon[29187]: pgmap v556: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.8 MiB/s wr, 342 op/s 2024-04-23T18:36:04.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:04 smithi177 ceph-mon[32308]: pgmap v556: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.8 MiB/s wr, 342 op/s 2024-04-23T18:36:04.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:04 smithi188 ceph-mon[32694]: pgmap v556: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.8 MiB/s wr, 342 op/s 2024-04-23T18:36:05.573 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 23.60 MB/sec execute 159 sec latency 17476.593 ms 2024-04-23T18:36:05.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:05 smithi050 ceph-mon[29187]: osdmap e118: 12 total, 12 up, 12 in 2024-04-23T18:36:05.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:05 smithi177 ceph-mon[32308]: osdmap e118: 12 total, 12 up, 12 in 2024-04-23T18:36:05.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:05 smithi188 ceph-mon[32694]: osdmap e118: 12 total, 12 up, 12 in 2024-04-23T18:36:06.572 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 23.45 MB/sec execute 160 sec latency 18476.655 ms 2024-04-23T18:36:06.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:06 smithi050 ceph-mon[29187]: pgmap v558: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 4.1 MiB/s wr, 227 op/s 2024-04-23T18:36:06.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:06 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:06.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:06 smithi177 ceph-mon[32308]: pgmap v558: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 4.1 MiB/s wr, 227 op/s 2024-04-23T18:36:06.795 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:06 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:06.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:06 smithi188 ceph-mon[32694]: pgmap v558: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 4.1 MiB/s wr, 227 op/s 2024-04-23T18:36:06.845 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:06 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:07.572 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 23.31 MB/sec execute 161 sec latency 19476.773 ms 2024-04-23T18:36:07.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:07 smithi050 ceph-mon[29187]: pgmap v559: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 5.3 MiB/s wr, 172 op/s 2024-04-23T18:36:07.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:07 smithi177 ceph-mon[32308]: pgmap v559: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 5.3 MiB/s wr, 172 op/s 2024-04-23T18:36:07.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:07 smithi188 ceph-mon[32694]: pgmap v559: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 5.3 MiB/s wr, 172 op/s 2024-04-23T18:36:08.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:36:08.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.476+0000 7fc83b467640 1 -- 172.21.15.50:0/2132121570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc834100d60 msgr2=0x7fc834101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:08.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.476+0000 7fc83b467640 1 --2- 172.21.15.50:0/2132121570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc834100d60 0x7fc834101140 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fc82c009900 tx=0x7fc82c02a410 comp rx=0 tx=0).stop 2024-04-23T18:36:08.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.476+0000 7fc83b467640 1 -- 172.21.15.50:0/2132121570 shutdown_connections 2024-04-23T18:36:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.476+0000 7fc83b467640 1 --2- 172.21.15.50:0/2132121570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc83410e910 0x7fc834110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.476+0000 7fc83b467640 1 --2- 172.21.15.50:0/2132121570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc834101680 0x7fc83410e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:08.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.476+0000 7fc83b467640 1 --2- 172.21.15.50:0/2132121570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc834100d60 0x7fc834101140 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:08.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.476+0000 7fc83b467640 1 -- 172.21.15.50:0/2132121570 >> 172.21.15.50:0/2132121570 conn(0x7fc834077f50 msgr2=0x7fc834078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:08.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.476+0000 7fc83b467640 1 -- 172.21.15.50:0/2132121570 shutdown_connections 2024-04-23T18:36:08.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.476+0000 7fc83b467640 1 -- 172.21.15.50:0/2132121570 wait complete. 2024-04-23T18:36:08.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.477+0000 7fc83b467640 1 Processor -- start 2024-04-23T18:36:08.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.477+0000 7fc83b467640 1 -- start start 2024-04-23T18:36:08.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.477+0000 7fc83b467640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc834100d60 0x7fc8341a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:08.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.477+0000 7fc83b467640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc834101680 0x7fc8341a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:08.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.477+0000 7fc83b467640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc83410e910 0x7fc83419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:08.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.477+0000 7fc83b467640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc834114430 con 0x7fc834100d60 2024-04-23T18:36:08.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.477+0000 7fc83b467640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc8341142b0 con 0x7fc83410e910 2024-04-23T18:36:08.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.477+0000 7fc83b467640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc8341145b0 con 0x7fc834101680 2024-04-23T18:36:08.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.477+0000 7fc839203640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc834100d60 0x7fc8341a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:08.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.477+0000 7fc839203640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc834100d60 0x7fc8341a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39632/0 (socket says 172.21.15.50:39632) 2024-04-23T18:36:09.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.477+0000 7fc839203640 1 -- 172.21.15.50:0/1175041959 learned_addr learned my addr 172.21.15.50:0/1175041959 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:09.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.477+0000 7fc838a02640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc834101680 0x7fc8341a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:09.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc839a04640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc83410e910 0x7fc83419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:09.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc839203640 1 -- 172.21.15.50:0/1175041959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc834101680 msgr2=0x7fc8341a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:09.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc839203640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc834101680 0x7fc8341a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:09.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc839203640 1 -- 172.21.15.50:0/1175041959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc83410e910 msgr2=0x7fc83419a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:09.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc839203640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc83410e910 0x7fc83419a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:09.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc839203640 1 -- 172.21.15.50:0/1175041959 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83419af80 con 0x7fc834100d60 2024-04-23T18:36:09.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc839a04640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc83410e910 0x7fc83419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:36:09.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc839203640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc834100d60 0x7fc8341a0640 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fc82c005ab0 tx=0x7fc82c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:09.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc82a7fc640 1 -- 172.21.15.50:0/1175041959 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c03f070 con 0x7fc834100d60 2024-04-23T18:36:09.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc82a7fc640 1 -- 172.21.15.50:0/1175041959 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc82c031b60 con 0x7fc834100d60 2024-04-23T18:36:09.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc83b467640 1 -- 172.21.15.50:0/1175041959 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83419b210 con 0x7fc834100d60 2024-04-23T18:36:09.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc82a7fc640 1 -- 172.21.15.50:0/1175041959 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c035030 con 0x7fc834100d60 2024-04-23T18:36:09.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.478+0000 7fc83b467640 1 -- 172.21.15.50:0/1175041959 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8341a7490 con 0x7fc834100d60 2024-04-23T18:36:09.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.479+0000 7fc83b467640 1 -- 172.21.15.50:0/1175041959 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8340698e0 con 0x7fc834100d60 2024-04-23T18:36:09.393 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:08 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1175041959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:36:09.401 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:08 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1175041959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:36:09.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.480+0000 7fc82a7fc640 1 -- 172.21.15.50:0/1175041959 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc82c042070 con 0x7fc834100d60 2024-04-23T18:36:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.481+0000 7fc82a7fc640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc8200a3d20 0x7fc8200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:09.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.481+0000 7fc838a02640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc8200a3d20 0x7fc8200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:09.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.481+0000 7fc82a7fc640 1 -- 172.21.15.50:0/1175041959 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8586+0+0 (secure 0 0 0) 0x7fc82c0f0ff0 con 0x7fc834100d60 2024-04-23T18:36:09.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.482+0000 7fc838a02640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc8200a3d20 0x7fc8200a61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc83410f6e0 tx=0x7fc824005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:09.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.486+0000 7fc82a7fc640 1 -- 172.21.15.50:0/1175041959 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc82c0bd610 con 0x7fc834100d60 2024-04-23T18:36:09.527 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 23.16 MB/sec execute 162 sec latency 20476.841 ms 2024-04-23T18:36:09.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.714+0000 7fc83b467640 1 -- 172.21.15.50:0/1175041959 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc83402d5c0 con 0x7fc834100d60 2024-04-23T18:36:09.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.714+0000 7fc82a7fc640 1 -- 172.21.15.50:0/1175041959 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+11879 (secure 0 0 0) 0x7fc82c0bcf10 con 0x7fc834100d60 2024-04-23T18:36:09.560 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:36:09.572 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":57,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:36:04.975068+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27059":{"gid":27059,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/3902222090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3902222090},{"type":"v1","addr":"172.21.15.177:6837","nonce":3902222090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24499,24506]},"id":1}]} 2024-04-23T18:36:09.585 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 57 2024-04-23T18:36:09.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.717+0000 7fc83b467640 1 -- 172.21.15.50:0/1175041959 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc8200a3d20 msgr2=0x7fc8200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:09.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.717+0000 7fc83b467640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc8200a3d20 0x7fc8200a61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc83410f6e0 tx=0x7fc824005f90 comp rx=0 tx=0).stop 2024-04-23T18:36:09.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.717+0000 7fc83b467640 1 -- 172.21.15.50:0/1175041959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc834100d60 msgr2=0x7fc8341a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:09.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.717+0000 7fc83b467640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc834100d60 0x7fc8341a0640 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fc82c005ab0 tx=0x7fc82c003c10 comp rx=0 tx=0).stop 2024-04-23T18:36:09.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.717+0000 7fc83b467640 1 -- 172.21.15.50:0/1175041959 shutdown_connections 2024-04-23T18:36:09.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.717+0000 7fc83b467640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc8200a3d20 0x7fc8200a61e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:09.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.717+0000 7fc83b467640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc83410e910 0x7fc83419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:09.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.717+0000 7fc83b467640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc834101680 0x7fc8341a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:09.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.717+0000 7fc83b467640 1 --2- 172.21.15.50:0/1175041959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc834100d60 0x7fc8341a0640 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:09.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.717+0000 7fc83b467640 1 -- 172.21.15.50:0/1175041959 >> 172.21.15.50:0/1175041959 conn(0x7fc834077f50 msgr2=0x7fc83410fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:09.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.717+0000 7fc83b467640 1 -- 172.21.15.50:0/1175041959 shutdown_connections 2024-04-23T18:36:09.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:08.717+0000 7fc83b467640 1 -- 172.21.15.50:0/1175041959 wait complete. 2024-04-23T18:36:10.022 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id 7fa60ad4 --release --await' 2024-04-23T18:36:10.031 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id 7fa60ad4 --release --await 2024-04-23T18:36:10.061 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:09 smithi177 ceph-mon[32308]: pgmap v560: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 5.3 MiB/s wr, 172 op/s 2024-04-23T18:36:10.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:09 smithi188 ceph-mon[32694]: pgmap v560: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 5.3 MiB/s wr, 172 op/s 2024-04-23T18:36:10.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.805+0000 7f6c5ae60640 1 -- 172.21.15.50:0/2196973240 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c541072a0 msgr2=0x7f6c5410ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:10.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.805+0000 7f6c5ae60640 1 --2- 172.21.15.50:0/2196973240 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c541072a0 0x7f6c5410ee70 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c009ca0 tx=0x7f6c4c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:36:10.123 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.805+0000 7f6c5ae60640 1 -- 172.21.15.50:0/2196973240 shutdown_connections 2024-04-23T18:36:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.805+0000 7f6c5ae60640 1 --2- 172.21.15.50:0/2196973240 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c541072a0 0x7f6c5410ee70 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.805+0000 7f6c5ae60640 1 --2- 172.21.15.50:0/2196973240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c54102b80 0x7f6c54106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.805+0000 7f6c5ae60640 1 --2- 172.21.15.50:0/2196973240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c541021d0 0x7f6c541025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:10.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.805+0000 7f6c5ae60640 1 -- 172.21.15.50:0/2196973240 >> 172.21.15.50:0/2196973240 conn(0x7f6c540fda10 msgr2=0x7f6c540ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:10.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.805+0000 7f6c5ae60640 1 -- 172.21.15.50:0/2196973240 shutdown_connections 2024-04-23T18:36:10.354 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.805+0000 7f6c5ae60640 1 -- 172.21.15.50:0/2196973240 wait complete. 2024-04-23T18:36:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c5ae60640 1 Processor -- start 2024-04-23T18:36:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c5ae60640 1 -- start start 2024-04-23T18:36:10.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c5ae60640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c541021d0 0x7f6c54074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c5ae60640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c541072a0 0x7f6c54074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c5ae60640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c54074cd0 0x7f6c541a68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:10.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c5ae60640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6c54112210 con 0x7f6c541021d0 2024-04-23T18:36:10.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c5ae60640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6c54112090 con 0x7f6c54074cd0 2024-04-23T18:36:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c5ae60640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6c54112390 con 0x7f6c541072a0 2024-04-23T18:36:10.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c58bfc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c541021d0 0x7f6c54074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c58bfc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c541021d0 0x7f6c54074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39658/0 (socket says 172.21.15.50:39658) 2024-04-23T18:36:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c58bfc640 1 -- 172.21.15.50:0/3333769771 learned_addr learned my addr 172.21.15.50:0/3333769771 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c593fd640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c54074cd0 0x7f6c541a68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c58bfc640 1 -- 172.21.15.50:0/3333769771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c541072a0 msgr2=0x7f6c54074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:10.741 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c541072a0 0x7f6c54074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c58bfc640 1 -- 172.21.15.50:0/3333769771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c54074cd0 msgr2=0x7f6c541a68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:10.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c54074cd0 0x7f6c541a68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c58bfc640 1 -- 172.21.15.50:0/3333769771 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c541a6f40 con 0x7f6c541021d0 2024-04-23T18:36:10.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c593fd640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c54074cd0 0x7f6c541a68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:36:10.908 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c541021d0 0x7f6c54074250 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f6c4800d950 tx=0x7f6c4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.806+0000 7f6c51ffb640 1 -- 172.21.15.50:0/3333769771 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c48017030 con 0x7f6c541021d0 2024-04-23T18:36:10.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.807+0000 7f6c51ffb640 1 -- 172.21.15.50:0/3333769771 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c48004020 con 0x7f6c541021d0 2024-04-23T18:36:10.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.807+0000 7f6c5ae60640 1 -- 172.21.15.50:0/3333769771 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c541a7230 con 0x7f6c541021d0 2024-04-23T18:36:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.807+0000 7f6c51ffb640 1 -- 172.21.15.50:0/3333769771 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c480047a0 con 0x7f6c541021d0 2024-04-23T18:36:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.807+0000 7f6c5ae60640 1 -- 172.21.15.50:0/3333769771 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c541a7780 con 0x7f6c541021d0 2024-04-23T18:36:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.809+0000 7f6c51ffb640 1 -- 172.21.15.50:0/3333769771 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6c48009a30 con 0x7f6c541021d0 2024-04-23T18:36:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.809+0000 7f6c5db74740 1 Processor -- start 2024-04-23T18:36:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.809+0000 7f6c5db74740 1 -- start start 2024-04-23T18:36:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c5db74740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095620 0x55db0d095a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c5db74740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095f40 0x55db0d0a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c5db74740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d0a0800 0x55db0d0a2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c5db74740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55db0cffa970 con 0x55db0d0a0800 2024-04-23T18:36:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c5db74740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55db0cf974a0 con 0x55db0d095620 2024-04-23T18:36:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c5db74740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55db0cff6930 con 0x55db0d095f40 2024-04-23T18:36:11.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c53fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d0a0800 0x55db0d0a2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:11.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c53fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d0a0800 0x55db0d0a2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39670/0 (socket says 172.21.15.50:39670) 2024-04-23T18:36:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c53fff640 1 -- 172.21.15.50:0/3115114566 learned_addr learned my addr 172.21.15.50:0/3115114566 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:11.260 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/3115114566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095f40 0x55db0d0a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:11.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c51ffb640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6c440a3dc0 0x7f6c440a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:11.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c593fd640 1 --2- 172.21.15.50:0/3115114566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095620 0x55db0d095a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:11.297 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c53fff640 1 -- 172.21.15.50:0/3115114566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095f40 msgr2=0x55db0d0a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c53fff640 1 --2- 172.21.15.50:0/3115114566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095f40 0x55db0d0a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c593fd640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6c440a3dc0 0x7f6c440a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c53fff640 1 -- 172.21.15.50:0/3115114566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095620 msgr2=0x55db0d095a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c53fff640 1 --2- 172.21.15.50:0/3115114566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095620 0x55db0d095a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c53fff640 1 -- 172.21.15.50:0/3115114566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db0cd0d210 con 0x55db0d0a0800 2024-04-23T18:36:11.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c51ffb640 1 -- 172.21.15.50:0/3333769771 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8586+0+0 (secure 0 0 0) 0x7f6c480d0330 con 0x7f6c541021d0 2024-04-23T18:36:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.810+0000 7f6c53fff640 1 --2- 172.21.15.50:0/3115114566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d0a0800 0x55db0d0a2bf0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000ba20 tx=0x7f6c4000bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf74ea414baea0b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c3effd640 1 -- 172.21.15.50:0/3115114566 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c40016070 con 0x55db0d0a0800 2024-04-23T18:36:11.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c593fd640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6c440a3dc0 0x7f6c440a6280 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c009a60 tx=0x7f6c4c02a8c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c3effd640 1 -- 172.21.15.50:0/3115114566 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c400054d0 con 0x55db0d0a0800 2024-04-23T18:36:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c3effd640 1 -- 172.21.15.50:0/3115114566 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c40004ce0 con 0x55db0d0a0800 2024-04-23T18:36:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c5db74740 1 -- 172.21.15.50:0/3115114566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d0a0800 msgr2=0x55db0d0a2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c5db74740 1 --2- 172.21.15.50:0/3115114566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d0a0800 0x55db0d0a2bf0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000ba20 tx=0x7f6c4000bee0 comp rx=0 tx=0).stop 2024-04-23T18:36:11.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c5db74740 1 -- 172.21.15.50:0/3115114566 shutdown_connections 2024-04-23T18:36:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c5db74740 1 --2- 172.21.15.50:0/3115114566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d0a0800 0x55db0d0a2bf0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c5db74740 1 --2- 172.21.15.50:0/3115114566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095f40 0x55db0d0a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c5db74740 1 --2- 172.21.15.50:0/3115114566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095620 0x55db0d095a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c5db74740 1 -- 172.21.15.50:0/3115114566 >> 172.21.15.50:0/3115114566 conn(0x55db0d091b30 msgr2=0x55db0d091f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c5db74740 1 -- 172.21.15.50:0/3115114566 shutdown_connections 2024-04-23T18:36:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.811+0000 7f6c5db74740 1 -- 172.21.15.50:0/3115114566 wait complete. 2024-04-23T18:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c5db74740 1 -- start start 2024-04-23T18:36:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c5db74740 1 Processor -- start 2024-04-23T18:36:11.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c5db74740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095620 0x55db0d0a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c5db74740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095f40 0x55db0d09d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c5db74740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d0a0800 0x55db0d09d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:11.846 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c5db74740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55db0d0a3430 con 0x55db0d095f40 2024-04-23T18:36:11.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c5db74740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55db0cf950b0 con 0x55db0d0a0800 2024-04-23T18:36:11.894 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c5db74740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55db0d0a35b0 con 0x55db0d095620 2024-04-23T18:36:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c5db74740 10 client.0 _ll_register_callbacks cb 0x55db0cf82d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c53fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095f40 0x55db0d09d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c53fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095f40 0x55db0d09d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39674/0 (socket says 172.21.15.50:39674) 2024-04-23T18:36:11.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c53fff640 1 -- 172.21.15.50:0/1633843954 learned_addr learned my addr 172.21.15.50:0/1633843954 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:11.982 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095620 0x55db0d0a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:11.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.812+0000 7f6c593fd640 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d0a0800 0x55db0d09d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:12.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c53fff640 1 -- 172.21.15.50:0/1633843954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095620 msgr2=0x55db0d0a00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:12.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c53fff640 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095620 0x55db0d0a00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c53fff640 1 -- 172.21.15.50:0/1633843954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d0a0800 msgr2=0x55db0d09d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:12.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c53fff640 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d0a0800 0x55db0d09d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:12.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c53fff640 1 -- 172.21.15.50:0/1633843954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db0cdd9620 con 0x55db0d095f40 2024-04-23T18:36:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c593fd640 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d0a0800 0x55db0d09d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:36:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c53fff640 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095f40 0x55db0d09d3c0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000b9f0 tx=0x7f6c400052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3eabf5fc1cc898e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c33fff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:36:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c33fff640 1 -- 172.21.15.50:0/1633843954 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c40016030 con 0x55db0d095f40 2024-04-23T18:36:12.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c33fff640 1 -- 172.21.15.50:0/1633843954 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8586+0+0 (secure 0 0 0) 0x7f6c4000da40 con 0x55db0d095f40 2024-04-23T18:36:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c5db74740 1 -- 172.21.15.50:0/1633843954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c400054d0 con 0x55db0d095f40 2024-04-23T18:36:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c33fff640 1 -- 172.21.15.50:0/1633843954 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f6c4000dc40 con 0x55db0d095f40 2024-04-23T18:36:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c5db74740 10 client.17649 fetch_fsmap learned FSMap version 57 2024-04-23T18:36:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c5db74740 1 -- 172.21.15.50:0/1633843954 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55db0d0af2d0 con 0x55db0d095f40 2024-04-23T18:36:12.324 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c33fff640 1 -- 172.21.15.50:0/1633843954 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c400054d0 con 0x55db0d095f40 2024-04-23T18:36:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.813+0000 7f6c33fff640 1 -- 172.21.15.50:0/1633843954 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 57) ==== 8350+0+0 (secure 0 0 0) 0x7f6c4002a030 con 0x55db0d095f40 2024-04-23T18:36:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.814+0000 7f6c5db74740 10 client.17649 fetch_fsmap finished waiting for FSMap version 57 2024-04-23T18:36:12.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.814+0000 7f6c5db74740 10 client.17649 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:36:12.367 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.814+0000 7f6c5db74740 20 client.17649 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.814+0000 7f6c5db74740 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55db0d0b06e0 0x55db0d0b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:12.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.814+0000 7f6c5db74740 4 client.17649 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:36:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.814+0000 7f6c5db74740 1 -- 172.21.15.50:0/1633843954 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db0d008390 con 0x55db0d0b06e0 2024-04-23T18:36:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.814+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55db0d0b06e0 0x55db0d0b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.814+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55db0d0b06e0 0x55db0d0b2ad0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=350b0914fad718c7 server_cookie=dd1e9d12953d93cc in_seq=0 out_seq=0 2024-04-23T18:36:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.814+0000 7f6c33fff640 10 client.17649 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:12.505 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c33fff640 1 -- 172.21.15.50:0/1633843954 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db0d008390 con 0x55db0d0b06e0 2024-04-23T18:36:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c33fff640 10 client.17649 handle_command_reply: tid=0 2024-04-23T18:36:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c5db74740 1 client.17649 shutdown 2024-04-23T18:36:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c5db74740 1 -- 172.21.15.50:0/1633843954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095f40 msgr2=0x55db0d09d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:12.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c5db74740 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095f40 0x55db0d09d3c0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000b9f0 tx=0x7f6c400052d0 comp rx=0 tx=0).stop 2024-04-23T18:36:12.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c5db74740 1 -- 172.21.15.50:0/1633843954 shutdown_connections 2024-04-23T18:36:12.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c5db74740 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55db0d0b06e0 0x55db0d0b2ad0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:12.651 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c5db74740 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d0a0800 0x55db0d09d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:12.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c5db74740 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095f40 0x55db0d09d3c0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c5db74740 1 --2- 172.21.15.50:0/1633843954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095620 0x55db0d0a00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c33fff640 0 client.17649 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:12.695 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c5db74740 1 -- 172.21.15.50:0/1633843954 >> 172.21.15.50:0/1633843954 conn(0x55db0d091b30 msgr2=0x55db0d097860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c5db74740 1 -- 172.21.15.50:0/1633843954 shutdown_connections 2024-04-23T18:36:12.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c5db74740 1 -- 172.21.15.50:0/1633843954 wait complete. 2024-04-23T18:36:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.815+0000 7f6c5db74740 20 client.17649 trim_cache size 0 max 16384 2024-04-23T18:36:12.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.830+0000 7f6c5db74740 1 Processor -- start 2024-04-23T18:36:12.835 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.830+0000 7f6c5db74740 1 -- start start 2024-04-23T18:36:12.847 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.830+0000 7f6c5db74740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d091b30 0x55db0d0ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.830+0000 7f6c5db74740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095620 0x55db0d0b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.830+0000 7f6c5db74740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095f40 0x55db0d0b8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:12.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.830+0000 7f6c5db74740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6c541b27d0 con 0x55db0d091b30 2024-04-23T18:36:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.830+0000 7f6c5db74740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6c541b0040 con 0x55db0d095620 2024-04-23T18:36:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.830+0000 7f6c5db74740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6c541adeb0 con 0x55db0d095f40 2024-04-23T18:36:12.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.830+0000 7f6c58bfc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d091b30 0x55db0d0ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.830+0000 7f6c58bfc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d091b30 0x55db0d0ac2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39686/0 (socket says 172.21.15.50:39686) 2024-04-23T18:36:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c58bfc640 1 -- 172.21.15.50:0/153156606 learned_addr learned my addr 172.21.15.50:0/153156606 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c53fff640 1 --2- 172.21.15.50:0/153156606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095620 0x55db0d0b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:13.042 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c593fd640 1 --2- 172.21.15.50:0/153156606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095f40 0x55db0d0b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c58bfc640 1 -- 172.21.15.50:0/153156606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095f40 msgr2=0x55db0d0b8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:13.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/153156606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095f40 0x55db0d0b8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:13.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c58bfc640 1 -- 172.21.15.50:0/153156606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095620 msgr2=0x55db0d0b82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/153156606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095620 0x55db0d0b82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c58bfc640 1 -- 172.21.15.50:0/153156606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db0d095d70 con 0x55db0d091b30 2024-04-23T18:36:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c53fff640 1 --2- 172.21.15.50:0/153156606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095620 0x55db0d0b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:36:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c593fd640 1 --2- 172.21.15.50:0/153156606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095f40 0x55db0d0b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:36:13.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/153156606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d091b30 0x55db0d0ac2f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f6c48098c60 tx=0x7f6c4809c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d165aa69907d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c3e7fc640 1 -- 172.21.15.50:0/153156606 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c4809c990 con 0x55db0d091b30 2024-04-23T18:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c3e7fc640 1 -- 172.21.15.50:0/153156606 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c4809c290 con 0x55db0d091b30 2024-04-23T18:36:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.831+0000 7f6c3e7fc640 1 -- 172.21.15.50:0/153156606 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c480cdcc0 con 0x55db0d091b30 2024-04-23T18:36:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.832+0000 7f6c5db74740 1 -- 172.21.15.50:0/153156606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d091b30 msgr2=0x55db0d0ac2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.832+0000 7f6c5db74740 1 --2- 172.21.15.50:0/153156606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d091b30 0x55db0d0ac2f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f6c48098c60 tx=0x7f6c4809c190 comp rx=0 tx=0).stop 2024-04-23T18:36:13.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.832+0000 7f6c5db74740 1 -- 172.21.15.50:0/153156606 shutdown_connections 2024-04-23T18:36:13.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.832+0000 7f6c5db74740 1 --2- 172.21.15.50:0/153156606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d095f40 0x55db0d0b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:13.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.832+0000 7f6c5db74740 1 --2- 172.21.15.50:0/153156606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095620 0x55db0d0b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:13.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.832+0000 7f6c5db74740 1 --2- 172.21.15.50:0/153156606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d091b30 0x55db0d0ac2f0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:13.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.832+0000 7f6c5db74740 1 -- 172.21.15.50:0/153156606 >> 172.21.15.50:0/153156606 conn(0x55db0d09da60 msgr2=0x55db0d090420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.832+0000 7f6c5db74740 1 -- 172.21.15.50:0/153156606 shutdown_connections 2024-04-23T18:36:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.832+0000 7f6c5db74740 1 -- 172.21.15.50:0/153156606 wait complete. 2024-04-23T18:36:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.832+0000 7f6c5db74740 1 -- start start 2024-04-23T18:36:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.832+0000 7f6c5db74740 1 Processor -- start 2024-04-23T18:36:13.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c5db74740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d091b30 0x55db0d0a8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c5db74740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095620 0x55db0d0a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:13.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c5db74740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095f40 0x55db0d0a6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c5db74740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6c541ae500 con 0x55db0d095620 2024-04-23T18:36:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c5db74740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6c541af970 con 0x55db0d095f40 2024-04-23T18:36:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c5db74740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6c541aebd0 con 0x55db0d091b30 2024-04-23T18:36:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c5db74740 10 client.0 _ll_register_callbacks cb 0x7f6c40005870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c53fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095620 0x55db0d0a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:13.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c53fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095620 0x55db0d0a5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39702/0 (socket says 172.21.15.50:39702) 2024-04-23T18:36:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c53fff640 1 -- 172.21.15.50:0/1814411402 learned_addr learned my addr 172.21.15.50:0/1814411402 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d091b30 0x55db0d0a8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c593fd640 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095f40 0x55db0d0a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:13.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c53fff640 1 -- 172.21.15.50:0/1814411402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d091b30 msgr2=0x55db0d0a8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:13.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c53fff640 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d091b30 0x55db0d0a8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c53fff640 1 -- 172.21.15.50:0/1814411402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095f40 msgr2=0x55db0d0a6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c53fff640 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095f40 0x55db0d0a6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:14.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c53fff640 1 -- 172.21.15.50:0/1814411402 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db0d0a69f0 con 0x55db0d095620 2024-04-23T18:36:14.037 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d091b30 0x55db0d0a8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:36:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c593fd640 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095f40 0x55db0d0a6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:36:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c53fff640 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095620 0x55db0d0a5e10 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f6c40003920 tx=0x7f6c4000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08b381025f1167e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:14.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c33fff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:36:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c33fff640 1 -- 172.21.15.50:0/1814411402 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c40016070 con 0x55db0d095620 2024-04-23T18:36:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c33fff640 1 -- 172.21.15.50:0/1814411402 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8586+0+0 (secure 0 0 0) 0x7f6c40005550 con 0x55db0d095620 2024-04-23T18:36:14.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c5db74740 1 -- 172.21.15.50:0/1814411402 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c4809c290 con 0x55db0d095620 2024-04-23T18:36:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c33fff640 1 -- 172.21.15.50:0/1814411402 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f6c4003b050 con 0x55db0d095620 2024-04-23T18:36:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c5db74740 10 client.17655 fetch_fsmap learned FSMap version 57 2024-04-23T18:36:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.833+0000 7f6c5db74740 1 -- 172.21.15.50:0/1814411402 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55db0d116a30 con 0x55db0d095620 2024-04-23T18:36:14.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.834+0000 7f6c33fff640 1 -- 172.21.15.50:0/1814411402 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c4809c290 con 0x55db0d095620 2024-04-23T18:36:14.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.834+0000 7f6c33fff640 1 -- 172.21.15.50:0/1814411402 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 57) ==== 8350+0+0 (secure 0 0 0) 0x7f6c40033020 con 0x55db0d095620 2024-04-23T18:36:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.834+0000 7f6c5db74740 10 client.17655 fetch_fsmap finished waiting for FSMap version 57 2024-04-23T18:36:14.298 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.834+0000 7f6c5db74740 10 client.17655 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:36:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.834+0000 7f6c5db74740 20 client.17655 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.834+0000 7f6c5db74740 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55db0d0a0800 0x55db0d1192a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.834+0000 7f6c5db74740 4 client.17655 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:36:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.834+0000 7f6c5db74740 1 -- 172.21.15.50:0/1814411402 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55db0d002200 con 0x55db0d0a0800 2024-04-23T18:36:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.834+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55db0d0a0800 0x55db0d1192a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.835+0000 7f6c58bfc640 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55db0d0a0800 0x55db0d1192a0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0789b5f7a622e6f server_cookie=33bb8b866c9ce0be in_seq=0 out_seq=0 2024-04-23T18:36:14.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.835+0000 7f6c33fff640 10 client.17655 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:14.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c33fff640 1 -- 172.21.15.50:0/1814411402 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55db0d002200 con 0x55db0d0a0800 2024-04-23T18:36:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c33fff640 10 client.17655 handle_command_reply: tid=0 2024-04-23T18:36:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c5db74740 1 client.17655 shutdown 2024-04-23T18:36:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c5db74740 1 -- 172.21.15.50:0/1814411402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095620 msgr2=0x55db0d0a5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c5db74740 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095620 0x55db0d0a5e10 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f6c40003920 tx=0x7f6c4000bf60 comp rx=0 tx=0).stop 2024-04-23T18:36:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c5db74740 1 -- 172.21.15.50:0/1814411402 shutdown_connections 2024-04-23T18:36:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c5db74740 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55db0d0a0800 0x55db0d1192a0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:14.685 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c5db74740 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db0d095f40 0x55db0d0a6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c5db74740 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55db0d095620 0x55db0d0a5e10 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c5db74740 1 --2- 172.21.15.50:0/1814411402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55db0d091b30 0x55db0d0a8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:14.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c33fff640 0 client.17655 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c5db74740 1 -- 172.21.15.50:0/1814411402 >> 172.21.15.50:0/1814411402 conn(0x55db0d09da60 msgr2=0x55db0d090cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c5db74740 1 -- 172.21.15.50:0/1814411402 shutdown_connections 2024-04-23T18:36:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c5db74740 1 -- 172.21.15.50:0/1814411402 wait complete. 2024-04-23T18:36:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.954+0000 7f6c5db74740 20 client.17655 trim_cache size 0 max 16384 2024-04-23T18:36:14.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:36:14.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:36:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:36:15.018 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:36:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.955+0000 7f6c5ae60640 1 -- 172.21.15.50:0/3333769771 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6c440a3dc0 msgr2=0x7f6c440a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.955+0000 7f6c5ae60640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6c440a3dc0 0x7f6c440a6280 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c009a60 tx=0x7f6c4c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:36:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.955+0000 7f6c5ae60640 1 -- 172.21.15.50:0/3333769771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c541021d0 msgr2=0x7f6c54074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.955+0000 7f6c5ae60640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c541021d0 0x7f6c54074250 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f6c4800d950 tx=0x7f6c4800de10 comp rx=0 tx=0).stop 2024-04-23T18:36:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.955+0000 7f6c5ae60640 1 -- 172.21.15.50:0/3333769771 shutdown_connections 2024-04-23T18:36:15.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.955+0000 7f6c5ae60640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6c440a3dc0 0x7f6c440a6280 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:15.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.955+0000 7f6c5ae60640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c54074cd0 0x7f6c541a68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.955+0000 7f6c5ae60640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c541072a0 0x7f6c54074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.955+0000 7f6c5ae60640 1 --2- 172.21.15.50:0/3333769771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c541021d0 0x7f6c54074250 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.955+0000 7f6c5ae60640 1 -- 172.21.15.50:0/3333769771 >> 172.21.15.50:0/3333769771 conn(0x7f6c540fda10 msgr2=0x7f6c540ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.955+0000 7f6c5ae60640 1 -- 172.21.15.50:0/3333769771 shutdown_connections 2024-04-23T18:36:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:08.955+0000 7f6c5ae60640 1 -- 172.21.15.50:0/3333769771 wait complete. 2024-04-23T18:36:15.398 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:08 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1175041959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:36:15.455 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 23.02 MB/sec execute 163 sec latency 21476.875 ms 2024-04-23T18:36:15.504 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:09 smithi050 ceph-mon[29187]: pgmap v560: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 5.3 MiB/s wr, 172 op/s 2024-04-23T18:36:15.574 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 22.88 MB/sec execute 164 sec latency 22476.935 ms 2024-04-23T18:36:15.597 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 22.74 MB/sec execute 165 sec latency 23476.993 ms 2024-04-23T18:36:15.618 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:12 smithi050 ceph-mon[29187]: pgmap v561: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 5.3 MiB/s wr, 172 op/s 2024-04-23T18:36:15.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:12 smithi050 ceph-mon[29187]: osdmap e119: 12 total, 12 up, 12 in 2024-04-23T18:36:15.724 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 22.61 MB/sec execute 166 sec latency 24477.083 ms 2024-04-23T18:36:15.771 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 22.47 MB/sec execute 167 sec latency 25477.197 ms 2024-04-23T18:36:15.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:14 smithi050 ceph-mon[29187]: pgmap v563: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 6.3 MiB/s wr, 224 op/s 2024-04-23T18:36:15.796 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 22.34 MB/sec execute 168 sec latency 26477.278 ms 2024-04-23T18:36:15.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:12 smithi188 ceph-mon[32694]: pgmap v561: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 5.3 MiB/s wr, 172 op/s 2024-04-23T18:36:15.819 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:12 smithi188 ceph-mon[32694]: osdmap e119: 12 total, 12 up, 12 in 2024-04-23T18:36:15.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:14 smithi188 ceph-mon[32694]: pgmap v563: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 6.3 MiB/s wr, 224 op/s 2024-04-23T18:36:15.943 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:15 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:16.005 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:12 smithi177 ceph-mon[32308]: pgmap v561: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 5.3 MiB/s wr, 172 op/s 2024-04-23T18:36:16.081 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:12 smithi177 ceph-mon[32308]: osdmap e119: 12 total, 12 up, 12 in 2024-04-23T18:36:16.090 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:14 smithi177 ceph-mon[32308]: pgmap v563: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 6.3 MiB/s wr, 224 op/s 2024-04-23T18:36:16.157 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:15 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:16.168 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:15 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:16.226 INFO:tasks.workunit.client.0.smithi050.stdout: 10 121897 22.21 MB/sec execute 169 sec latency 27477.338 ms 2024-04-23T18:36:16.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.628+0000 7fcdafc74640 1 -- 172.21.15.50:0/2324631703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda81016d0 msgr2=0x7fcda810be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:16.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.628+0000 7fcdafc74640 1 --2- 172.21.15.50:0/2324631703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda81016d0 0x7fcda810be30 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fcd98009900 tx=0x7fcd9802a410 comp rx=0 tx=0).stop 2024-04-23T18:36:16.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.628+0000 7fcdafc74640 1 -- 172.21.15.50:0/2324631703 shutdown_connections 2024-04-23T18:36:16.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.628+0000 7fcdafc74640 1 --2- 172.21.15.50:0/2324631703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcda810c4f0 0x7fcda810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:16.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.628+0000 7fcdafc74640 1 --2- 172.21.15.50:0/2324631703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda81016d0 0x7fcda810be30 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.628+0000 7fcdafc74640 1 --2- 172.21.15.50:0/2324631703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcda8100d20 0x7fcda8101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.628+0000 7fcdafc74640 1 -- 172.21.15.50:0/2324631703 >> 172.21.15.50:0/2324631703 conn(0x7fcda8076150 msgr2=0x7fcda8076560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:16.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.628+0000 7fcdafc74640 1 -- 172.21.15.50:0/2324631703 shutdown_connections 2024-04-23T18:36:16.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.628+0000 7fcdafc74640 1 -- 172.21.15.50:0/2324631703 wait complete. 2024-04-23T18:36:16.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdafc74640 1 Processor -- start 2024-04-23T18:36:16.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdafc74640 1 -- start start 2024-04-23T18:36:16.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdafc74640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda8100d20 0x7fcda8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:16.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdafc74640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcda810c4f0 0x7fcda8198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:16.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdafc74640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcda819cfe0 0x7fcda819d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:16.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdafc74640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81121a0 con 0x7fcda8100d20 2024-04-23T18:36:16.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdafc74640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcda8112020 con 0x7fcda810c4f0 2024-04-23T18:36:16.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdafc74640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcda8112320 con 0x7fcda819cfe0 2024-04-23T18:36:16.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdada10640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda8100d20 0x7fcda8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:16.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdada10640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda8100d20 0x7fcda8198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51892/0 (socket says 172.21.15.50:51892) 2024-04-23T18:36:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdada10640 1 -- 172.21.15.50:0/3427435357 learned_addr learned my addr 172.21.15.50:0/3427435357 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:16.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdad20f640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcda810c4f0 0x7fcda8198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:16.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.629+0000 7fcdae211640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcda819cfe0 0x7fcda819d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:16.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.630+0000 7fcdada10640 1 -- 172.21.15.50:0/3427435357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcda819cfe0 msgr2=0x7fcda819d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:16.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.630+0000 7fcdada10640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcda819cfe0 0x7fcda819d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:16.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.630+0000 7fcdada10640 1 -- 172.21.15.50:0/3427435357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcda810c4f0 msgr2=0x7fcda8198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:16.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.630+0000 7fcdada10640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcda810c4f0 0x7fcda8198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:16.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.630+0000 7fcdada10640 1 -- 172.21.15.50:0/3427435357 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda819dad0 con 0x7fcda8100d20 2024-04-23T18:36:16.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.630+0000 7fcdada10640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda8100d20 0x7fcda8198720 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fcda400d950 tx=0x7fcda400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:17.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.630+0000 7fcd9effd640 1 -- 172.21.15.50:0/3427435357 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda4017070 con 0x7fcda8100d20 2024-04-23T18:36:17.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.630+0000 7fcdafc74640 1 -- 172.21.15.50:0/3427435357 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcda819ddc0 con 0x7fcda8100d20 2024-04-23T18:36:17.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.630+0000 7fcd9effd640 1 -- 172.21.15.50:0/3427435357 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcda4009c30 con 0x7fcda8100d20 2024-04-23T18:36:17.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.630+0000 7fcd9effd640 1 -- 172.21.15.50:0/3427435357 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda400f620 con 0x7fcda8100d20 2024-04-23T18:36:17.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.630+0000 7fcdafc74640 1 -- 172.21.15.50:0/3427435357 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda81a5580 con 0x7fcda8100d20 2024-04-23T18:36:17.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.632+0000 7fcd9effd640 1 -- 172.21.15.50:0/3427435357 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fcda4010460 con 0x7fcda8100d20 2024-04-23T18:36:17.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.633+0000 7fcdb2982740 1 Processor -- start 2024-04-23T18:36:17.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.633+0000 7fcdb2982740 1 -- start start 2024-04-23T18:36:17.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.633+0000 7fcdb2982740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd7a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.633+0000 7fcdb2982740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd84f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:17.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.633+0000 7fcdb2982740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd87870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.633+0000 7fcdb2982740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56538dcdf6d0 con 0x56538dd7a490 2024-04-23T18:36:17.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.633+0000 7fcd9effd640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcd940a3d20 0x7fcd940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:17.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.633+0000 7fcdb2982740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56538dc7c1a0 con 0x56538dd7adb0 2024-04-23T18:36:17.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.633+0000 7fcdb2982740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56538dcdb690 con 0x56538dd85480 2024-04-23T18:36:17.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdada10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd84f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:17.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdada10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd84f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:44054/0 (socket says 172.21.15.50:44054) 2024-04-23T18:36:17.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdada10640 1 -- 172.21.15.50:0/30771992 learned_addr learned my addr 172.21.15.50:0/30771992 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdae211640 1 --2- 172.21.15.50:0/30771992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd7a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:17.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdad20f640 1 --2- 172.21.15.50:0/30771992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd87870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:17.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdae211640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcd940a3d20 0x7fcd940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:17.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcd9effd640 1 -- 172.21.15.50:0/3427435357 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8542+0+0 (secure 0 0 0) 0x7fcda40d03a0 con 0x7fcda8100d20 2024-04-23T18:36:17.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdada10640 1 -- 172.21.15.50:0/30771992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 msgr2=0x56538dd87870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:17.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdada10640 1 --2- 172.21.15.50:0/30771992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd87870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:17.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdada10640 1 -- 172.21.15.50:0/30771992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 msgr2=0x56538dd7a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdada10640 1 --2- 172.21.15.50:0/30771992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd7a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:17.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdada10640 1 -- 172.21.15.50:0/30771992 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56538dc865b0 con 0x56538dd7adb0 2024-04-23T18:36:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdada10640 1 --2- 172.21.15.50:0/30771992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd84f40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcda4098db0 tx=0x7fcda4005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f9d4e8d62ace88c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:17.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.634+0000 7fcdae211640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcd940a3d20 0x7fcd940a61e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcda0005c40 tx=0x7fcda0009500 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:17.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.635+0000 7fcd93fff640 1 -- 172.21.15.50:0/30771992 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda4020070 con 0x56538dd7adb0 2024-04-23T18:36:17.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.635+0000 7fcd93fff640 1 -- 172.21.15.50:0/30771992 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcda409c3e0 con 0x56538dd7adb0 2024-04-23T18:36:17.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.635+0000 7fcd93fff640 1 -- 172.21.15.50:0/30771992 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda4098a50 con 0x56538dd7adb0 2024-04-23T18:36:17.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.636+0000 7fcdb2982740 1 -- 172.21.15.50:0/30771992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 msgr2=0x56538dd84f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:17.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.636+0000 7fcdb2982740 1 --2- 172.21.15.50:0/30771992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd84f40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcda4098db0 tx=0x7fcda4005880 comp rx=0 tx=0).stop 2024-04-23T18:36:17.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.636+0000 7fcdb2982740 1 -- 172.21.15.50:0/30771992 shutdown_connections 2024-04-23T18:36:17.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.636+0000 7fcdb2982740 1 --2- 172.21.15.50:0/30771992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd87870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:17.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.636+0000 7fcdb2982740 1 --2- 172.21.15.50:0/30771992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd84f40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:17.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.636+0000 7fcdb2982740 1 --2- 172.21.15.50:0/30771992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd7a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:17.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.636+0000 7fcdb2982740 1 -- 172.21.15.50:0/30771992 >> 172.21.15.50:0/30771992 conn(0x56538dd767b0 msgr2=0x56538dd76b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.636+0000 7fcdb2982740 1 -- 172.21.15.50:0/30771992 shutdown_connections 2024-04-23T18:36:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.636+0000 7fcdb2982740 1 -- 172.21.15.50:0/30771992 wait complete. 2024-04-23T18:36:17.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.636+0000 7fcdb2982740 1 -- start start 2024-04-23T18:36:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.636+0000 7fcdb2982740 1 Processor -- start 2024-04-23T18:36:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdb2982740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd84ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:17.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdb2982740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd821c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:17.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdb2982740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd82900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:17.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdb2982740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56538dd88250 con 0x56538dd7a490 2024-04-23T18:36:17.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdada10640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd84ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:18.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdb2982740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56538dc79b20 con 0x56538dd7adb0 2024-04-23T18:36:18.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdb2982740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56538dd883d0 con 0x56538dd85480 2024-04-23T18:36:18.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdad20f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd821c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdada10640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd84ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51910/0 (socket says 172.21.15.50:51910) 2024-04-23T18:36:18.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdada10640 1 -- 172.21.15.50:0/3046275007 learned_addr learned my addr 172.21.15.50:0/3046275007 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:18.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdb2982740 10 client.0 _ll_register_callbacks cb 0x56538dc79530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:18.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdae211640 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd82900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:18.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdad20f640 1 -- 172.21.15.50:0/3046275007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 msgr2=0x56538dd82900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdad20f640 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd82900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:18.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdad20f640 1 -- 172.21.15.50:0/3046275007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 msgr2=0x56538dd84ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdad20f640 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd84ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdad20f640 1 -- 172.21.15.50:0/3046275007 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56538d9f1920 con 0x56538dd7adb0 2024-04-23T18:36:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdae211640 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd82900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:36:18.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.637+0000 7fcdada10640 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd84ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:36:18.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcdad20f640 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd821c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcd9802a930 tx=0x7fcd98032f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acf7eccedddb1f73 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:18.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcd917fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:36:18.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcd917fa640 1 -- 172.21.15.50:0/3046275007 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd98035040 con 0x56538dd7adb0 2024-04-23T18:36:18.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcd917fa640 1 -- 172.21.15.50:0/3046275007 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8542+0+0 (secure 0 0 0) 0x7fcd980317e0 con 0x56538dd7adb0 2024-04-23T18:36:18.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcdb2982740 1 -- 172.21.15.50:0/3046275007 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcda409c3e0 con 0x56538dd7adb0 2024-04-23T18:36:18.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcd917fa640 1 -- 172.21.15.50:0/3046275007 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fcd98004040 con 0x56538dd7adb0 2024-04-23T18:36:18.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcdb2982740 10 client.27164 fetch_fsmap learned FSMap version 58 2024-04-23T18:36:18.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcdb2982740 1 -- 172.21.15.50:0/3046275007 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x56538dabdd30 con 0x56538dd7adb0 2024-04-23T18:36:18.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcd917fa640 1 -- 172.21.15.50:0/3046275007 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcda409c3e0 con 0x56538dd7adb0 2024-04-23T18:36:18.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcd917fa640 1 -- 172.21.15.50:0/3046275007 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 58) ==== 8346+0+0 (secure 0 0 0) 0x7fcd980047e0 con 0x56538dd7adb0 2024-04-23T18:36:18.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcdb2982740 10 client.27164 fetch_fsmap finished waiting for FSMap version 58 2024-04-23T18:36:18.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcdb2982740 10 client.27164 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:36:18.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcdb2982740 20 client.27164 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:18.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcdb2982740 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56538dd955a0 0x56538dd979f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:18.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcdb2982740 4 client.27164 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:36:18.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.638+0000 7fcdb2982740 1 -- 172.21.15.50:0/3046275007 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56538dc9d500 con 0x56538dd955a0 2024-04-23T18:36:18.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.639+0000 7fcdada10640 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56538dd955a0 0x56538dd979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:18.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.639+0000 7fcd917fa640 10 client.27164 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:18.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.639+0000 7fcdada10640 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56538dd955a0 0x56538dd979f0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b1d3f2bcdc92ac7 server_cookie=50abb47c2efef930 in_seq=0 out_seq=0 2024-04-23T18:36:18.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.639+0000 7fcd917fa640 1 -- 172.21.15.50:0/3046275007 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56538dc9d500 con 0x56538dd955a0 2024-04-23T18:36:18.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.639+0000 7fcd917fa640 10 client.27164 handle_command_reply: tid=0 2024-04-23T18:36:18.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcdb2982740 1 client.27164 shutdown 2024-04-23T18:36:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcdb2982740 1 -- 172.21.15.50:0/3046275007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 msgr2=0x56538dd821c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:18.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcdb2982740 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd821c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcd9802a930 tx=0x7fcd98032f60 comp rx=0 tx=0).stop 2024-04-23T18:36:18.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcdb2982740 1 -- 172.21.15.50:0/3046275007 shutdown_connections 2024-04-23T18:36:18.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcdb2982740 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56538dd955a0 0x56538dd979f0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:19.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcdb2982740 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd82900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:19.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcdb2982740 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd821c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:19.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcdb2982740 1 --2- 172.21.15.50:0/3046275007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd84ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:19.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcd917fa640 0 client.27164 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:19.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcdb2982740 1 -- 172.21.15.50:0/3046275007 >> 172.21.15.50:0/3046275007 conn(0x56538dd767b0 msgr2=0x56538dd7c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:19.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcdb2982740 1 -- 172.21.15.50:0/3046275007 shutdown_connections 2024-04-23T18:36:19.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcdb2982740 1 -- 172.21.15.50:0/3046275007 wait complete. 2024-04-23T18:36:19.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.640+0000 7fcdb2982740 20 client.27164 trim_cache size 0 max 16384 2024-04-23T18:36:19.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.672+0000 7fcdb2982740 1 Processor -- start 2024-04-23T18:36:19.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.672+0000 7fcdb2982740 1 -- start start 2024-04-23T18:36:19.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.672+0000 7fcdb2982740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd7a490 0x56538dd91110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:19.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdb2982740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd84a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:19.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdb2982740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd85480 0x56538dd84f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:19.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdb2982740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81ae260 con 0x56538dd85480 2024-04-23T18:36:19.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdb2982740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81abb00 con 0x56538dd7adb0 2024-04-23T18:36:19.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdb2982740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81a9970 con 0x56538dd7a490 2024-04-23T18:36:19.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdada10640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd7a490 0x56538dd91110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:19.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdada10640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd7a490 0x56538dd91110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:41180/0 (socket says 172.21.15.50:41180) 2024-04-23T18:36:19.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdada10640 1 -- 172.21.15.50:0/598771576 learned_addr learned my addr 172.21.15.50:0/598771576 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:19.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdae211640 1 --2- 172.21.15.50:0/598771576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd85480 0x56538dd84f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:19.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdad20f640 1 --2- 172.21.15.50:0/598771576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd84a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:19.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdae211640 1 -- 172.21.15.50:0/598771576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd7a490 msgr2=0x56538dd91110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:19.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdae211640 1 --2- 172.21.15.50:0/598771576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd7a490 0x56538dd91110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:19.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdae211640 1 -- 172.21.15.50:0/598771576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 msgr2=0x56538dd84a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:19.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdae211640 1 --2- 172.21.15.50:0/598771576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd84a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:19.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.673+0000 7fcdae211640 1 -- 172.21.15.50:0/598771576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56538dd7abe0 con 0x56538dd85480 2024-04-23T18:36:19.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.674+0000 7fcdae211640 1 --2- 172.21.15.50:0/598771576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd85480 0x56538dd84f40 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fcda0011fb0 tx=0x7fcda0005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7627bb7f7c7032cc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:19.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.674+0000 7fcd937fe640 1 -- 172.21.15.50:0/598771576 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda00118d0 con 0x56538dd85480 2024-04-23T18:36:19.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.674+0000 7fcd937fe640 1 -- 172.21.15.50:0/598771576 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcda000fcc0 con 0x56538dd85480 2024-04-23T18:36:19.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.674+0000 7fcd937fe640 1 -- 172.21.15.50:0/598771576 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda0005020 con 0x56538dd85480 2024-04-23T18:36:19.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.675+0000 7fcdb2982740 1 -- 172.21.15.50:0/598771576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd85480 msgr2=0x56538dd84f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:19.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.675+0000 7fcdb2982740 1 --2- 172.21.15.50:0/598771576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd85480 0x56538dd84f40 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fcda0011fb0 tx=0x7fcda0005bd0 comp rx=0 tx=0).stop 2024-04-23T18:36:19.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.676+0000 7fcdb2982740 1 -- 172.21.15.50:0/598771576 shutdown_connections 2024-04-23T18:36:19.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.676+0000 7fcdb2982740 1 --2- 172.21.15.50:0/598771576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd85480 0x56538dd84f40 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:19.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.676+0000 7fcdb2982740 1 --2- 172.21.15.50:0/598771576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd84a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:19.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.676+0000 7fcdb2982740 1 --2- 172.21.15.50:0/598771576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd7a490 0x56538dd91110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:19.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.676+0000 7fcdb2982740 1 -- 172.21.15.50:0/598771576 >> 172.21.15.50:0/598771576 conn(0x56538dd767b0 msgr2=0x56538dd7bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.677+0000 7fcdb2982740 1 -- 172.21.15.50:0/598771576 shutdown_connections 2024-04-23T18:36:19.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.677+0000 7fcdb2982740 1 -- 172.21.15.50:0/598771576 wait complete. 2024-04-23T18:36:19.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.677+0000 7fcdb2982740 1 -- start start 2024-04-23T18:36:19.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.677+0000 7fcdb2982740 1 Processor -- start 2024-04-23T18:36:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.678+0000 7fcdb2982740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd8d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:19.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.678+0000 7fcdb2982740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd8db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:19.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.678+0000 7fcdb2982740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd885e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:19.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.678+0000 7fcdb2982740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81a9fc0 con 0x56538dd7a490 2024-04-23T18:36:19.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.678+0000 7fcdb2982740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81ab430 con 0x56538dd7adb0 2024-04-23T18:36:19.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.678+0000 7fcdb2982740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81aa690 con 0x56538dd85480 2024-04-23T18:36:19.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.678+0000 7fcdb2982740 10 client.0 _ll_register_callbacks cb 0x7fcd98032bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:19.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.678+0000 7fcdae211640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd885e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:19.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.678+0000 7fcdae211640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd885e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:41186/0 (socket says 172.21.15.50:41186) 2024-04-23T18:36:19.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.678+0000 7fcdae211640 1 -- 172.21.15.50:0/2836646058 learned_addr learned my addr 172.21.15.50:0/2836646058 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:20.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.678+0000 7fcdad20f640 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd8db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:20.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.678+0000 7fcdada10640 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd8d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:20.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.679+0000 7fcdae211640 1 -- 172.21.15.50:0/2836646058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 msgr2=0x56538dd8db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:20.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.679+0000 7fcdae211640 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd8db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:20.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.679+0000 7fcdae211640 1 -- 172.21.15.50:0/2836646058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 msgr2=0x56538dd8d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:20.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.679+0000 7fcdae211640 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd8d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:20.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.679+0000 7fcdae211640 1 -- 172.21.15.50:0/2836646058 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56538dd88e10 con 0x56538dd85480 2024-04-23T18:36:20.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.679+0000 7fcdad20f640 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd8db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:36:20.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.679+0000 7fcdae211640 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd885e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcda0013010 tx=0x7fcda0011b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ba23662feb9c635 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:20.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.679+0000 7fcd917fa640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:36:20.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.679+0000 7fcd917fa640 1 -- 172.21.15.50:0/2836646058 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda00045b0 con 0x56538dd85480 2024-04-23T18:36:20.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.679+0000 7fcdb2982740 1 -- 172.21.15.50:0/2836646058 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcda000fcc0 con 0x56538dd85480 2024-04-23T18:36:20.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.679+0000 7fcd917fa640 1 -- 172.21.15.50:0/2836646058 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8542+0+0 (secure 0 0 0) 0x7fcda0029660 con 0x56538dd85480 2024-04-23T18:36:20.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.680+0000 7fcd917fa640 1 -- 172.21.15.50:0/2836646058 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fcda0029860 con 0x56538dd85480 2024-04-23T18:36:20.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.680+0000 7fcdb2982740 10 client.27220 fetch_fsmap learned FSMap version 58 2024-04-23T18:36:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.680+0000 7fcdb2982740 1 -- 172.21.15.50:0/2836646058 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x56538de04da0 con 0x56538dd85480 2024-04-23T18:36:20.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.680+0000 7fcd917fa640 1 -- 172.21.15.50:0/2836646058 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcda000fcc0 con 0x56538dd85480 2024-04-23T18:36:20.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.680+0000 7fcd917fa640 1 -- 172.21.15.50:0/2836646058 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 58) ==== 8346+0+0 (secure 0 0 0) 0x7fcda0031070 con 0x56538dd85480 2024-04-23T18:36:20.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.680+0000 7fcdb2982740 10 client.27220 fetch_fsmap finished waiting for FSMap version 58 2024-04-23T18:36:20.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.680+0000 7fcdb2982740 10 client.27220 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:36:20.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.680+0000 7fcdb2982740 20 client.27220 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.680+0000 7fcdb2982740 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56538de05420 0x56538de07840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.680+0000 7fcdb2982740 4 client.27220 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "7fa60ad4", "release": true, "await": true}] 2024-04-23T18:36:20.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.680+0000 7fcdb2982740 1 -- 172.21.15.50:0/2836646058 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "7fa60ad4", "release": true, "await": true}) -- 0x56538dced0f0 con 0x56538de05420 2024-04-23T18:36:20.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.681+0000 7fcdada10640 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56538de05420 0x56538de07840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:20.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.681+0000 7fcdada10640 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56538de05420 0x56538de07840 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1cfbca2939661a4 server_cookie=81d7309426121782 in_seq=0 out_seq=0 2024-04-23T18:36:20.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.681+0000 7fcd917fa640 10 client.27220 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:20.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.712+0000 7fcd917fa640 1 -- 172.21.15.50:0/2836646058 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56538dced0f0 con 0x56538de05420 2024-04-23T18:36:20.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.712+0000 7fcd917fa640 10 client.27220 handle_command_reply: tid=0 2024-04-23T18:36:20.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.712+0000 7fcdb2982740 1 client.27220 shutdown 2024-04-23T18:36:20.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.712+0000 7fcdb2982740 1 -- 172.21.15.50:0/2836646058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 msgr2=0x56538dd885e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:20.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.712+0000 7fcdb2982740 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd885e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcda0013010 tx=0x7fcda0011b00 comp rx=0 tx=0).stop 2024-04-23T18:36:20.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdb2982740 1 -- 172.21.15.50:0/2836646058 shutdown_connections 2024-04-23T18:36:20.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdb2982740 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x56538de05420 0x56538de07840 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:20.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdb2982740 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56538dd85480 0x56538dd885e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:20.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdb2982740 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56538dd7adb0 0x56538dd8db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:20.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcd917fa640 0 client.27220 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:20.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdb2982740 1 --2- 172.21.15.50:0/2836646058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56538dd7a490 0x56538dd8d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:20.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdb2982740 1 -- 172.21.15.50:0/2836646058 >> 172.21.15.50:0/2836646058 conn(0x56538dd767b0 msgr2=0x56538dd7c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:20.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdb2982740 1 -- 172.21.15.50:0/2836646058 shutdown_connections 2024-04-23T18:36:20.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdb2982740 1 -- 172.21.15.50:0/2836646058 wait complete. 2024-04-23T18:36:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdb2982740 20 client.27220 trim_cache size 0 max 16384 2024-04-23T18:36:20.975 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:36:20.982 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 58, 2024-04-23T18:36:21.029 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:36:21.036 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 60, 2024-04-23T18:36:21.070 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:36:21.135 INFO:teuthology.orchestra.run.smithi050.stdout: "7fa60ad4": { 2024-04-23T18:36:21.193 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 60, 2024-04-23T18:36:21.205 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:36:21.264 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:36:21.276 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:36:21.287 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:36:21.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:36:21.310 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:36:21.319 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 79.7, 2024-04-23T18:36:21.327 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:36:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:36:21.347 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:36:21.356 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:36:21.365 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:36:21.407 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:36:21.419 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:36:21.487 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:36:21.543 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:36:21.608 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:36:21.617 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:36:21.658 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:36:21.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:16 smithi188 ceph-mon[32694]: pgmap v564: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 5.3 MiB/s wr, 189 op/s 2024-04-23T18:36:21.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:18 smithi188 ceph-mon[32694]: pgmap v565: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 253 op/s 2024-04-23T18:36:21.689 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:20 smithi188 ceph-mon[32694]: pgmap v566: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 253 op/s 2024-04-23T18:36:21.699 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:16 smithi177 ceph-mon[32308]: pgmap v564: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 5.3 MiB/s wr, 189 op/s 2024-04-23T18:36:21.708 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:18 smithi177 ceph-mon[32308]: pgmap v565: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 253 op/s 2024-04-23T18:36:21.718 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:20 smithi177 ceph-mon[32308]: pgmap v566: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 253 op/s 2024-04-23T18:36:21.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdafc74640 1 -- 172.21.15.50:0/3427435357 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcd940a3d20 msgr2=0x7fcd940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:21.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdafc74640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcd940a3d20 0x7fcd940a61e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcda0005c40 tx=0x7fcda0009500 comp rx=0 tx=0).stop 2024-04-23T18:36:21.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdafc74640 1 -- 172.21.15.50:0/3427435357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda8100d20 msgr2=0x7fcda8198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:21.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.713+0000 7fcdafc74640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda8100d20 0x7fcda8198720 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fcda400d950 tx=0x7fcda400de10 comp rx=0 tx=0).stop 2024-04-23T18:36:21.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.714+0000 7fcdafc74640 1 -- 172.21.15.50:0/3427435357 shutdown_connections 2024-04-23T18:36:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.714+0000 7fcdafc74640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fcd940a3d20 0x7fcd940a61e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:21.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.714+0000 7fcdafc74640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcda819cfe0 0x7fcda819d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.714+0000 7fcdafc74640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcda810c4f0 0x7fcda8198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:21.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.714+0000 7fcdafc74640 1 --2- 172.21.15.50:0/3427435357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda8100d20 0x7fcda8198720 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:22.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.714+0000 7fcdafc74640 1 -- 172.21.15.50:0/3427435357 >> 172.21.15.50:0/3427435357 conn(0x7fcda8076150 msgr2=0x7fcda80fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:22.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.714+0000 7fcdafc74640 1 -- 172.21.15.50:0/3427435357 shutdown_connections 2024-04-23T18:36:22.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:15.714+0000 7fcdafc74640 1 -- 172.21.15.50:0/3427435357 wait complete. 2024-04-23T18:36:22.060 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:16 smithi050 ceph-mon[29187]: pgmap v564: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 5.3 MiB/s wr, 189 op/s 2024-04-23T18:36:22.069 INFO:tasks.workunit.client.0.smithi050.stdout: 10 122188 22.15 MB/sec execute 170 sec latency 27642.153 ms 2024-04-23T18:36:22.079 INFO:tasks.workunit.client.0.smithi050.stdout: 10 122616 22.12 MB/sec execute 171 sec latency 155.127 ms 2024-04-23T18:36:22.089 INFO:tasks.workunit.client.0.smithi050.stdout: 10 123092 22.13 MB/sec execute 172 sec latency 119.867 ms 2024-04-23T18:36:22.098 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '7fa60ad4', seconds elapsed: 34.6 2024-04-23T18:36:22.107 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.7 2024-04-23T18:36:22.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:36:22.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:18 smithi050 ceph-mon[29187]: pgmap v565: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 253 op/s 2024-04-23T18:36:22.147 INFO:tasks.workunit.client.0.smithi050.stdout: 10 123636 22.24 MB/sec execute 173 sec latency 124.330 ms 2024-04-23T18:36:22.187 INFO:tasks.workunit.client.0.smithi050.stdout: 10 124143 22.25 MB/sec execute 174 sec latency 103.936 ms 2024-04-23T18:36:22.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:20 smithi050 ceph-mon[29187]: pgmap v566: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 253 op/s 2024-04-23T18:36:22.257 INFO:tasks.workunit.client.0.smithi050.stdout: 10 124564 22.23 MB/sec execute 175 sec latency 140.771 ms 2024-04-23T18:36:22.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.055+0000 7f73fd594640 1 -- 172.21.15.50:0/3418671732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f81021d0 msgr2=0x7f73f81025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.055+0000 7f73fd594640 1 --2- 172.21.15.50:0/3418671732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f81021d0 0x7f73f81025b0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009900 tx=0x7f73e802a410 comp rx=0 tx=0).stop 2024-04-23T18:36:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.055+0000 7f73fd594640 1 -- 172.21.15.50:0/3418671732 shutdown_connections 2024-04-23T18:36:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.055+0000 7f73fd594640 1 --2- 172.21.15.50:0/3418671732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73f81072a0 0x7f73f810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.055+0000 7f73fd594640 1 --2- 172.21.15.50:0/3418671732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8102b80 0x7f73f8106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:22.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.055+0000 7f73fd594640 1 --2- 172.21.15.50:0/3418671732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f81021d0 0x7f73f81025b0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.055+0000 7f73fd594640 1 -- 172.21.15.50:0/3418671732 >> 172.21.15.50:0/3418671732 conn(0x7f73f80793e0 msgr2=0x7f73f80797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.055+0000 7f73fd594640 1 -- 172.21.15.50:0/3418671732 shutdown_connections 2024-04-23T18:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.055+0000 7f73fd594640 1 -- 172.21.15.50:0/3418671732 wait complete. 2024-04-23T18:36:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73fd594640 1 Processor -- start 2024-04-23T18:36:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73fd594640 1 -- start start 2024-04-23T18:36:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73fd594640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f81021d0 0x7f73f8074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73fd594640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73f81072a0 0x7f73f8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:22.526 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73fd594640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8074cd0 0x7f73f81a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73fd594640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f73f8112230 con 0x7f73f81021d0 2024-04-23T18:36:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73fd594640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81120b0 con 0x7f73f81072a0 2024-04-23T18:36:22.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73fd594640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81123b0 con 0x7f73f8074cd0 2024-04-23T18:36:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73f6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f81021d0 0x7f73f8074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73f6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f81021d0 0x7f73f8074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58166/0 (socket says 172.21.15.50:58166) 2024-04-23T18:36:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73f6ffd640 1 -- 172.21.15.50:0/1740675921 learned_addr learned my addr 172.21.15.50:0/1740675921 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73effff640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73f81072a0 0x7f73f8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.056+0000 7f73f77fe640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8074cd0 0x7f73f81a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:22.870 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.057+0000 7f73f6ffd640 1 -- 172.21.15.50:0/1740675921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8074cd0 msgr2=0x7f73f81a6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:22.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.057+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8074cd0 0x7f73f81a6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.057+0000 7f73f6ffd640 1 -- 172.21.15.50:0/1740675921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73f81072a0 msgr2=0x7f73f8074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.057+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73f81072a0 0x7f73f8074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:22.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.057+0000 7f73f6ffd640 1 -- 172.21.15.50:0/1740675921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f81a6f60 con 0x7f73f81021d0 2024-04-23T18:36:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.057+0000 7f73f77fe640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8074cd0 0x7f73f81a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:36:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.057+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f81021d0 0x7f73f8074250 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f73e8005ab0 tx=0x7f73e8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.057+0000 7f73f4ff9640 1 -- 172.21.15.50:0/1740675921 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e803f070 con 0x7f73f81021d0 2024-04-23T18:36:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.057+0000 7f73f4ff9640 1 -- 172.21.15.50:0/1740675921 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e8031b60 con 0x7f73f81021d0 2024-04-23T18:36:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.057+0000 7f73fd594640 1 -- 172.21.15.50:0/1740675921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f81a71f0 con 0x7f73f81021d0 2024-04-23T18:36:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.057+0000 7f73fd594640 1 -- 172.21.15.50:0/1740675921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f81a76e0 con 0x7f73f81021d0 2024-04-23T18:36:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.057+0000 7f73f4ff9640 1 -- 172.21.15.50:0/1740675921 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8035030 con 0x7f73f81021d0 2024-04-23T18:36:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.059+0000 7f73f4ff9640 1 -- 172.21.15.50:0/1740675921 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f73e8042070 con 0x7f73f81021d0 2024-04-23T18:36:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.060+0000 7f7400388740 1 Processor -- start 2024-04-23T18:36:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.060+0000 7f7400388740 1 -- start start 2024-04-23T18:36:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.060+0000 7f73f4ff9640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f73e40a3df0 0x7f73e40a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:23.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.060+0000 7f73f77fe640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f73e40a3df0 0x7f73e40a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.060+0000 7f73f4ff9640 1 -- 172.21.15.50:0/1740675921 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8542+0+0 (secure 0 0 0) 0x7f73e80f0ed0 con 0x7f73f81021d0 2024-04-23T18:36:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.060+0000 7f7400388740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 0x5579b313da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:23.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.060+0000 7f7400388740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313df40 0x5579b31482c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.060+0000 7f7400388740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 0x5579b314abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.060+0000 7f7400388740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5579b30a2970 con 0x5579b313d620 2024-04-23T18:36:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.060+0000 7f7400388740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5579b303f4a0 con 0x5579b313df40 2024-04-23T18:36:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.060+0000 7f7400388740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5579b309e930 con 0x5579b3148800 2024-04-23T18:36:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73f6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 0x5579b313da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73f77fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 0x5579b314abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:23.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73f6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 0x5579b313da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58172/0 (socket says 172.21.15.50:58172) 2024-04-23T18:36:23.367 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73f6ffd640 1 -- 172.21.15.50:0/43570070 learned_addr learned my addr 172.21.15.50:0/43570070 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73effff640 1 --2- 172.21.15.50:0/43570070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313df40 0x5579b31482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73f6ffd640 1 -- 172.21.15.50:0/43570070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 msgr2=0x5579b314abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/43570070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 0x5579b314abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73f6ffd640 1 -- 172.21.15.50:0/43570070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313df40 msgr2=0x5579b31482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/43570070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313df40 0x5579b31482c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73f6ffd640 1 -- 172.21.15.50:0/43570070 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579b2db5210 con 0x5579b313d620 2024-04-23T18:36:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73f77fe640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f73e40a3df0 0x7f73e40a62b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f73f0005fd0 tx=0x7f73f0011f20 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:23.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/43570070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 0x5579b313da00 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f73e80bcea0 tx=0x7f73e8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71bfccaa072de410 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:23.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73ed7fa640 1 -- 172.21.15.50:0/43570070 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e80d6070 con 0x5579b313d620 2024-04-23T18:36:23.634 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73ed7fa640 1 -- 172.21.15.50:0/43570070 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e80c5ad0 con 0x5579b313d620 2024-04-23T18:36:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.061+0000 7f73ed7fa640 1 -- 172.21.15.50:0/43570070 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8049070 con 0x5579b313d620 2024-04-23T18:36:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.062+0000 7f7400388740 1 -- 172.21.15.50:0/43570070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 msgr2=0x5579b313da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.062+0000 7f7400388740 1 --2- 172.21.15.50:0/43570070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 0x5579b313da00 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f73e80bcea0 tx=0x7f73e8009260 comp rx=0 tx=0).stop 2024-04-23T18:36:23.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.062+0000 7f7400388740 1 -- 172.21.15.50:0/43570070 shutdown_connections 2024-04-23T18:36:23.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.062+0000 7f7400388740 1 --2- 172.21.15.50:0/43570070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 0x5579b314abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.062+0000 7f7400388740 1 --2- 172.21.15.50:0/43570070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313df40 0x5579b31482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.062+0000 7f7400388740 1 --2- 172.21.15.50:0/43570070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 0x5579b313da00 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:23.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.062+0000 7f7400388740 1 -- 172.21.15.50:0/43570070 >> 172.21.15.50:0/43570070 conn(0x5579b3139b30 msgr2=0x5579b3139f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.062+0000 7f7400388740 1 -- 172.21.15.50:0/43570070 shutdown_connections 2024-04-23T18:36:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.062+0000 7f7400388740 1 -- 172.21.15.50:0/43570070 wait complete. 2024-04-23T18:36:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.062+0000 7f7400388740 1 -- start start 2024-04-23T18:36:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f7400388740 1 Processor -- start 2024-04-23T18:36:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f7400388740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313d620 0x5579b3148150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:24.011 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f7400388740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 0x5579b3145440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:24.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f7400388740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 0x5579b31459a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f7400388740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5579b314b1f0 con 0x5579b313df40 2024-04-23T18:36:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f7400388740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5579b303d0b0 con 0x5579b313d620 2024-04-23T18:36:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f7400388740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5579b314b370 con 0x5579b3148800 2024-04-23T18:36:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f7400388740 10 client.0 _ll_register_callbacks cb 0x5579b302ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:24.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73effff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 0x5579b3145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73effff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 0x5579b3145440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58188/0 (socket says 172.21.15.50:58188) 2024-04-23T18:36:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73effff640 1 -- 172.21.15.50:0/4203787221 learned_addr learned my addr 172.21.15.50:0/4203787221 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:24.127 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313d620 0x5579b3148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:24.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73f77fe640 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 0x5579b31459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:24.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73effff640 1 -- 172.21.15.50:0/4203787221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 msgr2=0x5579b31459a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73effff640 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 0x5579b31459a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:24.300 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73effff640 1 -- 172.21.15.50:0/4203787221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313d620 msgr2=0x5579b3148150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73effff640 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313d620 0x5579b3148150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73effff640 1 -- 172.21.15.50:0/4203787221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579b2e81620 con 0x5579b313df40 2024-04-23T18:36:24.364 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313d620 0x5579b3148150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:36:24.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73effff640 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 0x5579b3145440 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f73e00126e0 tx=0x7f73e0012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c167f1ce1c5c10 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.063+0000 7f73de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:36:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f73de7fc640 1 -- 172.21.15.50:0/4203787221 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e000f670 con 0x5579b313df40 2024-04-23T18:36:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f73de7fc640 1 -- 172.21.15.50:0/4203787221 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8542+0+0 (secure 0 0 0) 0x7f73e000f810 con 0x5579b313df40 2024-04-23T18:36:24.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f7400388740 1 -- 172.21.15.50:0/4203787221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73e80c5ad0 con 0x5579b313df40 2024-04-23T18:36:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f73de7fc640 1 -- 172.21.15.50:0/4203787221 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f73e000fa10 con 0x5579b313df40 2024-04-23T18:36:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f7400388740 10 client.17685 fetch_fsmap learned FSMap version 58 2024-04-23T18:36:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f7400388740 1 -- 172.21.15.50:0/4203787221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5579b31570f0 con 0x5579b313df40 2024-04-23T18:36:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f73de7fc640 1 -- 172.21.15.50:0/4203787221 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e80c5ad0 con 0x5579b313df40 2024-04-23T18:36:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f73de7fc640 1 -- 172.21.15.50:0/4203787221 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 58) ==== 8346+0+0 (secure 0 0 0) 0x7f73e0010bb0 con 0x5579b313df40 2024-04-23T18:36:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f7400388740 10 client.17685 fetch_fsmap finished waiting for FSMap version 58 2024-04-23T18:36:24.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f7400388740 10 client.17685 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:36:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f7400388740 20 client.17685 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f7400388740 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5579b3158670 0x5579b315aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f7400388740 4 client.17685 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:36:24.681 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f7400388740 1 -- 172.21.15.50:0/4203787221 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579b30b0390 con 0x5579b3158670 2024-04-23T18:36:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.064+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5579b3158670 0x5579b315aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.065+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5579b3158670 0x5579b315aa80 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1060b2943793fdd server_cookie=92762fd5d0a6b7a3 in_seq=0 out_seq=0 2024-04-23T18:36:24.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.065+0000 7f73de7fc640 10 client.17685 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f73de7fc640 1 -- 172.21.15.50:0/4203787221 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5579b30b0390 con 0x5579b3158670 2024-04-23T18:36:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f73de7fc640 10 client.17685 handle_command_reply: tid=0 2024-04-23T18:36:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f7400388740 1 client.17685 shutdown 2024-04-23T18:36:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f7400388740 1 -- 172.21.15.50:0/4203787221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 msgr2=0x5579b3145440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f7400388740 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 0x5579b3145440 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f73e00126e0 tx=0x7f73e0012ae0 comp rx=0 tx=0).stop 2024-04-23T18:36:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f7400388740 1 -- 172.21.15.50:0/4203787221 shutdown_connections 2024-04-23T18:36:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f7400388740 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5579b3158670 0x5579b315aa80 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f7400388740 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 0x5579b31459a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f7400388740 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 0x5579b3145440 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f7400388740 1 --2- 172.21.15.50:0/4203787221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313d620 0x5579b3148150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f73de7fc640 0 client.17685 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f7400388740 1 -- 172.21.15.50:0/4203787221 >> 172.21.15.50:0/4203787221 conn(0x5579b3139b30 msgr2=0x5579b313f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f7400388740 1 -- 172.21.15.50:0/4203787221 shutdown_connections 2024-04-23T18:36:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f7400388740 1 -- 172.21.15.50:0/4203787221 wait complete. 2024-04-23T18:36:25.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.066+0000 7f7400388740 20 client.17685 trim_cache size 0 max 16384 2024-04-23T18:36:25.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f7400388740 1 Processor -- start 2024-04-23T18:36:25.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f7400388740 1 -- start start 2024-04-23T18:36:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f7400388740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b313d620 0x5579b3153f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f7400388740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 0x5579b3160390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f7400388740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b3148800 0x5579b31608d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:25.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f7400388740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81b2680 con 0x5579b313df40 2024-04-23T18:36:25.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f7400388740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81aff20 con 0x5579b3148800 2024-04-23T18:36:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f7400388740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81add90 con 0x5579b313d620 2024-04-23T18:36:25.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f73effff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 0x5579b3160390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f73effff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 0x5579b3160390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58194/0 (socket says 172.21.15.50:58194) 2024-04-23T18:36:25.208 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f73effff640 1 -- 172.21.15.50:0/491749101 learned_addr learned my addr 172.21.15.50:0/491749101 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:25.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/491749101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b313d620 0x5579b3153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.083+0000 7f73f77fe640 1 --2- 172.21.15.50:0/491749101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b3148800 0x5579b31608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.084+0000 7f73effff640 1 -- 172.21.15.50:0/491749101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b313d620 msgr2=0x5579b3153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.084+0000 7f73effff640 1 --2- 172.21.15.50:0/491749101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b313d620 0x5579b3153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.084+0000 7f73effff640 1 -- 172.21.15.50:0/491749101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b3148800 msgr2=0x5579b31608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.084+0000 7f73effff640 1 --2- 172.21.15.50:0/491749101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b3148800 0x5579b31608d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.084+0000 7f73effff640 1 -- 172.21.15.50:0/491749101 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579b31443f0 con 0x5579b313df40 2024-04-23T18:36:25.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.084+0000 7f73effff640 1 --2- 172.21.15.50:0/491749101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 0x5579b3160390 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f73e0012420 tx=0x7f73e000f930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874a100c01a11dc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.084+0000 7f73ecff9640 1 -- 172.21.15.50:0/491749101 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e001c6b0 con 0x5579b313df40 2024-04-23T18:36:25.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.084+0000 7f73ecff9640 1 -- 172.21.15.50:0/491749101 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e001c850 con 0x5579b313df40 2024-04-23T18:36:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.084+0000 7f73ecff9640 1 -- 172.21.15.50:0/491749101 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e0005550 con 0x5579b313df40 2024-04-23T18:36:25.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.085+0000 7f7400388740 1 -- 172.21.15.50:0/491749101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 msgr2=0x5579b3160390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.085+0000 7f7400388740 1 --2- 172.21.15.50:0/491749101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 0x5579b3160390 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f73e0012420 tx=0x7f73e000f930 comp rx=0 tx=0).stop 2024-04-23T18:36:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.085+0000 7f7400388740 1 -- 172.21.15.50:0/491749101 shutdown_connections 2024-04-23T18:36:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.085+0000 7f7400388740 1 --2- 172.21.15.50:0/491749101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b3148800 0x5579b31608d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:26.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.085+0000 7f7400388740 1 --2- 172.21.15.50:0/491749101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313df40 0x5579b3160390 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.085+0000 7f7400388740 1 --2- 172.21.15.50:0/491749101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b313d620 0x5579b3153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.085+0000 7f7400388740 1 -- 172.21.15.50:0/491749101 >> 172.21.15.50:0/491749101 conn(0x5579b3139b30 msgr2=0x5579b3138cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:26.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.085+0000 7f7400388740 1 -- 172.21.15.50:0/491749101 shutdown_connections 2024-04-23T18:36:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.085+0000 7f7400388740 1 -- 172.21.15.50:0/491749101 wait complete. 2024-04-23T18:36:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.085+0000 7f7400388740 1 -- start start 2024-04-23T18:36:26.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f7400388740 1 Processor -- start 2024-04-23T18:36:26.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f7400388740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 0x5579b314e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:26.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f7400388740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313df40 0x5579b314e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f7400388740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 0x5579b314eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:26.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f7400388740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81ae3e0 con 0x5579b313d620 2024-04-23T18:36:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f7400388740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81af850 con 0x5579b313df40 2024-04-23T18:36:26.319 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f7400388740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81aeab0 con 0x5579b3148800 2024-04-23T18:36:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f7400388740 10 client.0 _ll_register_callbacks cb 0x7f73e0009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73f6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 0x5579b314e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73f6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 0x5579b314e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58210/0 (socket says 172.21.15.50:58210) 2024-04-23T18:36:26.400 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73f6ffd640 1 -- 172.21.15.50:0/2409898265 learned_addr learned my addr 172.21.15.50:0/2409898265 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73f77fe640 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 0x5579b314eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:26.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73effff640 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313df40 0x5579b314e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73f6ffd640 1 -- 172.21.15.50:0/2409898265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 msgr2=0x5579b314eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 0x5579b314eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:26.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73f6ffd640 1 -- 172.21.15.50:0/2409898265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313df40 msgr2=0x5579b314e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313df40 0x5579b314e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73f6ffd640 1 -- 172.21.15.50:0/2409898265 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579b313dd70 con 0x5579b313d620 2024-04-23T18:36:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73f6ffd640 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 0x5579b314e020 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f73e80ee6c0 tx=0x7f73e8031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf9884806defda1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:36:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.086+0000 7f73de7fc640 1 -- 172.21.15.50:0/2409898265 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8049040 con 0x5579b313d620 2024-04-23T18:36:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f7400388740 1 -- 172.21.15.50:0/2409898265 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73e001c850 con 0x5579b313d620 2024-04-23T18:36:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f73de7fc640 1 -- 172.21.15.50:0/2409898265 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8542+0+0 (secure 0 0 0) 0x7f73e80ee890 con 0x5579b313d620 2024-04-23T18:36:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f73de7fc640 1 -- 172.21.15.50:0/2409898265 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f73e80d6040 con 0x5579b313d620 2024-04-23T18:36:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f7400388740 10 client.17691 fetch_fsmap learned FSMap version 58 2024-04-23T18:36:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f7400388740 1 -- 172.21.15.50:0/2409898265 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5579b31bee30 con 0x5579b313d620 2024-04-23T18:36:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f73de7fc640 1 -- 172.21.15.50:0/2409898265 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e001c850 con 0x5579b313d620 2024-04-23T18:36:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f73de7fc640 1 -- 172.21.15.50:0/2409898265 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 58) ==== 8346+0+0 (secure 0 0 0) 0x7f73e803b460 con 0x5579b313d620 2024-04-23T18:36:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f7400388740 10 client.17691 fetch_fsmap finished waiting for FSMap version 58 2024-04-23T18:36:26.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f7400388740 10 client.17691 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:36:27.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f7400388740 20 client.17691 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f7400388740 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5579b31bf3c0 0x5579b31c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f7400388740 4 client.17691 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:36:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f7400388740 1 -- 172.21.15.50:0/2409898265 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5579b30aa200 con 0x5579b31bf3c0 2024-04-23T18:36:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.087+0000 7f73effff640 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5579b31bf3c0 0x5579b31c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.088+0000 7f73effff640 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5579b31bf3c0 0x5579b31c17e0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b38631509009381b server_cookie=4179fffedb4484d3 in_seq=0 out_seq=0 2024-04-23T18:36:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.088+0000 7f73de7fc640 10 client.17691 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f73de7fc640 1 -- 172.21.15.50:0/2409898265 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5579b30aa200 con 0x5579b31bf3c0 2024-04-23T18:36:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f73de7fc640 10 client.17691 handle_command_reply: tid=0 2024-04-23T18:36:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f7400388740 1 client.17691 shutdown 2024-04-23T18:36:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f7400388740 1 -- 172.21.15.50:0/2409898265 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 msgr2=0x5579b314e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f7400388740 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 0x5579b314e020 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f73e80ee6c0 tx=0x7f73e8031400 comp rx=0 tx=0).stop 2024-04-23T18:36:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f7400388740 1 -- 172.21.15.50:0/2409898265 shutdown_connections 2024-04-23T18:36:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f7400388740 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5579b31bf3c0 0x5579b31c17e0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f7400388740 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5579b3148800 0x5579b314eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f7400388740 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5579b313df40 0x5579b314e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f73de7fc640 0 client.17691 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:27.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f7400388740 1 --2- 172.21.15.50:0/2409898265 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5579b313d620 0x5579b314e020 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f7400388740 1 -- 172.21.15.50:0/2409898265 >> 172.21.15.50:0/2409898265 conn(0x5579b3139b30 msgr2=0x5579b3152f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:27.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f7400388740 1 -- 172.21.15.50:0/2409898265 shutdown_connections 2024-04-23T18:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f7400388740 1 -- 172.21.15.50:0/2409898265 wait complete. 2024-04-23T18:36:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.246+0000 7f7400388740 20 client.17691 trim_cache size 0 max 16384 2024-04-23T18:36:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:36:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:36:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:36:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:36:27.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.247+0000 7f73fd594640 1 -- 172.21.15.50:0/1740675921 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f73e40a3df0 msgr2=0x7f73e40a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.247+0000 7f73fd594640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f73e40a3df0 0x7f73e40a62b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f73f0005fd0 tx=0x7f73f0011f20 comp rx=0 tx=0).stop 2024-04-23T18:36:27.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.247+0000 7f73fd594640 1 -- 172.21.15.50:0/1740675921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f81021d0 msgr2=0x7f73f8074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.247+0000 7f73fd594640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f81021d0 0x7f73f8074250 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f73e8005ab0 tx=0x7f73e8003c10 comp rx=0 tx=0).stop 2024-04-23T18:36:27.631 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.247+0000 7f73fd594640 1 -- 172.21.15.50:0/1740675921 shutdown_connections 2024-04-23T18:36:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.247+0000 7f73fd594640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f73e40a3df0 0x7f73e40a62b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.247+0000 7f73fd594640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73f8074cd0 0x7f73f81a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.247+0000 7f73fd594640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73f81072a0 0x7f73f8074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.247+0000 7f73fd594640 1 --2- 172.21.15.50:0/1740675921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f81021d0 0x7f73f8074250 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.247+0000 7f73fd594640 1 -- 172.21.15.50:0/1740675921 >> 172.21.15.50:0/1740675921 conn(0x7f73f80793e0 msgr2=0x7f73f80ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.247+0000 7f73fd594640 1 -- 172.21.15.50:0/1740675921 shutdown_connections 2024-04-23T18:36:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:22.247+0000 7f73fd594640 1 -- 172.21.15.50:0/1740675921 wait complete. 2024-04-23T18:36:27.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:22 smithi177 ceph-mon[32308]: pgmap v567: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 253 op/s 2024-04-23T18:36:27.951 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:36:22 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: ignoring --setuser ceph since I am not root 2024-04-23T18:36:27.963 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:36:22 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: ignoring --setgroup ceph since I am not root 2024-04-23T18:36:27.972 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:36:22 smithi177 ceph-mds[66745]: -- 172.21.15.177:0/977097045 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e8ca605a40 con 0x55e8ca5d9000 2024-04-23T18:36:27.979 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:36:22 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: starting mds.h at 2024-04-23T18:36:27.988 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:22 smithi188 ceph-mon[32694]: pgmap v567: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 253 op/s 2024-04-23T18:36:27.994 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:23 smithi188 ceph-mon[32694]: daemon mds.h restarted 2024-04-23T18:36:28.003 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:23 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3501841921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:36:28.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:24 smithi188 ceph-mon[32694]: pgmap v568: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 9.0 MiB/s wr, 532 op/s 2024-04-23T18:36:28.020 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:24 smithi188 ceph-mon[32694]: osdmap e120: 12 total, 12 up, 12 in 2024-04-23T18:36:28.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:24 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6836/2754597719,v1:172.21.15.177:6837/2754597719] up:boot 2024-04-23T18:36:28.090 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:24 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:36:28.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:24 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:36:28.234 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:24 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:28.315 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:25 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:28.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:26 smithi188 ceph-mon[32694]: pgmap v570: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.0 MiB/s wr, 506 op/s 2024-04-23T18:36:28.374 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:23 smithi177 ceph-mon[32308]: daemon mds.h restarted 2024-04-23T18:36:28.381 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:23 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3501841921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:36:28.392 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:24 smithi177 ceph-mon[32308]: pgmap v568: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 9.0 MiB/s wr, 532 op/s 2024-04-23T18:36:28.402 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:24 smithi177 ceph-mon[32308]: osdmap e120: 12 total, 12 up, 12 in 2024-04-23T18:36:28.410 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:24 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6836/2754597719,v1:172.21.15.177:6837/2754597719] up:boot 2024-04-23T18:36:28.416 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:24 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:36:28.423 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:24 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:36:28.429 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:24 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:28.436 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:25 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:28.442 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:26 smithi177 ceph-mon[32308]: pgmap v570: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.0 MiB/s wr, 506 op/s 2024-04-23T18:36:28.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.448+0000 7fa9418fa640 1 -- 172.21.15.50:0/2030975606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa93c1021a0 msgr2=0x7fa93c102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:28.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.448+0000 7fa9418fa640 1 --2- 172.21.15.50:0/2030975606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa93c1021a0 0x7fa93c102580 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009900 tx=0x7fa92c02a410 comp rx=0 tx=0).stop 2024-04-23T18:36:28.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.448+0000 7fa9418fa640 1 -- 172.21.15.50:0/2030975606 shutdown_connections 2024-04-23T18:36:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.448+0000 7fa9418fa640 1 --2- 172.21.15.50:0/2030975606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa93c107270 0x7fa93c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:28.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.448+0000 7fa9418fa640 1 --2- 172.21.15.50:0/2030975606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93c102b50 0x7fa93c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:28.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.448+0000 7fa9418fa640 1 --2- 172.21.15.50:0/2030975606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa93c1021a0 0x7fa93c102580 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:28.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.448+0000 7fa9418fa640 1 -- 172.21.15.50:0/2030975606 >> 172.21.15.50:0/2030975606 conn(0x7fa93c079350 msgr2=0x7fa93c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:28.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.448+0000 7fa9418fa640 1 -- 172.21.15.50:0/2030975606 shutdown_connections 2024-04-23T18:36:28.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.448+0000 7fa9418fa640 1 -- 172.21.15.50:0/2030975606 wait complete. 2024-04-23T18:36:28.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.449+0000 7fa9418fa640 1 Processor -- start 2024-04-23T18:36:28.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.449+0000 7fa9418fa640 1 -- start start 2024-04-23T18:36:28.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.449+0000 7fa9418fa640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa93c1021a0 0x7fa93c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:28.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.449+0000 7fa9418fa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa93c107270 0x7fa93c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:28.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.449+0000 7fa9418fa640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93c1985b0 0x7fa93c198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:28.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.449+0000 7fa9418fa640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c1121d0 con 0x7fa93c1021a0 2024-04-23T18:36:28.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.449+0000 7fa9418fa640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c112050 con 0x7fa93c107270 2024-04-23T18:36:28.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.449+0000 7fa9418fa640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c112350 con 0x7fa93c1985b0 2024-04-23T18:36:28.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa93affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa93c1021a0 0x7fa93c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:28.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa93affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa93c1021a0 0x7fa93c19e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58224/0 (socket says 172.21.15.50:58224) 2024-04-23T18:36:28.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa93affd640 1 -- 172.21.15.50:0/3501841921 learned_addr learned my addr 172.21.15.50:0/3501841921 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa93a7fc640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa93c107270 0x7fa93c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:29.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa93b7fe640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93c1985b0 0x7fa93c198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:29.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa93affd640 1 -- 172.21.15.50:0/3501841921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93c1985b0 msgr2=0x7fa93c198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:29.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa93affd640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93c1985b0 0x7fa93c198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa93affd640 1 -- 172.21.15.50:0/3501841921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa93c107270 msgr2=0x7fa93c19e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa93affd640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa93c107270 0x7fa93c19e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa93affd640 1 -- 172.21.15.50:0/3501841921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa93c1990d0 con 0x7fa93c1021a0 2024-04-23T18:36:29.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa93a7fc640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa93c107270 0x7fa93c19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:36:29.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa93affd640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa93c1021a0 0x7fa93c19e400 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fa92c005940 tx=0x7fa92c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:29.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa9408f8640 1 -- 172.21.15.50:0/3501841921 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c03f030 con 0x7fa93c1021a0 2024-04-23T18:36:29.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa9408f8640 1 -- 172.21.15.50:0/3501841921 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa92c0044c0 con 0x7fa93c1021a0 2024-04-23T18:36:29.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa9408f8640 1 -- 172.21.15.50:0/3501841921 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c035070 con 0x7fa93c1021a0 2024-04-23T18:36:29.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa9418fa640 1 -- 172.21.15.50:0/3501841921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa93c199360 con 0x7fa93c1021a0 2024-04-23T18:36:29.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.450+0000 7fa9418fa640 1 -- 172.21.15.50:0/3501841921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa93c1a5550 con 0x7fa93c1021a0 2024-04-23T18:36:29.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.451+0000 7fa9418fa640 1 -- 172.21.15.50:0/3501841921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa93c068d70 con 0x7fa93c1021a0 2024-04-23T18:36:29.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.453+0000 7fa9408f8640 1 -- 172.21.15.50:0/3501841921 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa92c042030 con 0x7fa93c1021a0 2024-04-23T18:36:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.454+0000 7fa9408f8640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa9240a3d20 0x7fa9240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:29.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.454+0000 7fa9408f8640 1 -- 172.21.15.50:0/3501841921 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8542+0+0 (secure 0 0 0) 0x7fa92c0f0b50 con 0x7fa93c1021a0 2024-04-23T18:36:29.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.454+0000 7fa93a7fc640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa9240a3d20 0x7fa9240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.454+0000 7fa93a7fc640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa9240a3d20 0x7fa9240a61e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa93c101b70 tx=0x7fa930005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:29.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.459+0000 7fa9408f8640 1 -- 172.21.15.50:0/3501841921 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa92c0bcb90 con 0x7fa93c1021a0 2024-04-23T18:36:29.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:22 smithi050 ceph-mon[29187]: pgmap v567: 97 pgs: 97 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 253 op/s 2024-04-23T18:36:29.602 INFO:tasks.workunit.client.0.smithi050.stdout: 10 125036 22.26 MB/sec execute 176 sec latency 107.682 ms 2024-04-23T18:36:29.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.761+0000 7fa9418fa640 1 -- 172.21.15.50:0/3501841921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa93c199ae0 con 0x7fa93c1021a0 2024-04-23T18:36:29.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.762+0000 7fa9408f8640 1 -- 172.21.15.50:0/3501841921 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+11877 (secure 0 0 0) 0x7fa92c0c0e90 con 0x7fa93c1021a0 2024-04-23T18:36:29.623 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:36:29.630 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":58,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:36:13.866479+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27059":{"gid":27059,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/3902222090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3902222090},{"type":"v1","addr":"172.21.15.177:6837","nonce":3902222090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:36:29.643 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:28 smithi188 ceph-mon[32694]: pgmap v571: 97 pgs: 97 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.1 MiB/s wr, 644 op/s 2024-04-23T18:36:29.650 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:28 smithi177 ceph-mon[32308]: pgmap v571: 97 pgs: 97 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.1 MiB/s wr, 644 op/s 2024-04-23T18:36:29.659 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 58 2024-04-23T18:36:29.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.764+0000 7fa9418fa640 1 -- 172.21.15.50:0/3501841921 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa9240a3d20 msgr2=0x7fa9240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:29.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.764+0000 7fa9418fa640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa9240a3d20 0x7fa9240a61e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa93c101b70 tx=0x7fa930005f90 comp rx=0 tx=0).stop 2024-04-23T18:36:29.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.764+0000 7fa9418fa640 1 -- 172.21.15.50:0/3501841921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa93c1021a0 msgr2=0x7fa93c19e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:29.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.764+0000 7fa9418fa640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa93c1021a0 0x7fa93c19e400 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fa92c005940 tx=0x7fa92c0040c0 comp rx=0 tx=0).stop 2024-04-23T18:36:29.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.764+0000 7fa9418fa640 1 -- 172.21.15.50:0/3501841921 shutdown_connections 2024-04-23T18:36:29.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.764+0000 7fa9418fa640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa9240a3d20 0x7fa9240a61e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:29.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.764+0000 7fa9418fa640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa93c1985b0 0x7fa93c198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:29.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.764+0000 7fa9418fa640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa93c107270 0x7fa93c19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:29.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.764+0000 7fa9418fa640 1 --2- 172.21.15.50:0/3501841921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa93c1021a0 0x7fa93c19e400 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:29.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.764+0000 7fa9418fa640 1 -- 172.21.15.50:0/3501841921 >> 172.21.15.50:0/3501841921 conn(0x7fa93c079350 msgr2=0x7fa93c100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:29.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.764+0000 7fa9418fa640 1 -- 172.21.15.50:0/3501841921 shutdown_connections 2024-04-23T18:36:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:22.764+0000 7fa9418fa640 1 -- 172.21.15.50:0/3501841921 wait complete. 2024-04-23T18:36:30.017 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 79.7 --await-for 42.0' 2024-04-23T18:36:30.026 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 79.7 --await-for 42.0 2024-04-23T18:36:30.034 INFO:tasks.workunit.client.0.smithi050.stdout: 10 125631 22.33 MB/sec execute 177 sec latency 144.303 ms 2024-04-23T18:36:30.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:23 smithi050 ceph-mon[29187]: daemon mds.h restarted 2024-04-23T18:36:30.064 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:23 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3501841921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:36:30.074 INFO:tasks.workunit.client.0.smithi050.stdout: 10 126134 22.34 MB/sec execute 178 sec latency 86.851 ms 2024-04-23T18:36:30.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:24 smithi050 ceph-mon[29187]: pgmap v568: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 9.0 MiB/s wr, 532 op/s 2024-04-23T18:36:30.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:24 smithi050 ceph-mon[29187]: osdmap e120: 12 total, 12 up, 12 in 2024-04-23T18:36:30.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:24 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6836/2754597719,v1:172.21.15.177:6837/2754597719] up:boot 2024-04-23T18:36:30.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:24 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T18:36:30.292 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:24 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:36:30.344 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:24 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:30.358 INFO:tasks.workunit.client.0.smithi050.stdout: 10 126786 22.42 MB/sec execute 179 sec latency 81.688 ms 2024-04-23T18:36:30.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:25 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:30.485 INFO:tasks.workunit.client.0.smithi050.stdout: 10 127325 22.46 MB/sec execute 180 sec latency 84.803 ms 2024-04-23T18:36:30.493 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:26 smithi050 ceph-mon[29187]: pgmap v570: 97 pgs: 97 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.0 MiB/s wr, 506 op/s 2024-04-23T18:36:30.502 INFO:tasks.workunit.client.0.smithi050.stdout: 10 127863 22.49 MB/sec execute 181 sec latency 95.197 ms 2024-04-23T18:36:30.511 INFO:tasks.workunit.client.0.smithi050.stdout: 10 128378 22.53 MB/sec execute 182 sec latency 104.235 ms 2024-04-23T18:36:30.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:28 smithi050 ceph-mon[29187]: pgmap v571: 97 pgs: 97 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.1 MiB/s wr, 644 op/s 2024-04-23T18:36:30.528 INFO:tasks.workunit.client.0.smithi050.stdout: 10 128872 22.51 MB/sec execute 183 sec latency 112.220 ms 2024-04-23T18:36:30.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:30 smithi177 ceph-mon[32308]: pgmap v572: 97 pgs: 97 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.1 MiB/s wr, 644 op/s 2024-04-23T18:36:30.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:30 smithi050 ceph-mon[29187]: pgmap v572: 97 pgs: 97 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.1 MiB/s wr, 644 op/s 2024-04-23T18:36:30.573 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129494 22.58 MB/sec execute 184 sec latency 100.473 ms 2024-04-23T18:36:30.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:30 smithi188 ceph-mon[32694]: pgmap v572: 97 pgs: 97 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.1 MiB/s wr, 644 op/s 2024-04-23T18:36:30.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.678+0000 7f700758e640 1 -- 172.21.15.50:0/1358440330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70001031e0 msgr2=0x7f70001035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:30.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.678+0000 7f700758e640 1 --2- 172.21.15.50:0/1358440330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70001031e0 0x7f70001035c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc009900 tx=0x7f6ffc02a510 comp rx=0 tx=0).stop 2024-04-23T18:36:30.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.678+0000 7f700758e640 1 -- 172.21.15.50:0/1358440330 shutdown_connections 2024-04-23T18:36:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.678+0000 7f700758e640 1 --2- 172.21.15.50:0/1358440330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f700010c520 0x7f700010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:30.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.678+0000 7f700758e640 1 --2- 172.21.15.50:0/1358440330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7000103b00 0x7f700010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:30.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.678+0000 7f700758e640 1 --2- 172.21.15.50:0/1358440330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70001031e0 0x7f70001035c0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:30.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.678+0000 7f700758e640 1 -- 172.21.15.50:0/1358440330 >> 172.21.15.50:0/1358440330 conn(0x7f7000077f50 msgr2=0x7f7000078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:30.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.678+0000 7f700758e640 1 -- 172.21.15.50:0/1358440330 shutdown_connections 2024-04-23T18:36:30.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.678+0000 7f700758e640 1 -- 172.21.15.50:0/1358440330 wait complete. 2024-04-23T18:36:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f700758e640 1 Processor -- start 2024-04-23T18:36:30.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f700758e640 1 -- start start 2024-04-23T18:36:30.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f700758e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70001031e0 0x7f7000100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:30.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f700758e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7000103b00 0x7f7000101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:30.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f700758e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700010c520 0x7f70001018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:30.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f700758e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7000112190 con 0x7f700010c520 2024-04-23T18:36:30.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f700758e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7000112010 con 0x7f7000103b00 2024-04-23T18:36:30.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f700758e640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7000112310 con 0x7f70001031e0 2024-04-23T18:36:31.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f7005b2b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700010c520 0x7f70001018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:31.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f7005b2b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700010c520 0x7f70001018b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58256/0 (socket says 172.21.15.50:58256) 2024-04-23T18:36:31.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f7005b2b640 1 -- 172.21.15.50:0/1548693519 learned_addr learned my addr 172.21.15.50:0/1548693519 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:31.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f700532a640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70001031e0 0x7f7000100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:31.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.679+0000 7f7004b29640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7000103b00 0x7f7000101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:31.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.680+0000 7f7005b2b640 1 -- 172.21.15.50:0/1548693519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70001031e0 msgr2=0x7f7000100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:31.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.680+0000 7f7005b2b640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70001031e0 0x7f7000100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:31.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.680+0000 7f7005b2b640 1 -- 172.21.15.50:0/1548693519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7000103b00 msgr2=0x7f7000101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:31.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.680+0000 7f7005b2b640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7000103b00 0x7f7000101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:31.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.680+0000 7f7005b2b640 1 -- 172.21.15.50:0/1548693519 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70001a6c30 con 0x7f700010c520 2024-04-23T18:36:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.680+0000 7f7005b2b640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700010c520 0x7f70001018b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800eb10 tx=0x7f6ff800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.680+0000 7f6ff67fc640 1 -- 172.21.15.50:0/1548693519 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8004b20 con 0x7f700010c520 2024-04-23T18:36:31.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.680+0000 7f6ff67fc640 1 -- 172.21.15.50:0/1548693519 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ff8004cc0 con 0x7f700010c520 2024-04-23T18:36:31.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.680+0000 7f700758e640 1 -- 172.21.15.50:0/1548693519 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70001a6f20 con 0x7f700010c520 2024-04-23T18:36:31.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.680+0000 7f6ff67fc640 1 -- 172.21.15.50:0/1548693519 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8010670 con 0x7f700010c520 2024-04-23T18:36:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.680+0000 7f700758e640 1 -- 172.21.15.50:0/1548693519 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70001a73f0 con 0x7f700010c520 2024-04-23T18:36:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.683+0000 7f6ff67fc640 1 -- 172.21.15.50:0/1548693519 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6ff8005020 con 0x7f700010c520 2024-04-23T18:36:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.683+0000 7f700a37c740 1 Processor -- start 2024-04-23T18:36:31.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.683+0000 7f700a37c740 1 -- start start 2024-04-23T18:36:31.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700a37c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a7c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:31.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700a37c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a86b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700a37c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6a89480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:31.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700a37c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a0c69e1370 con 0x55a0c6a87090 2024-04-23T18:36:31.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700a37c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a0c697e2e0 con 0x55a0c6a7c9c0 2024-04-23T18:36:31.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700a37c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a0c69dd330 con 0x55a0c6a7c0a0 2024-04-23T18:36:31.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f6ff67fc640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6fec0a3d20 0x7f6fec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:31.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f7004b29640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a7c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:31.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700532a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a86b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:31.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f7004b29640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a7c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:56390/0 (socket says 172.21.15.50:56390) 2024-04-23T18:36:31.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f7004b29640 1 -- 172.21.15.50:0/4190664884 learned_addr learned my addr 172.21.15.50:0/4190664884 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:31.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f7004b29640 1 --2- 172.21.15.50:0/4190664884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6a89480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:31.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f7005b2b640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6fec0a3d20 0x7f6fec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:31.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f6ff67fc640 1 -- 172.21.15.50:0/1548693519 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7f6ff80d0cd0 con 0x7f700010c520 2024-04-23T18:36:31.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700532a640 1 -- 172.21.15.50:0/4190664884 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c0a0 msgr2=0x55a0c6a7c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:31.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700532a640 1 --2- 172.21.15.50:0/4190664884 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a7c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:31.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700532a640 1 -- 172.21.15.50:0/4190664884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 msgr2=0x55a0c6a89480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:31.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700532a640 1 --2- 172.21.15.50:0/4190664884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6a89480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:31.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700532a640 1 -- 172.21.15.50:0/4190664884 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0c67bfd80 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f7005b2b640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6fec0a3d20 0x7f6fec0a61e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80ce4e0 tx=0x7f6ff801bb30 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:32.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.684+0000 7f700532a640 1 --2- 172.21.15.50:0/4190664884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a86b50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc009230 tx=0x7f6ffc032bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=904162232820a6a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.685+0000 7f6feb7fe640 1 -- 172.21.15.50:0/4190664884 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc040070 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.685+0000 7f6feb7fe640 1 -- 172.21.15.50:0/4190664884 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ffc004030 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.685+0000 7f6feb7fe640 1 -- 172.21.15.50:0/4190664884 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc035040 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.686+0000 7f700a37c740 1 -- 172.21.15.50:0/4190664884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c9c0 msgr2=0x55a0c6a86b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:32.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.686+0000 7f700a37c740 1 --2- 172.21.15.50:0/4190664884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a86b50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc009230 tx=0x7f6ffc032bd0 comp rx=0 tx=0).stop 2024-04-23T18:36:32.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.686+0000 7f700a37c740 1 -- 172.21.15.50:0/4190664884 shutdown_connections 2024-04-23T18:36:32.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.686+0000 7f700a37c740 1 --2- 172.21.15.50:0/4190664884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6a89480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:32.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.686+0000 7f700a37c740 1 --2- 172.21.15.50:0/4190664884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a86b50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:32.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.686+0000 7f700a37c740 1 --2- 172.21.15.50:0/4190664884 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a7c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:32.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.686+0000 7f700a37c740 1 -- 172.21.15.50:0/4190664884 >> 172.21.15.50:0/4190664884 conn(0x55a0c6a783c0 msgr2=0x55a0c6a787a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:32.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.686+0000 7f700a37c740 1 -- 172.21.15.50:0/4190664884 shutdown_connections 2024-04-23T18:36:32.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.686+0000 7f700a37c740 1 -- 172.21.15.50:0/4190664884 wait complete. 2024-04-23T18:36:32.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.686+0000 7f700a37c740 1 -- start start 2024-04-23T18:36:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.686+0000 7f700a37c740 1 Processor -- start 2024-04-23T18:36:32.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.687+0000 7f700a37c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a84050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:32.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.687+0000 7f700a37c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a84590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:32.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.687+0000 7f700a37c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6a84ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:32.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.687+0000 7f7004b29640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a84050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:32.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.687+0000 7f7004b29640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a84050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:47738/0 (socket says 172.21.15.50:47738) 2024-04-23T18:36:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.687+0000 7f7004b29640 1 -- 172.21.15.50:0/3091298268 learned_addr learned my addr 172.21.15.50:0/3091298268 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:32.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.687+0000 7f700a37c740 1 -- 172.21.15.50:0/3091298268 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a0c6a89e20 con 0x55a0c6a87090 2024-04-23T18:36:32.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.687+0000 7f700532a640 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a84590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:32.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.687+0000 7f700a37c740 1 -- 172.21.15.50:0/3091298268 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a0c697bfb0 con 0x55a0c6a7c0a0 2024-04-23T18:36:32.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.687+0000 7f700a37c740 1 -- 172.21.15.50:0/3091298268 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a0c6a89fa0 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.687+0000 7f7004b29640 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6a84ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:32.478 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.688+0000 7f700a37c740 10 client.0 _ll_register_callbacks cb 0x55a0c697b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:32.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.688+0000 7f700532a640 1 -- 172.21.15.50:0/3091298268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c0a0 msgr2=0x55a0c6a84050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:32.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.688+0000 7f700532a640 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a84050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:32.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.688+0000 7f700532a640 1 -- 172.21.15.50:0/3091298268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 msgr2=0x55a0c6a84ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:32.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.688+0000 7f700532a640 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6a84ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:32.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.688+0000 7f700532a640 1 -- 172.21.15.50:0/3091298268 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0c6820510 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.688+0000 7f7004b29640 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6a84ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:36:32.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.688+0000 7f700532a640 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a84590 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x55a0c6a7d660 tx=0x7f6ffc009590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43267e2b9bb2068d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:32.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.688+0000 7f6fe8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:36:32.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.688+0000 7f6fe8ff9640 1 -- 172.21.15.50:0/3091298268 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc040070 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.688+0000 7f6fe8ff9640 1 -- 172.21.15.50:0/3091298268 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7f6ffc0316a0 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.688+0000 7f700a37c740 1 -- 172.21.15.50:0/3091298268 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ffc004030 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.689+0000 7f6fe8ff9640 1 -- 172.21.15.50:0/3091298268 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f6ffc035040 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.689+0000 7f700a37c740 10 client.27259 fetch_fsmap learned FSMap version 62 2024-04-23T18:36:32.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.689+0000 7f700a37c740 1 -- 172.21.15.50:0/3091298268 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55a0c66b3670 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.689+0000 7f6fe8ff9640 1 -- 172.21.15.50:0/3091298268 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ffc004030 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.689+0000 7f6fe8ff9640 1 -- 172.21.15.50:0/3091298268 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f6ffc004a90 con 0x55a0c6a7c9c0 2024-04-23T18:36:32.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.690+0000 7f700a37c740 10 client.27259 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:36:32.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.690+0000 7f700a37c740 10 client.27259 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:36:32.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.690+0000 7f700a37c740 20 client.27259 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:32.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.690+0000 7f700a37c740 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0c6a97560 0x55a0c6a99a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:32.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.690+0000 7f700a37c740 4 client.27259 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:36:32.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.690+0000 7f700a37c740 1 -- 172.21.15.50:0/3091298268 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0c69eed90 con 0x55a0c6a97560 2024-04-23T18:36:32.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.690+0000 7f7004b29640 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0c6a97560 0x55a0c6a99a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:33.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.691+0000 7f7004b29640 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0c6a97560 0x55a0c6a99a00 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bec3a152401caa37 server_cookie=5a3f4b4179cfa89a in_seq=0 out_seq=0 2024-04-23T18:36:33.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.691+0000 7f6fe8ff9640 10 client.27259 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:33.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f6fe8ff9640 1 -- 172.21.15.50:0/3091298268 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a0c69eed90 con 0x55a0c6a97560 2024-04-23T18:36:33.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f6fe8ff9640 10 client.27259 handle_command_reply: tid=0 2024-04-23T18:36:33.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f700a37c740 1 client.27259 shutdown 2024-04-23T18:36:33.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f700a37c740 1 -- 172.21.15.50:0/3091298268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c9c0 msgr2=0x55a0c6a84590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:33.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f700a37c740 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a84590 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x55a0c6a7d660 tx=0x7f6ffc009590 comp rx=0 tx=0).stop 2024-04-23T18:36:33.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f700a37c740 1 -- 172.21.15.50:0/3091298268 shutdown_connections 2024-04-23T18:36:33.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f700a37c740 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0c6a97560 0x55a0c6a99a00 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:33.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f700a37c740 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6a84ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:33.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f700a37c740 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a84590 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:33.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f700a37c740 1 --2- 172.21.15.50:0/3091298268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a84050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:33.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f700a37c740 1 -- 172.21.15.50:0/3091298268 >> 172.21.15.50:0/3091298268 conn(0x55a0c6a783c0 msgr2=0x55a0c6a7e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:33.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f6fe8ff9640 0 client.27259 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:33.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f700a37c740 1 -- 172.21.15.50:0/3091298268 shutdown_connections 2024-04-23T18:36:33.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.692+0000 7f700a37c740 1 -- 172.21.15.50:0/3091298268 wait complete. 2024-04-23T18:36:33.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.693+0000 7f700a37c740 20 client.27259 trim_cache size 0 max 16384 2024-04-23T18:36:33.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f700a37c740 1 Processor -- start 2024-04-23T18:36:33.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f700a37c740 1 -- start start 2024-04-23T18:36:33.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f700a37c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a7e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f700a37c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a77480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:33.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f700a37c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a87090 0x55a0c6a812b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:33.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f700a37c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f70001b25e0 con 0x55a0c6a7c0a0 2024-04-23T18:36:33.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f700a37c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f70001afeb0 con 0x55a0c6a7c9c0 2024-04-23T18:36:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f700a37c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f70001add00 con 0x55a0c6a87090 2024-04-23T18:36:33.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f7004b29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a7e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:33.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f700532a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a77480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:33.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f7004b29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a7e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58294/0 (socket says 172.21.15.50:58294) 2024-04-23T18:36:33.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f7004b29640 1 -- 172.21.15.50:0/1719366012 learned_addr learned my addr 172.21.15.50:0/1719366012 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:33.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f7004b29640 1 --2- 172.21.15.50:0/1719366012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a87090 0x55a0c6a812b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:33.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f7004b29640 1 -- 172.21.15.50:0/1719366012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a87090 msgr2=0x55a0c6a812b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:33.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f7004b29640 1 --2- 172.21.15.50:0/1719366012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a87090 0x55a0c6a812b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:33.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f7004b29640 1 -- 172.21.15.50:0/1719366012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c9c0 msgr2=0x55a0c6a77480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:33.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f7004b29640 1 --2- 172.21.15.50:0/1719366012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a77480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f7004b29640 1 -- 172.21.15.50:0/1719366012 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0c6a825a0 con 0x55a0c6a7c0a0 2024-04-23T18:36:33.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f700532a640 1 --2- 172.21.15.50:0/1719366012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a77480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:36:33.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.713+0000 7f7004b29640 1 --2- 172.21.15.50:0/1719366012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a7e910 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000b750 tx=0x7f6ff000b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a7679674695457 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:33.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.714+0000 7f6feaffd640 1 -- 172.21.15.50:0/1719366012 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0010df0 con 0x55a0c6a7c0a0 2024-04-23T18:36:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.714+0000 7f6feaffd640 1 -- 172.21.15.50:0/1719366012 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ff0024040 con 0x55a0c6a7c0a0 2024-04-23T18:36:33.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.714+0000 7f6feaffd640 1 -- 172.21.15.50:0/1719366012 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff00226a0 con 0x55a0c6a7c0a0 2024-04-23T18:36:33.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.714+0000 7f700a37c740 1 -- 172.21.15.50:0/1719366012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a7c0a0 msgr2=0x55a0c6a7e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:33.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.714+0000 7f700a37c740 1 --2- 172.21.15.50:0/1719366012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a7e910 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000b750 tx=0x7f6ff000b410 comp rx=0 tx=0).stop 2024-04-23T18:36:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.714+0000 7f700a37c740 1 -- 172.21.15.50:0/1719366012 shutdown_connections 2024-04-23T18:36:33.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.714+0000 7f700a37c740 1 --2- 172.21.15.50:0/1719366012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a87090 0x55a0c6a812b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:33.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.714+0000 7f700a37c740 1 --2- 172.21.15.50:0/1719366012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a77480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:33.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.714+0000 7f700a37c740 1 --2- 172.21.15.50:0/1719366012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a7e910 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:33.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.714+0000 7f700a37c740 1 -- 172.21.15.50:0/1719366012 >> 172.21.15.50:0/1719366012 conn(0x55a0c6a783c0 msgr2=0x55a0c6a76d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:33.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.714+0000 7f700a37c740 1 -- 172.21.15.50:0/1719366012 shutdown_connections 2024-04-23T18:36:33.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f700a37c740 1 -- 172.21.15.50:0/1719366012 wait complete. 2024-04-23T18:36:34.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f700a37c740 1 -- start start 2024-04-23T18:36:34.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f700a37c740 1 Processor -- start 2024-04-23T18:36:34.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f700a37c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a9d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:34.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f700a37c740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a9d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f700a37c740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6afca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:34.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f700a37c740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f70001ae370 con 0x55a0c6a87090 2024-04-23T18:36:34.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f700a37c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f70001af7e0 con 0x55a0c6a7c0a0 2024-04-23T18:36:34.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f700a37c740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f70001aea40 con 0x55a0c6a7c9c0 2024-04-23T18:36:34.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f700a37c740 10 client.0 _ll_register_callbacks cb 0x7f6ffc032b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f7004b29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6afca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:34.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f7004b29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6afca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58296/0 (socket says 172.21.15.50:58296) 2024-04-23T18:36:34.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f700532a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a9d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:34.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f7004b29640 1 -- 172.21.15.50:0/4203138159 learned_addr learned my addr 172.21.15.50:0/4203138159 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:34.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.715+0000 7f7004b29640 1 --2- 172.21.15.50:0/4203138159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a9d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:34.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f7004b29640 1 -- 172.21.15.50:0/4203138159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c9c0 msgr2=0x55a0c6a9d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:34.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f7004b29640 1 --2- 172.21.15.50:0/4203138159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a9d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:34.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f7004b29640 1 -- 172.21.15.50:0/4203138159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c0a0 msgr2=0x55a0c6a9d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:34.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f7004b29640 1 --2- 172.21.15.50:0/4203138159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a9d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:34.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f7004b29640 1 -- 172.21.15.50:0/4203138159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0c6afcfa0 con 0x55a0c6a87090 2024-04-23T18:36:34.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f7004b29640 1 --2- 172.21.15.50:0/4203138159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6afca30 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0011010 tx=0x7f6ff0024940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e16bc20ccd763679 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:34.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f6fe8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:36:34.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f6fe8ff9640 1 -- 172.21.15.50:0/4203138159 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0012e30 con 0x55a0c6a87090 2024-04-23T18:36:34.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f6fe8ff9640 1 -- 172.21.15.50:0/4203138159 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7f6ff0004020 con 0x55a0c6a87090 2024-04-23T18:36:34.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f700a37c740 1 -- 172.21.15.50:0/4203138159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ff0024040 con 0x55a0c6a87090 2024-04-23T18:36:34.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f6fe8ff9640 1 -- 172.21.15.50:0/4203138159 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f6ff0004220 con 0x55a0c6a87090 2024-04-23T18:36:34.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f700a37c740 10 client.17718 fetch_fsmap learned FSMap version 62 2024-04-23T18:36:34.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f700a37c740 1 -- 172.21.15.50:0/4203138159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55a0c6afe280 con 0x55a0c6a87090 2024-04-23T18:36:34.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.716+0000 7f6fe8ff9640 1 -- 172.21.15.50:0/4203138159 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ff0024040 con 0x55a0c6a87090 2024-04-23T18:36:34.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.717+0000 7f6fe8ff9640 1 -- 172.21.15.50:0/4203138159 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f6ff0003990 con 0x55a0c6a87090 2024-04-23T18:36:34.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.717+0000 7f700a37c740 10 client.17718 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:36:34.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.717+0000 7f700a37c740 10 client.17718 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:36:34.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.717+0000 7f700a37c740 20 client.17718 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:34.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.717+0000 7f700a37c740 1 --2- 172.21.15.50:0/4203138159 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0c6afe940 0x55a0c6b00d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:34.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.717+0000 7f700a37c740 4 client.17718 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.7, "await_for": 42.0}] 2024-04-23T18:36:34.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.717+0000 7f700a37c740 1 -- 172.21.15.50:0/4203138159 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.7, "await_for": 42.0}) -- 0x55a0c69e8c00 con 0x55a0c6afe940 2024-04-23T18:36:34.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.717+0000 7f700532a640 1 --2- 172.21.15.50:0/4203138159 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0c6afe940 0x55a0c6b00d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:34.897 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.717+0000 7f6fe8ff9640 10 client.17718 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:34.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:30.717+0000 7f700532a640 1 --2- 172.21.15.50:0/4203138159 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0c6afe940 0x55a0c6b00d30 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20ef85fb96cd1248 server_cookie=d8f1a6595dfd5164 in_seq=0 out_seq=0 2024-04-23T18:36:34.953 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:31 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:34.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:32 smithi188 ceph-mon[32694]: pgmap v573: 97 pgs: 97 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.1 MiB/s wr, 644 op/s 2024-04-23T18:36:34.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:34 smithi188 ceph-mon[32694]: pgmap v574: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.4 MiB/s wr, 494 op/s 2024-04-23T18:36:34.978 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:31 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:34.985 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:32 smithi177 ceph-mon[32308]: pgmap v573: 97 pgs: 97 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.1 MiB/s wr, 644 op/s 2024-04-23T18:36:34.991 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:34 smithi177 ceph-mon[32308]: pgmap v574: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.4 MiB/s wr, 494 op/s 2024-04-23T18:36:34.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f6fe8ff9640 1 -- 172.21.15.50:0/4203138159 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55a0c69e8c00 con 0x55a0c6afe940 2024-04-23T18:36:35.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f6fe8ff9640 10 client.17718 handle_command_reply: tid=0 2024-04-23T18:36:35.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f700a37c740 1 client.17718 shutdown 2024-04-23T18:36:35.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f700a37c740 1 -- 172.21.15.50:0/4203138159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 msgr2=0x55a0c6afca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:35.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f700a37c740 1 --2- 172.21.15.50:0/4203138159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6afca30 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0011010 tx=0x7f6ff0024940 comp rx=0 tx=0).stop 2024-04-23T18:36:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f700a37c740 1 -- 172.21.15.50:0/4203138159 shutdown_connections 2024-04-23T18:36:35.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f700a37c740 1 --2- 172.21.15.50:0/4203138159 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a0c6afe940 0x55a0c6b00d30 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:35.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f700a37c740 1 --2- 172.21.15.50:0/4203138159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a0c6a87090 0x55a0c6afca30 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:35.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f700a37c740 1 --2- 172.21.15.50:0/4203138159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a0c6a7c9c0 0x55a0c6a9d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:35.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f6fe8ff9640 0 client.17718 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:35.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f700a37c740 1 --2- 172.21.15.50:0/4203138159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0c6a7c0a0 0x55a0c6a9d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:35.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f700a37c740 1 -- 172.21.15.50:0/4203138159 >> 172.21.15.50:0/4203138159 conn(0x55a0c6a783c0 msgr2=0x55a0c6a77600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:35.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f700a37c740 1 -- 172.21.15.50:0/4203138159 shutdown_connections 2024-04-23T18:36:35.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f700a37c740 1 -- 172.21.15.50:0/4203138159 wait complete. 2024-04-23T18:36:35.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.293+0000 7f700a37c740 20 client.17718 trim_cache size 0 max 16384 2024-04-23T18:36:35.322 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:36:35.331 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 62, 2024-04-23T18:36:35.341 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:36:35.350 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 62, 2024-04-23T18:36:35.360 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:36:35.370 INFO:teuthology.orchestra.run.smithi050.stdout: "c3d7e5f6": { 2024-04-23T18:36:35.388 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 62, 2024-04-23T18:36:35.433 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:36:35.441 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:36:35.503 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:36:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:36:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:36:35.625 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:36:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 79.7, 2024-04-23T18:36:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:36:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:36:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:36:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:36:35.704 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:36:35.711 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:36:35.718 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:36:35.724 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:36:35.731 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:36:35.739 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:36:35.778 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:36:35.795 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:36:35.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.294+0000 7f700758e640 1 -- 172.21.15.50:0/1548693519 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6fec0a3d20 msgr2=0x7f6fec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:35.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.294+0000 7f700758e640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6fec0a3d20 0x7f6fec0a61e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80ce4e0 tx=0x7f6ff801bb30 comp rx=0 tx=0).stop 2024-04-23T18:36:35.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.294+0000 7f700758e640 1 -- 172.21.15.50:0/1548693519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700010c520 msgr2=0x7f70001018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.294+0000 7f700758e640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700010c520 0x7f70001018b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800eb10 tx=0x7f6ff800c2d0 comp rx=0 tx=0).stop 2024-04-23T18:36:36.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.294+0000 7f700758e640 1 -- 172.21.15.50:0/1548693519 shutdown_connections 2024-04-23T18:36:36.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.294+0000 7f700758e640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f6fec0a3d20 0x7f6fec0a61e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:36.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.294+0000 7f700758e640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700010c520 0x7f70001018b0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:36.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.294+0000 7f700758e640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7000103b00 0x7f7000101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:36.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.294+0000 7f700758e640 1 --2- 172.21.15.50:0/1548693519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70001031e0 0x7f7000100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:36.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.294+0000 7f700758e640 1 -- 172.21.15.50:0/1548693519 >> 172.21.15.50:0/1548693519 conn(0x7f7000077f50 msgr2=0x7f70000fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:36.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.294+0000 7f700758e640 1 -- 172.21.15.50:0/1548693519 shutdown_connections 2024-04-23T18:36:36.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:31.294+0000 7f700758e640 1 -- 172.21.15.50:0/1548693519 wait complete. 2024-04-23T18:36:36.267 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 22.51 MB/sec execute 185 sec latency 649.950 ms 2024-04-23T18:36:36.276 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:31 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:36:36.285 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 22.39 MB/sec execute 186 sec latency 1650.073 ms 2024-04-23T18:36:36.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:32 smithi050 ceph-mon[29187]: pgmap v573: 97 pgs: 97 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.1 MiB/s wr, 644 op/s 2024-04-23T18:36:36.326 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 22.27 MB/sec execute 187 sec latency 2650.190 ms 2024-04-23T18:36:36.386 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 22.15 MB/sec execute 188 sec latency 3650.298 ms 2024-04-23T18:36:36.426 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:34 smithi050 ceph-mon[29187]: pgmap v574: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.4 MiB/s wr, 494 op/s 2024-04-23T18:36:36.488 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c3d7e5f6', quiesce took 14.4 seconds. Will release after: 19.7 2024-04-23T18:36:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.346+0000 7f303d50f640 1 -- 172.21.15.50:0/617052225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3038107290 msgr2=0x7f303810ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.346+0000 7f303d50f640 1 --2- 172.21.15.50:0/617052225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3038107290 0x7f303810ee60 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f3030009ca0 tx=0x7f303002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:36:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.347+0000 7f303d50f640 1 -- 172.21.15.50:0/617052225 shutdown_connections 2024-04-23T18:36:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.347+0000 7f303d50f640 1 --2- 172.21.15.50:0/617052225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3038107290 0x7f303810ee60 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.347+0000 7f303d50f640 1 --2- 172.21.15.50:0/617052225 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3038102b70 0x7f3038106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.347+0000 7f303d50f640 1 --2- 172.21.15.50:0/617052225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30381021c0 0x7f30381025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.347+0000 7f303d50f640 1 -- 172.21.15.50:0/617052225 >> 172.21.15.50:0/617052225 conn(0x7f30380fda20 msgr2=0x7f30380ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.347+0000 7f303d50f640 1 -- 172.21.15.50:0/617052225 shutdown_connections 2024-04-23T18:36:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.347+0000 7f303d50f640 1 -- 172.21.15.50:0/617052225 wait complete. 2024-04-23T18:36:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.347+0000 7f303d50f640 1 Processor -- start 2024-04-23T18:36:36.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f303d50f640 1 -- start start 2024-04-23T18:36:36.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f303d50f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30381021c0 0x7f3038198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f303d50f640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3038107290 0x7f3038198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:36.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f303d50f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f303819cfb0 0x7f303819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f303d50f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3038112200 con 0x7f30381021c0 2024-04-23T18:36:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f303d50f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3038112080 con 0x7f303819cfb0 2024-04-23T18:36:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f303d50f640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3038112380 con 0x7f3038107290 2024-04-23T18:36:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f3036ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30381021c0 0x7f3038198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:36.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f3036ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30381021c0 0x7f3038198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39134/0 (socket says 172.21.15.50:39134) 2024-04-23T18:36:36.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f3036ffd640 1 -- 172.21.15.50:0/3278524235 learned_addr learned my addr 172.21.15.50:0/3278524235 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f30367fc640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3038107290 0x7f3038198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f30377fe640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f303819cfb0 0x7f303819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:36.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f3036ffd640 1 -- 172.21.15.50:0/3278524235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3038107290 msgr2=0x7f3038198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:36.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f3036ffd640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3038107290 0x7f3038198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:36.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f3036ffd640 1 -- 172.21.15.50:0/3278524235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f303819cfb0 msgr2=0x7f303819d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:36.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f3036ffd640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f303819cfb0 0x7f303819d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.348+0000 7f3036ffd640 1 -- 172.21.15.50:0/3278524235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303819daa0 con 0x7f30381021c0 2024-04-23T18:36:36.921 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.349+0000 7f3036ffd640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30381021c0 0x7f3038198740 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f302800d950 tx=0x7f302800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.349+0000 7f301ffff640 1 -- 172.21.15.50:0/3278524235 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3028017070 con 0x7f30381021c0 2024-04-23T18:36:36.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.349+0000 7f303d50f640 1 -- 172.21.15.50:0/3278524235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f303819dd90 con 0x7f30381021c0 2024-04-23T18:36:36.994 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.349+0000 7f301ffff640 1 -- 172.21.15.50:0/3278524235 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3028009c30 con 0x7f30381021c0 2024-04-23T18:36:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.349+0000 7f301ffff640 1 -- 172.21.15.50:0/3278524235 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302800f620 con 0x7f30381021c0 2024-04-23T18:36:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.349+0000 7f303d50f640 1 -- 172.21.15.50:0/3278524235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30381a55d0 con 0x7f30381021c0 2024-04-23T18:36:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.351+0000 7f301ffff640 1 -- 172.21.15.50:0/3278524235 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3028010460 con 0x7f30381021c0 2024-04-23T18:36:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.351+0000 7f3040221740 1 Processor -- start 2024-04-23T18:36:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.351+0000 7f3040221740 1 -- start start 2024-04-23T18:36:37.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.351+0000 7f3040221740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 0x55b19d3788e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:37.252 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.351+0000 7f3040221740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378e20 0x55b19d3876d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:37.260 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.351+0000 7f3040221740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 0x55b19d38a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:37.269 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f3040221740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b19d2dd970 con 0x55b19d387c10 2024-04-23T18:36:37.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f3040221740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b19d27a4a0 con 0x55b19d378e20 2024-04-23T18:36:37.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f3040221740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b19d2d9930 con 0x55b19d378500 2024-04-23T18:36:37.298 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f30367fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 0x55b19d38a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:37.308 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f30367fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 0x55b19d38a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39140/0 (socket says 172.21.15.50:39140) 2024-04-23T18:36:37.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f30367fc640 1 -- 172.21.15.50:0/3862708752 learned_addr learned my addr 172.21.15.50:0/3862708752 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:37.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f30377fe640 1 --2- 172.21.15.50:0/3862708752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 0x55b19d3788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f3036ffd640 1 --2- 172.21.15.50:0/3862708752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378e20 0x55b19d3876d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f30367fc640 1 -- 172.21.15.50:0/3862708752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 msgr2=0x55b19d3788e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f30367fc640 1 --2- 172.21.15.50:0/3862708752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 0x55b19d3788e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f30367fc640 1 -- 172.21.15.50:0/3862708752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378e20 msgr2=0x55b19d3876d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f301ffff640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f30200a3dc0 0x7f30200a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:37.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f30367fc640 1 --2- 172.21.15.50:0/3862708752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378e20 0x55b19d3876d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:37.679 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f30367fc640 1 -- 172.21.15.50:0/3862708752 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b19cff0210 con 0x55b19d387c10 2024-04-23T18:36:37.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f30377fe640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f30200a3dc0 0x7f30200a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f301ffff640 1 -- 172.21.15.50:0/3278524235 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7f30280d0470 con 0x7f30381021c0 2024-04-23T18:36:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f30367fc640 1 --2- 172.21.15.50:0/3862708752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 0x55b19d38a000 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f302c004650 tx=0x7f302c00bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1bed09b661c10c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:37.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.352+0000 7f301d7fa640 1 -- 172.21.15.50:0/3862708752 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c003bd0 con 0x55b19d387c10 2024-04-23T18:36:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.353+0000 7f301d7fa640 1 -- 172.21.15.50:0/3862708752 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f302c015070 con 0x55b19d387c10 2024-04-23T18:36:37.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.353+0000 7f301d7fa640 1 -- 172.21.15.50:0/3862708752 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c004cd0 con 0x55b19d387c10 2024-04-23T18:36:37.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.353+0000 7f30377fe640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f30200a3dc0 0x7f30200a6280 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3030005e30 tx=0x7f3030005c30 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.353+0000 7f3040221740 1 -- 172.21.15.50:0/3862708752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 msgr2=0x55b19d38a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.353+0000 7f3040221740 1 --2- 172.21.15.50:0/3862708752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 0x55b19d38a000 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f302c004650 tx=0x7f302c00bb20 comp rx=0 tx=0).stop 2024-04-23T18:36:37.747 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.353+0000 7f3040221740 1 -- 172.21.15.50:0/3862708752 shutdown_connections 2024-04-23T18:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.353+0000 7f3040221740 1 --2- 172.21.15.50:0/3862708752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 0x55b19d38a000 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.353+0000 7f3040221740 1 --2- 172.21.15.50:0/3862708752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378e20 0x55b19d3876d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:37.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.353+0000 7f3040221740 1 --2- 172.21.15.50:0/3862708752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 0x55b19d3788e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:37.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.353+0000 7f3040221740 1 -- 172.21.15.50:0/3862708752 >> 172.21.15.50:0/3862708752 conn(0x55b19d374b30 msgr2=0x55b19d374f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.353+0000 7f3040221740 1 -- 172.21.15.50:0/3862708752 shutdown_connections 2024-04-23T18:36:37.991 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.353+0000 7f3040221740 1 -- 172.21.15.50:0/3862708752 wait complete. 2024-04-23T18:36:38.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f3040221740 1 -- start start 2024-04-23T18:36:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f3040221740 1 Processor -- start 2024-04-23T18:36:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f3040221740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 0x55b19d3874d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:38.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f3040221740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378e20 0x55b19d3847c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f3040221740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 0x55b19d384d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f3040221740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b19d38a840 con 0x55b19d387c10 2024-04-23T18:36:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f3040221740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b19d2780b0 con 0x55b19d378e20 2024-04-23T18:36:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f3040221740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b19d38a9c0 con 0x55b19d378500 2024-04-23T18:36:38.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f3040221740 10 client.0 _ll_register_callbacks cb 0x55b19d265d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f30377fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 0x55b19d384d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f30377fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 0x55b19d384d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39152/0 (socket says 172.21.15.50:39152) 2024-04-23T18:36:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f30377fe640 1 -- 172.21.15.50:0/1587719109 learned_addr learned my addr 172.21.15.50:0/1587719109 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:38.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f3036ffd640 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 0x55b19d3874d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:38.210 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f30367fc640 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378e20 0x55b19d3847c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:38.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f30377fe640 1 -- 172.21.15.50:0/1587719109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 msgr2=0x55b19d3874d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:38.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f30377fe640 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 0x55b19d3874d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:38.323 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f30377fe640 1 -- 172.21.15.50:0/1587719109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378e20 msgr2=0x55b19d3847c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:38.368 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f30377fe640 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378e20 0x55b19d3847c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.354+0000 7f30377fe640 1 -- 172.21.15.50:0/1587719109 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b19d0bc620 con 0x55b19d387c10 2024-04-23T18:36:38.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f30377fe640 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 0x55b19d384d00 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f3030004af0 tx=0x7f303002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ae4af03ebd2120 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f301aff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:36:38.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f301aff5640 1 -- 172.21.15.50:0/1587719109 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3030003b80 con 0x55b19d387c10 2024-04-23T18:36:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f301aff5640 1 -- 172.21.15.50:0/1587719109 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7f3030003d20 con 0x55b19d387c10 2024-04-23T18:36:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f3040221740 1 -- 172.21.15.50:0/1587719109 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f302c015070 con 0x55b19d387c10 2024-04-23T18:36:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f301aff5640 1 -- 172.21.15.50:0/1587719109 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f3030004040 con 0x55b19d387c10 2024-04-23T18:36:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f3040221740 10 client.17730 fetch_fsmap learned FSMap version 62 2024-04-23T18:36:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f3040221740 1 -- 172.21.15.50:0/1587719109 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55b19d396780 con 0x55b19d387c10 2024-04-23T18:36:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f301aff5640 1 -- 172.21.15.50:0/1587719109 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f302c015070 con 0x55b19d387c10 2024-04-23T18:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f301aff5640 1 -- 172.21.15.50:0/1587719109 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f3030051070 con 0x55b19d387c10 2024-04-23T18:36:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f3040221740 10 client.17730 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:36:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f3040221740 10 client.17730 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:36:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f3040221740 20 client.17730 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:38.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f3040221740 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b19d397c80 0x55b19d39a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:38.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f3040221740 4 client.17730 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:36:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.355+0000 7f3040221740 1 -- 172.21.15.50:0/1587719109 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b19d2eb390 con 0x55b19d397c80 2024-04-23T18:36:38.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.356+0000 7f3036ffd640 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b19d397c80 0x55b19d39a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.356+0000 7f3036ffd640 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b19d397c80 0x55b19d39a0b0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dd0ce50b7efd91f server_cookie=42178f50a926e929 in_seq=0 out_seq=0 2024-04-23T18:36:38.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.356+0000 7f301aff5640 10 client.17730 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:38.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.357+0000 7f301aff5640 1 -- 172.21.15.50:0/1587719109 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b19d2eb390 con 0x55b19d397c80 2024-04-23T18:36:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.357+0000 7f301aff5640 10 client.17730 handle_command_reply: tid=0 2024-04-23T18:36:38.805 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.357+0000 7f3040221740 1 client.17730 shutdown 2024-04-23T18:36:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.357+0000 7f3040221740 1 -- 172.21.15.50:0/1587719109 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 msgr2=0x55b19d384d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.358+0000 7f3040221740 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 0x55b19d384d00 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f3030004af0 tx=0x7f303002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:36:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.358+0000 7f3040221740 1 -- 172.21.15.50:0/1587719109 shutdown_connections 2024-04-23T18:36:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.358+0000 7f3040221740 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b19d397c80 0x55b19d39a0b0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:38.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.358+0000 7f3040221740 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d387c10 0x55b19d384d00 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:38.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.358+0000 7f3040221740 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378e20 0x55b19d3847c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.358+0000 7f3040221740 1 --2- 172.21.15.50:0/1587719109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 0x55b19d3874d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.358+0000 7f301aff5640 0 client.17730 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:39.030 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.358+0000 7f3040221740 1 -- 172.21.15.50:0/1587719109 >> 172.21.15.50:0/1587719109 conn(0x55b19d374b30 msgr2=0x55b19d37a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:39.039 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.358+0000 7f3040221740 1 -- 172.21.15.50:0/1587719109 shutdown_connections 2024-04-23T18:36:39.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.358+0000 7f3040221740 1 -- 172.21.15.50:0/1587719109 wait complete. 2024-04-23T18:36:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.358+0000 7f3040221740 20 client.17730 trim_cache size 0 max 16384 2024-04-23T18:36:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.373+0000 7f3040221740 1 Processor -- start 2024-04-23T18:36:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.373+0000 7f3040221740 1 -- start start 2024-04-23T18:36:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.373+0000 7f3040221740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378500 0x55b19d393770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:39.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.373+0000 7f3040221740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 0x55b19d37fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:39.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.373+0000 7f3040221740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d387c10 0x55b19d387000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.373+0000 7f3040221740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f30381ae130 con 0x55b19d378e20 2024-04-23T18:36:39.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.373+0000 7f3040221740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f30381aba00 con 0x55b19d378500 2024-04-23T18:36:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.373+0000 7f3040221740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f30381a9890 con 0x55b19d387c10 2024-04-23T18:36:39.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.373+0000 7f30367fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 0x55b19d37fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:39.207 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.373+0000 7f30367fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 0x55b19d37fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39158/0 (socket says 172.21.15.50:39158) 2024-04-23T18:36:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.373+0000 7f30367fc640 1 -- 172.21.15.50:0/1207888042 learned_addr learned my addr 172.21.15.50:0/1207888042 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.373+0000 7f3036ffd640 1 --2- 172.21.15.50:0/1207888042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378500 0x55b19d393770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.374+0000 7f30377fe640 1 --2- 172.21.15.50:0/1207888042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d387c10 0x55b19d387000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.374+0000 7f30367fc640 1 -- 172.21.15.50:0/1207888042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d387c10 msgr2=0x55b19d387000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:39.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.374+0000 7f30367fc640 1 --2- 172.21.15.50:0/1207888042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d387c10 0x55b19d387000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.374+0000 7f30367fc640 1 -- 172.21.15.50:0/1207888042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378500 msgr2=0x55b19d393770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.374+0000 7f30367fc640 1 --2- 172.21.15.50:0/1207888042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378500 0x55b19d393770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:39.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.374+0000 7f30367fc640 1 -- 172.21.15.50:0/1207888042 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b19d374960 con 0x55b19d378e20 2024-04-23T18:36:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.374+0000 7f3036ffd640 1 --2- 172.21.15.50:0/1207888042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378500 0x55b19d393770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:36:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.374+0000 7f30367fc640 1 --2- 172.21.15.50:0/1207888042 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 0x55b19d37fd80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f302c0039c0 tx=0x7f302c004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4896bb179d506c59 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:39.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.374+0000 7f301cff9640 1 -- 172.21.15.50:0/1207888042 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c003bd0 con 0x55b19d378e20 2024-04-23T18:36:39.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.374+0000 7f301cff9640 1 -- 172.21.15.50:0/1207888042 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f302c015070 con 0x55b19d378e20 2024-04-23T18:36:39.526 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.374+0000 7f301cff9640 1 -- 172.21.15.50:0/1207888042 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c019420 con 0x55b19d378e20 2024-04-23T18:36:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 -- 172.21.15.50:0/1207888042 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 msgr2=0x55b19d37fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 --2- 172.21.15.50:0/1207888042 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 0x55b19d37fd80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f302c0039c0 tx=0x7f302c004950 comp rx=0 tx=0).stop 2024-04-23T18:36:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 -- 172.21.15.50:0/1207888042 shutdown_connections 2024-04-23T18:36:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 --2- 172.21.15.50:0/1207888042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d387c10 0x55b19d387000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:39.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 --2- 172.21.15.50:0/1207888042 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 0x55b19d37fd80 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:39.677 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 --2- 172.21.15.50:0/1207888042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d378500 0x55b19d393770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:39.743 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 -- 172.21.15.50:0/1207888042 >> 172.21.15.50:0/1207888042 conn(0x55b19d374b30 msgr2=0x55b19d37f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 -- 172.21.15.50:0/1207888042 shutdown_connections 2024-04-23T18:36:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 -- 172.21.15.50:0/1207888042 wait complete. 2024-04-23T18:36:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 -- start start 2024-04-23T18:36:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 Processor -- start 2024-04-23T18:36:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 0x55b19d390090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 0x55b19d38b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:39.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d387c10 0x55b19d38b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:39.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f30381a9ee0 con 0x55b19d378e20 2024-04-23T18:36:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f30381ab330 con 0x55b19d387c10 2024-04-23T18:36:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f30381aa590 con 0x55b19d378500 2024-04-23T18:36:39.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f3040221740 10 client.0 _ll_register_callbacks cb 0x7f3030032f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:39.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f30367fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 0x55b19d38b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f30367fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 0x55b19d38b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39162/0 (socket says 172.21.15.50:39162) 2024-04-23T18:36:39.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.375+0000 7f30367fc640 1 -- 172.21.15.50:0/3376839273 learned_addr learned my addr 172.21.15.50:0/3376839273 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:39.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f3036ffd640 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 0x55b19d390090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:40.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f30377fe640 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d387c10 0x55b19d38b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:40.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f30367fc640 1 -- 172.21.15.50:0/3376839273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 msgr2=0x55b19d390090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f30367fc640 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 0x55b19d390090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f30367fc640 1 -- 172.21.15.50:0/3376839273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d387c10 msgr2=0x55b19d38b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f30367fc640 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d387c10 0x55b19d38b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f30367fc640 1 -- 172.21.15.50:0/3376839273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b19d38a720 con 0x55b19d378e20 2024-04-23T18:36:40.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f30377fe640 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d387c10 0x55b19d38b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:36:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f30367fc640 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 0x55b19d38b400 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f302c005fe0 tx=0x7f302c0048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d93befa07707870 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f301aff5640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:36:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f301aff5640 1 -- 172.21.15.50:0/3376839273 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c032070 con 0x55b19d378e20 2024-04-23T18:36:40.282 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f301aff5640 1 -- 172.21.15.50:0/3376839273 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7f302c011440 con 0x55b19d378e20 2024-04-23T18:36:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f3040221740 1 -- 172.21.15.50:0/3376839273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f302c015070 con 0x55b19d378e20 2024-04-23T18:36:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f301aff5640 1 -- 172.21.15.50:0/3376839273 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f302c011640 con 0x55b19d378e20 2024-04-23T18:36:40.309 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f3040221740 10 client.17736 fetch_fsmap learned FSMap version 62 2024-04-23T18:36:40.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f3040221740 1 -- 172.21.15.50:0/3376839273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55b19d3fe0d0 con 0x55b19d378e20 2024-04-23T18:36:40.350 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.376+0000 7f301aff5640 1 -- 172.21.15.50:0/3376839273 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f302c015070 con 0x55b19d378e20 2024-04-23T18:36:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.377+0000 7f301aff5640 1 -- 172.21.15.50:0/3376839273 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f302c023070 con 0x55b19d378e20 2024-04-23T18:36:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.377+0000 7f3040221740 10 client.17736 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:36:40.473 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.377+0000 7f3040221740 10 client.17736 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:36:40.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.377+0000 7f3040221740 20 client.17736 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.377+0000 7f3040221740 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b19d3fe5a0 0x55b19d400990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.377+0000 7f3040221740 4 client.17736 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:36:40.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.377+0000 7f3040221740 1 -- 172.21.15.50:0/3376839273 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b19d2e5200 con 0x55b19d3fe5a0 2024-04-23T18:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.377+0000 7f3036ffd640 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b19d3fe5a0 0x55b19d400990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.377+0000 7f3036ffd640 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b19d3fe5a0 0x55b19d400990 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=764800a089241177 server_cookie=702ded57edbc5ec0 in_seq=0 out_seq=0 2024-04-23T18:36:40.650 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.377+0000 7f301aff5640 10 client.17736 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f301aff5640 1 -- 172.21.15.50:0/3376839273 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b19d2e5200 con 0x55b19d3fe5a0 2024-04-23T18:36:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f301aff5640 10 client.17736 handle_command_reply: tid=0 2024-04-23T18:36:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f3040221740 1 client.17736 shutdown 2024-04-23T18:36:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f3040221740 1 -- 172.21.15.50:0/3376839273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 msgr2=0x55b19d38b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f3040221740 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 0x55b19d38b400 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f302c005fe0 tx=0x7f302c0048b0 comp rx=0 tx=0).stop 2024-04-23T18:36:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f3040221740 1 -- 172.21.15.50:0/3376839273 shutdown_connections 2024-04-23T18:36:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f3040221740 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b19d3fe5a0 0x55b19d400990 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:40.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f3040221740 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b19d387c10 0x55b19d38b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f301aff5640 0 client.17736 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:41.022 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f3040221740 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b19d378e20 0x55b19d38b400 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:41.226 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f3040221740 1 --2- 172.21.15.50:0/3376839273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b19d378500 0x55b19d390090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f3040221740 1 -- 172.21.15.50:0/3376839273 >> 172.21.15.50:0/3376839273 conn(0x55b19d374b30 msgr2=0x55b19d373420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f3040221740 1 -- 172.21.15.50:0/3376839273 shutdown_connections 2024-04-23T18:36:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f3040221740 1 -- 172.21.15.50:0/3376839273 wait complete. 2024-04-23T18:36:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.534+0000 7f3040221740 20 client.17736 trim_cache size 0 max 16384 2024-04-23T18:36:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:36:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:36:41.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:36:41.366 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:36:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.535+0000 7f303d50f640 1 -- 172.21.15.50:0/3278524235 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f30200a3dc0 msgr2=0x7f30200a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:41.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.535+0000 7f303d50f640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f30200a3dc0 0x7f30200a6280 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3030005e30 tx=0x7f3030005c30 comp rx=0 tx=0).stop 2024-04-23T18:36:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.535+0000 7f303d50f640 1 -- 172.21.15.50:0/3278524235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30381021c0 msgr2=0x7f3038198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.535+0000 7f303d50f640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30381021c0 0x7f3038198740 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f302800d950 tx=0x7f302800de10 comp rx=0 tx=0).stop 2024-04-23T18:36:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.535+0000 7f303d50f640 1 -- 172.21.15.50:0/3278524235 shutdown_connections 2024-04-23T18:36:41.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.535+0000 7f303d50f640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f30200a3dc0 0x7f30200a6280 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.535+0000 7f303d50f640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f303819cfb0 0x7f303819d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.535+0000 7f303d50f640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3038107290 0x7f3038198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.535+0000 7f303d50f640 1 --2- 172.21.15.50:0/3278524235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30381021c0 0x7f3038198740 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.535+0000 7f303d50f640 1 -- 172.21.15.50:0/3278524235 >> 172.21.15.50:0/3278524235 conn(0x7f30380fda20 msgr2=0x7f30380ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:41.681 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.535+0000 7f303d50f640 1 -- 172.21.15.50:0/3278524235 shutdown_connections 2024-04-23T18:36:41.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:35.535+0000 7f303d50f640 1 -- 172.21.15.50:0/3278524235 wait complete. 2024-04-23T18:36:41.699 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 22.03 MB/sec execute 189 sec latency 4650.366 ms 2024-04-23T18:36:41.710 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 21.92 MB/sec execute 190 sec latency 5650.404 ms 2024-04-23T18:36:41.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:36 smithi050 ceph-mon[29187]: pgmap v575: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.5 MiB/s wr, 427 op/s 2024-04-23T18:36:41.727 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 21.80 MB/sec execute 191 sec latency 6650.517 ms 2024-04-23T18:36:41.735 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 21.69 MB/sec execute 192 sec latency 7650.621 ms 2024-04-23T18:36:41.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:38 smithi050 ceph-mon[29187]: pgmap v576: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.9 MiB/s wr, 490 op/s 2024-04-23T18:36:41.757 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 21.58 MB/sec execute 193 sec latency 8650.692 ms 2024-04-23T18:36:41.766 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 21.46 MB/sec execute 194 sec latency 9650.735 ms 2024-04-23T18:36:41.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:40 smithi050 ceph-mon[29187]: pgmap v577: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.1 MiB/s wr, 257 op/s 2024-04-23T18:36:41.817 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 21.35 MB/sec execute 195 sec latency 10650.777 ms 2024-04-23T18:36:41.827 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:36 smithi188 ceph-mon[32694]: pgmap v575: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.5 MiB/s wr, 427 op/s 2024-04-23T18:36:41.886 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:38 smithi188 ceph-mon[32694]: pgmap v576: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.9 MiB/s wr, 490 op/s 2024-04-23T18:36:41.942 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:40 smithi188 ceph-mon[32694]: pgmap v577: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.1 MiB/s wr, 257 op/s 2024-04-23T18:36:41.996 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:36 smithi177 ceph-mon[32308]: pgmap v575: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.5 MiB/s wr, 427 op/s 2024-04-23T18:36:42.005 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:38 smithi177 ceph-mon[32308]: pgmap v576: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.9 MiB/s wr, 490 op/s 2024-04-23T18:36:42.067 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:40 smithi177 ceph-mon[32308]: pgmap v577: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.1 MiB/s wr, 257 op/s 2024-04-23T18:36:42.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:42 smithi177 ceph-mon[32308]: pgmap v578: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.1 MiB/s wr, 257 op/s 2024-04-23T18:36:42.574 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 21.25 MB/sec execute 196 sec latency 11650.906 ms 2024-04-23T18:36:42.583 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:42 smithi188 ceph-mon[32694]: pgmap v578: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.1 MiB/s wr, 257 op/s 2024-04-23T18:36:42.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:42 smithi050 ceph-mon[29187]: pgmap v578: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.1 MiB/s wr, 257 op/s 2024-04-23T18:36:43.574 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 21.14 MB/sec execute 197 sec latency 12650.983 ms 2024-04-23T18:36:44.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:44 smithi177 ceph-mon[32308]: pgmap v579: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.1 MiB/s wr, 263 op/s 2024-04-23T18:36:44.574 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 21.03 MB/sec execute 198 sec latency 13651.100 ms 2024-04-23T18:36:44.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:44 smithi188 ceph-mon[32694]: pgmap v579: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.1 MiB/s wr, 263 op/s 2024-04-23T18:36:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:44 smithi050 ceph-mon[29187]: pgmap v579: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.1 MiB/s wr, 263 op/s 2024-04-23T18:36:45.574 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 20.93 MB/sec execute 199 sec latency 14651.177 ms 2024-04-23T18:36:46.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:46 smithi177 ceph-mon[32308]: pgmap v580: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 566 KiB/s wr, 84 op/s 2024-04-23T18:36:46.574 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 20.82 MB/sec execute 200 sec latency 15651.292 ms 2024-04-23T18:36:46.583 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:46 smithi188 ceph-mon[32694]: pgmap v580: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 566 KiB/s wr, 84 op/s 2024-04-23T18:36:46.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:46 smithi050 ceph-mon[29187]: pgmap v580: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 566 KiB/s wr, 84 op/s 2024-04-23T18:36:47.575 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 20.72 MB/sec execute 201 sec latency 16651.442 ms 2024-04-23T18:36:48.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:48 smithi177 ceph-mon[32308]: pgmap v581: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 1.1 MiB/s wr, 96 op/s 2024-04-23T18:36:48.574 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 20.61 MB/sec execute 202 sec latency 17651.510 ms 2024-04-23T18:36:48.620 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:48 smithi188 ceph-mon[32694]: pgmap v581: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 1.1 MiB/s wr, 96 op/s 2024-04-23T18:36:48.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.634+0000 7f74189d8640 1 -- 172.21.15.50:0/3813629645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7414103b20 msgr2=0x7f741410c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.634+0000 7f74189d8640 1 --2- 172.21.15.50:0/3813629645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7414103b20 0x7f741410c000 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f7408009900 tx=0x7f740802a480 comp rx=0 tx=0).stop 2024-04-23T18:36:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.634+0000 7f74189d8640 1 -- 172.21.15.50:0/3813629645 shutdown_connections 2024-04-23T18:36:48.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.634+0000 7f74189d8640 1 --2- 172.21.15.50:0/3813629645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f741410c5d0 0x7f741410ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.634+0000 7f74189d8640 1 --2- 172.21.15.50:0/3813629645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7414103b20 0x7f741410c000 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.634+0000 7f74189d8640 1 --2- 172.21.15.50:0/3813629645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7414103200 0x7f74141035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.634+0000 7f74189d8640 1 -- 172.21.15.50:0/3813629645 >> 172.21.15.50:0/3813629645 conn(0x7f74140771d0 msgr2=0x7f74140775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.634+0000 7f74189d8640 1 -- 172.21.15.50:0/3813629645 shutdown_connections 2024-04-23T18:36:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.634+0000 7f74189d8640 1 -- 172.21.15.50:0/3813629645 wait complete. 2024-04-23T18:36:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f74189d8640 1 Processor -- start 2024-04-23T18:36:48.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f74189d8640 1 -- start start 2024-04-23T18:36:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f74189d8640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7414103200 0x7f7414100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:48.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f74189d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7414103b20 0x7f7414101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:48.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f74189d8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f741410c5d0 0x7f74141018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f74189d8640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7414112210 con 0x7f741410c5d0 2024-04-23T18:36:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f74189d8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7414112090 con 0x7f7414103b20 2024-04-23T18:36:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f74189d8640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7414112390 con 0x7f7414103200 2024-04-23T18:36:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f7412d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f741410c5d0 0x7f74141018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f7412d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f741410c5d0 0x7f74141018d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55644/0 (socket says 172.21.15.50:55644) 2024-04-23T18:36:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f7412d9d640 1 -- 172.21.15.50:0/1235858203 learned_addr learned my addr 172.21.15.50:0/1235858203 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f741259c640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7414103200 0x7f7414100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:49.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f7411d9b640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7414103b20 0x7f7414101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:49.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f7412d9d640 1 -- 172.21.15.50:0/1235858203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7414103200 msgr2=0x7f7414100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:49.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f7412d9d640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7414103200 0x7f7414100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f7412d9d640 1 -- 172.21.15.50:0/1235858203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7414103b20 msgr2=0x7f7414101390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f7412d9d640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7414103b20 0x7f7414101390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.635+0000 7f7412d9d640 1 -- 172.21.15.50:0/1235858203 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74141a6c10 con 0x7f741410c5d0 2024-04-23T18:36:49.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.636+0000 7f7412d9d640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f741410c5d0 0x7f74141018d0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f740c00eb40 tx=0x7f740c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.636+0000 7f74037fe640 1 -- 172.21.15.50:0/1235858203 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c004b20 con 0x7f741410c5d0 2024-04-23T18:36:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.636+0000 7f74037fe640 1 -- 172.21.15.50:0/1235858203 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f740c004cc0 con 0x7f741410c5d0 2024-04-23T18:36:49.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.636+0000 7f74189d8640 1 -- 172.21.15.50:0/1235858203 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74141a6f00 con 0x7f741410c5d0 2024-04-23T18:36:49.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.636+0000 7f74037fe640 1 -- 172.21.15.50:0/1235858203 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c010720 con 0x7f741410c5d0 2024-04-23T18:36:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.636+0000 7f74189d8640 1 -- 172.21.15.50:0/1235858203 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74141a73d0 con 0x7f741410c5d0 2024-04-23T18:36:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.638+0000 7f74037fe640 1 -- 172.21.15.50:0/1235858203 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f740c005560 con 0x7f741410c5d0 2024-04-23T18:36:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f741b7c8740 1 Processor -- start 2024-04-23T18:36:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f741b7c8740 1 -- start start 2024-04-23T18:36:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f741b7c8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21b620 0x5652fd21ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f741b7c8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd21bf40 0x5652fd2262c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f741b7c8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 0x5652fd228bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f741b7c8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5652fd180970 con 0x5652fd226800 2024-04-23T18:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f741b7c8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5652fd11d4a0 con 0x5652fd21b620 2024-04-23T18:36:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f741b7c8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5652fd17c930 con 0x5652fd21bf40 2024-04-23T18:36:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f74037fe640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f73fc0a3e40 0x7f73fc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f7411d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 0x5652fd228bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f741259c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd21bf40 0x5652fd2262c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:49.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f7411d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 0x5652fd228bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55654/0 (socket says 172.21.15.50:55654) 2024-04-23T18:36:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f7411d9b640 1 -- 172.21.15.50:0/319823483 learned_addr learned my addr 172.21.15.50:0/319823483 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f7411d9b640 1 --2- 172.21.15.50:0/319823483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21b620 0x5652fd21ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.639+0000 7f7412d9d640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f73fc0a3e40 0x7f73fc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.640+0000 7f74037fe640 1 -- 172.21.15.50:0/1235858203 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7f740c0d0eb0 con 0x7f741410c5d0 2024-04-23T18:36:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.640+0000 7f741259c640 1 -- 172.21.15.50:0/319823483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21b620 msgr2=0x5652fd21ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:49.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.640+0000 7f741259c640 1 --2- 172.21.15.50:0/319823483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21b620 0x5652fd21ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.640+0000 7f741259c640 1 -- 172.21.15.50:0/319823483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 msgr2=0x5652fd228bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.640+0000 7f741259c640 1 --2- 172.21.15.50:0/319823483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 0x5652fd228bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.640+0000 7f741259c640 1 -- 172.21.15.50:0/319823483 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652fce93210 con 0x5652fd21bf40 2024-04-23T18:36:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.640+0000 7f7412d9d640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f73fc0a3e40 0x7f73fc0a6300 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f740c09cd90 tx=0x7f740c00af40 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.640+0000 7f741259c640 1 --2- 172.21.15.50:0/319823483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd21bf40 0x5652fd2262c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7404009c00 tx=0x7f7404011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd054d1e5a02bbfd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.640+0000 7f7400ff9640 1 -- 172.21.15.50:0/319823483 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404004050 con 0x5652fd21bf40 2024-04-23T18:36:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.640+0000 7f7400ff9640 1 -- 172.21.15.50:0/319823483 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74040054d0 con 0x5652fd21bf40 2024-04-23T18:36:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.640+0000 7f7400ff9640 1 -- 172.21.15.50:0/319823483 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404004ca0 con 0x5652fd21bf40 2024-04-23T18:36:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.641+0000 7f741b7c8740 1 -- 172.21.15.50:0/319823483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd21bf40 msgr2=0x5652fd2262c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.641+0000 7f741b7c8740 1 --2- 172.21.15.50:0/319823483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd21bf40 0x5652fd2262c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7404009c00 tx=0x7f7404011810 comp rx=0 tx=0).stop 2024-04-23T18:36:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.641+0000 7f741b7c8740 1 -- 172.21.15.50:0/319823483 shutdown_connections 2024-04-23T18:36:49.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.641+0000 7f741b7c8740 1 --2- 172.21.15.50:0/319823483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 0x5652fd228bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.641+0000 7f741b7c8740 1 --2- 172.21.15.50:0/319823483 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd21bf40 0x5652fd2262c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:49.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.641+0000 7f741b7c8740 1 --2- 172.21.15.50:0/319823483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21b620 0x5652fd21ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:49.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.641+0000 7f741b7c8740 1 -- 172.21.15.50:0/319823483 >> 172.21.15.50:0/319823483 conn(0x5652fd217b30 msgr2=0x5652fd217f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.641+0000 7f741b7c8740 1 -- 172.21.15.50:0/319823483 shutdown_connections 2024-04-23T18:36:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.641+0000 7f741b7c8740 1 -- 172.21.15.50:0/319823483 wait complete. 2024-04-23T18:36:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.641+0000 7f741b7c8740 1 -- start start 2024-04-23T18:36:49.991 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f741b7c8740 1 Processor -- start 2024-04-23T18:36:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f741b7c8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21b620 0x5652fd226080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f741b7c8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21bf40 0x5652fd223370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f741b7c8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd226800 0x5652fd2238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f741b7c8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5652fd229430 con 0x5652fd21b620 2024-04-23T18:36:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f741b7c8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5652fd11b0b0 con 0x5652fd21bf40 2024-04-23T18:36:50.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f741b7c8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5652fd2295b0 con 0x5652fd226800 2024-04-23T18:36:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f741b7c8740 10 client.0 _ll_register_callbacks cb 0x5652fd108d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f7411d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21b620 0x5652fd226080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f7411d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21b620 0x5652fd226080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55660/0 (socket says 172.21.15.50:55660) 2024-04-23T18:36:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f7411d9b640 1 -- 172.21.15.50:0/4219119534 learned_addr learned my addr 172.21.15.50:0/4219119534 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f741259c640 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21bf40 0x5652fd223370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f7411d9b640 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd226800 0x5652fd2238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f7411d9b640 1 -- 172.21.15.50:0/4219119534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd226800 msgr2=0x5652fd2238b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f7411d9b640 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd226800 0x5652fd2238b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:50.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f7411d9b640 1 -- 172.21.15.50:0/4219119534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21bf40 msgr2=0x5652fd223370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f7411d9b640 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21bf40 0x5652fd223370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f7411d9b640 1 -- 172.21.15.50:0/4219119534 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652fcf5f620 con 0x5652fd21b620 2024-04-23T18:36:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f7411d9b640 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21b620 0x5652fd226080 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f7408009900 tx=0x7f7408041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a73c7a0a4b9bd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.642+0000 7f73f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:36:50.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f73f9ffb640 1 -- 172.21.15.50:0/4219119534 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408035050 con 0x5652fd21b620 2024-04-23T18:36:50.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f73f9ffb640 1 -- 172.21.15.50:0/4219119534 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7f7408004100 con 0x5652fd21b620 2024-04-23T18:36:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f741b7c8740 1 -- 172.21.15.50:0/4219119534 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74040054d0 con 0x5652fd21b620 2024-04-23T18:36:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f73f9ffb640 1 -- 172.21.15.50:0/4219119534 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f7408004300 con 0x5652fd21b620 2024-04-23T18:36:50.453 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f741b7c8740 10 client.17748 fetch_fsmap learned FSMap version 62 2024-04-23T18:36:50.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f741b7c8740 1 -- 172.21.15.50:0/4219119534 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5652fd2352c0 con 0x5652fd21b620 2024-04-23T18:36:50.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f73f9ffb640 1 -- 172.21.15.50:0/4219119534 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74040054d0 con 0x5652fd21b620 2024-04-23T18:36:50.504 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f73f9ffb640 1 -- 172.21.15.50:0/4219119534 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f7408055500 con 0x5652fd21b620 2024-04-23T18:36:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f741b7c8740 10 client.17748 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:36:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f741b7c8740 10 client.17748 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:36:50.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f741b7c8740 20 client.17748 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f741b7c8740 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5652fd236730 0x5652fd238b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f741b7c8740 4 client.17748 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:36:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.643+0000 7f741b7c8740 1 -- 172.21.15.50:0/4219119534 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652fd18e390 con 0x5652fd236730 2024-04-23T18:36:50.626 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.644+0000 7f741259c640 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5652fd236730 0x5652fd238b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.644+0000 7f741259c640 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5652fd236730 0x5652fd238b20 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a613a1b30d6e0e49 server_cookie=a2dbeadb1cc2a88c in_seq=0 out_seq=0 2024-04-23T18:36:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.644+0000 7f73f9ffb640 10 client.17748 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:50.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f73f9ffb640 1 -- 172.21.15.50:0/4219119534 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5652fd18e390 con 0x5652fd236730 2024-04-23T18:36:50.695 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f73f9ffb640 10 client.17748 handle_command_reply: tid=0 2024-04-23T18:36:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f741b7c8740 1 client.17748 shutdown 2024-04-23T18:36:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f741b7c8740 1 -- 172.21.15.50:0/4219119534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21b620 msgr2=0x5652fd226080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f741b7c8740 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21b620 0x5652fd226080 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f7408009900 tx=0x7f7408041040 comp rx=0 tx=0).stop 2024-04-23T18:36:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f741b7c8740 1 -- 172.21.15.50:0/4219119534 shutdown_connections 2024-04-23T18:36:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f741b7c8740 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5652fd236730 0x5652fd238b20 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f741b7c8740 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd226800 0x5652fd2238b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f741b7c8740 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21bf40 0x5652fd223370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f741b7c8740 1 --2- 172.21.15.50:0/4219119534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21b620 0x5652fd226080 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:50.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f73f9ffb640 0 client.17748 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:50.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f741b7c8740 1 -- 172.21.15.50:0/4219119534 >> 172.21.15.50:0/4219119534 conn(0x5652fd217b30 msgr2=0x5652fd21d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:50.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f741b7c8740 1 -- 172.21.15.50:0/4219119534 shutdown_connections 2024-04-23T18:36:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.645+0000 7f741b7c8740 1 -- 172.21.15.50:0/4219119534 wait complete. 2024-04-23T18:36:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.646+0000 7f741b7c8740 20 client.17748 trim_cache size 0 max 16384 2024-04-23T18:36:51.027 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:50 smithi188 ceph-mon[32694]: pgmap v582: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 597 KiB/s wr, 18 op/s 2024-04-23T18:36:51.033 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:50 smithi177 ceph-mon[32308]: pgmap v582: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 597 KiB/s wr, 18 op/s 2024-04-23T18:36:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f741b7c8740 1 Processor -- start 2024-04-23T18:36:51.051 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f741b7c8740 1 -- start start 2024-04-23T18:36:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f741b7c8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd21b620 0x5652fd225bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f741b7c8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21bf40 0x5652fd23e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f741b7c8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 0x5652fd23e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:51.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f741b7c8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f74141b25a0 con 0x5652fd226800 2024-04-23T18:36:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f741b7c8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f74141afe70 con 0x5652fd21bf40 2024-04-23T18:36:51.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f741b7c8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f74141adcc0 con 0x5652fd21b620 2024-04-23T18:36:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f7411d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 0x5652fd23e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:51.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f7411d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 0x5652fd23e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55668/0 (socket says 172.21.15.50:55668) 2024-04-23T18:36:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f7411d9b640 1 -- 172.21.15.50:0/1193506791 learned_addr learned my addr 172.21.15.50:0/1193506791 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:51.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f741259c640 1 --2- 172.21.15.50:0/1193506791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21bf40 0x5652fd23e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:51.740 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f7411d9b640 1 --2- 172.21.15.50:0/1193506791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd21b620 0x5652fd225bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:51.748 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f7411d9b640 1 -- 172.21.15.50:0/1193506791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd21b620 msgr2=0x5652fd225bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:51.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f7411d9b640 1 --2- 172.21.15.50:0/1193506791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd21b620 0x5652fd225bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:51.769 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f7411d9b640 1 -- 172.21.15.50:0/1193506791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21bf40 msgr2=0x5652fd23e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f7411d9b640 1 --2- 172.21.15.50:0/1193506791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21bf40 0x5652fd23e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:51.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f7411d9b640 1 -- 172.21.15.50:0/1193506791 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652fd238e90 con 0x5652fd226800 2024-04-23T18:36:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f741259c640 1 --2- 172.21.15.50:0/1193506791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21bf40 0x5652fd23e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:36:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.661+0000 7f7411d9b640 1 --2- 172.21.15.50:0/1193506791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 0x5652fd23e8d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f7408031a90 tx=0x7f7408055f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c668809ed43c2b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:51.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.662+0000 7f73fbfff640 1 -- 172.21.15.50:0/1193506791 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740805b070 con 0x5652fd226800 2024-04-23T18:36:51.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.662+0000 7f73fbfff640 1 -- 172.21.15.50:0/1193506791 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74080044a0 con 0x5652fd226800 2024-04-23T18:36:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.662+0000 7f73fbfff640 1 -- 172.21.15.50:0/1193506791 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408035040 con 0x5652fd226800 2024-04-23T18:36:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.662+0000 7f741b7c8740 1 -- 172.21.15.50:0/1193506791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 msgr2=0x5652fd23e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.662+0000 7f741b7c8740 1 --2- 172.21.15.50:0/1193506791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 0x5652fd23e8d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f7408031a90 tx=0x7f7408055f40 comp rx=0 tx=0).stop 2024-04-23T18:36:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.662+0000 7f741b7c8740 1 -- 172.21.15.50:0/1193506791 shutdown_connections 2024-04-23T18:36:51.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.662+0000 7f741b7c8740 1 --2- 172.21.15.50:0/1193506791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd226800 0x5652fd23e8d0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.662+0000 7f741b7c8740 1 --2- 172.21.15.50:0/1193506791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21bf40 0x5652fd23e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.662+0000 7f741b7c8740 1 --2- 172.21.15.50:0/1193506791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd21b620 0x5652fd225bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.662+0000 7f741b7c8740 1 -- 172.21.15.50:0/1193506791 >> 172.21.15.50:0/1193506791 conn(0x5652fd217b30 msgr2=0x5652fd216cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.662+0000 7f741b7c8740 1 -- 172.21.15.50:0/1193506791 shutdown_connections 2024-04-23T18:36:52.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.662+0000 7f741b7c8740 1 -- 172.21.15.50:0/1193506791 wait complete. 2024-04-23T18:36:52.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741b7c8740 1 -- start start 2024-04-23T18:36:52.037 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741b7c8740 1 Processor -- start 2024-04-23T18:36:52.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741b7c8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21b620 0x5652fd22ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741b7c8740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21bf40 0x5652fd22bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:52.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741b7c8740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd226800 0x5652fd22c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:52.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741b7c8740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f74141ae330 con 0x5652fd21bf40 2024-04-23T18:36:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741b7c8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f74141af7a0 con 0x5652fd21b620 2024-04-23T18:36:52.138 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741b7c8740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f74141aea00 con 0x5652fd226800 2024-04-23T18:36:52.186 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741b7c8740 10 client.0 _ll_register_callbacks cb 0x7f7408032e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:52.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21bf40 0x5652fd22bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21bf40 0x5652fd22bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55682/0 (socket says 172.21.15.50:55682) 2024-04-23T18:36:52.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741259c640 1 -- 172.21.15.50:0/3038671992 learned_addr learned my addr 172.21.15.50:0/3038671992 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741259c640 1 -- 172.21.15.50:0/3038671992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd226800 msgr2=0x5652fd22c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:52.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741259c640 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd226800 0x5652fd22c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:52.283 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f7411d9b640 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21b620 0x5652fd22ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:52.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741259c640 1 -- 172.21.15.50:0/3038671992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21b620 msgr2=0x5652fd22ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741259c640 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21b620 0x5652fd22ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741259c640 1 -- 172.21.15.50:0/3038671992 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652fd22c910 con 0x5652fd21bf40 2024-04-23T18:36:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f7411d9b640 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21b620 0x5652fd22ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:36:52.324 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f741259c640 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21bf40 0x5652fd22bd90 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x5652fd21cbc0 tx=0x7f7404005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d752e66b29e30aef server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:52.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.663+0000 7f73f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:36:52.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f73f9ffb640 1 -- 172.21.15.50:0/3038671992 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404034060 con 0x5652fd21bf40 2024-04-23T18:36:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f73f9ffb640 1 -- 172.21.15.50:0/3038671992 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7f7404004050 con 0x5652fd21bf40 2024-04-23T18:36:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f741b7c8740 1 -- 172.21.15.50:0/3038671992 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74080044a0 con 0x5652fd21bf40 2024-04-23T18:36:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f73f9ffb640 1 -- 172.21.15.50:0/3038671992 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f7404004250 con 0x5652fd21bf40 2024-04-23T18:36:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f741b7c8740 10 client.17754 fetch_fsmap learned FSMap version 62 2024-04-23T18:36:52.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f741b7c8740 1 -- 172.21.15.50:0/3038671992 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5652fd29ca50 con 0x5652fd21bf40 2024-04-23T18:36:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f73f9ffb640 1 -- 172.21.15.50:0/3038671992 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74080044a0 con 0x5652fd21bf40 2024-04-23T18:36:52.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f73f9ffb640 1 -- 172.21.15.50:0/3038671992 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f74040035f0 con 0x5652fd21bf40 2024-04-23T18:36:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f741b7c8740 10 client.17754 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:36:52.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f741b7c8740 10 client.17754 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:36:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f741b7c8740 20 client.17754 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f741b7c8740 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5652fd29d070 0x5652fd29f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f741b7c8740 4 client.17754 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:36:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f741b7c8740 1 -- 172.21.15.50:0/3038671992 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5652fd188200 con 0x5652fd29d070 2024-04-23T18:36:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.664+0000 7f7411d9b640 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5652fd29d070 0x5652fd29f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.665+0000 7f7411d9b640 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5652fd29d070 0x5652fd29f490 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8767a7532ecf4fe server_cookie=eaf8d841429c4df in_seq=0 out_seq=0 2024-04-23T18:36:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.665+0000 7f73f9ffb640 10 client.17754 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:52.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:48 smithi050 ceph-mon[29187]: pgmap v581: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 1.1 MiB/s wr, 96 op/s 2024-04-23T18:36:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f73f9ffb640 1 -- 172.21.15.50:0/3038671992 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5652fd188200 con 0x5652fd29d070 2024-04-23T18:36:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f73f9ffb640 10 client.17754 handle_command_reply: tid=0 2024-04-23T18:36:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f741b7c8740 1 client.17754 shutdown 2024-04-23T18:36:52.745 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f741b7c8740 1 -- 172.21.15.50:0/3038671992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21bf40 msgr2=0x5652fd22bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:52.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f741b7c8740 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21bf40 0x5652fd22bd90 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x5652fd21cbc0 tx=0x7f7404005d30 comp rx=0 tx=0).stop 2024-04-23T18:36:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f741b7c8740 1 -- 172.21.15.50:0/3038671992 shutdown_connections 2024-04-23T18:36:52.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f741b7c8740 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5652fd29d070 0x5652fd29f490 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:52.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f741b7c8740 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5652fd226800 0x5652fd22c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:52.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f741b7c8740 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5652fd21bf40 0x5652fd22bd90 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:52.856 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f73f9ffb640 0 client.17754 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:52.863 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f741b7c8740 1 --2- 172.21.15.50:0/3038671992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652fd21b620 0x5652fd22ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f741b7c8740 1 -- 172.21.15.50:0/3038671992 >> 172.21.15.50:0/3038671992 conn(0x5652fd217b30 msgr2=0x5652fd216420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:52.881 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f741b7c8740 1 -- 172.21.15.50:0/3038671992 shutdown_connections 2024-04-23T18:36:52.891 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.724+0000 7f741b7c8740 1 -- 172.21.15.50:0/3038671992 wait complete. 2024-04-23T18:36:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f741b7c8740 20 client.17754 trim_cache size 0 max 16384 2024-04-23T18:36:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:36:52.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:36:52.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:36:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:36:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f74189d8640 1 -- 172.21.15.50:0/1235858203 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f73fc0a3e40 msgr2=0x7f73fc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:53.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f74189d8640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f73fc0a3e40 0x7f73fc0a6300 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f740c09cd90 tx=0x7f740c00af40 comp rx=0 tx=0).stop 2024-04-23T18:36:53.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f74189d8640 1 -- 172.21.15.50:0/1235858203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f741410c5d0 msgr2=0x7f74141018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:53.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f74189d8640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f741410c5d0 0x7f74141018d0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f740c00eb40 tx=0x7f740c00c2d0 comp rx=0 tx=0).stop 2024-04-23T18:36:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f74189d8640 1 -- 172.21.15.50:0/1235858203 shutdown_connections 2024-04-23T18:36:53.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f74189d8640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f73fc0a3e40 0x7f73fc0a6300 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:53.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f74189d8640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f741410c5d0 0x7f74141018d0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f74189d8640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7414103b20 0x7f7414101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:53.194 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f74189d8640 1 --2- 172.21.15.50:0/1235858203 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7414103200 0x7f7414100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:53.202 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f74189d8640 1 -- 172.21.15.50:0/1235858203 >> 172.21.15.50:0/1235858203 conn(0x7f74140771d0 msgr2=0x7f74140fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:53.208 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f74189d8640 1 -- 172.21.15.50:0/1235858203 shutdown_connections 2024-04-23T18:36:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:36:48.725+0000 7f74189d8640 1 -- 172.21.15.50:0/1235858203 wait complete. 2024-04-23T18:36:53.228 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 20.51 MB/sec execute 203 sec latency 18651.583 ms 2024-04-23T18:36:53.236 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 20.41 MB/sec execute 204 sec latency 19651.634 ms 2024-04-23T18:36:53.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:50 smithi050 ceph-mon[29187]: pgmap v582: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 597 KiB/s wr, 18 op/s 2024-04-23T18:36:53.253 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 20.31 MB/sec execute 205 sec latency 20651.719 ms 2024-04-23T18:36:53.261 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 20.21 MB/sec execute 206 sec latency 21651.830 ms 2024-04-23T18:36:53.305 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:52 smithi188 ceph-mon[32694]: pgmap v583: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 597 KiB/s wr, 18 op/s 2024-04-23T18:36:53.348 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:52 smithi177 ceph-mon[32308]: pgmap v583: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 597 KiB/s wr, 18 op/s 2024-04-23T18:36:53.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:52 smithi050 ceph-mon[29187]: pgmap v583: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 597 KiB/s wr, 18 op/s 2024-04-23T18:36:53.575 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 20.12 MB/sec execute 207 sec latency 22651.883 ms 2024-04-23T18:36:54.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:54 smithi177 ceph-mon[32308]: pgmap v584: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 877 KiB/s wr, 44 op/s 2024-04-23T18:36:54.575 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 20.02 MB/sec execute 208 sec latency 23651.941 ms 2024-04-23T18:36:54.583 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:54 smithi188 ceph-mon[32694]: pgmap v584: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 877 KiB/s wr, 44 op/s 2024-04-23T18:36:54.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:54 smithi050 ceph-mon[29187]: pgmap v584: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 877 KiB/s wr, 44 op/s 2024-04-23T18:36:55.575 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 19.92 MB/sec execute 209 sec latency 24651.979 ms 2024-04-23T18:36:56.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:36:56.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.330+0000 7fa68c387640 1 -- 172.21.15.50:0/2733401882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684101ac0 msgr2=0x7fa684105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:56.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.330+0000 7fa68c387640 1 --2- 172.21.15.50:0/2733401882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684101ac0 0x7fa684105ab0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fa674009900 tx=0x7fa67402a410 comp rx=0 tx=0).stop 2024-04-23T18:36:56.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.330+0000 7fa68c387640 1 -- 172.21.15.50:0/2733401882 shutdown_connections 2024-04-23T18:36:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.330+0000 7fa68c387640 1 --2- 172.21.15.50:0/2733401882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa684106190 0x7fa68410dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:56.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.330+0000 7fa68c387640 1 --2- 172.21.15.50:0/2733401882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684101ac0 0x7fa684105ab0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:56.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.330+0000 7fa68c387640 1 --2- 172.21.15.50:0/2733401882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa684101110 0x7fa6841014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:56.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.330+0000 7fa68c387640 1 -- 172.21.15.50:0/2733401882 >> 172.21.15.50:0/2733401882 conn(0x7fa684077fb0 msgr2=0x7fa6840783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:56.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.330+0000 7fa68c387640 1 -- 172.21.15.50:0/2733401882 shutdown_connections 2024-04-23T18:36:56.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.330+0000 7fa68c387640 1 -- 172.21.15.50:0/2733401882 wait complete. 2024-04-23T18:36:56.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.331+0000 7fa68c387640 1 Processor -- start 2024-04-23T18:36:56.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.331+0000 7fa68c387640 1 -- start start 2024-04-23T18:36:56.590 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:56 smithi188 ceph-mon[32694]: pgmap v585: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 858 KiB/s wr, 38 op/s 2024-04-23T18:36:56.736 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:56 smithi177 ceph-mon[32308]: pgmap v585: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 858 KiB/s wr, 38 op/s 2024-04-23T18:36:56.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.331+0000 7fa68c387640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa684101110 0x7fa684102b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:56.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.331+0000 7fa68c387640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684106190 0x7fa684103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.331+0000 7fa68c387640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6841035d0 0x7fa6841a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:56.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.331+0000 7fa68c387640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841110d0 con 0x7fa684106190 2024-04-23T18:36:56.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.331+0000 7fa68c387640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa684110f50 con 0x7fa6841035d0 2024-04-23T18:36:56.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.331+0000 7fa68c387640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa684111250 con 0x7fa684101110 2024-04-23T18:36:56.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.331+0000 7fa689922640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684106190 0x7fa684103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:56.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.331+0000 7fa689922640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684106190 0x7fa684103090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35952/0 (socket says 172.21.15.50:35952) 2024-04-23T18:36:56.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.331+0000 7fa689922640 1 -- 172.21.15.50:0/899485544 learned_addr learned my addr 172.21.15.50:0/899485544 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:56.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.331+0000 7fa68a123640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa684101110 0x7fa684102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:56.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.332+0000 7fa68a924640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6841035d0 0x7fa6841a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:56.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.332+0000 7fa689922640 1 -- 172.21.15.50:0/899485544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa684101110 msgr2=0x7fa684102b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:56.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.332+0000 7fa689922640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa684101110 0x7fa684102b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:56.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.332+0000 7fa689922640 1 -- 172.21.15.50:0/899485544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6841035d0 msgr2=0x7fa6841a6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:56.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.332+0000 7fa689922640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6841035d0 0x7fa6841a6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:56.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.332+0000 7fa689922640 1 -- 172.21.15.50:0/899485544 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6841a7070 con 0x7fa684106190 2024-04-23T18:36:57.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.332+0000 7fa689922640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684106190 0x7fa684103090 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fa674006010 tx=0x7fa674004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:57.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.332+0000 7fa67b7fe640 1 -- 172.21.15.50:0/899485544 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67403f030 con 0x7fa684106190 2024-04-23T18:36:57.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.332+0000 7fa67b7fe640 1 -- 172.21.15.50:0/899485544 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa674003aa0 con 0x7fa684106190 2024-04-23T18:36:57.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.332+0000 7fa68c387640 1 -- 172.21.15.50:0/899485544 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6841a7300 con 0x7fa684106190 2024-04-23T18:36:57.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.332+0000 7fa67b7fe640 1 -- 172.21.15.50:0/899485544 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa674035070 con 0x7fa684106190 2024-04-23T18:36:57.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.332+0000 7fa68c387640 1 -- 172.21.15.50:0/899485544 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6841a77f0 con 0x7fa684106190 2024-04-23T18:36:57.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.333+0000 7fa68c387640 1 -- 172.21.15.50:0/899485544 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa684102590 con 0x7fa684106190 2024-04-23T18:36:57.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.335+0000 7fa67b7fe640 1 -- 172.21.15.50:0/899485544 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa674042030 con 0x7fa684106190 2024-04-23T18:36:57.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.335+0000 7fa67b7fe640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa6700a3d50 0x7fa6700a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:57.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.335+0000 7fa68a123640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa6700a3d50 0x7fa6700a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:57.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.335+0000 7fa67b7fe640 1 -- 172.21.15.50:0/899485544 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7fa6740f0840 con 0x7fa684106190 2024-04-23T18:36:57.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.336+0000 7fa68a123640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa6700a3d50 0x7fa6700a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa68000bbe0 tx=0x7fa68000b390 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:57.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.340+0000 7fa67b7fe640 1 -- 172.21.15.50:0/899485544 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa6740bc840 con 0x7fa684106190 2024-04-23T18:36:57.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.566+0000 7fa68c387640 1 -- 172.21.15.50:0/899485544 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa68410a3d0 con 0x7fa684106190 2024-04-23T18:36:57.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.566+0000 7fa67b7fe640 1 -- 172.21.15.50:0/899485544 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+11875 (secure 0 0 0) 0x7fa6740c0b40 con 0x7fa684106190 2024-04-23T18:36:57.285 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:36:57.309 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":62,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:36:29.984583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27194":{"gid":27194,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2754597719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2754597719},{"type":"v1","addr":"172.21.15.177:6837","nonce":2754597719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:36:57.357 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 62 2024-04-23T18:36:57.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.568+0000 7fa68c387640 1 -- 172.21.15.50:0/899485544 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa6700a3d50 msgr2=0x7fa6700a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:57.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.568+0000 7fa68c387640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa6700a3d50 0x7fa6700a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa68000bbe0 tx=0x7fa68000b390 comp rx=0 tx=0).stop 2024-04-23T18:36:57.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.568+0000 7fa68c387640 1 -- 172.21.15.50:0/899485544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684106190 msgr2=0x7fa684103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:57.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.568+0000 7fa68c387640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684106190 0x7fa684103090 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fa674006010 tx=0x7fa674004060 comp rx=0 tx=0).stop 2024-04-23T18:36:57.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.568+0000 7fa68c387640 1 -- 172.21.15.50:0/899485544 shutdown_connections 2024-04-23T18:36:57.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.568+0000 7fa68c387640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fa6700a3d50 0x7fa6700a6210 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:57.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.568+0000 7fa68c387640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6841035d0 0x7fa6841a6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:57.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.568+0000 7fa68c387640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684106190 0x7fa684103090 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:57.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.568+0000 7fa68c387640 1 --2- 172.21.15.50:0/899485544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa684101110 0x7fa684102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:57.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.568+0000 7fa68c387640 1 -- 172.21.15.50:0/899485544 >> 172.21.15.50:0/899485544 conn(0x7fa684077fb0 msgr2=0x7fa68406a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:57.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.569+0000 7fa68c387640 1 -- 172.21.15.50:0/899485544 shutdown_connections 2024-04-23T18:36:57.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:56.569+0000 7fa68c387640 1 -- 172.21.15.50:0/899485544 wait complete. 2024-04-23T18:36:57.550 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:56 smithi050 ceph-mon[29187]: pgmap v585: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 858 KiB/s wr, 38 op/s 2024-04-23T18:36:57.559 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id c3d7e5f6 --release --await' 2024-04-23T18:36:57.567 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id c3d7e5f6 --release --await 2024-04-23T18:36:57.602 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 19.83 MB/sec execute 210 sec latency 25652.010 ms 2024-04-23T18:36:57.656 INFO:tasks.workunit.client.0.smithi050.stdout: 10 129639 19.73 MB/sec execute 211 sec latency 26652.112 ms 2024-04-23T18:36:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:57 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/899485544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:36:57.675 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:57 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/899485544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:36:57.691 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:57 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/899485544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:36:57.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.793+0000 7ff5add4f640 1 -- 172.21.15.50:0/3828106331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a8100450 msgr2=0x7ff5a81008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.793+0000 7ff5add4f640 1 --2- 172.21.15.50:0/3828106331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a8100450 0x7ff5a81008d0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7ff59c009930 tx=0x7ff59c02a420 comp rx=0 tx=0).stop 2024-04-23T18:36:57.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.793+0000 7ff5add4f640 1 -- 172.21.15.50:0/3828106331 shutdown_connections 2024-04-23T18:36:57.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.793+0000 7ff5add4f640 1 --2- 172.21.15.50:0/3828106331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5a8100e10 0x7ff5a810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:57.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.793+0000 7ff5add4f640 1 --2- 172.21.15.50:0/3828106331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a8100450 0x7ff5a81008d0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.793+0000 7ff5add4f640 1 --2- 172.21.15.50:0/3828106331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5a8106f30 0x7ff5a8107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:57.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.793+0000 7ff5add4f640 1 -- 172.21.15.50:0/3828106331 >> 172.21.15.50:0/3828106331 conn(0x7ff5a8077410 msgr2=0x7ff5a80fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:57.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.793+0000 7ff5add4f640 1 -- 172.21.15.50:0/3828106331 shutdown_connections 2024-04-23T18:36:57.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.794+0000 7ff5add4f640 1 -- 172.21.15.50:0/3828106331 wait complete. 2024-04-23T18:36:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.794+0000 7ff5add4f640 1 Processor -- start 2024-04-23T18:36:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.794+0000 7ff5add4f640 1 -- start start 2024-04-23T18:36:57.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.794+0000 7ff5add4f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5a8100450 0x7ff5a8101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:58.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.794+0000 7ff5add4f640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5a8100e10 0x7ff5a8101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:58.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5add4f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a8106f30 0x7ff5a81020b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:58.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5add4f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81122e0 con 0x7ff5a8106f30 2024-04-23T18:36:58.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5add4f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a8112160 con 0x7ff5a8100450 2024-04-23T18:36:58.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5add4f640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a8112460 con 0x7ff5a8100e10 2024-04-23T18:36:58.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a8106f30 0x7ff5a81020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:58.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a8106f30 0x7ff5a81020b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35976/0 (socket says 172.21.15.50:35976) 2024-04-23T18:36:58.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a7fff640 1 -- 172.21.15.50:0/736295393 learned_addr learned my addr 172.21.15.50:0/736295393 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:58.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5a8100450 0x7ff5a8101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:58.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5a8100e10 0x7ff5a8101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:58.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a7fff640 1 -- 172.21.15.50:0/736295393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5a8100e10 msgr2=0x7ff5a8101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:58.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a7fff640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5a8100e10 0x7ff5a8101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:58.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a7fff640 1 -- 172.21.15.50:0/736295393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5a8100450 msgr2=0x7ff5a8101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:58.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a7fff640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5a8100450 0x7ff5a8101630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:58.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a7fff640 1 -- 172.21.15.50:0/736295393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a81a6d20 con 0x7ff5a8106f30 2024-04-23T18:36:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a7fff640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a8106f30 0x7ff5a81020b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000eb10 tx=0x7ff5a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a4ff9640 1 -- 172.21.15.50:0/736295393 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0004b20 con 0x7ff5a8106f30 2024-04-23T18:36:58.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a4ff9640 1 -- 172.21.15.50:0/736295393 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5a0004cc0 con 0x7ff5a8106f30 2024-04-23T18:36:58.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5add4f640 1 -- 172.21.15.50:0/736295393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a81a7010 con 0x7ff5a8106f30 2024-04-23T18:36:58.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5a4ff9640 1 -- 172.21.15.50:0/736295393 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0010670 con 0x7ff5a8106f30 2024-04-23T18:36:58.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.795+0000 7ff5add4f640 1 -- 172.21.15.50:0/736295393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a81a74e0 con 0x7ff5a8106f30 2024-04-23T18:36:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.798+0000 7ff5a4ff9640 1 -- 172.21.15.50:0/736295393 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff5a0005020 con 0x7ff5a8106f30 2024-04-23T18:36:58.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.798+0000 7ff5b0a6d740 1 Processor -- start 2024-04-23T18:36:58.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.798+0000 7ff5b0a6d740 1 -- start start 2024-04-23T18:36:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.798+0000 7ff5b0a6d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf828a490 0x559cf828a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:58.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.798+0000 7ff5b0a6d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 0x559cf8294f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:58.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.798+0000 7ff5b0a6d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf8295480 0x559cf8297870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:58.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.798+0000 7ff5b0a6d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559cf81ef6d0 con 0x559cf828adb0 2024-04-23T18:36:58.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.798+0000 7ff5b0a6d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559cf818c1a0 con 0x559cf828a490 2024-04-23T18:36:58.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.798+0000 7ff5b0a6d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559cf81eb690 con 0x559cf8295480 2024-04-23T18:36:58.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a4ff9640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5900a3da0 0x7ff5900a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:58.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 0x559cf8294f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:58.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 0x559cf8294f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35984/0 (socket says 172.21.15.50:35984) 2024-04-23T18:36:58.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a77fe640 1 -- 172.21.15.50:0/1666567804 learned_addr learned my addr 172.21.15.50:0/1666567804 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:58.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/1666567804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf8295480 0x559cf8297870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:58.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a7fff640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5900a3da0 0x7ff5900a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:58.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/1666567804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf828a490 0x559cf828a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:58.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a4ff9640 1 -- 172.21.15.50:0/736295393 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7ff5a00d0db0 con 0x7ff5a8106f30 2024-04-23T18:36:58.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a77fe640 1 -- 172.21.15.50:0/1666567804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf8295480 msgr2=0x559cf8297870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:58.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/1666567804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf8295480 0x559cf8297870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:58.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a77fe640 1 -- 172.21.15.50:0/1666567804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf828a490 msgr2=0x559cf828a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:58.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/1666567804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf828a490 0x559cf828a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:58.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a77fe640 1 -- 172.21.15.50:0/1666567804 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cf81965b0 con 0x559cf828adb0 2024-04-23T18:36:58.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/1666567804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf8295480 0x559cf8297870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:36:58.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/1666567804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 0x559cf8294f40 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ff59800b690 tx=0x7ff59800bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1297ec9b9dfe4c2c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:58.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff5a7fff640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5900a3da0 0x7ff5900a6260 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff5a009ccd0 tx=0x7ff5a00bd040 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:58.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff58dffb640 1 -- 172.21.15.50:0/1666567804 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598003bd0 con 0x559cf828adb0 2024-04-23T18:36:58.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff58dffb640 1 -- 172.21.15.50:0/1666567804 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff598015070 con 0x559cf828adb0 2024-04-23T18:36:58.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.799+0000 7ff58dffb640 1 -- 172.21.15.50:0/1666567804 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598004d10 con 0x559cf828adb0 2024-04-23T18:36:58.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.800+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/1666567804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 msgr2=0x559cf8294f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:58.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.800+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/1666567804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 0x559cf8294f40 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ff59800b690 tx=0x7ff59800bb50 comp rx=0 tx=0).stop 2024-04-23T18:36:58.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.800+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/1666567804 shutdown_connections 2024-04-23T18:36:59.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.800+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/1666567804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf8295480 0x559cf8297870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:59.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.800+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/1666567804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 0x559cf8294f40 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:59.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.800+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/1666567804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf828a490 0x559cf828a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:59.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.800+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/1666567804 >> 172.21.15.50:0/1666567804 conn(0x559cf82867b0 msgr2=0x559cf8286b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:59.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.800+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/1666567804 shutdown_connections 2024-04-23T18:36:59.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.800+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/1666567804 wait complete. 2024-04-23T18:36:59.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.800+0000 7ff5b0a6d740 1 -- start start 2024-04-23T18:36:59.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5b0a6d740 1 Processor -- start 2024-04-23T18:36:59.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5b0a6d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 0x559cf8294ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:59.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5b0a6d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf828adb0 0x559cf82921c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:59.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5b0a6d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf8295480 0x559cf8292900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:59.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5b0a6d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559cf8298310 con 0x559cf828a490 2024-04-23T18:36:59.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5b0a6d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559cf8189b20 con 0x559cf8295480 2024-04-23T18:36:59.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5b0a6d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559cf8298490 con 0x559cf828adb0 2024-04-23T18:36:59.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5b0a6d740 10 client.0 _ll_register_callbacks cb 0x559cf8189530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:36:59.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5a6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 0x559cf8294ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:59.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5a6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 0x559cf8294ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35990/0 (socket says 172.21.15.50:35990) 2024-04-23T18:36:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5a6ffd640 1 -- 172.21.15.50:0/2651010630 learned_addr learned my addr 172.21.15.50:0/2651010630 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:36:59.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf8295480 0x559cf8292900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:59.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf828adb0 0x559cf82921c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:59.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5a6ffd640 1 -- 172.21.15.50:0/2651010630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf828adb0 msgr2=0x559cf82921c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:59.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf828adb0 0x559cf82921c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:59.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5a6ffd640 1 -- 172.21.15.50:0/2651010630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf8295480 msgr2=0x559cf8292900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:36:59.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf8295480 0x559cf8292900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:59.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.801+0000 7ff5a6ffd640 1 -- 172.21.15.50:0/2651010630 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cf7f01920 con 0x559cf828a490 2024-04-23T18:36:59.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 0x559cf8294ed0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7ff59c0096c0 tx=0x7ff59c02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26edf1d1038684a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:36:59.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff586ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:36:59.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff586ffd640 1 -- 172.21.15.50:0/2651010630 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c035040 con 0x559cf828a490 2024-04-23T18:36:59.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff586ffd640 1 -- 172.21.15.50:0/2651010630 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7ff59c003b90 con 0x559cf828a490 2024-04-23T18:36:59.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/2651010630 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff598015070 con 0x559cf828a490 2024-04-23T18:36:59.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff586ffd640 1 -- 172.21.15.50:0/2651010630 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7ff59c003d90 con 0x559cf828a490 2024-04-23T18:36:59.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff5b0a6d740 10 client.17772 fetch_fsmap learned FSMap version 62 2024-04-23T18:36:59.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/2651010630 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x559cf7fcdd30 con 0x559cf828a490 2024-04-23T18:36:59.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff586ffd640 1 -- 172.21.15.50:0/2651010630 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff598015070 con 0x559cf828a490 2024-04-23T18:36:59.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff586ffd640 1 -- 172.21.15.50:0/2651010630 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7ff59c03a040 con 0x559cf828a490 2024-04-23T18:36:59.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff5b0a6d740 10 client.17772 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:36:59.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff5b0a6d740 10 client.17772 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:36:59.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff5b0a6d740 20 client.17772 populate_metadata read hostname 'smithi050' 2024-04-23T18:36:59.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559cf82a55a0 0x559cf82a79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:36:59.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff5b0a6d740 4 client.17772 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:36:59.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.802+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/2651010630 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559cf81ad500 con 0x559cf82a55a0 2024-04-23T18:36:59.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.803+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559cf82a55a0 0x559cf82a79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:36:59.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.803+0000 7ff586ffd640 10 client.17772 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.803+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559cf82a55a0 0x559cf82a79b0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60b5ac296b9e6a4b server_cookie=cab4383ed0e39dfa in_seq=0 out_seq=0 2024-04-23T18:36:59.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff586ffd640 1 -- 172.21.15.50:0/2651010630 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559cf81ad500 con 0x559cf82a55a0 2024-04-23T18:36:59.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff586ffd640 10 client.17772 handle_command_reply: tid=0 2024-04-23T18:36:59.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff5b0a6d740 1 client.17772 shutdown 2024-04-23T18:36:59.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/2651010630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 msgr2=0x559cf8294ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:36:59.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 0x559cf8294ed0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7ff59c0096c0 tx=0x7ff59c02a420 comp rx=0 tx=0).stop 2024-04-23T18:36:59.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/2651010630 shutdown_connections 2024-04-23T18:36:59.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559cf82a55a0 0x559cf82a79b0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:59.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf8295480 0x559cf8292900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:59.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff586ffd640 0 client.17772 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:36:59.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf828adb0 0x559cf82921c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:59.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/2651010630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 0x559cf8294ed0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:36:59.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/2651010630 >> 172.21.15.50:0/2651010630 conn(0x559cf82867b0 msgr2=0x559cf82859d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:36:59.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/2651010630 shutdown_connections 2024-04-23T18:36:59.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/2651010630 wait complete. 2024-04-23T18:37:00.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.804+0000 7ff5b0a6d740 20 client.17772 trim_cache size 0 max 16384 2024-04-23T18:37:00.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.821+0000 7ff5b0a6d740 1 Processor -- start 2024-04-23T18:37:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5b0a6d740 1 -- start start 2024-04-23T18:37:00.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5b0a6d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf828a490 0x559cf8286150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:00.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5b0a6d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 0x559cf8294a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5b0a6d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf8295480 0x559cf8294f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:00.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5b0a6d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81b2690 con 0x559cf828adb0 2024-04-23T18:37:00.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5b0a6d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81aff60 con 0x559cf828a490 2024-04-23T18:37:00.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5b0a6d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81addb0 con 0x559cf8295480 2024-04-23T18:37:00.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 0x559cf8294a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:00.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 0x559cf8294a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36004/0 (socket says 172.21.15.50:36004) 2024-04-23T18:37:00.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5a77fe640 1 -- 172.21.15.50:0/3581046662 learned_addr learned my addr 172.21.15.50:0/3581046662 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:00.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/3581046662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf828a490 0x559cf8286150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:00.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/3581046662 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf8295480 0x559cf8294f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:00.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5a77fe640 1 -- 172.21.15.50:0/3581046662 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf8295480 msgr2=0x559cf8294f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:00.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/3581046662 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf8295480 0x559cf8294f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:00.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5a77fe640 1 -- 172.21.15.50:0/3581046662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf828a490 msgr2=0x559cf8286150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:00.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/3581046662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf828a490 0x559cf8286150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:00.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5a77fe640 1 -- 172.21.15.50:0/3581046662 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cf8290710 con 0x559cf828adb0 2024-04-23T18:37:00.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/3581046662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 0x559cf8294a00 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff59800b690 tx=0x7ff5980047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2bd786eff6957cd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:00.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.822+0000 7ff58d7fa640 1 -- 172.21.15.50:0/3581046662 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598030070 con 0x559cf828adb0 2024-04-23T18:37:00.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.823+0000 7ff58d7fa640 1 -- 172.21.15.50:0/3581046662 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff598015040 con 0x559cf828adb0 2024-04-23T18:37:00.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.823+0000 7ff58d7fa640 1 -- 172.21.15.50:0/3581046662 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5980116a0 con 0x559cf828adb0 2024-04-23T18:37:00.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.823+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/3581046662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 msgr2=0x559cf8294a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:00.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.823+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/3581046662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 0x559cf8294a00 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff59800b690 tx=0x7ff5980047e0 comp rx=0 tx=0).stop 2024-04-23T18:37:00.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.823+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/3581046662 shutdown_connections 2024-04-23T18:37:00.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.823+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/3581046662 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf8295480 0x559cf8294f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:00.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.823+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/3581046662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828adb0 0x559cf8294a00 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:00.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.823+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/3581046662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf828a490 0x559cf8286150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:00.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.823+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/3581046662 >> 172.21.15.50:0/3581046662 conn(0x559cf82867b0 msgr2=0x559cf828d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:00.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.823+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/3581046662 shutdown_connections 2024-04-23T18:37:00.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.823+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/3581046662 wait complete. 2024-04-23T18:37:00.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.823+0000 7ff5b0a6d740 1 -- start start 2024-04-23T18:37:00.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5b0a6d740 1 Processor -- start 2024-04-23T18:37:00.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5b0a6d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 0x559cf82964d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:00.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5b0a6d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf828adb0 0x559cf8296a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:00.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5b0a6d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf8295480 0x559cf829bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:00.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5b0a6d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81ae420 con 0x559cf828a490 2024-04-23T18:37:00.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5b0a6d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81af890 con 0x559cf8295480 2024-04-23T18:37:00.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5b0a6d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81aeaf0 con 0x559cf828adb0 2024-04-23T18:37:00.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5b0a6d740 10 client.0 _ll_register_callbacks cb 0x7ff59c003610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:37:00.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5a6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 0x559cf82964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:00.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5a6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 0x559cf82964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36020/0 (socket says 172.21.15.50:36020) 2024-04-23T18:37:00.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5a6ffd640 1 -- 172.21.15.50:0/649240830 learned_addr learned my addr 172.21.15.50:0/649240830 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:00.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf828adb0 0x559cf8296a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:00.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf8295480 0x559cf829bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:00.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5a6ffd640 1 -- 172.21.15.50:0/649240830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf828adb0 msgr2=0x559cf8296a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:00.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf828adb0 0x559cf8296a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:00.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5a6ffd640 1 -- 172.21.15.50:0/649240830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf8295480 msgr2=0x559cf829bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf8295480 0x559cf829bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.824+0000 7ff5a6ffd640 1 -- 172.21.15.50:0/649240830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cf82972a0 con 0x559cf828a490 2024-04-23T18:37:00.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff5a6ffd640 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 0x559cf82964d0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff59c003720 tx=0x7ff59c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add63a8b5fe50e54 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:00.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff586ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:37:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff586ffd640 1 -- 172.21.15.50:0/649240830 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c035070 con 0x559cf828a490 2024-04-23T18:37:00.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/649240830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff598015040 con 0x559cf828a490 2024-04-23T18:37:00.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff586ffd640 1 -- 172.21.15.50:0/649240830 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7ff59c003b90 con 0x559cf828a490 2024-04-23T18:37:01.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff586ffd640 1 -- 172.21.15.50:0/649240830 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7ff59c04cd30 con 0x559cf828a490 2024-04-23T18:37:01.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff5b0a6d740 10 client.17778 fetch_fsmap learned FSMap version 62 2024-04-23T18:37:01.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/649240830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x559cf830c130 con 0x559cf828a490 2024-04-23T18:37:01.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff586ffd640 1 -- 172.21.15.50:0/649240830 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff598015040 con 0x559cf828a490 2024-04-23T18:37:01.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff586ffd640 1 -- 172.21.15.50:0/649240830 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7ff59c05d020 con 0x559cf828a490 2024-04-23T18:37:01.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff5b0a6d740 10 client.17778 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:37:01.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff5b0a6d740 10 client.17778 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:37:01.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff5b0a6d740 20 client.17778 populate_metadata read hostname 'smithi050' 2024-04-23T18:37:01.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559cf830c790 0x559cf830eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:01.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff5b0a6d740 4 client.17778 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c3d7e5f6", "release": true, "await": true}] 2024-04-23T18:37:01.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.825+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/649240830 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c3d7e5f6", "release": true, "await": true}) -- 0x559cf81fd0f0 con 0x559cf830c790 2024-04-23T18:37:01.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.826+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559cf830c790 0x559cf830eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:01.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.826+0000 7ff5a77fe640 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559cf830c790 0x559cf830eb80 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e5d21b26f2a75e5 server_cookie=9fb94cd1fde3626 in_seq=0 out_seq=0 2024-04-23T18:37:01.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.826+0000 7ff586ffd640 10 client.17778 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:01.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.851+0000 7ff586ffd640 1 -- 172.21.15.50:0/649240830 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x559cf81fd0f0 con 0x559cf830c790 2024-04-23T18:37:01.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.851+0000 7ff586ffd640 10 client.17778 handle_command_reply: tid=0 2024-04-23T18:37:01.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.851+0000 7ff5b0a6d740 1 client.17778 shutdown 2024-04-23T18:37:01.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/649240830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 msgr2=0x559cf82964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:01.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 0x559cf82964d0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff59c003720 tx=0x7ff59c004040 comp rx=0 tx=0).stop 2024-04-23T18:37:01.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/649240830 shutdown_connections 2024-04-23T18:37:01.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559cf830c790 0x559cf830eb80 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:01.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cf8295480 0x559cf829bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:01.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff586ffd640 0 client.17778 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:01.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559cf828adb0 0x559cf8296a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:01.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5b0a6d740 1 --2- 172.21.15.50:0/649240830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559cf828a490 0x559cf82964d0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:01.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/649240830 >> 172.21.15.50:0/649240830 conn(0x559cf82867b0 msgr2=0x559cf828bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:01.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/649240830 shutdown_connections 2024-04-23T18:37:01.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5b0a6d740 1 -- 172.21.15.50:0/649240830 wait complete. 2024-04-23T18:37:01.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5b0a6d740 20 client.17778 trim_cache size 0 max 16384 2024-04-23T18:37:01.791 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:37:01.798 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 62, 2024-04-23T18:37:01.806 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:37:01.814 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 64, 2024-04-23T18:37:01.825 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:37:01.871 INFO:teuthology.orchestra.run.smithi050.stdout: "c3d7e5f6": { 2024-04-23T18:37:01.976 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 64, 2024-04-23T18:37:01.986 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:37:01.994 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:37:02.003 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:37:02.052 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:37:02.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:37:02.070 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:37:02.078 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 79.7, 2024-04-23T18:37:02.088 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:37:02.097 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:37:02.105 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:37:02.113 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:37:02.121 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:37:02.129 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:37:02.136 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:37:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:37:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:37:02.242 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:37:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:37:02.260 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:37:02.286 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:58 smithi188 ceph-mon[32694]: pgmap v586: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 2.3 MiB/s wr, 90 op/s 2024-04-23T18:37:02.326 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:36:59 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:37:02.337 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:00 smithi188 ceph-mon[32694]: pgmap v587: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-04-23T18:37:02.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:00 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:37:02.356 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:00 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:37:02.365 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:00 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:37:02.373 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:58 smithi177 ceph-mon[32308]: pgmap v586: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 2.3 MiB/s wr, 90 op/s 2024-04-23T18:37:02.382 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:36:59 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:37:02.391 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:00 smithi177 ceph-mon[32308]: pgmap v587: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-04-23T18:37:02.402 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:00 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:37:02.411 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:00 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:37:02.420 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:00 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:37:02.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5add4f640 1 -- 172.21.15.50:0/736295393 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5900a3da0 msgr2=0x7ff5900a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:02.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5add4f640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5900a3da0 0x7ff5900a6260 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff5a009ccd0 tx=0x7ff5a00bd040 comp rx=0 tx=0).stop 2024-04-23T18:37:02.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5add4f640 1 -- 172.21.15.50:0/736295393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a8106f30 msgr2=0x7ff5a81020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:02.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.852+0000 7ff5add4f640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a8106f30 0x7ff5a81020b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000eb10 tx=0x7ff5a000c2d0 comp rx=0 tx=0).stop 2024-04-23T18:37:02.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.853+0000 7ff5add4f640 1 -- 172.21.15.50:0/736295393 shutdown_connections 2024-04-23T18:37:02.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.853+0000 7ff5add4f640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5900a3da0 0x7ff5900a6260 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:02.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.853+0000 7ff5add4f640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a8106f30 0x7ff5a81020b0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:02.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.853+0000 7ff5add4f640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5a8100e10 0x7ff5a8101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:02.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.853+0000 7ff5add4f640 1 --2- 172.21.15.50:0/736295393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5a8100450 0x7ff5a8101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:02.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.853+0000 7ff5add4f640 1 -- 172.21.15.50:0/736295393 >> 172.21.15.50:0/736295393 conn(0x7ff5a8077410 msgr2=0x7ff5a8105110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:02.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.853+0000 7ff5add4f640 1 -- 172.21.15.50:0/736295393 shutdown_connections 2024-04-23T18:37:02.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:36:57.853+0000 7ff5add4f640 1 -- 172.21.15.50:0/736295393 wait complete. 2024-04-23T18:37:02.640 INFO:tasks.workunit.client.0.smithi050.stdout: 10 130034 19.77 MB/sec execute 212 sec latency 26953.378 ms 2024-04-23T18:37:02.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:58 smithi050 ceph-mon[29187]: pgmap v586: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 2.3 MiB/s wr, 90 op/s 2024-04-23T18:37:02.658 INFO:tasks.workunit.client.0.smithi050.stdout: 10 130484 19.78 MB/sec execute 213 sec latency 144.729 ms 2024-04-23T18:37:02.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:36:59 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:37:02.675 INFO:tasks.workunit.client.0.smithi050.stdout: 10 130782 19.75 MB/sec execute 214 sec latency 399.478 ms 2024-04-23T18:37:02.684 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:00 smithi050 ceph-mon[29187]: pgmap v587: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-04-23T18:37:02.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:00 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:37:02.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:00 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:37:02.770 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:00 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:37:02.779 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c3d7e5f6', seconds elapsed: 40.7 2024-04-23T18:37:02.798 INFO:tasks.workunit.client.0.smithi050.stdout: 10 130782 19.66 MB/sec execute 215 sec latency 1151.141 ms 2024-04-23T18:37:02.835 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:02 smithi188 ceph-mon[32694]: pgmap v588: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-04-23T18:37:02.845 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:02 smithi177 ceph-mon[32308]: pgmap v588: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-04-23T18:37:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.825+0000 7f63cf810640 1 -- 172.21.15.50:0/3337415930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63c8074c20 msgr2=0x7f63c810e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:02.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.825+0000 7f63cf810640 1 --2- 172.21.15.50:0/3337415930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63c8074c20 0x7f63c810e2d0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009900 tx=0x7f63b802a480 comp rx=0 tx=0).stop 2024-04-23T18:37:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.825+0000 7f63cf810640 1 -- 172.21.15.50:0/3337415930 shutdown_connections 2024-04-23T18:37:02.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.825+0000 7f63cf810640 1 --2- 172.21.15.50:0/3337415930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63c810e810 0x7f63c8110c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.825+0000 7f63cf810640 1 --2- 172.21.15.50:0/3337415930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63c8074c20 0x7f63c810e2d0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:02.908 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.825+0000 7f63cf810640 1 --2- 172.21.15.50:0/3337415930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c8074300 0x7f63c80746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:02.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.825+0000 7f63cf810640 1 -- 172.21.15.50:0/3337415930 >> 172.21.15.50:0/3337415930 conn(0x7f63c80fda20 msgr2=0x7f63c80ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.825+0000 7f63cf810640 1 -- 172.21.15.50:0/3337415930 shutdown_connections 2024-04-23T18:37:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.825+0000 7f63cf810640 1 -- 172.21.15.50:0/3337415930 wait complete. 2024-04-23T18:37:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.825+0000 7f63cf810640 1 Processor -- start 2024-04-23T18:37:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cf810640 1 -- start start 2024-04-23T18:37:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cf810640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63c8074300 0x7f63c81a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cf810640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c8074c20 0x7f63c81a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:03.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cf810640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63c810e810 0x7f63c819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cf810640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f63c81143d0 con 0x7f63c810e810 2024-04-23T18:37:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cf810640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f63c8114250 con 0x7f63c8074300 2024-04-23T18:37:03.127 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cf810640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f63c8114550 con 0x7f63c8074c20 2024-04-23T18:37:03.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cddad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63c810e810 0x7f63c819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cddad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63c810e810 0x7f63c819a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58046/0 (socket says 172.21.15.50:58046) 2024-04-23T18:37:03.153 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cddad640 1 -- 172.21.15.50:0/1663930288 learned_addr learned my addr 172.21.15.50:0/1663930288 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:03.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63ccdab640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c8074c20 0x7f63c81a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cd5ac640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63c8074300 0x7f63c81a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cddad640 1 -- 172.21.15.50:0/1663930288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c8074c20 msgr2=0x7f63c81a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cddad640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c8074c20 0x7f63c81a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:03.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cddad640 1 -- 172.21.15.50:0/1663930288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63c8074300 msgr2=0x7f63c81a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cddad640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63c8074300 0x7f63c81a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cddad640 1 -- 172.21.15.50:0/1663930288 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c819afd0 con 0x7f63c810e810 2024-04-23T18:37:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cd5ac640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63c8074300 0x7f63c81a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:37:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.826+0000 7f63cddad640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63c810e810 0x7f63c819a710 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f63c000eb10 tx=0x7f63c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.827+0000 7f63be7fc640 1 -- 172.21.15.50:0/1663930288 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0004b20 con 0x7f63c810e810 2024-04-23T18:37:03.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.827+0000 7f63be7fc640 1 -- 172.21.15.50:0/1663930288 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63c0004cc0 con 0x7f63c810e810 2024-04-23T18:37:03.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.827+0000 7f63cf810640 1 -- 172.21.15.50:0/1663930288 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c819b2c0 con 0x7f63c810e810 2024-04-23T18:37:03.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.827+0000 7f63be7fc640 1 -- 172.21.15.50:0/1663930288 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0010670 con 0x7f63c810e810 2024-04-23T18:37:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.827+0000 7f63cf810640 1 -- 172.21.15.50:0/1663930288 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c81a75d0 con 0x7f63c810e810 2024-04-23T18:37:03.459 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.829+0000 7f63be7fc640 1 -- 172.21.15.50:0/1663930288 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f63c0005020 con 0x7f63c810e810 2024-04-23T18:37:03.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63d2604740 1 Processor -- start 2024-04-23T18:37:03.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63d2604740 1 -- start start 2024-04-23T18:37:03.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63d2604740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 0x56439644ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63d2604740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644bf40 0x5643964562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:03.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63d2604740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564396456800 0x564396458bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:03.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63d2604740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5643963b0970 con 0x56439644bf40 2024-04-23T18:37:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63d2604740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56439634d4a0 con 0x564396456800 2024-04-23T18:37:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63d2604740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5643963ac930 con 0x56439644b620 2024-04-23T18:37:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63be7fc640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f63b40a3e40 0x7f63b40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63cd5ac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644bf40 0x5643964562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:03.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63cd5ac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644bf40 0x5643964562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58056/0 (socket says 172.21.15.50:58056) 2024-04-23T18:37:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63cd5ac640 1 -- 172.21.15.50:0/1854697224 learned_addr learned my addr 172.21.15.50:0/1854697224 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63cddad640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f63b40a3e40 0x7f63b40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:03.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63ccdab640 1 --2- 172.21.15.50:0/1854697224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 0x56439644ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63ccdab640 1 --2- 172.21.15.50:0/1854697224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564396456800 0x564396458bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.830+0000 7f63be7fc640 1 -- 172.21.15.50:0/1663930288 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7f63c00d0d50 con 0x7f63c810e810 2024-04-23T18:37:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.831+0000 7f63cd5ac640 1 -- 172.21.15.50:0/1854697224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 msgr2=0x56439644ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.831+0000 7f63cd5ac640 1 --2- 172.21.15.50:0/1854697224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 0x56439644ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.831+0000 7f63cd5ac640 1 -- 172.21.15.50:0/1854697224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564396456800 msgr2=0x564396458bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:03.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.831+0000 7f63cd5ac640 1 --2- 172.21.15.50:0/1854697224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564396456800 0x564396458bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.831+0000 7f63cd5ac640 1 -- 172.21.15.50:0/1854697224 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643960c3210 con 0x56439644bf40 2024-04-23T18:37:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.831+0000 7f63cddad640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f63b40a3e40 0x7f63b40a6300 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f63c009c9a0 tx=0x7f63c0092130 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.831+0000 7f63cd5ac640 1 --2- 172.21.15.50:0/1854697224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644bf40 0x5643964562c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009d40 tx=0x7f63c40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ca9a4205831247 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.831+0000 7f63b37fe640 1 -- 172.21.15.50:0/1854697224 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c4012650 con 0x56439644bf40 2024-04-23T18:37:03.847 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.831+0000 7f63b37fe640 1 -- 172.21.15.50:0/1854697224 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63c4009e30 con 0x56439644bf40 2024-04-23T18:37:03.855 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.831+0000 7f63b37fe640 1 -- 172.21.15.50:0/1854697224 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c400fc20 con 0x56439644bf40 2024-04-23T18:37:03.903 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.832+0000 7f63d2604740 1 -- 172.21.15.50:0/1854697224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644bf40 msgr2=0x5643964562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:03.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.832+0000 7f63d2604740 1 --2- 172.21.15.50:0/1854697224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644bf40 0x5643964562c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009d40 tx=0x7f63c40117b0 comp rx=0 tx=0).stop 2024-04-23T18:37:03.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.832+0000 7f63d2604740 1 -- 172.21.15.50:0/1854697224 shutdown_connections 2024-04-23T18:37:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.832+0000 7f63d2604740 1 --2- 172.21.15.50:0/1854697224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564396456800 0x564396458bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.832+0000 7f63d2604740 1 --2- 172.21.15.50:0/1854697224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644bf40 0x5643964562c0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.832+0000 7f63d2604740 1 --2- 172.21.15.50:0/1854697224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 0x56439644ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:04.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.832+0000 7f63d2604740 1 -- 172.21.15.50:0/1854697224 >> 172.21.15.50:0/1854697224 conn(0x564396447b30 msgr2=0x564396447f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.832+0000 7f63d2604740 1 -- 172.21.15.50:0/1854697224 shutdown_connections 2024-04-23T18:37:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.832+0000 7f63d2604740 1 -- 172.21.15.50:0/1854697224 wait complete. 2024-04-23T18:37:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.832+0000 7f63d2604740 1 -- start start 2024-04-23T18:37:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.832+0000 7f63d2604740 1 Processor -- start 2024-04-23T18:37:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63d2604740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644b620 0x5643964560f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63d2604740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644bf40 0x5643964533e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63d2604740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564396456800 0x564396453920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63d2604740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564396459430 con 0x56439644b620 2024-04-23T18:37:04.189 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63d2604740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56439634b0b0 con 0x564396456800 2024-04-23T18:37:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63d2604740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5643964595b0 con 0x56439644bf40 2024-04-23T18:37:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63d2604740 10 client.0 _ll_register_callbacks cb 0x564396338d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:37:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63ccdab640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644b620 0x5643964560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:04.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63ccdab640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644b620 0x5643964560f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58066/0 (socket says 172.21.15.50:58066) 2024-04-23T18:37:04.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63ccdab640 1 -- 172.21.15.50:0/7608688 learned_addr learned my addr 172.21.15.50:0/7608688 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63ccdab640 1 -- 172.21.15.50:0/7608688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644bf40 msgr2=0x5643964533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63ccdab640 1 --2- 172.21.15.50:0/7608688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644bf40 0x5643964533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63ccdab640 1 -- 172.21.15.50:0/7608688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564396456800 msgr2=0x564396453920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63ccdab640 1 --2- 172.21.15.50:0/7608688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564396456800 0x564396453920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63ccdab640 1 -- 172.21.15.50:0/7608688 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56439618f620 con 0x56439644b620 2024-04-23T18:37:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63ccdab640 1 --2- 172.21.15.50:0/7608688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644b620 0x5643964560f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f63b802a9a0 tx=0x7f63b8005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ab8c82767e1030 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.833+0000 7f63b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:37:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63b0ff9640 1 -- 172.21.15.50:0/7608688 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8035070 con 0x56439644b620 2024-04-23T18:37:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63b0ff9640 1 -- 172.21.15.50:0/7608688 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7f63b8004240 con 0x56439644b620 2024-04-23T18:37:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63d2604740 1 -- 172.21.15.50:0/7608688 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63c4009e30 con 0x56439644b620 2024-04-23T18:37:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63b0ff9640 1 -- 172.21.15.50:0/7608688 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f63b8004440 con 0x56439644b620 2024-04-23T18:37:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63d2604740 10 client.17790 fetch_fsmap learned FSMap version 62 2024-04-23T18:37:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63d2604740 1 -- 172.21.15.50:0/7608688 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x564396465300 con 0x56439644b620 2024-04-23T18:37:04.531 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63b0ff9640 1 -- 172.21.15.50:0/7608688 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63c4009e30 con 0x56439644b620 2024-04-23T18:37:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63b0ff9640 1 -- 172.21.15.50:0/7608688 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f63b8053610 con 0x56439644b620 2024-04-23T18:37:04.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63d2604740 10 client.17790 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:37:04.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63d2604740 10 client.17790 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:37:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63d2604740 20 client.17790 populate_metadata read hostname 'smithi050' 2024-04-23T18:37:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63d2604740 1 --2- 172.21.15.50:0/7608688 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5643964667c0 0x564396468c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:04.684 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63d2604740 4 client.17790 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:37:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.834+0000 7f63d2604740 1 -- 172.21.15.50:0/7608688 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643963be390 con 0x5643964667c0 2024-04-23T18:37:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.835+0000 7f63cd5ac640 1 --2- 172.21.15.50:0/7608688 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5643964667c0 0x564396468c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.835+0000 7f63b0ff9640 10 client.17790 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:04.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.835+0000 7f63cd5ac640 1 --2- 172.21.15.50:0/7608688 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5643964667c0 0x564396468c50 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a30c9e04968fb62 server_cookie=b4acb6f01b8c77e4 in_seq=0 out_seq=0 2024-04-23T18:37:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63b0ff9640 1 -- 172.21.15.50:0/7608688 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5643963be390 con 0x5643964667c0 2024-04-23T18:37:04.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63b0ff9640 10 client.17790 handle_command_reply: tid=0 2024-04-23T18:37:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63d2604740 1 client.17790 shutdown 2024-04-23T18:37:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63d2604740 1 -- 172.21.15.50:0/7608688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644b620 msgr2=0x5643964560f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:04.748 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63d2604740 1 --2- 172.21.15.50:0/7608688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644b620 0x5643964560f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f63b802a9a0 tx=0x7f63b8005be0 comp rx=0 tx=0).stop 2024-04-23T18:37:04.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63d2604740 1 -- 172.21.15.50:0/7608688 shutdown_connections 2024-04-23T18:37:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63d2604740 1 --2- 172.21.15.50:0/7608688 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5643964667c0 0x564396468c50 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:04.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63d2604740 1 --2- 172.21.15.50:0/7608688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564396456800 0x564396453920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:04.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63d2604740 1 --2- 172.21.15.50:0/7608688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644bf40 0x5643964533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:04.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63d2604740 1 --2- 172.21.15.50:0/7608688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x56439644b620 0x5643964560f0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63b0ff9640 0 client.17790 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:04.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63d2604740 1 -- 172.21.15.50:0/7608688 >> 172.21.15.50:0/7608688 conn(0x564396447b30 msgr2=0x564396446cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63d2604740 1 -- 172.21.15.50:0/7608688 shutdown_connections 2024-04-23T18:37:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63d2604740 1 -- 172.21.15.50:0/7608688 wait complete. 2024-04-23T18:37:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.836+0000 7f63d2604740 20 client.17790 trim_cache size 0 max 16384 2024-04-23T18:37:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.851+0000 7f63d2604740 1 Processor -- start 2024-04-23T18:37:04.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.851+0000 7f63d2604740 1 -- start start 2024-04-23T18:37:04.991 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63d2604740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 0x5643964622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63d2604740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439644bf40 0x564396455c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63d2604740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 0x56439646e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63d2604740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f63c81b25f0 con 0x564396456800 2024-04-23T18:37:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63d2604740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f63c81afec0 con 0x56439644bf40 2024-04-23T18:37:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63d2604740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f63c81add10 con 0x56439644b620 2024-04-23T18:37:05.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63ccdab640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 0x56439646e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:05.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63ccdab640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 0x56439646e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58076/0 (socket says 172.21.15.50:58076) 2024-04-23T18:37:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63ccdab640 1 -- 172.21.15.50:0/1318747273 learned_addr learned my addr 172.21.15.50:0/1318747273 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63cd5ac640 1 --2- 172.21.15.50:0/1318747273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439644bf40 0x564396455c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63ccdab640 1 --2- 172.21.15.50:0/1318747273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 0x5643964622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:05.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63ccdab640 1 -- 172.21.15.50:0/1318747273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 msgr2=0x5643964622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:05.269 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63ccdab640 1 --2- 172.21.15.50:0/1318747273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 0x5643964622f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:05.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63ccdab640 1 -- 172.21.15.50:0/1318747273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439644bf40 msgr2=0x564396455c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:05.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63ccdab640 1 --2- 172.21.15.50:0/1318747273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439644bf40 0x564396455c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63ccdab640 1 -- 172.21.15.50:0/1318747273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564396451950 con 0x564396456800 2024-04-23T18:37:05.308 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63ccdab640 1 --2- 172.21.15.50:0/1318747273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 0x56439646e560 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f63b8000c00 tx=0x7f63b8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f07fe4a8f72515a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.852+0000 7f63b2ffd640 1 -- 172.21.15.50:0/1318747273 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8035050 con 0x564396456800 2024-04-23T18:37:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.853+0000 7f63b2ffd640 1 -- 172.21.15.50:0/1318747273 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63b8058070 con 0x564396456800 2024-04-23T18:37:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.853+0000 7f63b2ffd640 1 -- 172.21.15.50:0/1318747273 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b80325d0 con 0x564396456800 2024-04-23T18:37:05.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.853+0000 7f63d2604740 1 -- 172.21.15.50:0/1318747273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 msgr2=0x56439646e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.853+0000 7f63d2604740 1 --2- 172.21.15.50:0/1318747273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 0x56439646e560 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f63b8000c00 tx=0x7f63b8005fd0 comp rx=0 tx=0).stop 2024-04-23T18:37:05.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.853+0000 7f63d2604740 1 -- 172.21.15.50:0/1318747273 shutdown_connections 2024-04-23T18:37:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.853+0000 7f63d2604740 1 --2- 172.21.15.50:0/1318747273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 0x56439646e560 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.853+0000 7f63d2604740 1 --2- 172.21.15.50:0/1318747273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439644bf40 0x564396455c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.853+0000 7f63d2604740 1 --2- 172.21.15.50:0/1318747273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 0x5643964622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:05.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.853+0000 7f63d2604740 1 -- 172.21.15.50:0/1318747273 >> 172.21.15.50:0/1318747273 conn(0x564396447b30 msgr2=0x56439644d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.853+0000 7f63d2604740 1 -- 172.21.15.50:0/1318747273 shutdown_connections 2024-04-23T18:37:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.853+0000 7f63d2604740 1 -- 172.21.15.50:0/1318747273 wait complete. 2024-04-23T18:37:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.853+0000 7f63d2604740 1 -- start start 2024-04-23T18:37:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63d2604740 1 Processor -- start 2024-04-23T18:37:05.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63d2604740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 0x56439645ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63d2604740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439644bf40 0x56439644fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:05.580 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63d2604740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 0x564396450370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63d2604740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f63c81ae380 con 0x564396456800 2024-04-23T18:37:05.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63d2604740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f63c81af7f0 con 0x56439644bf40 2024-04-23T18:37:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63d2604740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f63c81aea50 con 0x56439644b620 2024-04-23T18:37:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63d2604740 10 client.0 _ll_register_callbacks cb 0x7f63b8032f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:37:05.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63ccdab640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 0x564396450370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:05.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63ccdab640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 0x564396450370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58088/0 (socket says 172.21.15.50:58088) 2024-04-23T18:37:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63ccdab640 1 -- 172.21.15.50:0/3907881629 learned_addr learned my addr 172.21.15.50:0/3907881629 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:05.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63cd5ac640 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439644bf40 0x56439644fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63ccdab640 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 0x56439645ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63ccdab640 1 -- 172.21.15.50:0/3907881629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 msgr2=0x56439645ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63ccdab640 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 0x56439645ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:05.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63ccdab640 1 -- 172.21.15.50:0/3907881629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439644bf40 msgr2=0x56439644fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.854+0000 7f63ccdab640 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439644bf40 0x56439644fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:05.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63ccdab640 1 -- 172.21.15.50:0/3907881629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564396450bf0 con 0x564396456800 2024-04-23T18:37:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63ccdab640 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 0x564396450370 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f63b8005c40 tx=0x7f63b8031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4652facbca53f00e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:37:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63b0ff9640 1 -- 172.21.15.50:0/3907881629 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8035070 con 0x564396456800 2024-04-23T18:37:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63b0ff9640 1 -- 172.21.15.50:0/3907881629 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8498+0+0 (secure 0 0 0) 0x7f63b8053670 con 0x564396456800 2024-04-23T18:37:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63d2604740 1 -- 172.21.15.50:0/3907881629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63b8058070 con 0x564396456800 2024-04-23T18:37:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63b0ff9640 1 -- 172.21.15.50:0/3907881629 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f63b8032370 con 0x564396456800 2024-04-23T18:37:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63d2604740 10 client.17796 fetch_fsmap learned FSMap version 62 2024-04-23T18:37:05.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63d2604740 1 -- 172.21.15.50:0/3907881629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5643964cce30 con 0x564396456800 2024-04-23T18:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63b0ff9640 1 -- 172.21.15.50:0/3907881629 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63b8058070 con 0x564396456800 2024-04-23T18:37:06.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63b0ff9640 1 -- 172.21.15.50:0/3907881629 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f63b805d790 con 0x564396456800 2024-04-23T18:37:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63d2604740 10 client.17796 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:37:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63d2604740 10 client.17796 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:37:06.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.855+0000 7f63d2604740 20 client.17796 populate_metadata read hostname 'smithi050' 2024-04-23T18:37:06.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.856+0000 7f63d2604740 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5643964cd470 0x5643964cf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.856+0000 7f63d2604740 4 client.17796 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:37:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.856+0000 7f63d2604740 1 -- 172.21.15.50:0/3907881629 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564396447980 con 0x5643964cd470 2024-04-23T18:37:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.856+0000 7f63cd5ac640 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5643964cd470 0x5643964cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.856+0000 7f63cd5ac640 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5643964cd470 0x5643964cf890 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2b9a2afa7eb99bf server_cookie=eb7342378a374e31 in_seq=0 out_seq=0 2024-04-23T18:37:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:01.856+0000 7f63b0ff9640 10 client.17796 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.121+0000 7f63b0ff9640 1 -- 172.21.15.50:0/3907881629 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564396447980 con 0x5643964cd470 2024-04-23T18:37:06.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.121+0000 7f63b0ff9640 10 client.17796 handle_command_reply: tid=0 2024-04-23T18:37:06.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.121+0000 7f63d2604740 1 client.17796 shutdown 2024-04-23T18:37:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.121+0000 7f63d2604740 1 -- 172.21.15.50:0/3907881629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 msgr2=0x564396450370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:06.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.121+0000 7f63d2604740 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 0x564396450370 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f63b8005c40 tx=0x7f63b8031400 comp rx=0 tx=0).stop 2024-04-23T18:37:06.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.121+0000 7f63d2604740 1 -- 172.21.15.50:0/3907881629 shutdown_connections 2024-04-23T18:37:06.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.121+0000 7f63d2604740 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5643964cd470 0x5643964cf890 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:06.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.121+0000 7f63d2604740 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564396456800 0x564396450370 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.121+0000 7f63d2604740 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439644bf40 0x56439644fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:06.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.121+0000 7f63b0ff9640 0 client.17796 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.121+0000 7f63d2604740 1 --2- 172.21.15.50:0/3907881629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56439644b620 0x56439645ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.122+0000 7f63d2604740 1 -- 172.21.15.50:0/3907881629 >> 172.21.15.50:0/3907881629 conn(0x564396447b30 msgr2=0x56439644e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.122+0000 7f63d2604740 1 -- 172.21.15.50:0/3907881629 shutdown_connections 2024-04-23T18:37:07.012 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.122+0000 7f63d2604740 1 -- 172.21.15.50:0/3907881629 wait complete. 2024-04-23T18:37:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.122+0000 7f63d2604740 20 client.17796 trim_cache size 0 max 16384 2024-04-23T18:37:07.029 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:37:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:37:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:37:07.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:37:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.122+0000 7f63cf810640 1 -- 172.21.15.50:0/1663930288 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f63b40a3e40 msgr2=0x7f63b40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:07.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.122+0000 7f63cf810640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f63b40a3e40 0x7f63b40a6300 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f63c009c9a0 tx=0x7f63c0092130 comp rx=0 tx=0).stop 2024-04-23T18:37:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.122+0000 7f63cf810640 1 -- 172.21.15.50:0/1663930288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63c810e810 msgr2=0x7f63c819a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.122+0000 7f63cf810640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63c810e810 0x7f63c819a710 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f63c000eb10 tx=0x7f63c000c2d0 comp rx=0 tx=0).stop 2024-04-23T18:37:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.123+0000 7f63cf810640 1 -- 172.21.15.50:0/1663930288 shutdown_connections 2024-04-23T18:37:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.123+0000 7f63cf810640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f63b40a3e40 0x7f63b40a6300 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.123+0000 7f63cf810640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63c810e810 0x7f63c819a710 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.123+0000 7f63cf810640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63c8074c20 0x7f63c81a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:07.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.123+0000 7f63cf810640 1 --2- 172.21.15.50:0/1663930288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63c8074300 0x7f63c81a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.123+0000 7f63cf810640 1 -- 172.21.15.50:0/1663930288 >> 172.21.15.50:0/1663930288 conn(0x7f63c80fda20 msgr2=0x7f63c80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.123+0000 7f63cf810640 1 -- 172.21.15.50:0/1663930288 shutdown_connections 2024-04-23T18:37:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:02.123+0000 7f63cf810640 1 -- 172.21.15.50:0/1663930288 wait complete. 2024-04-23T18:37:07.339 INFO:tasks.workunit.client.0.smithi050.stdout: 10 130964 19.61 MB/sec execute 216 sec latency 1564.299 ms 2024-04-23T18:37:07.347 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:02 smithi050 ceph-mon[29187]: pgmap v588: 97 pgs: 97 active+clean; 1.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-04-23T18:37:07.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:04 smithi177 ceph-mon[32308]: pgmap v589: 97 pgs: 97 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.1 MiB/s wr, 249 op/s 2024-04-23T18:37:07.365 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:05 smithi177 ceph-mon[32308]: osdmap e121: 12 total, 12 up, 12 in 2024-04-23T18:37:07.376 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:06 smithi177 ceph-mon[32308]: pgmap v591: 97 pgs: 97 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.8 MiB/s wr, 267 op/s 2024-04-23T18:37:07.382 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:04 smithi188 ceph-mon[32694]: pgmap v589: 97 pgs: 97 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.1 MiB/s wr, 249 op/s 2024-04-23T18:37:07.391 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:05 smithi188 ceph-mon[32694]: osdmap e121: 12 total, 12 up, 12 in 2024-04-23T18:37:07.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:06 smithi188 ceph-mon[32694]: pgmap v591: 97 pgs: 97 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.8 MiB/s wr, 267 op/s 2024-04-23T18:37:07.410 INFO:tasks.workunit.client.0.smithi050.stdout: 10 131241 19.60 MB/sec execute 217 sec latency 191.303 ms 2024-04-23T18:37:07.459 INFO:tasks.workunit.client.0.smithi050.stdout: 10 131577 19.60 MB/sec execute 218 sec latency 306.939 ms 2024-04-23T18:37:07.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:04 smithi050 ceph-mon[29187]: pgmap v589: 97 pgs: 97 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.1 MiB/s wr, 249 op/s 2024-04-23T18:37:07.506 INFO:tasks.workunit.client.0.smithi050.stdout: 10 131780 19.54 MB/sec execute 219 sec latency 513.961 ms 2024-04-23T18:37:07.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:05 smithi050 ceph-mon[29187]: osdmap e121: 12 total, 12 up, 12 in 2024-04-23T18:37:07.540 INFO:tasks.workunit.client.0.smithi050.stdout: 10 131997 19.52 MB/sec execute 220 sec latency 537.301 ms 2024-04-23T18:37:07.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:06 smithi050 ceph-mon[29187]: pgmap v591: 97 pgs: 97 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.8 MiB/s wr, 267 op/s 2024-04-23T18:37:07.599 INFO:tasks.workunit.client.0.smithi050.stdout: 10 132425 19.56 MB/sec execute 221 sec latency 591.977 ms 2024-04-23T18:37:08.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 132851 19.54 MB/sec execute 222 sec latency 99.218 ms 2024-04-23T18:37:08.586 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:08 smithi188 ceph-mon[32694]: pgmap v592: 97 pgs: 5 active+clean+snaptrim, 81 active+clean, 11 active+clean+snaptrim_wait; 1.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.6 MiB/s wr, 559 op/s 2024-04-23T18:37:08.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:08 smithi050 ceph-mon[29187]: pgmap v592: 97 pgs: 5 active+clean+snaptrim, 81 active+clean, 11 active+clean+snaptrim_wait; 1.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.6 MiB/s wr, 559 op/s 2024-04-23T18:37:08.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:08 smithi177 ceph-mon[32308]: pgmap v592: 97 pgs: 5 active+clean+snaptrim, 81 active+clean, 11 active+clean+snaptrim_wait; 1.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.6 MiB/s wr, 559 op/s 2024-04-23T18:37:09.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 133388 19.54 MB/sec execute 223 sec latency 130.922 ms 2024-04-23T18:37:10.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 134081 19.66 MB/sec execute 224 sec latency 104.040 ms 2024-04-23T18:37:10.594 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:10 smithi188 ceph-mon[32694]: pgmap v593: 97 pgs: 8 active+clean+snaptrim, 73 active+clean, 16 active+clean+snaptrim_wait; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 657 op/s 2024-04-23T18:37:10.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:10 smithi050 ceph-mon[29187]: pgmap v593: 97 pgs: 8 active+clean+snaptrim, 73 active+clean, 16 active+clean+snaptrim_wait; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 657 op/s 2024-04-23T18:37:10.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:10 smithi177 ceph-mon[32308]: pgmap v593: 97 pgs: 8 active+clean+snaptrim, 73 active+clean, 16 active+clean+snaptrim_wait; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 657 op/s 2024-04-23T18:37:11.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 134695 19.72 MB/sec execute 225 sec latency 78.268 ms 2024-04-23T18:37:12.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 135315 19.81 MB/sec execute 226 sec latency 103.381 ms 2024-04-23T18:37:12.608 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:12 smithi188 ceph-mon[32694]: pgmap v594: 97 pgs: 8 active+clean+snaptrim, 73 active+clean, 16 active+clean+snaptrim_wait; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 657 op/s 2024-04-23T18:37:12.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:12 smithi050 ceph-mon[29187]: pgmap v594: 97 pgs: 8 active+clean+snaptrim, 73 active+clean, 16 active+clean+snaptrim_wait; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 657 op/s 2024-04-23T18:37:12.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:12 smithi177 ceph-mon[32308]: pgmap v594: 97 pgs: 8 active+clean+snaptrim, 73 active+clean, 16 active+clean+snaptrim_wait; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 657 op/s 2024-04-23T18:37:13.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 135812 19.87 MB/sec execute 227 sec latency 80.963 ms 2024-04-23T18:37:14.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 136188 19.83 MB/sec execute 228 sec latency 104.183 ms 2024-04-23T18:37:14.636 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:14 smithi188 ceph-mon[32694]: pgmap v595: 97 pgs: 97 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.1 MiB/s wr, 695 op/s 2024-04-23T18:37:14.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:14 smithi050 ceph-mon[29187]: pgmap v595: 97 pgs: 97 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.1 MiB/s wr, 695 op/s 2024-04-23T18:37:14.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:14 smithi177 ceph-mon[32308]: pgmap v595: 97 pgs: 97 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.1 MiB/s wr, 695 op/s 2024-04-23T18:37:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.224+0000 7fdd407f9640 1 -- 172.21.15.50:0/4089725975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd38103af0 msgr2=0x7fdd3810bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.224+0000 7fdd407f9640 1 --2- 172.21.15.50:0/4089725975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd38103af0 0x7fdd3810bfd0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009900 tx=0x7fdd2802a480 comp rx=0 tx=0).stop 2024-04-23T18:37:15.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.224+0000 7fdd407f9640 1 -- 172.21.15.50:0/4089725975 shutdown_connections 2024-04-23T18:37:15.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.224+0000 7fdd407f9640 1 --2- 172.21.15.50:0/4089725975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd3810c5a0 0x7fdd3810e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.224+0000 7fdd407f9640 1 --2- 172.21.15.50:0/4089725975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd38103af0 0x7fdd3810bfd0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:15.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.224+0000 7fdd407f9640 1 --2- 172.21.15.50:0/4089725975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd381031d0 0x7fdd381035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:15.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.224+0000 7fdd407f9640 1 -- 172.21.15.50:0/4089725975 >> 172.21.15.50:0/4089725975 conn(0x7fdd38077fa0 msgr2=0x7fdd380783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.224+0000 7fdd407f9640 1 -- 172.21.15.50:0/4089725975 shutdown_connections 2024-04-23T18:37:15.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.224+0000 7fdd407f9640 1 -- 172.21.15.50:0/4089725975 wait complete. 2024-04-23T18:37:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.225+0000 7fdd407f9640 1 Processor -- start 2024-04-23T18:37:15.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.225+0000 7fdd407f9640 1 -- start start 2024-04-23T18:37:15.314 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.225+0000 7fdd407f9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd381031d0 0x7fdd381a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:15.322 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.225+0000 7fdd407f9640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd38103af0 0x7fdd381a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:15.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.225+0000 7fdd407f9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd3810c5a0 0x7fdd3819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.225+0000 7fdd407f9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdd381121d0 con 0x7fdd381031d0 2024-04-23T18:37:15.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.225+0000 7fdd407f9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd38112050 con 0x7fdd3810c5a0 2024-04-23T18:37:15.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.225+0000 7fdd407f9640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdd38112350 con 0x7fdd38103af0 2024-04-23T18:37:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.225+0000 7fdd3e595640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd381031d0 0x7fdd381a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd3e595640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd381031d0 0x7fdd381a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51990/0 (socket says 172.21.15.50:51990) 2024-04-23T18:37:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd3e595640 1 -- 172.21.15.50:0/2162194170 learned_addr learned my addr 172.21.15.50:0/2162194170 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:15.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd3810c5a0 0x7fdd3819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd3dd94640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd38103af0 0x7fdd381a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd3e595640 1 -- 172.21.15.50:0/2162194170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd38103af0 msgr2=0x7fdd381a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd3e595640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd38103af0 0x7fdd381a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd3e595640 1 -- 172.21.15.50:0/2162194170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd3810c5a0 msgr2=0x7fdd3819a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd3e595640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd3810c5a0 0x7fdd3819a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd3e595640 1 -- 172.21.15.50:0/2162194170 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3819afc0 con 0x7fdd381031d0 2024-04-23T18:37:15.585 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd3dd94640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd38103af0 0x7fdd381a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:37:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd3e595640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd381031d0 0x7fdd381a0680 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400e9f0 tx=0x7fdd3400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:15.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd2f7fe640 1 -- 172.21.15.50:0/2162194170 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3400ccc0 con 0x7fdd381031d0 2024-04-23T18:37:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd2f7fe640 1 -- 172.21.15.50:0/2162194170 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3400ce60 con 0x7fdd381031d0 2024-04-23T18:37:15.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd407f9640 1 -- 172.21.15.50:0/2162194170 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd3819b2b0 con 0x7fdd381031d0 2024-04-23T18:37:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd2f7fe640 1 -- 172.21.15.50:0/2162194170 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd34010430 con 0x7fdd381031d0 2024-04-23T18:37:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.226+0000 7fdd407f9640 1 -- 172.21.15.50:0/2162194170 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd381a74d0 con 0x7fdd381031d0 2024-04-23T18:37:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd2f7fe640 1 -- 172.21.15.50:0/2162194170 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdd340046c0 con 0x7fdd381031d0 2024-04-23T18:37:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd435df740 1 Processor -- start 2024-04-23T18:37:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd435df740 1 -- start start 2024-04-23T18:37:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd435df740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc08a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd435df740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc08f40 0x55645bc132c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:15.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd435df740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc13800 0x55645bc15bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd435df740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55645bb6d970 con 0x55645bc08f40 2024-04-23T18:37:15.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd435df740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55645bb0a4a0 con 0x55645bc08620 2024-04-23T18:37:15.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd435df740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55645bb69930 con 0x55645bc13800 2024-04-23T18:37:15.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd3ed96640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc08a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd3e595640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc08f40 0x55645bc132c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd3ed96640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc08a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:56798/0 (socket says 172.21.15.50:56798) 2024-04-23T18:37:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.229+0000 7fdd3ed96640 1 -- 172.21.15.50:0/1429181584 learned_addr learned my addr 172.21.15.50:0/1429181584 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:15.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd3dd94640 1 --2- 172.21.15.50:0/1429181584 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc13800 0x55645bc15bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:15.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd2f7fe640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdd240a3dc0 0x7fdd240a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:15.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd3ed96640 1 -- 172.21.15.50:0/1429181584 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc13800 msgr2=0x55645bc15bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/1429181584 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc13800 0x55645bc15bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:15.981 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd3ed96640 1 -- 172.21.15.50:0/1429181584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc08f40 msgr2=0x55645bc132c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:15.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/1429181584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc08f40 0x55645bc132c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd3ed96640 1 -- 172.21.15.50:0/1429181584 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55645b880210 con 0x55645bc08620 2024-04-23T18:37:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd2f7fe640 1 -- 172.21.15.50:0/2162194170 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8586+0+0 (secure 0 0 0) 0x7fdd340d0cc0 con 0x7fdd381031d0 2024-04-23T18:37:16.017 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdd240a3dc0 0x7fdd240a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/1429181584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc08a00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdd30011590 tx=0x7fdd30009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd68ecba717aace3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd2cff9640 1 -- 172.21.15.50:0/1429181584 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30017050 con 0x55645bc08620 2024-04-23T18:37:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd2cff9640 1 -- 172.21.15.50:0/1429181584 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3001bbf0 con 0x55645bc08620 2024-04-23T18:37:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd2cff9640 1 -- 172.21.15.50:0/1429181584 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30026b00 con 0x55645bc08620 2024-04-23T18:37:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.230+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdd240a3dc0 0x7fdd240a6280 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdd30023000 tx=0x7fdd30024000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.231+0000 7fdd435df740 1 -- 172.21.15.50:0/1429181584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 msgr2=0x55645bc08a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.231+0000 7fdd435df740 1 --2- 172.21.15.50:0/1429181584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc08a00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdd30011590 tx=0x7fdd30009f90 comp rx=0 tx=0).stop 2024-04-23T18:37:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.231+0000 7fdd435df740 1 -- 172.21.15.50:0/1429181584 shutdown_connections 2024-04-23T18:37:16.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.231+0000 7fdd435df740 1 --2- 172.21.15.50:0/1429181584 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc13800 0x55645bc15bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:16.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.231+0000 7fdd435df740 1 --2- 172.21.15.50:0/1429181584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc08f40 0x55645bc132c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.231+0000 7fdd435df740 1 --2- 172.21.15.50:0/1429181584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc08a00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.231+0000 7fdd435df740 1 -- 172.21.15.50:0/1429181584 >> 172.21.15.50:0/1429181584 conn(0x55645bc04b30 msgr2=0x55645bc04f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.231+0000 7fdd435df740 1 -- 172.21.15.50:0/1429181584 shutdown_connections 2024-04-23T18:37:16.195 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.231+0000 7fdd435df740 1 -- 172.21.15.50:0/1429181584 wait complete. 2024-04-23T18:37:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.231+0000 7fdd435df740 1 -- start start 2024-04-23T18:37:16.210 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd435df740 1 Processor -- start 2024-04-23T18:37:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd435df740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc13190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:16.226 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd435df740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc08f40 0x55645bc10480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd435df740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc13800 0x55645bc10a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:16.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd435df740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55645bc16430 con 0x55645bc08f40 2024-04-23T18:37:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd435df740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55645bb080b0 con 0x55645bc08620 2024-04-23T18:37:16.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd435df740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55645bc165b0 con 0x55645bc13800 2024-04-23T18:37:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd435df740 10 client.0 _ll_register_callbacks cb 0x55645baf5d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:37:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd3dd94640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc08f40 0x55645bc10480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd3e595640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc13190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:16.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd3e595640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc13190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:56806/0 (socket says 172.21.15.50:56806) 2024-04-23T18:37:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd3dd94640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc08f40 0x55645bc10480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52008/0 (socket says 172.21.15.50:52008) 2024-04-23T18:37:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd3e595640 1 -- 172.21.15.50:0/3597149871 learned_addr learned my addr 172.21.15.50:0/3597149871 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc13800 0x55645bc10a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd3dd94640 1 -- 172.21.15.50:0/3597149871 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc13800 msgr2=0x55645bc10a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd3dd94640 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc13800 0x55645bc10a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:16.426 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd3dd94640 1 -- 172.21.15.50:0/3597149871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 msgr2=0x55645bc13190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd3dd94640 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc13190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd3dd94640 1 -- 172.21.15.50:0/3597149871 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55645b94c620 con 0x55645bc08f40 2024-04-23T18:37:16.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.232+0000 7fdd3e595640 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc13190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:37:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd3dd94640 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc08f40 0x55645bc10480 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fdd3810e2e0 tx=0x7fdd28032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476bb1e8c12f88c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:16.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:37:16.470 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd21ffb640 1 -- 172.21.15.50:0/3597149871 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28040070 con 0x55645bc08f40 2024-04-23T18:37:16.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd21ffb640 1 -- 172.21.15.50:0/3597149871 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8586+0+0 (secure 0 0 0) 0x7fdd28004040 con 0x55645bc08f40 2024-04-23T18:37:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd435df740 1 -- 172.21.15.50:0/3597149871 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd3001bbf0 con 0x55645bc08f40 2024-04-23T18:37:16.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd21ffb640 1 -- 172.21.15.50:0/3597149871 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fdd28035040 con 0x55645bc08f40 2024-04-23T18:37:16.500 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd435df740 10 client.17808 fetch_fsmap learned FSMap version 62 2024-04-23T18:37:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd435df740 1 -- 172.21.15.50:0/3597149871 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55645bc22380 con 0x55645bc08f40 2024-04-23T18:37:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd21ffb640 1 -- 172.21.15.50:0/3597149871 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3001bbf0 con 0x55645bc08f40 2024-04-23T18:37:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd21ffb640 1 -- 172.21.15.50:0/3597149871 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fdd280046c0 con 0x55645bc08f40 2024-04-23T18:37:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd435df740 10 client.17808 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:37:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd435df740 10 client.17808 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:37:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd435df740 20 client.17808 populate_metadata read hostname 'smithi050' 2024-04-23T18:37:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.233+0000 7fdd435df740 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55645bc23840 0x55645bc25c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.234+0000 7fdd435df740 4 client.17808 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:37:16.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.234+0000 7fdd435df740 1 -- 172.21.15.50:0/3597149871 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55645bb7b390 con 0x55645bc23840 2024-04-23T18:37:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.234+0000 7fdd3e595640 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55645bc23840 0x55645bc25c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.234+0000 7fdd3e595640 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55645bc23840 0x55645bc25c30 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11cefed1923b30fa server_cookie=70b7701b73663650 in_seq=0 out_seq=0 2024-04-23T18:37:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.234+0000 7fdd21ffb640 10 client.17808 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd21ffb640 1 -- 172.21.15.50:0/3597149871 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55645bb7b390 con 0x55645bc23840 2024-04-23T18:37:16.709 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd21ffb640 10 client.17808 handle_command_reply: tid=0 2024-04-23T18:37:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd435df740 1 client.17808 shutdown 2024-04-23T18:37:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd435df740 1 -- 172.21.15.50:0/3597149871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc08f40 msgr2=0x55645bc10480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd435df740 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc08f40 0x55645bc10480 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fdd3810e2e0 tx=0x7fdd28032e90 comp rx=0 tx=0).stop 2024-04-23T18:37:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd435df740 1 -- 172.21.15.50:0/3597149871 shutdown_connections 2024-04-23T18:37:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd435df740 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55645bc23840 0x55645bc25c30 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:16.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd435df740 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc13800 0x55645bc10a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd435df740 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc08f40 0x55645bc10480 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd21ffb640 0 client.17808 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd435df740 1 --2- 172.21.15.50:0/3597149871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc13190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd435df740 1 -- 172.21.15.50:0/3597149871 >> 172.21.15.50:0/3597149871 conn(0x55645bc04b30 msgr2=0x55645bc0a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:16.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd435df740 1 -- 172.21.15.50:0/3597149871 shutdown_connections 2024-04-23T18:37:16.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd435df740 1 -- 172.21.15.50:0/3597149871 wait complete. 2024-04-23T18:37:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.235+0000 7fdd435df740 20 client.17808 trim_cache size 0 max 16384 2024-04-23T18:37:16.893 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.252+0000 7fdd435df740 1 Processor -- start 2024-04-23T18:37:16.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.252+0000 7fdd435df740 1 -- start start 2024-04-23T18:37:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.252+0000 7fdd435df740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc1f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.252+0000 7fdd435df740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc08f40 0x55645bc2b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.252+0000 7fdd435df740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 0x55645bc2ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:16.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.252+0000 7fdd435df740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdd381b27a0 con 0x55645bc13800 2024-04-23T18:37:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.252+0000 7fdd435df740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd381b0070 con 0x55645bc08620 2024-04-23T18:37:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.252+0000 7fdd435df740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdd381adee0 con 0x55645bc08f40 2024-04-23T18:37:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.252+0000 7fdd3ed96640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 0x55645bc2ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.252+0000 7fdd3e595640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc1f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd3ed96640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 0x55645bc2ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52018/0 (socket says 172.21.15.50:52018) 2024-04-23T18:37:17.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd3dd94640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc08f40 0x55645bc2b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:17.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd3ed96640 1 -- 172.21.15.50:0/3612717543 learned_addr learned my addr 172.21.15.50:0/3612717543 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd3ed96640 1 -- 172.21.15.50:0/3612717543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc08f40 msgr2=0x55645bc2b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:17.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/3612717543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc08f40 0x55645bc2b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd3ed96640 1 -- 172.21.15.50:0/3612717543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 msgr2=0x55645bc1f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:17.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/3612717543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc1f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:17.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd3ed96640 1 -- 172.21.15.50:0/3612717543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55645bc05470 con 0x55645bc13800 2024-04-23T18:37:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd3e595640 1 --2- 172.21.15.50:0/3612717543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc1f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:37:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/3612717543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 0x55645bc2ba40 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fdd30011ee0 tx=0x7fdd30011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a604792e4754633 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd23fff640 1 -- 172.21.15.50:0/3612717543 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30026440 con 0x55645bc13800 2024-04-23T18:37:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd23fff640 1 -- 172.21.15.50:0/3612717543 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd30026850 con 0x55645bc13800 2024-04-23T18:37:17.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.253+0000 7fdd23fff640 1 -- 172.21.15.50:0/3612717543 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3001bc50 con 0x55645bc13800 2024-04-23T18:37:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.254+0000 7fdd435df740 1 -- 172.21.15.50:0/3612717543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 msgr2=0x55645bc2ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.254+0000 7fdd435df740 1 --2- 172.21.15.50:0/3612717543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 0x55645bc2ba40 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fdd30011ee0 tx=0x7fdd30011f90 comp rx=0 tx=0).stop 2024-04-23T18:37:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.254+0000 7fdd435df740 1 -- 172.21.15.50:0/3612717543 shutdown_connections 2024-04-23T18:37:17.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.254+0000 7fdd435df740 1 --2- 172.21.15.50:0/3612717543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 0x55645bc2ba40 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.254+0000 7fdd435df740 1 --2- 172.21.15.50:0/3612717543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc08f40 0x55645bc2b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.254+0000 7fdd435df740 1 --2- 172.21.15.50:0/3612717543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc1f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:17.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.254+0000 7fdd435df740 1 -- 172.21.15.50:0/3612717543 >> 172.21.15.50:0/3612717543 conn(0x55645bc04b30 msgr2=0x55645bc03cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:17.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.254+0000 7fdd435df740 1 -- 172.21.15.50:0/3612717543 shutdown_connections 2024-04-23T18:37:17.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.254+0000 7fdd435df740 1 -- 172.21.15.50:0/3612717543 wait complete. 2024-04-23T18:37:17.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.254+0000 7fdd435df740 1 -- start start 2024-04-23T18:37:17.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd435df740 1 Processor -- start 2024-04-23T18:37:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd435df740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc193a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd435df740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc08f40 0x55645bc198e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd435df740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 0x55645bc19e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:17.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd435df740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdd381ae530 con 0x55645bc13800 2024-04-23T18:37:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd435df740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd381af9a0 con 0x55645bc08620 2024-04-23T18:37:17.835 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd435df740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdd381aec00 con 0x55645bc08f40 2024-04-23T18:37:17.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd435df740 10 client.0 _ll_register_callbacks cb 0x7fdd28032b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd3ed96640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 0x55645bc19e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd3dd94640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc08f40 0x55645bc198e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:17.981 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd3ed96640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 0x55645bc19e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52024/0 (socket says 172.21.15.50:52024) 2024-04-23T18:37:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd3ed96640 1 -- 172.21.15.50:0/3841690518 learned_addr learned my addr 172.21.15.50:0/3841690518 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd3e595640 1 --2- 172.21.15.50:0/3841690518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc193a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:18.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd3ed96640 1 -- 172.21.15.50:0/3841690518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc08f40 msgr2=0x55645bc198e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:18.011 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/3841690518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc08f40 0x55645bc198e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:18.018 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd3ed96640 1 -- 172.21.15.50:0/3841690518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 msgr2=0x55645bc193a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:18.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/3841690518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc193a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd3ed96640 1 -- 172.21.15.50:0/3841690518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55645bc88ee0 con 0x55645bc13800 2024-04-23T18:37:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd3ed96640 1 --2- 172.21.15.50:0/3841690518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 0x55645bc19e20 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009f40 tx=0x7fdd3000f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e4963df8f5e073 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.255+0000 7fdd21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:37:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd21ffb640 1 -- 172.21.15.50:0/3841690518 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30017070 con 0x55645bc13800 2024-04-23T18:37:18.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd21ffb640 1 -- 172.21.15.50:0/3841690518 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8586+0+0 (secure 0 0 0) 0x7fdd30026440 con 0x55645bc13800 2024-04-23T18:37:18.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd435df740 1 -- 172.21.15.50:0/3841690518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd30026850 con 0x55645bc13800 2024-04-23T18:37:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd21ffb640 1 -- 172.21.15.50:0/3841690518 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fdd30026640 con 0x55645bc13800 2024-04-23T18:37:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd435df740 10 client.17814 fetch_fsmap learned FSMap version 62 2024-04-23T18:37:18.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd435df740 1 -- 172.21.15.50:0/3841690518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55645bc8a1d0 con 0x55645bc13800 2024-04-23T18:37:18.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd21ffb640 1 -- 172.21.15.50:0/3841690518 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd30026850 con 0x55645bc13800 2024-04-23T18:37:18.205 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd21ffb640 1 -- 172.21.15.50:0/3841690518 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fdd30045030 con 0x55645bc13800 2024-04-23T18:37:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd435df740 10 client.17814 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:37:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd435df740 10 client.17814 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:37:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd435df740 20 client.17814 populate_metadata read hostname 'smithi050' 2024-04-23T18:37:18.239 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd435df740 1 --2- 172.21.15.50:0/3841690518 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55645bc8a6f0 0x55645bc8cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:18.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd435df740 4 client.17814 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:37:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.256+0000 7fdd435df740 1 -- 172.21.15.50:0/3841690518 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55645bb75200 con 0x55645bc8a6f0 2024-04-23T18:37:18.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.257+0000 7fdd3e595640 1 --2- 172.21.15.50:0/3841690518 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55645bc8a6f0 0x55645bc8cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:18.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.257+0000 7fdd3e595640 1 --2- 172.21.15.50:0/3841690518 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55645bc8a6f0 0x55645bc8cae0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19145f0a2c8633ea server_cookie=456f83faa2c6a3bb in_seq=0 out_seq=0 2024-04-23T18:37:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.257+0000 7fdd21ffb640 10 client.17814 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:18.291 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:16 smithi177 ceph-mon[32308]: pgmap v596: 97 pgs: 97 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.7 MiB/s wr, 662 op/s 2024-04-23T18:37:18.300 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:16 smithi177 ceph-mon[32308]: osdmap e122: 12 total, 12 up, 12 in 2024-04-23T18:37:18.308 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:16 smithi188 ceph-mon[32694]: pgmap v596: 97 pgs: 97 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.7 MiB/s wr, 662 op/s 2024-04-23T18:37:18.316 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:16 smithi188 ceph-mon[32694]: osdmap e122: 12 total, 12 up, 12 in 2024-04-23T18:37:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd21ffb640 1 -- 172.21.15.50:0/3841690518 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55645bb75200 con 0x55645bc8a6f0 2024-04-23T18:37:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd21ffb640 10 client.17814 handle_command_reply: tid=0 2024-04-23T18:37:18.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd435df740 1 client.17814 shutdown 2024-04-23T18:37:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd435df740 1 -- 172.21.15.50:0/3841690518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 msgr2=0x55645bc19e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:18.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd435df740 1 --2- 172.21.15.50:0/3841690518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 0x55645bc19e20 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009f40 tx=0x7fdd3000f430 comp rx=0 tx=0).stop 2024-04-23T18:37:18.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd435df740 1 -- 172.21.15.50:0/3841690518 shutdown_connections 2024-04-23T18:37:18.471 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd435df740 1 --2- 172.21.15.50:0/3841690518 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55645bc8a6f0 0x55645bc8cae0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd21ffb640 0 client.17814 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd435df740 1 --2- 172.21.15.50:0/3841690518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55645bc13800 0x55645bc19e20 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd435df740 1 --2- 172.21.15.50:0/3841690518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55645bc08f40 0x55645bc198e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd435df740 1 --2- 172.21.15.50:0/3841690518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55645bc08620 0x55645bc193a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd435df740 1 -- 172.21.15.50:0/3841690518 >> 172.21.15.50:0/3841690518 conn(0x55645bc04b30 msgr2=0x55645bc03cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd435df740 1 -- 172.21.15.50:0/3841690518 shutdown_connections 2024-04-23T18:37:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd435df740 1 -- 172.21.15.50:0/3841690518 wait complete. 2024-04-23T18:37:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.469+0000 7fdd435df740 20 client.17814 trim_cache size 0 max 16384 2024-04-23T18:37:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:37:18.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:37:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:37:18.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:37:18.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.470+0000 7fdd407f9640 1 -- 172.21.15.50:0/2162194170 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdd240a3dc0 msgr2=0x7fdd240a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.470+0000 7fdd407f9640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdd240a3dc0 0x7fdd240a6280 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdd30023000 tx=0x7fdd30024000 comp rx=0 tx=0).stop 2024-04-23T18:37:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.470+0000 7fdd407f9640 1 -- 172.21.15.50:0/2162194170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd381031d0 msgr2=0x7fdd381a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:18.740 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.470+0000 7fdd407f9640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd381031d0 0x7fdd381a0680 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400e9f0 tx=0x7fdd3400edf0 comp rx=0 tx=0).stop 2024-04-23T18:37:18.749 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.470+0000 7fdd407f9640 1 -- 172.21.15.50:0/2162194170 shutdown_connections 2024-04-23T18:37:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.470+0000 7fdd407f9640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdd240a3dc0 0x7fdd240a6280 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:18.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.470+0000 7fdd407f9640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd3810c5a0 0x7fdd3819a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:18.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.470+0000 7fdd407f9640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd38103af0 0x7fdd381a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.470+0000 7fdd407f9640 1 --2- 172.21.15.50:0/2162194170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd381031d0 0x7fdd381a0680 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.470+0000 7fdd407f9640 1 -- 172.21.15.50:0/2162194170 >> 172.21.15.50:0/2162194170 conn(0x7fdd38077fa0 msgr2=0x7fdd38108520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:18.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.470+0000 7fdd407f9640 1 -- 172.21.15.50:0/2162194170 shutdown_connections 2024-04-23T18:37:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:15.470+0000 7fdd407f9640 1 -- 172.21.15.50:0/2162194170 wait complete. 2024-04-23T18:37:18.820 INFO:tasks.workunit.client.0.smithi050.stdout: 10 136545 19.80 MB/sec execute 229 sec latency 157.340 ms 2024-04-23T18:37:18.830 INFO:tasks.workunit.client.0.smithi050.stdout: 10 137010 19.83 MB/sec execute 230 sec latency 285.838 ms 2024-04-23T18:37:18.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:16 smithi050 ceph-mon[29187]: pgmap v596: 97 pgs: 97 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.7 MiB/s wr, 662 op/s 2024-04-23T18:37:18.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:16 smithi050 ceph-mon[29187]: osdmap e122: 12 total, 12 up, 12 in 2024-04-23T18:37:18.885 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:18 smithi177 ceph-mon[32308]: pgmap v598: 97 pgs: 97 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 500 op/s 2024-04-23T18:37:18.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:18 smithi188 ceph-mon[32694]: pgmap v598: 97 pgs: 97 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 500 op/s 2024-04-23T18:37:18.977 INFO:tasks.workunit.client.0.smithi050.stdout: 10 137712 19.89 MB/sec execute 231 sec latency 91.976 ms 2024-04-23T18:37:18.985 INFO:tasks.workunit.client.0.smithi050.stdout: 10 138409 19.99 MB/sec execute 232 sec latency 75.726 ms 2024-04-23T18:37:18.995 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:18 smithi050 ceph-mon[29187]: pgmap v598: 97 pgs: 97 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 500 op/s 2024-04-23T18:37:19.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 138961 20.07 MB/sec execute 233 sec latency 200.707 ms 2024-04-23T18:37:20.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 139390 20.08 MB/sec execute 234 sec latency 144.647 ms 2024-04-23T18:37:20.611 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:20 smithi188 ceph-mon[32694]: pgmap v599: 97 pgs: 97 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 512 op/s 2024-04-23T18:37:20.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:20 smithi050 ceph-mon[29187]: pgmap v599: 97 pgs: 97 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 512 op/s 2024-04-23T18:37:20.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:20 smithi177 ceph-mon[32308]: pgmap v599: 97 pgs: 97 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 512 op/s 2024-04-23T18:37:21.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 139806 20.06 MB/sec execute 235 sec latency 141.898 ms 2024-04-23T18:37:22.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 140370 20.11 MB/sec execute 236 sec latency 81.646 ms 2024-04-23T18:37:23.593 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:22 smithi177 ceph-mon[32308]: pgmap v600: 97 pgs: 97 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 512 op/s 2024-04-23T18:37:23.621 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:22 smithi188 ceph-mon[32694]: pgmap v600: 97 pgs: 97 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 512 op/s 2024-04-23T18:37:23.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:22 smithi050 ceph-mon[29187]: pgmap v600: 97 pgs: 97 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 512 op/s 2024-04-23T18:37:23.662 INFO:tasks.workunit.client.0.smithi050.stdout: 10 140898 20.13 MB/sec execute 237 sec latency 104.053 ms 2024-04-23T18:37:24.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 141489 20.15 MB/sec execute 238 sec latency 93.847 ms 2024-04-23T18:37:25.123 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:24 smithi177 ceph-mon[32308]: pgmap v601: 97 pgs: 97 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.9 MiB/s wr, 497 op/s 2024-04-23T18:37:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:24 smithi188 ceph-mon[32694]: pgmap v601: 97 pgs: 97 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.9 MiB/s wr, 497 op/s 2024-04-23T18:37:25.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:24 smithi050 ceph-mon[29187]: pgmap v601: 97 pgs: 97 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.9 MiB/s wr, 497 op/s 2024-04-23T18:37:25.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 142084 20.23 MB/sec execute 239 sec latency 90.069 ms 2024-04-23T18:37:26.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 142630 20.30 MB/sec execute 240 sec latency 131.448 ms 2024-04-23T18:37:26.597 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:26 smithi188 ceph-mon[32694]: pgmap v602: 97 pgs: 97 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.9 MiB/s wr, 497 op/s 2024-04-23T18:37:26.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:26 smithi050 ceph-mon[29187]: pgmap v602: 97 pgs: 97 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.9 MiB/s wr, 497 op/s 2024-04-23T18:37:26.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:26 smithi177 ceph-mon[32308]: pgmap v602: 97 pgs: 97 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.9 MiB/s wr, 497 op/s 2024-04-23T18:37:27.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 143133 20.31 MB/sec execute 241 sec latency 79.697 ms 2024-04-23T18:37:28.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.571+0000 7f33e1e91640 1 -- 172.21.15.50:0/1954402457 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33dc1010f0 msgr2=0x7f33dc1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:28.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.571+0000 7f33e1e91640 1 --2- 172.21.15.50:0/1954402457 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33dc1010f0 0x7f33dc1014d0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009900 tx=0x7f33cc02a410 comp rx=0 tx=0).stop 2024-04-23T18:37:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.571+0000 7f33e1e91640 1 -- 172.21.15.50:0/1954402457 shutdown_connections 2024-04-23T18:37:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.571+0000 7f33e1e91640 1 --2- 172.21.15.50:0/1954402457 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33dc106170 0x7f33dc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:28.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.571+0000 7f33e1e91640 1 --2- 172.21.15.50:0/1954402457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33dc101aa0 0x7f33dc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:28.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.571+0000 7f33e1e91640 1 --2- 172.21.15.50:0/1954402457 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33dc1010f0 0x7f33dc1014d0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:28.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.571+0000 7f33e1e91640 1 -- 172.21.15.50:0/1954402457 >> 172.21.15.50:0/1954402457 conn(0x7f33dc077fd0 msgr2=0x7f33dc0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.571+0000 7f33e1e91640 1 -- 172.21.15.50:0/1954402457 shutdown_connections 2024-04-23T18:37:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.571+0000 7f33e1e91640 1 -- 172.21.15.50:0/1954402457 wait complete. 2024-04-23T18:37:28.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:28 smithi188 ceph-mon[32694]: pgmap v603: 97 pgs: 97 active+clean; 1.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 617 op/s 2024-04-23T18:37:28.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33e1e91640 1 Processor -- start 2024-04-23T18:37:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33e1e91640 1 -- start start 2024-04-23T18:37:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33e1e91640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33dc1010f0 0x7f33dc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33e1e91640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33dc106170 0x7f33dc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:28.766 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33e1e91640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33dc19cfd0 0x7f33dc19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33e1e91640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1121f0 con 0x7f33dc19cfd0 2024-04-23T18:37:28.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33e1e91640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc112070 con 0x7f33dc106170 2024-04-23T18:37:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33e1e91640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc112370 con 0x7f33dc1010f0 2024-04-23T18:37:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33dbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33dc19cfd0 0x7f33dc19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33daffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33dc106170 0x7f33dc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:28.880 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33daffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33dc106170 0x7f33dc198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:60724/0 (socket says 172.21.15.50:60724) 2024-04-23T18:37:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33dbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33dc19cfd0 0x7f33dc19d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56868/0 (socket says 172.21.15.50:56868) 2024-04-23T18:37:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33daffd640 1 -- 172.21.15.50:0/2047483083 learned_addr learned my addr 172.21.15.50:0/2047483083 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.572+0000 7f33db7fe640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33dc1010f0 0x7f33dc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.573+0000 7f33daffd640 1 -- 172.21.15.50:0/2047483083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33dc1010f0 msgr2=0x7f33dc198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:28.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.573+0000 7f33daffd640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33dc1010f0 0x7f33dc198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.573+0000 7f33daffd640 1 -- 172.21.15.50:0/2047483083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33dc19cfd0 msgr2=0x7f33dc19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:28.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.573+0000 7f33daffd640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33dc19cfd0 0x7f33dc19d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:28.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.573+0000 7f33daffd640 1 -- 172.21.15.50:0/2047483083 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33dc19dac0 con 0x7f33dc106170 2024-04-23T18:37:28.996 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.573+0000 7f33db7fe640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33dc1010f0 0x7f33dc198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:37:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.573+0000 7f33daffd640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33dc106170 0x7f33dc198c50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f33d000e9c0 tx=0x7f33d000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.573+0000 7f33d8ff9640 1 -- 172.21.15.50:0/2047483083 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d000cd80 con 0x7f33dc106170 2024-04-23T18:37:29.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.573+0000 7f33e1e91640 1 -- 172.21.15.50:0/2047483083 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33dc19ddb0 con 0x7f33dc106170 2024-04-23T18:37:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.573+0000 7f33d8ff9640 1 -- 172.21.15.50:0/2047483083 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33d00054d0 con 0x7f33dc106170 2024-04-23T18:37:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.573+0000 7f33d8ff9640 1 -- 172.21.15.50:0/2047483083 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0010430 con 0x7f33dc106170 2024-04-23T18:37:29.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.573+0000 7f33e1e91640 1 -- 172.21.15.50:0/2047483083 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33dc10e970 con 0x7f33dc106170 2024-04-23T18:37:29.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33e4ba9740 1 Processor -- start 2024-04-23T18:37:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33e4ba9740 1 -- start start 2024-04-23T18:37:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33d8ff9640 1 -- 172.21.15.50:0/2047483083 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f33d0004fd0 con 0x7f33dc106170 2024-04-23T18:37:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33e4ba9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 0x55c393558a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33e4ba9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c3935632c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33e4ba9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393563800 0x55c393565bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33e4ba9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c3934bd970 con 0x55c393558620 2024-04-23T18:37:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33e4ba9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c39345a4a0 con 0x55c393558f40 2024-04-23T18:37:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33e4ba9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c3934b9930 con 0x55c393563800 2024-04-23T18:37:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33dbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 0x55c393558a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:29.214 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33dbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 0x55c393558a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56874/0 (socket says 172.21.15.50:56874) 2024-04-23T18:37:29.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33dbfff640 1 -- 172.21.15.50:0/1229438997 learned_addr learned my addr 172.21.15.50:0/1229438997 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33db7fe640 1 --2- 172.21.15.50:0/1229438997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c3935632c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:29.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.576+0000 7f33daffd640 1 --2- 172.21.15.50:0/1229438997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393563800 0x55c393565bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.577+0000 7f33dbfff640 1 -- 172.21.15.50:0/1229438997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393563800 msgr2=0x55c393565bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:29.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.577+0000 7f33dbfff640 1 --2- 172.21.15.50:0/1229438997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393563800 0x55c393565bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.577+0000 7f33dbfff640 1 -- 172.21.15.50:0/1229438997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 msgr2=0x55c3935632c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.577+0000 7f33dbfff640 1 --2- 172.21.15.50:0/1229438997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c3935632c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:29.370 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.577+0000 7f33dbfff640 1 -- 172.21.15.50:0/1229438997 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3931d0210 con 0x55c393558620 2024-04-23T18:37:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.577+0000 7f33dbfff640 1 --2- 172.21.15.50:0/1229438997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 0x55c393558a00 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f33d4009f90 tx=0x7f33d4010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc89eae113449d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.577+0000 7f33d8ff9640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f33c40a3dc0 0x7f33c40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.577+0000 7f33c1ffb640 1 -- 172.21.15.50:0/1229438997 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d4011700 con 0x55c393558620 2024-04-23T18:37:29.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.577+0000 7f33c1ffb640 1 -- 172.21.15.50:0/1229438997 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33d4011d50 con 0x55c393558620 2024-04-23T18:37:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.577+0000 7f33d8ff9640 1 -- 172.21.15.50:0/2047483083 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8542+0+0 (secure 0 0 0) 0x7f33d00d0d20 con 0x7f33dc106170 2024-04-23T18:37:29.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.577+0000 7f33db7fe640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f33c40a3dc0 0x7f33c40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.577+0000 7f33c1ffb640 1 -- 172.21.15.50:0/1229438997 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d4019da0 con 0x55c393558620 2024-04-23T18:37:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.578+0000 7f33e4ba9740 1 -- 172.21.15.50:0/1229438997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 msgr2=0x55c393558a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.578+0000 7f33db7fe640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f33c40a3dc0 0x7f33c40a6280 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f33cc031440 tx=0x7f33cc005b50 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:29.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.578+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/1229438997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 0x55c393558a00 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f33d4009f90 tx=0x7f33d4010860 comp rx=0 tx=0).stop 2024-04-23T18:37:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.578+0000 7f33e4ba9740 1 -- 172.21.15.50:0/1229438997 shutdown_connections 2024-04-23T18:37:29.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.578+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/1229438997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393563800 0x55c393565bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.578+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/1229438997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c3935632c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.578+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/1229438997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 0x55c393558a00 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.578+0000 7f33e4ba9740 1 -- 172.21.15.50:0/1229438997 >> 172.21.15.50:0/1229438997 conn(0x55c393554b30 msgr2=0x55c393554f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.578+0000 7f33e4ba9740 1 -- 172.21.15.50:0/1229438997 shutdown_connections 2024-04-23T18:37:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.578+0000 7f33e4ba9740 1 -- 172.21.15.50:0/1229438997 wait complete. 2024-04-23T18:37:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33e4ba9740 1 -- start start 2024-04-23T18:37:29.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33e4ba9740 1 Processor -- start 2024-04-23T18:37:29.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33e4ba9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 0x55c3935631a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:29.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33e4ba9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c393560490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:29.650 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33e4ba9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393563800 0x55c393560a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33e4ba9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c393566430 con 0x55c393558620 2024-04-23T18:37:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33e4ba9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c3934580b0 con 0x55c393558f40 2024-04-23T18:37:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33e4ba9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c3935665b0 con 0x55c393563800 2024-04-23T18:37:29.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33e4ba9740 10 client.0 _ll_register_callbacks cb 0x55c393445d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:37:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33db7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 0x55c3935631a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:29.716 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33db7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 0x55c3935631a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56882/0 (socket says 172.21.15.50:56882) 2024-04-23T18:37:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33daffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c393560490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33db7fe640 1 -- 172.21.15.50:0/3536638400 learned_addr learned my addr 172.21.15.50:0/3536638400 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.579+0000 7f33dbfff640 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393563800 0x55c393560a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33daffd640 1 -- 172.21.15.50:0/3536638400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393563800 msgr2=0x55c393560a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33daffd640 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393563800 0x55c393560a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:29.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33daffd640 1 -- 172.21.15.50:0/3536638400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 msgr2=0x55c3935631a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:29.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33daffd640 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 0x55c3935631a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33daffd640 1 -- 172.21.15.50:0/3536638400 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c39329c620 con 0x55c393558f40 2024-04-23T18:37:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33db7fe640 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 0x55c3935631a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:37:29.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33daffd640 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c393560490 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f33d00ce400 tx=0x7f33d009cae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0d1ab6e6757dc01 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33baffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:37:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33baffd640 1 -- 172.21.15.50:0/3536638400 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d00ce880 con 0x55c393558f40 2024-04-23T18:37:29.915 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3536638400 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33d4011d50 con 0x55c393558f40 2024-04-23T18:37:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33baffd640 1 -- 172.21.15.50:0/3536638400 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8542+0+0 (secure 0 0 0) 0x7f33d009ceb0 con 0x55c393558f40 2024-04-23T18:37:29.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33baffd640 1 -- 172.21.15.50:0/3536638400 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f33d009d350 con 0x55c393558f40 2024-04-23T18:37:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33e4ba9740 10 client.27320 fetch_fsmap learned FSMap version 62 2024-04-23T18:37:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3536638400 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55c3935723a0 con 0x55c393558f40 2024-04-23T18:37:29.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.580+0000 7f33baffd640 1 -- 172.21.15.50:0/3536638400 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33d4011d50 con 0x55c393558f40 2024-04-23T18:37:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.581+0000 7f33baffd640 1 -- 172.21.15.50:0/3536638400 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f33d00a5430 con 0x55c393558f40 2024-04-23T18:37:29.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.581+0000 7f33e4ba9740 10 client.27320 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:37:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.581+0000 7f33e4ba9740 10 client.27320 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:37:30.029 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.581+0000 7f33e4ba9740 20 client.27320 populate_metadata read hostname 'smithi050' 2024-04-23T18:37:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.581+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c393573860 0x55c393575cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.581+0000 7f33e4ba9740 4 client.27320 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:37:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.581+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3536638400 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3934cb390 con 0x55c393573860 2024-04-23T18:37:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.581+0000 7f33db7fe640 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c393573860 0x55c393575cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.581+0000 7f33baffd640 10 client.27320 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.581+0000 7f33db7fe640 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c393573860 0x55c393575cb0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b3a93e1989d2bfa server_cookie=62b4d37f108422b in_seq=0 out_seq=0 2024-04-23T18:37:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33baffd640 1 -- 172.21.15.50:0/3536638400 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c3934cb390 con 0x55c393573860 2024-04-23T18:37:30.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33baffd640 10 client.27320 handle_command_reply: tid=0 2024-04-23T18:37:30.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33e4ba9740 1 client.27320 shutdown 2024-04-23T18:37:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3536638400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 msgr2=0x55c393560490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:30.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c393560490 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f33d00ce400 tx=0x7f33d009cae0 comp rx=0 tx=0).stop 2024-04-23T18:37:30.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3536638400 shutdown_connections 2024-04-23T18:37:30.185 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c393573860 0x55c393575cb0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393563800 0x55c393560a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33baffd640 0 client.27320 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c393560490 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3536638400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558620 0x55c3935631a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3536638400 >> 172.21.15.50:0/3536638400 conn(0x55c393554b30 msgr2=0x55c39355a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3536638400 shutdown_connections 2024-04-23T18:37:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3536638400 wait complete. 2024-04-23T18:37:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.582+0000 7f33e4ba9740 20 client.27320 trim_cache size 0 max 16384 2024-04-23T18:37:30.306 INFO:tasks.workunit.client.0.smithi050.stdout: 10 143685 20.36 MB/sec execute 242 sec latency 105.312 ms 2024-04-23T18:37:30.315 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:28 smithi177 ceph-mon[32308]: pgmap v603: 97 pgs: 97 active+clean; 1.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 617 op/s 2024-04-23T18:37:30.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.599+0000 7f33e4ba9740 1 Processor -- start 2024-04-23T18:37:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33e4ba9740 1 -- start start 2024-04-23T18:37:30.355 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33e4ba9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393558620 0x55c39356f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33e4ba9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c39357b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33e4ba9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393563800 0x55c39357bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33e4ba9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1b2730 con 0x55c393563800 2024-04-23T18:37:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33e4ba9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1b0000 con 0x55c393558f40 2024-04-23T18:37:30.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33e4ba9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1ade70 con 0x55c393558620 2024-04-23T18:37:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33dbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393563800 0x55c39357bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:30.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33daffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c39357b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33dbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393563800 0x55c39357bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56886/0 (socket says 172.21.15.50:56886) 2024-04-23T18:37:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33dbfff640 1 -- 172.21.15.50:0/3119243273 learned_addr learned my addr 172.21.15.50:0/3119243273 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33db7fe640 1 --2- 172.21.15.50:0/3119243273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393558620 0x55c39356f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:30.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33dbfff640 1 -- 172.21.15.50:0/3119243273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393558620 msgr2=0x55c39356f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33dbfff640 1 --2- 172.21.15.50:0/3119243273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393558620 0x55c39356f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33dbfff640 1 -- 172.21.15.50:0/3119243273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 msgr2=0x55c39357b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:30.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33dbfff640 1 --2- 172.21.15.50:0/3119243273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c39357b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:30.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.600+0000 7f33dbfff640 1 -- 172.21.15.50:0/3119243273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c39355f480 con 0x55c393563800 2024-04-23T18:37:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.601+0000 7f33dbfff640 1 --2- 172.21.15.50:0/3119243273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393563800 0x55c39357bb00 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f33d4000f80 tx=0x7f33d4009940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6118300cf195f4f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.601+0000 7f33c17fa640 1 -- 172.21.15.50:0/3119243273 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d4028070 con 0x55c393563800 2024-04-23T18:37:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.601+0000 7f33c17fa640 1 -- 172.21.15.50:0/3119243273 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33d4009b70 con 0x55c393563800 2024-04-23T18:37:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.601+0000 7f33c17fa640 1 -- 172.21.15.50:0/3119243273 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d4004280 con 0x55c393563800 2024-04-23T18:37:30.555 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.602+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3119243273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393563800 msgr2=0x55c39357bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.602+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3119243273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393563800 0x55c39357bb00 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f33d4000f80 tx=0x7f33d4009940 comp rx=0 tx=0).stop 2024-04-23T18:37:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.602+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3119243273 shutdown_connections 2024-04-23T18:37:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.602+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3119243273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393563800 0x55c39357bb00 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.602+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3119243273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393558f40 0x55c39357b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.602+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3119243273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393558620 0x55c39356f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.602+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3119243273 >> 172.21.15.50:0/3119243273 conn(0x55c393554b30 msgr2=0x55c393553cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.602+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3119243273 shutdown_connections 2024-04-23T18:37:30.684 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.602+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3119243273 wait complete. 2024-04-23T18:37:30.693 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.602+0000 7f33e4ba9740 1 -- start start 2024-04-23T18:37:30.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.602+0000 7f33e4ba9740 1 Processor -- start 2024-04-23T18:37:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33e4ba9740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393558620 0x55c393569600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:30.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33e4ba9740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558f40 0x55c393569b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33e4ba9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393563800 0x55c3935d8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33e4ba9740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1ae4c0 con 0x55c393558f40 2024-04-23T18:37:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33e4ba9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1af930 con 0x55c393563800 2024-04-23T18:37:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33e4ba9740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1aeb90 con 0x55c393558620 2024-04-23T18:37:30.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33e4ba9740 10 client.0 _ll_register_callbacks cb 0x7f33d0005680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:37:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33daffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558f40 0x55c393569b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33daffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558f40 0x55c393569b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56888/0 (socket says 172.21.15.50:56888) 2024-04-23T18:37:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33daffd640 1 -- 172.21.15.50:0/3744785787 learned_addr learned my addr 172.21.15.50:0/3744785787 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33db7fe640 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393558620 0x55c393569600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33dbfff640 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393563800 0x55c3935d8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33daffd640 1 -- 172.21.15.50:0/3744785787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393558620 msgr2=0x55c393569600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33daffd640 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393558620 0x55c393569600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33daffd640 1 -- 172.21.15.50:0/3744785787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393563800 msgr2=0x55c3935d8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33daffd640 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393563800 0x55c3935d8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33daffd640 1 -- 172.21.15.50:0/3744785787 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c39356a410 con 0x55c393558f40 2024-04-23T18:37:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33dbfff640 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393563800 0x55c3935d8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:37:31.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33daffd640 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558f40 0x55c393569b40 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f33d0099620 tx=0x7f33d00049b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22632f90b22eae7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:31.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.603+0000 7f33baffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:37:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33baffd640 1 -- 172.21.15.50:0/3744785787 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0066060 con 0x55c393558f40 2024-04-23T18:37:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33baffd640 1 -- 172.21.15.50:0/3744785787 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8542+0+0 (secure 0 0 0) 0x7f33d0061bd0 con 0x55c393558f40 2024-04-23T18:37:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3744785787 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33d4009b70 con 0x55c393558f40 2024-04-23T18:37:31.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33baffd640 1 -- 172.21.15.50:0/3744785787 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f33d00d3040 con 0x55c393558f40 2024-04-23T18:37:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33e4ba9740 10 client.17832 fetch_fsmap learned FSMap version 62 2024-04-23T18:37:31.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3744785787 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55c3935da460 con 0x55c393558f40 2024-04-23T18:37:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33baffd640 1 -- 172.21.15.50:0/3744785787 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33d4009b70 con 0x55c393558f40 2024-04-23T18:37:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33baffd640 1 -- 172.21.15.50:0/3744785787 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f33d009c0a0 con 0x55c393558f40 2024-04-23T18:37:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33e4ba9740 10 client.17832 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:37:31.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33e4ba9740 10 client.17832 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:37:31.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33e4ba9740 20 client.17832 populate_metadata read hostname 'smithi050' 2024-04-23T18:37:31.187 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c3935dac10 0x55c3935dd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:31.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33e4ba9740 4 client.17832 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:37:31.204 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3744785787 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c3934c5200 con 0x55c3935dac10 2024-04-23T18:37:31.212 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.604+0000 7f33db7fe640 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c3935dac10 0x55c3935dd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:31.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.605+0000 7f33db7fe640 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c3935dac10 0x55c3935dd030 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82ecaad9f3d90135 server_cookie=cec34ad7b3c34d5e in_seq=0 out_seq=0 2024-04-23T18:37:31.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:28.605+0000 7f33baffd640 10 client.17832 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:31.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:28 smithi050 ceph-mon[29187]: pgmap v603: 97 pgs: 97 active+clean; 1.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 9.6 MiB/s wr, 617 op/s 2024-04-23T18:37:31.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33baffd640 1 -- 172.21.15.50:0/3744785787 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c3934c5200 con 0x55c3935dac10 2024-04-23T18:37:31.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33baffd640 10 client.17832 handle_command_reply: tid=0 2024-04-23T18:37:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33e4ba9740 1 client.17832 shutdown 2024-04-23T18:37:31.283 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3744785787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558f40 msgr2=0x55c393569b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:31.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558f40 0x55c393569b40 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f33d0099620 tx=0x7f33d00049b0 comp rx=0 tx=0).stop 2024-04-23T18:37:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3744785787 shutdown_connections 2024-04-23T18:37:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c3935dac10 0x55c3935dd030 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:31.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c393563800 0x55c3935d8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33baffd640 0 client.17832 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c393558f40 0x55c393569b40 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:31.368 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33e4ba9740 1 --2- 172.21.15.50:0/3744785787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c393558620 0x55c393569600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:31.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3744785787 >> 172.21.15.50:0/3744785787 conn(0x55c393554b30 msgr2=0x55c39356e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3744785787 shutdown_connections 2024-04-23T18:37:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33e4ba9740 1 -- 172.21.15.50:0/3744785787 wait complete. 2024-04-23T18:37:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.030+0000 7f33e4ba9740 20 client.17832 trim_cache size 0 max 16384 2024-04-23T18:37:31.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:37:31.431 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:37:31.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:37:31.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:37:31.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.031+0000 7f33e1e91640 1 -- 172.21.15.50:0/2047483083 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f33c40a3dc0 msgr2=0x7f33c40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:31.459 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.031+0000 7f33e1e91640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f33c40a3dc0 0x7f33c40a6280 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f33cc031440 tx=0x7f33cc005b50 comp rx=0 tx=0).stop 2024-04-23T18:37:31.467 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.031+0000 7f33e1e91640 1 -- 172.21.15.50:0/2047483083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33dc106170 msgr2=0x7f33dc198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.031+0000 7f33e1e91640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33dc106170 0x7f33dc198c50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f33d000e9c0 tx=0x7f33d000ee80 comp rx=0 tx=0).stop 2024-04-23T18:37:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.031+0000 7f33e1e91640 1 -- 172.21.15.50:0/2047483083 shutdown_connections 2024-04-23T18:37:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.031+0000 7f33e1e91640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f33c40a3dc0 0x7f33c40a6280 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:31.571 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.031+0000 7f33e1e91640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33dc19cfd0 0x7f33dc19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:31.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.031+0000 7f33e1e91640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33dc106170 0x7f33dc198c50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:31.585 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.031+0000 7f33e1e91640 1 --2- 172.21.15.50:0/2047483083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f33dc1010f0 0x7f33dc198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.031+0000 7f33e1e91640 1 -- 172.21.15.50:0/2047483083 >> 172.21.15.50:0/2047483083 conn(0x7f33dc077fd0 msgr2=0x7f33dc0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:31.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.031+0000 7f33e1e91640 1 -- 172.21.15.50:0/2047483083 shutdown_connections 2024-04-23T18:37:31.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:29.031+0000 7f33e1e91640 1 -- 172.21.15.50:0/2047483083 wait complete. 2024-04-23T18:37:31.611 INFO:tasks.workunit.client.0.smithi050.stdout: 10 144062 20.38 MB/sec execute 243 sec latency 440.664 ms 2024-04-23T18:37:31.634 INFO:tasks.workunit.client.0.smithi050.stdout: 10 144525 20.37 MB/sec execute 244 sec latency 215.216 ms 2024-04-23T18:37:31.643 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:30 smithi177 ceph-mon[32308]: pgmap v604: 97 pgs: 97 active+clean; 1.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.6 MiB/s wr, 455 op/s 2024-04-23T18:37:31.650 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:30 smithi188 ceph-mon[32694]: pgmap v604: 97 pgs: 97 active+clean; 1.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.6 MiB/s wr, 455 op/s 2024-04-23T18:37:31.658 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:30 smithi050 ceph-mon[29187]: pgmap v604: 97 pgs: 97 active+clean; 1.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.6 MiB/s wr, 455 op/s 2024-04-23T18:37:31.664 INFO:tasks.workunit.client.0.smithi050.stdout: 10 145066 20.38 MB/sec execute 245 sec latency 108.463 ms 2024-04-23T18:37:32.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 145734 20.47 MB/sec execute 246 sec latency 87.656 ms 2024-04-23T18:37:32.604 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:32 smithi188 ceph-mon[32694]: pgmap v605: 97 pgs: 97 active+clean; 1.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 350 op/s 2024-04-23T18:37:32.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:32 smithi050 ceph-mon[29187]: pgmap v605: 97 pgs: 97 active+clean; 1.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 350 op/s 2024-04-23T18:37:32.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:32 smithi177 ceph-mon[32308]: pgmap v605: 97 pgs: 97 active+clean; 1.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 350 op/s 2024-04-23T18:37:33.576 INFO:tasks.workunit.client.0.smithi050.stdout: 10 146250 20.51 MB/sec execute 247 sec latency 178.696 ms 2024-04-23T18:37:34.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 146831 20.53 MB/sec execute 248 sec latency 123.948 ms 2024-04-23T18:37:34.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:34 smithi050 ceph-mon[29187]: pgmap v606: 97 pgs: 97 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.8 MiB/s wr, 599 op/s 2024-04-23T18:37:34.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:34 smithi177 ceph-mon[32308]: pgmap v606: 97 pgs: 97 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.8 MiB/s wr, 599 op/s 2024-04-23T18:37:34.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:34 smithi188 ceph-mon[32694]: pgmap v606: 97 pgs: 97 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.8 MiB/s wr, 599 op/s 2024-04-23T18:37:35.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 147388 20.59 MB/sec execute 249 sec latency 92.356 ms 2024-04-23T18:37:36.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 147978 20.62 MB/sec execute 250 sec latency 115.368 ms 2024-04-23T18:37:36.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:36 smithi050 ceph-mon[29187]: pgmap v607: 97 pgs: 97 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 422 op/s 2024-04-23T18:37:36.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:36 smithi177 ceph-mon[32308]: pgmap v607: 97 pgs: 97 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 422 op/s 2024-04-23T18:37:36.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:36 smithi188 ceph-mon[32694]: pgmap v607: 97 pgs: 97 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 422 op/s 2024-04-23T18:37:37.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 148501 20.65 MB/sec execute 251 sec latency 123.422 ms 2024-04-23T18:37:38.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 148970 20.66 MB/sec execute 252 sec latency 172.666 ms 2024-04-23T18:37:38.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:38 smithi050 ceph-mon[29187]: pgmap v608: 97 pgs: 97 active+clean; 1.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.9 MiB/s wr, 546 op/s 2024-04-23T18:37:38.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:38 smithi177 ceph-mon[32308]: pgmap v608: 97 pgs: 97 active+clean; 1.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.9 MiB/s wr, 546 op/s 2024-04-23T18:37:38.839 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:38 smithi188 ceph-mon[32694]: pgmap v608: 97 pgs: 97 active+clean; 1.3 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.9 MiB/s wr, 546 op/s 2024-04-23T18:37:39.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 149506 20.72 MB/sec execute 253 sec latency 113.933 ms 2024-04-23T18:37:40.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 150046 20.74 MB/sec execute 254 sec latency 94.429 ms 2024-04-23T18:37:40.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:40 smithi050 ceph-mon[29187]: pgmap v609: 97 pgs: 97 active+clean; 1.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.9 MiB/s wr, 495 op/s 2024-04-23T18:37:40.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:40 smithi177 ceph-mon[32308]: pgmap v609: 97 pgs: 97 active+clean; 1.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.9 MiB/s wr, 495 op/s 2024-04-23T18:37:40.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:40 smithi188 ceph-mon[32694]: pgmap v609: 97 pgs: 97 active+clean; 1.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.9 MiB/s wr, 495 op/s 2024-04-23T18:37:41.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 150605 20.78 MB/sec execute 255 sec latency 86.899 ms 2024-04-23T18:37:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.136+0000 7ff8be9f9640 1 -- 172.21.15.50:0/4100352304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b810c520 msgr2=0x7ff8b810e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.136+0000 7ff8be9f9640 1 --2- 172.21.15.50:0/4100352304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b810c520 0x7ff8b810e9c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009be0 tx=0x7ff8b002a750 comp rx=0 tx=0).stop 2024-04-23T18:37:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.136+0000 7ff8be9f9640 1 -- 172.21.15.50:0/4100352304 shutdown_connections 2024-04-23T18:37:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.136+0000 7ff8be9f9640 1 --2- 172.21.15.50:0/4100352304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b810c520 0x7ff8b810e9c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:42.205 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.136+0000 7ff8be9f9640 1 --2- 172.21.15.50:0/4100352304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8b8103a20 0x7ff8b810bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.136+0000 7ff8be9f9640 1 --2- 172.21.15.50:0/4100352304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8b8103100 0x7ff8b81034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.136+0000 7ff8be9f9640 1 -- 172.21.15.50:0/4100352304 >> 172.21.15.50:0/4100352304 conn(0x7ff8b80fc920 msgr2=0x7ff8b80fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.136+0000 7ff8be9f9640 1 -- 172.21.15.50:0/4100352304 shutdown_connections 2024-04-23T18:37:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.136+0000 7ff8be9f9640 1 -- 172.21.15.50:0/4100352304 wait complete. 2024-04-23T18:37:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.137+0000 7ff8be9f9640 1 Processor -- start 2024-04-23T18:37:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.137+0000 7ff8be9f9640 1 -- start start 2024-04-23T18:37:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8be9f9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8b8103100 0x7ff8b8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8be9f9640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8b8103a20 0x7ff8b8198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8be9f9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b810c520 0x7ff8b819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8be9f9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b8112190 con 0x7ff8b8103100 2024-04-23T18:37:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8be9f9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b8112010 con 0x7ff8b810c520 2024-04-23T18:37:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8be9f9640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b8112310 con 0x7ff8b8103a20 2024-04-23T18:37:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8b8103100 0x7ff8b8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8b8103100 0x7ff8b8198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53048/0 (socket says 172.21.15.50:53048) 2024-04-23T18:37:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8b7fff640 1 -- 172.21.15.50:0/3909282144 learned_addr learned my addr 172.21.15.50:0/3909282144 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8bcf96640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b810c520 0x7ff8b819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8b77fe640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8b8103a20 0x7ff8b8198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8b7fff640 1 -- 172.21.15.50:0/3909282144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8b8103a20 msgr2=0x7ff8b8198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:42.458 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8b8103a20 0x7ff8b8198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:42.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8b7fff640 1 -- 172.21.15.50:0/3909282144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b810c520 msgr2=0x7ff8b819cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b810c520 0x7ff8b819cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8b7fff640 1 -- 172.21.15.50:0/3909282144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b819d6d0 con 0x7ff8b8103100 2024-04-23T18:37:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8b8103100 0x7ff8b8198730 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800e9c0 tx=0x7ff8a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8b57fa640 1 -- 172.21.15.50:0/3909282144 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a800ccc0 con 0x7ff8b8103100 2024-04-23T18:37:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8b57fa640 1 -- 172.21.15.50:0/3909282144 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8a80054d0 con 0x7ff8b8103100 2024-04-23T18:37:42.562 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.138+0000 7ff8be9f9640 1 -- 172.21.15.50:0/3909282144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b819d9c0 con 0x7ff8b8103100 2024-04-23T18:37:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.139+0000 7ff8b57fa640 1 -- 172.21.15.50:0/3909282144 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8010430 con 0x7ff8b8103100 2024-04-23T18:37:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.139+0000 7ff8be9f9640 1 -- 172.21.15.50:0/3909282144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b81a5250 con 0x7ff8b8103100 2024-04-23T18:37:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.141+0000 7ff8c17df740 1 Processor -- start 2024-04-23T18:37:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.141+0000 7ff8b57fa640 1 -- 172.21.15.50:0/3909282144 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff8a80105d0 con 0x7ff8b8103100 2024-04-23T18:37:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.141+0000 7ff8c17df740 1 -- start start 2024-04-23T18:37:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.141+0000 7ff8c17df740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44c620 0x564eba44ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.141+0000 7ff8c17df740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44cf40 0x564eba4572c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:42.657 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.141+0000 7ff8c17df740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 0x564eba459bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:42.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.141+0000 7ff8c17df740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564eba3b1970 con 0x564eba457800 2024-04-23T18:37:42.672 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.141+0000 7ff8c17df740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564eba34e4a0 con 0x564eba44cf40 2024-04-23T18:37:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.141+0000 7ff8c17df740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564eba3ad930 con 0x564eba44c620 2024-04-23T18:37:42.685 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.141+0000 7ff8b77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 0x564eba459bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:42.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8b77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 0x564eba459bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53050/0 (socket says 172.21.15.50:53050) 2024-04-23T18:37:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8bcf96640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44c620 0x564eba44ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:42.740 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8b77fe640 1 -- 172.21.15.50:0/3857546331 learned_addr learned my addr 172.21.15.50:0/3857546331 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:42.750 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/3857546331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44cf40 0x564eba4572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8b77fe640 1 -- 172.21.15.50:0/3857546331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44c620 msgr2=0x564eba44ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8b77fe640 1 --2- 172.21.15.50:0/3857546331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44c620 0x564eba44ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:42.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8b77fe640 1 -- 172.21.15.50:0/3857546331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44cf40 msgr2=0x564eba4572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8b77fe640 1 --2- 172.21.15.50:0/3857546331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44cf40 0x564eba4572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:42.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8b77fe640 1 -- 172.21.15.50:0/3857546331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eba0c4210 con 0x564eba457800 2024-04-23T18:37:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8b57fa640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff8a00a3dc0 0x7ff8a00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8b77fe640 1 --2- 172.21.15.50:0/3857546331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 0x564eba459bf0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009c00 tx=0x7ff8ac011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252f80318e550c40 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:42.909 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8b57fa640 1 -- 172.21.15.50:0/3909282144 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8542+0+0 (secure 0 0 0) 0x7ff8a80d0c00 con 0x7ff8b8103100 2024-04-23T18:37:42.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff8bcf96640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff8a00a3dc0 0x7ff8a00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:42.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff89e7fc640 1 -- 172.21.15.50:0/3857546331 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac004230 con 0x564eba457800 2024-04-23T18:37:42.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff89e7fc640 1 -- 172.21.15.50:0/3857546331 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8ac0043d0 con 0x564eba457800 2024-04-23T18:37:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.142+0000 7ff89e7fc640 1 -- 172.21.15.50:0/3857546331 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac019690 con 0x564eba457800 2024-04-23T18:37:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.143+0000 7ff8c17df740 1 -- 172.21.15.50:0/3857546331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 msgr2=0x564eba459bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:42.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.143+0000 7ff8c17df740 1 --2- 172.21.15.50:0/3857546331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 0x564eba459bf0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009c00 tx=0x7ff8ac011810 comp rx=0 tx=0).stop 2024-04-23T18:37:42.979 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.143+0000 7ff8bcf96640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff8a00a3dc0 0x7ff8a00a6280 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0006010 tx=0x7ff8b0004a40 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.143+0000 7ff8c17df740 1 -- 172.21.15.50:0/3857546331 shutdown_connections 2024-04-23T18:37:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.143+0000 7ff8c17df740 1 --2- 172.21.15.50:0/3857546331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 0x564eba459bf0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.143+0000 7ff8c17df740 1 --2- 172.21.15.50:0/3857546331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44cf40 0x564eba4572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.143+0000 7ff8c17df740 1 --2- 172.21.15.50:0/3857546331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44c620 0x564eba44ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.143+0000 7ff8c17df740 1 -- 172.21.15.50:0/3857546331 >> 172.21.15.50:0/3857546331 conn(0x564eba448b30 msgr2=0x564eba448f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:43.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.143+0000 7ff8c17df740 1 -- 172.21.15.50:0/3857546331 shutdown_connections 2024-04-23T18:37:43.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.143+0000 7ff8c17df740 1 -- 172.21.15.50:0/3857546331 wait complete. 2024-04-23T18:37:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8c17df740 1 -- start start 2024-04-23T18:37:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8c17df740 1 Processor -- start 2024-04-23T18:37:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8c17df740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 0x564eba457110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8c17df740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44cf40 0x564eba454400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8c17df740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba457800 0x564eba454940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8c17df740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564eba45a430 con 0x564eba44c620 2024-04-23T18:37:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8c17df740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564eba34c0b0 con 0x564eba44cf40 2024-04-23T18:37:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8c17df740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x564eba45a5b0 con 0x564eba457800 2024-04-23T18:37:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 0x564eba457110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8c17df740 10 client.0 _ll_register_callbacks cb 0x564eba339d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:37:43.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8b77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44cf40 0x564eba454400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8b77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44cf40 0x564eba454400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:38280/0 (socket says 172.21.15.50:38280) 2024-04-23T18:37:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 0x564eba457110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53054/0 (socket says 172.21.15.50:53054) 2024-04-23T18:37:43.334 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8b77fe640 1 -- 172.21.15.50:0/3638226880 learned_addr learned my addr 172.21.15.50:0/3638226880 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:43.342 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.144+0000 7ff8bcf96640 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba457800 0x564eba454940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8b77fe640 1 -- 172.21.15.50:0/3638226880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba457800 msgr2=0x564eba454940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8b77fe640 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba457800 0x564eba454940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:43.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8b77fe640 1 -- 172.21.15.50:0/3638226880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 msgr2=0x564eba457110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8b77fe640 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 0x564eba457110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8b77fe640 1 -- 172.21.15.50:0/3638226880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eba190620 con 0x564eba44cf40 2024-04-23T18:37:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8bcf96640 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba457800 0x564eba454940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:37:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 0x564eba457110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:37:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8b77fe640 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44cf40 0x564eba454400 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac000c00 tx=0x7ff8ac0038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f9a8a43976ee0db server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8977fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:37:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8977fe640 1 -- 172.21.15.50:0/3638226880 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac028020 con 0x564eba44cf40 2024-04-23T18:37:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8c17df740 1 -- 172.21.15.50:0/3638226880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8ac0043d0 con 0x564eba44cf40 2024-04-23T18:37:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8977fe640 1 -- 172.21.15.50:0/3638226880 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8542+0+0 (secure 0 0 0) 0x7ff8ac005240 con 0x564eba44cf40 2024-04-23T18:37:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8977fe640 1 -- 172.21.15.50:0/3638226880 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7ff8ac033050 con 0x564eba44cf40 2024-04-23T18:37:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8c17df740 10 client.27338 fetch_fsmap learned FSMap version 62 2024-04-23T18:37:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.145+0000 7ff8c17df740 1 -- 172.21.15.50:0/3638226880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x564eba46a720 con 0x564eba44cf40 2024-04-23T18:37:43.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.146+0000 7ff8977fe640 1 -- 172.21.15.50:0/3638226880 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8ac0043d0 con 0x564eba44cf40 2024-04-23T18:37:43.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.146+0000 7ff8977fe640 1 -- 172.21.15.50:0/3638226880 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7ff8ac018070 con 0x564eba44cf40 2024-04-23T18:37:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.146+0000 7ff8c17df740 10 client.27338 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:37:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.146+0000 7ff8c17df740 10 client.27338 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:37:43.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.146+0000 7ff8c17df740 20 client.27338 populate_metadata read hostname 'smithi050' 2024-04-23T18:37:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.146+0000 7ff8c17df740 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564eba46bb30 0x564eba46df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.146+0000 7ff8c17df740 4 client.27338 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:37:43.685 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.146+0000 7ff8c17df740 1 -- 172.21.15.50:0/3638226880 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564eba3bf390 con 0x564eba46bb30 2024-04-23T18:37:43.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.146+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564eba46bb30 0x564eba46df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.146+0000 7ff8977fe640 10 client.27338 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:43.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.146+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564eba46bb30 0x564eba46df20 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd7ec44325171c2c server_cookie=89b1457483cb09f5 in_seq=0 out_seq=0 2024-04-23T18:37:43.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8977fe640 1 -- 172.21.15.50:0/3638226880 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564eba3bf390 con 0x564eba46bb30 2024-04-23T18:37:43.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8977fe640 10 client.27338 handle_command_reply: tid=0 2024-04-23T18:37:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8c17df740 1 client.27338 shutdown 2024-04-23T18:37:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8c17df740 1 -- 172.21.15.50:0/3638226880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44cf40 msgr2=0x564eba454400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:43.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8c17df740 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44cf40 0x564eba454400 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac000c00 tx=0x7ff8ac0038a0 comp rx=0 tx=0).stop 2024-04-23T18:37:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8c17df740 1 -- 172.21.15.50:0/3638226880 shutdown_connections 2024-04-23T18:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8c17df740 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564eba46bb30 0x564eba46df20 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:43.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8c17df740 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba457800 0x564eba454940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8977fe640 0 client.27338 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8c17df740 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44cf40 0x564eba454400 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8c17df740 1 --2- 172.21.15.50:0/3638226880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 0x564eba457110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:43.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8c17df740 1 -- 172.21.15.50:0/3638226880 >> 172.21.15.50:0/3638226880 conn(0x564eba448b30 msgr2=0x564eba44e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8c17df740 1 -- 172.21.15.50:0/3638226880 shutdown_connections 2024-04-23T18:37:43.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8c17df740 1 -- 172.21.15.50:0/3638226880 wait complete. 2024-04-23T18:37:43.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.148+0000 7ff8c17df740 20 client.27338 trim_cache size 0 max 16384 2024-04-23T18:37:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.165+0000 7ff8c17df740 1 Processor -- start 2024-04-23T18:37:43.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.165+0000 7ff8c17df740 1 -- start start 2024-04-23T18:37:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8c17df740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44c620 0x564eba44f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8c17df740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44cf40 0x564eba456c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:43.982 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8c17df740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 0x564eba4736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8c17df740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b81ae330 con 0x564eba457800 2024-04-23T18:37:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8c17df740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b81abc00 con 0x564eba44c620 2024-04-23T18:37:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8c17df740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b81a9a70 con 0x564eba44cf40 2024-04-23T18:37:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8bcf96640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 0x564eba4736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44c620 0x564eba44f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8bcf96640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 0x564eba4736d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53060/0 (socket says 172.21.15.50:53060) 2024-04-23T18:37:44.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44c620 0x564eba44f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:38282/0 (socket says 172.21.15.50:38282) 2024-04-23T18:37:44.127 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8bcf96640 1 -- 172.21.15.50:0/2941665732 learned_addr learned my addr 172.21.15.50:0/2941665732 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8b77fe640 1 --2- 172.21.15.50:0/2941665732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44cf40 0x564eba456c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8b7fff640 1 -- 172.21.15.50:0/2941665732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44cf40 msgr2=0x564eba456c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/2941665732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44cf40 0x564eba456c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8b7fff640 1 -- 172.21.15.50:0/2941665732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 msgr2=0x564eba4736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/2941665732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 0x564eba4736d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8b7fff640 1 -- 172.21.15.50:0/2941665732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eba444500 con 0x564eba44c620 2024-04-23T18:37:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8b77fe640 1 --2- 172.21.15.50:0/2941665732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44cf40 0x564eba456c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:37:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8bcf96640 1 --2- 172.21.15.50:0/2941665732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 0x564eba4736d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:37:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.166+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/2941665732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44c620 0x564eba44f970 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80ce340 tx=0x7ff8a80d4750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b66d12ee8c4d887 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.167+0000 7ff89dffb640 1 -- 172.21.15.50:0/2941665732 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80b6070 con 0x564eba44c620 2024-04-23T18:37:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.167+0000 7ff89dffb640 1 -- 172.21.15.50:0/2941665732 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8a809d290 con 0x564eba44c620 2024-04-23T18:37:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.167+0000 7ff89dffb640 1 -- 172.21.15.50:0/2941665732 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80a0f40 con 0x564eba44c620 2024-04-23T18:37:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.167+0000 7ff8c17df740 1 -- 172.21.15.50:0/2941665732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44c620 msgr2=0x564eba44f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.167+0000 7ff8c17df740 1 --2- 172.21.15.50:0/2941665732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44c620 0x564eba44f970 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80ce340 tx=0x7ff8a80d4750 comp rx=0 tx=0).stop 2024-04-23T18:37:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.168+0000 7ff8c17df740 1 -- 172.21.15.50:0/2941665732 shutdown_connections 2024-04-23T18:37:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.168+0000 7ff8c17df740 1 --2- 172.21.15.50:0/2941665732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba457800 0x564eba4736d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.168+0000 7ff8c17df740 1 --2- 172.21.15.50:0/2941665732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44cf40 0x564eba456c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:44.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.168+0000 7ff8c17df740 1 --2- 172.21.15.50:0/2941665732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba44c620 0x564eba44f970 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.168+0000 7ff8c17df740 1 -- 172.21.15.50:0/2941665732 >> 172.21.15.50:0/2941665732 conn(0x564eba448b30 msgr2=0x564eba44f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:44.445 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.168+0000 7ff8c17df740 1 -- 172.21.15.50:0/2941665732 shutdown_connections 2024-04-23T18:37:44.453 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.168+0000 7ff8c17df740 1 -- 172.21.15.50:0/2941665732 wait complete. 2024-04-23T18:37:44.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.168+0000 7ff8c17df740 1 -- start start 2024-04-23T18:37:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.168+0000 7ff8c17df740 1 Processor -- start 2024-04-23T18:37:44.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8c17df740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 0x564eba463cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8c17df740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44cf40 0x564eba45f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8c17df740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba457800 0x564eba45f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8c17df740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b81aa0c0 con 0x564eba44c620 2024-04-23T18:37:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8c17df740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b81ab530 con 0x564eba457800 2024-04-23T18:37:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8c17df740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b81aa790 con 0x564eba44cf40 2024-04-23T18:37:44.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8c17df740 10 client.0 _ll_register_callbacks cb 0x7ff8ac009a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:37:44.572 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 0x564eba463cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 0x564eba463cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53076/0 (socket says 172.21.15.50:53076) 2024-04-23T18:37:44.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8b7fff640 1 -- 172.21.15.50:0/1898093011 learned_addr learned my addr 172.21.15.50:0/1898093011 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:44.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8b7fff640 1 -- 172.21.15.50:0/1898093011 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44cf40 msgr2=0x564eba45f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/1898093011 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44cf40 0x564eba45f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8b7fff640 1 -- 172.21.15.50:0/1898093011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba457800 msgr2=0x564eba45f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/1898093011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba457800 0x564eba45f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:44.696 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8b7fff640 1 -- 172.21.15.50:0/1898093011 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eba45faa0 con 0x564eba44c620 2024-04-23T18:37:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8b7fff640 1 --2- 172.21.15.50:0/1898093011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 0x564eba463cd0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80ce3a0 tx=0x7ff8a80ce2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fafbd2416cfab server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8977fe640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:37:44.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.169+0000 7ff8977fe640 1 -- 172.21.15.50:0/1898093011 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80b6040 con 0x564eba44c620 2024-04-23T18:37:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8977fe640 1 -- 172.21.15.50:0/1898093011 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8542+0+0 (secure 0 0 0) 0x7ff8a80bd0b0 con 0x564eba44c620 2024-04-23T18:37:44.744 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8c17df740 1 -- 172.21.15.50:0/1898093011 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8a809d290 con 0x564eba44c620 2024-04-23T18:37:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8977fe640 1 -- 172.21.15.50:0/1898093011 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7ff8a809c6d0 con 0x564eba44c620 2024-04-23T18:37:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8c17df740 10 client.17850 fetch_fsmap learned FSMap version 62 2024-04-23T18:37:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8c17df740 1 -- 172.21.15.50:0/1898093011 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x564eba4d1b70 con 0x564eba44c620 2024-04-23T18:37:44.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8977fe640 1 -- 172.21.15.50:0/1898093011 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8a809d290 con 0x564eba44c620 2024-04-23T18:37:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8977fe640 1 -- 172.21.15.50:0/1898093011 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7ff8a8083080 con 0x564eba44c620 2024-04-23T18:37:44.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8c17df740 10 client.17850 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:37:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8c17df740 10 client.17850 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:37:44.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8c17df740 20 client.17850 populate_metadata read hostname 'smithi050' 2024-04-23T18:37:44.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8c17df740 1 --2- 172.21.15.50:0/1898093011 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564eba46bb30 0x564eba4d4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8c17df740 4 client.17850 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:37:44.953 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8c17df740 1 -- 172.21.15.50:0/1898093011 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564eba454cd0 con 0x564eba46bb30 2024-04-23T18:37:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.170+0000 7ff8b77fe640 1 --2- 172.21.15.50:0/1898093011 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564eba46bb30 0x564eba4d4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.171+0000 7ff8b77fe640 1 --2- 172.21.15.50:0/1898093011 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564eba46bb30 0x564eba4d4220 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b97fc6860191fb9 server_cookie=e2c30c2cdba00519 in_seq=0 out_seq=0 2024-04-23T18:37:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.171+0000 7ff8977fe640 10 client.17850 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:44.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:42 smithi188 ceph-mon[32694]: pgmap v610: 97 pgs: 97 active+clean; 1.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.2 MiB/s wr, 453 op/s 2024-04-23T18:37:44.999 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:44 smithi188 ceph-mon[32694]: pgmap v611: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 11 MiB/s wr, 658 op/s 2024-04-23T18:37:45.008 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:42 smithi177 ceph-mon[32308]: pgmap v610: 97 pgs: 97 active+clean; 1.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.2 MiB/s wr, 453 op/s 2024-04-23T18:37:45.058 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:44 smithi177 ceph-mon[32308]: pgmap v611: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 11 MiB/s wr, 658 op/s 2024-04-23T18:37:45.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.423+0000 7ff8977fe640 1 -- 172.21.15.50:0/1898093011 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564eba454cd0 con 0x564eba46bb30 2024-04-23T18:37:45.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.423+0000 7ff8977fe640 10 client.17850 handle_command_reply: tid=0 2024-04-23T18:37:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.423+0000 7ff8c17df740 1 client.17850 shutdown 2024-04-23T18:37:45.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.424+0000 7ff8c17df740 1 -- 172.21.15.50:0/1898093011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 msgr2=0x564eba463cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:45.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.424+0000 7ff8c17df740 1 --2- 172.21.15.50:0/1898093011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 0x564eba463cd0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80ce3a0 tx=0x7ff8a80ce2b0 comp rx=0 tx=0).stop 2024-04-23T18:37:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.424+0000 7ff8c17df740 1 -- 172.21.15.50:0/1898093011 shutdown_connections 2024-04-23T18:37:45.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.424+0000 7ff8c17df740 1 --2- 172.21.15.50:0/1898093011 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x564eba46bb30 0x564eba4d4220 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.424+0000 7ff8c17df740 1 --2- 172.21.15.50:0/1898093011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564eba457800 0x564eba45f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.424+0000 7ff8c17df740 1 --2- 172.21.15.50:0/1898093011 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564eba44cf40 0x564eba45f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:45.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.424+0000 7ff8c17df740 1 --2- 172.21.15.50:0/1898093011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564eba44c620 0x564eba463cd0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:45.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.424+0000 7ff8c17df740 1 -- 172.21.15.50:0/1898093011 >> 172.21.15.50:0/1898093011 conn(0x564eba448b30 msgr2=0x564eba447cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.424+0000 7ff8977fe640 0 client.17850 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.424+0000 7ff8c17df740 1 -- 172.21.15.50:0/1898093011 shutdown_connections 2024-04-23T18:37:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.424+0000 7ff8c17df740 1 -- 172.21.15.50:0/1898093011 wait complete. 2024-04-23T18:37:45.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.424+0000 7ff8c17df740 20 client.17850 trim_cache size 0 max 16384 2024-04-23T18:37:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:37:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:37:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:37:45.323 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:37:45.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.425+0000 7ff8be9f9640 1 -- 172.21.15.50:0/3909282144 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff8a00a3dc0 msgr2=0x7ff8a00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.425+0000 7ff8be9f9640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff8a00a3dc0 0x7ff8a00a6280 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0006010 tx=0x7ff8b0004a40 comp rx=0 tx=0).stop 2024-04-23T18:37:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.425+0000 7ff8be9f9640 1 -- 172.21.15.50:0/3909282144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8b8103100 msgr2=0x7ff8b8198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:45.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.425+0000 7ff8be9f9640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8b8103100 0x7ff8b8198730 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800e9c0 tx=0x7ff8a800ee80 comp rx=0 tx=0).stop 2024-04-23T18:37:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.425+0000 7ff8be9f9640 1 -- 172.21.15.50:0/3909282144 shutdown_connections 2024-04-23T18:37:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.425+0000 7ff8be9f9640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff8a00a3dc0 0x7ff8a00a6280 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.425+0000 7ff8be9f9640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b810c520 0x7ff8b819cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:45.509 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.425+0000 7ff8be9f9640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8b8103a20 0x7ff8b8198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.425+0000 7ff8be9f9640 1 --2- 172.21.15.50:0/3909282144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8b8103100 0x7ff8b8198730 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.425+0000 7ff8be9f9640 1 -- 172.21.15.50:0/3909282144 >> 172.21.15.50:0/3909282144 conn(0x7ff8b80fc920 msgr2=0x7ff8b80fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.425+0000 7ff8be9f9640 1 -- 172.21.15.50:0/3909282144 shutdown_connections 2024-04-23T18:37:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:42.426+0000 7ff8be9f9640 1 -- 172.21.15.50:0/3909282144 wait complete. 2024-04-23T18:37:45.543 INFO:tasks.workunit.client.0.smithi050.stdout: 10 151015 20.77 MB/sec execute 256 sec latency 220.489 ms 2024-04-23T18:37:45.550 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:42 smithi050 ceph-mon[29187]: pgmap v610: 97 pgs: 97 active+clean; 1.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.2 MiB/s wr, 453 op/s 2024-04-23T18:37:45.609 INFO:tasks.workunit.client.0.smithi050.stdout: 10 151362 20.76 MB/sec execute 257 sec latency 297.533 ms 2024-04-23T18:37:45.617 INFO:tasks.workunit.client.0.smithi050.stdout: 10 151777 20.77 MB/sec execute 258 sec latency 544.710 ms 2024-04-23T18:37:45.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:44 smithi050 ceph-mon[29187]: pgmap v611: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 11 MiB/s wr, 658 op/s 2024-04-23T18:37:45.634 INFO:tasks.workunit.client.0.smithi050.stdout: 10 152379 20.82 MB/sec execute 259 sec latency 72.524 ms 2024-04-23T18:37:46.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 152945 20.87 MB/sec execute 260 sec latency 134.832 ms 2024-04-23T18:37:46.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:46 smithi050 ceph-mon[29187]: pgmap v612: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 409 op/s 2024-04-23T18:37:46.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:46 smithi177 ceph-mon[32308]: pgmap v612: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 409 op/s 2024-04-23T18:37:46.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:46 smithi188 ceph-mon[32694]: pgmap v612: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 409 op/s 2024-04-23T18:37:47.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 153431 20.87 MB/sec execute 261 sec latency 132.094 ms 2024-04-23T18:37:48.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 153890 20.90 MB/sec execute 262 sec latency 269.851 ms 2024-04-23T18:37:48.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:48 smithi188 ceph-mon[32694]: pgmap v613: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 623 op/s 2024-04-23T18:37:48.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:48 smithi050 ceph-mon[29187]: pgmap v613: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 623 op/s 2024-04-23T18:37:48.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:48 smithi177 ceph-mon[32308]: pgmap v613: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 623 op/s 2024-04-23T18:37:49.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 154280 20.90 MB/sec execute 263 sec latency 201.572 ms 2024-04-23T18:37:50.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 154813 20.91 MB/sec execute 264 sec latency 99.255 ms 2024-04-23T18:37:50.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:50 smithi050 ceph-mon[29187]: pgmap v614: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 601 op/s 2024-04-23T18:37:50.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:50 smithi177 ceph-mon[32308]: pgmap v614: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 601 op/s 2024-04-23T18:37:50.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:50 smithi188 ceph-mon[32694]: pgmap v614: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 601 op/s 2024-04-23T18:37:51.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 155455 20.98 MB/sec execute 265 sec latency 80.429 ms 2024-04-23T18:37:52.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 156064 21.03 MB/sec execute 266 sec latency 94.919 ms 2024-04-23T18:37:52.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:52 smithi050 ceph-mon[29187]: pgmap v615: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.6 MiB/s wr, 521 op/s 2024-04-23T18:37:52.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:52 smithi177 ceph-mon[32308]: pgmap v615: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.6 MiB/s wr, 521 op/s 2024-04-23T18:37:52.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:52 smithi188 ceph-mon[32694]: pgmap v615: 97 pgs: 97 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.6 MiB/s wr, 521 op/s 2024-04-23T18:37:53.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 156417 21.01 MB/sec execute 267 sec latency 244.934 ms 2024-04-23T18:37:54.577 INFO:tasks.workunit.client.0.smithi050.stdout: 10 156902 21.03 MB/sec execute 268 sec latency 200.076 ms 2024-04-23T18:37:54.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:54 smithi050 ceph-mon[29187]: pgmap v616: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 747 op/s 2024-04-23T18:37:54.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:54 smithi177 ceph-mon[32308]: pgmap v616: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 747 op/s 2024-04-23T18:37:54.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:54 smithi188 ceph-mon[32694]: pgmap v616: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 747 op/s 2024-04-23T18:37:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.531+0000 7f2450e72640 1 -- 172.21.15.50:0/2951149776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244c101330 msgr2=0x7f244c101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:55.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.531+0000 7f2450e72640 1 --2- 172.21.15.50:0/2951149776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244c101330 0x7f244c101710 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f243c009900 tx=0x7f243c02a410 comp rx=0 tx=0).stop 2024-04-23T18:37:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.531+0000 7f2450e72640 1 -- 172.21.15.50:0/2951149776 shutdown_connections 2024-04-23T18:37:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.531+0000 7f2450e72640 1 --2- 172.21.15.50:0/2951149776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c1061d0 0x7f244c10dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.531+0000 7f2450e72640 1 --2- 172.21.15.50:0/2951149776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f244c101ce0 0x7f244c105af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.531+0000 7f2450e72640 1 --2- 172.21.15.50:0/2951149776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244c101330 0x7f244c101710 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.531+0000 7f2450e72640 1 -- 172.21.15.50:0/2951149776 >> 172.21.15.50:0/2951149776 conn(0x7f244c077d30 msgr2=0x7f244c078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:55.608 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.531+0000 7f2450e72640 1 -- 172.21.15.50:0/2951149776 shutdown_connections 2024-04-23T18:37:55.620 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.531+0000 7f2450e72640 1 -- 172.21.15.50:0/2951149776 wait complete. 2024-04-23T18:37:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.532+0000 7f2450e72640 1 Processor -- start 2024-04-23T18:37:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.532+0000 7f2450e72640 1 -- start start 2024-04-23T18:37:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.532+0000 7f2450e72640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244c101330 0x7f244c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:55.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.532+0000 7f2450e72640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f244c1061d0 0x7f244c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:55.856 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.532+0000 7f2450e72640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c19cfd0 0x7f244c19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:55.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.532+0000 7f2450e72640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f244c111160 con 0x7f244c101330 2024-04-23T18:37:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.532+0000 7f2450e72640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f244c110fe0 con 0x7f244c19cfd0 2024-04-23T18:37:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.532+0000 7f2450e72640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1112e0 con 0x7f244c1061d0 2024-04-23T18:37:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.532+0000 7f244a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244c101330 0x7f244c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:56.029 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.532+0000 7f244a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244c101330 0x7f244c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54402/0 (socket says 172.21.15.50:54402) 2024-04-23T18:37:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.532+0000 7f2449d9b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f244c1061d0 0x7f244c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.532+0000 7f244a59c640 1 -- 172.21.15.50:0/2305246762 learned_addr learned my addr 172.21.15.50:0/2305246762 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:56.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f244ad9d640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c19cfd0 0x7f244c19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f244a59c640 1 -- 172.21.15.50:0/2305246762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f244c1061d0 msgr2=0x7f244c198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f244a59c640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f244c1061d0 0x7f244c198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:56.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f244a59c640 1 -- 172.21.15.50:0/2305246762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c19cfd0 msgr2=0x7f244c19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f244a59c640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c19cfd0 0x7f244c19d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:56.171 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f244a59c640 1 -- 172.21.15.50:0/2305246762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244c19dac0 con 0x7f244c101330 2024-04-23T18:37:56.237 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f244ad9d640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c19cfd0 0x7f244c19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:37:56.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f2449d9b640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f244c1061d0 0x7f244c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:37:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f244a59c640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244c101330 0x7f244c198710 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f243c005ab0 tx=0x7f243c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f243b7fe640 1 -- 172.21.15.50:0/2305246762 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c03f070 con 0x7f244c101330 2024-04-23T18:37:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f243b7fe640 1 -- 172.21.15.50:0/2305246762 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f243c031b50 con 0x7f244c101330 2024-04-23T18:37:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f243b7fe640 1 -- 172.21.15.50:0/2305246762 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c035030 con 0x7f244c101330 2024-04-23T18:37:56.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f2450e72640 1 -- 172.21.15.50:0/2305246762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244c19dd50 con 0x7f244c101330 2024-04-23T18:37:56.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.533+0000 7f2450e72640 1 -- 172.21.15.50:0/2305246762 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244c1a5570 con 0x7f244c101330 2024-04-23T18:37:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.535+0000 7f243b7fe640 1 -- 172.21.15.50:0/2305246762 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f243c042070 con 0x7f244c101330 2024-04-23T18:37:56.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.536+0000 7f2453b85740 1 Processor -- start 2024-04-23T18:37:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.536+0000 7f2453b85740 1 -- start start 2024-04-23T18:37:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.536+0000 7f2453b85740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd7ed920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.536+0000 7f243b7fe640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f24340a3e40 0x7f24340a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.536+0000 7f2453b85740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 0x55f7bd7fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.536+0000 7f2453b85740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7ff000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.536+0000 7f2453b85740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f7bd752970 con 0x55f7bd7fcc10 2024-04-23T18:37:56.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.536+0000 7f2453b85740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f7bd6ef4a0 con 0x55f7bd7ede60 2024-04-23T18:37:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.536+0000 7f2453b85740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f7bd74e930 con 0x55f7bd7ed540 2024-04-23T18:37:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f244a59c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd7ed920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:56.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f2449d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 0x55f7bd7fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f2449d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 0x55f7bd7fc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:40380/0 (socket says 172.21.15.50:40380) 2024-04-23T18:37:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f244ad9d640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f24340a3e40 0x7f24340a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:56.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f2449d9b640 1 -- 172.21.15.50:0/1065118599 learned_addr learned my addr 172.21.15.50:0/1065118599 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f244ad9d640 1 --2- 172.21.15.50:0/1065118599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7ff000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:56.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f243b7fe640 1 -- 172.21.15.50:0/2305246762 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8542+0+0 (secure 0 0 0) 0x7f243c0f0d60 con 0x7f244c101330 2024-04-23T18:37:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f2449d9b640 1 -- 172.21.15.50:0/1065118599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 msgr2=0x55f7bd7ed920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:56.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f2449d9b640 1 --2- 172.21.15.50:0/1065118599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd7ed920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f2449d9b640 1 -- 172.21.15.50:0/1065118599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7fcc10 msgr2=0x55f7bd7ff000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:56.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f2449d9b640 1 --2- 172.21.15.50:0/1065118599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7ff000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f2449d9b640 1 -- 172.21.15.50:0/1065118599 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7bd465210 con 0x55f7bd7ede60 2024-04-23T18:37:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f244ad9d640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f24340a3e40 0x7f24340a6300 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2444005fd0 tx=0x7f24440116d0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f2449d9b640 1 --2- 172.21.15.50:0/1065118599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 0x55f7bd7fc6d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2440011470 tx=0x7f2440011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9abd56049dc1d6b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:56.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f2438ff9640 1 -- 172.21.15.50:0/1065118599 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244001e070 con 0x55f7bd7ede60 2024-04-23T18:37:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f2438ff9640 1 -- 172.21.15.50:0/1065118599 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24400035f0 con 0x55f7bd7ede60 2024-04-23T18:37:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.537+0000 7f2438ff9640 1 -- 172.21.15.50:0/1065118599 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24400044e0 con 0x55f7bd7ede60 2024-04-23T18:37:56.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.538+0000 7f2453b85740 1 -- 172.21.15.50:0/1065118599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 msgr2=0x55f7bd7fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.538+0000 7f2453b85740 1 --2- 172.21.15.50:0/1065118599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 0x55f7bd7fc6d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2440011470 tx=0x7f2440011930 comp rx=0 tx=0).stop 2024-04-23T18:37:57.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.538+0000 7f2453b85740 1 -- 172.21.15.50:0/1065118599 shutdown_connections 2024-04-23T18:37:57.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.538+0000 7f2453b85740 1 --2- 172.21.15.50:0/1065118599 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7ff000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.538+0000 7f2453b85740 1 --2- 172.21.15.50:0/1065118599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 0x55f7bd7fc6d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.538+0000 7f2453b85740 1 --2- 172.21.15.50:0/1065118599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd7ed920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.538+0000 7f2453b85740 1 -- 172.21.15.50:0/1065118599 >> 172.21.15.50:0/1065118599 conn(0x55f7bd7e9b30 msgr2=0x55f7bd7e9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.538+0000 7f2453b85740 1 -- 172.21.15.50:0/1065118599 shutdown_connections 2024-04-23T18:37:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.538+0000 7f2453b85740 1 -- 172.21.15.50:0/1065118599 wait complete. 2024-04-23T18:37:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f2453b85740 1 -- start start 2024-04-23T18:37:57.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f2453b85740 1 Processor -- start 2024-04-23T18:37:57.261 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f2453b85740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd7fc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:57.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f2453b85740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 0x55f7bd7f9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f2453b85740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7f9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f2453b85740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55f7bd7ff840 con 0x55f7bd7fcc10 2024-04-23T18:37:57.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f2453b85740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f7bd6ed0b0 con 0x55f7bd7ede60 2024-04-23T18:37:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f2453b85740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f7bd7ff9c0 con 0x55f7bd7ed540 2024-04-23T18:37:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f2453b85740 10 client.0 _ll_register_callbacks cb 0x55f7bd6dad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:37:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f2449d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 0x55f7bd7f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:57.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f244ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7f9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f244a59c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd7fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f244ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7f9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54416/0 (socket says 172.21.15.50:54416) 2024-04-23T18:37:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f244a59c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd7fc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:59588/0 (socket says 172.21.15.50:59588) 2024-04-23T18:37:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f2449d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 0x55f7bd7f9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:40388/0 (socket says 172.21.15.50:40388) 2024-04-23T18:37:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.539+0000 7f244ad9d640 1 -- 172.21.15.50:0/3424899027 learned_addr learned my addr 172.21.15.50:0/3424899027 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f244ad9d640 1 -- 172.21.15.50:0/3424899027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 msgr2=0x55f7bd7fc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:57.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f244ad9d640 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd7fc530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f244ad9d640 1 -- 172.21.15.50:0/3424899027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 msgr2=0x55f7bd7f9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:57.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f244ad9d640 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 0x55f7bd7f9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f244ad9d640 1 -- 172.21.15.50:0/3424899027 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7bd531620 con 0x55f7bd7fcc10 2024-04-23T18:37:57.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f244a59c640 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd7fc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:37:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f2449d9b640 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 0x55f7bd7f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:37:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f244ad9d640 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7f9d60 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2444003ec0 tx=0x7f244401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0677ae1f1cded77 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f2431ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:37:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f2431ffb640 1 -- 172.21.15.50:0/3424899027 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444024070 con 0x55f7bd7fcc10 2024-04-23T18:37:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f2431ffb640 1 -- 172.21.15.50:0/3424899027 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8542+0+0 (secure 0 0 0) 0x7f244401f660 con 0x55f7bd7fcc10 2024-04-23T18:37:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f2453b85740 1 -- 172.21.15.50:0/3424899027 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24400035f0 con 0x55f7bd7fcc10 2024-04-23T18:37:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f2431ffb640 1 -- 172.21.15.50:0/3424899027 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f244401f860 con 0x55f7bd7fcc10 2024-04-23T18:37:57.994 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f2453b85740 10 client.17862 fetch_fsmap learned FSMap version 62 2024-04-23T18:37:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f2453b85740 1 -- 172.21.15.50:0/3424899027 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55f7bd80b790 con 0x55f7bd7fcc10 2024-04-23T18:37:58.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.540+0000 7f2431ffb640 1 -- 172.21.15.50:0/3424899027 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24400035f0 con 0x55f7bd7fcc10 2024-04-23T18:37:58.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.541+0000 7f2431ffb640 1 -- 172.21.15.50:0/3424899027 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f2444028930 con 0x55f7bd7fcc10 2024-04-23T18:37:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.541+0000 7f2453b85740 10 client.17862 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:37:58.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.541+0000 7f2453b85740 10 client.17862 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:37:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.541+0000 7f2453b85740 20 client.17862 populate_metadata read hostname 'smithi050' 2024-04-23T18:37:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.541+0000 7f2453b85740 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f7bd80cca0 0x55f7bd80f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.541+0000 7f2453b85740 4 client.17862 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:37:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.541+0000 7f2453b85740 1 -- 172.21.15.50:0/3424899027 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7bd760390 con 0x55f7bd80cca0 2024-04-23T18:37:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.541+0000 7f244a59c640 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f7bd80cca0 0x55f7bd80f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.541+0000 7f2431ffb640 10 client.17862 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.541+0000 7f244a59c640 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f7bd80cca0 0x55f7bd80f0b0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4df4065fe7ce4e4f server_cookie=a69c9c95cd687bce in_seq=0 out_seq=0 2024-04-23T18:37:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2431ffb640 1 -- 172.21.15.50:0/3424899027 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7bd760390 con 0x55f7bd80cca0 2024-04-23T18:37:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2431ffb640 10 client.17862 handle_command_reply: tid=0 2024-04-23T18:37:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2453b85740 1 client.17862 shutdown 2024-04-23T18:37:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2453b85740 1 -- 172.21.15.50:0/3424899027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7fcc10 msgr2=0x55f7bd7f9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2453b85740 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7f9d60 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2444003ec0 tx=0x7f244401d000 comp rx=0 tx=0).stop 2024-04-23T18:37:58.400 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2453b85740 1 -- 172.21.15.50:0/3424899027 shutdown_connections 2024-04-23T18:37:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2453b85740 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f7bd80cca0 0x55f7bd80f0b0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2453b85740 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7f9d60 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2453b85740 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ede60 0x55f7bd7f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2453b85740 1 --2- 172.21.15.50:0/3424899027 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd7fc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2431ffb640 0 client.17862 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:37:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2453b85740 1 -- 172.21.15.50:0/3424899027 >> 172.21.15.50:0/3424899027 conn(0x55f7bd7e9b30 msgr2=0x55f7bd7ef9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2453b85740 1 -- 172.21.15.50:0/3424899027 shutdown_connections 2024-04-23T18:37:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2453b85740 1 -- 172.21.15.50:0/3424899027 wait complete. 2024-04-23T18:37:58.554 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.544+0000 7f2453b85740 20 client.17862 trim_cache size 0 max 16384 2024-04-23T18:37:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.562+0000 7f2453b85740 1 Processor -- start 2024-04-23T18:37:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.562+0000 7f2453b85740 1 -- start start 2024-04-23T18:37:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2453b85740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd808730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2453b85740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 0x55f7bd7f4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2453b85740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7fc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2453b85740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1ae1f0 con 0x55f7bd7ede60 2024-04-23T18:37:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2453b85740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1abac0 con 0x55f7bd7fcc10 2024-04-23T18:37:58.857 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2453b85740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1a9930 con 0x55f7bd7ed540 2024-04-23T18:37:58.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2449d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 0x55f7bd7f4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:58.901 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2449d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 0x55f7bd7f4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54426/0 (socket says 172.21.15.50:54426) 2024-04-23T18:37:58.911 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2449d9b640 1 -- 172.21.15.50:0/3681918576 learned_addr learned my addr 172.21.15.50:0/3681918576 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:58.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f244ad9d640 1 --2- 172.21.15.50:0/3681918576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7fc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f244a59c640 1 --2- 172.21.15.50:0/3681918576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd808730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:58.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2449d9b640 1 -- 172.21.15.50:0/3681918576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 msgr2=0x55f7bd808730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2449d9b640 1 --2- 172.21.15.50:0/3681918576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd808730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2449d9b640 1 -- 172.21.15.50:0/3681918576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7fcc10 msgr2=0x55f7bd7fc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2449d9b640 1 --2- 172.21.15.50:0/3681918576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7fc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2449d9b640 1 -- 172.21.15.50:0/3681918576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7bd7f7d90 con 0x55f7bd7ede60 2024-04-23T18:37:59.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2449d9b640 1 --2- 172.21.15.50:0/3681918576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 0x55f7bd7f4d80 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2440011470 tx=0x7f24400096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64374dab44fa70c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:59.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.563+0000 7f2433fff640 1 -- 172.21.15.50:0/3681918576 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2440028020 con 0x55f7bd7ede60 2024-04-23T18:37:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.564+0000 7f2433fff640 1 -- 172.21.15.50:0/3681918576 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2440005570 con 0x55f7bd7ede60 2024-04-23T18:37:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.564+0000 7f2433fff640 1 -- 172.21.15.50:0/3681918576 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244001e040 con 0x55f7bd7ede60 2024-04-23T18:37:59.188 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.564+0000 7f2453b85740 1 -- 172.21.15.50:0/3681918576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 msgr2=0x55f7bd7f4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:37:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.564+0000 7f2453b85740 1 --2- 172.21.15.50:0/3681918576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 0x55f7bd7f4d80 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2440011470 tx=0x7f24400096c0 comp rx=0 tx=0).stop 2024-04-23T18:37:59.260 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.564+0000 7f2453b85740 1 -- 172.21.15.50:0/3681918576 shutdown_connections 2024-04-23T18:37:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.564+0000 7f2453b85740 1 --2- 172.21.15.50:0/3681918576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7fc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:59.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.564+0000 7f2453b85740 1 --2- 172.21.15.50:0/3681918576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 0x55f7bd7f4d80 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.564+0000 7f2453b85740 1 --2- 172.21.15.50:0/3681918576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7ed540 0x55f7bd808730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:59.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.564+0000 7f2453b85740 1 -- 172.21.15.50:0/3681918576 >> 172.21.15.50:0/3681918576 conn(0x55f7bd7e9b30 msgr2=0x55f7bd7f4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:37:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.564+0000 7f2453b85740 1 -- 172.21.15.50:0/3681918576 shutdown_connections 2024-04-23T18:37:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.564+0000 7f2453b85740 1 -- 172.21.15.50:0/3681918576 wait complete. 2024-04-23T18:37:59.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2453b85740 1 -- start start 2024-04-23T18:37:59.398 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2453b85740 1 Processor -- start 2024-04-23T18:37:59.459 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2453b85740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ed540 0x55f7bd804c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2453b85740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 0x55f7bd805180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2453b85740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7f5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:37:59.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2453b85740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1a9f80 con 0x55f7bd7ede60 2024-04-23T18:37:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2453b85740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1ab3f0 con 0x55f7bd7ed540 2024-04-23T18:37:59.608 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2453b85740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1aa650 con 0x55f7bd7fcc10 2024-04-23T18:37:59.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2453b85740 10 client.0 _ll_register_callbacks cb 0x7f2444003c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:37:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2449d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 0x55f7bd805180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:59.672 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2449d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 0x55f7bd805180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54432/0 (socket says 172.21.15.50:54432) 2024-04-23T18:37:59.683 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f244a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ed540 0x55f7bd804c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2449d9b640 1 -- 172.21.15.50:0/1246137744 learned_addr learned my addr 172.21.15.50:0/1246137744 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:37:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f244ad9d640 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7f5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:37:59.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2449d9b640 1 -- 172.21.15.50:0/1246137744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7fcc10 msgr2=0x55f7bd7f5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2449d9b640 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7f5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2449d9b640 1 -- 172.21.15.50:0/1246137744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ed540 msgr2=0x55f7bd804c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:37:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.565+0000 7f2449d9b640 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ed540 0x55f7bd804c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:37:59.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.566+0000 7f2449d9b640 1 -- 172.21.15.50:0/1246137744 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7bd7f6490 con 0x55f7bd7ede60 2024-04-23T18:37:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.566+0000 7f244a59c640 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ed540 0x55f7bd804c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:37:59.838 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.566+0000 7f2449d9b640 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 0x55f7bd805180 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f2440011440 tx=0x7f2440009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150a55aeded5a8d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:37:59.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.566+0000 7f2431ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:37:59.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.566+0000 7f2431ffb640 1 -- 172.21.15.50:0/1246137744 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2440028070 con 0x55f7bd7ede60 2024-04-23T18:37:59.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.566+0000 7f2431ffb640 1 -- 172.21.15.50:0/1246137744 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8542+0+0 (secure 0 0 0) 0x7f24400035f0 con 0x55f7bd7ede60 2024-04-23T18:37:59.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.566+0000 7f2453b85740 1 -- 172.21.15.50:0/1246137744 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2440005570 con 0x55f7bd7ede60 2024-04-23T18:38:00.002 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.566+0000 7f2431ffb640 1 -- 172.21.15.50:0/1246137744 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f244003d050 con 0x55f7bd7ede60 2024-04-23T18:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.566+0000 7f2453b85740 10 client.17868 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.566+0000 7f2453b85740 1 -- 172.21.15.50:0/1246137744 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55f7bd872d40 con 0x55f7bd7ede60 2024-04-23T18:38:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.566+0000 7f2431ffb640 1 -- 172.21.15.50:0/1246137744 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2440005570 con 0x55f7bd7ede60 2024-04-23T18:38:00.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.566+0000 7f2431ffb640 1 -- 172.21.15.50:0/1246137744 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f2440024480 con 0x55f7bd7ede60 2024-04-23T18:38:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.567+0000 7f2453b85740 10 client.17868 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.567+0000 7f2453b85740 10 client.17868 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.567+0000 7f2453b85740 20 client.17868 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.567+0000 7f2453b85740 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f7bd8731b0 0x55f7bd8755a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.567+0000 7f2453b85740 4 client.17868 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:38:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.567+0000 7f2453b85740 1 -- 172.21.15.50:0/1246137744 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f7bd7fa0f0 con 0x55f7bd8731b0 2024-04-23T18:38:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.567+0000 7f244a59c640 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f7bd8731b0 0x55f7bd8755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.567+0000 7f244a59c640 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f7bd8731b0 0x55f7bd8755a0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2403b7fbd0398970 server_cookie=f396ea6f239a5d15 in_seq=0 out_seq=0 2024-04-23T18:38:00.326 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.567+0000 7f2431ffb640 10 client.17868 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:00.336 INFO:tasks.workunit.client.0.smithi050.stdout: 10 157446 21.06 MB/sec execute 269 sec latency 97.372 ms 2024-04-23T18:38:00.380 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2431ffb640 1 -- 172.21.15.50:0/1246137744 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f7bd7fa0f0 con 0x55f7bd8731b0 2024-04-23T18:38:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2431ffb640 10 client.17868 handle_command_reply: tid=0 2024-04-23T18:38:00.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2453b85740 1 client.17868 shutdown 2024-04-23T18:38:00.588 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2453b85740 1 -- 172.21.15.50:0/1246137744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 msgr2=0x55f7bd805180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2453b85740 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 0x55f7bd805180 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f2440011440 tx=0x7f2440009bd0 comp rx=0 tx=0).stop 2024-04-23T18:38:00.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2453b85740 1 -- 172.21.15.50:0/1246137744 shutdown_connections 2024-04-23T18:38:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2453b85740 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55f7bd8731b0 0x55f7bd8755a0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2431ffb640 0 client.17868 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2453b85740 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f7bd7fcc10 0x55f7bd7f5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2453b85740 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55f7bd7ede60 0x55f7bd805180 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2453b85740 1 --2- 172.21.15.50:0/1246137744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f7bd7ed540 0x55f7bd804c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2453b85740 1 -- 172.21.15.50:0/1246137744 >> 172.21.15.50:0/1246137744 conn(0x55f7bd7e9b30 msgr2=0x55f7bd7f4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2453b85740 1 -- 172.21.15.50:0/1246137744 shutdown_connections 2024-04-23T18:38:00.855 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.830+0000 7f2453b85740 1 -- 172.21.15.50:0/1246137744 wait complete. 2024-04-23T18:38:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.831+0000 7f2453b85740 20 client.17868 trim_cache size 0 max 16384 2024-04-23T18:38:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:38:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:38:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:38:00.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:38:00.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.831+0000 7f2450e72640 1 -- 172.21.15.50:0/2305246762 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f24340a3e40 msgr2=0x7f24340a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.831+0000 7f2450e72640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f24340a3e40 0x7f24340a6300 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2444005fd0 tx=0x7f24440116d0 comp rx=0 tx=0).stop 2024-04-23T18:38:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.831+0000 7f2450e72640 1 -- 172.21.15.50:0/2305246762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244c101330 msgr2=0x7f244c198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:01.072 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.831+0000 7f2450e72640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244c101330 0x7f244c198710 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f243c005ab0 tx=0x7f243c003b90 comp rx=0 tx=0).stop 2024-04-23T18:38:01.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.831+0000 7f2450e72640 1 -- 172.21.15.50:0/2305246762 shutdown_connections 2024-04-23T18:38:01.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.831+0000 7f2450e72640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f24340a3e40 0x7f24340a6300 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.831+0000 7f2450e72640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c19cfd0 0x7f244c19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:01.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.831+0000 7f2450e72640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f244c1061d0 0x7f244c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:01.167 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.831+0000 7f2450e72640 1 --2- 172.21.15.50:0/2305246762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244c101330 0x7f244c198710 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:01.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.831+0000 7f2450e72640 1 -- 172.21.15.50:0/2305246762 >> 172.21.15.50:0/2305246762 conn(0x7f244c077d30 msgr2=0x7f244c100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:01.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.831+0000 7f2450e72640 1 -- 172.21.15.50:0/2305246762 shutdown_connections 2024-04-23T18:38:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:37:55.832+0000 7f2450e72640 1 -- 172.21.15.50:0/2305246762 wait complete. 2024-04-23T18:38:01.253 INFO:tasks.workunit.client.0.smithi050.stdout: 10 157902 21.07 MB/sec execute 270 sec latency 254.442 ms 2024-04-23T18:38:01.282 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:56 smithi050 ceph-mon[29187]: pgmap v617: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.8 MiB/s wr, 542 op/s 2024-04-23T18:38:01.295 INFO:tasks.workunit.client.0.smithi050.stdout: 10 158444 21.11 MB/sec execute 271 sec latency 97.347 ms 2024-04-23T18:38:01.332 INFO:tasks.workunit.client.0.smithi050.stdout: 10 158764 21.11 MB/sec execute 272 sec latency 334.022 ms 2024-04-23T18:38:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:56 smithi188 ceph-mon[32694]: pgmap v617: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.8 MiB/s wr, 542 op/s 2024-04-23T18:38:01.381 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:37:58 smithi188 ceph-mon[32694]: pgmap v618: 97 pgs: 97 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 698 op/s 2024-04-23T18:38:01.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:00 smithi188 ceph-mon[32694]: pgmap v619: 97 pgs: 97 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 622 op/s 2024-04-23T18:38:01.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:00 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:38:01.448 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:56 smithi177 ceph-mon[32308]: pgmap v617: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.8 MiB/s wr, 542 op/s 2024-04-23T18:38:01.495 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:37:58 smithi177 ceph-mon[32308]: pgmap v618: 97 pgs: 97 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 698 op/s 2024-04-23T18:38:01.505 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:00 smithi177 ceph-mon[32308]: pgmap v619: 97 pgs: 97 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 622 op/s 2024-04-23T18:38:01.515 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:00 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:38:01.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:37:58 smithi050 ceph-mon[29187]: pgmap v618: 97 pgs: 97 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 698 op/s 2024-04-23T18:38:01.535 INFO:tasks.workunit.client.0.smithi050.stdout: 10 159200 21.12 MB/sec execute 273 sec latency 129.790 ms 2024-04-23T18:38:01.546 INFO:tasks.workunit.client.0.smithi050.stdout: 10 159550 21.09 MB/sec execute 274 sec latency 233.991 ms 2024-04-23T18:38:01.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:00 smithi050 ceph-mon[29187]: pgmap v619: 97 pgs: 97 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 622 op/s 2024-04-23T18:38:01.614 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:00 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:38:01.622 INFO:tasks.workunit.client.0.smithi050.stdout: 10 159918 21.05 MB/sec execute 275 sec latency 109.699 ms 2024-04-23T18:38:01.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:01 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:38:01.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:01 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:38:01.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:01 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:38:01.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:01 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:38:01.804 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:01 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:38:01.853 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:01 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:38:01.870 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:01 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:38:01.886 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:01 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:38:01.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:01 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:38:02.578 INFO:tasks.workunit.client.0.smithi050.stdout: 10 160241 21.06 MB/sec execute 276 sec latency 337.426 ms 2024-04-23T18:38:02.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:02 smithi050 ceph-mon[29187]: pgmap v620: 97 pgs: 97 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.8 MiB/s wr, 520 op/s 2024-04-23T18:38:02.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:02 smithi177 ceph-mon[32308]: pgmap v620: 97 pgs: 97 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.8 MiB/s wr, 520 op/s 2024-04-23T18:38:02.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:02 smithi188 ceph-mon[32694]: pgmap v620: 97 pgs: 97 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.8 MiB/s wr, 520 op/s 2024-04-23T18:38:03.578 INFO:tasks.workunit.client.0.smithi050.stdout: 10 160581 21.06 MB/sec execute 277 sec latency 208.548 ms 2024-04-23T18:38:04.578 INFO:tasks.workunit.client.0.smithi050.stdout: 10 160990 21.07 MB/sec execute 278 sec latency 293.383 ms 2024-04-23T18:38:04.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:04 smithi050 ceph-mon[29187]: pgmap v621: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 14 MiB/s wr, 842 op/s 2024-04-23T18:38:04.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:04 smithi177 ceph-mon[32308]: pgmap v621: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 14 MiB/s wr, 842 op/s 2024-04-23T18:38:04.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:04 smithi188 ceph-mon[32694]: pgmap v621: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 14 MiB/s wr, 842 op/s 2024-04-23T18:38:05.578 INFO:tasks.workunit.client.0.smithi050.stdout: 10 161270 21.05 MB/sec execute 279 sec latency 163.133 ms 2024-04-23T18:38:05.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:05 smithi050 ceph-mon[29187]: osdmap e123: 12 total, 12 up, 12 in 2024-04-23T18:38:05.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:05 smithi050 ceph-mon[29187]: pgmap v623: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 739 op/s 2024-04-23T18:38:05.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:05 smithi177 ceph-mon[32308]: osdmap e123: 12 total, 12 up, 12 in 2024-04-23T18:38:05.793 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:05 smithi177 ceph-mon[32308]: pgmap v623: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 739 op/s 2024-04-23T18:38:05.829 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:05 smithi188 ceph-mon[32694]: osdmap e123: 12 total, 12 up, 12 in 2024-04-23T18:38:05.895 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:05 smithi188 ceph-mon[32694]: pgmap v623: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 739 op/s 2024-04-23T18:38:06.578 INFO:tasks.workunit.client.0.smithi050.stdout: 10 161613 21.07 MB/sec execute 280 sec latency 137.030 ms 2024-04-23T18:38:07.578 INFO:tasks.workunit.client.0.smithi050.stdout: 10 161997 21.05 MB/sec execute 281 sec latency 175.618 ms 2024-04-23T18:38:08.193 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:07 smithi050 ceph-mon[29187]: pgmap v624: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 737 op/s 2024-04-23T18:38:08.286 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:07 smithi177 ceph-mon[32308]: pgmap v624: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 737 op/s 2024-04-23T18:38:08.328 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:07 smithi188 ceph-mon[32694]: pgmap v624: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 737 op/s 2024-04-23T18:38:08.579 INFO:tasks.workunit.client.0.smithi050.stdout: 10 162444 21.04 MB/sec execute 282 sec latency 119.932 ms 2024-04-23T18:38:08.959 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.967+0000 7ff5277fe640 1 -- 172.21.15.50:0/2190641832 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff520031c50 con 0x7ff528100e10 2024-04-23T18:38:08.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.967+0000 7ff52ecef640 1 -- 172.21.15.50:0/2190641832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff528100e10 msgr2=0x7ff52810ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.967+0000 7ff52ecef640 1 --2- 172.21.15.50:0/2190641832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff528100e10 0x7ff52810ede0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7ff520009cd0 tx=0x7ff52002a8e0 comp rx=0 tx=0).stop 2024-04-23T18:38:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.967+0000 7ff52ecef640 1 -- 172.21.15.50:0/2190641832 shutdown_connections 2024-04-23T18:38:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.967+0000 7ff52ecef640 1 --2- 172.21.15.50:0/2190641832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff528100e10 0x7ff52810ede0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:09.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.967+0000 7ff52ecef640 1 --2- 172.21.15.50:0/2190641832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff528100450 0x7ff5281008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.967+0000 7ff52ecef640 1 --2- 172.21.15.50:0/2190641832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff528106f30 0x7ff528107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.967+0000 7ff52ecef640 1 -- 172.21.15.50:0/2190641832 >> 172.21.15.50:0/2190641832 conn(0x7ff5280773e0 msgr2=0x7ff5280fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.967+0000 7ff52ecef640 1 -- 172.21.15.50:0/2190641832 shutdown_connections 2024-04-23T18:38:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.967+0000 7ff52ecef640 1 -- 172.21.15.50:0/2190641832 wait complete. 2024-04-23T18:38:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.968+0000 7ff52ecef640 1 Processor -- start 2024-04-23T18:38:09.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.968+0000 7ff52ecef640 1 -- start start 2024-04-23T18:38:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ecef640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff528100450 0x7ff5281a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ecef640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff528100e10 0x7ff5281a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ecef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff528106f30 0x7ff52819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ecef640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff528103490 con 0x7ff528100450 2024-04-23T18:38:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ecef640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff528103310 con 0x7ff528106f30 2024-04-23T18:38:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ecef640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff5281147a0 con 0x7ff528100e10 2024-04-23T18:38:09.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ca8b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff528100450 0x7ff5281a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:09.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ca8b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff528100450 0x7ff5281a0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40926/0 (socket says 172.21.15.50:40926) 2024-04-23T18:38:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ca8b640 1 -- 172.21.15.50:0/3707039172 learned_addr learned my addr 172.21.15.50:0/3707039172 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:09.822 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52d28c640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff528106f30 0x7ff52819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ca8b640 1 -- 172.21.15.50:0/3707039172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff528100e10 msgr2=0x7ff5281a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ca8b640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff528100e10 0x7ff5281a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ca8b640 1 -- 172.21.15.50:0/3707039172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff528106f30 msgr2=0x7ff52819a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ca8b640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff528106f30 0x7ff52819a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ca8b640 1 -- 172.21.15.50:0/3707039172 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52819af10 con 0x7ff528100450 2024-04-23T18:38:09.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.969+0000 7ff52ca8b640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff528100450 0x7ff5281a0590 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff51c00e9f0 tx=0x7ff51c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.970+0000 7ff525ffb640 1 -- 172.21.15.50:0/3707039172 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c00cd80 con 0x7ff528100450 2024-04-23T18:38:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.970+0000 7ff52ecef640 1 -- 172.21.15.50:0/3707039172 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52819b200 con 0x7ff528100450 2024-04-23T18:38:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.970+0000 7ff525ffb640 1 -- 172.21.15.50:0/3707039172 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff51c0108e0 con 0x7ff528100450 2024-04-23T18:38:10.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.970+0000 7ff52ecef640 1 -- 172.21.15.50:0/3707039172 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5281a7510 con 0x7ff528100450 2024-04-23T18:38:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.970+0000 7ff525ffb640 1 -- 172.21.15.50:0/3707039172 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c00cd80 con 0x7ff528100450 2024-04-23T18:38:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.973+0000 7ff525ffb640 1 -- 172.21.15.50:0/3707039172 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff51c010430 con 0x7ff528100450 2024-04-23T18:38:10.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.973+0000 7ff531a0a740 1 Processor -- start 2024-04-23T18:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.973+0000 7ff531a0a740 1 -- start start 2024-04-23T18:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.973+0000 7ff525ffb640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5180a41d0 0x7ff5180a6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff531a0a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556837472620 0x556837472a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff531a0a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472f40 0x55683747d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff531a0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 0x55683747fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff531a0a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5568373d7970 con 0x556837472f40 2024-04-23T18:38:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff531a0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5568373744a0 con 0x55683747d800 2024-04-23T18:38:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff531a0a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5568373d3930 con 0x556837472620 2024-04-23T18:38:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff525ffb640 1 -- 172.21.15.50:0/3707039172 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8586+0+0 (secure 0 0 0) 0x7ff51c00cb80 con 0x7ff528100450 2024-04-23T18:38:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff52d28c640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5180a41d0 0x7ff5180a6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff527fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472f40 0x55683747d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff52d28c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 0x55683747fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:10.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff52d28c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 0x55683747fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:45412/0 (socket says 172.21.15.50:45412) 2024-04-23T18:38:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff52d28c640 1 -- 172.21.15.50:0/3512461923 learned_addr learned my addr 172.21.15.50:0/3512461923 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:10.362 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.974+0000 7ff52ca8b640 1 --2- 172.21.15.50:0/3512461923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556837472620 0x556837472a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:10.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.975+0000 7ff52d28c640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5180a41d0 0x7ff5180a6690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff520009a60 tx=0x7ff5200099f0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:10.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.975+0000 7ff527fff640 1 -- 172.21.15.50:0/3512461923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556837472620 msgr2=0x556837472a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:10.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.975+0000 7ff527fff640 1 --2- 172.21.15.50:0/3512461923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556837472620 0x556837472a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.975+0000 7ff527fff640 1 -- 172.21.15.50:0/3512461923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 msgr2=0x55683747fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.975+0000 7ff527fff640 1 --2- 172.21.15.50:0/3512461923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 0x55683747fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.975+0000 7ff527fff640 1 -- 172.21.15.50:0/3512461923 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568370ea210 con 0x556837472f40 2024-04-23T18:38:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.975+0000 7ff527fff640 1 --2- 172.21.15.50:0/3512461923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472f40 0x55683747d2c0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ff51400a9b0 tx=0x7ff51400ae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5034de783d5c8e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:10.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.976+0000 7ff512ffd640 1 -- 172.21.15.50:0/3512461923 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51400cd00 con 0x556837472f40 2024-04-23T18:38:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.976+0000 7ff512ffd640 1 -- 172.21.15.50:0/3512461923 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff51400f900 con 0x556837472f40 2024-04-23T18:38:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.976+0000 7ff512ffd640 1 -- 172.21.15.50:0/3512461923 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff514018940 con 0x556837472f40 2024-04-23T18:38:10.524 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.977+0000 7ff531a0a740 1 -- 172.21.15.50:0/3512461923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472f40 msgr2=0x55683747d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:10.534 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.977+0000 7ff531a0a740 1 --2- 172.21.15.50:0/3512461923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472f40 0x55683747d2c0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ff51400a9b0 tx=0x7ff51400ae70 comp rx=0 tx=0).stop 2024-04-23T18:38:10.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.977+0000 7ff531a0a740 1 -- 172.21.15.50:0/3512461923 shutdown_connections 2024-04-23T18:38:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.977+0000 7ff531a0a740 1 --2- 172.21.15.50:0/3512461923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 0x55683747fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.977+0000 7ff531a0a740 1 --2- 172.21.15.50:0/3512461923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472f40 0x55683747d2c0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.977+0000 7ff531a0a740 1 --2- 172.21.15.50:0/3512461923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556837472620 0x556837472a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.977+0000 7ff531a0a740 1 -- 172.21.15.50:0/3512461923 >> 172.21.15.50:0/3512461923 conn(0x55683746eb30 msgr2=0x55683746ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.977+0000 7ff531a0a740 1 -- 172.21.15.50:0/3512461923 shutdown_connections 2024-04-23T18:38:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.977+0000 7ff531a0a740 1 -- 172.21.15.50:0/3512461923 wait complete. 2024-04-23T18:38:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.977+0000 7ff531a0a740 1 -- start start 2024-04-23T18:38:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.977+0000 7ff531a0a740 1 Processor -- start 2024-04-23T18:38:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.978+0000 7ff531a0a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x55683747d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.978+0000 7ff531a0a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556837472f40 0x55683747a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.978+0000 7ff531a0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 0x55683747aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:10.802 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.978+0000 7ff531a0a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556837480430 con 0x556837472620 2024-04-23T18:38:10.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.978+0000 7ff531a0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5568373720b0 con 0x55683747d800 2024-04-23T18:38:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.978+0000 7ff531a0a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5568374805b0 con 0x556837472f40 2024-04-23T18:38:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.978+0000 7ff531a0a740 10 client.0 _ll_register_callbacks cb 0x55683735fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:38:10.835 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.978+0000 7ff52d28c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 0x55683747aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.978+0000 7ff52ca8b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x55683747d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:10.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.978+0000 7ff52d28c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 0x55683747aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:45428/0 (socket says 172.21.15.50:45428) 2024-04-23T18:38:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.978+0000 7ff52d28c640 1 -- 172.21.15.50:0/961485307 learned_addr learned my addr 172.21.15.50:0/961485307 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:10.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.978+0000 7ff527fff640 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556837472f40 0x55683747a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:10.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff52d28c640 1 -- 172.21.15.50:0/961485307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556837472f40 msgr2=0x55683747a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:10.947 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff52d28c640 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556837472f40 0x55683747a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff52d28c640 1 -- 172.21.15.50:0/961485307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 msgr2=0x55683747d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff52d28c640 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x55683747d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff52d28c640 1 -- 172.21.15.50:0/961485307 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568371b6620 con 0x55683747d800 2024-04-23T18:38:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff52ca8b640 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x55683747d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:38:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff52d28c640 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 0x55683747aa00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff520033040 tx=0x7ff520046000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d0a98b3913501f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff507fff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:38:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff507fff640 1 -- 172.21.15.50:0/961485307 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff520004490 con 0x55683747d800 2024-04-23T18:38:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff531a0a740 1 -- 172.21.15.50:0/961485307 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff51400f900 con 0x55683747d800 2024-04-23T18:38:11.103 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff507fff640 1 -- 172.21.15.50:0/961485307 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8586+0+0 (secure 0 0 0) 0x7ff520004630 con 0x55683747d800 2024-04-23T18:38:11.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff507fff640 1 -- 172.21.15.50:0/961485307 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7ff520004830 con 0x55683747d800 2024-04-23T18:38:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff531a0a740 10 client.27371 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.979+0000 7ff531a0a740 1 -- 172.21.15.50:0/961485307 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55683748c470 con 0x55683747d800 2024-04-23T18:38:11.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.980+0000 7ff507fff640 1 -- 172.21.15.50:0/961485307 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff51400f900 con 0x55683747d800 2024-04-23T18:38:11.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.980+0000 7ff507fff640 1 -- 172.21.15.50:0/961485307 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7ff520049070 con 0x55683747d800 2024-04-23T18:38:11.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.980+0000 7ff531a0a740 10 client.27371 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:11.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.980+0000 7ff531a0a740 10 client.27371 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:38:11.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.980+0000 7ff531a0a740 20 client.27371 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:11.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.980+0000 7ff531a0a740 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55683748d930 0x55683748fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.980+0000 7ff531a0a740 4 client.27371 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:38:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.980+0000 7ff531a0a740 1 -- 172.21.15.50:0/961485307 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568373e5390 con 0x55683748d930 2024-04-23T18:38:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.980+0000 7ff52ca8b640 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55683748d930 0x55683748fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:11.378 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.981+0000 7ff52ca8b640 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55683748d930 0x55683748fd20 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df1c66cc06d30d1 server_cookie=e1c93d453e89beae in_seq=0 out_seq=0 2024-04-23T18:38:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.981+0000 7ff507fff640 10 client.27371 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:11.399 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.982+0000 7ff507fff640 1 -- 172.21.15.50:0/961485307 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5568373e5390 con 0x55683748d930 2024-04-23T18:38:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.982+0000 7ff507fff640 10 client.27371 handle_command_reply: tid=0 2024-04-23T18:38:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.982+0000 7ff531a0a740 1 client.27371 shutdown 2024-04-23T18:38:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.982+0000 7ff531a0a740 1 -- 172.21.15.50:0/961485307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 msgr2=0x55683747aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.982+0000 7ff531a0a740 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 0x55683747aa00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff520033040 tx=0x7ff520046000 comp rx=0 tx=0).stop 2024-04-23T18:38:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.983+0000 7ff531a0a740 1 -- 172.21.15.50:0/961485307 shutdown_connections 2024-04-23T18:38:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.983+0000 7ff531a0a740 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55683748d930 0x55683748fd20 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.983+0000 7ff531a0a740 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55683747d800 0x55683747aa00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.983+0000 7ff531a0a740 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556837472f40 0x55683747a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.983+0000 7ff531a0a740 1 --2- 172.21.15.50:0/961485307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x55683747d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.983+0000 7ff507fff640 0 client.27371 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.983+0000 7ff531a0a740 1 -- 172.21.15.50:0/961485307 >> 172.21.15.50:0/961485307 conn(0x55683746eb30 msgr2=0x5568374748b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.983+0000 7ff531a0a740 1 -- 172.21.15.50:0/961485307 shutdown_connections 2024-04-23T18:38:11.682 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.983+0000 7ff531a0a740 1 -- 172.21.15.50:0/961485307 wait complete. 2024-04-23T18:38:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:08.983+0000 7ff531a0a740 20 client.27371 trim_cache size 0 max 16384 2024-04-23T18:38:11.709 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:09 smithi188 ceph-mon[32694]: pgmap v625: 97 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 75 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 11 MiB/s wr, 704 op/s 2024-04-23T18:38:11.725 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:09 smithi177 ceph-mon[32308]: pgmap v625: 97 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 75 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 11 MiB/s wr, 704 op/s 2024-04-23T18:38:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.003+0000 7ff531a0a740 1 Processor -- start 2024-04-23T18:38:11.753 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.003+0000 7ff531a0a740 1 -- start start 2024-04-23T18:38:11.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.003+0000 7ff531a0a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x556837489350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.003+0000 7ff531a0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556837472f40 0x556837495630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:11.787 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.003+0000 7ff531a0a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55683747d800 0x556837495b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.003+0000 7ff531a0a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5281b2650 con 0x556837472620 2024-04-23T18:38:11.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.003+0000 7ff531a0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff5281aff20 con 0x556837472f40 2024-04-23T18:38:11.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.003+0000 7ff531a0a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff5281add90 con 0x55683747d800 2024-04-23T18:38:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.003+0000 7ff52ca8b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x556837489350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.003+0000 7ff52ca8b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x556837489350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40950/0 (socket says 172.21.15.50:40950) 2024-04-23T18:38:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.003+0000 7ff52ca8b640 1 -- 172.21.15.50:0/2929253791 learned_addr learned my addr 172.21.15.50:0/2929253791 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.003+0000 7ff52d28c640 1 --2- 172.21.15.50:0/2929253791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55683747d800 0x556837495b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.004+0000 7ff527fff640 1 --2- 172.21.15.50:0/2929253791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556837472f40 0x556837495630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:12.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.004+0000 7ff52ca8b640 1 -- 172.21.15.50:0/2929253791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55683747d800 msgr2=0x556837495b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:12.038 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.004+0000 7ff52ca8b640 1 --2- 172.21.15.50:0/2929253791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55683747d800 0x556837495b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:12.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.004+0000 7ff52ca8b640 1 -- 172.21.15.50:0/2929253791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556837472f40 msgr2=0x556837495630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:12.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.004+0000 7ff52ca8b640 1 --2- 172.21.15.50:0/2929253791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556837472f40 0x556837495630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:12.067 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.004+0000 7ff52ca8b640 1 -- 172.21.15.50:0/2929253791 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568374789f0 con 0x556837472620 2024-04-23T18:38:12.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.004+0000 7ff52ca8b640 1 --2- 172.21.15.50:0/2929253791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x556837489350 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ff51c09c4f0 tx=0x7ff51c018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7779b5a7d3bdad server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.004+0000 7ff5127fc640 1 -- 172.21.15.50:0/2929253791 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c09a600 con 0x556837472620 2024-04-23T18:38:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.004+0000 7ff5127fc640 1 -- 172.21.15.50:0/2929253791 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff51c09a860 con 0x556837472620 2024-04-23T18:38:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.004+0000 7ff5127fc640 1 -- 172.21.15.50:0/2929253791 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c098510 con 0x556837472620 2024-04-23T18:38:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.005+0000 7ff531a0a740 1 -- 172.21.15.50:0/2929253791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 msgr2=0x556837489350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.005+0000 7ff531a0a740 1 --2- 172.21.15.50:0/2929253791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x556837489350 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ff51c09c4f0 tx=0x7ff51c018420 comp rx=0 tx=0).stop 2024-04-23T18:38:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.005+0000 7ff531a0a740 1 -- 172.21.15.50:0/2929253791 shutdown_connections 2024-04-23T18:38:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.005+0000 7ff531a0a740 1 --2- 172.21.15.50:0/2929253791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55683747d800 0x556837495b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.005+0000 7ff531a0a740 1 --2- 172.21.15.50:0/2929253791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556837472f40 0x556837495630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.005+0000 7ff531a0a740 1 --2- 172.21.15.50:0/2929253791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x556837489350 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:12.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.005+0000 7ff531a0a740 1 -- 172.21.15.50:0/2929253791 >> 172.21.15.50:0/2929253791 conn(0x55683746eb30 msgr2=0x55683746dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.005+0000 7ff531a0a740 1 -- 172.21.15.50:0/2929253791 shutdown_connections 2024-04-23T18:38:12.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.005+0000 7ff531a0a740 1 -- 172.21.15.50:0/2929253791 wait complete. 2024-04-23T18:38:12.321 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff531a0a740 1 -- start start 2024-04-23T18:38:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff531a0a740 1 Processor -- start 2024-04-23T18:38:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff531a0a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x5568374839d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff531a0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556837472f40 0x556837483f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:12.354 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff531a0a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55683747d800 0x5568374fb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:12.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff531a0a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5281ae3e0 con 0x556837472620 2024-04-23T18:38:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff531a0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff5281af850 con 0x556837472f40 2024-04-23T18:38:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff531a0a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff5281aeab0 con 0x55683747d800 2024-04-23T18:38:12.392 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff531a0a740 10 client.0 _ll_register_callbacks cb 0x7ff5200316e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:38:12.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff52ca8b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x5568374839d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff527fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556837472f40 0x556837483f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff52ca8b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x5568374839d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40954/0 (socket says 172.21.15.50:40954) 2024-04-23T18:38:12.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff52ca8b640 1 -- 172.21.15.50:0/441302929 learned_addr learned my addr 172.21.15.50:0/441302929 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.006+0000 7ff52d28c640 1 --2- 172.21.15.50:0/441302929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55683747d800 0x5568374fb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.007+0000 7ff527fff640 1 -- 172.21.15.50:0/441302929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55683747d800 msgr2=0x5568374fb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:12.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.007+0000 7ff527fff640 1 --2- 172.21.15.50:0/441302929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55683747d800 0x5568374fb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.007+0000 7ff527fff640 1 -- 172.21.15.50:0/441302929 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 msgr2=0x5568374839d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:12.582 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.007+0000 7ff527fff640 1 --2- 172.21.15.50:0/441302929 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x5568374839d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:12.595 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.007+0000 7ff527fff640 1 -- 172.21.15.50:0/441302929 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568374fbc00 con 0x556837472f40 2024-04-23T18:38:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.007+0000 7ff527fff640 1 --2- 172.21.15.50:0/441302929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556837472f40 0x556837483f10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff51400a2a0 tx=0x7ff514018400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=824b7a9a4991d80 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:12.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.007+0000 7ff507fff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:38:12.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.007+0000 7ff507fff640 1 -- 172.21.15.50:0/441302929 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff514029060 con 0x556837472f40 2024-04-23T18:38:12.641 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.007+0000 7ff507fff640 1 -- 172.21.15.50:0/441302929 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8586+0+0 (secure 0 0 0) 0x7ff514004220 con 0x556837472f40 2024-04-23T18:38:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff531a0a740 1 -- 172.21.15.50:0/441302929 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff51c09a860 con 0x556837472f40 2024-04-23T18:38:12.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff507fff640 1 -- 172.21.15.50:0/441302929 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7ff514004440 con 0x556837472f40 2024-04-23T18:38:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff531a0a740 10 client.27377 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff531a0a740 1 -- 172.21.15.50:0/441302929 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5568374fcec0 con 0x556837472f40 2024-04-23T18:38:12.724 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff507fff640 1 -- 172.21.15.50:0/441302929 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff51401e050 con 0x556837472f40 2024-04-23T18:38:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff507fff640 1 -- 172.21.15.50:0/441302929 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7ff5140047a0 con 0x556837472f40 2024-04-23T18:38:12.795 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff531a0a740 10 client.27377 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff531a0a740 10 client.27377 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:38:12.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff531a0a740 20 client.27377 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff531a0a740 1 --2- 172.21.15.50:0/441302929 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5568374fd250 0x5568374ff640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:12.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff531a0a740 4 client.27377 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:38:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff531a0a740 1 -- 172.21.15.50:0/441302929 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7ff51c09a860 con 0x5568374fd250 2024-04-23T18:38:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.008+0000 7ff52ca8b640 1 --2- 172.21.15.50:0/441302929 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5568374fd250 0x5568374ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.009+0000 7ff507fff640 10 client.27377 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.009+0000 7ff52ca8b640 1 --2- 172.21.15.50:0/441302929 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5568374fd250 0x5568374ff640 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ea7d55c9eb4601c server_cookie=c39f6eb421bf752e in_seq=0 out_seq=0 2024-04-23T18:38:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.296+0000 7ff507fff640 1 -- 172.21.15.50:0/441302929 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7ff51c09a860 con 0x5568374fd250 2024-04-23T18:38:12.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.296+0000 7ff507fff640 10 client.27377 handle_command_reply: tid=0 2024-04-23T18:38:12.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.296+0000 7ff531a0a740 1 client.27377 shutdown 2024-04-23T18:38:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.296+0000 7ff531a0a740 1 -- 172.21.15.50:0/441302929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556837472f40 msgr2=0x556837483f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.296+0000 7ff531a0a740 1 --2- 172.21.15.50:0/441302929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556837472f40 0x556837483f10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff51400a2a0 tx=0x7ff514018400 comp rx=0 tx=0).stop 2024-04-23T18:38:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.297+0000 7ff531a0a740 1 -- 172.21.15.50:0/441302929 shutdown_connections 2024-04-23T18:38:13.038 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.297+0000 7ff531a0a740 1 --2- 172.21.15.50:0/441302929 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5568374fd250 0x5568374ff640 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.297+0000 7ff507fff640 0 client.27377 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:13.096 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.297+0000 7ff531a0a740 1 --2- 172.21.15.50:0/441302929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55683747d800 0x5568374fb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.297+0000 7ff531a0a740 1 --2- 172.21.15.50:0/441302929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556837472f40 0x556837483f10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.297+0000 7ff531a0a740 1 --2- 172.21.15.50:0/441302929 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556837472620 0x5568374839d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.297+0000 7ff531a0a740 1 -- 172.21.15.50:0/441302929 >> 172.21.15.50:0/441302929 conn(0x55683746eb30 msgr2=0x556837488340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.297+0000 7ff531a0a740 1 -- 172.21.15.50:0/441302929 shutdown_connections 2024-04-23T18:38:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.297+0000 7ff531a0a740 1 -- 172.21.15.50:0/441302929 wait complete. 2024-04-23T18:38:13.185 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.297+0000 7ff531a0a740 20 client.27377 trim_cache size 0 max 16384 2024-04-23T18:38:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:38:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:38:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:38:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:38:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.298+0000 7ff52ecef640 1 -- 172.21.15.50:0/3707039172 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5180a41d0 msgr2=0x7ff5180a6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:13.247 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.298+0000 7ff52ecef640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5180a41d0 0x7ff5180a6690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff520009a60 tx=0x7ff5200099f0 comp rx=0 tx=0).stop 2024-04-23T18:38:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.298+0000 7ff52ecef640 1 -- 172.21.15.50:0/3707039172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff528100450 msgr2=0x7ff5281a0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.298+0000 7ff52ecef640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff528100450 0x7ff5281a0590 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff51c00e9f0 tx=0x7ff51c00eeb0 comp rx=0 tx=0).stop 2024-04-23T18:38:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.298+0000 7ff52ecef640 1 -- 172.21.15.50:0/3707039172 shutdown_connections 2024-04-23T18:38:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.299+0000 7ff52ecef640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff5180a41d0 0x7ff5180a6690 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:13.365 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.299+0000 7ff52ecef640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff528106f30 0x7ff52819a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:13.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.299+0000 7ff52ecef640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff528100e10 0x7ff5281a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:13.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.299+0000 7ff52ecef640 1 --2- 172.21.15.50:0/3707039172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff528100450 0x7ff5281a0590 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:13.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.299+0000 7ff52ecef640 1 -- 172.21.15.50:0/3707039172 >> 172.21.15.50:0/3707039172 conn(0x7ff5280773e0 msgr2=0x7ff528104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.299+0000 7ff52ecef640 1 -- 172.21.15.50:0/3707039172 shutdown_connections 2024-04-23T18:38:13.457 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:09.300+0000 7ff52ecef640 1 -- 172.21.15.50:0/3707039172 wait complete. 2024-04-23T18:38:13.467 INFO:tasks.workunit.client.0.smithi050.stdout: 10 162697 21.03 MB/sec execute 283 sec latency 436.026 ms 2024-04-23T18:38:13.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:09 smithi050 ceph-mon[29187]: pgmap v625: 97 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 75 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 11 MiB/s wr, 704 op/s 2024-04-23T18:38:13.488 INFO:tasks.workunit.client.0.smithi050.stdout: 10 163170 21.03 MB/sec execute 284 sec latency 270.709 ms 2024-04-23T18:38:13.497 INFO:tasks.workunit.client.0.smithi050.stdout: 10 163711 21.05 MB/sec execute 285 sec latency 111.439 ms 2024-04-23T18:38:13.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:11 smithi050 ceph-mon[29187]: pgmap v626: 97 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 75 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 11 MiB/s wr, 704 op/s 2024-04-23T18:38:13.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:11 smithi188 ceph-mon[32694]: pgmap v626: 97 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 75 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 11 MiB/s wr, 704 op/s 2024-04-23T18:38:13.531 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:11 smithi177 ceph-mon[32308]: pgmap v626: 97 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 75 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 11 MiB/s wr, 704 op/s 2024-04-23T18:38:13.544 INFO:tasks.workunit.client.0.smithi050.stdout: 10 164385 21.12 MB/sec execute 286 sec latency 87.906 ms 2024-04-23T18:38:13.578 INFO:tasks.workunit.client.0.smithi050.stdout: 10 164928 21.17 MB/sec execute 287 sec latency 85.151 ms 2024-04-23T18:38:14.192 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:13 smithi050 ceph-mon[29187]: pgmap v627: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 663 op/s 2024-04-23T18:38:14.286 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:13 smithi177 ceph-mon[32308]: pgmap v627: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 663 op/s 2024-04-23T18:38:14.328 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:13 smithi188 ceph-mon[32694]: pgmap v627: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 663 op/s 2024-04-23T18:38:14.578 INFO:tasks.workunit.client.0.smithi050.stdout: 10 165273 21.16 MB/sec execute 288 sec latency 346.956 ms 2024-04-23T18:38:15.578 INFO:tasks.workunit.client.0.smithi050.stdout: 10 165619 21.13 MB/sec execute 289 sec latency 344.752 ms 2024-04-23T18:38:15.970 INFO:tasks.quiescer.fs.[cephfs]:Total duration (54.0) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T18:38:15.991 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.0 2024-04-23T18:38:16.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:38:16.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.195+0000 7f4f39be2640 1 -- 172.21.15.50:0/3322895008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f34103b00 msgr2=0x7f4f3410bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:16.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.195+0000 7f4f39be2640 1 --2- 172.21.15.50:0/3322895008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f34103b00 0x7f4f3410bfe0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009900 tx=0x7f4f2802a510 comp rx=0 tx=0).stop 2024-04-23T18:38:16.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.195+0000 7f4f39be2640 1 -- 172.21.15.50:0/3322895008 shutdown_connections 2024-04-23T18:38:16.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.195+0000 7f4f39be2640 1 --2- 172.21.15.50:0/3322895008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f3410c520 0x7f4f3410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:16.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.195+0000 7f4f39be2640 1 --2- 172.21.15.50:0/3322895008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f34103b00 0x7f4f3410bfe0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:16.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.195+0000 7f4f39be2640 1 --2- 172.21.15.50:0/3322895008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f341031e0 0x7f4f341035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:16.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.195+0000 7f4f39be2640 1 -- 172.21.15.50:0/3322895008 >> 172.21.15.50:0/3322895008 conn(0x7f4f34077f50 msgr2=0x7f4f34078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:16.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.196+0000 7f4f39be2640 1 -- 172.21.15.50:0/3322895008 shutdown_connections 2024-04-23T18:38:16.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.196+0000 7f4f39be2640 1 -- 172.21.15.50:0/3322895008 wait complete. 2024-04-23T18:38:16.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.196+0000 7f4f39be2640 1 Processor -- start 2024-04-23T18:38:16.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.196+0000 7f4f39be2640 1 -- start start 2024-04-23T18:38:16.489 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:15 smithi188 ceph-mon[32694]: pgmap v628: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 11 MiB/s wr, 636 op/s 2024-04-23T18:38:16.500 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:15 smithi177 ceph-mon[32308]: pgmap v628: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 11 MiB/s wr, 636 op/s 2024-04-23T18:38:16.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f39be2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f341031e0 0x7f4f34100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:16.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f39be2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f34103b00 0x7f4f34101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:16.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f39be2640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f3410c520 0x7f4f341018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:16.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f39be2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341121b0 con 0x7f4f34103b00 2024-04-23T18:38:16.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f39be2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4f34112030 con 0x7f4f341031e0 2024-04-23T18:38:16.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f39be2640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4f34112330 con 0x7f4f3410c520 2024-04-23T18:38:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f32ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f34103b00 0x7f4f34101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:16.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f337fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f341031e0 0x7f4f34100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:16.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f32ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f34103b00 0x7f4f34101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48654/0 (socket says 172.21.15.50:48654) 2024-04-23T18:38:16.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f32ffd640 1 -- 172.21.15.50:0/2352063549 learned_addr learned my addr 172.21.15.50:0/2352063549 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:16.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f33fff640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f3410c520 0x7f4f341018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:16.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f337fe640 1 -- 172.21.15.50:0/2352063549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f3410c520 msgr2=0x7f4f341018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:16.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f337fe640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f3410c520 0x7f4f341018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:16.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f337fe640 1 -- 172.21.15.50:0/2352063549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f34103b00 msgr2=0x7f4f34101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:16.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f337fe640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f34103b00 0x7f4f34101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:16.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f337fe640 1 -- 172.21.15.50:0/2352063549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f341a6c10 con 0x7f4f341031e0 2024-04-23T18:38:16.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f32ffd640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f34103b00 0x7f4f34101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:38:16.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.197+0000 7f4f337fe640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f341031e0 0x7f4f34100e30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4f2400e9f0 tx=0x7f4f2400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.198+0000 7f4f30ff9640 1 -- 172.21.15.50:0/2352063549 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2400cd80 con 0x7f4f341031e0 2024-04-23T18:38:17.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.198+0000 7f4f30ff9640 1 -- 172.21.15.50:0/2352063549 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f240035f0 con 0x7f4f341031e0 2024-04-23T18:38:17.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.198+0000 7f4f39be2640 1 -- 172.21.15.50:0/2352063549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f341a6f00 con 0x7f4f341031e0 2024-04-23T18:38:17.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.198+0000 7f4f30ff9640 1 -- 172.21.15.50:0/2352063549 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24010670 con 0x7f4f341031e0 2024-04-23T18:38:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.198+0000 7f4f39be2640 1 -- 172.21.15.50:0/2352063549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f341a7370 con 0x7f4f341031e0 2024-04-23T18:38:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.199+0000 7f4f39be2640 1 -- 172.21.15.50:0/2352063549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f34068d70 con 0x7f4f341031e0 2024-04-23T18:38:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.200+0000 7f4f30ff9640 1 -- 172.21.15.50:0/2352063549 <== mon.2 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4f240037a0 con 0x7f4f341031e0 2024-04-23T18:38:17.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:15 smithi050 ceph-mon[29187]: pgmap v628: 97 pgs: 97 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 11 MiB/s wr, 636 op/s 2024-04-23T18:38:17.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.201+0000 7f4f30ff9640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f4f1c0a3d20 0x7f4f1c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.201+0000 7f4f32ffd640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f4f1c0a3d20 0x7f4f1c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:17.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.201+0000 7f4f30ff9640 1 -- 172.21.15.50:0/2352063549 <== mon.2 v2:172.21.15.177:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8586+0+0 (secure 0 0 0) 0x7f4f240d0a60 con 0x7f4f341031e0 2024-04-23T18:38:17.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.201+0000 7f4f32ffd640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f4f1c0a3d20 0x7f4f1c0a61e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4f28031440 tx=0x7f4f280058d0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:17.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.205+0000 7f4f30ff9640 1 -- 172.21.15.50:0/2352063549 <== mon.2 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f2409d0c0 con 0x7f4f341031e0 2024-04-23T18:38:17.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.472+0000 7f4f39be2640 1 -- 172.21.15.50:0/2352063549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f34102360 con 0x7f4f341031e0 2024-04-23T18:38:17.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.473+0000 7f4f30ff9640 1 -- 172.21.15.50:0/2352063549 <== mon.2 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+11875 (secure 0 0 0) 0x7f4f2409c9c0 con 0x7f4f341031e0 2024-04-23T18:38:17.343 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:38:17.357 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":62,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:36:29.984583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27194":{"gid":27194,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2754597719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2754597719},{"type":"v1","addr":"172.21.15.177:6837","nonce":2754597719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:38:17.372 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 62 2024-04-23T18:38:17.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.475+0000 7f4f39be2640 1 -- 172.21.15.50:0/2352063549 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f4f1c0a3d20 msgr2=0x7f4f1c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:17.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.475+0000 7f4f39be2640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f4f1c0a3d20 0x7f4f1c0a61e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4f28031440 tx=0x7f4f280058d0 comp rx=0 tx=0).stop 2024-04-23T18:38:17.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.475+0000 7f4f39be2640 1 -- 172.21.15.50:0/2352063549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f341031e0 msgr2=0x7f4f34100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:17.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.475+0000 7f4f39be2640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f341031e0 0x7f4f34100e30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4f2400e9f0 tx=0x7f4f2400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:38:17.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.475+0000 7f4f39be2640 1 -- 172.21.15.50:0/2352063549 shutdown_connections 2024-04-23T18:38:17.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.475+0000 7f4f39be2640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f4f1c0a3d20 0x7f4f1c0a61e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:17.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.475+0000 7f4f39be2640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f3410c520 0x7f4f341018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:17.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.475+0000 7f4f39be2640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f34103b00 0x7f4f34101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:17.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.475+0000 7f4f39be2640 1 --2- 172.21.15.50:0/2352063549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f341031e0 0x7f4f34100e30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:17.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.475+0000 7f4f39be2640 1 -- 172.21.15.50:0/2352063549 >> 172.21.15.50:0/2352063549 conn(0x7f4f34077f50 msgr2=0x7f4f340fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:17.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.475+0000 7f4f39be2640 1 -- 172.21.15.50:0/2352063549 shutdown_connections 2024-04-23T18:38:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:16.475+0000 7f4f39be2640 1 -- 172.21.15.50:0/2352063549 wait complete. 2024-04-23T18:38:17.760 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 87.0 --await-for 42.0' 2024-04-23T18:38:17.788 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 87.0 --await-for 42.0 2024-04-23T18:38:17.800 INFO:tasks.workunit.client.0.smithi050.stdout: 10 166070 21.14 MB/sec execute 290 sec latency 110.121 ms 2024-04-23T18:38:17.811 INFO:tasks.workunit.client.0.smithi050.stdout: 10 166660 21.16 MB/sec execute 291 sec latency 122.233 ms 2024-04-23T18:38:17.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:17 smithi050 ceph-mon[29187]: osdmap e124: 12 total, 12 up, 12 in 2024-04-23T18:38:17.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:17 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2352063549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:38:17.925 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:17 smithi188 ceph-mon[32694]: osdmap e124: 12 total, 12 up, 12 in 2024-04-23T18:38:17.956 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:17 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2352063549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:38:17.965 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:17 smithi177 ceph-mon[32308]: osdmap e124: 12 total, 12 up, 12 in 2024-04-23T18:38:17.975 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:17 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2352063549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:38:18.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.070+0000 7efe0cbe2640 1 -- 172.21.15.50:0/3367659917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe08107260 msgr2=0x7efe0810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:18.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.070+0000 7efe0cbe2640 1 --2- 172.21.15.50:0/3367659917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe08107260 0x7efe0810ee30 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7efe00009ca0 tx=0x7efe0002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:38:18.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.070+0000 7efe0cbe2640 1 -- 172.21.15.50:0/3367659917 shutdown_connections 2024-04-23T18:38:18.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.070+0000 7efe0cbe2640 1 --2- 172.21.15.50:0/3367659917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe08107260 0x7efe0810ee30 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:18.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.070+0000 7efe0cbe2640 1 --2- 172.21.15.50:0/3367659917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe08102b40 0x7efe08106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:18.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.070+0000 7efe0cbe2640 1 --2- 172.21.15.50:0/3367659917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe08102190 0x7efe08102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:18.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.070+0000 7efe0cbe2640 1 -- 172.21.15.50:0/3367659917 >> 172.21.15.50:0/3367659917 conn(0x7efe08079310 msgr2=0x7efe08079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:18.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.070+0000 7efe0cbe2640 1 -- 172.21.15.50:0/3367659917 shutdown_connections 2024-04-23T18:38:18.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.070+0000 7efe0cbe2640 1 -- 172.21.15.50:0/3367659917 wait complete. 2024-04-23T18:38:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.070+0000 7efe0cbe2640 1 Processor -- start 2024-04-23T18:38:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0cbe2640 1 -- start start 2024-04-23T18:38:18.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0cbe2640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe08102190 0x7efe081a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0cbe2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe08107260 0x7efe081a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0cbe2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe0819a770 0x7efe0819abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:18.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0cbe2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efe081121d0 con 0x7efe08107260 2024-04-23T18:38:18.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0cbe2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efe08112050 con 0x7efe0819a770 2024-04-23T18:38:18.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0cbe2640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efe08112350 con 0x7efe08102190 2024-04-23T18:38:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe05d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe08107260 0x7efe081a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:18.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe06d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe0819a770 0x7efe0819abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe05d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe08107260 0x7efe081a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48668/0 (socket says 172.21.15.50:48668) 2024-04-23T18:38:18.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe06d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe0819a770 0x7efe0819abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:42224/0 (socket says 172.21.15.50:42224) 2024-04-23T18:38:18.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0659c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe08102190 0x7efe081a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:18.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe05d9b640 1 -- 172.21.15.50:0/2778852492 learned_addr learned my addr 172.21.15.50:0/2778852492 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:18.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0659c640 1 -- 172.21.15.50:0/2778852492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe0819a770 msgr2=0x7efe0819abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0659c640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe0819a770 0x7efe0819abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:18.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0659c640 1 -- 172.21.15.50:0/2778852492 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe08107260 msgr2=0x7efe081a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:18.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0659c640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe08107260 0x7efe081a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:18.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe0659c640 1 -- 172.21.15.50:0/2778852492 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0819b370 con 0x7efe08102190 2024-04-23T18:38:18.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe06d9d640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe0819a770 0x7efe0819abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:38:18.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.071+0000 7efe05d9b640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe08107260 0x7efe081a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:38:18.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.072+0000 7efe0659c640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe08102190 0x7efe081a06b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efdf800e9f0 tx=0x7efdf800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:18.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.072+0000 7efdf77fe640 1 -- 172.21.15.50:0/2778852492 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf800ccc0 con 0x7efe08102190 2024-04-23T18:38:18.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.072+0000 7efe0cbe2640 1 -- 172.21.15.50:0/2778852492 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe081a7450 con 0x7efe08102190 2024-04-23T18:38:18.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.072+0000 7efdf77fe640 1 -- 172.21.15.50:0/2778852492 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdf8005020 con 0x7efe08102190 2024-04-23T18:38:18.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.072+0000 7efdf77fe640 1 -- 172.21.15.50:0/2778852492 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8004a00 con 0x7efe08102190 2024-04-23T18:38:18.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.072+0000 7efe0cbe2640 1 -- 172.21.15.50:0/2778852492 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe081a7850 con 0x7efe08102190 2024-04-23T18:38:18.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.075+0000 7efdf77fe640 1 -- 172.21.15.50:0/2778852492 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7efdf8004ba0 con 0x7efe08102190 2024-04-23T18:38:18.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efdf77fe640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efdf00a3d20 0x7efdf00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:18.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe0f9d5740 1 Processor -- start 2024-04-23T18:38:18.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe05d9b640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efdf00a3d20 0x7efdf00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:18.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efdf77fe640 1 -- 172.21.15.50:0/2778852492 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7efdf80d0800 con 0x7efe08102190 2024-04-23T18:38:18.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe0f9d5740 1 -- start start 2024-04-23T18:38:18.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe0f9d5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1beb98480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:18.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe0f9d5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb989c0 0x55a1beba2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:18.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe0f9d5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1beba5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:19.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe0f9d5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a1beafd370 con 0x55a1beba3090 2024-04-23T18:38:19.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe0f9d5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a1bea9a2e0 con 0x55a1beb989c0 2024-04-23T18:38:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe0f9d5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a1beaf9330 con 0x55a1beb980a0 2024-04-23T18:38:19.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe05d9b640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efdf00a3d20 0x7efdf00a61e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efe0819bfc0 tx=0x7efdfc005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:19.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe06d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1beba5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:19.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe06d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1beba5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48670/0 (socket says 172.21.15.50:48670) 2024-04-23T18:38:19.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe06d9d640 1 -- 172.21.15.50:0/528593590 learned_addr learned my addr 172.21.15.50:0/528593590 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:19.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.076+0000 7efe0659c640 1 --2- 172.21.15.50:0/528593590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1beb98480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:19.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.077+0000 7efe05d9b640 1 --2- 172.21.15.50:0/528593590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb989c0 0x55a1beba2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.077+0000 7efe06d9d640 1 -- 172.21.15.50:0/528593590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 msgr2=0x55a1beb98480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:19.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.077+0000 7efe06d9d640 1 --2- 172.21.15.50:0/528593590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1beb98480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:19.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.077+0000 7efe06d9d640 1 -- 172.21.15.50:0/528593590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb989c0 msgr2=0x55a1beba2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:19.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.077+0000 7efe06d9d640 1 --2- 172.21.15.50:0/528593590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb989c0 0x55a1beba2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:19.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.077+0000 7efe06d9d640 1 -- 172.21.15.50:0/528593590 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1be8dbd80 con 0x55a1beba3090 2024-04-23T18:38:19.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.077+0000 7efe0659c640 1 --2- 172.21.15.50:0/528593590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1beb98480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:38:19.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.077+0000 7efe06d9d640 1 --2- 172.21.15.50:0/528593590 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1beba5480 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7efe00033040 tx=0x7efe0002f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562d7ee931198afe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:19.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.077+0000 7efdf4ff9640 1 -- 172.21.15.50:0/528593590 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe00031450 con 0x55a1beba3090 2024-04-23T18:38:19.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.077+0000 7efdf4ff9640 1 -- 172.21.15.50:0/528593590 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe00036040 con 0x55a1beba3090 2024-04-23T18:38:19.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.077+0000 7efdf4ff9640 1 -- 172.21.15.50:0/528593590 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe000043c0 con 0x55a1beba3090 2024-04-23T18:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.078+0000 7efe0f9d5740 1 -- 172.21.15.50:0/528593590 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 msgr2=0x55a1beba5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:19.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.078+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/528593590 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1beba5480 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7efe00033040 tx=0x7efe0002f310 comp rx=0 tx=0).stop 2024-04-23T18:38:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.078+0000 7efe0f9d5740 1 -- 172.21.15.50:0/528593590 shutdown_connections 2024-04-23T18:38:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.078+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/528593590 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1beba5480 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:20.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.078+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/528593590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb989c0 0x55a1beba2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:20.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.078+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/528593590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1beb98480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:20.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.078+0000 7efe0f9d5740 1 -- 172.21.15.50:0/528593590 >> 172.21.15.50:0/528593590 conn(0x55a1beb943c0 msgr2=0x55a1beb947a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:20.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.078+0000 7efe0f9d5740 1 -- 172.21.15.50:0/528593590 shutdown_connections 2024-04-23T18:38:20.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.078+0000 7efe0f9d5740 1 -- 172.21.15.50:0/528593590 wait complete. 2024-04-23T18:38:20.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.078+0000 7efe0f9d5740 1 -- start start 2024-04-23T18:38:20.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe0f9d5740 1 Processor -- start 2024-04-23T18:38:20.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe0f9d5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb980a0 0x55a1beba0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:20.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe0f9d5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb989c0 0x55a1beba0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:20.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe0f9d5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1beba0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:20.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe0f9d5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a1beba5ca0 con 0x55a1beba3090 2024-04-23T18:38:20.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe0f9d5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a1bea97fb0 con 0x55a1beb980a0 2024-04-23T18:38:20.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe0f9d5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a1beba5e20 con 0x55a1beb989c0 2024-04-23T18:38:20.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe0f9d5740 10 client.0 _ll_register_callbacks cb 0x55a1bea979a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:38:20.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe05d9b640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb989c0 0x55a1beba0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:20.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe06d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1beba0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:20.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe06d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1beba0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48682/0 (socket says 172.21.15.50:48682) 2024-04-23T18:38:20.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe06d9d640 1 -- 172.21.15.50:0/1023660301 learned_addr learned my addr 172.21.15.50:0/1023660301 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:20.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.079+0000 7efe0659c640 1 --2- 172.21.15.50:0/1023660301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb980a0 0x55a1beba0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:20.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.080+0000 7efe06d9d640 1 -- 172.21.15.50:0/1023660301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb989c0 msgr2=0x55a1beba0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:20.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.080+0000 7efe06d9d640 1 --2- 172.21.15.50:0/1023660301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb989c0 0x55a1beba0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.080+0000 7efe06d9d640 1 -- 172.21.15.50:0/1023660301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb980a0 msgr2=0x55a1beba0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:20.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.080+0000 7efe06d9d640 1 --2- 172.21.15.50:0/1023660301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb980a0 0x55a1beba0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:20.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.080+0000 7efe06d9d640 1 -- 172.21.15.50:0/1023660301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1be93c510 con 0x55a1beba3090 2024-04-23T18:38:20.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.080+0000 7efe06d9d640 1 --2- 172.21.15.50:0/1023660301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1beba0aa0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7efe00009bf0 tx=0x7efe00031680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6003442aecd51e83 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:20.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.080+0000 7efdedffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:38:20.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.080+0000 7efdedffb640 1 -- 172.21.15.50:0/1023660301 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe000317f0 con 0x55a1beba3090 2024-04-23T18:38:20.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.081+0000 7efe0f9d5740 1 -- 172.21.15.50:0/1023660301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe00036040 con 0x55a1beba3090 2024-04-23T18:38:20.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.081+0000 7efdedffb640 1 -- 172.21.15.50:0/1023660301 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7efe00004120 con 0x55a1beba3090 2024-04-23T18:38:20.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.081+0000 7efdedffb640 1 -- 172.21.15.50:0/1023660301 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7efe0005e050 con 0x55a1beba3090 2024-04-23T18:38:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.081+0000 7efe0f9d5740 10 client.17904 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:20.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.081+0000 7efe0f9d5740 1 -- 172.21.15.50:0/1023660301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55a1be7cf670 con 0x55a1beba3090 2024-04-23T18:38:20.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.082+0000 7efdedffb640 1 -- 172.21.15.50:0/1023660301 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe00036040 con 0x55a1beba3090 2024-04-23T18:38:20.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.082+0000 7efdedffb640 1 -- 172.21.15.50:0/1023660301 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7efe0004f070 con 0x55a1beba3090 2024-04-23T18:38:20.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.082+0000 7efe0f9d5740 10 client.17904 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:20.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.082+0000 7efe0f9d5740 10 client.17904 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:38:20.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.082+0000 7efe0f9d5740 20 client.17904 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:21.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.082+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/1023660301 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a1bebbb900 0x55a1bebbdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:21.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.082+0000 7efe0f9d5740 4 client.17904 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:38:21.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.082+0000 7efe0f9d5740 1 -- 172.21.15.50:0/1023660301 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1beb0ad90 con 0x55a1bebbb900 2024-04-23T18:38:21.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.082+0000 7efe0659c640 1 --2- 172.21.15.50:0/1023660301 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a1bebbb900 0x55a1bebbdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:21.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.083+0000 7efdedffb640 10 client.17904 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:21.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.083+0000 7efe0659c640 1 --2- 172.21.15.50:0/1023660301 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a1bebbb900 0x55a1bebbdcf0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa7f2a6d5b7b083b server_cookie=a986ff6915bcd46 in_seq=0 out_seq=0 2024-04-23T18:38:21.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.086+0000 7efdedffb640 1 -- 172.21.15.50:0/1023660301 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a1beb0ad90 con 0x55a1bebbb900 2024-04-23T18:38:21.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.086+0000 7efdedffb640 10 client.17904 handle_command_reply: tid=0 2024-04-23T18:38:21.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.086+0000 7efe0f9d5740 1 client.17904 shutdown 2024-04-23T18:38:21.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.086+0000 7efe0f9d5740 1 -- 172.21.15.50:0/1023660301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 msgr2=0x55a1beba0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:21.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.086+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/1023660301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1beba0aa0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7efe00009bf0 tx=0x7efe00031680 comp rx=0 tx=0).stop 2024-04-23T18:38:21.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.087+0000 7efe0f9d5740 1 -- 172.21.15.50:0/1023660301 shutdown_connections 2024-04-23T18:38:21.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.087+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/1023660301 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a1bebbb900 0x55a1bebbdcf0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.087+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/1023660301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1beba0aa0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.087+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/1023660301 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb989c0 0x55a1beba0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:21.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.087+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/1023660301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb980a0 0x55a1beba0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:21.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.087+0000 7efe0f9d5740 1 -- 172.21.15.50:0/1023660301 >> 172.21.15.50:0/1023660301 conn(0x55a1beb943c0 msgr2=0x55a1beb9a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.087+0000 7efdedffb640 0 client.17904 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.087+0000 7efe0f9d5740 1 -- 172.21.15.50:0/1023660301 shutdown_connections 2024-04-23T18:38:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.087+0000 7efe0f9d5740 1 -- 172.21.15.50:0/1023660301 wait complete. 2024-04-23T18:38:21.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.087+0000 7efe0f9d5740 20 client.17904 trim_cache size 0 max 16384 2024-04-23T18:38:21.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.107+0000 7efe0f9d5740 1 Processor -- start 2024-04-23T18:38:21.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.107+0000 7efe0f9d5740 1 -- start start 2024-04-23T18:38:21.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.107+0000 7efe0f9d5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1beba08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:21.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.107+0000 7efe0f9d5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb989c0 0x55a1beb9a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:21.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe0f9d5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1bebc34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:21.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe0f9d5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efe081ae250 con 0x55a1beba3090 2024-04-23T18:38:21.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe0f9d5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efe081abb20 con 0x55a1beb989c0 2024-04-23T18:38:21.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe0f9d5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efe081a99b0 con 0x55a1beb980a0 2024-04-23T18:38:21.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe06d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1bebc34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe0659c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1beba08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe06d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1bebc34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48694/0 (socket says 172.21.15.50:48694) 2024-04-23T18:38:21.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe0659c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1beba08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:44476/0 (socket says 172.21.15.50:44476) 2024-04-23T18:38:21.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe06d9d640 1 -- 172.21.15.50:0/3816815222 learned_addr learned my addr 172.21.15.50:0/3816815222 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:21.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe05d9b640 1 --2- 172.21.15.50:0/3816815222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb989c0 0x55a1beb9a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:21.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe0659c640 1 -- 172.21.15.50:0/3816815222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb989c0 msgr2=0x55a1beb9a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:21.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe0659c640 1 --2- 172.21.15.50:0/3816815222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb989c0 0x55a1beb9a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe0659c640 1 -- 172.21.15.50:0/3816815222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 msgr2=0x55a1bebc34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe0659c640 1 --2- 172.21.15.50:0/3816815222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1bebc34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe0659c640 1 -- 172.21.15.50:0/3816815222 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1beb987f0 con 0x55a1beb980a0 2024-04-23T18:38:21.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe06d9d640 1 --2- 172.21.15.50:0/3816815222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1bebc34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:38:21.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efe0659c640 1 --2- 172.21.15.50:0/3816815222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1beba08c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efdf80cdee0 tx=0x7efdf80096e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=577c3c405cbfcc7b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:22.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.108+0000 7efdeffff640 1 -- 172.21.15.50:0/3816815222 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf80b9070 con 0x55a1beb980a0 2024-04-23T18:38:22.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.109+0000 7efdeffff640 1 -- 172.21.15.50:0/3816815222 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdf80a0ae0 con 0x55a1beb980a0 2024-04-23T18:38:22.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.109+0000 7efdeffff640 1 -- 172.21.15.50:0/3816815222 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf801c040 con 0x55a1beb980a0 2024-04-23T18:38:22.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.109+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3816815222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 msgr2=0x55a1beba08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:22.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.109+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/3816815222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1beba08c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efdf80cdee0 tx=0x7efdf80096e0 comp rx=0 tx=0).stop 2024-04-23T18:38:22.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.109+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3816815222 shutdown_connections 2024-04-23T18:38:22.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.109+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/3816815222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beba3090 0x55a1bebc34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:22.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.109+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/3816815222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beb989c0 0x55a1beb9a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:22.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.109+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/3816815222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1beba08c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:22.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.109+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3816815222 >> 172.21.15.50:0/3816815222 conn(0x55a1beb943c0 msgr2=0x55a1beb99900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:22.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.109+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3816815222 shutdown_connections 2024-04-23T18:38:22.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.109+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3816815222 wait complete. 2024-04-23T18:38:22.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0f9d5740 1 -- start start 2024-04-23T18:38:22.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0f9d5740 1 Processor -- start 2024-04-23T18:38:22.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0f9d5740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1bebab0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:22.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0f9d5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beb989c0 0x55a1bebab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:22.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0f9d5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beba3090 0x55a1beba8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0f9d5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efe081aa000 con 0x55a1beb989c0 2024-04-23T18:38:22.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0f9d5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efe081ab450 con 0x55a1beba3090 2024-04-23T18:38:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0f9d5740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efe081aa6b0 con 0x55a1beb980a0 2024-04-23T18:38:22.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0f9d5740 10 client.0 _ll_register_callbacks cb 0x7efe00009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:38:22.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0659c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1bebab0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe05d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beb989c0 0x55a1bebab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:22.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0659c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1bebab0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:44482/0 (socket says 172.21.15.50:44482) 2024-04-23T18:38:22.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0659c640 1 -- 172.21.15.50:0/3196598386 learned_addr learned my addr 172.21.15.50:0/3196598386 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe06d9d640 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beba3090 0x55a1beba8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:22.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0659c640 1 -- 172.21.15.50:0/3196598386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beba3090 msgr2=0x55a1beba8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:22.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0659c640 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beba3090 0x55a1beba8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:22.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0659c640 1 -- 172.21.15.50:0/3196598386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beb989c0 msgr2=0x55a1bebab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:22.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0659c640 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beb989c0 0x55a1bebab5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe0659c640 1 -- 172.21.15.50:0/3196598386 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1beba87b0 con 0x55a1beb980a0 2024-04-23T18:38:22.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe05d9b640 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beb989c0 0x55a1bebab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:38:22.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.110+0000 7efe06d9d640 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beba3090 0x55a1beba8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:38:22.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.111+0000 7efe0659c640 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1bebab0a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efdf8098b90 tx=0x7efdf809a7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7169178d8f96ee9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:22.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.111+0000 7efdedffb640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:38:22.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.111+0000 7efdedffb640 1 -- 172.21.15.50:0/3196598386 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf801c050 con 0x55a1beb980a0 2024-04-23T18:38:22.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.111+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3196598386 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdf80a0ae0 con 0x55a1beb980a0 2024-04-23T18:38:22.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.111+0000 7efdedffb640 1 -- 172.21.15.50:0/3196598386 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7efdf809aa30 con 0x55a1beb980a0 2024-04-23T18:38:22.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.111+0000 7efdedffb640 1 -- 172.21.15.50:0/3196598386 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7efdf80b9070 con 0x55a1beb980a0 2024-04-23T18:38:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.111+0000 7efe0f9d5740 10 client.27433 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:22.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.111+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3196598386 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55a1bec22730 con 0x55a1beb980a0 2024-04-23T18:38:22.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.111+0000 7efdedffb640 1 -- 172.21.15.50:0/3196598386 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdf80a0ae0 con 0x55a1beb980a0 2024-04-23T18:38:22.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.112+0000 7efdedffb640 1 -- 172.21.15.50:0/3196598386 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7efdf80a4f10 con 0x55a1beb980a0 2024-04-23T18:38:22.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.112+0000 7efe0f9d5740 10 client.27433 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:22.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.112+0000 7efe0f9d5740 10 client.27433 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:38:22.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.112+0000 7efe0f9d5740 20 client.27433 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:22.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.112+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a1bec22ea0 0x55a1bec252c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:22.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.112+0000 7efe0f9d5740 4 client.27433 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.0, "await_for": 42.0}] 2024-04-23T18:38:23.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.112+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3196598386 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.0, "await_for": 42.0}) -- 0x55a1beb04c00 con 0x55a1bec22ea0 2024-04-23T18:38:23.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.112+0000 7efe05d9b640 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a1bec22ea0 0x55a1bec252c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:23.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.112+0000 7efdedffb640 10 client.27433 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.112+0000 7efe05d9b640 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a1bec22ea0 0x55a1bec252c0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5527c50cf755fab4 server_cookie=180364962d74714a in_seq=0 out_seq=0 2024-04-23T18:38:23.099 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167135 21.19 MB/sec execute 292 sec latency 435.361 ms 2024-04-23T18:38:23.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:18 smithi177 ceph-mon[32308]: pgmap v630: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 672 op/s 2024-04-23T18:38:23.144 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:20 smithi177 ceph-mon[32308]: pgmap v631: 97 pgs: 97 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 628 op/s 2024-04-23T18:38:23.151 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:22 smithi177 ceph-mon[32308]: pgmap v632: 97 pgs: 97 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 628 op/s 2024-04-23T18:38:23.159 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:18 smithi188 ceph-mon[32694]: pgmap v630: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 672 op/s 2024-04-23T18:38:23.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:20 smithi188 ceph-mon[32694]: pgmap v631: 97 pgs: 97 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 628 op/s 2024-04-23T18:38:23.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:22 smithi188 ceph-mon[32694]: pgmap v632: 97 pgs: 97 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 628 op/s 2024-04-23T18:38:23.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efdedffb640 1 -- 172.21.15.50:0/3196598386 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55a1beb04c00 con 0x55a1bec22ea0 2024-04-23T18:38:23.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efdedffb640 10 client.27433 handle_command_reply: tid=0 2024-04-23T18:38:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efe0f9d5740 1 client.27433 shutdown 2024-04-23T18:38:23.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3196598386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 msgr2=0x55a1bebab0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:23.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1bebab0a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efdf8098b90 tx=0x7efdf809a7d0 comp rx=0 tx=0).stop 2024-04-23T18:38:23.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3196598386 shutdown_connections 2024-04-23T18:38:23.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a1bec22ea0 0x55a1bec252c0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:23.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1beba3090 0x55a1beba8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:23.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efdedffb640 0 client.27433 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:23.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a1beb989c0 0x55a1bebab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efe0f9d5740 1 --2- 172.21.15.50:0/3196598386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a1beb980a0 0x55a1bebab0a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:23.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3196598386 >> 172.21.15.50:0/3196598386 conn(0x55a1beb943c0 msgr2=0x55a1beb9a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:23.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3196598386 shutdown_connections 2024-04-23T18:38:23.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efe0f9d5740 1 -- 172.21.15.50:0/3196598386 wait complete. 2024-04-23T18:38:23.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.682+0000 7efe0f9d5740 20 client.27433 trim_cache size 0 max 16384 2024-04-23T18:38:23.669 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:38:23.678 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 62, 2024-04-23T18:38:23.685 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:38:23.744 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 66, 2024-04-23T18:38:23.752 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:38:23.761 INFO:teuthology.orchestra.run.smithi050.stdout: "525eaceb": { 2024-04-23T18:38:23.829 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 66, 2024-04-23T18:38:23.838 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:38:23.847 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:38:23.855 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:38:23.916 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:38:23.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:38:23.972 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:38:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 87.0, 2024-04-23T18:38:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:38:24.035 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:38:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:38:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:38:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:38:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:38:24.124 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:38:24.132 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:38:24.141 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:38:24.195 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:38:24.205 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:38:24.212 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:38:24.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.683+0000 7efe0cbe2640 1 -- 172.21.15.50:0/2778852492 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efdf00a3d20 msgr2=0x7efdf00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:24.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.683+0000 7efe0cbe2640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efdf00a3d20 0x7efdf00a61e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efe0819bfc0 tx=0x7efdfc005f90 comp rx=0 tx=0).stop 2024-04-23T18:38:24.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.683+0000 7efe0cbe2640 1 -- 172.21.15.50:0/2778852492 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe08102190 msgr2=0x7efe081a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:24.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.683+0000 7efe0cbe2640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe08102190 0x7efe081a06b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efdf800e9f0 tx=0x7efdf800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:38:24.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.683+0000 7efe0cbe2640 1 -- 172.21.15.50:0/2778852492 shutdown_connections 2024-04-23T18:38:24.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.683+0000 7efe0cbe2640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7efdf00a3d20 0x7efdf00a61e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:24.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.683+0000 7efe0cbe2640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe0819a770 0x7efe0819abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:24.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.683+0000 7efe0cbe2640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe08107260 0x7efe081a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:24.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.683+0000 7efe0cbe2640 1 --2- 172.21.15.50:0/2778852492 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe08102190 0x7efe081a06b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:24.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.683+0000 7efe0cbe2640 1 -- 172.21.15.50:0/2778852492 >> 172.21.15.50:0/2778852492 conn(0x7efe08079310 msgr2=0x7efe081001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:24.478 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.683+0000 7efe0cbe2640 1 -- 172.21.15.50:0/2778852492 shutdown_connections 2024-04-23T18:38:24.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:18.683+0000 7efe0cbe2640 1 -- 172.21.15.50:0/2778852492 wait complete. 2024-04-23T18:38:25.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:18 smithi050 ceph-mon[29187]: pgmap v630: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 672 op/s 2024-04-23T18:38:25.019 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:24 smithi177 ceph-mon[32308]: pgmap v633: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 12 MiB/s wr, 482 op/s 2024-04-23T18:38:25.104 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:24 smithi188 ceph-mon[32694]: pgmap v633: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 12 MiB/s wr, 482 op/s 2024-04-23T18:38:25.152 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 21.11 MB/sec execute 293 sec latency 1380.015 ms 2024-04-23T18:38:25.179 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 21.04 MB/sec execute 294 sec latency 2380.144 ms 2024-04-23T18:38:25.188 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:20 smithi050 ceph-mon[29187]: pgmap v631: 97 pgs: 97 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 628 op/s 2024-04-23T18:38:25.241 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.97 MB/sec execute 295 sec latency 3380.222 ms 2024-04-23T18:38:25.284 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '525eaceb', quiesce took 9.3 seconds. Will release after: 27.0 2024-04-23T18:38:25.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.404+0000 7fe34ace8640 1 -- 172.21.15.50:0/2717974521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34410c5d0 msgr2=0x7fe34410ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:25.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.404+0000 7fe34ace8640 1 --2- 172.21.15.50:0/2717974521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34410c5d0 0x7fe34410ea00 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009ca0 tx=0x7fe33c02a950 comp rx=0 tx=0).stop 2024-04-23T18:38:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.404+0000 7fe34ace8640 1 -- 172.21.15.50:0/2717974521 shutdown_connections 2024-04-23T18:38:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.404+0000 7fe34ace8640 1 --2- 172.21.15.50:0/2717974521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34410c5d0 0x7fe34410ea00 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.404+0000 7fe34ace8640 1 --2- 172.21.15.50:0/2717974521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344103b20 0x7fe34410c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:25.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.404+0000 7fe34ace8640 1 --2- 172.21.15.50:0/2717974521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe344103200 0x7fe3441035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:25.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.404+0000 7fe34ace8640 1 -- 172.21.15.50:0/2717974521 >> 172.21.15.50:0/2717974521 conn(0x7fe3440771d0 msgr2=0x7fe3440775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:25.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.404+0000 7fe34ace8640 1 -- 172.21.15.50:0/2717974521 shutdown_connections 2024-04-23T18:38:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.404+0000 7fe34ace8640 1 -- 172.21.15.50:0/2717974521 wait complete. 2024-04-23T18:38:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe34ace8640 1 Processor -- start 2024-04-23T18:38:25.606 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe34ace8640 1 -- start start 2024-04-23T18:38:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe34ace8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe344103200 0x7fe344198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe34ace8640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344103b20 0x7fe344198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe34ace8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34410c5d0 0x7fe34419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:25.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe34ace8640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe3441121f0 con 0x7fe34410c5d0 2024-04-23T18:38:25.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe34ace8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe344112070 con 0x7fe344103200 2024-04-23T18:38:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe34ace8640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe344112370 con 0x7fe344103b20 2024-04-23T18:38:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe349285640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34410c5d0 0x7fe34419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:25.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe349285640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34410c5d0 0x7fe34419cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60412/0 (socket says 172.21.15.50:60412) 2024-04-23T18:38:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe349285640 1 -- 172.21.15.50:0/1220653224 learned_addr learned my addr 172.21.15.50:0/1220653224 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe343fff640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344103b20 0x7fe344198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe348a84640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe344103200 0x7fe344198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:25.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe349285640 1 -- 172.21.15.50:0/1220653224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344103b20 msgr2=0x7fe344198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:25.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe349285640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344103b20 0x7fe344198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:25.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe349285640 1 -- 172.21.15.50:0/1220653224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe344103200 msgr2=0x7fe344198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe349285640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe344103200 0x7fe344198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.405+0000 7fe349285640 1 -- 172.21.15.50:0/1220653224 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34419d610 con 0x7fe34410c5d0 2024-04-23T18:38:25.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.406+0000 7fe349285640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34410c5d0 0x7fe34419cee0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fe33c004af0 tx=0x7fe33c02f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:26.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.406+0000 7fe341ffb640 1 -- 172.21.15.50:0/1220653224 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c034040 con 0x7fe34410c5d0 2024-04-23T18:38:26.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.406+0000 7fe341ffb640 1 -- 172.21.15.50:0/1220653224 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe33c02fe80 con 0x7fe34410c5d0 2024-04-23T18:38:26.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.406+0000 7fe34ace8640 1 -- 172.21.15.50:0/1220653224 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe34419d8a0 con 0x7fe34410c5d0 2024-04-23T18:38:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.406+0000 7fe341ffb640 1 -- 172.21.15.50:0/1220653224 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c004670 con 0x7fe34410c5d0 2024-04-23T18:38:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.406+0000 7fe34ace8640 1 -- 172.21.15.50:0/1220653224 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3441a5190 con 0x7fe34410c5d0 2024-04-23T18:38:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.409+0000 7fe34d9fe740 1 Processor -- start 2024-04-23T18:38:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.409+0000 7fe341ffb640 1 -- 172.21.15.50:0/1220653224 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe33c037020 con 0x7fe34410c5d0 2024-04-23T18:38:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.409+0000 7fe34d9fe740 1 -- start start 2024-04-23T18:38:26.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe34d9fe740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55730776e620 0x55730776ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:26.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe34d9fe740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x5573077792c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe34d9fe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557307779800 0x55730777bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe34d9fe740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5573076d3970 con 0x55730776ef40 2024-04-23T18:38:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe34d9fe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5573076704a0 con 0x557307779800 2024-04-23T18:38:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe34d9fe740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5573076cf930 con 0x55730776e620 2024-04-23T18:38:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe348a84640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x5573077792c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:26.399 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe348a84640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x5573077792c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60426/0 (socket says 172.21.15.50:60426) 2024-04-23T18:38:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe348a84640 1 -- 172.21.15.50:0/806767359 learned_addr learned my addr 172.21.15.50:0/806767359 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:26.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe343fff640 1 --2- 172.21.15.50:0/806767359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55730776e620 0x55730776ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe343fff640 1 --2- 172.21.15.50:0/806767359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557307779800 0x55730777bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe341ffb640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe32c0a3dc0 0x7fe32c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe348a84640 1 -- 172.21.15.50:0/806767359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55730776e620 msgr2=0x55730776ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:26.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe348a84640 1 --2- 172.21.15.50:0/806767359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55730776e620 0x55730776ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe348a84640 1 -- 172.21.15.50:0/806767359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557307779800 msgr2=0x55730777bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:26.603 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe348a84640 1 --2- 172.21.15.50:0/806767359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557307779800 0x55730777bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe348a84640 1 -- 172.21.15.50:0/806767359 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573073e6210 con 0x55730776ef40 2024-04-23T18:38:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe343fff640 1 --2- 172.21.15.50:0/806767359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557307779800 0x55730777bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:38:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe349285640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe32c0a3dc0 0x7fe32c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:26.673 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe348a84640 1 --2- 172.21.15.50:0/806767359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x5573077792c0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fe33400bcd0 tx=0x7fe3340107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e27dd9bbdff893 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:26.681 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.410+0000 7fe341ffb640 1 -- 172.21.15.50:0/1220653224 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fe33c0f1520 con 0x7fe34410c5d0 2024-04-23T18:38:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.411+0000 7fe32affd640 1 -- 172.21.15.50:0/806767359 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe334005220 con 0x55730776ef40 2024-04-23T18:38:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.411+0000 7fe32affd640 1 -- 172.21.15.50:0/806767359 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe334004c50 con 0x55730776ef40 2024-04-23T18:38:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.411+0000 7fe32affd640 1 -- 172.21.15.50:0/806767359 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe334011680 con 0x55730776ef40 2024-04-23T18:38:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.411+0000 7fe34d9fe740 1 -- 172.21.15.50:0/806767359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 msgr2=0x5573077792c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.411+0000 7fe349285640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe32c0a3dc0 0x7fe32c0a6280 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0b99c0 tx=0x7fe33c004140 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.411+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/806767359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x5573077792c0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fe33400bcd0 tx=0x7fe3340107a0 comp rx=0 tx=0).stop 2024-04-23T18:38:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.411+0000 7fe34d9fe740 1 -- 172.21.15.50:0/806767359 shutdown_connections 2024-04-23T18:38:26.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.411+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/806767359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557307779800 0x55730777bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:26.876 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.411+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/806767359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x5573077792c0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:26.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.411+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/806767359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55730776e620 0x55730776ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.411+0000 7fe34d9fe740 1 -- 172.21.15.50:0/806767359 >> 172.21.15.50:0/806767359 conn(0x55730776ab30 msgr2=0x55730776af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.412+0000 7fe34d9fe740 1 -- 172.21.15.50:0/806767359 shutdown_connections 2024-04-23T18:38:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.412+0000 7fe34d9fe740 1 -- 172.21.15.50:0/806767359 wait complete. 2024-04-23T18:38:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.412+0000 7fe34d9fe740 1 -- start start 2024-04-23T18:38:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.412+0000 7fe34d9fe740 1 Processor -- start 2024-04-23T18:38:27.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.412+0000 7fe34d9fe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55730776e620 0x557307779170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:27.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.412+0000 7fe34d9fe740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x557307776460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.412+0000 7fe34d9fe740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557307779800 0x5573077769c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.412+0000 7fe34d9fe740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55730777c430 con 0x55730776ef40 2024-04-23T18:38:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.412+0000 7fe34d9fe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55730766e0b0 con 0x55730776e620 2024-04-23T18:38:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.412+0000 7fe34d9fe740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55730777c5b0 con 0x557307779800 2024-04-23T18:38:27.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe34d9fe740 10 client.0 _ll_register_callbacks cb 0x55730765bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:38:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe348a84640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x557307776460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe348a84640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x557307776460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60432/0 (socket says 172.21.15.50:60432) 2024-04-23T18:38:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe348a84640 1 -- 172.21.15.50:0/3059961419 learned_addr learned my addr 172.21.15.50:0/3059961419 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe343fff640 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55730776e620 0x557307779170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe343fff640 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557307779800 0x5573077769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:27.350 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe348a84640 1 -- 172.21.15.50:0/3059961419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557307779800 msgr2=0x5573077769c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe348a84640 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557307779800 0x5573077769c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:27.367 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe348a84640 1 -- 172.21.15.50:0/3059961419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55730776e620 msgr2=0x557307779170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe348a84640 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55730776e620 0x557307779170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe348a84640 1 -- 172.21.15.50:0/3059961419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573074b2620 con 0x55730776ef40 2024-04-23T18:38:27.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe348a84640 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x557307776460 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fe33400bca0 tx=0x7fe3340048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e6f24024bc2f52 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe323fff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:38:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe323fff640 1 -- 172.21.15.50:0/3059961419 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33402a060 con 0x55730776ef40 2024-04-23T18:38:27.458 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe323fff640 1 -- 172.21.15.50:0/3059961419 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fe334005010 con 0x55730776ef40 2024-04-23T18:38:27.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe34d9fe740 1 -- 172.21.15.50:0/3059961419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe334004c50 con 0x55730776ef40 2024-04-23T18:38:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe323fff640 1 -- 172.21.15.50:0/3059961419 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fe334005210 con 0x55730776ef40 2024-04-23T18:38:27.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.413+0000 7fe34d9fe740 10 client.17922 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.414+0000 7fe34d9fe740 1 -- 172.21.15.50:0/3059961419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x557307788310 con 0x55730776ef40 2024-04-23T18:38:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.414+0000 7fe323fff640 1 -- 172.21.15.50:0/3059961419 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe334004c50 con 0x55730776ef40 2024-04-23T18:38:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.414+0000 7fe323fff640 1 -- 172.21.15.50:0/3059961419 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fe33401b430 con 0x55730776ef40 2024-04-23T18:38:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.414+0000 7fe34d9fe740 10 client.17922 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.414+0000 7fe34d9fe740 10 client.17922 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:38:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.414+0000 7fe34d9fe740 20 client.17922 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:27.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.414+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557307789710 0x55730778bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:27.724 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.414+0000 7fe34d9fe740 4 client.17922 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:38:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.414+0000 7fe34d9fe740 1 -- 172.21.15.50:0/3059961419 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573076e1390 con 0x557307789710 2024-04-23T18:38:27.804 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.414+0000 7fe343fff640 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557307789710 0x55730778bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.415+0000 7fe343fff640 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557307789710 0x55730778bb00 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4de2981ff759afbd server_cookie=fc8bb8afffe9480f in_seq=0 out_seq=0 2024-04-23T18:38:27.834 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.415+0000 7fe323fff640 10 client.17922 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:27.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe323fff640 1 -- 172.21.15.50:0/3059961419 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5573076e1390 con 0x557307789710 2024-04-23T18:38:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe323fff640 10 client.17922 handle_command_reply: tid=0 2024-04-23T18:38:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe34d9fe740 1 client.17922 shutdown 2024-04-23T18:38:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe34d9fe740 1 -- 172.21.15.50:0/3059961419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 msgr2=0x557307776460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:27.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x557307776460 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fe33400bca0 tx=0x7fe3340048f0 comp rx=0 tx=0).stop 2024-04-23T18:38:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe34d9fe740 1 -- 172.21.15.50:0/3059961419 shutdown_connections 2024-04-23T18:38:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x557307789710 0x55730778bb00 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557307779800 0x5573077769c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x557307776460 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:28.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe323fff640 0 client.17922 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/3059961419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55730776e620 0x557307779170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe34d9fe740 1 -- 172.21.15.50:0/3059961419 >> 172.21.15.50:0/3059961419 conn(0x55730776ab30 msgr2=0x557307770860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe34d9fe740 1 -- 172.21.15.50:0/3059961419 shutdown_connections 2024-04-23T18:38:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe34d9fe740 1 -- 172.21.15.50:0/3059961419 wait complete. 2024-04-23T18:38:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.416+0000 7fe34d9fe740 20 client.17922 trim_cache size 0 max 16384 2024-04-23T18:38:28.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.431+0000 7fe34d9fe740 1 Processor -- start 2024-04-23T18:38:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.431+0000 7fe34d9fe740 1 -- start start 2024-04-23T18:38:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.431+0000 7fe34d9fe740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776e620 0x5573077852f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:28.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.431+0000 7fe34d9fe740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55730776ef40 0x557307778ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:28.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.431+0000 7fe34d9fe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557307779800 0x5573077910d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:28.354 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.431+0000 7fe34d9fe740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe3441b0400 con 0x55730776e620 2024-04-23T18:38:28.364 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.431+0000 7fe34d9fe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe3441adcd0 con 0x557307779800 2024-04-23T18:38:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.431+0000 7fe34d9fe740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe3441abb40 con 0x55730776ef40 2024-04-23T18:38:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe343fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776e620 0x5573077852f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:28.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe343fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776e620 0x5573077852f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60440/0 (socket says 172.21.15.50:60440) 2024-04-23T18:38:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe343fff640 1 -- 172.21.15.50:0/755512337 learned_addr learned my addr 172.21.15.50:0/755512337 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:28.522 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe348a84640 1 --2- 172.21.15.50:0/755512337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55730776ef40 0x557307778ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:28.534 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe343fff640 1 --2- 172.21.15.50:0/755512337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557307779800 0x5573077910d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:28.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe343fff640 1 -- 172.21.15.50:0/755512337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55730776ef40 msgr2=0x557307778ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:28.555 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe343fff640 1 --2- 172.21.15.50:0/755512337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55730776ef40 0x557307778ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe343fff640 1 -- 172.21.15.50:0/755512337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557307779800 msgr2=0x5573077910d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe343fff640 1 --2- 172.21.15.50:0/755512337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557307779800 0x5573077910d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:28.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe343fff640 1 -- 172.21.15.50:0/755512337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55730777bfb0 con 0x55730776e620 2024-04-23T18:38:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe343fff640 1 --2- 172.21.15.50:0/755512337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776e620 0x5573077852f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fe338009ea0 tx=0x7fe338004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f575bf6434fd8e78 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:28.671 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe32a7fc640 1 -- 172.21.15.50:0/755512337 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe338010ac0 con 0x55730776e620 2024-04-23T18:38:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe32a7fc640 1 -- 172.21.15.50:0/755512337 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe338010c60 con 0x55730776e620 2024-04-23T18:38:28.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.432+0000 7fe32a7fc640 1 -- 172.21.15.50:0/755512337 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe338004a00 con 0x55730776e620 2024-04-23T18:38:28.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.433+0000 7fe34d9fe740 1 -- 172.21.15.50:0/755512337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776e620 msgr2=0x5573077852f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:28.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.433+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/755512337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776e620 0x5573077852f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fe338009ea0 tx=0x7fe338004730 comp rx=0 tx=0).stop 2024-04-23T18:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.433+0000 7fe34d9fe740 1 -- 172.21.15.50:0/755512337 shutdown_connections 2024-04-23T18:38:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.433+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/755512337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557307779800 0x5573077910d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:28.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.433+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/755512337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55730776ef40 0x557307778ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:28.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.433+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/755512337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776e620 0x5573077852f0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.433+0000 7fe34d9fe740 1 -- 172.21.15.50:0/755512337 >> 172.21.15.50:0/755512337 conn(0x55730776ab30 msgr2=0x557307769cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:29.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.433+0000 7fe34d9fe740 1 -- 172.21.15.50:0/755512337 shutdown_connections 2024-04-23T18:38:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.433+0000 7fe34d9fe740 1 -- 172.21.15.50:0/755512337 wait complete. 2024-04-23T18:38:29.091 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.433+0000 7fe34d9fe740 1 -- start start 2024-04-23T18:38:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.433+0000 7fe34d9fe740 1 Processor -- start 2024-04-23T18:38:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe34d9fe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55730776e620 0x5573077ec6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe34d9fe740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x5573077ecbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:29.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe34d9fe740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557307779800 0x5573077e6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe34d9fe740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe3441ac190 con 0x55730776ef40 2024-04-23T18:38:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe34d9fe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe3441ad600 con 0x55730776e620 2024-04-23T18:38:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe34d9fe740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe3441ac860 con 0x557307779800 2024-04-23T18:38:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe34d9fe740 10 client.0 _ll_register_callbacks cb 0x7fe33400b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:38:29.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe348a84640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x5573077ecbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe348a84640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x5573077ecbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60450/0 (socket says 172.21.15.50:60450) 2024-04-23T18:38:29.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe348a84640 1 -- 172.21.15.50:0/1178468859 learned_addr learned my addr 172.21.15.50:0/1178468859 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:29.399 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe343fff640 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55730776e620 0x5573077ec6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:29.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe348a84640 1 -- 172.21.15.50:0/1178468859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557307779800 msgr2=0x5573077e6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:29.505 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe348a84640 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557307779800 0x5573077e6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe348a84640 1 -- 172.21.15.50:0/1178468859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55730776e620 msgr2=0x5573077ec6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe348a84640 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55730776e620 0x5573077ec6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe348a84640 1 -- 172.21.15.50:0/1178468859 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573077e71a0 con 0x55730776ef40 2024-04-23T18:38:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe343fff640 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55730776e620 0x5573077ec6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:38:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe348a84640 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x5573077ecbe0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fe334005d00 tx=0x7fe334000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4daeee22f86e1566 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:29.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe323fff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:38:29.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe323fff640 1 -- 172.21.15.50:0/1178468859 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33402a040 con 0x55730776ef40 2024-04-23T18:38:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe323fff640 1 -- 172.21.15.50:0/1178468859 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fe3340055c0 con 0x55730776ef40 2024-04-23T18:38:29.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.434+0000 7fe34d9fe740 1 -- 172.21.15.50:0/1178468859 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe338010c60 con 0x55730776ef40 2024-04-23T18:38:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.435+0000 7fe323fff640 1 -- 172.21.15.50:0/1178468859 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fe33403e050 con 0x55730776ef40 2024-04-23T18:38:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.435+0000 7fe34d9fe740 10 client.17928 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:30.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.435+0000 7fe34d9fe740 1 -- 172.21.15.50:0/1178468859 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5573077f3eb0 con 0x55730776ef40 2024-04-23T18:38:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.435+0000 7fe323fff640 1 -- 172.21.15.50:0/1178468859 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe338010c60 con 0x55730776ef40 2024-04-23T18:38:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.435+0000 7fe323fff640 1 -- 172.21.15.50:0/1178468859 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fe3340118f0 con 0x55730776ef40 2024-04-23T18:38:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.435+0000 7fe34d9fe740 10 client.17928 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:30.127 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.435+0000 7fe34d9fe740 10 client.17928 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:38:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.435+0000 7fe34d9fe740 20 client.17928 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:30.189 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.435+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5573077f4660 0x5573077f6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:30.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.435+0000 7fe34d9fe740 4 client.17928 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:38:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.435+0000 7fe34d9fe740 1 -- 172.21.15.50:0/1178468859 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5573076db200 con 0x5573077f4660 2024-04-23T18:38:30.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.435+0000 7fe343fff640 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5573077f4660 0x5573077f6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:30.283 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.436+0000 7fe323fff640 10 client.17928 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.436+0000 7fe343fff640 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5573077f4660 0x5573077f6ac0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ba902373afdcb0a server_cookie=b1b332ca10cdd68 in_seq=0 out_seq=0 2024-04-23T18:38:30.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:22 smithi050 ceph-mon[29187]: pgmap v632: 97 pgs: 97 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 628 op/s 2024-04-23T18:38:30.315 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.90 MB/sec execute 296 sec latency 4380.262 ms 2024-04-23T18:38:30.324 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:26 smithi188 ceph-mon[32694]: pgmap v634: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 12 MiB/s wr, 482 op/s 2024-04-23T18:38:30.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:28 smithi188 ceph-mon[32694]: pgmap v635: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 12 MiB/s wr, 552 op/s 2024-04-23T18:38:30.424 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:26 smithi177 ceph-mon[32308]: pgmap v634: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 12 MiB/s wr, 482 op/s 2024-04-23T18:38:30.435 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:28 smithi177 ceph-mon[32308]: pgmap v635: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 12 MiB/s wr, 552 op/s 2024-04-23T18:38:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe323fff640 1 -- 172.21.15.50:0/1178468859 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5573076db200 con 0x5573077f4660 2024-04-23T18:38:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe323fff640 10 client.17928 handle_command_reply: tid=0 2024-04-23T18:38:30.510 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe34d9fe740 1 client.17928 shutdown 2024-04-23T18:38:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe34d9fe740 1 -- 172.21.15.50:0/1178468859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 msgr2=0x5573077ecbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x5573077ecbe0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fe334005d00 tx=0x7fe334000bc0 comp rx=0 tx=0).stop 2024-04-23T18:38:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe34d9fe740 1 -- 172.21.15.50:0/1178468859 shutdown_connections 2024-04-23T18:38:30.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5573077f4660 0x5573077f6ac0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:30.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x557307779800 0x5573077e6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55730776ef40 0x5573077ecbe0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe323fff640 0 client.17928 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe34d9fe740 1 --2- 172.21.15.50:0/1178468859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55730776e620 0x5573077ec6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe34d9fe740 1 -- 172.21.15.50:0/1178468859 >> 172.21.15.50:0/1178468859 conn(0x55730776ab30 msgr2=0x5573077842e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:30.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe34d9fe740 1 -- 172.21.15.50:0/1178468859 shutdown_connections 2024-04-23T18:38:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe34d9fe740 1 -- 172.21.15.50:0/1178468859 wait complete. 2024-04-23T18:38:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.660+0000 7fe34d9fe740 20 client.17928 trim_cache size 0 max 16384 2024-04-23T18:38:30.785 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:38:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:38:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:38:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:38:30.855 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.661+0000 7fe34ace8640 1 -- 172.21.15.50:0/1220653224 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe32c0a3dc0 msgr2=0x7fe32c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:30.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.661+0000 7fe34ace8640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe32c0a3dc0 0x7fe32c0a6280 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0b99c0 tx=0x7fe33c004140 comp rx=0 tx=0).stop 2024-04-23T18:38:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.661+0000 7fe34ace8640 1 -- 172.21.15.50:0/1220653224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34410c5d0 msgr2=0x7fe34419cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.661+0000 7fe34ace8640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34410c5d0 0x7fe34419cee0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fe33c004af0 tx=0x7fe33c02f5d0 comp rx=0 tx=0).stop 2024-04-23T18:38:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.661+0000 7fe34ace8640 1 -- 172.21.15.50:0/1220653224 shutdown_connections 2024-04-23T18:38:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.661+0000 7fe34ace8640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe32c0a3dc0 0x7fe32c0a6280 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.661+0000 7fe34ace8640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34410c5d0 0x7fe34419cee0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.661+0000 7fe34ace8640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344103b20 0x7fe344198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:31.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.661+0000 7fe34ace8640 1 --2- 172.21.15.50:0/1220653224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe344103200 0x7fe344198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.661+0000 7fe34ace8640 1 -- 172.21.15.50:0/1220653224 >> 172.21.15.50:0/1220653224 conn(0x7fe3440771d0 msgr2=0x7fe3440fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.661+0000 7fe34ace8640 1 -- 172.21.15.50:0/1220653224 shutdown_connections 2024-04-23T18:38:31.212 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:22.661+0000 7fe34ace8640 1 -- 172.21.15.50:0/1220653224 wait complete. 2024-04-23T18:38:31.272 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.83 MB/sec execute 297 sec latency 5380.417 ms 2024-04-23T18:38:31.283 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.76 MB/sec execute 298 sec latency 6380.513 ms 2024-04-23T18:38:31.294 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:30 smithi188 ceph-mon[32694]: pgmap v636: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 7.8 MiB/s wr, 379 op/s 2024-04-23T18:38:31.334 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:30 smithi177 ceph-mon[32308]: pgmap v636: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 7.8 MiB/s wr, 379 op/s 2024-04-23T18:38:31.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:24 smithi050 ceph-mon[29187]: pgmap v633: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 12 MiB/s wr, 482 op/s 2024-04-23T18:38:31.364 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.69 MB/sec execute 299 sec latency 7380.597 ms 2024-04-23T18:38:31.378 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.62 MB/sec execute 300 sec latency 8380.723 ms 2024-04-23T18:38:31.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:26 smithi050 ceph-mon[29187]: pgmap v634: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 12 MiB/s wr, 482 op/s 2024-04-23T18:38:31.443 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.55 MB/sec execute 301 sec latency 9380.848 ms 2024-04-23T18:38:31.499 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.49 MB/sec execute 302 sec latency 10380.951 ms 2024-04-23T18:38:31.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:28 smithi050 ceph-mon[29187]: pgmap v635: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 12 MiB/s wr, 552 op/s 2024-04-23T18:38:31.595 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.42 MB/sec execute 303 sec latency 11381.043 ms 2024-04-23T18:38:31.609 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.35 MB/sec execute 304 sec latency 12381.118 ms 2024-04-23T18:38:31.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:30 smithi050 ceph-mon[29187]: pgmap v636: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 7.8 MiB/s wr, 379 op/s 2024-04-23T18:38:31.635 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.28 MB/sec execute 305 sec latency 13381.234 ms 2024-04-23T18:38:32.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:32 smithi177 ceph-mon[32308]: pgmap v637: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.9 MiB/s wr, 305 op/s 2024-04-23T18:38:32.582 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.22 MB/sec execute 306 sec latency 14381.364 ms 2024-04-23T18:38:32.647 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:32 smithi188 ceph-mon[32694]: pgmap v637: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.9 MiB/s wr, 305 op/s 2024-04-23T18:38:32.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:32 smithi050 ceph-mon[29187]: pgmap v637: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.9 MiB/s wr, 305 op/s 2024-04-23T18:38:33.580 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.15 MB/sec execute 307 sec latency 15381.479 ms 2024-04-23T18:38:34.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:34 smithi177 ceph-mon[32308]: pgmap v638: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 358 op/s 2024-04-23T18:38:34.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:34 smithi188 ceph-mon[32694]: pgmap v638: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 358 op/s 2024-04-23T18:38:34.587 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.09 MB/sec execute 308 sec latency 16381.573 ms 2024-04-23T18:38:34.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:34 smithi050 ceph-mon[29187]: pgmap v638: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 358 op/s 2024-04-23T18:38:35.580 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 20.02 MB/sec execute 309 sec latency 17381.641 ms 2024-04-23T18:38:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.761+0000 7fec27057640 1 -- 172.21.15.50:0/1779268135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec20101a90 msgr2=0x7fec20105ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.761+0000 7fec27057640 1 --2- 172.21.15.50:0/1779268135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec20101a90 0x7fec20105ad0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fec10009900 tx=0x7fec1002a410 comp rx=0 tx=0).stop 2024-04-23T18:38:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.762+0000 7fec27057640 1 -- 172.21.15.50:0/1779268135 shutdown_connections 2024-04-23T18:38:35.783 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.762+0000 7fec27057640 1 --2- 172.21.15.50:0/1779268135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec201061b0 0x7fec2010dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.762+0000 7fec27057640 1 --2- 172.21.15.50:0/1779268135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec20101a90 0x7fec20105ad0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.762+0000 7fec27057640 1 --2- 172.21.15.50:0/1779268135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec201010e0 0x7fec201014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.762+0000 7fec27057640 1 -- 172.21.15.50:0/1779268135 >> 172.21.15.50:0/1779268135 conn(0x7fec200771d0 msgr2=0x7fec200775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.762+0000 7fec27057640 1 -- 172.21.15.50:0/1779268135 shutdown_connections 2024-04-23T18:38:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.762+0000 7fec27057640 1 -- 172.21.15.50:0/1779268135 wait complete. 2024-04-23T18:38:35.988 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.762+0000 7fec27057640 1 Processor -- start 2024-04-23T18:38:35.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.762+0000 7fec27057640 1 -- start start 2024-04-23T18:38:36.007 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.762+0000 7fec27057640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec201010e0 0x7fec20198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.762+0000 7fec27057640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec201061b0 0x7fec20198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec27057640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec2019cf20 0x7fec2019d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec27057640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fec201110f0 con 0x7fec201010e0 2024-04-23T18:38:36.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec27057640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fec20110f70 con 0x7fec201061b0 2024-04-23T18:38:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec27057640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fec20111270 con 0x7fec2019cf20 2024-04-23T18:38:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec24df3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec201010e0 0x7fec20198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec24df3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec201010e0 0x7fec20198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48258/0 (socket says 172.21.15.50:48258) 2024-04-23T18:38:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec24df3640 1 -- 172.21.15.50:0/2375916494 learned_addr learned my addr 172.21.15.50:0/2375916494 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec17fff640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec201061b0 0x7fec20198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec255f4640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec2019cf20 0x7fec2019d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:36.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec24df3640 1 -- 172.21.15.50:0/2375916494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec2019cf20 msgr2=0x7fec2019d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:36.319 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec24df3640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec2019cf20 0x7fec2019d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec24df3640 1 -- 172.21.15.50:0/2375916494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec201061b0 msgr2=0x7fec20198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec24df3640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec201061b0 0x7fec20198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec24df3640 1 -- 172.21.15.50:0/2375916494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec2019da10 con 0x7fec201010e0 2024-04-23T18:38:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec24df3640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec201010e0 0x7fec20198660 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00e9f0 tx=0x7fec1c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec15ffb640 1 -- 172.21.15.50:0/2375916494 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1c00cd80 con 0x7fec201010e0 2024-04-23T18:38:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec15ffb640 1 -- 172.21.15.50:0/2375916494 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec1c0054d0 con 0x7fec201010e0 2024-04-23T18:38:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec15ffb640 1 -- 172.21.15.50:0/2375916494 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1c010690 con 0x7fec201010e0 2024-04-23T18:38:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.763+0000 7fec27057640 1 -- 172.21.15.50:0/2375916494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec2019dd00 con 0x7fec201010e0 2024-04-23T18:38:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.764+0000 7fec27057640 1 -- 172.21.15.50:0/2375916494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec201a5470 con 0x7fec201010e0 2024-04-23T18:38:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.766+0000 7fec15ffb640 1 -- 172.21.15.50:0/2375916494 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fec1c004e90 con 0x7fec201010e0 2024-04-23T18:38:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.766+0000 7fec29d68740 1 Processor -- start 2024-04-23T18:38:36.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.766+0000 7fec29d68740 1 -- start start 2024-04-23T18:38:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec29d68740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce360920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec29d68740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce360e60 0x5565ce36f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec29d68740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce36fc10 0x5565ce372000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec29d68740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5565ce2c5970 con 0x5565ce36fc10 2024-04-23T18:38:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec29d68740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5565ce2624a0 con 0x5565ce360e60 2024-04-23T18:38:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec29d68740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5565ce2c1930 con 0x5565ce360540 2024-04-23T18:38:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec15ffb640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fec080a3dc0 0x7fec080a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec17fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce36fc10 0x5565ce372000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec17fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce36fc10 0x5565ce372000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48272/0 (socket says 172.21.15.50:48272) 2024-04-23T18:38:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec17fff640 1 -- 172.21.15.50:0/2297856213 learned_addr learned my addr 172.21.15.50:0/2297856213 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec24df3640 1 --2- 172.21.15.50:0/2297856213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce360e60 0x5565ce36f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec255f4640 1 --2- 172.21.15.50:0/2297856213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce360920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec255f4640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fec080a3dc0 0x7fec080a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec15ffb640 1 -- 172.21.15.50:0/2375916494 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fec1c0d0d20 con 0x7fec201010e0 2024-04-23T18:38:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec17fff640 1 -- 172.21.15.50:0/2297856213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 msgr2=0x5565ce360920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:36.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec17fff640 1 --2- 172.21.15.50:0/2297856213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce360920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:36.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec17fff640 1 -- 172.21.15.50:0/2297856213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce360e60 msgr2=0x5565ce36f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:36.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec17fff640 1 --2- 172.21.15.50:0/2297856213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce360e60 0x5565ce36f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec17fff640 1 -- 172.21.15.50:0/2297856213 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565cdfd8210 con 0x5565ce36fc10 2024-04-23T18:38:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.767+0000 7fec17fff640 1 --2- 172.21.15.50:0/2297856213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce36fc10 0x5565ce372000 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fec10009900 tx=0x7fec10032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b391153525d7caf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.768+0000 7fec255f4640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fec080a3dc0 0x7fec080a6280 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fec18009fd0 tx=0x7fec18005bd0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:37.003 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.768+0000 7fec06ffd640 1 -- 172.21.15.50:0/2297856213 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec10040070 con 0x5565ce36fc10 2024-04-23T18:38:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.768+0000 7fec06ffd640 1 -- 172.21.15.50:0/2297856213 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec10004030 con 0x5565ce36fc10 2024-04-23T18:38:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.768+0000 7fec06ffd640 1 -- 172.21.15.50:0/2297856213 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec10035040 con 0x5565ce36fc10 2024-04-23T18:38:37.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.768+0000 7fec29d68740 1 -- 172.21.15.50:0/2297856213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce36fc10 msgr2=0x5565ce372000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.768+0000 7fec29d68740 1 --2- 172.21.15.50:0/2297856213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce36fc10 0x5565ce372000 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fec10009900 tx=0x7fec10032b20 comp rx=0 tx=0).stop 2024-04-23T18:38:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.768+0000 7fec29d68740 1 -- 172.21.15.50:0/2297856213 shutdown_connections 2024-04-23T18:38:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.768+0000 7fec29d68740 1 --2- 172.21.15.50:0/2297856213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce36fc10 0x5565ce372000 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.768+0000 7fec29d68740 1 --2- 172.21.15.50:0/2297856213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce360e60 0x5565ce36f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.768+0000 7fec29d68740 1 --2- 172.21.15.50:0/2297856213 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce360920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:37.223 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.768+0000 7fec29d68740 1 -- 172.21.15.50:0/2297856213 >> 172.21.15.50:0/2297856213 conn(0x5565ce35cb30 msgr2=0x5565ce35cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.768+0000 7fec29d68740 1 -- 172.21.15.50:0/2297856213 shutdown_connections 2024-04-23T18:38:37.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.769+0000 7fec29d68740 1 -- 172.21.15.50:0/2297856213 wait complete. 2024-04-23T18:38:37.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.769+0000 7fec29d68740 1 -- start start 2024-04-23T18:38:37.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.769+0000 7fec29d68740 1 Processor -- start 2024-04-23T18:38:37.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.769+0000 7fec29d68740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce36f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:37.324 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.769+0000 7fec29d68740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 0x5565ce36c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.769+0000 7fec29d68740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 0x5565ce36cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:37.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.769+0000 7fec29d68740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5565ce3727e0 con 0x5565ce360e60 2024-04-23T18:38:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.769+0000 7fec29d68740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5565ce2600b0 con 0x5565ce36fc10 2024-04-23T18:38:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.769+0000 7fec29d68740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5565ce372960 con 0x5565ce360540 2024-04-23T18:38:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.769+0000 7fec29d68740 10 client.0 _ll_register_callbacks cb 0x5565ce24dd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:38:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec17fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 0x5565ce36c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec17fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 0x5565ce36c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48284/0 (socket says 172.21.15.50:48284) 2024-04-23T18:38:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec17fff640 1 -- 172.21.15.50:0/3491101200 learned_addr learned my addr 172.21.15.50:0/3491101200 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec255f4640 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 0x5565ce36cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec24df3640 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce36f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec17fff640 1 -- 172.21.15.50:0/3491101200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 msgr2=0x5565ce36f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:37.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec17fff640 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce36f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:37.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec17fff640 1 -- 172.21.15.50:0/3491101200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 msgr2=0x5565ce36cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec17fff640 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 0x5565ce36cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:37.613 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec17fff640 1 -- 172.21.15.50:0/3491101200 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565ce0a4620 con 0x5565ce360e60 2024-04-23T18:38:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec17fff640 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 0x5565ce36c8e0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fec10000c00 tx=0x7fec10009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b3773065de9e75 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:37.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7febfffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:38:37.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7febfffff640 1 -- 172.21.15.50:0/3491101200 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec10040070 con 0x5565ce360e60 2024-04-23T18:38:37.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7febfffff640 1 -- 172.21.15.50:0/3491101200 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fec10005280 con 0x5565ce360e60 2024-04-23T18:38:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec29d68740 1 -- 172.21.15.50:0/3491101200 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec10004030 con 0x5565ce360e60 2024-04-23T18:38:37.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7febfffff640 1 -- 172.21.15.50:0/3491101200 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fec10056050 con 0x5565ce360e60 2024-04-23T18:38:37.793 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec29d68740 10 client.17940 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:37.803 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.770+0000 7fec29d68740 1 -- 172.21.15.50:0/3491101200 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5565ce37e790 con 0x5565ce360e60 2024-04-23T18:38:37.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.771+0000 7febfffff640 1 -- 172.21.15.50:0/3491101200 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec10004030 con 0x5565ce360e60 2024-04-23T18:38:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.771+0000 7febfffff640 1 -- 172.21.15.50:0/3491101200 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fec10004b70 con 0x5565ce360e60 2024-04-23T18:38:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.771+0000 7fec29d68740 10 client.17940 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:37.835 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.771+0000 7fec29d68740 10 client.17940 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:38:37.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.771+0000 7fec29d68740 20 client.17940 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.771+0000 7fec29d68740 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5565ce37fae0 0x5565ce381ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.771+0000 7fec29d68740 4 client.17940 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:38:37.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.771+0000 7fec29d68740 1 -- 172.21.15.50:0/3491101200 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565ce2d3390 con 0x5565ce37fae0 2024-04-23T18:38:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.771+0000 7fec24df3640 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5565ce37fae0 0x5565ce381ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.772+0000 7febfffff640 10 client.17940 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.772+0000 7fec24df3640 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5565ce37fae0 0x5565ce381ed0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2430ca966dc5f9c7 server_cookie=dfc607df05491b22 in_seq=0 out_seq=0 2024-04-23T18:38:38.025 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.772+0000 7febfffff640 1 -- 172.21.15.50:0/3491101200 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565ce2d3390 con 0x5565ce37fae0 2024-04-23T18:38:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.772+0000 7febfffff640 10 client.17940 handle_command_reply: tid=0 2024-04-23T18:38:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.772+0000 7fec29d68740 1 client.17940 shutdown 2024-04-23T18:38:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.772+0000 7fec29d68740 1 -- 172.21.15.50:0/3491101200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 msgr2=0x5565ce36c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.772+0000 7fec29d68740 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 0x5565ce36c8e0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fec10000c00 tx=0x7fec10009580 comp rx=0 tx=0).stop 2024-04-23T18:38:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.773+0000 7fec29d68740 1 -- 172.21.15.50:0/3491101200 shutdown_connections 2024-04-23T18:38:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.773+0000 7fec29d68740 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5565ce37fae0 0x5565ce381ed0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.773+0000 7fec29d68740 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 0x5565ce36cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.773+0000 7fec29d68740 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 0x5565ce36c8e0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:38.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.773+0000 7febfffff640 0 client.17940 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.773+0000 7fec29d68740 1 --2- 172.21.15.50:0/3491101200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce36f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.773+0000 7fec29d68740 1 -- 172.21.15.50:0/3491101200 >> 172.21.15.50:0/3491101200 conn(0x5565ce35cb30 msgr2=0x5565ce3629a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:38.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.773+0000 7fec29d68740 1 -- 172.21.15.50:0/3491101200 shutdown_connections 2024-04-23T18:38:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.773+0000 7fec29d68740 1 -- 172.21.15.50:0/3491101200 wait complete. 2024-04-23T18:38:38.282 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.773+0000 7fec29d68740 20 client.17940 trim_cache size 0 max 16384 2024-04-23T18:38:38.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.788+0000 7fec29d68740 1 Processor -- start 2024-04-23T18:38:38.300 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.788+0000 7fec29d68740 1 -- start start 2024-04-23T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.788+0000 7fec29d68740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce36ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:38.384 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.788+0000 7fec29d68740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 0x5565ce367d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.788+0000 7fec29d68740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 0x5565ce36f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:38.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.788+0000 7fec29d68740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fec201ae180 con 0x5565ce360e60 2024-04-23T18:38:38.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.788+0000 7fec29d68740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fec201aba50 con 0x5565ce36fc10 2024-04-23T18:38:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.788+0000 7fec29d68740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fec201a98c0 con 0x5565ce360540 2024-04-23T18:38:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec17fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 0x5565ce367d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:38.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec17fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 0x5565ce367d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48286/0 (socket says 172.21.15.50:48286) 2024-04-23T18:38:38.514 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec17fff640 1 -- 172.21.15.50:0/3238051438 learned_addr learned my addr 172.21.15.50:0/3238051438 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:38.526 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec255f4640 1 --2- 172.21.15.50:0/3238051438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 0x5565ce36f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec24df3640 1 --2- 172.21.15.50:0/3238051438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce36ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec17fff640 1 -- 172.21.15.50:0/3238051438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 msgr2=0x5565ce36ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec17fff640 1 --2- 172.21.15.50:0/3238051438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce36ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:38.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec17fff640 1 -- 172.21.15.50:0/3238051438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 msgr2=0x5565ce36f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec17fff640 1 --2- 172.21.15.50:0/3238051438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 0x5565ce36f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec17fff640 1 -- 172.21.15.50:0/3238051438 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565ce36d260 con 0x5565ce360e60 2024-04-23T18:38:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec24df3640 1 --2- 172.21.15.50:0/3238051438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce36ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:38:38.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec17fff640 1 --2- 172.21.15.50:0/3238051438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 0x5565ce367d80 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fec10009900 tx=0x7fec10031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97f2e84339511dc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:38.804 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec067fc640 1 -- 172.21.15.50:0/3238051438 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1005f070 con 0x5565ce360e60 2024-04-23T18:38:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec067fc640 1 -- 172.21.15.50:0/3238051438 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec1004d070 con 0x5565ce360e60 2024-04-23T18:38:38.865 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.789+0000 7fec067fc640 1 -- 172.21.15.50:0/3238051438 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec10040070 con 0x5565ce360e60 2024-04-23T18:38:38.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.790+0000 7fec29d68740 1 -- 172.21.15.50:0/3238051438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 msgr2=0x5565ce367d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.790+0000 7fec29d68740 1 --2- 172.21.15.50:0/3238051438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 0x5565ce367d80 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fec10009900 tx=0x7fec10031400 comp rx=0 tx=0).stop 2024-04-23T18:38:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.790+0000 7fec29d68740 1 -- 172.21.15.50:0/3238051438 shutdown_connections 2024-04-23T18:38:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.790+0000 7fec29d68740 1 --2- 172.21.15.50:0/3238051438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 0x5565ce36f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.790+0000 7fec29d68740 1 --2- 172.21.15.50:0/3238051438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360e60 0x5565ce367d80 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.790+0000 7fec29d68740 1 --2- 172.21.15.50:0/3238051438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360540 0x5565ce36ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.790+0000 7fec29d68740 1 -- 172.21.15.50:0/3238051438 >> 172.21.15.50:0/3238051438 conn(0x5565ce35cb30 msgr2=0x5565ce367670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:38.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.790+0000 7fec29d68740 1 -- 172.21.15.50:0/3238051438 shutdown_connections 2024-04-23T18:38:39.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.790+0000 7fec29d68740 1 -- 172.21.15.50:0/3238051438 wait complete. 2024-04-23T18:38:39.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.790+0000 7fec29d68740 1 -- start start 2024-04-23T18:38:39.087 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.790+0000 7fec29d68740 1 Processor -- start 2024-04-23T18:38:39.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec29d68740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360540 0x5565ce384bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:39.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec29d68740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360e60 0x5565ce385110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec29d68740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 0x5565ce3820b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:39.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec29d68740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fec201a9f10 con 0x5565ce360540 2024-04-23T18:38:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec29d68740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fec201ab380 con 0x5565ce36fc10 2024-04-23T18:38:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec29d68740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fec201aa5e0 con 0x5565ce360e60 2024-04-23T18:38:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec29d68740 10 client.0 _ll_register_callbacks cb 0x7fec10005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:38:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec24df3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360540 0x5565ce384bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec24df3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360540 0x5565ce384bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48294/0 (socket says 172.21.15.50:48294) 2024-04-23T18:38:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec24df3640 1 -- 172.21.15.50:0/1776537407 learned_addr learned my addr 172.21.15.50:0/1776537407 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec255f4640 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 0x5565ce3820b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec17fff640 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360e60 0x5565ce385110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec24df3640 1 -- 172.21.15.50:0/1776537407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360e60 msgr2=0x5565ce385110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec24df3640 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360e60 0x5565ce385110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec24df3640 1 -- 172.21.15.50:0/1776537407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 msgr2=0x5565ce3820b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec24df3640 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 0x5565ce3820b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec24df3640 1 -- 172.21.15.50:0/1776537407 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565ce382840 con 0x5565ce360540 2024-04-23T18:38:39.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7fec24df3640 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360540 0x5565ce384bd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fec1c099620 tx=0x7fec1c098f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2cdb07ef0f2516 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:39.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7febfffff640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:38:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7febfffff640 1 -- 172.21.15.50:0/1776537407 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1c0b61b0 con 0x5565ce360540 2024-04-23T18:38:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.791+0000 7febfffff640 1 -- 172.21.15.50:0/1776537407 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fec1c0046c0 con 0x5565ce360540 2024-04-23T18:38:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7fec29d68740 1 -- 172.21.15.50:0/1776537407 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec1004d070 con 0x5565ce360540 2024-04-23T18:38:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7febfffff640 1 -- 172.21.15.50:0/1776537407 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fec1c09cc50 con 0x5565ce360540 2024-04-23T18:38:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7fec29d68740 10 client.17946 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7fec29d68740 1 -- 172.21.15.50:0/1776537407 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5565ce3e56c0 con 0x5565ce360540 2024-04-23T18:38:39.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7febfffff640 1 -- 172.21.15.50:0/1776537407 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec1004d070 con 0x5565ce360540 2024-04-23T18:38:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7febfffff640 1 -- 172.21.15.50:0/1776537407 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fec1c0ce680 con 0x5565ce360540 2024-04-23T18:38:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7fec29d68740 10 client.17946 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7fec29d68740 10 client.17946 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:38:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7fec29d68740 20 client.17946 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7fec29d68740 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5565ce3e5d90 0x5565ce3e8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7fec29d68740 4 client.17946 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:38:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7fec29d68740 1 -- 172.21.15.50:0/1776537407 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5565ce2cd200 con 0x5565ce3e5d90 2024-04-23T18:38:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.792+0000 7fec17fff640 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5565ce3e5d90 0x5565ce3e8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.793+0000 7fec17fff640 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5565ce3e5d90 0x5565ce3e8180 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=713e929dc11ef814 server_cookie=399757e897bc2114 in_seq=0 out_seq=0 2024-04-23T18:38:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.793+0000 7febfffff640 10 client.17946 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:39.772 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.858+0000 7febfffff640 1 -- 172.21.15.50:0/1776537407 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5565ce2cd200 con 0x5565ce3e5d90 2024-04-23T18:38:39.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.858+0000 7febfffff640 10 client.17946 handle_command_reply: tid=0 2024-04-23T18:38:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.858+0000 7fec29d68740 1 client.17946 shutdown 2024-04-23T18:38:39.842 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.859+0000 7fec29d68740 1 -- 172.21.15.50:0/1776537407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360540 msgr2=0x5565ce384bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.859+0000 7fec29d68740 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360540 0x5565ce384bd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fec1c099620 tx=0x7fec1c098f30 comp rx=0 tx=0).stop 2024-04-23T18:38:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.859+0000 7fec29d68740 1 -- 172.21.15.50:0/1776537407 shutdown_connections 2024-04-23T18:38:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.859+0000 7fec29d68740 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5565ce3e5d90 0x5565ce3e8180 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:39.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.859+0000 7fec29d68740 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565ce36fc10 0x5565ce3820b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:39.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.859+0000 7fec29d68740 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5565ce360e60 0x5565ce385110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.859+0000 7fec29d68740 1 --2- 172.21.15.50:0/1776537407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5565ce360540 0x5565ce384bd0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:40.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.859+0000 7fec29d68740 1 -- 172.21.15.50:0/1776537407 >> 172.21.15.50:0/1776537407 conn(0x5565ce35cb30 msgr2=0x5565ce36b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.859+0000 7febfffff640 0 client.17946 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.859+0000 7fec29d68740 1 -- 172.21.15.50:0/1776537407 shutdown_connections 2024-04-23T18:38:40.723 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.859+0000 7fec29d68740 1 -- 172.21.15.50:0/1776537407 wait complete. 2024-04-23T18:38:40.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.860+0000 7fec29d68740 20 client.17946 trim_cache size 0 max 16384 2024-04-23T18:38:40.775 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:38:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:38:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:38:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:38:40.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:36 smithi188 ceph-mon[32694]: pgmap v639: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 3.2 MiB/s wr, 191 op/s 2024-04-23T18:38:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:38 smithi188 ceph-mon[32694]: pgmap v640: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 3.9 MiB/s wr, 229 op/s 2024-04-23T18:38:40.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:40 smithi188 ceph-mon[32694]: pgmap v641: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 3.0 MiB/s wr, 166 op/s 2024-04-23T18:38:40.929 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:36 smithi177 ceph-mon[32308]: pgmap v639: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 3.2 MiB/s wr, 191 op/s 2024-04-23T18:38:40.937 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:38 smithi177 ceph-mon[32308]: pgmap v640: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 3.9 MiB/s wr, 229 op/s 2024-04-23T18:38:40.945 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:40 smithi177 ceph-mon[32308]: pgmap v641: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 3.0 MiB/s wr, 166 op/s 2024-04-23T18:38:40.955 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.860+0000 7fec27057640 1 -- 172.21.15.50:0/2375916494 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fec080a3dc0 msgr2=0x7fec080a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:40.963 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.860+0000 7fec27057640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fec080a3dc0 0x7fec080a6280 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fec18009fd0 tx=0x7fec18005bd0 comp rx=0 tx=0).stop 2024-04-23T18:38:40.971 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.860+0000 7fec27057640 1 -- 172.21.15.50:0/2375916494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec201010e0 msgr2=0x7fec20198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:41.028 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.860+0000 7fec27057640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec201010e0 0x7fec20198660 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00e9f0 tx=0x7fec1c00edf0 comp rx=0 tx=0).stop 2024-04-23T18:38:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.861+0000 7fec27057640 1 -- 172.21.15.50:0/2375916494 shutdown_connections 2024-04-23T18:38:41.078 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.861+0000 7fec27057640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fec080a3dc0 0x7fec080a6280 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.861+0000 7fec27057640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fec2019cf20 0x7fec2019d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:41.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.861+0000 7fec27057640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec201061b0 0x7fec20198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.861+0000 7fec27057640 1 --2- 172.21.15.50:0/2375916494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec201010e0 0x7fec20198660 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.861+0000 7fec27057640 1 -- 172.21.15.50:0/2375916494 >> 172.21.15.50:0/2375916494 conn(0x7fec200771d0 msgr2=0x7fec2006aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:41.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.861+0000 7fec27057640 1 -- 172.21.15.50:0/2375916494 shutdown_connections 2024-04-23T18:38:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:35.861+0000 7fec27057640 1 -- 172.21.15.50:0/2375916494 wait complete. 2024-04-23T18:38:41.156 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.96 MB/sec execute 310 sec latency 18381.681 ms 2024-04-23T18:38:41.207 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:36 smithi050 ceph-mon[29187]: pgmap v639: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 3.2 MiB/s wr, 191 op/s 2024-04-23T18:38:41.218 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.89 MB/sec execute 311 sec latency 19381.774 ms 2024-04-23T18:38:41.226 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.83 MB/sec execute 312 sec latency 20381.876 ms 2024-04-23T18:38:41.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:38 smithi050 ceph-mon[29187]: pgmap v640: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 3.9 MiB/s wr, 229 op/s 2024-04-23T18:38:41.244 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.77 MB/sec execute 313 sec latency 21381.994 ms 2024-04-23T18:38:41.252 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.70 MB/sec execute 314 sec latency 22382.044 ms 2024-04-23T18:38:41.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:40 smithi050 ceph-mon[29187]: pgmap v641: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 3.0 MiB/s wr, 166 op/s 2024-04-23T18:38:41.580 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.64 MB/sec execute 315 sec latency 23382.083 ms 2024-04-23T18:38:42.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:42 smithi177 ceph-mon[32308]: pgmap v642: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 2.2 MiB/s wr, 114 op/s 2024-04-23T18:38:42.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:42 smithi188 ceph-mon[32694]: pgmap v642: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 2.2 MiB/s wr, 114 op/s 2024-04-23T18:38:42.586 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.58 MB/sec execute 316 sec latency 24382.163 ms 2024-04-23T18:38:42.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:42 smithi050 ceph-mon[29187]: pgmap v642: 97 pgs: 97 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 2.2 MiB/s wr, 114 op/s 2024-04-23T18:38:43.581 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.52 MB/sec execute 317 sec latency 25382.305 ms 2024-04-23T18:38:44.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:44 smithi177 ceph-mon[32308]: pgmap v643: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 4.0 MiB/s wr, 162 op/s 2024-04-23T18:38:44.595 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.45 MB/sec execute 318 sec latency 26382.354 ms 2024-04-23T18:38:44.621 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:44 smithi188 ceph-mon[32694]: pgmap v643: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 4.0 MiB/s wr, 162 op/s 2024-04-23T18:38:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:44 smithi050 ceph-mon[29187]: pgmap v643: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 4.0 MiB/s wr, 162 op/s 2024-04-23T18:38:45.581 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.39 MB/sec execute 319 sec latency 27382.416 ms 2024-04-23T18:38:46.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:46 smithi177 ceph-mon[32308]: pgmap v644: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 2.8 MiB/s wr, 109 op/s 2024-04-23T18:38:46.623 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.33 MB/sec execute 320 sec latency 28382.456 ms 2024-04-23T18:38:46.631 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:46 smithi188 ceph-mon[32694]: pgmap v644: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 2.8 MiB/s wr, 109 op/s 2024-04-23T18:38:46.715 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:46 smithi050 ceph-mon[29187]: pgmap v644: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 2.8 MiB/s wr, 109 op/s 2024-04-23T18:38:47.581 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.27 MB/sec execute 321 sec latency 29382.592 ms 2024-04-23T18:38:48.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:48 smithi177 ceph-mon[32308]: pgmap v645: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 3.9 MiB/s wr, 151 op/s 2024-04-23T18:38:48.588 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.21 MB/sec execute 322 sec latency 30382.736 ms 2024-04-23T18:38:48.653 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:48 smithi188 ceph-mon[32694]: pgmap v645: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 3.9 MiB/s wr, 151 op/s 2024-04-23T18:38:48.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:48 smithi050 ceph-mon[29187]: pgmap v645: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 3.9 MiB/s wr, 151 op/s 2024-04-23T18:38:48.960 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.966+0000 7fc11067c640 1 -- 172.21.15.50:0/3662150208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc108101a60 msgr2=0x7fc108105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:48.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.966+0000 7fc11067c640 1 --2- 172.21.15.50:0/3662150208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc108101a60 0x7fc108105aa0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009900 tx=0x7fc0f802a410 comp rx=0 tx=0).stop 2024-04-23T18:38:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.966+0000 7fc11067c640 1 -- 172.21.15.50:0/3662150208 shutdown_connections 2024-04-23T18:38:48.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.966+0000 7fc11067c640 1 --2- 172.21.15.50:0/3662150208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc108106180 0x7fc10810dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.966+0000 7fc11067c640 1 --2- 172.21.15.50:0/3662150208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc108101a60 0x7fc108105aa0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.966+0000 7fc11067c640 1 --2- 172.21.15.50:0/3662150208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1081010b0 0x7fc108101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.966+0000 7fc11067c640 1 -- 172.21.15.50:0/3662150208 >> 172.21.15.50:0/3662150208 conn(0x7fc108078000 msgr2=0x7fc108078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:49.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.966+0000 7fc11067c640 1 -- 172.21.15.50:0/3662150208 shutdown_connections 2024-04-23T18:38:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.966+0000 7fc11067c640 1 -- 172.21.15.50:0/3662150208 wait complete. 2024-04-23T18:38:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc11067c640 1 Processor -- start 2024-04-23T18:38:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc11067c640 1 -- start start 2024-04-23T18:38:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc11067c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1081010b0 0x7fc1081986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc11067c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc108106180 0x7fc108198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc11067c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc10819cf20 0x7fc10819d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:49.224 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc11067c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc108112230 con 0x7fc108106180 2024-04-23T18:38:49.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc11067c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc1081120b0 con 0x7fc10819cf20 2024-04-23T18:38:49.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc11067c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc1081123b0 con 0x7fc1081010b0 2024-04-23T18:38:49.258 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc10dc17640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc108106180 0x7fc108198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc10dc17640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc108106180 0x7fc108198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56106/0 (socket says 172.21.15.50:56106) 2024-04-23T18:38:49.273 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc10dc17640 1 -- 172.21.15.50:0/1236160284 learned_addr learned my addr 172.21.15.50:0/1236160284 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:49.281 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc10ec19640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc10819cf20 0x7fc10819d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc10e418640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1081010b0 0x7fc1081986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:49.298 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc10dc17640 1 -- 172.21.15.50:0/1236160284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1081010b0 msgr2=0x7fc1081986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc10dc17640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1081010b0 0x7fc1081986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:49.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc10dc17640 1 -- 172.21.15.50:0/1236160284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc10819cf20 msgr2=0x7fc10819d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc10dc17640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc10819cf20 0x7fc10819d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.967+0000 7fc10dc17640 1 -- 172.21.15.50:0/1236160284 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10819da10 con 0x7fc108106180 2024-04-23T18:38:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.968+0000 7fc10dc17640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc108106180 0x7fc108198bf0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8005ab0 tx=0x7fc0f8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.968+0000 7fc0ff7fe640 1 -- 172.21.15.50:0/1236160284 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f803f070 con 0x7fc108106180 2024-04-23T18:38:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.968+0000 7fc0ff7fe640 1 -- 172.21.15.50:0/1236160284 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0f8031b50 con 0x7fc108106180 2024-04-23T18:38:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.968+0000 7fc11067c640 1 -- 172.21.15.50:0/1236160284 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc10819dca0 con 0x7fc108106180 2024-04-23T18:38:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.968+0000 7fc0ff7fe640 1 -- 172.21.15.50:0/1236160284 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8035030 con 0x7fc108106180 2024-04-23T18:38:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.968+0000 7fc11067c640 1 -- 172.21.15.50:0/1236160284 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1081a5460 con 0x7fc108106180 2024-04-23T18:38:49.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.971+0000 7fc0ff7fe640 1 -- 172.21.15.50:0/1236160284 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc0f8042070 con 0x7fc108106180 2024-04-23T18:38:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.971+0000 7fc113392740 1 Processor -- start 2024-04-23T18:38:49.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.971+0000 7fc113392740 1 -- start start 2024-04-23T18:38:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.971+0000 7fc113392740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ac540 0x559e3e7ac920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:49.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.971+0000 7fc113392740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 0x559e3e7bb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.971+0000 7fc113392740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7bbc10 0x559e3e7be000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:49.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.971+0000 7fc113392740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559e3e711970 con 0x559e3e7ace60 2024-04-23T18:38:49.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.971+0000 7fc113392740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559e3e6ae4a0 con 0x559e3e7bbc10 2024-04-23T18:38:49.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.971+0000 7fc113392740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559e3e70d930 con 0x559e3e7ac540 2024-04-23T18:38:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10e418640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 0x559e3e7bb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:49.636 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10e418640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 0x559e3e7bb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56118/0 (socket says 172.21.15.50:56118) 2024-04-23T18:38:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10e418640 1 -- 172.21.15.50:0/1342022935 learned_addr learned my addr 172.21.15.50:0/1342022935 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10ec19640 1 --2- 172.21.15.50:0/1342022935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ac540 0x559e3e7ac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc0ff7fe640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc0f40a3dc0 0x7fc0f40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10dc17640 1 --2- 172.21.15.50:0/1342022935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7bbc10 0x559e3e7be000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10ec19640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc0f40a3dc0 0x7fc0f40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10e418640 1 -- 172.21.15.50:0/1342022935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ac540 msgr2=0x559e3e7ac920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10e418640 1 --2- 172.21.15.50:0/1342022935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ac540 0x559e3e7ac920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10e418640 1 -- 172.21.15.50:0/1342022935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7bbc10 msgr2=0x559e3e7be000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10e418640 1 --2- 172.21.15.50:0/1342022935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7bbc10 0x559e3e7be000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10e418640 1 -- 172.21.15.50:0/1342022935 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e3e424210 con 0x559e3e7ace60 2024-04-23T18:38:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10dc17640 1 --2- 172.21.15.50:0/1342022935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7bbc10 0x559e3e7be000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:38:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc0ff7fe640 1 -- 172.21.15.50:0/1236160284 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fc0f80f0f70 con 0x7fc108106180 2024-04-23T18:38:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10e418640 1 --2- 172.21.15.50:0/1342022935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 0x559e3e7bb6d0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fc104009c80 tx=0x7fc1040117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55172fdaf632442f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:49.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc10ec19640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc0f40a3dc0 0x7fc0f40a6280 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc100005fd0 tx=0x7fc100004ab0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.972+0000 7fc0fcff9640 1 -- 172.21.15.50:0/1342022935 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc104012450 con 0x559e3e7ace60 2024-04-23T18:38:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.973+0000 7fc0fcff9640 1 -- 172.21.15.50:0/1342022935 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc104012aa0 con 0x559e3e7ace60 2024-04-23T18:38:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.973+0000 7fc0fcff9640 1 -- 172.21.15.50:0/1342022935 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10400fae0 con 0x559e3e7ace60 2024-04-23T18:38:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.973+0000 7fc113392740 1 -- 172.21.15.50:0/1342022935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 msgr2=0x559e3e7bb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:49.989 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.973+0000 7fc113392740 1 --2- 172.21.15.50:0/1342022935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 0x559e3e7bb6d0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fc104009c80 tx=0x7fc1040117b0 comp rx=0 tx=0).stop 2024-04-23T18:38:49.998 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.973+0000 7fc113392740 1 -- 172.21.15.50:0/1342022935 shutdown_connections 2024-04-23T18:38:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.973+0000 7fc113392740 1 --2- 172.21.15.50:0/1342022935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7bbc10 0x559e3e7be000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:50.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.973+0000 7fc113392740 1 --2- 172.21.15.50:0/1342022935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 0x559e3e7bb6d0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.973+0000 7fc113392740 1 --2- 172.21.15.50:0/1342022935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ac540 0x559e3e7ac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.973+0000 7fc113392740 1 -- 172.21.15.50:0/1342022935 >> 172.21.15.50:0/1342022935 conn(0x559e3e7a8b30 msgr2=0x559e3e7a8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.973+0000 7fc113392740 1 -- 172.21.15.50:0/1342022935 shutdown_connections 2024-04-23T18:38:50.107 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.973+0000 7fc113392740 1 -- 172.21.15.50:0/1342022935 wait complete. 2024-04-23T18:38:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc113392740 1 -- start start 2024-04-23T18:38:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc113392740 1 Processor -- start 2024-04-23T18:38:50.132 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc113392740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7ac540 0x559e3e7bb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc113392740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ace60 0x559e3e7b87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:50.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc113392740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7bbc10 0x559e3e7b8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:50.153 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc113392740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x559e3e7be840 con 0x559e3e7bbc10 2024-04-23T18:38:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc113392740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559e3e6ac0b0 con 0x559e3e7ac540 2024-04-23T18:38:50.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc113392740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559e3e7be9c0 con 0x559e3e7ace60 2024-04-23T18:38:50.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc113392740 10 client.0 _ll_register_callbacks cb 0x559e3e699d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:38:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc10ec19640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7bbc10 0x559e3e7b8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc10e418640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7ac540 0x559e3e7bb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc10ec19640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7bbc10 0x559e3e7b8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56130/0 (socket says 172.21.15.50:56130) 2024-04-23T18:38:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc10ec19640 1 -- 172.21.15.50:0/1369548926 learned_addr learned my addr 172.21.15.50:0/1369548926 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.974+0000 7fc10dc17640 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ace60 0x559e3e7b87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.975+0000 7fc10e418640 1 -- 172.21.15.50:0/1369548926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ace60 msgr2=0x559e3e7b87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.975+0000 7fc10e418640 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ace60 0x559e3e7b87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:50.367 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.975+0000 7fc10e418640 1 -- 172.21.15.50:0/1369548926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7bbc10 msgr2=0x559e3e7b8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.975+0000 7fc10e418640 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7bbc10 0x559e3e7b8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.975+0000 7fc10e418640 1 -- 172.21.15.50:0/1369548926 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e3e4f0620 con 0x559e3e7ac540 2024-04-23T18:38:50.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.975+0000 7fc10ec19640 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7bbc10 0x559e3e7b8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:38:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.975+0000 7fc10e418640 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7ac540 0x559e3e7bb500 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc10400f600 tx=0x7fc104009fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=625188ec24a4ca5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.975+0000 7fc0f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:38:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.975+0000 7fc0f1ffb640 1 -- 172.21.15.50:0/1369548926 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc104028020 con 0x559e3e7ac540 2024-04-23T18:38:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.975+0000 7fc0f1ffb640 1 -- 172.21.15.50:0/1369548926 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fc104012450 con 0x559e3e7ac540 2024-04-23T18:38:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.975+0000 7fc113392740 1 -- 172.21.15.50:0/1369548926 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc104012aa0 con 0x559e3e7ac540 2024-04-23T18:38:50.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.976+0000 7fc0f1ffb640 1 -- 172.21.15.50:0/1369548926 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fc104033050 con 0x559e3e7ac540 2024-04-23T18:38:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.976+0000 7fc113392740 10 client.27443 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.976+0000 7fc113392740 1 -- 172.21.15.50:0/1369548926 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x559e3e7ca710 con 0x559e3e7ac540 2024-04-23T18:38:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.976+0000 7fc0f1ffb640 1 -- 172.21.15.50:0/1369548926 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc104012aa0 con 0x559e3e7ac540 2024-04-23T18:38:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.976+0000 7fc0f1ffb640 1 -- 172.21.15.50:0/1369548926 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fc1040035f0 con 0x559e3e7ac540 2024-04-23T18:38:50.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.976+0000 7fc113392740 10 client.27443 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:50.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.976+0000 7fc113392740 10 client.27443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:38:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.976+0000 7fc113392740 20 client.27443 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.976+0000 7fc113392740 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559e3e7cbb70 0x559e3e7cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.976+0000 7fc113392740 4 client.27443 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:38:51.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.976+0000 7fc113392740 1 -- 172.21.15.50:0/1369548926 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e3e71f390 con 0x559e3e7cbb70 2024-04-23T18:38:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.976+0000 7fc10dc17640 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559e3e7cbb70 0x559e3e7cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.977+0000 7fc10dc17640 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559e3e7cbb70 0x559e3e7cdf60 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=449ae029957b5577 server_cookie=93c90cfd19c59b44 in_seq=0 out_seq=0 2024-04-23T18:38:51.181 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.977+0000 7fc0f1ffb640 10 client.27443 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.977+0000 7fc0f1ffb640 1 -- 172.21.15.50:0/1369548926 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e3e71f390 con 0x559e3e7cbb70 2024-04-23T18:38:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.977+0000 7fc0f1ffb640 10 client.27443 handle_command_reply: tid=0 2024-04-23T18:38:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.977+0000 7fc113392740 1 client.27443 shutdown 2024-04-23T18:38:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.978+0000 7fc113392740 1 -- 172.21.15.50:0/1369548926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7ac540 msgr2=0x559e3e7bb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.978+0000 7fc113392740 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7ac540 0x559e3e7bb500 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc10400f600 tx=0x7fc104009fa0 comp rx=0 tx=0).stop 2024-04-23T18:38:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.978+0000 7fc113392740 1 -- 172.21.15.50:0/1369548926 shutdown_connections 2024-04-23T18:38:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.978+0000 7fc113392740 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559e3e7cbb70 0x559e3e7cdf60 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:51.355 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.978+0000 7fc113392740 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7bbc10 0x559e3e7b8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.978+0000 7fc113392740 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ace60 0x559e3e7b87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.978+0000 7fc113392740 1 --2- 172.21.15.50:0/1369548926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7ac540 0x559e3e7bb500 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.978+0000 7fc0f1ffb640 0 client.27443 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:51.390 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.978+0000 7fc113392740 1 -- 172.21.15.50:0/1369548926 >> 172.21.15.50:0/1369548926 conn(0x559e3e7a8b30 msgr2=0x559e3e7ae9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.978+0000 7fc113392740 1 -- 172.21.15.50:0/1369548926 shutdown_connections 2024-04-23T18:38:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.978+0000 7fc113392740 1 -- 172.21.15.50:0/1369548926 wait complete. 2024-04-23T18:38:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.978+0000 7fc113392740 20 client.27443 trim_cache size 0 max 16384 2024-04-23T18:38:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.993+0000 7fc113392740 1 Processor -- start 2024-04-23T18:38:51.432 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.993+0000 7fc113392740 1 -- start start 2024-04-23T18:38:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.993+0000 7fc113392740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7a8b30 0x559e3e7c7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.993+0000 7fc113392740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ac540 0x559e3e7b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.993+0000 7fc113392740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ace60 0x559e3e7bb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.993+0000 7fc113392740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc1081ae1b0 con 0x559e3e7ac540 2024-04-23T18:38:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.993+0000 7fc113392740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc1081aba80 con 0x559e3e7a8b30 2024-04-23T18:38:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.993+0000 7fc113392740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc1081a9910 con 0x559e3e7ace60 2024-04-23T18:38:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.994+0000 7fc10e418640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7a8b30 0x559e3e7c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.994+0000 7fc10e418640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7a8b30 0x559e3e7c7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:34196/0 (socket says 172.21.15.50:34196) 2024-04-23T18:38:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.994+0000 7fc10e418640 1 -- 172.21.15.50:0/1485293793 learned_addr learned my addr 172.21.15.50:0/1485293793 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:51.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.994+0000 7fc10dc17640 1 --2- 172.21.15.50:0/1485293793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ac540 0x559e3e7b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:51.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.994+0000 7fc10ec19640 1 --2- 172.21.15.50:0/1485293793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ace60 0x559e3e7bb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:51.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.994+0000 7fc10e418640 1 -- 172.21.15.50:0/1485293793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ace60 msgr2=0x559e3e7bb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.994+0000 7fc10e418640 1 --2- 172.21.15.50:0/1485293793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ace60 0x559e3e7bb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:51.649 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.994+0000 7fc10e418640 1 -- 172.21.15.50:0/1485293793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ac540 msgr2=0x559e3e7b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:51.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.994+0000 7fc10e418640 1 --2- 172.21.15.50:0/1485293793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ac540 0x559e3e7b3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:51.665 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.994+0000 7fc10e418640 1 -- 172.21.15.50:0/1485293793 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e3e7a8960 con 0x559e3e7a8b30 2024-04-23T18:38:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.994+0000 7fc10e418640 1 --2- 172.21.15.50:0/1485293793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7a8b30 0x559e3e7c7700 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc104003c00 tx=0x7fc104013f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c9767523da47aec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.995+0000 7fc0f3fff640 1 -- 172.21.15.50:0/1485293793 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc104028040 con 0x559e3e7a8b30 2024-04-23T18:38:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.995+0000 7fc0f3fff640 1 -- 172.21.15.50:0/1485293793 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc10403c070 con 0x559e3e7a8b30 2024-04-23T18:38:51.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.995+0000 7fc0f3fff640 1 -- 172.21.15.50:0/1485293793 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1040035f0 con 0x559e3e7a8b30 2024-04-23T18:38:51.729 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.995+0000 7fc113392740 1 -- 172.21.15.50:0/1485293793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7a8b30 msgr2=0x559e3e7c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:51.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.995+0000 7fc113392740 1 --2- 172.21.15.50:0/1485293793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7a8b30 0x559e3e7c7700 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc104003c00 tx=0x7fc104013f70 comp rx=0 tx=0).stop 2024-04-23T18:38:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.995+0000 7fc113392740 1 -- 172.21.15.50:0/1485293793 shutdown_connections 2024-04-23T18:38:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.995+0000 7fc113392740 1 --2- 172.21.15.50:0/1485293793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ace60 0x559e3e7bb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:51.814 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.995+0000 7fc113392740 1 --2- 172.21.15.50:0/1485293793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ac540 0x559e3e7b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.995+0000 7fc113392740 1 --2- 172.21.15.50:0/1485293793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7a8b30 0x559e3e7c7700 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.995+0000 7fc113392740 1 -- 172.21.15.50:0/1485293793 >> 172.21.15.50:0/1485293793 conn(0x559e3e7b8e90 msgr2=0x559e3e7b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:51.839 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.995+0000 7fc113392740 1 -- 172.21.15.50:0/1485293793 shutdown_connections 2024-04-23T18:38:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.995+0000 7fc113392740 1 -- 172.21.15.50:0/1485293793 wait complete. 2024-04-23T18:38:51.856 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc113392740 1 -- start start 2024-04-23T18:38:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc113392740 1 Processor -- start 2024-04-23T18:38:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc113392740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7a8b30 0x559e3e7c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc113392740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ac540 0x559e3e7b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc113392740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 0x559e3e7b5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc113392740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc1081a9f60 con 0x559e3e7ace60 2024-04-23T18:38:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc113392740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc1081ab3b0 con 0x559e3e7a8b30 2024-04-23T18:38:51.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc113392740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc1081aa610 con 0x559e3e7ac540 2024-04-23T18:38:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc113392740 10 client.0 _ll_register_callbacks cb 0x7fc1040056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:38:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc10ec19640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 0x559e3e7b5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc10dc17640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ac540 0x559e3e7b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:52.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc10ec19640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 0x559e3e7b5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56152/0 (socket says 172.21.15.50:56152) 2024-04-23T18:38:52.086 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc10dc17640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ac540 0x559e3e7b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:48882/0 (socket says 172.21.15.50:48882) 2024-04-23T18:38:52.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc10ec19640 1 -- 172.21.15.50:0/1079834983 learned_addr learned my addr 172.21.15.50:0/1079834983 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:52.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.996+0000 7fc10e418640 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7a8b30 0x559e3e7c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:52.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc10dc17640 1 -- 172.21.15.50:0/1079834983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7a8b30 msgr2=0x559e3e7c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:52.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc10dc17640 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7a8b30 0x559e3e7c3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:52.133 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc10dc17640 1 -- 172.21.15.50:0/1079834983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 msgr2=0x559e3e7b5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:52.141 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc10dc17640 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 0x559e3e7b5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:52.149 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc10dc17640 1 -- 172.21.15.50:0/1079834983 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e3e7b5950 con 0x559e3e7ac540 2024-04-23T18:38:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc10ec19640 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 0x559e3e7b5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:38:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc10dc17640 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ac540 0x559e3e7b4ed0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80ee650 tx=0x7fc0f8031f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78958dd6e2d1c915 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:52.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc0f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:38:52.204 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc0f1ffb640 1 -- 172.21.15.50:0/1079834983 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8049040 con 0x559e3e7ac540 2024-04-23T18:38:52.214 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc0f1ffb640 1 -- 172.21.15.50:0/1079834983 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fc0f80bb1a0 con 0x559e3e7ac540 2024-04-23T18:38:52.222 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc113392740 1 -- 172.21.15.50:0/1079834983 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc10403c070 con 0x559e3e7ac540 2024-04-23T18:38:52.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc0f1ffb640 1 -- 172.21.15.50:0/1079834983 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fc0f80bcea0 con 0x559e3e7ac540 2024-04-23T18:38:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc113392740 10 client.27481 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.997+0000 7fc113392740 1 -- 172.21.15.50:0/1079834983 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x559e3e831ab0 con 0x559e3e7ac540 2024-04-23T18:38:52.336 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.998+0000 7fc0f1ffb640 1 -- 172.21.15.50:0/1079834983 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc10403c070 con 0x559e3e7ac540 2024-04-23T18:38:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.998+0000 7fc0f1ffb640 1 -- 172.21.15.50:0/1079834983 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fc0f80b9510 con 0x559e3e7ac540 2024-04-23T18:38:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.998+0000 7fc113392740 10 client.27481 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.998+0000 7fc113392740 10 client.27481 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:38:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.998+0000 7fc113392740 20 client.27481 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:52.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.998+0000 7fc113392740 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559e3e7bbc10 0x559e3e834220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:52.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.998+0000 7fc113392740 4 client.27481 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:38:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.998+0000 7fc113392740 1 -- 172.21.15.50:0/1079834983 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559e3e719200 con 0x559e3e7bbc10 2024-04-23T18:38:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.998+0000 7fc10e418640 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559e3e7bbc10 0x559e3e834220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:52.417 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.999+0000 7fc10e418640 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559e3e7bbc10 0x559e3e834220 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c287c2f3b4044cea server_cookie=c52eb628e6babbf4 in_seq=0 out_seq=0 2024-04-23T18:38:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:48.999+0000 7fc0f1ffb640 10 client.27481 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc0f1ffb640 1 -- 172.21.15.50:0/1079834983 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559e3e719200 con 0x559e3e7bbc10 2024-04-23T18:38:52.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc0f1ffb640 10 client.27481 handle_command_reply: tid=0 2024-04-23T18:38:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc113392740 1 client.27481 shutdown 2024-04-23T18:38:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc113392740 1 -- 172.21.15.50:0/1079834983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ac540 msgr2=0x559e3e7b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc113392740 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ac540 0x559e3e7b4ed0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80ee650 tx=0x7fc0f8031f50 comp rx=0 tx=0).stop 2024-04-23T18:38:52.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc113392740 1 -- 172.21.15.50:0/1079834983 shutdown_connections 2024-04-23T18:38:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc113392740 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x559e3e7bbc10 0x559e3e834220 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc113392740 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x559e3e7ace60 0x559e3e7b5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc113392740 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559e3e7ac540 0x559e3e7b4ed0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc113392740 1 --2- 172.21.15.50:0/1079834983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e3e7a8b30 0x559e3e7c3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc0f1ffb640 0 client.27481 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc113392740 1 -- 172.21.15.50:0/1079834983 >> 172.21.15.50:0/1079834983 conn(0x559e3e7b8e90 msgr2=0x559e3e7b3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc113392740 1 -- 172.21.15.50:0/1079834983 shutdown_connections 2024-04-23T18:38:52.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc113392740 1 -- 172.21.15.50:0/1079834983 wait complete. 2024-04-23T18:38:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.040+0000 7fc113392740 20 client.27481 trim_cache size 0 max 16384 2024-04-23T18:38:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:38:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:38:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:38:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:38:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.041+0000 7fc11067c640 1 -- 172.21.15.50:0/1236160284 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc0f40a3dc0 msgr2=0x7fc0f40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:52.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.041+0000 7fc11067c640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc0f40a3dc0 0x7fc0f40a6280 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc100005fd0 tx=0x7fc100004ab0 comp rx=0 tx=0).stop 2024-04-23T18:38:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.041+0000 7fc11067c640 1 -- 172.21.15.50:0/1236160284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc108106180 msgr2=0x7fc108198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:52.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.041+0000 7fc11067c640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc108106180 0x7fc108198bf0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8005ab0 tx=0x7fc0f8003b90 comp rx=0 tx=0).stop 2024-04-23T18:38:52.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.041+0000 7fc11067c640 1 -- 172.21.15.50:0/1236160284 shutdown_connections 2024-04-23T18:38:52.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.041+0000 7fc11067c640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fc0f40a3dc0 0x7fc0f40a6280 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:52.797 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.041+0000 7fc11067c640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc10819cf20 0x7fc10819d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:52.805 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.041+0000 7fc11067c640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc108106180 0x7fc108198bf0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.041+0000 7fc11067c640 1 --2- 172.21.15.50:0/1236160284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1081010b0 0x7fc1081986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:52.819 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.041+0000 7fc11067c640 1 -- 172.21.15.50:0/1236160284 >> 172.21.15.50:0/1236160284 conn(0x7fc108078000 msgr2=0x7fc1080fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.041+0000 7fc11067c640 1 -- 172.21.15.50:0/1236160284 shutdown_connections 2024-04-23T18:38:52.834 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:38:49.041+0000 7fc11067c640 1 -- 172.21.15.50:0/1236160284 wait complete. 2024-04-23T18:38:52.842 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.15 MB/sec execute 323 sec latency 31382.799 ms 2024-04-23T18:38:52.849 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.09 MB/sec execute 324 sec latency 32382.911 ms 2024-04-23T18:38:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:50 smithi050 ceph-mon[29187]: pgmap v646: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 4.1 MiB/s wr, 140 op/s 2024-04-23T18:38:52.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:38:52.895 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 19.04 MB/sec execute 325 sec latency 33382.967 ms 2024-04-23T18:38:52.903 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 18.98 MB/sec execute 326 sec latency 34383.096 ms 2024-04-23T18:38:52.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:52 smithi050 ceph-mon[29187]: pgmap v647: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.8 MiB/s wr, 117 op/s 2024-04-23T18:38:52.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:50 smithi188 ceph-mon[32694]: pgmap v646: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 4.1 MiB/s wr, 140 op/s 2024-04-23T18:38:52.987 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:52 smithi188 ceph-mon[32694]: pgmap v647: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.8 MiB/s wr, 117 op/s 2024-04-23T18:38:52.995 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:50 smithi177 ceph-mon[32308]: pgmap v646: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 4.1 MiB/s wr, 140 op/s 2024-04-23T18:38:53.002 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:52 smithi177 ceph-mon[32308]: pgmap v647: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.8 MiB/s wr, 117 op/s 2024-04-23T18:38:53.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.100+0000 7fda0f35c640 1 -- 172.21.15.50:0/895433234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0810c520 msgr2=0x7fda0810e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:53.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.100+0000 7fda0f35c640 1 --2- 172.21.15.50:0/895433234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0810c520 0x7fda0810e9c0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fda00009ca0 tx=0x7fda0002a9e0 comp rx=0 tx=0).stop 2024-04-23T18:38:53.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.100+0000 7fda0f35c640 1 -- 172.21.15.50:0/895433234 shutdown_connections 2024-04-23T18:38:53.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.100+0000 7fda0f35c640 1 --2- 172.21.15.50:0/895433234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0810c520 0x7fda0810e9c0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:53.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.100+0000 7fda0f35c640 1 --2- 172.21.15.50:0/895433234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda08103b00 0x7fda0810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:53.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.100+0000 7fda0f35c640 1 --2- 172.21.15.50:0/895433234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda081031e0 0x7fda081035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:53.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.100+0000 7fda0f35c640 1 -- 172.21.15.50:0/895433234 >> 172.21.15.50:0/895433234 conn(0x7fda08077f50 msgr2=0x7fda08078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:53.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.100+0000 7fda0f35c640 1 -- 172.21.15.50:0/895433234 shutdown_connections 2024-04-23T18:38:53.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.100+0000 7fda0f35c640 1 -- 172.21.15.50:0/895433234 wait complete. 2024-04-23T18:38:53.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.101+0000 7fda0f35c640 1 Processor -- start 2024-04-23T18:38:53.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.101+0000 7fda0f35c640 1 -- start start 2024-04-23T18:38:53.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.101+0000 7fda0f35c640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda081031e0 0x7fda08198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:53.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.101+0000 7fda0f35c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda08103b00 0x7fda08198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:53.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.101+0000 7fda0f35c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0810c520 0x7fda0819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:53.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.101+0000 7fda0f35c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fda08112190 con 0x7fda0810c520 2024-04-23T18:38:53.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.101+0000 7fda0f35c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fda08112010 con 0x7fda08103b00 2024-04-23T18:38:53.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.101+0000 7fda0f35c640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fda08112310 con 0x7fda081031e0 2024-04-23T18:38:53.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.101+0000 7fda0d8f9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0810c520 0x7fda0819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:53.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.101+0000 7fda0d8f9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0810c520 0x7fda0819ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46484/0 (socket says 172.21.15.50:46484) 2024-04-23T18:38:53.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.101+0000 7fda0d8f9640 1 -- 172.21.15.50:0/29838468 learned_addr learned my addr 172.21.15.50:0/29838468 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:53.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fda0d0f8640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda081031e0 0x7fda08198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:53.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fda0c8f7640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda08103b00 0x7fda08198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:53.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fda0d8f9640 1 -- 172.21.15.50:0/29838468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda081031e0 msgr2=0x7fda08198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:53.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fda0d8f9640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda081031e0 0x7fda08198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:53.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fda0d8f9640 1 -- 172.21.15.50:0/29838468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda08103b00 msgr2=0x7fda08198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:53.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fda0d8f9640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda08103b00 0x7fda08198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:53.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fda0d8f9640 1 -- 172.21.15.50:0/29838468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda0819d600 con 0x7fda0810c520 2024-04-23T18:38:53.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fda0d8f9640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0810c520 0x7fda0819ced0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fda00006010 tx=0x7fda0002fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:53.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fd9fe7fc640 1 -- 172.21.15.50:0/29838468 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00033040 con 0x7fda0810c520 2024-04-23T18:38:53.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fd9fe7fc640 1 -- 172.21.15.50:0/29838468 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda0002e7a0 con 0x7fda0810c520 2024-04-23T18:38:53.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fda0f35c640 1 -- 172.21.15.50:0/29838468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda0819d890 con 0x7fda0810c520 2024-04-23T18:38:53.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fd9fe7fc640 1 -- 172.21.15.50:0/29838468 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00043630 con 0x7fda0810c520 2024-04-23T18:38:53.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.102+0000 7fda0f35c640 1 -- 172.21.15.50:0/29838468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda081a5180 con 0x7fda0810c520 2024-04-23T18:38:53.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.103+0000 7fda0f35c640 1 -- 172.21.15.50:0/29838468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda08068d70 con 0x7fda0810c520 2024-04-23T18:38:53.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.105+0000 7fd9fe7fc640 1 -- 172.21.15.50:0/29838468 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fda00036070 con 0x7fda0810c520 2024-04-23T18:38:53.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.106+0000 7fd9fe7fc640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd9f40a3d20 0x7fd9f40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:53.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.106+0000 7fda0d0f8640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd9f40a3d20 0x7fd9f40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:53.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.106+0000 7fd9fe7fc640 1 -- 172.21.15.50:0/29838468 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fda000f1d30 con 0x7fda0810c520 2024-04-23T18:38:53.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.106+0000 7fda0d0f8640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd9f40a3d20 0x7fd9f40a61e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fda0400abe0 tx=0x7fda0400a2d0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:53.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.109+0000 7fd9fe7fc640 1 -- 172.21.15.50:0/29838468 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda0003e070 con 0x7fda0810c520 2024-04-23T18:38:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.341+0000 7fda0f35c640 1 -- 172.21.15.50:0/29838468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda08199960 con 0x7fda0810c520 2024-04-23T18:38:53.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.341+0000 7fd9fe7fc640 1 -- 172.21.15.50:0/29838468 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+11875 (secure 0 0 0) 0x7fda08199960 con 0x7fda0810c520 2024-04-23T18:38:53.779 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:38:53.838 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":62,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:36:29.984583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27194":{"gid":27194,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2754597719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2754597719},{"type":"v1","addr":"172.21.15.177:6837","nonce":2754597719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:38:53.848 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 62 2024-04-23T18:38:53.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.343+0000 7fda0f35c640 1 -- 172.21.15.50:0/29838468 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd9f40a3d20 msgr2=0x7fd9f40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:53.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.343+0000 7fda0f35c640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd9f40a3d20 0x7fd9f40a61e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fda0400abe0 tx=0x7fda0400a2d0 comp rx=0 tx=0).stop 2024-04-23T18:38:53.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.343+0000 7fda0f35c640 1 -- 172.21.15.50:0/29838468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0810c520 msgr2=0x7fda0819ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:53.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.343+0000 7fda0f35c640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0810c520 0x7fda0819ced0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fda00006010 tx=0x7fda0002fbe0 comp rx=0 tx=0).stop 2024-04-23T18:38:53.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.344+0000 7fda0f35c640 1 -- 172.21.15.50:0/29838468 shutdown_connections 2024-04-23T18:38:53.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.344+0000 7fda0f35c640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fd9f40a3d20 0x7fd9f40a61e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:53.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.344+0000 7fda0f35c640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda0810c520 0x7fda0819ced0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:53.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.344+0000 7fda0f35c640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda08103b00 0x7fda08198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:54.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.344+0000 7fda0f35c640 1 --2- 172.21.15.50:0/29838468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda081031e0 0x7fda08198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:54.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.344+0000 7fda0f35c640 1 -- 172.21.15.50:0/29838468 >> 172.21.15.50:0/29838468 conn(0x7fda08077f50 msgr2=0x7fda080fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:54.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.344+0000 7fda0f35c640 1 -- 172.21.15.50:0/29838468 shutdown_connections 2024-04-23T18:38:54.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:53.344+0000 7fda0f35c640 1 -- 172.21.15.50:0/29838468 wait complete. 2024-04-23T18:38:54.036 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id 525eaceb --release --await' 2024-04-23T18:38:54.044 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id 525eaceb --release --await 2024-04-23T18:38:54.069 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167141 18.92 MB/sec execute 327 sec latency 35383.183 ms 2024-04-23T18:38:54.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.225+0000 7f2089fef640 1 -- 172.21.15.50:0/2425895057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084101a50 msgr2=0x7f2084105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:54.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.225+0000 7f2089fef640 1 --2- 172.21.15.50:0/2425895057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084101a50 0x7f2084105a90 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f2078009900 tx=0x7f207802a410 comp rx=0 tx=0).stop 2024-04-23T18:38:54.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.225+0000 7f2089fef640 1 -- 172.21.15.50:0/2425895057 shutdown_connections 2024-04-23T18:38:54.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.225+0000 7f2089fef640 1 --2- 172.21.15.50:0/2425895057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2084106170 0x7f208410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:54.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.225+0000 7f2089fef640 1 --2- 172.21.15.50:0/2425895057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084101a50 0x7f2084105a90 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.225+0000 7f2089fef640 1 --2- 172.21.15.50:0/2425895057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20841010a0 0x7f2084101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.225+0000 7f2089fef640 1 -- 172.21.15.50:0/2425895057 >> 172.21.15.50:0/2425895057 conn(0x7f2084077eb0 msgr2=0x7f20840782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:54.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.225+0000 7f2089fef640 1 -- 172.21.15.50:0/2425895057 shutdown_connections 2024-04-23T18:38:54.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.225+0000 7f2089fef640 1 -- 172.21.15.50:0/2425895057 wait complete. 2024-04-23T18:38:54.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.226+0000 7f2089fef640 1 Processor -- start 2024-04-23T18:38:54.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.226+0000 7f2089fef640 1 -- start start 2024-04-23T18:38:54.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.226+0000 7f2089fef640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20841010a0 0x7f2084198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:54.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.226+0000 7f2089fef640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084106170 0x7f2084198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:54.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.226+0000 7f2089fef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f208419d000 0x7f208419d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:54.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.226+0000 7f2089fef640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f20841110b0 con 0x7f2084106170 2024-04-23T18:38:54.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.226+0000 7f2089fef640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2084110f30 con 0x7f208419d000 2024-04-23T18:38:54.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.226+0000 7f2089fef640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2084111230 con 0x7f20841010a0 2024-04-23T18:38:54.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.226+0000 7f2082ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084106170 0x7f2084198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:54.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2082ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084106170 0x7f2084198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46514/0 (socket says 172.21.15.50:46514) 2024-04-23T18:38:54.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2082ffd640 1 -- 172.21.15.50:0/1952679334 learned_addr learned my addr 172.21.15.50:0/1952679334 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:54.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f20837fe640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20841010a0 0x7f2084198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:54.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2083fff640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f208419d000 0x7f208419d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:54.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2082ffd640 1 -- 172.21.15.50:0/1952679334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20841010a0 msgr2=0x7f2084198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2082ffd640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20841010a0 0x7f2084198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:54.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2082ffd640 1 -- 172.21.15.50:0/1952679334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f208419d000 msgr2=0x7f208419d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:54.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2082ffd640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f208419d000 0x7f208419d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2082ffd640 1 -- 172.21.15.50:0/1952679334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f208419daf0 con 0x7f2084106170 2024-04-23T18:38:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2082ffd640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084106170 0x7f2084198c80 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f2078006010 tx=0x7f2078004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:54.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2080ff9640 1 -- 172.21.15.50:0/1952679334 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207803f030 con 0x7f2084106170 2024-04-23T18:38:54.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2089fef640 1 -- 172.21.15.50:0/1952679334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f208419dd80 con 0x7f2084106170 2024-04-23T18:38:54.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2080ff9640 1 -- 172.21.15.50:0/1952679334 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2078003aa0 con 0x7f2084106170 2024-04-23T18:38:54.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2080ff9640 1 -- 172.21.15.50:0/1952679334 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2078035070 con 0x7f2084106170 2024-04-23T18:38:54.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.227+0000 7f2089fef640 1 -- 172.21.15.50:0/1952679334 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20841a55a0 con 0x7f2084106170 2024-04-23T18:38:54.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f2080ff9640 1 -- 172.21.15.50:0/1952679334 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2078042030 con 0x7f2084106170 2024-04-23T18:38:54.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f208cdd7740 1 Processor -- start 2024-04-23T18:38:54.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f208cdd7740 1 -- start start 2024-04-23T18:38:54.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f208cdd7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1490 0x55a03f3b1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:54.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f208cdd7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1db0 0x55a03f3bbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:54.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f208cdd7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3bc480 0x55a03f3be870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:55.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f208cdd7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a03f3166d0 con 0x55a03f3b1490 2024-04-23T18:38:55.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f208cdd7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a03f2b31a0 con 0x55a03f3b1db0 2024-04-23T18:38:55.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f208cdd7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a03f312690 con 0x55a03f3bc480 2024-04-23T18:38:55.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f2083fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1490 0x55a03f3b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:55.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f2083fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1490 0x55a03f3b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46518/0 (socket says 172.21.15.50:46518) 2024-04-23T18:38:55.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f2083fff640 1 -- 172.21.15.50:0/1513468693 learned_addr learned my addr 172.21.15.50:0/1513468693 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:55.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f2080ff9640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f206c0a3d20 0x7f206c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:55.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f20837fe640 1 --2- 172.21.15.50:0/1513468693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1db0 0x55a03f3bbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:55.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f2082ffd640 1 --2- 172.21.15.50:0/1513468693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3bc480 0x55a03f3be870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:55.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.230+0000 7f2083fff640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f206c0a3d20 0x7f206c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:55.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.231+0000 7f2080ff9640 1 -- 172.21.15.50:0/1952679334 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7f20780f0c40 con 0x7f2084106170 2024-04-23T18:38:55.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.231+0000 7f2083fff640 1 -- 172.21.15.50:0/1513468693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3bc480 msgr2=0x55a03f3be870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:55.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.231+0000 7f2083fff640 1 --2- 172.21.15.50:0/1513468693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3bc480 0x55a03f3be870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:55.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.231+0000 7f2083fff640 1 -- 172.21.15.50:0/1513468693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1db0 msgr2=0x55a03f3bbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:55.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.231+0000 7f2083fff640 1 --2- 172.21.15.50:0/1513468693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1db0 0x55a03f3bbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.231+0000 7f2083fff640 1 -- 172.21.15.50:0/1513468693 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a03f2bd5b0 con 0x55a03f3b1490 2024-04-23T18:38:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.231+0000 7f20837fe640 1 --2- 172.21.15.50:0/1513468693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1db0 0x55a03f3bbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:38:55.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.231+0000 7f2083fff640 1 --2- 172.21.15.50:0/1513468693 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1490 0x55a03f3b1870 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f207c0129f0 tx=0x7f207c012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a44740e5f2db6c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:55.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.231+0000 7f2069ffb640 1 -- 172.21.15.50:0/1513468693 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c023070 con 0x55a03f3b1490 2024-04-23T18:38:55.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.231+0000 7f2069ffb640 1 -- 172.21.15.50:0/1513468693 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f207c0046e0 con 0x55a03f3b1490 2024-04-23T18:38:55.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.231+0000 7f2069ffb640 1 -- 172.21.15.50:0/1513468693 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c01f990 con 0x55a03f3b1490 2024-04-23T18:38:55.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.232+0000 7f208cdd7740 1 -- 172.21.15.50:0/1513468693 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1490 msgr2=0x55a03f3b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:55.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.232+0000 7f2083fff640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f206c0a3d20 0x7f206c0a61e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f207c022000 tx=0x7f207c021000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:55.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.232+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1513468693 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1490 0x55a03f3b1870 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f207c0129f0 tx=0x7f207c012eb0 comp rx=0 tx=0).stop 2024-04-23T18:38:55.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.232+0000 7f208cdd7740 1 -- 172.21.15.50:0/1513468693 shutdown_connections 2024-04-23T18:38:55.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.232+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1513468693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3bc480 0x55a03f3be870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:55.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.232+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1513468693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1db0 0x55a03f3bbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:55.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.232+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1513468693 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1490 0x55a03f3b1870 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:55.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.232+0000 7f208cdd7740 1 -- 172.21.15.50:0/1513468693 >> 172.21.15.50:0/1513468693 conn(0x55a03f3ad7b0 msgr2=0x55a03f3adb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:55.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.232+0000 7f208cdd7740 1 -- 172.21.15.50:0/1513468693 shutdown_connections 2024-04-23T18:38:55.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.232+0000 7f208cdd7740 1 -- 172.21.15.50:0/1513468693 wait complete. 2024-04-23T18:38:55.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.232+0000 7f208cdd7740 1 -- start start 2024-04-23T18:38:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.232+0000 7f208cdd7740 1 Processor -- start 2024-04-23T18:38:55.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f208cdd7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3b1490 0x55a03f3bbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f208cdd7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1db0 0x55a03f3b91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:55.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f208cdd7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3bc480 0x55a03f3b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:55.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f208cdd7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55a03f3bf310 con 0x55a03f3bc480 2024-04-23T18:38:55.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f208cdd7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a03f2b0b20 con 0x55a03f3b1db0 2024-04-23T18:38:55.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f208cdd7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55a03f3bf490 con 0x55a03f3b1490 2024-04-23T18:38:55.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f208cdd7740 10 client.0 _ll_register_callbacks cb 0x55a03f2b0530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:38:55.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2083fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3bc480 0x55a03f3b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:55.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2083fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3bc480 0x55a03f3b9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46528/0 (socket says 172.21.15.50:46528) 2024-04-23T18:38:55.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2083fff640 1 -- 172.21.15.50:0/1777872337 learned_addr learned my addr 172.21.15.50:0/1777872337 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:55.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f20837fe640 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3b1490 0x55a03f3bbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:55.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2082ffd640 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1db0 0x55a03f3b91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:55.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2083fff640 1 -- 172.21.15.50:0/1777872337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3b1490 msgr2=0x55a03f3bbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:55.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2083fff640 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3b1490 0x55a03f3bbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:55.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2083fff640 1 -- 172.21.15.50:0/1777872337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1db0 msgr2=0x55a03f3b91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:55.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2083fff640 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1db0 0x55a03f3b91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:55.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2083fff640 1 -- 172.21.15.50:0/1777872337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a03f028920 con 0x55a03f3bc480 2024-04-23T18:38:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2083fff640 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3bc480 0x55a03f3b9860 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f207c009b20 tx=0x7f207c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=266b73de975b0486 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:55.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2062ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:38:55.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2062ffd640 1 -- 172.21.15.50:0/1777872337 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c023070 con 0x55a03f3bc480 2024-04-23T18:38:56.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f2062ffd640 1 -- 172.21.15.50:0/1777872337 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7f207c0042d0 con 0x55a03f3bc480 2024-04-23T18:38:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.233+0000 7f208cdd7740 1 -- 172.21.15.50:0/1777872337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f207c0046e0 con 0x55a03f3bc480 2024-04-23T18:38:56.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.234+0000 7f2062ffd640 1 -- 172.21.15.50:0/1777872337 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f207c0044d0 con 0x55a03f3bc480 2024-04-23T18:38:56.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.234+0000 7f208cdd7740 10 client.17982 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.234+0000 7f208cdd7740 1 -- 172.21.15.50:0/1777872337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55a03f0f4d30 con 0x55a03f3bc480 2024-04-23T18:38:56.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.234+0000 7f2062ffd640 1 -- 172.21.15.50:0/1777872337 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f207c0046e0 con 0x55a03f3bc480 2024-04-23T18:38:56.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.234+0000 7f2062ffd640 1 -- 172.21.15.50:0/1777872337 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f207c02e070 con 0x55a03f3bc480 2024-04-23T18:38:56.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.234+0000 7f208cdd7740 10 client.17982 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:56.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.234+0000 7f208cdd7740 10 client.17982 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:38:56.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.234+0000 7f208cdd7740 20 client.17982 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:56.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.234+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a03f3cc4f0 0x55a03f3ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:56.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.234+0000 7f208cdd7740 4 client.17982 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:38:56.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.234+0000 7f208cdd7740 1 -- 172.21.15.50:0/1777872337 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a03f2d4500 con 0x55a03f3cc4f0 2024-04-23T18:38:56.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.234+0000 7f20837fe640 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a03f3cc4f0 0x55a03f3ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:56.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.235+0000 7f2062ffd640 10 client.17982 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:56.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.235+0000 7f20837fe640 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a03f3cc4f0 0x55a03f3ce8e0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d9d60ca9808cc4 server_cookie=e6837a3ae99144a4 in_seq=0 out_seq=0 2024-04-23T18:38:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.235+0000 7f2062ffd640 1 -- 172.21.15.50:0/1777872337 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a03f2d4500 con 0x55a03f3cc4f0 2024-04-23T18:38:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.235+0000 7f2062ffd640 10 client.17982 handle_command_reply: tid=0 2024-04-23T18:38:56.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f208cdd7740 1 client.17982 shutdown 2024-04-23T18:38:56.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f208cdd7740 1 -- 172.21.15.50:0/1777872337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3bc480 msgr2=0x55a03f3b9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:56.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3bc480 0x55a03f3b9860 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f207c009b20 tx=0x7f207c009240 comp rx=0 tx=0).stop 2024-04-23T18:38:56.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f208cdd7740 1 -- 172.21.15.50:0/1777872337 shutdown_connections 2024-04-23T18:38:56.478 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a03f3cc4f0 0x55a03f3ce8e0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:56.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3bc480 0x55a03f3b9860 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:56.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1db0 0x55a03f3b91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:56.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f2062ffd640 0 client.17982 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:56.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1777872337 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3b1490 0x55a03f3bbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:56.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f208cdd7740 1 -- 172.21.15.50:0/1777872337 >> 172.21.15.50:0/1777872337 conn(0x55a03f3ad7b0 msgr2=0x55a03f3ac9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:56.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f208cdd7740 1 -- 172.21.15.50:0/1777872337 shutdown_connections 2024-04-23T18:38:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f208cdd7740 1 -- 172.21.15.50:0/1777872337 wait complete. 2024-04-23T18:38:56.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.236+0000 7f208cdd7740 20 client.17982 trim_cache size 0 max 16384 2024-04-23T18:38:56.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f208cdd7740 1 Processor -- start 2024-04-23T18:38:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f208cdd7740 1 -- start start 2024-04-23T18:38:56.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f208cdd7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1490 0x55a03f3c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:56.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f208cdd7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 0x55a03f3ad150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:56.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f208cdd7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3bc480 0x55a03f3b7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:56.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f208cdd7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f20841b04f0 con 0x55a03f3b1db0 2024-04-23T18:38:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f208cdd7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f20841addc0 con 0x55a03f3b1490 2024-04-23T18:38:56.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f208cdd7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f20841abc30 con 0x55a03f3bc480 2024-04-23T18:38:56.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f2082ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 0x55a03f3ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:56.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f2082ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 0x55a03f3ad150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46542/0 (socket says 172.21.15.50:46542) 2024-04-23T18:38:56.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f2082ffd640 1 -- 172.21.15.50:0/1045587812 learned_addr learned my addr 172.21.15.50:0/1045587812 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:56.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f2083fff640 1 --2- 172.21.15.50:0/1045587812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3bc480 0x55a03f3b7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:56.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f2082ffd640 1 -- 172.21.15.50:0/1045587812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3bc480 msgr2=0x55a03f3b7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f20837fe640 1 --2- 172.21.15.50:0/1045587812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1490 0x55a03f3c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:56.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f2082ffd640 1 --2- 172.21.15.50:0/1045587812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3bc480 0x55a03f3b7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:56.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f2082ffd640 1 -- 172.21.15.50:0/1045587812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1490 msgr2=0x55a03f3c81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:56.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f2082ffd640 1 --2- 172.21.15.50:0/1045587812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1490 0x55a03f3c81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:56.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.254+0000 7f2082ffd640 1 -- 172.21.15.50:0/1045587812 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a03f3b1be0 con 0x55a03f3b1db0 2024-04-23T18:38:56.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f2082ffd640 1 --2- 172.21.15.50:0/1045587812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 0x55a03f3ad150 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f2078031440 tx=0x7f2078003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49cc72def41ea58 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:56.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f20697fa640 1 -- 172.21.15.50:0/1045587812 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2078049040 con 0x55a03f3b1db0 2024-04-23T18:38:56.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f20697fa640 1 -- 172.21.15.50:0/1045587812 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20780bcbd0 con 0x55a03f3b1db0 2024-04-23T18:38:56.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f20697fa640 1 -- 172.21.15.50:0/1045587812 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20780b9240 con 0x55a03f3b1db0 2024-04-23T18:38:57.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f208cdd7740 1 -- 172.21.15.50:0/1045587812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 msgr2=0x55a03f3ad150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:57.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1045587812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 0x55a03f3ad150 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f2078031440 tx=0x7f2078003f50 comp rx=0 tx=0).stop 2024-04-23T18:38:57.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f208cdd7740 1 -- 172.21.15.50:0/1045587812 shutdown_connections 2024-04-23T18:38:57.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1045587812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3bc480 0x55a03f3b7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:57.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1045587812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 0x55a03f3ad150 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:57.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f208cdd7740 1 --2- 172.21.15.50:0/1045587812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3b1490 0x55a03f3c81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:57.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f208cdd7740 1 -- 172.21.15.50:0/1045587812 >> 172.21.15.50:0/1045587812 conn(0x55a03f3ad7b0 msgr2=0x55a03f3b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:57.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f208cdd7740 1 -- 172.21.15.50:0/1045587812 shutdown_connections 2024-04-23T18:38:57.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.255+0000 7f208cdd7740 1 -- 172.21.15.50:0/1045587812 wait complete. 2024-04-23T18:38:57.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f208cdd7740 1 -- start start 2024-04-23T18:38:57.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f208cdd7740 1 Processor -- start 2024-04-23T18:38:57.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f208cdd7740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3b1490 0x55a03f3c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:57.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f208cdd7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 0x55a03f3c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:57.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f208cdd7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3bc480 0x55a03f3bf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:57.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f208cdd7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f20841ac280 con 0x55a03f3b1db0 2024-04-23T18:38:57.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f208cdd7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f20841ad6f0 con 0x55a03f3bc480 2024-04-23T18:38:57.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f208cdd7740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f20841ac950 con 0x55a03f3b1490 2024-04-23T18:38:57.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f208cdd7740 10 client.0 _ll_register_callbacks cb 0x7f207c0127f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:38:57.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f2082ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 0x55a03f3c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:57.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f2082ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 0x55a03f3c4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46546/0 (socket says 172.21.15.50:46546) 2024-04-23T18:38:57.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f2082ffd640 1 -- 172.21.15.50:0/4275608421 learned_addr learned my addr 172.21.15.50:0/4275608421 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:57.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f2083fff640 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3bc480 0x55a03f3bf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:57.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.256+0000 7f20837fe640 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3b1490 0x55a03f3c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:57.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2082ffd640 1 -- 172.21.15.50:0/4275608421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3b1490 msgr2=0x55a03f3c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:57.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2082ffd640 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3b1490 0x55a03f3c46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:57.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2082ffd640 1 -- 172.21.15.50:0/4275608421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3bc480 msgr2=0x55a03f3bf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:38:57.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2082ffd640 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3bc480 0x55a03f3bf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:57.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2082ffd640 1 -- 172.21.15.50:0/4275608421 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a03f3cec50 con 0x55a03f3b1db0 2024-04-23T18:38:57.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2083fff640 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3bc480 0x55a03f3bf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:38:57.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2082ffd640 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 0x55a03f3c4c30 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f20780b95a0 tx=0x7f20780bcad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee1e03d830cb8cb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:57.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2062ffd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:38:57.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2062ffd640 1 -- 172.21.15.50:0/4275608421 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2078049070 con 0x55a03f3b1db0 2024-04-23T18:38:57.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f208cdd7740 1 -- 172.21.15.50:0/4275608421 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20780bcbd0 con 0x55a03f3b1db0 2024-04-23T18:38:57.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2062ffd640 1 -- 172.21.15.50:0/4275608421 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7f20780b9240 con 0x55a03f3b1db0 2024-04-23T18:38:57.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2062ffd640 1 -- 172.21.15.50:0/4275608421 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f20780c0f80 con 0x55a03f3b1db0 2024-04-23T18:38:57.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f208cdd7740 10 client.17988 fetch_fsmap learned FSMap version 62 2024-04-23T18:38:57.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f208cdd7740 1 -- 172.21.15.50:0/4275608421 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55a03f433310 con 0x55a03f3b1db0 2024-04-23T18:38:57.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2062ffd640 1 -- 172.21.15.50:0/4275608421 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20780bcbd0 con 0x55a03f3b1db0 2024-04-23T18:38:57.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f2062ffd640 1 -- 172.21.15.50:0/4275608421 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f20780bce30 con 0x55a03f3b1db0 2024-04-23T18:38:57.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f208cdd7740 10 client.17988 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:38:57.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f208cdd7740 10 client.17988 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:38:57.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f208cdd7740 20 client.17988 populate_metadata read hostname 'smithi050' 2024-04-23T18:38:57.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f208cdd7740 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a03f433790 0x55a03f435b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:57.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f208cdd7740 4 client.17988 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "525eaceb", "release": true, "await": true}] 2024-04-23T18:38:57.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.257+0000 7f208cdd7740 1 -- 172.21.15.50:0/4275608421 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "525eaceb", "release": true, "await": true}) -- 0x55a03f3240f0 con 0x55a03f433790 2024-04-23T18:38:57.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.258+0000 7f20837fe640 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a03f433790 0x55a03f435b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:57.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.258+0000 7f2062ffd640 10 client.17988 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:57.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.258+0000 7f20837fe640 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a03f433790 0x55a03f435b80 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b92df2e00a939fe3 server_cookie=bdded998ebc7012d in_seq=0 out_seq=0 2024-04-23T18:38:57.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f2062ffd640 1 -- 172.21.15.50:0/4275608421 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55a03f3240f0 con 0x55a03f433790 2024-04-23T18:38:57.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f2062ffd640 10 client.17988 handle_command_reply: tid=0 2024-04-23T18:38:57.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f208cdd7740 1 client.17988 shutdown 2024-04-23T18:38:57.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f208cdd7740 1 -- 172.21.15.50:0/4275608421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 msgr2=0x55a03f3c4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f208cdd7740 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 0x55a03f3c4c30 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f20780b95a0 tx=0x7f20780bcad0 comp rx=0 tx=0).stop 2024-04-23T18:38:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f208cdd7740 1 -- 172.21.15.50:0/4275608421 shutdown_connections 2024-04-23T18:38:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f208cdd7740 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55a03f433790 0x55a03f435b80 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:57.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f208cdd7740 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a03f3bc480 0x55a03f3bf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:57.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f208cdd7740 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55a03f3b1db0 0x55a03f3c4c30 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:57.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f208cdd7740 1 --2- 172.21.15.50:0/4275608421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a03f3b1490 0x55a03f3c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:57.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f2062ffd640 0 client.17988 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:38:57.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f208cdd7740 1 -- 172.21.15.50:0/4275608421 >> 172.21.15.50:0/4275608421 conn(0x55a03f3ad7b0 msgr2=0x55a03f3b2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:57.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f208cdd7740 1 -- 172.21.15.50:0/4275608421 shutdown_connections 2024-04-23T18:38:57.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f208cdd7740 1 -- 172.21.15.50:0/4275608421 wait complete. 2024-04-23T18:38:58.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.287+0000 7f208cdd7740 20 client.17988 trim_cache size 0 max 16384 2024-04-23T18:38:58.042 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:38:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 62, 2024-04-23T18:38:58.075 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:38:58.083 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 68, 2024-04-23T18:38:58.092 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:38:58.102 INFO:teuthology.orchestra.run.smithi050.stdout: "525eaceb": { 2024-04-23T18:38:58.111 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 68, 2024-04-23T18:38:58.118 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:38:58.127 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:38:58.136 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:38:58.144 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:38:58.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:38:58.176 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:38:58.183 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 87.0, 2024-04-23T18:38:58.190 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:38:58.197 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:38:58.258 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:38:58.316 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:38:58.323 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:38:58.330 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:38:58.337 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:38:58.343 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:38:58.353 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:38:58.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:38:58.366 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:38:58.376 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:38:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.288+0000 7f2089fef640 1 -- 172.21.15.50:0/1952679334 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f206c0a3d20 msgr2=0x7f206c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.288+0000 7f2089fef640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f206c0a3d20 0x7f206c0a61e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f207c022000 tx=0x7f207c021000 comp rx=0 tx=0).stop 2024-04-23T18:38:58.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.288+0000 7f2089fef640 1 -- 172.21.15.50:0/1952679334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084106170 msgr2=0x7f2084198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:58.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.288+0000 7f2089fef640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084106170 0x7f2084198c80 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f2078006010 tx=0x7f2078004060 comp rx=0 tx=0).stop 2024-04-23T18:38:58.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.288+0000 7f2089fef640 1 -- 172.21.15.50:0/1952679334 shutdown_connections 2024-04-23T18:38:58.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.288+0000 7f2089fef640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f206c0a3d20 0x7f206c0a61e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:58.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.288+0000 7f2089fef640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f208419d000 0x7f208419d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:58.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.288+0000 7f2089fef640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084106170 0x7f2084198c80 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:58.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.288+0000 7f2089fef640 1 --2- 172.21.15.50:0/1952679334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20841010a0 0x7f2084198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:58.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.288+0000 7f2089fef640 1 -- 172.21.15.50:0/1952679334 >> 172.21.15.50:0/1952679334 conn(0x7f2084077eb0 msgr2=0x7f208406a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.288+0000 7f2089fef640 1 -- 172.21.15.50:0/1952679334 shutdown_connections 2024-04-23T18:38:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:54.288+0000 7f2089fef640 1 -- 172.21.15.50:0/1952679334 wait complete. 2024-04-23T18:38:58.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:54 smithi050 ceph-mon[29187]: pgmap v648: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 5.1 MiB/s wr, 175 op/s 2024-04-23T18:38:58.610 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:54 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/29838468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:38:58.617 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '525eaceb', seconds elapsed: 42.6 2024-04-23T18:38:58.627 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.0 2024-04-23T18:38:58.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:38:58.642 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167320 18.90 MB/sec execute 328 sec latency 36125.803 ms 2024-04-23T18:38:58.650 INFO:tasks.workunit.client.0.smithi050.stdout: 10 167865 18.94 MB/sec execute 329 sec latency 116.362 ms 2024-04-23T18:38:58.672 INFO:tasks.workunit.client.0.smithi050.stdout: 10 168372 18.97 MB/sec execute 330 sec latency 76.498 ms 2024-04-23T18:38:58.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:56 smithi050 ceph-mon[29187]: pgmap v649: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 3.4 MiB/s wr, 127 op/s 2024-04-23T18:38:58.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:54 smithi188 ceph-mon[32694]: pgmap v648: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 5.1 MiB/s wr, 175 op/s 2024-04-23T18:38:58.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:54 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/29838468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:38:58.701 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:56 smithi188 ceph-mon[32694]: pgmap v649: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 3.4 MiB/s wr, 127 op/s 2024-04-23T18:38:58.766 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:38:58 smithi188 ceph-mon[32694]: pgmap v650: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 196 op/s 2024-04-23T18:38:58.796 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:54 smithi177 ceph-mon[32308]: pgmap v648: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 5.1 MiB/s wr, 175 op/s 2024-04-23T18:38:58.805 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:54 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/29838468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:38:58.816 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:56 smithi177 ceph-mon[32308]: pgmap v649: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 3.4 MiB/s wr, 127 op/s 2024-04-23T18:38:58.829 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:38:58 smithi177 ceph-mon[32308]: pgmap v650: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 196 op/s 2024-04-23T18:38:58.838 INFO:tasks.workunit.client.0.smithi050.stdout: 10 168848 18.98 MB/sec execute 331 sec latency 100.178 ms 2024-04-23T18:38:58.846 INFO:tasks.workunit.client.0.smithi050.stdout: 10 169317 18.99 MB/sec execute 332 sec latency 97.674 ms 2024-04-23T18:38:58.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:38:58 smithi050 ceph-mon[29187]: pgmap v650: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 196 op/s 2024-04-23T18:38:58.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.002+0000 7fe0cbae8640 1 -- 172.21.15.50:0/1731238927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0c41021a0 msgr2=0x7fe0c4102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.002+0000 7fe0cbae8640 1 --2- 172.21.15.50:0/1731238927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0c41021a0 0x7fe0c4102580 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0009900 tx=0x7fe0c002a410 comp rx=0 tx=0).stop 2024-04-23T18:38:59.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.002+0000 7fe0cbae8640 1 -- 172.21.15.50:0/1731238927 shutdown_connections 2024-04-23T18:38:59.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.002+0000 7fe0cbae8640 1 --2- 172.21.15.50:0/1731238927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0c4107270 0x7fe0c410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:59.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.002+0000 7fe0cbae8640 1 --2- 172.21.15.50:0/1731238927 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe0c4102b50 0x7fe0c4106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.002+0000 7fe0cbae8640 1 --2- 172.21.15.50:0/1731238927 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0c41021a0 0x7fe0c4102580 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:59.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.002+0000 7fe0cbae8640 1 -- 172.21.15.50:0/1731238927 >> 172.21.15.50:0/1731238927 conn(0x7fe0c4079350 msgr2=0x7fe0c4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:59.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.002+0000 7fe0cbae8640 1 -- 172.21.15.50:0/1731238927 shutdown_connections 2024-04-23T18:38:59.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.002+0000 7fe0cbae8640 1 -- 172.21.15.50:0/1731238927 wait complete. 2024-04-23T18:38:59.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0cbae8640 1 Processor -- start 2024-04-23T18:38:59.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0cbae8640 1 -- start start 2024-04-23T18:38:59.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0cbae8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0c41021a0 0x7fe0c419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:59.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0cbae8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0c4107270 0x7fe0c419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:59.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0cbae8640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe0c41985d0 0x7fe0c4198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:59.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0c9083640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0c4107270 0x7fe0c419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:59.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0c9884640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0c41021a0 0x7fe0c419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:59.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0c9083640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0c4107270 0x7fe0c419e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46568/0 (socket says 172.21.15.50:46568) 2024-04-23T18:38:59.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0c9083640 1 -- 172.21.15.50:0/800289047 learned_addr learned my addr 172.21.15.50:0/800289047 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:38:59.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c41121d0 con 0x7fe0c4107270 2024-04-23T18:38:59.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c4112050 con 0x7fe0c41021a0 2024-04-23T18:38:59.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c4112350 con 0x7fe0c41985d0 2024-04-23T18:38:59.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.003+0000 7fe0ca085640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe0c41985d0 0x7fe0c4198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:59.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.004+0000 7fe0c9083640 1 -- 172.21.15.50:0/800289047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe0c41985d0 msgr2=0x7fe0c4198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:59.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.004+0000 7fe0c9083640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe0c41985d0 0x7fe0c4198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:59.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.004+0000 7fe0c9083640 1 -- 172.21.15.50:0/800289047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0c41021a0 msgr2=0x7fe0c419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.004+0000 7fe0c9083640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0c41021a0 0x7fe0c419e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:59.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.004+0000 7fe0c9083640 1 -- 172.21.15.50:0/800289047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c41990f0 con 0x7fe0c4107270 2024-04-23T18:38:59.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.004+0000 7fe0ca085640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe0c41985d0 0x7fe0c4198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:38:59.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.004+0000 7fe0c9083640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0c4107270 0x7fe0c419e960 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400e9f0 tx=0x7fe0b400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:59.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.004+0000 7fe0baffd640 1 -- 172.21.15.50:0/800289047 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b400ccc0 con 0x7fe0c4107270 2024-04-23T18:38:59.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.005+0000 7fe0baffd640 1 -- 172.21.15.50:0/800289047 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0b40054d0 con 0x7fe0c4107270 2024-04-23T18:38:59.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.005+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c41993e0 con 0x7fe0c4107270 2024-04-23T18:38:59.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.005+0000 7fe0baffd640 1 -- 172.21.15.50:0/800289047 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4010430 con 0x7fe0c4107270 2024-04-23T18:38:59.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.005+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c41a5580 con 0x7fe0c4107270 2024-04-23T18:38:59.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.008+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c4068d70 con 0x7fe0c4107270 2024-04-23T18:38:59.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.009+0000 7fe0baffd640 1 -- 172.21.15.50:0/800289047 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe0b40046c0 con 0x7fe0c4107270 2024-04-23T18:38:59.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.010+0000 7fe0baffd640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe0b00a3d20 0x7fe0b00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:38:59.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.010+0000 7fe0baffd640 1 -- 172.21.15.50:0/800289047 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fe0b40d0b50 con 0x7fe0c4107270 2024-04-23T18:38:59.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.010+0000 7fe0c9884640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe0b00a3d20 0x7fe0b00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:38:59.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.011+0000 7fe0c9884640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe0b00a3d20 0x7fe0b00a61e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0031440 tx=0x7fe0c0005b50 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:38:59.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.016+0000 7fe0baffd640 1 -- 172.21.15.50:0/800289047 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe0b409d1e0 con 0x7fe0c4107270 2024-04-23T18:38:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.361+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0c410a4a0 con 0x7fe0c4107270 2024-04-23T18:38:59.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.362+0000 7fe0baffd640 1 -- 172.21.15.50:0/800289047 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+11875 (secure 0 0 0) 0x7fe0b409cae0 con 0x7fe0c4107270 2024-04-23T18:38:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:38:59.810 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":62,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:36:29.984583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27194":{"gid":27194,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2754597719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2754597719},{"type":"v1","addr":"172.21.15.177:6837","nonce":2754597719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:38:59.820 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 62 2024-04-23T18:38:59.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.365+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe0b00a3d20 msgr2=0x7fe0b00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:59.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.365+0000 7fe0cbae8640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe0b00a3d20 0x7fe0b00a61e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0031440 tx=0x7fe0c0005b50 comp rx=0 tx=0).stop 2024-04-23T18:38:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.365+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0c4107270 msgr2=0x7fe0c419e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:38:59.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.365+0000 7fe0cbae8640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0c4107270 0x7fe0c419e960 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400e9f0 tx=0x7fe0b400edf0 comp rx=0 tx=0).stop 2024-04-23T18:38:59.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.365+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 shutdown_connections 2024-04-23T18:38:59.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.365+0000 7fe0cbae8640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe0b00a3d20 0x7fe0b00a61e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:59.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.365+0000 7fe0cbae8640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe0c41985d0 0x7fe0c4198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:59.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.365+0000 7fe0cbae8640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0c4107270 0x7fe0c419e960 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:59.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.365+0000 7fe0cbae8640 1 --2- 172.21.15.50:0/800289047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0c41021a0 0x7fe0c419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:38:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.365+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 >> 172.21.15.50:0/800289047 conn(0x7fe0c4079350 msgr2=0x7fe0c41003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:38:59.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.365+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 shutdown_connections 2024-04-23T18:38:59.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:38:59.365+0000 7fe0cbae8640 1 -- 172.21.15.50:0/800289047 wait complete. 2024-04-23T18:39:00.008 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db / --timeout 60 --expiration 87.0 --await-for 42.0' 2024-04-23T18:39:00.021 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db / --timeout 60 --expiration 87.0 --await-for 42.0 2024-04-23T18:39:00.099 INFO:tasks.workunit.client.0.smithi050.stdout: 10 169896 19.02 MB/sec execute 333 sec latency 90.608 ms 2024-04-23T18:39:00.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.270+0000 7fb00959a640 1 -- 172.21.15.50:0/1058957034 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0040318a0 con 0x7fb00c107250 2024-04-23T18:39:00.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.270+0000 7fb010a59640 1 -- 172.21.15.50:0/1058957034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb00c107250 msgr2=0x7fb00c10ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:00.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.270+0000 7fb010a59640 1 --2- 172.21.15.50:0/1058957034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb00c107250 0x7fb00c10ee20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb004009900 tx=0x7fb00402a410 comp rx=0 tx=0).stop 2024-04-23T18:39:00.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.270+0000 7fb010a59640 1 -- 172.21.15.50:0/1058957034 shutdown_connections 2024-04-23T18:39:00.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.270+0000 7fb010a59640 1 --2- 172.21.15.50:0/1058957034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb00c107250 0x7fb00c10ee20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:00.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.270+0000 7fb010a59640 1 --2- 172.21.15.50:0/1058957034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb00c102b30 0x7fb00c106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:00.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.270+0000 7fb010a59640 1 --2- 172.21.15.50:0/1058957034 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c102180 0x7fb00c102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:00.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.270+0000 7fb010a59640 1 -- 172.21.15.50:0/1058957034 >> 172.21.15.50:0/1058957034 conn(0x7fb00c079350 msgr2=0x7fb00c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:00.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.271+0000 7fb010a59640 1 -- 172.21.15.50:0/1058957034 shutdown_connections 2024-04-23T18:39:00.458 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.272+0000 7fb010a59640 1 -- 172.21.15.50:0/1058957034 wait complete. 2024-04-23T18:39:00.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb010a59640 1 Processor -- start 2024-04-23T18:39:00.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb010a59640 1 -- start start 2024-04-23T18:39:00.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb010a59640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb00c102180 0x7fb00c1164f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:00.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb010a59640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c107250 0x7fb00c116a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:00.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb010a59640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb00c111650 0x7fb00c111b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:00.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb010a59640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c110000 con 0x7fb00c107250 2024-04-23T18:39:00.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb010a59640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c10fe80 con 0x7fb00c102180 2024-04-23T18:39:00.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb010a59640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c110180 con 0x7fb00c111650 2024-04-23T18:39:00.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb00a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb00c102180 0x7fb00c1164f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:00.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb00ad9d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb00c111650 0x7fb00c111b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:00.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb009d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c107250 0x7fb00c116a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:00.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb00a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb00c102180 0x7fb00c1164f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:35674/0 (socket says 172.21.15.50:35674) 2024-04-23T18:39:00.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb00ad9d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb00c111650 0x7fb00c111b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:55340/0 (socket says 172.21.15.50:55340) 2024-04-23T18:39:00.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.273+0000 7fb00a59c640 1 -- 172.21.15.50:0/93195567 learned_addr learned my addr 172.21.15.50:0/93195567 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:00.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.274+0000 7fb009d9b640 1 -- 172.21.15.50:0/93195567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb00c111650 msgr2=0x7fb00c111b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.274+0000 7fb009d9b640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb00c111650 0x7fb00c111b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:00.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.274+0000 7fb009d9b640 1 -- 172.21.15.50:0/93195567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb00c102180 msgr2=0x7fb00c1164f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:00.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.274+0000 7fb009d9b640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb00c102180 0x7fb00c1164f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:00.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.274+0000 7fb009d9b640 1 -- 172.21.15.50:0/93195567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00c112040 con 0x7fb00c107250 2024-04-23T18:39:00.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.274+0000 7fb00ad9d640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb00c111650 0x7fb00c111b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:39:00.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.274+0000 7fb009d9b640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c107250 0x7fb00c116a30 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fb00000e9f0 tx=0x7fb00000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:00.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.274+0000 7faffb7fe640 1 -- 172.21.15.50:0/93195567 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00000ccc0 con 0x7fb00c107250 2024-04-23T18:39:00.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.274+0000 7fb010a59640 1 -- 172.21.15.50:0/93195567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb00c112330 con 0x7fb00c107250 2024-04-23T18:39:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.274+0000 7fb010a59640 1 -- 172.21.15.50:0/93195567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb00c1b44a0 con 0x7fb00c107250 2024-04-23T18:39:00.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.274+0000 7faffb7fe640 1 -- 172.21.15.50:0/93195567 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb00000ce60 con 0x7fb00c107250 2024-04-23T18:39:01.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.274+0000 7faffb7fe640 1 -- 172.21.15.50:0/93195567 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000010430 con 0x7fb00c107250 2024-04-23T18:39:01.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.277+0000 7fb01376d740 1 Processor -- start 2024-04-23T18:39:01.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.277+0000 7fb01376d740 1 -- start start 2024-04-23T18:39:01.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.277+0000 7faffb7fe640 1 -- 172.21.15.50:0/93195567 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb000004f60 con 0x7fb00c107250 2024-04-23T18:39:01.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb01376d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c96d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:01.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb01376d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d9c0 0x55c11c977b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:01.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb01376d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c978090 0x55c11c97a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:01.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb01376d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c11c8d2370 con 0x55c11c96d9c0 2024-04-23T18:39:01.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb01376d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c11c86f2e0 con 0x55c11c96d0a0 2024-04-23T18:39:01.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb01376d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c11c8ce330 con 0x55c11c978090 2024-04-23T18:39:01.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb00ad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c96d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:01.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb00ad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c96d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:35680/0 (socket says 172.21.15.50:35680) 2024-04-23T18:39:01.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb00ad9d640 1 -- 172.21.15.50:0/1045049017 learned_addr learned my addr 172.21.15.50:0/1045049017 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb00a59c640 1 --2- 172.21.15.50:0/1045049017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d9c0 0x55c11c977b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:01.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb009d9b640 1 --2- 172.21.15.50:0/1045049017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c978090 0x55c11c97a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:01.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb00ad9d640 1 -- 172.21.15.50:0/1045049017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c978090 msgr2=0x55c11c97a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:01.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb00ad9d640 1 --2- 172.21.15.50:0/1045049017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c978090 0x55c11c97a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:01.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb00ad9d640 1 -- 172.21.15.50:0/1045049017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d9c0 msgr2=0x55c11c977b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:01.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb00ad9d640 1 --2- 172.21.15.50:0/1045049017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d9c0 0x55c11c977b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:01.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.278+0000 7fb00ad9d640 1 -- 172.21.15.50:0/1045049017 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c11c6b0d80 con 0x55c11c96d0a0 2024-04-23T18:39:01.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.279+0000 7fb00ad9d640 1 --2- 172.21.15.50:0/1045049017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c96d480 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb00402aa00 tx=0x7fb0040044e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f80b92039f1e955 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:01.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.279+0000 7faff8ff9640 1 -- 172.21.15.50:0/1045049017 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004040070 con 0x55c11c96d0a0 2024-04-23T18:39:01.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.279+0000 7faff8ff9640 1 -- 172.21.15.50:0/1045049017 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0040047a0 con 0x55c11c96d0a0 2024-04-23T18:39:01.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.279+0000 7faff8ff9640 1 -- 172.21.15.50:0/1045049017 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004035040 con 0x55c11c96d0a0 2024-04-23T18:39:01.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.279+0000 7faffb7fe640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faff40a3e70 0x7faff40a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:01.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.279+0000 7fb00a59c640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faff40a3e70 0x7faff40a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:02.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.279+0000 7faffb7fe640 1 -- 172.21.15.50:0/93195567 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fb0000d0800 con 0x7fb00c107250 2024-04-23T18:39:02.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.280+0000 7fb01376d740 1 -- 172.21.15.50:0/1045049017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 msgr2=0x55c11c96d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:02.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.280+0000 7fb01376d740 1 --2- 172.21.15.50:0/1045049017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c96d480 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb00402aa00 tx=0x7fb0040044e0 comp rx=0 tx=0).stop 2024-04-23T18:39:02.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.280+0000 7fb00a59c640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faff40a3e70 0x7faff40a6330 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7faffc009fd0 tx=0x7faffc009800 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:02.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.280+0000 7fb01376d740 1 -- 172.21.15.50:0/1045049017 shutdown_connections 2024-04-23T18:39:02.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.280+0000 7fb01376d740 1 --2- 172.21.15.50:0/1045049017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c978090 0x55c11c97a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:02.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.280+0000 7fb01376d740 1 --2- 172.21.15.50:0/1045049017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d9c0 0x55c11c977b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.280+0000 7fb01376d740 1 --2- 172.21.15.50:0/1045049017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c96d480 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:02.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.280+0000 7fb01376d740 1 -- 172.21.15.50:0/1045049017 >> 172.21.15.50:0/1045049017 conn(0x55c11c9693c0 msgr2=0x55c11c9697a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:02.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.280+0000 7fb01376d740 1 -- 172.21.15.50:0/1045049017 shutdown_connections 2024-04-23T18:39:02.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.280+0000 7fb01376d740 1 -- 172.21.15.50:0/1045049017 wait complete. 2024-04-23T18:39:02.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.280+0000 7fb01376d740 1 -- start start 2024-04-23T18:39:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.280+0000 7fb01376d740 1 Processor -- start 2024-04-23T18:39:02.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb01376d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c974ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:02.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb01376d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c96d9c0 0x55c11c975530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:02.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb01376d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c978090 0x55c11c975a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb01376d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c11c97ae20 con 0x55c11c978090 2024-04-23T18:39:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb01376d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c11c86cfb0 con 0x55c11c96d0a0 2024-04-23T18:39:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb01376d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c11c97afa0 con 0x55c11c96d9c0 2024-04-23T18:39:02.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb01376d740 10 client.0 _ll_register_callbacks cb 0x55c11c86c9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:39:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb00a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c974ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:02.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb00a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c974ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:35692/0 (socket says 172.21.15.50:35692) 2024-04-23T18:39:02.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb00a59c640 1 -- 172.21.15.50:0/3626406493 learned_addr learned my addr 172.21.15.50:0/3626406493 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:02.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb009d9b640 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c96d9c0 0x55c11c975530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:02.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb00ad9d640 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c978090 0x55c11c975a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:02.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb00a59c640 1 -- 172.21.15.50:0/3626406493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c96d9c0 msgr2=0x55c11c975530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:02.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb00a59c640 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c96d9c0 0x55c11c975530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:02.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb00a59c640 1 -- 172.21.15.50:0/3626406493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c978090 msgr2=0x55c11c975a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:02.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.281+0000 7fb00a59c640 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c978090 0x55c11c975a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:02.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.282+0000 7fb00a59c640 1 -- 172.21.15.50:0/3626406493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c11c711510 con 0x55c11c96d0a0 2024-04-23T18:39:02.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.282+0000 7fb00a59c640 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c974ff0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faffc012010 tx=0x7faffc004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5563ebfe50a6d141 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:02.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.282+0000 7faff1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:39:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.285+0000 7faff1ffb640 1 -- 172.21.15.50:0/3626406493 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc02c060 con 0x55c11c96d0a0 2024-04-23T18:39:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.285+0000 7faff1ffb640 1 -- 172.21.15.50:0/3626406493 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7faffc017030 con 0x55c11c96d0a0 2024-04-23T18:39:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.285+0000 7fb01376d740 1 -- 172.21.15.50:0/3626406493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0040047a0 con 0x55c11c96d0a0 2024-04-23T18:39:02.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.286+0000 7faff1ffb640 1 -- 172.21.15.50:0/3626406493 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7faffc01fb40 con 0x55c11c96d0a0 2024-04-23T18:39:02.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.286+0000 7fb01376d740 10 client.27488 fetch_fsmap learned FSMap version 62 2024-04-23T18:39:02.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.286+0000 7fb01376d740 1 -- 172.21.15.50:0/3626406493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55c11c5a4670 con 0x55c11c96d0a0 2024-04-23T18:39:02.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.286+0000 7faff1ffb640 1 -- 172.21.15.50:0/3626406493 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0040047a0 con 0x55c11c96d0a0 2024-04-23T18:39:02.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.286+0000 7faff1ffb640 1 -- 172.21.15.50:0/3626406493 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7faffc02f3c0 con 0x55c11c96d0a0 2024-04-23T18:39:02.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.286+0000 7fb01376d740 10 client.27488 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:39:02.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.286+0000 7fb01376d740 10 client.27488 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:39:02.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.287+0000 7fb01376d740 20 client.27488 populate_metadata read hostname 'smithi050' 2024-04-23T18:39:02.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.287+0000 7fb01376d740 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c11c988420 0x55c11c98a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:02.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.287+0000 7fb01376d740 4 client.27488 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:39:02.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.287+0000 7fb01376d740 1 -- 172.21.15.50:0/3626406493 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c11c8dfd90 con 0x55c11c988420 2024-04-23T18:39:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.287+0000 7fb009d9b640 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c11c988420 0x55c11c98a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:02.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.287+0000 7faff1ffb640 10 client.27488 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:02.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.287+0000 7fb009d9b640 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c11c988420 0x55c11c98a810 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b00147f28832a553 server_cookie=6ace8a49107909e7 in_seq=0 out_seq=0 2024-04-23T18:39:02.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.289+0000 7faff1ffb640 1 -- 172.21.15.50:0/3626406493 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c11c8dfd90 con 0x55c11c988420 2024-04-23T18:39:02.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.289+0000 7faff1ffb640 10 client.27488 handle_command_reply: tid=0 2024-04-23T18:39:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7fb01376d740 1 client.27488 shutdown 2024-04-23T18:39:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7fb01376d740 1 -- 172.21.15.50:0/3626406493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 msgr2=0x55c11c974ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:03.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7fb01376d740 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c974ff0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faffc012010 tx=0x7faffc004ab0 comp rx=0 tx=0).stop 2024-04-23T18:39:03.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7fb01376d740 1 -- 172.21.15.50:0/3626406493 shutdown_connections 2024-04-23T18:39:03.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7fb01376d740 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c11c988420 0x55c11c98a810 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7fb01376d740 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c978090 0x55c11c975a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:03.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7fb01376d740 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c96d9c0 0x55c11c975530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:03.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7fb01376d740 1 --2- 172.21.15.50:0/3626406493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c974ff0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:03.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7fb01376d740 1 -- 172.21.15.50:0/3626406493 >> 172.21.15.50:0/3626406493 conn(0x55c11c9693c0 msgr2=0x55c11c968600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:03.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7faff1ffb640 0 client.27488 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:03.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7fb01376d740 1 -- 172.21.15.50:0/3626406493 shutdown_connections 2024-04-23T18:39:03.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7fb01376d740 1 -- 172.21.15.50:0/3626406493 wait complete. 2024-04-23T18:39:03.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.290+0000 7fb01376d740 20 client.27488 trim_cache size 0 max 16384 2024-04-23T18:39:03.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb01376d740 1 Processor -- start 2024-04-23T18:39:03.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb01376d740 1 -- start start 2024-04-23T18:39:03.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb01376d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c968d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:03.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb01376d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d9c0 0x55c11c976f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:03.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb01376d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c978090 0x55c11c990120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:03.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb01376d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fafec0110a0 con 0x55c11c96d9c0 2024-04-23T18:39:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb01376d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fafec00e970 con 0x55c11c96d0a0 2024-04-23T18:39:03.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb01376d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fafec00c7e0 con 0x55c11c978090 2024-04-23T18:39:03.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb009d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d9c0 0x55c11c976f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:03.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb00a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c968d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:03.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb009d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d9c0 0x55c11c976f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46622/0 (socket says 172.21.15.50:46622) 2024-04-23T18:39:03.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb00a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c968d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:35702/0 (socket says 172.21.15.50:35702) 2024-04-23T18:39:03.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.313+0000 7fb009d9b640 1 -- 172.21.15.50:0/4289324324 learned_addr learned my addr 172.21.15.50:0/4289324324 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:03.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.314+0000 7fb00ad9d640 1 --2- 172.21.15.50:0/4289324324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c978090 0x55c11c990120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.314+0000 7fb00a59c640 1 -- 172.21.15.50:0/4289324324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c978090 msgr2=0x55c11c990120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:03.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.314+0000 7fb00a59c640 1 --2- 172.21.15.50:0/4289324324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c978090 0x55c11c990120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:03.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.314+0000 7fb00a59c640 1 -- 172.21.15.50:0/4289324324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d9c0 msgr2=0x55c11c976f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:03.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.314+0000 7fb00a59c640 1 --2- 172.21.15.50:0/4289324324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d9c0 0x55c11c976f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:03.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.314+0000 7fb00a59c640 1 -- 172.21.15.50:0/4289324324 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c11c96d7f0 con 0x55c11c96d0a0 2024-04-23T18:39:03.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.314+0000 7fb00a59c640 1 --2- 172.21.15.50:0/4289324324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c968d80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7faffc02f7e0 tx=0x7faffc009c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35be3ba56183a8c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:03.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.314+0000 7faff3fff640 1 -- 172.21.15.50:0/4289324324 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc035080 con 0x55c11c96d0a0 2024-04-23T18:39:03.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.314+0000 7faff3fff640 1 -- 172.21.15.50:0/4289324324 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faffc02fbd0 con 0x55c11c96d0a0 2024-04-23T18:39:03.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.314+0000 7faff3fff640 1 -- 172.21.15.50:0/4289324324 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc005580 con 0x55c11c96d0a0 2024-04-23T18:39:03.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.316+0000 7fb01376d740 1 -- 172.21.15.50:0/4289324324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 msgr2=0x55c11c968d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:03.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.316+0000 7fb01376d740 1 --2- 172.21.15.50:0/4289324324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c968d80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7faffc02f7e0 tx=0x7faffc009c10 comp rx=0 tx=0).stop 2024-04-23T18:39:03.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.316+0000 7fb01376d740 1 -- 172.21.15.50:0/4289324324 shutdown_connections 2024-04-23T18:39:03.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.316+0000 7fb01376d740 1 --2- 172.21.15.50:0/4289324324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c978090 0x55c11c990120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:03.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.316+0000 7fb01376d740 1 --2- 172.21.15.50:0/4289324324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d9c0 0x55c11c976f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:03.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.316+0000 7fb01376d740 1 --2- 172.21.15.50:0/4289324324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c96d0a0 0x55c11c968d80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:03.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.316+0000 7fb01376d740 1 -- 172.21.15.50:0/4289324324 >> 172.21.15.50:0/4289324324 conn(0x55c11c9693c0 msgr2=0x55c11c976820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:03.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.316+0000 7fb01376d740 1 -- 172.21.15.50:0/4289324324 shutdown_connections 2024-04-23T18:39:03.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.316+0000 7fb01376d740 1 -- 172.21.15.50:0/4289324324 wait complete. 2024-04-23T18:39:03.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.316+0000 7fb01376d740 1 -- start start 2024-04-23T18:39:03.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb01376d740 1 Processor -- start 2024-04-23T18:39:04.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb01376d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d0a0 0x55c11c980350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:04.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb01376d740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c96d9c0 0x55c11c97c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:04.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb01376d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c978090 0x55c11c97cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:04.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb01376d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fafec00ce30 con 0x55c11c96d0a0 2024-04-23T18:39:04.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb01376d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fafec00e2a0 con 0x55c11c978090 2024-04-23T18:39:04.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb01376d740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fafec00d500 con 0x55c11c96d9c0 2024-04-23T18:39:04.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb01376d740 10 client.0 _ll_register_callbacks cb 0x7faffc010810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:39:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb00ad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c978090 0x55c11c97cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:04.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb00a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d0a0 0x55c11c980350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:04.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb00ad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c978090 0x55c11c97cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:35712/0 (socket says 172.21.15.50:35712) 2024-04-23T18:39:04.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb00ad9d640 1 -- 172.21.15.50:0/3892511274 learned_addr learned my addr 172.21.15.50:0/3892511274 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:04.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.317+0000 7fb009d9b640 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c96d9c0 0x55c11c97c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:04.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.318+0000 7fb00ad9d640 1 -- 172.21.15.50:0/3892511274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c96d9c0 msgr2=0x55c11c97c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:04.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.318+0000 7fb00ad9d640 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c96d9c0 0x55c11c97c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:04.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.318+0000 7fb00ad9d640 1 -- 172.21.15.50:0/3892511274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d0a0 msgr2=0x55c11c980350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:04.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.318+0000 7fb00ad9d640 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d0a0 0x55c11c980350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.318+0000 7fb00ad9d640 1 -- 172.21.15.50:0/3892511274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c11c97d1a0 con 0x55c11c978090 2024-04-23T18:39:04.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.318+0000 7fb00a59c640 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d0a0 0x55c11c980350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:39:04.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.318+0000 7fb00ad9d640 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c978090 0x55c11c97cc60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb004031440 tx=0x7fb004003960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f4d7a5b92048f7c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:04.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.318+0000 7faff1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:39:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.318+0000 7faff1ffb640 1 -- 172.21.15.50:0/3892511274 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004053060 con 0x55c11c978090 2024-04-23T18:39:04.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.318+0000 7faff1ffb640 1 -- 172.21.15.50:0/3892511274 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7fb0040314a0 con 0x55c11c978090 2024-04-23T18:39:04.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.318+0000 7fb01376d740 1 -- 172.21.15.50:0/3892511274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faffc02fbd0 con 0x55c11c978090 2024-04-23T18:39:04.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.319+0000 7faff1ffb640 1 -- 172.21.15.50:0/3892511274 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fb004040040 con 0x55c11c978090 2024-04-23T18:39:04.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.319+0000 7fb01376d740 10 client.27494 fetch_fsmap learned FSMap version 62 2024-04-23T18:39:04.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.319+0000 7fb01376d740 1 -- 172.21.15.50:0/3892511274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55c11c9ef6b0 con 0x55c11c978090 2024-04-23T18:39:04.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.319+0000 7faff1ffb640 1 -- 172.21.15.50:0/3892511274 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faffc02fbd0 con 0x55c11c978090 2024-04-23T18:39:04.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.319+0000 7faff1ffb640 1 -- 172.21.15.50:0/3892511274 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fb00404e640 con 0x55c11c978090 2024-04-23T18:39:04.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.319+0000 7fb01376d740 10 client.27494 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:39:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.319+0000 7fb01376d740 10 client.27494 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:39:04.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.319+0000 7fb01376d740 20 client.27494 populate_metadata read hostname 'smithi050' 2024-04-23T18:39:04.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.319+0000 7fb01376d740 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c11c9efb00 0x55c11c9f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:04.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.319+0000 7fb01376d740 4 client.27494 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.0, "await_for": 42.0}] 2024-04-23T18:39:04.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.319+0000 7fb01376d740 1 -- 172.21.15.50:0/3892511274 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.0, "await_for": 42.0}) -- 0x55c11c8d9c00 con 0x55c11c9efb00 2024-04-23T18:39:04.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.319+0000 7fb00a59c640 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c11c9efb00 0x55c11c9f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:04.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.320+0000 7faff1ffb640 10 client.27494 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:04.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.320+0000 7fb00a59c640 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c11c9efb00 0x55c11c9f1ef0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f18c440d19621ca9 server_cookie=2118e87f502cd6f5 in_seq=0 out_seq=0 2024-04-23T18:39:04.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:00 smithi050 ceph-mon[29187]: pgmap v651: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.8 MiB/s wr, 184 op/s 2024-04-23T18:39:04.803 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:00 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/800289047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:39:04.813 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170381 19.06 MB/sec execute 334 sec latency 196.179 ms 2024-04-23T18:39:04.883 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:00 smithi188 ceph-mon[32694]: pgmap v651: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.8 MiB/s wr, 184 op/s 2024-04-23T18:39:04.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:00 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/800289047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:39:04.900 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:01 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:39:04.909 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:02 smithi188 ceph-mon[32694]: pgmap v652: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.9 MiB/s wr, 156 op/s 2024-04-23T18:39:04.982 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:05.055 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:05.068 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: pgmap v653: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 352 op/s 2024-04-23T18:39:05.078 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:05.089 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:05.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:05.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:05.128 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.139 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.152 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.219 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.267 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.280 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.294 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.357 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:03 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.372 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:00 smithi177 ceph-mon[32308]: pgmap v651: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.8 MiB/s wr, 184 op/s 2024-04-23T18:39:05.384 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:00 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/800289047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:39:05.497 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:01 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:39:05.507 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:02 smithi177 ceph-mon[32308]: pgmap v652: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.9 MiB/s wr, 156 op/s 2024-04-23T18:39:05.518 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:05.528 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:05.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: pgmap v653: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 352 op/s 2024-04-23T18:39:05.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:05.556 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:05.567 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:05.637 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:05.652 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.663 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.730 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.743 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.756 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.769 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.782 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.796 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:03 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:05.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7faff1ffb640 1 -- 172.21.15.50:0/3892511274 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55c11c8d9c00 con 0x55c11c9efb00 2024-04-23T18:39:05.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7faff1ffb640 10 client.27494 handle_command_reply: tid=0 2024-04-23T18:39:05.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7fb01376d740 1 client.27494 shutdown 2024-04-23T18:39:05.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7fb01376d740 1 -- 172.21.15.50:0/3892511274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c978090 msgr2=0x55c11c97cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:05.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7fb01376d740 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c978090 0x55c11c97cc60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb004031440 tx=0x7fb004003960 comp rx=0 tx=0).stop 2024-04-23T18:39:05.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7fb01376d740 1 -- 172.21.15.50:0/3892511274 shutdown_connections 2024-04-23T18:39:05.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7fb01376d740 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c11c9efb00 0x55c11c9f1ef0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:05.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7fb01376d740 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c11c978090 0x55c11c97cc60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:05.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7fb01376d740 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c11c96d9c0 0x55c11c97c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7faff1ffb640 0 client.27494 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:06.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7fb01376d740 1 --2- 172.21.15.50:0/3892511274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c11c96d0a0 0x55c11c980350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:06.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7fb01376d740 1 -- 172.21.15.50:0/3892511274 >> 172.21.15.50:0/3892511274 conn(0x55c11c9693c0 msgr2=0x55c11c9770b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:06.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7fb01376d740 1 -- 172.21.15.50:0/3892511274 shutdown_connections 2024-04-23T18:39:06.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7fb01376d740 1 -- 172.21.15.50:0/3892511274 wait complete. 2024-04-23T18:39:06.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.867+0000 7fb01376d740 20 client.27494 trim_cache size 0 max 16384 2024-04-23T18:39:06.130 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:39:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 62, 2024-04-23T18:39:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:39:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 70, 2024-04-23T18:39:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:39:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: "71fed75b": { 2024-04-23T18:39:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 70, 2024-04-23T18:39:06.255 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:39:06.264 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:39:06.278 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:39:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:39:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:39:06.383 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:39:06.394 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 87.0, 2024-04-23T18:39:06.406 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:39:06.416 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:39:06.431 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:39:06.442 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:39:06.454 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "QUIESCED", 2024-04-23T18:39:06.514 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:39:06.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:39:06.529 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:39:06.541 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:39:06.549 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:39:06.556 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:39:06.567 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:39:06.615 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-04-23T18:39:06.697 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:39:06.760 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-23T18:39:06.774 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: Unable to set osd_memory_target on smithi188 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T18:39:06.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:06.789 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:06.796 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:06.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:06.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-23T18:39:06.817 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-23T18:39:06.882 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:39:06.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:39:06.894 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:04 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:06.901 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:05 smithi188 ceph-mon[32694]: pgmap v654: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 294 op/s 2024-04-23T18:39:06.908 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:05 smithi188 ceph-mon[32694]: osdmap e125: 12 total, 12 up, 12 in 2024-04-23T18:39:06.914 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-04-23T18:39:06.921 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:39:06.975 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-23T18:39:07.047 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: Unable to set osd_memory_target on smithi188 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T18:39:07.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:07.122 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:07.132 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:07.143 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:07.153 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-23T18:39:07.163 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-23T18:39:07.173 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:39:07.183 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:39:07.255 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:04 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:07.264 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:05 smithi177 ceph-mon[32308]: pgmap v654: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 294 op/s 2024-04-23T18:39:07.328 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:05 smithi177 ceph-mon[32308]: osdmap e125: 12 total, 12 up, 12 in 2024-04-23T18:39:07.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.868+0000 7fb010a59640 1 -- 172.21.15.50:0/93195567 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faff40a3e70 msgr2=0x7faff40a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:07.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.868+0000 7fb010a59640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faff40a3e70 0x7faff40a6330 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7faffc009fd0 tx=0x7faffc009800 comp rx=0 tx=0).stop 2024-04-23T18:39:07.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.868+0000 7fb010a59640 1 -- 172.21.15.50:0/93195567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c107250 msgr2=0x7fb00c116a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:07.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.868+0000 7fb010a59640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c107250 0x7fb00c116a30 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fb00000e9f0 tx=0x7fb00000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:39:07.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.868+0000 7fb010a59640 1 -- 172.21.15.50:0/93195567 shutdown_connections 2024-04-23T18:39:07.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.868+0000 7fb010a59640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7faff40a3e70 0x7faff40a6330 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:07.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.868+0000 7fb010a59640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb00c111650 0x7fb00c111b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:07.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.868+0000 7fb010a59640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c107250 0x7fb00c116a30 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:07.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.868+0000 7fb010a59640 1 --2- 172.21.15.50:0/93195567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb00c102180 0x7fb00c1164f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:07.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.868+0000 7fb010a59640 1 -- 172.21.15.50:0/93195567 >> 172.21.15.50:0/93195567 conn(0x7fb00c079350 msgr2=0x7fb00c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:07.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.868+0000 7fb010a59640 1 -- 172.21.15.50:0/93195567 shutdown_connections 2024-04-23T18:39:07.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:00.868+0000 7fb010a59640 1 -- 172.21.15.50:0/93195567 wait complete. 2024-04-23T18:39:07.541 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 19.00 MB/sec execute 335 sec latency 1196.214 ms 2024-04-23T18:39:07.550 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:01 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:39:07.636 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '71fed75b', quiesce took 9.0 seconds. Will release after: 27.0 2024-04-23T18:39:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.132+0000 7f1495d4a640 1 -- 172.21.15.50:0/4290002258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14901072a0 msgr2=0x7f149010ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.132+0000 7f1495d4a640 1 --2- 172.21.15.50:0/4290002258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14901072a0 0x7f149010ee70 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f1488009ca0 tx=0x7f148802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:39:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.132+0000 7f1495d4a640 1 -- 172.21.15.50:0/4290002258 shutdown_connections 2024-04-23T18:39:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.132+0000 7f1495d4a640 1 --2- 172.21.15.50:0/4290002258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14901072a0 0x7f149010ee70 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.132+0000 7f1495d4a640 1 --2- 172.21.15.50:0/4290002258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1490102b80 0x7f1490106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.132+0000 7f1495d4a640 1 --2- 172.21.15.50:0/4290002258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14901021d0 0x7f14901025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:07.744 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.132+0000 7f1495d4a640 1 -- 172.21.15.50:0/4290002258 >> 172.21.15.50:0/4290002258 conn(0x7f14900793e0 msgr2=0x7f14900797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.132+0000 7f1495d4a640 1 -- 172.21.15.50:0/4290002258 shutdown_connections 2024-04-23T18:39:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.132+0000 7f1495d4a640 1 -- 172.21.15.50:0/4290002258 wait complete. 2024-04-23T18:39:07.865 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.132+0000 7f1495d4a640 1 Processor -- start 2024-04-23T18:39:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.132+0000 7f1495d4a640 1 -- start start 2024-04-23T18:39:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1495d4a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14901021d0 0x7f14901986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1495d4a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14901072a0 0x7f1490198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1495d4a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f149019ced0 0x7f149019d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1495d4a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1490112210 con 0x7f14901072a0 2024-04-23T18:39:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1495d4a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1490112090 con 0x7f149019ced0 2024-04-23T18:39:07.996 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1495d4a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1490112390 con 0x7f14901021d0 2024-04-23T18:39:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1486ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14901072a0 0x7f1490198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1486ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14901072a0 0x7f1490198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43772/0 (socket says 172.21.15.50:43772) 2024-04-23T18:39:08.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1486ffd640 1 -- 172.21.15.50:0/84675784 learned_addr learned my addr 172.21.15.50:0/84675784 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f148ffff640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f149019ced0 0x7f149019d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f148f7fe640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14901021d0 0x7f14901986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1486ffd640 1 -- 172.21.15.50:0/84675784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14901021d0 msgr2=0x7f14901986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1486ffd640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14901021d0 0x7f14901986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1486ffd640 1 -- 172.21.15.50:0/84675784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f149019ced0 msgr2=0x7f149019d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:08.156 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1486ffd640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f149019ced0 0x7f149019d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:08.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1486ffd640 1 -- 172.21.15.50:0/84675784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149019d9c0 con 0x7f14901072a0 2024-04-23T18:39:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.133+0000 7f1486ffd640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14901072a0 0x7f1490198bf0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f1478005fd0 tx=0x7f147800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.134+0000 7f148d7fa640 1 -- 172.21.15.50:0/84675784 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1478017070 con 0x7f14901072a0 2024-04-23T18:39:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.134+0000 7f1495d4a640 1 -- 172.21.15.50:0/84675784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f149019dcb0 con 0x7f14901072a0 2024-04-23T18:39:08.199 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.134+0000 7f148d7fa640 1 -- 172.21.15.50:0/84675784 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1478009c30 con 0x7f14901072a0 2024-04-23T18:39:08.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.134+0000 7f148d7fa640 1 -- 172.21.15.50:0/84675784 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1478003d30 con 0x7f14901072a0 2024-04-23T18:39:08.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.134+0000 7f1495d4a640 1 -- 172.21.15.50:0/84675784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14901a54f0 con 0x7f14901072a0 2024-04-23T18:39:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.136+0000 7f148d7fa640 1 -- 172.21.15.50:0/84675784 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1478003ed0 con 0x7f14901072a0 2024-04-23T18:39:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.136+0000 7f1498a5a740 1 Processor -- start 2024-04-23T18:39:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.136+0000 7f1498a5a740 1 -- start start 2024-04-23T18:39:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.136+0000 7f148d7fa640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f147c0a3dc0 0x7f147c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.136+0000 7f1498a5a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd449498e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:08.372 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.136+0000 7f1498a5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd449586d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.136+0000 7f1498a5a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 0x55bd4495b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.136+0000 7f1498a5a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bd448ae970 con 0x55bd44949500 2024-04-23T18:39:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.136+0000 7f1498a5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bd4484b4a0 con 0x55bd44949e20 2024-04-23T18:39:08.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.136+0000 7f1498a5a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bd448aa930 con 0x55bd44958c10 2024-04-23T18:39:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.136+0000 7f148ffff640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f147c0a3dc0 0x7f147c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.136+0000 7f148f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd449498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:08.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f148d7fa640 1 -- 172.21.15.50:0/84675784 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7f14780d03f0 con 0x7f14901072a0 2024-04-23T18:39:08.585 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f148f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd449498e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43778/0 (socket says 172.21.15.50:43778) 2024-04-23T18:39:08.594 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f1486ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd449586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:08.602 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f148f7fe640 1 -- 172.21.15.50:0/1497725368 learned_addr learned my addr 172.21.15.50:0/1497725368 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:08.611 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f148ffff640 1 --2- 172.21.15.50:0/1497725368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 0x55bd4495b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f148f7fe640 1 -- 172.21.15.50:0/1497725368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 msgr2=0x55bd4495b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f148f7fe640 1 --2- 172.21.15.50:0/1497725368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 0x55bd4495b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f148f7fe640 1 -- 172.21.15.50:0/1497725368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 msgr2=0x55bd449586d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f148f7fe640 1 --2- 172.21.15.50:0/1497725368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd449586d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f148f7fe640 1 -- 172.21.15.50:0/1497725368 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd445c1210 con 0x55bd44949500 2024-04-23T18:39:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f1486ffd640 1 --2- 172.21.15.50:0/1497725368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd449586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:39:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f148ffff640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f147c0a3dc0 0x7f147c0a6280 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55bd4494aa90 tx=0x7f14880099f0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:08.779 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f148f7fe640 1 --2- 172.21.15.50:0/1497725368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd449498e0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f148000bd20 tx=0x7f148000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2e73b820a7478e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f1485ffb640 1 -- 172.21.15.50:0/1497725368 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480015070 con 0x55bd44949500 2024-04-23T18:39:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f1485ffb640 1 -- 172.21.15.50:0/1497725368 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f148000be10 con 0x55bd44949500 2024-04-23T18:39:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.137+0000 7f1485ffb640 1 -- 172.21.15.50:0/1497725368 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14800106c0 con 0x55bd44949500 2024-04-23T18:39:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.138+0000 7f1498a5a740 1 -- 172.21.15.50:0/1497725368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 msgr2=0x55bd449498e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:08.965 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.138+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1497725368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd449498e0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f148000bd20 tx=0x7f148000f730 comp rx=0 tx=0).stop 2024-04-23T18:39:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.138+0000 7f1498a5a740 1 -- 172.21.15.50:0/1497725368 shutdown_connections 2024-04-23T18:39:08.982 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.138+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1497725368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 0x55bd4495b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:08.991 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.138+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1497725368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd449586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.138+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1497725368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd449498e0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:09.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.138+0000 7f1498a5a740 1 -- 172.21.15.50:0/1497725368 >> 172.21.15.50:0/1497725368 conn(0x55bd44945b30 msgr2=0x55bd44945f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:09.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.138+0000 7f1498a5a740 1 -- 172.21.15.50:0/1497725368 shutdown_connections 2024-04-23T18:39:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.138+0000 7f1498a5a740 1 -- 172.21.15.50:0/1497725368 wait complete. 2024-04-23T18:39:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.138+0000 7f1498a5a740 1 -- start start 2024-04-23T18:39:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f1498a5a740 1 Processor -- start 2024-04-23T18:39:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f1498a5a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd44958560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f1498a5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd44955850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f1498a5a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 0x55bd44955db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f1498a5a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55bd4495b840 con 0x55bd44949500 2024-04-23T18:39:09.264 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f1498a5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bd448490b0 con 0x55bd44949e20 2024-04-23T18:39:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f1498a5a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55bd4495b9c0 con 0x55bd44958c10 2024-04-23T18:39:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f1498a5a740 10 client.0 _ll_register_callbacks cb 0x55bd44836d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:39:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f148f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd44958560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f148f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd44958560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43792/0 (socket says 172.21.15.50:43792) 2024-04-23T18:39:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f148f7fe640 1 -- 172.21.15.50:0/116231494 learned_addr learned my addr 172.21.15.50:0/116231494 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f1486ffd640 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd44955850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f148ffff640 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 0x55bd44955db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:09.414 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f148f7fe640 1 -- 172.21.15.50:0/116231494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 msgr2=0x55bd44955db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:09.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f148f7fe640 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 0x55bd44955db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:09.427 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f148f7fe640 1 -- 172.21.15.50:0/116231494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 msgr2=0x55bd44955850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:09.435 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f148f7fe640 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd44955850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f148f7fe640 1 -- 172.21.15.50:0/116231494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd4468d620 con 0x55bd44949500 2024-04-23T18:39:09.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f148f7fe640 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd44958560 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f148000bcf0 tx=0x7f1480011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33afb94fdcdc3b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f146effd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:39:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.139+0000 7f146effd640 1 -- 172.21.15.50:0/116231494 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480029060 con 0x55bd44949500 2024-04-23T18:39:09.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f146effd640 1 -- 172.21.15.50:0/116231494 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7f1480011840 con 0x55bd44949500 2024-04-23T18:39:09.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f1498a5a740 1 -- 172.21.15.50:0/116231494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f148000be10 con 0x55bd44949500 2024-04-23T18:39:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f146effd640 1 -- 172.21.15.50:0/116231494 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f1480015050 con 0x55bd44949500 2024-04-23T18:39:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f1498a5a740 10 client.18024 fetch_fsmap learned FSMap version 62 2024-04-23T18:39:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f1498a5a740 1 -- 172.21.15.50:0/116231494 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55bd44967840 con 0x55bd44949500 2024-04-23T18:39:09.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f146effd640 1 -- 172.21.15.50:0/116231494 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f148000be10 con 0x55bd44949500 2024-04-23T18:39:09.695 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f146effd640 1 -- 172.21.15.50:0/116231494 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f148001a460 con 0x55bd44949500 2024-04-23T18:39:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f1498a5a740 10 client.18024 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:39:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f1498a5a740 10 client.18024 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:39:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f1498a5a740 20 client.18024 populate_metadata read hostname 'smithi050' 2024-04-23T18:39:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f1498a5a740 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd44968ca0 0x55bd4496b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:09.804 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f1498a5a740 4 client.18024 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:39:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f1498a5a740 1 -- 172.21.15.50:0/116231494 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd448bc390 con 0x55bd44968ca0 2024-04-23T18:39:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.140+0000 7f1486ffd640 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd44968ca0 0x55bd4496b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.141+0000 7f1486ffd640 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd44968ca0 0x55bd4496b090 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab9970011eecb7 server_cookie=d6d5412739ff78c6 in_seq=0 out_seq=0 2024-04-23T18:39:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.141+0000 7f146effd640 10 client.18024 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:09.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f146effd640 1 -- 172.21.15.50:0/116231494 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bd448bc390 con 0x55bd44968ca0 2024-04-23T18:39:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f146effd640 10 client.18024 handle_command_reply: tid=0 2024-04-23T18:39:10.060 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f1498a5a740 1 client.18024 shutdown 2024-04-23T18:39:10.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f1498a5a740 1 -- 172.21.15.50:0/116231494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 msgr2=0x55bd44958560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:10.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f1498a5a740 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd44958560 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f148000bcf0 tx=0x7f1480011440 comp rx=0 tx=0).stop 2024-04-23T18:39:10.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f1498a5a740 1 -- 172.21.15.50:0/116231494 shutdown_connections 2024-04-23T18:39:10.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f1498a5a740 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd44968ca0 0x55bd4496b090 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:10.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f1498a5a740 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 0x55bd44955db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:10.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f1498a5a740 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd44955850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f146effd640 0 client.18024 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f1498a5a740 1 --2- 172.21.15.50:0/116231494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd44958560 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f1498a5a740 1 -- 172.21.15.50:0/116231494 >> 172.21.15.50:0/116231494 conn(0x55bd44945b30 msgr2=0x55bd4494b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f1498a5a740 1 -- 172.21.15.50:0/116231494 shutdown_connections 2024-04-23T18:39:10.213 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f1498a5a740 1 -- 172.21.15.50:0/116231494 wait complete. 2024-04-23T18:39:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.142+0000 7f1498a5a740 20 client.18024 trim_cache size 0 max 16384 2024-04-23T18:39:10.230 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f1498a5a740 1 Processor -- start 2024-04-23T18:39:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f1498a5a740 1 -- start start 2024-04-23T18:39:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f1498a5a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd44958090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f1498a5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd44952ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:10.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f1498a5a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 0x55bd44953010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f1498a5a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f14901ae1d0 con 0x55bd44949500 2024-04-23T18:39:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f1498a5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14901abaa0 con 0x55bd44949e20 2024-04-23T18:39:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f1498a5a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f14901a9910 con 0x55bd44958c10 2024-04-23T18:39:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f148f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd44958090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f148f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd44958090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43808/0 (socket says 172.21.15.50:43808) 2024-04-23T18:39:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f148f7fe640 1 -- 172.21.15.50:0/1917992060 learned_addr learned my addr 172.21.15.50:0/1917992060 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:10.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f1486ffd640 1 --2- 172.21.15.50:0/1917992060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd44952ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:10.587 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f148f7fe640 1 -- 172.21.15.50:0/1917992060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 msgr2=0x55bd44953010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f148f7fe640 1 --2- 172.21.15.50:0/1917992060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 0x55bd44953010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f148f7fe640 1 -- 172.21.15.50:0/1917992060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 msgr2=0x55bd44952ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f148f7fe640 1 --2- 172.21.15.50:0/1917992060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd44952ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.157+0000 7f148f7fe640 1 -- 172.21.15.50:0/1917992060 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd44949c50 con 0x55bd44949500 2024-04-23T18:39:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f148f7fe640 1 --2- 172.21.15.50:0/1917992060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd44958090 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f148000ba80 tx=0x7f1480005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81370f3a06b764da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:10.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f14857fa640 1 -- 172.21.15.50:0/1917992060 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480029070 con 0x55bd44949500 2024-04-23T18:39:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f14857fa640 1 -- 172.21.15.50:0/1917992060 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f148001a460 con 0x55bd44949500 2024-04-23T18:39:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f14857fa640 1 -- 172.21.15.50:0/1917992060 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480011c50 con 0x55bd44949500 2024-04-23T18:39:10.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f1498a5a740 1 -- 172.21.15.50:0/1917992060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 msgr2=0x55bd44958090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1917992060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd44958090 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f148000ba80 tx=0x7f1480005850 comp rx=0 tx=0).stop 2024-04-23T18:39:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f1498a5a740 1 -- 172.21.15.50:0/1917992060 shutdown_connections 2024-04-23T18:39:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1917992060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44958c10 0x55bd44953010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:10.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1917992060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949e20 0x55bd44952ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:10.907 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1917992060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44949500 0x55bd44958090 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f1498a5a740 1 -- 172.21.15.50:0/1917992060 >> 172.21.15.50:0/1917992060 conn(0x55bd44945b30 msgr2=0x55bd44945f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f1498a5a740 1 -- 172.21.15.50:0/1917992060 shutdown_connections 2024-04-23T18:39:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.158+0000 7f1498a5a740 1 -- 172.21.15.50:0/1917992060 wait complete. 2024-04-23T18:39:11.026 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f1498a5a740 1 -- start start 2024-04-23T18:39:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f1498a5a740 1 Processor -- start 2024-04-23T18:39:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f1498a5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949500 0x55bd4494fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:11.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f1498a5a740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44949e20 0x55bd44950200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f1498a5a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44958c10 0x55bd44950740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f1498a5a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f14901a9f60 con 0x55bd44958c10 2024-04-23T18:39:11.080 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f1498a5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14901ab3d0 con 0x55bd44949500 2024-04-23T18:39:11.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f1498a5a740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f14901aa630 con 0x55bd44949e20 2024-04-23T18:39:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f1498a5a740 10 client.0 _ll_register_callbacks cb 0x7f14800056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:39:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f148ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44958c10 0x55bd44950740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f148ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44958c10 0x55bd44950740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43810/0 (socket says 172.21.15.50:43810) 2024-04-23T18:39:11.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f148ffff640 1 -- 172.21.15.50:0/1840400236 learned_addr learned my addr 172.21.15.50:0/1840400236 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f1486ffd640 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44949e20 0x55bd44950200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.159+0000 7f148f7fe640 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949500 0x55bd4494fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f148ffff640 1 -- 172.21.15.50:0/1840400236 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44949e20 msgr2=0x55bd44950200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f148ffff640 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44949e20 0x55bd44950200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f148ffff640 1 -- 172.21.15.50:0/1840400236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949500 msgr2=0x55bd4494fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f148ffff640 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949500 0x55bd4494fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:11.662 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f148ffff640 1 -- 172.21.15.50:0/1840400236 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd44956120 con 0x55bd44958c10 2024-04-23T18:39:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f148ffff640 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44958c10 0x55bd44950740 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f1488035000 tx=0x7f14880095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=406fce6bd6f3f644 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f146effd640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:39:11.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f146effd640 1 -- 172.21.15.50:0/1840400236 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488003d80 con 0x55bd44958c10 2024-04-23T18:39:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f146effd640 1 -- 172.21.15.50:0/1840400236 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8542+0+0 (secure 0 0 0) 0x7f1488031e00 con 0x55bd44958c10 2024-04-23T18:39:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f1498a5a740 1 -- 172.21.15.50:0/1840400236 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f148001a460 con 0x55bd44958c10 2024-04-23T18:39:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f146effd640 1 -- 172.21.15.50:0/1840400236 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f1488004040 con 0x55bd44958c10 2024-04-23T18:39:11.877 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f1498a5a740 10 client.18030 fetch_fsmap learned FSMap version 62 2024-04-23T18:39:11.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f1498a5a740 1 -- 172.21.15.50:0/1840400236 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55bd449d3780 con 0x55bd44958c10 2024-04-23T18:39:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f146effd640 1 -- 172.21.15.50:0/1840400236 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f148001a460 con 0x55bd44958c10 2024-04-23T18:39:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f146effd640 1 -- 172.21.15.50:0/1840400236 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f148802fa70 con 0x55bd44958c10 2024-04-23T18:39:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f1498a5a740 10 client.18030 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:39:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f1498a5a740 10 client.18030 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:39:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f1498a5a740 20 client.18030 populate_metadata read hostname 'smithi050' 2024-04-23T18:39:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd449d3ee0 0x55bd449d6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:11.992 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f1498a5a740 4 client.18030 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:39:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.160+0000 7f1498a5a740 1 -- 172.21.15.50:0/1840400236 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bd448b6200 con 0x55bd449d3ee0 2024-04-23T18:39:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.161+0000 7f148f7fe640 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd449d3ee0 0x55bd449d6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:12.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.161+0000 7f148f7fe640 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd449d3ee0 0x55bd449d6300 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46e5fd582f2092c2 server_cookie=cbb1dc92654c1b47 in_seq=0 out_seq=0 2024-04-23T18:39:12.035 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.161+0000 7f146effd640 10 client.18030 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f146effd640 1 -- 172.21.15.50:0/1840400236 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bd448b6200 con 0x55bd449d3ee0 2024-04-23T18:39:12.051 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f146effd640 10 client.18030 handle_command_reply: tid=0 2024-04-23T18:39:12.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1498a5a740 1 client.18030 shutdown 2024-04-23T18:39:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1498a5a740 1 -- 172.21.15.50:0/1840400236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44958c10 msgr2=0x55bd44950740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44958c10 0x55bd44950740 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f1488035000 tx=0x7f14880095c0 comp rx=0 tx=0).stop 2024-04-23T18:39:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1498a5a740 1 -- 172.21.15.50:0/1840400236 shutdown_connections 2024-04-23T18:39:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55bd449d3ee0 0x55bd449d6300 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55bd44958c10 0x55bd44950740 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bd44949e20 0x55bd44950200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1498a5a740 1 --2- 172.21.15.50:0/1840400236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd44949500 0x55bd4494fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:12.223 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1498a5a740 1 -- 172.21.15.50:0/1840400236 >> 172.21.15.50:0/1840400236 conn(0x55bd44945b30 msgr2=0x55bd4494b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f146effd640 0 client.18030 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:12.261 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1498a5a740 1 -- 172.21.15.50:0/1840400236 shutdown_connections 2024-04-23T18:39:12.317 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1498a5a740 1 -- 172.21.15.50:0/1840400236 wait complete. 2024-04-23T18:39:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1498a5a740 20 client.18030 trim_cache size 0 max 16384 2024-04-23T18:39:12.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:39:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:39:12.355 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:39:12.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:39:12.397 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1495d4a640 1 -- 172.21.15.50:0/84675784 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f147c0a3dc0 msgr2=0x7f147c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:12.472 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1495d4a640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f147c0a3dc0 0x7f147c0a6280 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55bd4494aa90 tx=0x7f14880099f0 comp rx=0 tx=0).stop 2024-04-23T18:39:12.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1495d4a640 1 -- 172.21.15.50:0/84675784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14901072a0 msgr2=0x7f1490198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1495d4a640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14901072a0 0x7f1490198bf0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f1478005fd0 tx=0x7f147800dde0 comp rx=0 tx=0).stop 2024-04-23T18:39:12.555 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1495d4a640 1 -- 172.21.15.50:0/84675784 shutdown_connections 2024-04-23T18:39:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1495d4a640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f147c0a3dc0 0x7f147c0a6280 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1495d4a640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f149019ced0 0x7f149019d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:12.583 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1495d4a640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14901072a0 0x7f1490198bf0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1495d4a640 1 --2- 172.21.15.50:0/84675784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14901021d0 0x7f14901986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:12.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1495d4a640 1 -- 172.21.15.50:0/84675784 >> 172.21.15.50:0/84675784 conn(0x7f14900793e0 msgr2=0x7f14900ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1495d4a640 1 -- 172.21.15.50:0/84675784 shutdown_connections 2024-04-23T18:39:12.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:02.300+0000 7f1495d4a640 1 -- 172.21.15.50:0/84675784 wait complete. 2024-04-23T18:39:12.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:02 smithi050 ceph-mon[29187]: pgmap v652: 97 pgs: 97 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.9 MiB/s wr, 156 op/s 2024-04-23T18:39:12.739 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.94 MB/sec execute 336 sec latency 2196.241 ms 2024-04-23T18:39:12.747 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.89 MB/sec execute 337 sec latency 3196.331 ms 2024-04-23T18:39:12.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:12.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:12.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: pgmap v653: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 352 op/s 2024-04-23T18:39:12.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:13.049 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:13.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:13.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:13.164 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:13.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:13.182 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:13.191 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:13.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:13.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:13.346 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:13.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:03 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:13.359 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.83 MB/sec execute 338 sec latency 4196.399 ms 2024-04-23T18:39:13.367 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-04-23T18:39:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: Unable to set osd_memory_target on smithi050 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-23T18:39:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi188 to 176.2M 2024-04-23T18:39:13.440 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: Unable to set osd_memory_target on smithi188 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T18:39:13.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:13.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:13.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:13.598 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T18:39:13.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-23T18:39:13.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-23T18:39:13.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:39:13.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:39:13.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:04 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:39:13.704 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.77 MB/sec execute 339 sec latency 5196.437 ms 2024-04-23T18:39:13.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:05 smithi050 ceph-mon[29187]: pgmap v654: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 294 op/s 2024-04-23T18:39:13.755 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:05 smithi050 ceph-mon[29187]: osdmap e125: 12 total, 12 up, 12 in 2024-04-23T18:39:13.762 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:07 smithi188 ceph-mon[32694]: pgmap v656: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.7 MiB/s wr, 313 op/s 2024-04-23T18:39:13.768 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:09 smithi188 ceph-mon[32694]: pgmap v657: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.1 MiB/s wr, 303 op/s 2024-04-23T18:39:13.775 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:11 smithi188 ceph-mon[32694]: pgmap v658: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.1 MiB/s wr, 303 op/s 2024-04-23T18:39:13.782 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:07 smithi177 ceph-mon[32308]: pgmap v656: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.7 MiB/s wr, 313 op/s 2024-04-23T18:39:13.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:09 smithi177 ceph-mon[32308]: pgmap v657: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.1 MiB/s wr, 303 op/s 2024-04-23T18:39:13.846 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:11 smithi177 ceph-mon[32308]: pgmap v658: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.1 MiB/s wr, 303 op/s 2024-04-23T18:39:13.904 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.72 MB/sec execute 340 sec latency 6196.503 ms 2024-04-23T18:39:13.911 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.66 MB/sec execute 341 sec latency 7196.640 ms 2024-04-23T18:39:13.918 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:07 smithi050 ceph-mon[29187]: pgmap v656: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.7 MiB/s wr, 313 op/s 2024-04-23T18:39:13.926 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.61 MB/sec execute 342 sec latency 8196.754 ms 2024-04-23T18:39:13.933 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.56 MB/sec execute 343 sec latency 9196.837 ms 2024-04-23T18:39:13.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:09 smithi050 ceph-mon[29187]: pgmap v657: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.1 MiB/s wr, 303 op/s 2024-04-23T18:39:13.950 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.50 MB/sec execute 344 sec latency 10196.985 ms 2024-04-23T18:39:13.978 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.45 MB/sec execute 345 sec latency 11197.066 ms 2024-04-23T18:39:14.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:11 smithi050 ceph-mon[29187]: pgmap v658: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.1 MiB/s wr, 303 op/s 2024-04-23T18:39:14.066 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.39 MB/sec execute 346 sec latency 12197.129 ms 2024-04-23T18:39:14.073 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.34 MB/sec execute 347 sec latency 13197.290 ms 2024-04-23T18:39:14.193 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:13 smithi050 ceph-mon[29187]: pgmap v659: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 150 KiB/s wr, 76 op/s 2024-04-23T18:39:14.286 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:13 smithi177 ceph-mon[32308]: pgmap v659: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 150 KiB/s wr, 76 op/s 2024-04-23T18:39:14.328 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:13 smithi188 ceph-mon[32694]: pgmap v659: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 150 KiB/s wr, 76 op/s 2024-04-23T18:39:14.583 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.29 MB/sec execute 348 sec latency 14197.411 ms 2024-04-23T18:39:15.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.401+0000 7fe9b37f7640 1 -- 172.21.15.50:0/362824143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ac100ff0 msgr2=0x7fe9ac1013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.401+0000 7fe9b37f7640 1 --2- 172.21.15.50:0/362824143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ac100ff0 0x7fe9ac1013d0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8009930 tx=0x7fe9a802a4d0 comp rx=0 tx=0).stop 2024-04-23T18:39:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.401+0000 7fe9b37f7640 1 -- 172.21.15.50:0/362824143 shutdown_connections 2024-04-23T18:39:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.401+0000 7fe9b37f7640 1 --2- 172.21.15.50:0/362824143 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac1063f0 0x7fe9ac1068a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.401+0000 7fe9b37f7640 1 --2- 172.21.15.50:0/362824143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9ac101910 0x7fe9ac105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.401+0000 7fe9b37f7640 1 --2- 172.21.15.50:0/362824143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ac100ff0 0x7fe9ac1013d0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:15.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.401+0000 7fe9b37f7640 1 -- 172.21.15.50:0/362824143 >> 172.21.15.50:0/362824143 conn(0x7fe9ac0771d0 msgr2=0x7fe9ac0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.401+0000 7fe9b37f7640 1 -- 172.21.15.50:0/362824143 shutdown_connections 2024-04-23T18:39:15.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.401+0000 7fe9b37f7640 1 -- 172.21.15.50:0/362824143 wait complete. 2024-04-23T18:39:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b37f7640 1 Processor -- start 2024-04-23T18:39:15.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b37f7640 1 -- start start 2024-04-23T18:39:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b37f7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac100ff0 0x7fe9ac1a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b37f7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ac101910 0x7fe9ac1a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b37f7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9ac1063f0 0x7fe9ac19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b37f7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1187f0 con 0x7fe9ac101910 2024-04-23T18:39:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b37f7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac118670 con 0x7fe9ac1063f0 2024-04-23T18:39:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b37f7640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac118970 con 0x7fe9ac100ff0 2024-04-23T18:39:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b0d92640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ac101910 0x7fe9ac1a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:15.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b1593640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac100ff0 0x7fe9ac1a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:15.866 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b1593640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac100ff0 0x7fe9ac1a4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:40768/0 (socket says 172.21.15.50:40768) 2024-04-23T18:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b0d92640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ac101910 0x7fe9ac1a4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42830/0 (socket says 172.21.15.50:42830) 2024-04-23T18:39:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b1593640 1 -- 172.21.15.50:0/1489548776 learned_addr learned my addr 172.21.15.50:0/1489548776 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.402+0000 7fe9b1d94640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9ac1063f0 0x7fe9ac19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.403+0000 7fe9b0d92640 1 -- 172.21.15.50:0/1489548776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac100ff0 msgr2=0x7fe9ac1a4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.403+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac100ff0 0x7fe9ac1a4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:15.914 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.403+0000 7fe9b0d92640 1 -- 172.21.15.50:0/1489548776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9ac1063f0 msgr2=0x7fe9ac19eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.403+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9ac1063f0 0x7fe9ac19eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:16.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.403+0000 7fe9b0d92640 1 -- 172.21.15.50:0/1489548776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ac19f230 con 0x7fe9ac101910 2024-04-23T18:39:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.403+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ac101910 0x7fe9ac1a4e40 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fe99c005fd0 tx=0x7fe99c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.403+0000 7fe9a27fc640 1 -- 172.21.15.50:0/1489548776 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c009c30 con 0x7fe9ac101910 2024-04-23T18:39:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.403+0000 7fe9a27fc640 1 -- 172.21.15.50:0/1489548776 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe99c009dd0 con 0x7fe9ac101910 2024-04-23T18:39:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.403+0000 7fe9b37f7640 1 -- 172.21.15.50:0/1489548776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9ac19f520 con 0x7fe9ac101910 2024-04-23T18:39:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.403+0000 7fe9a27fc640 1 -- 172.21.15.50:0/1489548776 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c010460 con 0x7fe9ac101910 2024-04-23T18:39:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.403+0000 7fe9b37f7640 1 -- 172.21.15.50:0/1489548776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9ac1ab7c0 con 0x7fe9ac101910 2024-04-23T18:39:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.406+0000 7fe9a27fc640 1 -- 172.21.15.50:0/1489548776 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe99c003e80 con 0x7fe9ac101910 2024-04-23T18:39:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.406+0000 7fe9b65e1740 1 Processor -- start 2024-04-23T18:39:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.406+0000 7fe9b65e1740 1 -- start start 2024-04-23T18:39:16.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.406+0000 7fe9b65e1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 0x5590e8f96a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.406+0000 7fe9b65e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96f40 0x5590e8fa12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:16.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.406+0000 7fe9b65e1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8fa3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.406+0000 7fe9b65e1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5590e8efb970 con 0x5590e8f96620 2024-04-23T18:39:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.406+0000 7fe9b65e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5590e8e984a0 con 0x5590e8f96f40 2024-04-23T18:39:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.406+0000 7fe9b65e1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5590e8ef7930 con 0x5590e8fa1800 2024-04-23T18:39:16.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.406+0000 7fe9b1d94640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 0x5590e8f96a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.406+0000 7fe9b1d94640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 0x5590e8f96a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42838/0 (socket says 172.21.15.50:42838) 2024-04-23T18:39:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.406+0000 7fe9b1d94640 1 -- 172.21.15.50:0/3039754364 learned_addr learned my addr 172.21.15.50:0/3039754364 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9a27fc640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe9980a3dc0 0x7fe9980a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/3039754364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8fa3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9b1593640 1 --2- 172.21.15.50:0/3039754364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96f40 0x5590e8fa12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9b1d94640 1 -- 172.21.15.50:0/3039754364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 msgr2=0x5590e8fa3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9b1d94640 1 --2- 172.21.15.50:0/3039754364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8fa3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:16.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9b1d94640 1 -- 172.21.15.50:0/3039754364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96f40 msgr2=0x5590e8fa12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:16.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9b1d94640 1 --2- 172.21.15.50:0/3039754364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96f40 0x5590e8fa12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9b1d94640 1 -- 172.21.15.50:0/3039754364 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590e8c0e210 con 0x5590e8f96620 2024-04-23T18:39:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/3039754364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8fa3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:39:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9b1d94640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe9980a3dc0 0x7fe9980a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9a27fc640 1 -- 172.21.15.50:0/1489548776 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8586+0+0 (secure 0 0 0) 0x7fe99c0d0370 con 0x7fe9ac101910 2024-04-23T18:39:16.709 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9b1d94640 1 --2- 172.21.15.50:0/3039754364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 0x5590e8f96a00 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4010440 tx=0x7fe9a4010900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7776cfce54b07ffa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9977fe640 1 -- 172.21.15.50:0/3039754364 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a401d400 con 0x5590e8f96620 2024-04-23T18:39:16.728 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9977fe640 1 -- 172.21.15.50:0/3039754364 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9a4013520 con 0x5590e8f96620 2024-04-23T18:39:16.794 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9977fe640 1 -- 172.21.15.50:0/3039754364 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4011840 con 0x5590e8f96620 2024-04-23T18:39:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.407+0000 7fe9b1d94640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe9980a3dc0 0x7fe9980a6280 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4021000 tx=0x7fe9a401e000 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.408+0000 7fe9b65e1740 1 -- 172.21.15.50:0/3039754364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 msgr2=0x5590e8f96a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:16.881 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.408+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/3039754364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 0x5590e8f96a00 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4010440 tx=0x7fe9a4010900 comp rx=0 tx=0).stop 2024-04-23T18:39:16.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.408+0000 7fe9b65e1740 1 -- 172.21.15.50:0/3039754364 shutdown_connections 2024-04-23T18:39:16.899 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.408+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/3039754364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8fa3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:16.908 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.408+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/3039754364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96f40 0x5590e8fa12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.408+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/3039754364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 0x5590e8f96a00 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.408+0000 7fe9b65e1740 1 -- 172.21.15.50:0/3039754364 >> 172.21.15.50:0/3039754364 conn(0x5590e8f92b30 msgr2=0x5590e8f92f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.408+0000 7fe9b65e1740 1 -- 172.21.15.50:0/3039754364 shutdown_connections 2024-04-23T18:39:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.408+0000 7fe9b65e1740 1 -- 172.21.15.50:0/3039754364 wait complete. 2024-04-23T18:39:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.408+0000 7fe9b65e1740 1 -- start start 2024-04-23T18:39:17.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b65e1740 1 Processor -- start 2024-04-23T18:39:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b65e1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8f96620 0x5590e8fa1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b65e1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 0x5590e8f9e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b65e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8fa1800 0x5590e8f9e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b65e1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5590e8fa4430 con 0x5590e8f96f40 2024-04-23T18:39:17.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b65e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5590e8e960b0 con 0x5590e8fa1800 2024-04-23T18:39:17.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b65e1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5590e8fa45b0 con 0x5590e8f96620 2024-04-23T18:39:17.285 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b65e1740 10 client.0 _ll_register_callbacks cb 0x5590e8e83d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:39:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b0d92640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 0x5590e8f9e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:17.300 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b0d92640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 0x5590e8f9e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42848/0 (socket says 172.21.15.50:42848) 2024-04-23T18:39:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b0d92640 1 -- 172.21.15.50:0/2871534271 learned_addr learned my addr 172.21.15.50:0/2871534271 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:17.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b1d94640 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8fa1800 0x5590e8f9e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:17.323 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.409+0000 7fe9b1593640 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8f96620 0x5590e8fa1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe9b0d92640 1 -- 172.21.15.50:0/2871534271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8f96620 msgr2=0x5590e8fa1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8f96620 0x5590e8fa1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe9b0d92640 1 -- 172.21.15.50:0/2871534271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8fa1800 msgr2=0x5590e8f9e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:17.556 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8fa1800 0x5590e8f9e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe9b0d92640 1 -- 172.21.15.50:0/2871534271 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590e8cda620 con 0x5590e8f96f40 2024-04-23T18:39:17.573 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 0x5590e8f9e410 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fe99c0ce080 tx=0x7fe99c0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae3af84c8a53c76 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:17.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe994ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:39:17.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe994ff9640 1 -- 172.21.15.50:0/2871534271 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c020070 con 0x5590e8f96f40 2024-04-23T18:39:17.599 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe994ff9640 1 -- 172.21.15.50:0/2871534271 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8586+0+0 (secure 0 0 0) 0x7fe99c0989f0 con 0x5590e8f96f40 2024-04-23T18:39:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe9b65e1740 1 -- 172.21.15.50:0/2871534271 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9a4013520 con 0x5590e8f96f40 2024-04-23T18:39:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe994ff9640 1 -- 172.21.15.50:0/2871534271 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fe99c0a4f40 con 0x5590e8f96f40 2024-04-23T18:39:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe9b65e1740 10 client.18042 fetch_fsmap learned FSMap version 62 2024-04-23T18:39:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe9b65e1740 1 -- 172.21.15.50:0/2871534271 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5590e8fb02d0 con 0x5590e8f96f40 2024-04-23T18:39:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe994ff9640 1 -- 172.21.15.50:0/2871534271 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9a4013520 con 0x5590e8f96f40 2024-04-23T18:39:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe994ff9640 1 -- 172.21.15.50:0/2871534271 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fe99c004170 con 0x5590e8f96f40 2024-04-23T18:39:17.716 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.410+0000 7fe9b65e1740 10 client.18042 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:39:17.724 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.411+0000 7fe9b65e1740 10 client.18042 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:39:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.411+0000 7fe9b65e1740 20 client.18042 populate_metadata read hostname 'smithi050' 2024-04-23T18:39:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.411+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5590e8fb1790 0x5590e8fb3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.411+0000 7fe9b65e1740 4 client.18042 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:39:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.411+0000 7fe9b65e1740 1 -- 172.21.15.50:0/2871534271 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590e8f09390 con 0x5590e8fb1790 2024-04-23T18:39:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.411+0000 7fe9b1593640 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5590e8fb1790 0x5590e8fb3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.411+0000 7fe9b1593640 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5590e8fb1790 0x5590e8fb3b80 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=282777ddcdbfab03 server_cookie=42ff737c15f227d in_seq=0 out_seq=0 2024-04-23T18:39:17.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.411+0000 7fe994ff9640 10 client.18042 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe994ff9640 1 -- 172.21.15.50:0/2871534271 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590e8f09390 con 0x5590e8fb1790 2024-04-23T18:39:17.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe994ff9640 10 client.18042 handle_command_reply: tid=0 2024-04-23T18:39:17.954 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe9b65e1740 1 client.18042 shutdown 2024-04-23T18:39:18.018 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe9b65e1740 1 -- 172.21.15.50:0/2871534271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 msgr2=0x5590e8f9e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 0x5590e8f9e410 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fe99c0ce080 tx=0x7fe99c0035f0 comp rx=0 tx=0).stop 2024-04-23T18:39:18.040 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe9b65e1740 1 -- 172.21.15.50:0/2871534271 shutdown_connections 2024-04-23T18:39:18.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5590e8fb1790 0x5590e8fb3b80 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:18.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8fa1800 0x5590e8f9e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 0x5590e8f9e410 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe994ff9640 0 client.18042 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:18.104 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/2871534271 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8f96620 0x5590e8fa1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:18.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe9b65e1740 1 -- 172.21.15.50:0/2871534271 >> 172.21.15.50:0/2871534271 conn(0x5590e8f92b30 msgr2=0x5590e8f988b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:18.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe9b65e1740 1 -- 172.21.15.50:0/2871534271 shutdown_connections 2024-04-23T18:39:18.274 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe9b65e1740 1 -- 172.21.15.50:0/2871534271 wait complete. 2024-04-23T18:39:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.412+0000 7fe9b65e1740 20 client.18042 trim_cache size 0 max 16384 2024-04-23T18:39:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.427+0000 7fe9b65e1740 1 Processor -- start 2024-04-23T18:39:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.427+0000 7fe9b65e1740 1 -- start start 2024-04-23T18:39:18.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b65e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96620 0x5590e8fad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:18.320 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b65e1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 0x5590e8f99260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b65e1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8fa0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b65e1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1b6ae0 con 0x5590e8f96f40 2024-04-23T18:39:18.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b65e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1b43b0 con 0x5590e8f96620 2024-04-23T18:39:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b65e1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1b2220 con 0x5590e8fa1800 2024-04-23T18:39:18.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b0d92640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 0x5590e8f99260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b0d92640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 0x5590e8f99260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42858/0 (socket says 172.21.15.50:42858) 2024-04-23T18:39:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b0d92640 1 -- 172.21.15.50:0/3268092618 learned_addr learned my addr 172.21.15.50:0/3268092618 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b1d94640 1 --2- 172.21.15.50:0/3268092618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8fa0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b0d92640 1 -- 172.21.15.50:0/3268092618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 msgr2=0x5590e8fa0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/3268092618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8fa0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:18.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b0d92640 1 -- 172.21.15.50:0/3268092618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96620 msgr2=0x5590e8fad2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:18.635 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/3268092618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96620 0x5590e8fad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b0d92640 1 -- 172.21.15.50:0/3268092618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590e8fa3bf0 con 0x5590e8f96f40 2024-04-23T18:39:18.650 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b1d94640 1 --2- 172.21.15.50:0/3268092618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8fa0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:39:18.657 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/3268092618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 0x5590e8f99260 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fe99c09c080 tx=0x7fe99c003850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a862f0034d8efa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:18.663 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe996ffd640 1 -- 172.21.15.50:0/3268092618 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c020070 con 0x5590e8f96f40 2024-04-23T18:39:18.671 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe996ffd640 1 -- 172.21.15.50:0/3268092618 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe99c0989f0 con 0x5590e8f96f40 2024-04-23T18:39:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.428+0000 7fe996ffd640 1 -- 172.21.15.50:0/3268092618 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c0a0150 con 0x5590e8f96f40 2024-04-23T18:39:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.429+0000 7fe9b65e1740 1 -- 172.21.15.50:0/3268092618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 msgr2=0x5590e8f99260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:18.757 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.429+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/3268092618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 0x5590e8f99260 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fe99c09c080 tx=0x7fe99c003850 comp rx=0 tx=0).stop 2024-04-23T18:39:18.764 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.429+0000 7fe9b65e1740 1 -- 172.21.15.50:0/3268092618 shutdown_connections 2024-04-23T18:39:18.774 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.429+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/3268092618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8fa0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.429+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/3268092618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96f40 0x5590e8f99260 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.429+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/3268092618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96620 0x5590e8fad2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:18.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.429+0000 7fe9b65e1740 1 -- 172.21.15.50:0/3268092618 >> 172.21.15.50:0/3268092618 conn(0x5590e8f92b30 msgr2=0x5590e8f98860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:18.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.429+0000 7fe9b65e1740 1 -- 172.21.15.50:0/3268092618 shutdown_connections 2024-04-23T18:39:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.429+0000 7fe9b65e1740 1 -- 172.21.15.50:0/3268092618 wait complete. 2024-04-23T18:39:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.429+0000 7fe9b65e1740 1 -- start start 2024-04-23T18:39:19.020 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b65e1740 1 Processor -- start 2024-04-23T18:39:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b65e1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 0x5590e8f9b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b65e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96f40 0x5590e8f9b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b65e1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8f9bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:19.079 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b65e1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1b2870 con 0x5590e8f96620 2024-04-23T18:39:19.089 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b65e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1b3ce0 con 0x5590e8f96f40 2024-04-23T18:39:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b65e1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1b2f40 con 0x5590e8fa1800 2024-04-23T18:39:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b65e1740 10 client.0 _ll_register_callbacks cb 0x7fe99c09c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:39:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b1593640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 0x5590e8f9b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b1593640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 0x5590e8f9b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42868/0 (socket says 172.21.15.50:42868) 2024-04-23T18:39:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b1593640 1 -- 172.21.15.50:0/114821052 learned_addr learned my addr 172.21.15.50:0/114821052 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96f40 0x5590e8f9b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b1d94640 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8f9bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:19.254 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b1593640 1 -- 172.21.15.50:0/114821052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 msgr2=0x5590e8f9bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b1593640 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8f9bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:19.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b1593640 1 -- 172.21.15.50:0/114821052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96f40 msgr2=0x5590e8f9b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b1593640 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96f40 0x5590e8f9b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b1593640 1 -- 172.21.15.50:0/114821052 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590e9016e60 con 0x5590e8f96620 2024-04-23T18:39:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b1d94640 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8f9bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:39:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.430+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96f40 0x5590e8f9b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:39:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe9b1593640 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 0x5590e8f9b2b0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8031440 tx=0x7fe9a8004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eaa9f890cf68fe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe994ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:39:19.455 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe994ff9640 1 -- 172.21.15.50:0/114821052 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8049020 con 0x5590e8f96620 2024-04-23T18:39:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe994ff9640 1 -- 172.21.15.50:0/114821052 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8586+0+0 (secure 0 0 0) 0x7fe9a8045dc0 con 0x5590e8f96620 2024-04-23T18:39:19.593 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe9b65e1740 1 -- 172.21.15.50:0/114821052 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe99c0989f0 con 0x5590e8f96620 2024-04-23T18:39:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe994ff9640 1 -- 172.21.15.50:0/114821052 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fe9a8054050 con 0x5590e8f96620 2024-04-23T18:39:19.605 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe9b65e1740 10 client.18048 fetch_fsmap learned FSMap version 62 2024-04-23T18:39:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe9b65e1740 1 -- 172.21.15.50:0/114821052 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5590e9018120 con 0x5590e8f96620 2024-04-23T18:39:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe994ff9640 1 -- 172.21.15.50:0/114821052 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe99c0989f0 con 0x5590e8f96620 2024-04-23T18:39:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe994ff9640 1 -- 172.21.15.50:0/114821052 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fe9a80035f0 con 0x5590e8f96620 2024-04-23T18:39:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe9b65e1740 10 client.18048 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:39:19.666 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe9b65e1740 10 client.18048 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:39:19.765 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe9b65e1740 20 client.18048 populate_metadata read hostname 'smithi050' 2024-04-23T18:39:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5590e90188d0 0x5590e901acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe9b65e1740 4 client.18048 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:39:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.431+0000 7fe9b65e1740 1 -- 172.21.15.50:0/114821052 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5590e8f03200 con 0x5590e90188d0 2024-04-23T18:39:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.432+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5590e90188d0 0x5590e901acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.432+0000 7fe9b0d92640 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5590e90188d0 0x5590e901acf0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7801e1bfa73c119 server_cookie=de07ae1c1c545dfa in_seq=0 out_seq=0 2024-04-23T18:39:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.432+0000 7fe994ff9640 10 client.18048 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe994ff9640 1 -- 172.21.15.50:0/114821052 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5590e8f03200 con 0x5590e90188d0 2024-04-23T18:39:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe994ff9640 10 client.18048 handle_command_reply: tid=0 2024-04-23T18:39:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe9b65e1740 1 client.18048 shutdown 2024-04-23T18:39:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe9b65e1740 1 -- 172.21.15.50:0/114821052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 msgr2=0x5590e8f9b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:20.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 0x5590e8f9b2b0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8031440 tx=0x7fe9a8004290 comp rx=0 tx=0).stop 2024-04-23T18:39:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe9b65e1740 1 -- 172.21.15.50:0/114821052 shutdown_connections 2024-04-23T18:39:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5590e90188d0 0x5590e901acf0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:20.054 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5590e8fa1800 0x5590e8f9bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:20.062 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590e8f96f40 0x5590e8f9b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe994ff9640 0 client.18048 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe9b65e1740 1 --2- 172.21.15.50:0/114821052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5590e8f96620 0x5590e8f9b2b0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe9b65e1740 1 -- 172.21.15.50:0/114821052 >> 172.21.15.50:0/114821052 conn(0x5590e8f92b30 msgr2=0x5590e8f9dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:20.225 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe9b65e1740 1 -- 172.21.15.50:0/114821052 shutdown_connections 2024-04-23T18:39:20.233 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe9b65e1740 1 -- 172.21.15.50:0/114821052 wait complete. 2024-04-23T18:39:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.479+0000 7fe9b65e1740 20 client.18048 trim_cache size 0 max 16384 2024-04-23T18:39:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:39:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:39:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:39:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:39:20.322 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.480+0000 7fe9b37f7640 1 -- 172.21.15.50:0/1489548776 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe9980a3dc0 msgr2=0x7fe9980a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:20.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.480+0000 7fe9b37f7640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe9980a3dc0 0x7fe9980a6280 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4021000 tx=0x7fe9a401e000 comp rx=0 tx=0).stop 2024-04-23T18:39:20.400 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.480+0000 7fe9b37f7640 1 -- 172.21.15.50:0/1489548776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ac101910 msgr2=0x7fe9ac1a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.480+0000 7fe9b37f7640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ac101910 0x7fe9ac1a4e40 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fe99c005fd0 tx=0x7fe99c00dde0 comp rx=0 tx=0).stop 2024-04-23T18:39:20.412 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.480+0000 7fe9b37f7640 1 -- 172.21.15.50:0/1489548776 shutdown_connections 2024-04-23T18:39:20.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.480+0000 7fe9b37f7640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe9980a3dc0 0x7fe9980a6280 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.480+0000 7fe9b37f7640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9ac1063f0 0x7fe9ac19eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.480+0000 7fe9b37f7640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9ac101910 0x7fe9ac1a4e40 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:20.506 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.480+0000 7fe9b37f7640 1 --2- 172.21.15.50:0/1489548776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9ac100ff0 0x7fe9ac1a4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:20.577 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.480+0000 7fe9b37f7640 1 -- 172.21.15.50:0/1489548776 >> 172.21.15.50:0/1489548776 conn(0x7fe9ac0771d0 msgr2=0x7fe9ac103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.480+0000 7fe9b37f7640 1 -- 172.21.15.50:0/1489548776 shutdown_connections 2024-04-23T18:39:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:15.480+0000 7fe9b37f7640 1 -- 172.21.15.50:0/1489548776 wait complete. 2024-04-23T18:39:20.662 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.24 MB/sec execute 349 sec latency 15197.496 ms 2024-04-23T18:39:20.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:15 smithi050 ceph-mon[29187]: pgmap v660: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 150 KiB/s wr, 76 op/s 2024-04-23T18:39:20.678 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:15 smithi188 ceph-mon[32694]: pgmap v660: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 150 KiB/s wr, 76 op/s 2024-04-23T18:39:20.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:17 smithi188 ceph-mon[32694]: osdmap e126: 12 total, 12 up, 12 in 2024-04-23T18:39:20.751 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:18 smithi188 ceph-mon[32694]: pgmap v662: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 221 KiB/s wr, 44 op/s 2024-04-23T18:39:20.758 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:20 smithi188 ceph-mon[32694]: pgmap v663: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 278 KiB/s wr, 23 op/s 2024-04-23T18:39:20.825 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:15 smithi177 ceph-mon[32308]: pgmap v660: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 150 KiB/s wr, 76 op/s 2024-04-23T18:39:20.832 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:17 smithi177 ceph-mon[32308]: osdmap e126: 12 total, 12 up, 12 in 2024-04-23T18:39:20.840 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:18 smithi177 ceph-mon[32308]: pgmap v662: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 221 KiB/s wr, 44 op/s 2024-04-23T18:39:20.846 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:20 smithi177 ceph-mon[32308]: pgmap v663: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 278 KiB/s wr, 23 op/s 2024-04-23T18:39:20.853 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.18 MB/sec execute 350 sec latency 16197.611 ms 2024-04-23T18:39:20.860 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.13 MB/sec execute 351 sec latency 17197.776 ms 2024-04-23T18:39:20.916 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:17 smithi050 ceph-mon[29187]: osdmap e126: 12 total, 12 up, 12 in 2024-04-23T18:39:20.974 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.08 MB/sec execute 352 sec latency 18197.906 ms 2024-04-23T18:39:21.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:18 smithi050 ceph-mon[29187]: pgmap v662: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 221 KiB/s wr, 44 op/s 2024-04-23T18:39:21.044 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 18.03 MB/sec execute 353 sec latency 19198.035 ms 2024-04-23T18:39:21.052 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.98 MB/sec execute 354 sec latency 20198.102 ms 2024-04-23T18:39:21.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:20 smithi050 ceph-mon[29187]: pgmap v663: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 278 KiB/s wr, 23 op/s 2024-04-23T18:39:21.584 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.93 MB/sec execute 355 sec latency 21198.219 ms 2024-04-23T18:39:22.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:22 smithi050 ceph-mon[29187]: pgmap v664: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 278 KiB/s wr, 23 op/s 2024-04-23T18:39:22.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:22 smithi177 ceph-mon[32308]: pgmap v664: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 278 KiB/s wr, 23 op/s 2024-04-23T18:39:22.582 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:22 smithi188 ceph-mon[32694]: pgmap v664: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 278 KiB/s wr, 23 op/s 2024-04-23T18:39:22.665 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.88 MB/sec execute 356 sec latency 22198.271 ms 2024-04-23T18:39:23.584 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.83 MB/sec execute 357 sec latency 23198.305 ms 2024-04-23T18:39:24.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:24 smithi177 ceph-mon[32308]: pgmap v665: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 236 KiB/s wr, 19 op/s 2024-04-23T18:39:24.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:24 smithi188 ceph-mon[32694]: pgmap v665: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 236 KiB/s wr, 19 op/s 2024-04-23T18:39:24.585 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.78 MB/sec execute 358 sec latency 24198.402 ms 2024-04-23T18:39:24.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:24 smithi050 ceph-mon[29187]: pgmap v665: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 236 KiB/s wr, 19 op/s 2024-04-23T18:39:25.584 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.73 MB/sec execute 359 sec latency 25198.450 ms 2024-04-23T18:39:26.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:26 smithi177 ceph-mon[32308]: pgmap v666: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 236 KiB/s wr, 19 op/s 2024-04-23T18:39:26.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:26 smithi188 ceph-mon[32694]: pgmap v666: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 236 KiB/s wr, 19 op/s 2024-04-23T18:39:26.586 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.68 MB/sec execute 360 sec latency 26198.519 ms 2024-04-23T18:39:26.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:26 smithi050 ceph-mon[29187]: pgmap v666: 97 pgs: 97 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 236 KiB/s wr, 19 op/s 2024-04-23T18:39:27.584 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.63 MB/sec execute 361 sec latency 27198.632 ms 2024-04-23T18:39:28.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:28 smithi177 ceph-mon[32308]: pgmap v667: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 2.3 MiB/s wr, 69 op/s 2024-04-23T18:39:28.609 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:28 smithi188 ceph-mon[32694]: pgmap v667: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 2.3 MiB/s wr, 69 op/s 2024-04-23T18:39:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.579+0000 7f7ae33e8640 1 -- 172.21.15.50:0/1414615686 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7adc1063f0 msgr2=0x7f7adc1068a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:28.635 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.579+0000 7f7ae33e8640 1 --2- 172.21.15.50:0/1414615686 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7adc1063f0 0x7f7adc1068a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009cd0 tx=0x7f7ad402a970 comp rx=0 tx=0).stop 2024-04-23T18:39:28.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.579+0000 7f7ae33e8640 1 -- 172.21.15.50:0/1414615686 shutdown_connections 2024-04-23T18:39:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.579+0000 7f7ae33e8640 1 --2- 172.21.15.50:0/1414615686 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7adc1063f0 0x7f7adc1068a0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.579+0000 7f7ae33e8640 1 --2- 172.21.15.50:0/1414615686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7adc101910 0x7f7adc105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.579+0000 7f7ae33e8640 1 --2- 172.21.15.50:0/1414615686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7adc100ff0 0x7f7adc1013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.579+0000 7f7ae33e8640 1 -- 172.21.15.50:0/1414615686 >> 172.21.15.50:0/1414615686 conn(0x7f7adc0771d0 msgr2=0x7f7adc0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:28.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.579+0000 7f7ae33e8640 1 -- 172.21.15.50:0/1414615686 shutdown_connections 2024-04-23T18:39:28.695 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.579+0000 7f7ae33e8640 1 -- 172.21.15.50:0/1414615686 wait complete. 2024-04-23T18:39:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.579+0000 7f7ae33e8640 1 Processor -- start 2024-04-23T18:39:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae33e8640 1 -- start start 2024-04-23T18:39:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae33e8640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7adc100ff0 0x7f7adc1a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae33e8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7adc101910 0x7f7adc1a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae33e8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7adc1063f0 0x7f7adc19ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae33e8640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1187f0 con 0x7f7adc1063f0 2024-04-23T18:39:28.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae33e8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc118670 con 0x7f7adc101910 2024-04-23T18:39:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae33e8640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc118970 con 0x7f7adc100ff0 2024-04-23T18:39:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae1985640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7adc1063f0 0x7f7adc19ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae1985640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7adc1063f0 0x7f7adc19ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49908/0 (socket says 172.21.15.50:49908) 2024-04-23T18:39:29.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae1985640 1 -- 172.21.15.50:0/3535025947 learned_addr learned my addr 172.21.15.50:0/3535025947 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae0983640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7adc101910 0x7f7adc1a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae1184640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7adc100ff0 0x7f7adc1a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae1985640 1 -- 172.21.15.50:0/3535025947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7adc100ff0 msgr2=0x7f7adc1a4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae1985640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7adc100ff0 0x7f7adc1a4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae1985640 1 -- 172.21.15.50:0/3535025947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7adc101910 msgr2=0x7f7adc1a4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae1985640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7adc101910 0x7f7adc1a4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae1985640 1 -- 172.21.15.50:0/3535025947 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7adc19f210 con 0x7f7adc1063f0 2024-04-23T18:39:29.193 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.580+0000 7f7ae1985640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7adc1063f0 0x7f7adc19ea90 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009df0 tx=0x7f7ad4003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.581+0000 7f7ad27fc640 1 -- 172.21.15.50:0/3535025947 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4047070 con 0x7f7adc1063f0 2024-04-23T18:39:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.581+0000 7f7ae33e8640 1 -- 172.21.15.50:0/3535025947 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7adc19f4a0 con 0x7f7adc1063f0 2024-04-23T18:39:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.581+0000 7f7ad27fc640 1 -- 172.21.15.50:0/3535025947 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ad402f830 con 0x7f7adc1063f0 2024-04-23T18:39:29.291 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.581+0000 7f7ad27fc640 1 -- 172.21.15.50:0/3535025947 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4034070 con 0x7f7adc1063f0 2024-04-23T18:39:29.348 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.581+0000 7f7ae33e8640 1 -- 172.21.15.50:0/3535025947 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7adc1ab750 con 0x7f7adc1063f0 2024-04-23T18:39:29.404 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.583+0000 7f7ad27fc640 1 -- 172.21.15.50:0/3535025947 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7ad404a070 con 0x7f7adc1063f0 2024-04-23T18:39:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae61d0740 1 Processor -- start 2024-04-23T18:39:29.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae61d0740 1 -- start start 2024-04-23T18:39:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae61d0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989620 0x5596e4989a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae61d0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 0x5596e49942c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:29.444 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae61d0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4994800 0x5596e4996bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae61d0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5596e48ee970 con 0x5596e4989f40 2024-04-23T18:39:29.459 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae61d0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5596e488b4a0 con 0x5596e4989620 2024-04-23T18:39:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae61d0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5596e48ea930 con 0x5596e4994800 2024-04-23T18:39:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ad27fc640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7ac80a3dc0 0x7f7ac80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:29.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae1184640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 0x5596e49942c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:29.491 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae1184640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 0x5596e49942c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49910/0 (socket says 172.21.15.50:49910) 2024-04-23T18:39:29.497 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae1184640 1 -- 172.21.15.50:0/3725108242 learned_addr learned my addr 172.21.15.50:0/3725108242 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae1985640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7ac80a3dc0 0x7f7ac80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae0983640 1 --2- 172.21.15.50:0/3725108242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4994800 0x5596e4996bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.584+0000 7f7ae0983640 1 --2- 172.21.15.50:0/3725108242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989620 0x5596e4989a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.585+0000 7f7ad27fc640 1 -- 172.21.15.50:0/3535025947 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7f7ad40f0e10 con 0x7f7adc1063f0 2024-04-23T18:39:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.585+0000 7f7ae1184640 1 -- 172.21.15.50:0/3725108242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4994800 msgr2=0x5596e4996bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:29.695 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.585+0000 7f7ae1184640 1 --2- 172.21.15.50:0/3725108242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4994800 0x5596e4996bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.585+0000 7f7ae1184640 1 -- 172.21.15.50:0/3725108242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989620 msgr2=0x5596e4989a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.585+0000 7f7ae1184640 1 --2- 172.21.15.50:0/3725108242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989620 0x5596e4989a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.585+0000 7f7ae1184640 1 -- 172.21.15.50:0/3725108242 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596e4601210 con 0x5596e4989f40 2024-04-23T18:39:29.727 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.585+0000 7f7ae1184640 1 --2- 172.21.15.50:0/3725108242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 0x5596e49942c0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800bc90 tx=0x7f7ad800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ace7566c5c3bd0d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.585+0000 7f7ae1985640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7ac80a3dc0 0x7f7ac80a6280 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40bcd30 tx=0x7f7ad408f890 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.585+0000 7f7ac77fe640 1 -- 172.21.15.50:0/3725108242 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8015070 con 0x5596e4989f40 2024-04-23T18:39:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.585+0000 7f7ac77fe640 1 -- 172.21.15.50:0/3725108242 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ad801a910 con 0x5596e4989f40 2024-04-23T18:39:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.585+0000 7f7ac77fe640 1 -- 172.21.15.50:0/3725108242 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad80118c0 con 0x5596e4989f40 2024-04-23T18:39:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.586+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3725108242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 msgr2=0x5596e49942c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:29.773 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.586+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/3725108242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 0x5596e49942c0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800bc90 tx=0x7f7ad800f730 comp rx=0 tx=0).stop 2024-04-23T18:39:29.850 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.586+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3725108242 shutdown_connections 2024-04-23T18:39:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.586+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/3725108242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4994800 0x5596e4996bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.586+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/3725108242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 0x5596e49942c0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.586+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/3725108242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989620 0x5596e4989a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:30.029 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.586+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3725108242 >> 172.21.15.50:0/3725108242 conn(0x5596e4985b30 msgr2=0x5596e4985f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.586+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3725108242 shutdown_connections 2024-04-23T18:39:30.092 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.586+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3725108242 wait complete. 2024-04-23T18:39:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.586+0000 7f7ae61d0740 1 -- start start 2024-04-23T18:39:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.586+0000 7f7ae61d0740 1 Processor -- start 2024-04-23T18:39:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae61d0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989620 0x5596e4994170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae61d0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989f40 0x5596e4991460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae61d0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4994800 0x5596e49919c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:30.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae61d0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5596e4997430 con 0x5596e4989620 2024-04-23T18:39:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae61d0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5596e48890b0 con 0x5596e4989f40 2024-04-23T18:39:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae61d0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5596e49975b0 con 0x5596e4994800 2024-04-23T18:39:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae61d0740 10 client.0 _ll_register_callbacks cb 0x5596e4876d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:39:30.263 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae0983640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989620 0x5596e4994170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:30.308 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae0983640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989620 0x5596e4994170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49922/0 (socket says 172.21.15.50:49922) 2024-04-23T18:39:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae0983640 1 -- 172.21.15.50:0/2353911086 learned_addr learned my addr 172.21.15.50:0/2353911086 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae1184640 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989f40 0x5596e4991460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae0983640 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4994800 0x5596e49919c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae0983640 1 -- 172.21.15.50:0/2353911086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4994800 msgr2=0x5596e49919c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae0983640 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4994800 0x5596e49919c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae0983640 1 -- 172.21.15.50:0/2353911086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989f40 msgr2=0x5596e4991460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae0983640 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989f40 0x5596e4991460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:30.501 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae0983640 1 -- 172.21.15.50:0/2353911086 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596e46cd620 con 0x5596e4989620 2024-04-23T18:39:30.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae1184640 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989f40 0x5596e4991460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:39:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae0983640 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989620 0x5596e4994170 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f7acc004370 tx=0x7f7acc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34a227123107c59 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ac4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:39:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ac4ff9640 1 -- 172.21.15.50:0/2353911086 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7acc005220 con 0x5596e4989620 2024-04-23T18:39:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ac4ff9640 1 -- 172.21.15.50:0/2353911086 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7f7acc0053c0 con 0x5596e4989620 2024-04-23T18:39:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.587+0000 7f7ae61d0740 1 -- 172.21.15.50:0/2353911086 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ad801a910 con 0x5596e4989620 2024-04-23T18:39:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.588+0000 7f7ac4ff9640 1 -- 172.21.15.50:0/2353911086 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f7acc027050 con 0x5596e4989620 2024-04-23T18:39:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.588+0000 7f7ae61d0740 10 client.18060 fetch_fsmap learned FSMap version 62 2024-04-23T18:39:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.588+0000 7f7ae61d0740 1 -- 172.21.15.50:0/2353911086 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5596e49a3400 con 0x5596e4989620 2024-04-23T18:39:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.588+0000 7f7ac4ff9640 1 -- 172.21.15.50:0/2353911086 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ad801a910 con 0x5596e4989620 2024-04-23T18:39:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.588+0000 7f7ac4ff9640 1 -- 172.21.15.50:0/2353911086 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f7acc010640 con 0x5596e4989620 2024-04-23T18:39:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.588+0000 7f7ae61d0740 10 client.18060 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:39:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.588+0000 7f7ae61d0740 10 client.18060 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:39:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.588+0000 7f7ae61d0740 20 client.18060 populate_metadata read hostname 'smithi050' 2024-04-23T18:39:30.725 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.588+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5596e49a4870 0x5596e49a6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.588+0000 7f7ae61d0740 4 client.18060 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:39:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.588+0000 7f7ae61d0740 1 -- 172.21.15.50:0/2353911086 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596e48fc390 con 0x5596e49a4870 2024-04-23T18:39:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.588+0000 7f7ae1184640 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5596e49a4870 0x5596e49a6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:30.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.589+0000 7f7ae1184640 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5596e49a4870 0x5596e49a6ca0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=378cc58597c5255e server_cookie=f37f9cd0b026d85 in_seq=0 out_seq=0 2024-04-23T18:39:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.589+0000 7f7ac4ff9640 10 client.18060 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:30.770 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ac4ff9640 1 -- 172.21.15.50:0/2353911086 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5596e48fc390 con 0x5596e49a4870 2024-04-23T18:39:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ac4ff9640 10 client.18060 handle_command_reply: tid=0 2024-04-23T18:39:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ae61d0740 1 client.18060 shutdown 2024-04-23T18:39:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ae61d0740 1 -- 172.21.15.50:0/2353911086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989620 msgr2=0x5596e4994170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989620 0x5596e4994170 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f7acc004370 tx=0x7f7acc004730 comp rx=0 tx=0).stop 2024-04-23T18:39:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ae61d0740 1 -- 172.21.15.50:0/2353911086 shutdown_connections 2024-04-23T18:39:30.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5596e49a4870 0x5596e49a6ca0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4994800 0x5596e49919c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989f40 0x5596e4991460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/2353911086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989620 0x5596e4994170 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:31.049 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ac4ff9640 0 client.18060 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:31.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ae61d0740 1 -- 172.21.15.50:0/2353911086 >> 172.21.15.50:0/2353911086 conn(0x5596e4985b30 msgr2=0x5596e498b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ae61d0740 1 -- 172.21.15.50:0/2353911086 shutdown_connections 2024-04-23T18:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ae61d0740 1 -- 172.21.15.50:0/2353911086 wait complete. 2024-04-23T18:39:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.590+0000 7f7ae61d0740 20 client.18060 trim_cache size 0 max 16384 2024-04-23T18:39:31.092 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.58 MB/sec execute 362 sec latency 28198.683 ms 2024-04-23T18:39:31.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:30 smithi188 ceph-mon[32694]: pgmap v668: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 3.1 MiB/s wr, 80 op/s 2024-04-23T18:39:31.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:30 smithi177 ceph-mon[32308]: pgmap v668: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 3.1 MiB/s wr, 80 op/s 2024-04-23T18:39:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.605+0000 7f7ae61d0740 1 Processor -- start 2024-04-23T18:39:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.605+0000 7f7ae61d0740 1 -- start start 2024-04-23T18:39:31.146 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.605+0000 7f7ae61d0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4989620 0x5596e49a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.605+0000 7f7ae61d0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 0x5596e4984b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:31.211 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.605+0000 7f7ae61d0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4994800 0x5596e49ac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:31.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.605+0000 7f7ae61d0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1b6840 con 0x5596e4989f40 2024-04-23T18:39:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.605+0000 7f7ae61d0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1b4110 con 0x5596e4994800 2024-04-23T18:39:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.605+0000 7f7ae61d0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1b1f80 con 0x5596e4989620 2024-04-23T18:39:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.605+0000 7f7ae1184640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 0x5596e4984b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.605+0000 7f7ae1184640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 0x5596e4984b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49924/0 (socket says 172.21.15.50:49924) 2024-04-23T18:39:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.605+0000 7f7ae1184640 1 -- 172.21.15.50:0/523625297 learned_addr learned my addr 172.21.15.50:0/523625297 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.605+0000 7f7ae0983640 1 --2- 172.21.15.50:0/523625297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4989620 0x5596e49a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.606+0000 7f7ae0983640 1 --2- 172.21.15.50:0/523625297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4994800 0x5596e49ac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:31.388 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.606+0000 7f7ae1184640 1 -- 172.21.15.50:0/523625297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4989620 msgr2=0x5596e49a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.606+0000 7f7ae1184640 1 --2- 172.21.15.50:0/523625297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4989620 0x5596e49a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:31.411 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.606+0000 7f7ae1184640 1 -- 172.21.15.50:0/523625297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4994800 msgr2=0x5596e49ac5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:31.420 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.606+0000 7f7ae1184640 1 --2- 172.21.15.50:0/523625297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4994800 0x5596e49ac5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:31.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.606+0000 7f7ae1184640 1 -- 172.21.15.50:0/523625297 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596e4990450 con 0x5596e4989f40 2024-04-23T18:39:31.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.606+0000 7f7ae0983640 1 --2- 172.21.15.50:0/523625297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4989620 0x5596e49a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:39:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.606+0000 7f7ae1184640 1 --2- 172.21.15.50:0/523625297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 0x5596e4984b30 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f7ad80114d0 tx=0x7f7ad800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7099c0b33a374b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:31.503 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.606+0000 7f7ac6ffd640 1 -- 172.21.15.50:0/523625297 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8015040 con 0x5596e4989f40 2024-04-23T18:39:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.606+0000 7f7ac6ffd640 1 -- 172.21.15.50:0/523625297 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ad80054d0 con 0x5596e4989f40 2024-04-23T18:39:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.606+0000 7f7ac6ffd640 1 -- 172.21.15.50:0/523625297 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8026080 con 0x5596e4989f40 2024-04-23T18:39:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.607+0000 7f7ae61d0740 1 -- 172.21.15.50:0/523625297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 msgr2=0x5596e4984b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:31.584 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.607+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/523625297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 0x5596e4984b30 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f7ad80114d0 tx=0x7f7ad800be00 comp rx=0 tx=0).stop 2024-04-23T18:39:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.607+0000 7f7ae61d0740 1 -- 172.21.15.50:0/523625297 shutdown_connections 2024-04-23T18:39:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.607+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/523625297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4994800 0x5596e49ac5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.607+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/523625297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4989f40 0x5596e4984b30 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.607+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/523625297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4989620 0x5596e49a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.607+0000 7f7ae61d0740 1 -- 172.21.15.50:0/523625297 >> 172.21.15.50:0/523625297 conn(0x5596e4985b30 msgr2=0x5596e4984420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.607+0000 7f7ae61d0740 1 -- 172.21.15.50:0/523625297 shutdown_connections 2024-04-23T18:39:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.607+0000 7f7ae61d0740 1 -- 172.21.15.50:0/523625297 wait complete. 2024-04-23T18:39:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.607+0000 7f7ae61d0740 1 -- start start 2024-04-23T18:39:31.896 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.607+0000 7f7ae61d0740 1 Processor -- start 2024-04-23T18:39:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.607+0000 7f7ae61d0740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4989620 0x5596e499a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:31.912 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae61d0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989f40 0x5596e499a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae61d0740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4994800 0x5596e499ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:31.973 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae61d0740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1b25d0 con 0x5596e4994800 2024-04-23T18:39:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae61d0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1b3a40 con 0x5596e4989f40 2024-04-23T18:39:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae61d0740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1b2ca0 con 0x5596e4989620 2024-04-23T18:39:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae61d0740 10 client.0 _ll_register_callbacks cb 0x7f7acc0041c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:39:32.056 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae0983640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4994800 0x5596e499ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae0983640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4994800 0x5596e499ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49936/0 (socket says 172.21.15.50:49936) 2024-04-23T18:39:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae0983640 1 -- 172.21.15.50:0/3587758960 learned_addr learned my addr 172.21.15.50:0/3587758960 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:32.171 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae1184640 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989f40 0x5596e499a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae0983640 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4989620 0x5596e499a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae0983640 1 -- 172.21.15.50:0/3587758960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4989620 msgr2=0x5596e499a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae0983640 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4989620 0x5596e499a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae0983640 1 -- 172.21.15.50:0/3587758960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989f40 msgr2=0x5596e499a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae0983640 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989f40 0x5596e499a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae0983640 1 -- 172.21.15.50:0/3587758960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596e4989d70 con 0x5596e4994800 2024-04-23T18:39:32.236 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae0983640 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4994800 0x5596e499ad60 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0042f0 tx=0x7f7acc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e474b9adc4de52b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:32.242 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ac4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:39:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ac4ff9640 1 -- 172.21.15.50:0/3587758960 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7acc0047a0 con 0x5596e4994800 2024-04-23T18:39:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ac4ff9640 1 -- 172.21.15.50:0/3587758960 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7f7acc004940 con 0x5596e4994800 2024-04-23T18:39:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3587758960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ad80054d0 con 0x5596e4994800 2024-04-23T18:39:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ac4ff9640 1 -- 172.21.15.50:0/3587758960 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f7acc004b40 con 0x5596e4994800 2024-04-23T18:39:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.608+0000 7f7ae61d0740 10 client.18066 fetch_fsmap learned FSMap version 62 2024-04-23T18:39:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.609+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3587758960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5596e4a0b000 con 0x5596e4994800 2024-04-23T18:39:32.567 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.609+0000 7f7ac4ff9640 1 -- 172.21.15.50:0/3587758960 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ad80054d0 con 0x5596e4994800 2024-04-23T18:39:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.609+0000 7f7ac4ff9640 1 -- 172.21.15.50:0/3587758960 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f7acc012480 con 0x5596e4994800 2024-04-23T18:39:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.609+0000 7f7ae61d0740 10 client.18066 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:39:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.609+0000 7f7ae61d0740 10 client.18066 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:39:32.711 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.609+0000 7f7ae61d0740 20 client.18066 populate_metadata read hostname 'smithi050' 2024-04-23T18:39:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.609+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5596e4a0b7e0 0x5596e4a0dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.609+0000 7f7ae61d0740 4 client.18066 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:39:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.609+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3587758960 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5596e48f6200 con 0x5596e4a0b7e0 2024-04-23T18:39:32.788 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.609+0000 7f7ae1184640 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5596e4a0b7e0 0x5596e4a0dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.610+0000 7f7ae1184640 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5596e4a0b7e0 0x5596e4a0dc00 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f53bdcb68566e4c server_cookie=efaf231867997693 in_seq=0 out_seq=0 2024-04-23T18:39:32.804 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.610+0000 7f7ac4ff9640 10 client.18066 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:32.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:28 smithi050 ceph-mon[29187]: pgmap v667: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 2.3 MiB/s wr, 69 op/s 2024-04-23T18:39:32.822 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:32 smithi188 ceph-mon[32694]: pgmap v669: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 3.1 MiB/s wr, 76 op/s 2024-04-23T18:39:32.831 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:32 smithi177 ceph-mon[32308]: pgmap v669: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 3.1 MiB/s wr, 76 op/s 2024-04-23T18:39:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ac4ff9640 1 -- 172.21.15.50:0/3587758960 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5596e48f6200 con 0x5596e4a0b7e0 2024-04-23T18:39:32.895 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ac4ff9640 10 client.18066 handle_command_reply: tid=0 2024-04-23T18:39:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ae61d0740 1 client.18066 shutdown 2024-04-23T18:39:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3587758960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4994800 msgr2=0x5596e499ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4994800 0x5596e499ad60 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0042f0 tx=0x7f7acc004730 comp rx=0 tx=0).stop 2024-04-23T18:39:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3587758960 shutdown_connections 2024-04-23T18:39:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x5596e4a0b7e0 0x5596e4a0dc00 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:33.110 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5596e4994800 0x5596e499ad60 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:33.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ac4ff9640 0 client.18066 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596e4989f40 0x5596e499a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ae61d0740 1 --2- 172.21.15.50:0/3587758960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5596e4989620 0x5596e499a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:33.139 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3587758960 >> 172.21.15.50:0/3587758960 conn(0x5596e4985b30 msgr2=0x5596e4984cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3587758960 shutdown_connections 2024-04-23T18:39:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ae61d0740 1 -- 172.21.15.50:0/3587758960 wait complete. 2024-04-23T18:39:33.164 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.726+0000 7f7ae61d0740 20 client.18066 trim_cache size 0 max 16384 2024-04-23T18:39:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:39:33.235 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:39:33.243 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:39:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:39:33.295 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.727+0000 7f7ae33e8640 1 -- 172.21.15.50:0/3535025947 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7ac80a3dc0 msgr2=0x7f7ac80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:33.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.727+0000 7f7ae33e8640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7ac80a3dc0 0x7f7ac80a6280 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40bcd30 tx=0x7f7ad408f890 comp rx=0 tx=0).stop 2024-04-23T18:39:33.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.727+0000 7f7ae33e8640 1 -- 172.21.15.50:0/3535025947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7adc1063f0 msgr2=0x7f7adc19ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.727+0000 7f7ae33e8640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7adc1063f0 0x7f7adc19ea90 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009df0 tx=0x7f7ad4003970 comp rx=0 tx=0).stop 2024-04-23T18:39:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.727+0000 7f7ae33e8640 1 -- 172.21.15.50:0/3535025947 shutdown_connections 2024-04-23T18:39:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.727+0000 7f7ae33e8640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7ac80a3dc0 0x7f7ac80a6280 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:33.416 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.727+0000 7f7ae33e8640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7adc1063f0 0x7f7adc19ea90 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.727+0000 7f7ae33e8640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7adc101910 0x7f7adc1a4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:33.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.727+0000 7f7ae33e8640 1 --2- 172.21.15.50:0/3535025947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7adc100ff0 0x7f7adc1a4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:33.442 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.727+0000 7f7ae33e8640 1 -- 172.21.15.50:0/3535025947 >> 172.21.15.50:0/3535025947 conn(0x7f7adc0771d0 msgr2=0x7f7adc103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.727+0000 7f7ae33e8640 1 -- 172.21.15.50:0/3535025947 shutdown_connections 2024-04-23T18:39:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:28.727+0000 7f7ae33e8640 1 -- 172.21.15.50:0/3535025947 wait complete. 2024-04-23T18:39:33.466 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.53 MB/sec execute 363 sec latency 29198.792 ms 2024-04-23T18:39:33.517 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.48 MB/sec execute 364 sec latency 30198.855 ms 2024-04-23T18:39:33.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:30 smithi050 ceph-mon[29187]: pgmap v668: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 3.1 MiB/s wr, 80 op/s 2024-04-23T18:39:33.533 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.44 MB/sec execute 365 sec latency 31198.954 ms 2024-04-23T18:39:33.541 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.39 MB/sec execute 366 sec latency 32199.032 ms 2024-04-23T18:39:33.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:32 smithi050 ceph-mon[29187]: pgmap v669: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 3.1 MiB/s wr, 76 op/s 2024-04-23T18:39:33.696 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.34 MB/sec execute 367 sec latency 33199.116 ms 2024-04-23T18:39:34.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:34 smithi177 ceph-mon[32308]: pgmap v670: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 3.3 MiB/s wr, 108 op/s 2024-04-23T18:39:34.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:34 smithi188 ceph-mon[32694]: pgmap v670: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 3.3 MiB/s wr, 108 op/s 2024-04-23T18:39:34.586 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.29 MB/sec execute 368 sec latency 34199.184 ms 2024-04-23T18:39:34.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:39:34.687 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:34 smithi050 ceph-mon[29187]: pgmap v670: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 3.3 MiB/s wr, 108 op/s 2024-04-23T18:39:34.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.785+0000 7f3b331bf640 1 -- 172.21.15.50:0/1528866462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b2c1010a0 msgr2=0x7f3b2c101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:34.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.785+0000 7f3b331bf640 1 --2- 172.21.15.50:0/1528866462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b2c1010a0 0x7f3b2c101480 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009900 tx=0x7f3b1c02a410 comp rx=0 tx=0).stop 2024-04-23T18:39:34.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.785+0000 7f3b331bf640 1 -- 172.21.15.50:0/1528866462 shutdown_connections 2024-04-23T18:39:34.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.785+0000 7f3b331bf640 1 --2- 172.21.15.50:0/1528866462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b2c106170 0x7f3b2c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:34.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.785+0000 7f3b331bf640 1 --2- 172.21.15.50:0/1528866462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b2c101a50 0x7f3b2c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:34.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.785+0000 7f3b331bf640 1 --2- 172.21.15.50:0/1528866462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b2c1010a0 0x7f3b2c101480 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:34.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.785+0000 7f3b331bf640 1 -- 172.21.15.50:0/1528866462 >> 172.21.15.50:0/1528866462 conn(0x7f3b2c077f50 msgr2=0x7f3b2c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:34.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.785+0000 7f3b331bf640 1 -- 172.21.15.50:0/1528866462 shutdown_connections 2024-04-23T18:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.785+0000 7f3b331bf640 1 -- 172.21.15.50:0/1528866462 wait complete. 2024-04-23T18:39:34.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b331bf640 1 Processor -- start 2024-04-23T18:39:35.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b331bf640 1 -- start start 2024-04-23T18:39:35.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b331bf640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b2c1010a0 0x7f3b2c10e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:35.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b331bf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b2c106170 0x7f3b2c10ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:35.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b331bf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b2c10f150 0x7f3b2c1a6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:35.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b331bf640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c1121f0 con 0x7f3b2c10f150 2024-04-23T18:39:35.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b331bf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c112070 con 0x7f3b2c106170 2024-04-23T18:39:35.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b331bf640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c112370 con 0x7f3b2c1010a0 2024-04-23T18:39:35.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b3175c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b2c10f150 0x7f3b2c1a6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:35.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b3175c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b2c10f150 0x7f3b2c1a6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56588/0 (socket says 172.21.15.50:56588) 2024-04-23T18:39:35.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b3175c640 1 -- 172.21.15.50:0/73478330 learned_addr learned my addr 172.21.15.50:0/73478330 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:35.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b2bfff640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b2c106170 0x7f3b2c10ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:35.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b30f5b640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b2c1010a0 0x7f3b2c10e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:35.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b3175c640 1 -- 172.21.15.50:0/73478330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b2c1010a0 msgr2=0x7f3b2c10e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:35.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b3175c640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b2c1010a0 0x7f3b2c10e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:35.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b3175c640 1 -- 172.21.15.50:0/73478330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b2c106170 msgr2=0x7f3b2c10ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:35.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b3175c640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b2c106170 0x7f3b2c10ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.786+0000 7f3b3175c640 1 -- 172.21.15.50:0/73478330 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2c1a7020 con 0x7f3b2c10f150 2024-04-23T18:39:35.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.787+0000 7f3b3175c640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b2c10f150 0x7f3b2c1a6920 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400eaa0 tx=0x7f3b2400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:35.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.787+0000 7f3b29ffb640 1 -- 172.21.15.50:0/73478330 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2400cdb0 con 0x7f3b2c10f150 2024-04-23T18:39:35.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.787+0000 7f3b29ffb640 1 -- 172.21.15.50:0/73478330 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b24005020 con 0x7f3b2c10f150 2024-04-23T18:39:35.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.787+0000 7f3b331bf640 1 -- 172.21.15.50:0/73478330 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2c1a7310 con 0x7f3b2c10f150 2024-04-23T18:39:35.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.787+0000 7f3b29ffb640 1 -- 172.21.15.50:0/73478330 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b240116d0 con 0x7f3b2c10f150 2024-04-23T18:39:35.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.787+0000 7f3b331bf640 1 -- 172.21.15.50:0/73478330 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b2c1a7860 con 0x7f3b2c10f150 2024-04-23T18:39:35.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.788+0000 7f3b331bf640 1 -- 172.21.15.50:0/73478330 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b2c068d70 con 0x7f3b2c10f150 2024-04-23T18:39:35.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.789+0000 7f3b29ffb640 1 -- 172.21.15.50:0/73478330 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3b240051d0 con 0x7f3b2c10f150 2024-04-23T18:39:35.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.790+0000 7f3b29ffb640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3b180a3d50 0x7f3b180a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:35.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.790+0000 7f3b30f5b640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3b180a3d50 0x7f3b180a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:35.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.790+0000 7f3b29ffb640 1 -- 172.21.15.50:0/73478330 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7f3b240d0900 con 0x7f3b2c10f150 2024-04-23T18:39:35.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.790+0000 7f3b30f5b640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3b180a3d50 0x7f3b180a6210 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c031440 tx=0x7f3b1c005b50 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:35.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:34.795+0000 7f3b29ffb640 1 -- 172.21.15.50:0/73478330 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b24016030 con 0x7f3b2c10f150 2024-04-23T18:39:35.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.021+0000 7f3b331bf640 1 -- 172.21.15.50:0/73478330 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b2c101a50 con 0x7f3b2c10f150 2024-04-23T18:39:35.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.022+0000 7f3b29ffb640 1 -- 172.21.15.50:0/73478330 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+11875 (secure 0 0 0) 0x7f3b24085e20 con 0x7f3b2c10f150 2024-04-23T18:39:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:39:35.653 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":62,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:36:29.984583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27194":{"gid":27194,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2754597719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2754597719},{"type":"v1","addr":"172.21.15.177:6837","nonce":2754597719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:39:35.708 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 62 2024-04-23T18:39:35.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.024+0000 7f3b331bf640 1 -- 172.21.15.50:0/73478330 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3b180a3d50 msgr2=0x7f3b180a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:35.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.024+0000 7f3b331bf640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3b180a3d50 0x7f3b180a6210 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c031440 tx=0x7f3b1c005b50 comp rx=0 tx=0).stop 2024-04-23T18:39:35.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.024+0000 7f3b331bf640 1 -- 172.21.15.50:0/73478330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b2c10f150 msgr2=0x7f3b2c1a6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:35.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.024+0000 7f3b331bf640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b2c10f150 0x7f3b2c1a6920 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400eaa0 tx=0x7f3b2400ef60 comp rx=0 tx=0).stop 2024-04-23T18:39:35.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.024+0000 7f3b331bf640 1 -- 172.21.15.50:0/73478330 shutdown_connections 2024-04-23T18:39:35.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.024+0000 7f3b331bf640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f3b180a3d50 0x7f3b180a6210 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.024+0000 7f3b331bf640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b2c10f150 0x7f3b2c1a6920 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:35.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.024+0000 7f3b331bf640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b2c106170 0x7f3b2c10ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:35.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.024+0000 7f3b331bf640 1 --2- 172.21.15.50:0/73478330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b2c1010a0 0x7f3b2c10e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:35.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.024+0000 7f3b331bf640 1 -- 172.21.15.50:0/73478330 >> 172.21.15.50:0/73478330 conn(0x7f3b2c077f50 msgr2=0x7f3b2c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:35.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.024+0000 7f3b331bf640 1 -- 172.21.15.50:0/73478330 shutdown_connections 2024-04-23T18:39:35.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:35.024+0000 7f3b331bf640 1 -- 172.21.15.50:0/73478330 wait complete. 2024-04-23T18:39:35.942 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --set-id 71fed75b --release --await' 2024-04-23T18:39:35.990 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --set-id 71fed75b --release --await 2024-04-23T18:39:35.999 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:35 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/73478330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:39:36.010 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170388 17.25 MB/sec execute 369 sec latency 35199.225 ms 2024-04-23T18:39:36.081 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:35 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/73478330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:39:36.090 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:35 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/73478330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:39:36.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.211+0000 7f7d3874a640 1 -- 172.21.15.50:0/2424377618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d30101b40 msgr2=0x7f7d3010dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:36.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.211+0000 7f7d3874a640 1 --2- 172.21.15.50:0/2424377618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d30101b40 0x7f7d3010dc80 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009cd0 tx=0x7f7d2802a8e0 comp rx=0 tx=0).stop 2024-04-23T18:39:36.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.211+0000 7f7d3874a640 1 -- 172.21.15.50:0/2424377618 shutdown_connections 2024-04-23T18:39:36.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.211+0000 7f7d3874a640 1 --2- 172.21.15.50:0/2424377618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d30101b40 0x7f7d3010dc80 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.211+0000 7f7d3874a640 1 --2- 172.21.15.50:0/2424377618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d301011a0 0x7f7d30101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.211+0000 7f7d3874a640 1 --2- 172.21.15.50:0/2424377618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d30107030 0x7f7d30107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:36.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.211+0000 7f7d3874a640 1 -- 172.21.15.50:0/2424377618 >> 172.21.15.50:0/2424377618 conn(0x7f7d30076c50 msgr2=0x7f7d30077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:36.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.211+0000 7f7d3874a640 1 -- 172.21.15.50:0/2424377618 shutdown_connections 2024-04-23T18:39:36.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.211+0000 7f7d3874a640 1 -- 172.21.15.50:0/2424377618 wait complete. 2024-04-23T18:39:36.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.211+0000 7f7d3874a640 1 Processor -- start 2024-04-23T18:39:36.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.211+0000 7f7d3874a640 1 -- start start 2024-04-23T18:39:36.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d3874a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d301011a0 0x7f7d30102b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:36.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d3874a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d30101b40 0x7f7d30103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:36.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d3874a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d30107030 0x7f7d301035d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:36.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d3874a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301110e0 con 0x7f7d30101b40 2024-04-23T18:39:36.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d3874a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7d30110f60 con 0x7f7d30107030 2024-04-23T18:39:36.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d3874a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7d30111260 con 0x7f7d301011a0 2024-04-23T18:39:36.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d35ce5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d30101b40 0x7f7d30103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:36.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d35ce5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d30101b40 0x7f7d30103090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56606/0 (socket says 172.21.15.50:56606) 2024-04-23T18:39:36.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d35ce5640 1 -- 172.21.15.50:0/2467235160 learned_addr learned my addr 172.21.15.50:0/2467235160 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:36.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d364e6640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d301011a0 0x7f7d30102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:36.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d30107030 0x7f7d301035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:36.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d35ce5640 1 -- 172.21.15.50:0/2467235160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d301011a0 msgr2=0x7f7d30102b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:36.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d35ce5640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d301011a0 0x7f7d30102b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:36.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d35ce5640 1 -- 172.21.15.50:0/2467235160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d30107030 msgr2=0x7f7d301035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:36.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d35ce5640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d30107030 0x7f7d301035d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:36.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d35ce5640 1 -- 172.21.15.50:0/2467235160 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d301a6cb0 con 0x7f7d30101b40 2024-04-23T18:39:36.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d30107030 0x7f7d301035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:39:36.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d35ce5640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d30101b40 0x7f7d30103090 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000e9f0 tx=0x7f7d2000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:36.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.212+0000 7f7d277fe640 1 -- 172.21.15.50:0/2467235160 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000cd80 con 0x7f7d30101b40 2024-04-23T18:39:36.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.213+0000 7f7d3874a640 1 -- 172.21.15.50:0/2467235160 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d301a6fa0 con 0x7f7d30101b40 2024-04-23T18:39:36.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.213+0000 7f7d277fe640 1 -- 172.21.15.50:0/2467235160 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d200054d0 con 0x7f7d30101b40 2024-04-23T18:39:36.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.213+0000 7f7d277fe640 1 -- 172.21.15.50:0/2467235160 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d20010690 con 0x7f7d30101b40 2024-04-23T18:39:36.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.213+0000 7f7d3874a640 1 -- 172.21.15.50:0/2467235160 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d301a7410 con 0x7f7d30101b40 2024-04-23T18:39:36.991 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:36 smithi188 ceph-mon[32694]: pgmap v671: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 3.3 MiB/s wr, 105 op/s 2024-04-23T18:39:37.042 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:36 smithi177 ceph-mon[32308]: pgmap v671: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 3.3 MiB/s wr, 105 op/s 2024-04-23T18:39:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.215+0000 7f7d277fe640 1 -- 172.21.15.50:0/2467235160 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7d20004e90 con 0x7f7d30101b40 2024-04-23T18:39:37.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.215+0000 7f7d3b45b740 1 Processor -- start 2024-04-23T18:39:37.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.215+0000 7f7d3b45b740 1 -- start start 2024-04-23T18:39:37.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d3b45b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527490 0x556a43527870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:37.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d3b45b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527db0 0x556a43531f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:37.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d3b45b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a43534870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:37.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d3b45b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556a4348c6d0 con 0x556a43532480 2024-04-23T18:39:37.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d3b45b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556a434291a0 con 0x556a43527490 2024-04-23T18:39:37.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d3b45b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556a43488690 con 0x556a43527db0 2024-04-23T18:39:37.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d277fe640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7d1c0a3d20 0x7f7d1c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:37.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d35ce5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a43534870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:37.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d35ce5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a43534870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56610/0 (socket says 172.21.15.50:56610) 2024-04-23T18:39:37.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d35ce5640 1 -- 172.21.15.50:0/1427793216 learned_addr learned my addr 172.21.15.50:0/1427793216 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:37.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7d1c0a3d20 0x7f7d1c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:37.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d364e6640 1 --2- 172.21.15.50:0/1427793216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527db0 0x556a43531f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:37.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/1427793216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527490 0x556a43527870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:37.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d277fe640 1 -- 172.21.15.50:0/2467235160 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7f7d200d0d50 con 0x7f7d30101b40 2024-04-23T18:39:37.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d35ce5640 1 -- 172.21.15.50:0/1427793216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527db0 msgr2=0x556a43531f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d35ce5640 1 --2- 172.21.15.50:0/1427793216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527db0 0x556a43531f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:37.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d35ce5640 1 -- 172.21.15.50:0/1427793216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527490 msgr2=0x556a43527870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:37.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d35ce5640 1 --2- 172.21.15.50:0/1427793216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527490 0x556a43527870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:37.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.216+0000 7f7d35ce5640 1 -- 172.21.15.50:0/1427793216 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a434335b0 con 0x556a43532480 2024-04-23T18:39:37.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.217+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7d1c0a3d20 0x7f7d1c0a61e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7d28033040 tx=0x7f7d28009910 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:37.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.217+0000 7f7d35ce5640 1 --2- 172.21.15.50:0/1427793216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a43534870 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000c6a0 tx=0x7f7d2009cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11808f3bd8a564 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:37.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.217+0000 7f7d24ff9640 1 -- 172.21.15.50:0/1427793216 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2009d3e0 con 0x556a43532480 2024-04-23T18:39:37.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.217+0000 7f7d24ff9640 1 -- 172.21.15.50:0/1427793216 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d2009cce0 con 0x556a43532480 2024-04-23T18:39:37.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.217+0000 7f7d24ff9640 1 -- 172.21.15.50:0/1427793216 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d200ce710 con 0x556a43532480 2024-04-23T18:39:37.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.217+0000 7f7d3b45b740 1 -- 172.21.15.50:0/1427793216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 msgr2=0x556a43534870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:37.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.217+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/1427793216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a43534870 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000c6a0 tx=0x7f7d2009cbe0 comp rx=0 tx=0).stop 2024-04-23T18:39:37.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.217+0000 7f7d3b45b740 1 -- 172.21.15.50:0/1427793216 shutdown_connections 2024-04-23T18:39:37.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.217+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/1427793216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a43534870 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:37.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/1427793216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527db0 0x556a43531f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:37.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/1427793216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527490 0x556a43527870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:37.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 -- 172.21.15.50:0/1427793216 >> 172.21.15.50:0/1427793216 conn(0x556a435237b0 msgr2=0x556a43523b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:37.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 -- 172.21.15.50:0/1427793216 shutdown_connections 2024-04-23T18:39:37.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 -- 172.21.15.50:0/1427793216 wait complete. 2024-04-23T18:39:37.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 -- start start 2024-04-23T18:39:37.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 Processor -- start 2024-04-23T18:39:37.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a43531ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:37.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 0x556a4352f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:37.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a4352f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:37.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x556a435352b0 con 0x556a43532480 2024-04-23T18:39:37.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556a43426b20 con 0x556a43527db0 2024-04-23T18:39:37.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x556a43535430 con 0x556a43527490 2024-04-23T18:39:37.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d3b45b740 10 client.0 _ll_register_callbacks cb 0x556a43426530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:39:37.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d36ce7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a4352f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:37.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d36ce7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a4352f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56612/0 (socket says 172.21.15.50:56612) 2024-04-23T18:39:38.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d36ce7640 1 -- 172.21.15.50:0/3108502872 learned_addr learned my addr 172.21.15.50:0/3108502872 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:38.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d364e6640 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a43531ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:38.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.218+0000 7f7d35ce5640 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 0x556a4352f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:38.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d36ce7640 1 -- 172.21.15.50:0/3108502872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 msgr2=0x556a43531ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:38.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a43531ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:38.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d36ce7640 1 -- 172.21.15.50:0/3108502872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 msgr2=0x556a4352f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:38.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 0x556a4352f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:38.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d36ce7640 1 -- 172.21.15.50:0/3108502872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a4319e920 con 0x556a43532480 2024-04-23T18:39:38.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a4352f900 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009430 tx=0x7f7d28046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26e7ae99cd8cece server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:38.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:39:38.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d19ffb640 1 -- 172.21.15.50:0/3108502872 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28052070 con 0x556a43532480 2024-04-23T18:39:38.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d3b45b740 1 -- 172.21.15.50:0/3108502872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d2009cce0 con 0x556a43532480 2024-04-23T18:39:38.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d19ffb640 1 -- 172.21.15.50:0/3108502872 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7f7d280047c0 con 0x556a43532480 2024-04-23T18:39:38.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d19ffb640 1 -- 172.21.15.50:0/3108502872 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f7d280305a0 con 0x556a43532480 2024-04-23T18:39:38.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d3b45b740 10 client.18084 fetch_fsmap learned FSMap version 62 2024-04-23T18:39:38.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.219+0000 7f7d3b45b740 1 -- 172.21.15.50:0/3108502872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x556a4326ad30 con 0x556a43532480 2024-04-23T18:39:38.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.220+0000 7f7d19ffb640 1 -- 172.21.15.50:0/3108502872 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d2009cce0 con 0x556a43532480 2024-04-23T18:39:38.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.220+0000 7f7d19ffb640 1 -- 172.21.15.50:0/3108502872 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f7d28030070 con 0x556a43532480 2024-04-23T18:39:38.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.220+0000 7f7d3b45b740 10 client.18084 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:39:38.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.220+0000 7f7d3b45b740 10 client.18084 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:39:38.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.220+0000 7f7d3b45b740 20 client.18084 populate_metadata read hostname 'smithi050' 2024-04-23T18:39:38.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.220+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556a435425c0 0x556a435449f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:38.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.220+0000 7f7d3b45b740 4 client.18084 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:39:38.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.220+0000 7f7d3b45b740 1 -- 172.21.15.50:0/3108502872 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a4344a500 con 0x556a435425c0 2024-04-23T18:39:38.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.220+0000 7f7d364e6640 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556a435425c0 0x556a435449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:38.458 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.221+0000 7f7d364e6640 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556a435425c0 0x556a435449f0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8eae072bcbfbf49e server_cookie=e69a36cc92a23b4a in_seq=0 out_seq=0 2024-04-23T18:39:38.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.221+0000 7f7d19ffb640 10 client.18084 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:38.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.221+0000 7f7d19ffb640 1 -- 172.21.15.50:0/3108502872 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a4344a500 con 0x556a435425c0 2024-04-23T18:39:38.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.221+0000 7f7d19ffb640 10 client.18084 handle_command_reply: tid=0 2024-04-23T18:39:38.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.221+0000 7f7d3b45b740 1 client.18084 shutdown 2024-04-23T18:39:38.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.222+0000 7f7d3b45b740 1 -- 172.21.15.50:0/3108502872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 msgr2=0x556a4352f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:38.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.222+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a4352f900 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009430 tx=0x7f7d28046000 comp rx=0 tx=0).stop 2024-04-23T18:39:38.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.222+0000 7f7d3b45b740 1 -- 172.21.15.50:0/3108502872 shutdown_connections 2024-04-23T18:39:38.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.222+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556a435425c0 0x556a435449f0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:38.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.222+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a4352f900 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:38.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.222+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 0x556a4352f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:38.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.222+0000 7f7d19ffb640 0 client.18084 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:38.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.222+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/3108502872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a43531ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:38.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.222+0000 7f7d3b45b740 1 -- 172.21.15.50:0/3108502872 >> 172.21.15.50:0/3108502872 conn(0x556a435237b0 msgr2=0x556a435295b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:38.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.222+0000 7f7d3b45b740 1 -- 172.21.15.50:0/3108502872 shutdown_connections 2024-04-23T18:39:38.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.222+0000 7f7d3b45b740 1 -- 172.21.15.50:0/3108502872 wait complete. 2024-04-23T18:39:38.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.222+0000 7f7d3b45b740 20 client.18084 trim_cache size 0 max 16384 2024-04-23T18:39:38.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.239+0000 7f7d3b45b740 1 Processor -- start 2024-04-23T18:39:38.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.239+0000 7f7d3b45b740 1 -- start start 2024-04-23T18:39:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d3b45b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a43529c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:38.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d3b45b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 0x556a43531a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:38.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d3b45b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a43531f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:38.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d3b45b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301b26f0 con 0x556a43532480 2024-04-23T18:39:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d3b45b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301affc0 con 0x556a43527db0 2024-04-23T18:39:38.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d3b45b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301ade30 con 0x556a43527490 2024-04-23T18:39:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d36ce7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a43531f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:39.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d36ce7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a43531f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56620/0 (socket says 172.21.15.50:56620) 2024-04-23T18:39:39.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d36ce7640 1 -- 172.21.15.50:0/1354064652 learned_addr learned my addr 172.21.15.50:0/1354064652 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d364e6640 1 --2- 172.21.15.50:0/1354064652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a43529c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:39.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d35ce5640 1 --2- 172.21.15.50:0/1354064652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 0x556a43531a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d36ce7640 1 -- 172.21.15.50:0/1354064652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 msgr2=0x556a43529c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:39.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/1354064652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a43529c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:39.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d36ce7640 1 -- 172.21.15.50:0/1354064652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 msgr2=0x556a43531a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:39.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/1354064652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 0x556a43531a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:39.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d36ce7640 1 -- 172.21.15.50:0/1354064652 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a4352d710 con 0x556a43532480 2024-04-23T18:39:39.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/1354064652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a43531f40 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f7d28030360 tx=0x7f7d28005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5aeb723ec3b23f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d1bfff640 1 -- 172.21.15.50:0/1354064652 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28052070 con 0x556a43532480 2024-04-23T18:39:39.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d1bfff640 1 -- 172.21.15.50:0/1354064652 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d28036040 con 0x556a43532480 2024-04-23T18:39:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.240+0000 7f7d1bfff640 1 -- 172.21.15.50:0/1354064652 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28030740 con 0x556a43532480 2024-04-23T18:39:39.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.241+0000 7f7d3b45b740 1 -- 172.21.15.50:0/1354064652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 msgr2=0x556a43531f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:39.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.241+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/1354064652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a43531f40 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f7d28030360 tx=0x7f7d28005c50 comp rx=0 tx=0).stop 2024-04-23T18:39:39.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.241+0000 7f7d3b45b740 1 -- 172.21.15.50:0/1354064652 shutdown_connections 2024-04-23T18:39:39.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.241+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/1354064652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a43531f40 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:39.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.241+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/1354064652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 0x556a43531a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:39.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.241+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/1354064652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a43529c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:39.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.241+0000 7f7d3b45b740 1 -- 172.21.15.50:0/1354064652 >> 172.21.15.50:0/1354064652 conn(0x556a435237b0 msgr2=0x556a435229d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:39.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.241+0000 7f7d3b45b740 1 -- 172.21.15.50:0/1354064652 shutdown_connections 2024-04-23T18:39:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.241+0000 7f7d3b45b740 1 -- 172.21.15.50:0/1354064652 wait complete. 2024-04-23T18:39:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.241+0000 7f7d3b45b740 1 -- start start 2024-04-23T18:39:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.241+0000 7f7d3b45b740 1 Processor -- start 2024-04-23T18:39:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d3b45b740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a4353a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:39.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d3b45b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 0x556a4352b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:39.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d3b45b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a4352b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:39.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d3b45b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301ae480 con 0x556a43532480 2024-04-23T18:39:39.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d3b45b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301af8f0 con 0x556a43527db0 2024-04-23T18:39:39.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d3b45b740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301aeb50 con 0x556a43527490 2024-04-23T18:39:39.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d3b45b740 10 client.0 _ll_register_callbacks cb 0x7f7d28009360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:39:39.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d36ce7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a4352b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d36ce7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a4352b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56626/0 (socket says 172.21.15.50:56626) 2024-04-23T18:39:39.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d36ce7640 1 -- 172.21.15.50:0/2391724725 learned_addr learned my addr 172.21.15.50:0/2391724725 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:39.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d35ce5640 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 0x556a4352b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:39.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d364e6640 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a4353a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:39.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d36ce7640 1 -- 172.21.15.50:0/2391724725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 msgr2=0x556a4353a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:39.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a4353a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:39.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d36ce7640 1 -- 172.21.15.50:0/2391724725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 msgr2=0x556a4352b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:39.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 0x556a4352b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d36ce7640 1 -- 172.21.15.50:0/2391724725 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a4352beb0 con 0x556a43532480 2024-04-23T18:39:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d364e6640 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a4353a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:39:39.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d36ce7640 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a4352b970 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f7d28004560 tx=0x7f7d2804caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e89d8125d73893 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:39:39.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d19ffb640 1 -- 172.21.15.50:0/2391724725 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28067060 con 0x556a43532480 2024-04-23T18:39:39.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d19ffb640 1 -- 172.21.15.50:0/2391724725 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7f7d28004d50 con 0x556a43532480 2024-04-23T18:39:39.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.242+0000 7f7d3b45b740 1 -- 172.21.15.50:0/2391724725 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d28036040 con 0x556a43532480 2024-04-23T18:39:39.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.243+0000 7f7d19ffb640 1 -- 172.21.15.50:0/2391724725 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f7d28052040 con 0x556a43532480 2024-04-23T18:39:39.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.243+0000 7f7d3b45b740 10 client.18090 fetch_fsmap learned FSMap version 62 2024-04-23T18:39:39.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.243+0000 7f7d3b45b740 1 -- 172.21.15.50:0/2391724725 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x556a4352d160 con 0x556a43532480 2024-04-23T18:39:39.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.243+0000 7f7d19ffb640 1 -- 172.21.15.50:0/2391724725 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d28036040 con 0x556a43532480 2024-04-23T18:39:39.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.243+0000 7f7d19ffb640 1 -- 172.21.15.50:0/2391724725 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f7d28005330 con 0x556a43532480 2024-04-23T18:39:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.243+0000 7f7d3b45b740 10 client.18090 fetch_fsmap finished waiting for FSMap version 62 2024-04-23T18:39:39.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.243+0000 7f7d3b45b740 10 client.18090 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:39:39.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.243+0000 7f7d3b45b740 20 client.18090 populate_metadata read hostname 'smithi050' 2024-04-23T18:39:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.243+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556a435a9f20 0x556a435ac340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:39.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.243+0000 7f7d3b45b740 4 client.18090 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "71fed75b", "release": true, "await": true}] 2024-04-23T18:39:39.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.243+0000 7f7d3b45b740 1 -- 172.21.15.50:0/2391724725 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "set_id": "71fed75b", "release": true, "await": true}) -- 0x556a4349a0f0 con 0x556a435a9f20 2024-04-23T18:39:39.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.243+0000 7f7d364e6640 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556a435a9f20 0x556a435ac340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:39.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.244+0000 7f7d19ffb640 10 client.18090 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:40.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.244+0000 7f7d364e6640 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556a435a9f20 0x556a435ac340 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a66ec5b14b317012 server_cookie=73ea20fdd0914291 in_seq=0 out_seq=0 2024-04-23T18:39:40.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d19ffb640 1 -- 172.21.15.50:0/2391724725 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x556a4349a0f0 con 0x556a435a9f20 2024-04-23T18:39:40.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d19ffb640 10 client.18090 handle_command_reply: tid=0 2024-04-23T18:39:40.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d3b45b740 1 client.18090 shutdown 2024-04-23T18:39:40.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d3b45b740 1 -- 172.21.15.50:0/2391724725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 msgr2=0x556a4352b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:40.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a4352b970 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f7d28004560 tx=0x7f7d2804caf0 comp rx=0 tx=0).stop 2024-04-23T18:39:40.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d3b45b740 1 -- 172.21.15.50:0/2391724725 shutdown_connections 2024-04-23T18:39:40.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x556a435a9f20 0x556a435ac340 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:40.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x556a43532480 0x556a4352b970 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:40.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a43527db0 0x556a4352b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:40.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d19ffb640 0 client.18090 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:40.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d3b45b740 1 --2- 172.21.15.50:0/2391724725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x556a43527490 0x556a4353a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d3b45b740 1 -- 172.21.15.50:0/2391724725 >> 172.21.15.50:0/2391724725 conn(0x556a435237b0 msgr2=0x556a43528c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:40.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d3b45b740 1 -- 172.21.15.50:0/2391724725 shutdown_connections 2024-04-23T18:39:40.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d3b45b740 1 -- 172.21.15.50:0/2391724725 wait complete. 2024-04-23T18:39:40.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.280+0000 7f7d3b45b740 20 client.18090 trim_cache size 0 max 16384 2024-04-23T18:39:40.273 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:39:40.352 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 62, 2024-04-23T18:39:40.362 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:39:40.373 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 72, 2024-04-23T18:39:40.381 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": { 2024-04-23T18:39:40.434 INFO:teuthology.orchestra.run.smithi050.stdout: "71fed75b": { 2024-04-23T18:39:40.442 INFO:teuthology.orchestra.run.smithi050.stdout: "version": 72, 2024-04-23T18:39:40.449 INFO:teuthology.orchestra.run.smithi050.stdout: "age_ref": 0.0, 2024-04-23T18:39:40.480 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:39:40.524 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:39:40.542 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:39:40.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:39:40.557 INFO:teuthology.orchestra.run.smithi050.stdout: "timeout": 60.0, 2024-04-23T18:39:40.564 INFO:teuthology.orchestra.run.smithi050.stdout: "expiration": 87.0, 2024-04-23T18:39:40.575 INFO:teuthology.orchestra.run.smithi050.stdout: "members": { 2024-04-23T18:39:40.583 INFO:teuthology.orchestra.run.smithi050.stdout: "file:/": { 2024-04-23T18:39:40.589 INFO:teuthology.orchestra.run.smithi050.stdout: "excluded": false, 2024-04-23T18:39:40.672 INFO:teuthology.orchestra.run.smithi050.stdout: "state": { 2024-04-23T18:39:40.681 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RELEASED", 2024-04-23T18:39:40.687 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0 2024-04-23T18:39:40.696 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:39:40.704 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:39:40.758 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:39:40.765 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:39:40.772 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:39:40.824 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:39:40.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:38 smithi188 ceph-mon[32694]: pgmap v672: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 3.6 MiB/s wr, 127 op/s 2024-04-23T18:39:40.954 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:40 smithi188 ceph-mon[32694]: pgmap v673: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-04-23T18:39:40.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:40 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:39:40.967 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:38 smithi177 ceph-mon[32308]: pgmap v672: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 3.6 MiB/s wr, 127 op/s 2024-04-23T18:39:40.973 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:40 smithi177 ceph-mon[32308]: pgmap v673: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-04-23T18:39:40.980 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:40 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:39:40.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.281+0000 7f7d3874a640 1 -- 172.21.15.50:0/2467235160 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7d1c0a3d20 msgr2=0x7f7d1c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:40.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.281+0000 7f7d3874a640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7d1c0a3d20 0x7f7d1c0a61e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7d28033040 tx=0x7f7d28009910 comp rx=0 tx=0).stop 2024-04-23T18:39:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.281+0000 7f7d3874a640 1 -- 172.21.15.50:0/2467235160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d30101b40 msgr2=0x7f7d30103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:41.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.281+0000 7f7d3874a640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d30101b40 0x7f7d30103090 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000e9f0 tx=0x7f7d2000edf0 comp rx=0 tx=0).stop 2024-04-23T18:39:41.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.281+0000 7f7d3874a640 1 -- 172.21.15.50:0/2467235160 shutdown_connections 2024-04-23T18:39:41.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.281+0000 7f7d3874a640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7d1c0a3d20 0x7f7d1c0a61e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:41.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.281+0000 7f7d3874a640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d30107030 0x7f7d301035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:41.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.281+0000 7f7d3874a640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d30101b40 0x7f7d30103090 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:41.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.281+0000 7f7d3874a640 1 --2- 172.21.15.50:0/2467235160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d301011a0 0x7f7d30102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:41.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.281+0000 7f7d3874a640 1 -- 172.21.15.50:0/2467235160 >> 172.21.15.50:0/2467235160 conn(0x7f7d30076c50 msgr2=0x7f7d30100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:41.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.281+0000 7f7d3874a640 1 -- 172.21.15.50:0/2467235160 shutdown_connections 2024-04-23T18:39:41.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:39:36.281+0000 7f7d3874a640 1 -- 172.21.15.50:0/2467235160 wait complete. 2024-04-23T18:39:41.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:36 smithi050 ceph-mon[29187]: pgmap v671: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 3.3 MiB/s wr, 105 op/s 2024-04-23T18:39:41.283 INFO:tasks.workunit.client.0.smithi050.stdout: 10 170590 17.22 MB/sec execute 370 sec latency 35904.368 ms 2024-04-23T18:39:41.290 INFO:tasks.workunit.client.0.smithi050.stdout: 10 171176 17.26 MB/sec execute 371 sec latency 74.119 ms 2024-04-23T18:39:41.298 INFO:tasks.workunit.client.0.smithi050.stdout: 10 171656 17.30 MB/sec execute 372 sec latency 361.641 ms 2024-04-23T18:39:41.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:38 smithi050 ceph-mon[29187]: pgmap v672: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 3.6 MiB/s wr, 127 op/s 2024-04-23T18:39:41.315 INFO:tasks.workunit.client.0.smithi050.stdout: 10 172122 17.31 MB/sec execute 373 sec latency 89.722 ms 2024-04-23T18:39:41.325 INFO:tasks.workunit.client.0.smithi050.stdout: 10 172594 17.33 MB/sec execute 374 sec latency 102.358 ms 2024-04-23T18:39:41.342 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '71fed75b', seconds elapsed: 42.7 2024-04-23T18:39:41.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:40 smithi050 ceph-mon[29187]: pgmap v673: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-04-23T18:39:41.362 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:40 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:39:41.585 INFO:tasks.workunit.client.0.smithi050.stdout: 10 173160 17.39 MB/sec execute 375 sec latency 80.631 ms 2024-04-23T18:39:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.823+0000 7f12276cb640 1 -- 172.21.15.50:0/363914640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12201031e0 msgr2=0x7f12201035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:41.826 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.823+0000 7f12276cb640 1 --2- 172.21.15.50:0/363914640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12201031e0 0x7f12201035c0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f1218009900 tx=0x7f121802a510 comp rx=0 tx=0).stop 2024-04-23T18:39:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.823+0000 7f12276cb640 1 -- 172.21.15.50:0/363914640 shutdown_connections 2024-04-23T18:39:41.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.823+0000 7f12276cb640 1 --2- 172.21.15.50:0/363914640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f122010c520 0x7f122010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.823+0000 7f12276cb640 1 --2- 172.21.15.50:0/363914640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1220103b00 0x7f122010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:42.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.823+0000 7f12276cb640 1 --2- 172.21.15.50:0/363914640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12201031e0 0x7f12201035c0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:42.095 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.823+0000 7f12276cb640 1 -- 172.21.15.50:0/363914640 >> 172.21.15.50:0/363914640 conn(0x7f1220077fd0 msgr2=0x7f12200783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.823+0000 7f12276cb640 1 -- 172.21.15.50:0/363914640 shutdown_connections 2024-04-23T18:39:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.823+0000 7f12276cb640 1 -- 172.21.15.50:0/363914640 wait complete. 2024-04-23T18:39:42.160 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f12276cb640 1 Processor -- start 2024-04-23T18:39:42.174 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f12276cb640 1 -- start start 2024-04-23T18:39:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f12276cb640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12201031e0 0x7f12201986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f12276cb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1220103b00 0x7f1220198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f12276cb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f122010c520 0x7f122019cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f12276cb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f12201121b0 con 0x7f1220103b00 2024-04-23T18:39:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f12276cb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1220112030 con 0x7f122010c520 2024-04-23T18:39:42.459 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f12276cb640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1220112330 con 0x7f12201031e0 2024-04-23T18:39:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f1224c66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1220103b00 0x7f1220198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:42.508 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f1224c66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1220103b00 0x7f1220198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49858/0 (socket says 172.21.15.50:49858) 2024-04-23T18:39:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f1224c66640 1 -- 172.21.15.50:0/638831050 learned_addr learned my addr 172.21.15.50:0/638831050 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f1225c68640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f122010c520 0x7f122019cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:42.601 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f1225467640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12201031e0 0x7f12201986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f1224c66640 1 -- 172.21.15.50:0/638831050 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12201031e0 msgr2=0x7f12201986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f1224c66640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12201031e0 0x7f12201986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:42.687 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f1224c66640 1 -- 172.21.15.50:0/638831050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f122010c520 msgr2=0x7f122019cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:42.696 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.824+0000 7f1224c66640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f122010c520 0x7f122019cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.825+0000 7f1224c66640 1 -- 172.21.15.50:0/638831050 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122019d6a0 con 0x7f1220103b00 2024-04-23T18:39:42.714 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.825+0000 7f1224c66640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1220103b00 0x7f1220198bf0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f121000ea60 tx=0x7f121000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:42.723 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.825+0000 7f12167fc640 1 -- 172.21.15.50:0/638831050 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121000cde0 con 0x7f1220103b00 2024-04-23T18:39:42.729 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.825+0000 7f12167fc640 1 -- 172.21.15.50:0/638831050 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12100035f0 con 0x7f1220103b00 2024-04-23T18:39:42.736 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.825+0000 7f12276cb640 1 -- 172.21.15.50:0/638831050 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f122019d990 con 0x7f1220103b00 2024-04-23T18:39:42.744 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:42 smithi188 ceph-mon[32694]: pgmap v674: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 664 KiB/s wr, 62 op/s 2024-04-23T18:39:42.751 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:42 smithi177 ceph-mon[32308]: pgmap v674: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 664 KiB/s wr, 62 op/s 2024-04-23T18:39:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.825+0000 7f12276cb640 1 -- 172.21.15.50:0/638831050 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12201a5220 con 0x7f1220103b00 2024-04-23T18:39:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.826+0000 7f12167fc640 1 -- 172.21.15.50:0/638831050 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210010690 con 0x7f1220103b00 2024-04-23T18:39:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.827+0000 7f12167fc640 1 -- 172.21.15.50:0/638831050 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f12100037a0 con 0x7f1220103b00 2024-04-23T18:39:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.828+0000 7f122a3dc740 1 Processor -- start 2024-04-23T18:39:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.828+0000 7f122a3dc740 1 -- start start 2024-04-23T18:39:42.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.828+0000 7f122a3dc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa8a3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:42.967 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.828+0000 7f122a3dc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8ae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:42.975 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.828+0000 7f122a3dc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8ae800 0x563eaa8b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:42.983 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.828+0000 7f122a3dc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563eaa808970 con 0x563eaa8a3620 2024-04-23T18:39:42.990 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.828+0000 7f122a3dc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563eaa7a54a0 con 0x563eaa8a3f40 2024-04-23T18:39:42.997 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.828+0000 7f12167fc640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f120c0a8250 0x7f120c0aa710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.828+0000 7f122a3dc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563eaa804930 con 0x563eaa8ae800 2024-04-23T18:39:43.011 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1225c68640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa8a3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1225467640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8ae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1225c68640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa8a3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49864/0 (socket says 172.21.15.50:49864) 2024-04-23T18:39:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1225c68640 1 -- 172.21.15.50:0/2319187467 learned_addr learned my addr 172.21.15.50:0/2319187467 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1225c68640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f120c0a8250 0x7f120c0aa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f12167fc640 1 -- 172.21.15.50:0/638831050 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7f12100d0a00 con 0x7f1220103b00 2024-04-23T18:39:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1224c66640 1 --2- 172.21.15.50:0/2319187467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8ae800 0x563eaa8b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:43.115 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1225467640 1 -- 172.21.15.50:0/2319187467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8ae800 msgr2=0x563eaa8b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:43.161 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1225467640 1 --2- 172.21.15.50:0/2319187467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8ae800 0x563eaa8b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1225467640 1 -- 172.21.15.50:0/2319187467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 msgr2=0x563eaa8a3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1225467640 1 --2- 172.21.15.50:0/2319187467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa8a3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1225467640 1 -- 172.21.15.50:0/2319187467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563eaa51b210 con 0x563eaa8a3f40 2024-04-23T18:39:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1225c68640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f120c0a8250 0x7f120c0aa710 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f121c0099e0 tx=0x7f121c009c10 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:43.294 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f1225467640 1 --2- 172.21.15.50:0/2319187467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8ae2c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1218009230 tx=0x7f1218032b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58fbfcc32b307ce3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:43.300 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f120b7fe640 1 -- 172.21.15.50:0/2319187467 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218040070 con 0x563eaa8a3f40 2024-04-23T18:39:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.829+0000 7f120b7fe640 1 -- 172.21.15.50:0/2319187467 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1218004030 con 0x563eaa8a3f40 2024-04-23T18:39:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.830+0000 7f120b7fe640 1 -- 172.21.15.50:0/2319187467 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218035040 con 0x563eaa8a3f40 2024-04-23T18:39:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.830+0000 7f122a3dc740 1 -- 172.21.15.50:0/2319187467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 msgr2=0x563eaa8ae2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:43.330 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.830+0000 7f122a3dc740 1 --2- 172.21.15.50:0/2319187467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8ae2c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1218009230 tx=0x7f1218032b20 comp rx=0 tx=0).stop 2024-04-23T18:39:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.830+0000 7f122a3dc740 1 -- 172.21.15.50:0/2319187467 shutdown_connections 2024-04-23T18:39:43.345 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.830+0000 7f122a3dc740 1 --2- 172.21.15.50:0/2319187467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8ae800 0x563eaa8b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.830+0000 7f122a3dc740 1 --2- 172.21.15.50:0/2319187467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8ae2c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.830+0000 7f122a3dc740 1 --2- 172.21.15.50:0/2319187467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa8a3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.830+0000 7f122a3dc740 1 -- 172.21.15.50:0/2319187467 >> 172.21.15.50:0/2319187467 conn(0x563eaa89fb30 msgr2=0x563eaa89ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:43.474 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.830+0000 7f122a3dc740 1 -- 172.21.15.50:0/2319187467 shutdown_connections 2024-04-23T18:39:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.830+0000 7f122a3dc740 1 -- 172.21.15.50:0/2319187467 wait complete. 2024-04-23T18:39:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f122a3dc740 1 -- start start 2024-04-23T18:39:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f122a3dc740 1 Processor -- start 2024-04-23T18:39:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f122a3dc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa8ae130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f122a3dc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8ab420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:43.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f122a3dc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8ae800 0x563eaa8ab960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f122a3dc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x563eaa8b13d0 con 0x563eaa8a3620 2024-04-23T18:39:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f122a3dc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563eaa7a30b0 con 0x563eaa8a3f40 2024-04-23T18:39:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f122a3dc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x563eaa8b1550 con 0x563eaa8ae800 2024-04-23T18:39:43.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f122a3dc740 10 client.0 _ll_register_callbacks cb 0x563eaa790d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:39:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f1225467640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa8ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f1224c66640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f1225467640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa8ae130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49868/0 (socket says 172.21.15.50:49868) 2024-04-23T18:39:43.635 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f1225467640 1 -- 172.21.15.50:0/1229173411 learned_addr learned my addr 172.21.15.50:0/1229173411 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:43.713 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.831+0000 7f1225c68640 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8ae800 0x563eaa8ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f1224c66640 1 -- 172.21.15.50:0/1229173411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8ae800 msgr2=0x563eaa8ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:43.782 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f1224c66640 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8ae800 0x563eaa8ab960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f1224c66640 1 -- 172.21.15.50:0/1229173411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 msgr2=0x563eaa8ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f1224c66640 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa8ae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f1224c66640 1 -- 172.21.15.50:0/1229173411 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563eaa5e7620 con 0x563eaa8a3f40 2024-04-23T18:39:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f1225467640 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa8ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:39:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f1224c66640 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8ab420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f12100ce140 tx=0x7f121009c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6fc35a5b45ee0ff server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:43.892 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f1208ff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:39:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f1208ff9640 1 -- 172.21.15.50:0/1229173411 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12100aa3f0 con 0x563eaa8a3f40 2024-04-23T18:39:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f122a3dc740 1 -- 172.21.15.50:0/1229173411 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1218004030 con 0x563eaa8a3f40 2024-04-23T18:39:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f1208ff9640 1 -- 172.21.15.50:0/1229173411 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7f1210099000 con 0x563eaa8a3f40 2024-04-23T18:39:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f1208ff9640 1 -- 172.21.15.50:0/1229173411 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f12100a64c0 con 0x563eaa8a3f40 2024-04-23T18:39:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f122a3dc740 10 client.27578 fetch_fsmap learned FSMap version 63 2024-04-23T18:39:43.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.832+0000 7f122a3dc740 1 -- 172.21.15.50:0/1229173411 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x563eaa8bd340 con 0x563eaa8a3f40 2024-04-23T18:39:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.833+0000 7f1208ff9640 1 -- 172.21.15.50:0/1229173411 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1218004030 con 0x563eaa8a3f40 2024-04-23T18:39:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.833+0000 7f1208ff9640 1 -- 172.21.15.50:0/1229173411 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 63) ==== 8342+0+0 (secure 0 0 0) 0x7f12100aa080 con 0x563eaa8a3f40 2024-04-23T18:39:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.833+0000 7f122a3dc740 10 client.27578 fetch_fsmap finished waiting for FSMap version 63 2024-04-23T18:39:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.833+0000 7f122a3dc740 10 client.27578 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:39:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.833+0000 7f122a3dc740 20 client.27578 populate_metadata read hostname 'smithi050' 2024-04-23T18:39:44.085 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.833+0000 7f122a3dc740 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563eaa8be800 0x563eaa8c0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:44.094 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.833+0000 7f122a3dc740 4 client.27578 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:39:44.102 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.833+0000 7f122a3dc740 1 -- 172.21.15.50:0/1229173411 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563eaa816390 con 0x563eaa8be800 2024-04-23T18:39:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.833+0000 7f1225467640 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563eaa8be800 0x563eaa8c0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:44.165 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.833+0000 7f1225467640 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563eaa8be800 0x563eaa8c0c30 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=743a8c7c0ed1cf4f server_cookie=6e93bd6c90a50088 in_seq=0 out_seq=0 2024-04-23T18:39:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.833+0000 7f1208ff9640 10 client.27578 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f1208ff9640 1 -- 172.21.15.50:0/1229173411 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563eaa816390 con 0x563eaa8be800 2024-04-23T18:39:44.190 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f1208ff9640 10 client.27578 handle_command_reply: tid=0 2024-04-23T18:39:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f122a3dc740 1 client.27578 shutdown 2024-04-23T18:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f122a3dc740 1 -- 172.21.15.50:0/1229173411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 msgr2=0x563eaa8ab420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f122a3dc740 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8ab420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f12100ce140 tx=0x7f121009c850 comp rx=0 tx=0).stop 2024-04-23T18:39:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f122a3dc740 1 -- 172.21.15.50:0/1229173411 shutdown_connections 2024-04-23T18:39:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f122a3dc740 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563eaa8be800 0x563eaa8c0c30 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f122a3dc740 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8ae800 0x563eaa8ab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f122a3dc740 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8ab420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f1208ff9640 0 client.27578 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f122a3dc740 1 --2- 172.21.15.50:0/1229173411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa8ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f122a3dc740 1 -- 172.21.15.50:0/1229173411 >> 172.21.15.50:0/1229173411 conn(0x563eaa89fb30 msgr2=0x563eaa8a58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f122a3dc740 1 -- 172.21.15.50:0/1229173411 shutdown_connections 2024-04-23T18:39:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f122a3dc740 1 -- 172.21.15.50:0/1229173411 wait complete. 2024-04-23T18:39:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.834+0000 7f122a3dc740 20 client.27578 trim_cache size 0 max 16384 2024-04-23T18:39:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.851+0000 7f122a3dc740 1 Processor -- start 2024-04-23T18:39:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.851+0000 7f122a3dc740 1 -- start start 2024-04-23T18:39:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.851+0000 7f122a3dc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa89fb30 0x563eaa8ba290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.851+0000 7f122a3dc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa89eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f122a3dc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f122a3dc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f12201ae270 con 0x563eaa8a3620 2024-04-23T18:39:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f122a3dc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f12201abb10 con 0x563eaa8a3f40 2024-04-23T18:39:44.621 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f122a3dc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f12201a9980 con 0x563eaa89fb30 2024-04-23T18:39:44.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f1224c66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa89eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:44.635 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f1224c66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa89eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49880/0 (socket says 172.21.15.50:49880) 2024-04-23T18:39:44.642 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f1224c66640 1 -- 172.21.15.50:0/1543162565 learned_addr learned my addr 172.21.15.50:0/1543162565 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:44.648 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f1225467640 1 --2- 172.21.15.50:0/1543162565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa89fb30 0x563eaa8ba290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f1225c68640 1 --2- 172.21.15.50:0/1543162565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f1224c66640 1 -- 172.21.15.50:0/1543162565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa89fb30 msgr2=0x563eaa8ba290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f1224c66640 1 --2- 172.21.15.50:0/1543162565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa89fb30 0x563eaa8ba290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f1224c66640 1 -- 172.21.15.50:0/1543162565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 msgr2=0x563eaa8c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f1224c66640 1 --2- 172.21.15.50:0/1543162565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f1224c66640 1 -- 172.21.15.50:0/1543162565 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563eaa89f5f0 con 0x563eaa8a3620 2024-04-23T18:39:44.848 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f1225c68640 1 --2- 172.21.15.50:0/1543162565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:39:44.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f1224c66640 1 --2- 172.21.15.50:0/1543162565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa89eb30 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f1210009230 tx=0x7f1210003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd23a98c6c3aab5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:44.919 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f120affd640 1 -- 172.21.15.50:0/1543162565 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121009c2f0 con 0x563eaa8a3620 2024-04-23T18:39:44.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f120affd640 1 -- 172.21.15.50:0/1543162565 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12100a5600 con 0x563eaa8a3620 2024-04-23T18:39:44.932 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.852+0000 7f120affd640 1 -- 172.21.15.50:0/1543162565 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12100a0d40 con 0x563eaa8a3620 2024-04-23T18:39:44.939 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.853+0000 7f122a3dc740 1 -- 172.21.15.50:0/1543162565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 msgr2=0x563eaa89eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:44.945 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.853+0000 7f122a3dc740 1 --2- 172.21.15.50:0/1543162565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa89eb30 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f1210009230 tx=0x7f1210003e90 comp rx=0 tx=0).stop 2024-04-23T18:39:44.951 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.853+0000 7f122a3dc740 1 -- 172.21.15.50:0/1543162565 shutdown_connections 2024-04-23T18:39:44.958 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.853+0000 7f122a3dc740 1 --2- 172.21.15.50:0/1543162565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.853+0000 7f122a3dc740 1 --2- 172.21.15.50:0/1543162565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa8a3620 0x563eaa89eb30 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:44.976 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.853+0000 7f122a3dc740 1 --2- 172.21.15.50:0/1543162565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa89fb30 0x563eaa8ba290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:45.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.853+0000 7f122a3dc740 1 -- 172.21.15.50:0/1543162565 >> 172.21.15.50:0/1543162565 conn(0x563eaa8abac0 msgr2=0x563eaa89e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.853+0000 7f122a3dc740 1 -- 172.21.15.50:0/1543162565 shutdown_connections 2024-04-23T18:39:45.034 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.853+0000 7f122a3dc740 1 -- 172.21.15.50:0/1543162565 wait complete. 2024-04-23T18:39:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f122a3dc740 1 -- start start 2024-04-23T18:39:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f122a3dc740 1 Processor -- start 2024-04-23T18:39:45.125 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f122a3dc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa89fb30 0x563eaa8b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f122a3dc740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8a3620 0x563eaa8b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:45.229 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f122a3dc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f122a3dc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f12201a9fd0 con 0x563eaa89fb30 2024-04-23T18:39:45.245 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f122a3dc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f12201ab440 con 0x563eaa8a3f40 2024-04-23T18:39:45.253 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f122a3dc740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f12201aa6a0 con 0x563eaa8a3620 2024-04-23T18:39:45.260 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f122a3dc740 10 client.0 _ll_register_callbacks cb 0x7f1210098d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:39:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f1225467640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa89fb30 0x563eaa8b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f1224c66640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8a3620 0x563eaa8b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f1225467640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa89fb30 0x563eaa8b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49896/0 (socket says 172.21.15.50:49896) 2024-04-23T18:39:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f1225467640 1 -- 172.21.15.50:0/3468032041 learned_addr learned my addr 172.21.15.50:0/3468032041 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:39:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.854+0000 7f1225c68640 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f1225467640 1 -- 172.21.15.50:0/3468032041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8a3620 msgr2=0x563eaa8b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:45.365 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f1225467640 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8a3620 0x563eaa8b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f1225467640 1 -- 172.21.15.50:0/3468032041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 msgr2=0x563eaa8b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:39:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f1225467640 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:45.477 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f1225467640 1 -- 172.21.15.50:0/3468032041 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563eaa923ec0 con 0x563eaa89fb30 2024-04-23T18:39:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f1224c66640 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8a3620 0x563eaa8b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:39:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f1225467640 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa89fb30 0x563eaa8b42d0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f1218000c00 tx=0x7f1218005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9525a70d616096ef server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:39:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f1208ff9640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:39:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f1208ff9640 1 -- 172.21.15.50:0/3468032041 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218056060 con 0x563eaa89fb30 2024-04-23T18:39:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f1208ff9640 1 -- 172.21.15.50:0/3468032041 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7f1218004040 con 0x563eaa89fb30 2024-04-23T18:39:45.589 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f122a3dc740 1 -- 172.21.15.50:0/3468032041 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12100a5600 con 0x563eaa89fb30 2024-04-23T18:39:45.598 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f1208ff9640 1 -- 172.21.15.50:0/3468032041 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f1218040040 con 0x563eaa89fb30 2024-04-23T18:39:45.606 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f122a3dc740 10 client.18108 fetch_fsmap learned FSMap version 63 2024-04-23T18:39:45.615 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f122a3dc740 1 -- 172.21.15.50:0/3468032041 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x563eaa925210 con 0x563eaa89fb30 2024-04-23T18:39:45.623 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.855+0000 7f1208ff9640 1 -- 172.21.15.50:0/3468032041 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12100a5600 con 0x563eaa89fb30 2024-04-23T18:39:45.633 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.856+0000 7f1208ff9640 1 -- 172.21.15.50:0/3468032041 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 63) ==== 8342+0+0 (secure 0 0 0) 0x7f1218005040 con 0x563eaa89fb30 2024-04-23T18:39:45.675 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.856+0000 7f122a3dc740 10 client.18108 fetch_fsmap finished waiting for FSMap version 63 2024-04-23T18:39:45.682 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.856+0000 7f122a3dc740 10 client.18108 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:39:45.689 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.856+0000 7f122a3dc740 20 client.18108 populate_metadata read hostname 'smithi050' 2024-04-23T18:39:45.707 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.856+0000 7f122a3dc740 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563eaa8ae800 0x563eaa927620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:39:45.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.856+0000 7f122a3dc740 4 client.18108 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:39:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.856+0000 7f122a3dc740 1 -- 172.21.15.50:0/3468032041 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563eaa810200 con 0x563eaa8ae800 2024-04-23T18:39:45.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.856+0000 7f1224c66640 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563eaa8ae800 0x563eaa927620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:39:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.856+0000 7f1224c66640 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563eaa8ae800 0x563eaa927620 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42d09b1a8facb04b server_cookie=dc68a1aec0febc29 in_seq=0 out_seq=0 2024-04-23T18:39:45.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:41.856+0000 7f1208ff9640 10 client.18108 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:45.853 INFO:tasks.workunit.client.0.smithi050.stdout: 10 173511 17.38 MB/sec execute 376 sec latency 853.001 ms 2024-04-23T18:39:45.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:42 smithi050 ceph-mon[29187]: pgmap v674: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 664 KiB/s wr, 62 op/s 2024-04-23T18:39:45.871 INFO:tasks.workunit.client.0.smithi050.stdout: 10 173996 17.40 MB/sec execute 377 sec latency 1853.030 ms 2024-04-23T18:39:45.880 INFO:tasks.workunit.client.0.smithi050.stdout: 10 174492 17.43 MB/sec execute 378 sec latency 2853.095 ms 2024-04-23T18:39:45.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:44 smithi050 ceph-mon[29187]: pgmap v675: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.8 MiB/s wr, 249 op/s 2024-04-23T18:39:45.896 INFO:tasks.workunit.client.0.smithi050.stdout: 10 175176 17.48 MB/sec execute 379 sec latency 3853.137 ms 2024-04-23T18:39:45.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:45 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:39:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:44 smithi188 ceph-mon[32694]: pgmap v675: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.8 MiB/s wr, 249 op/s 2024-04-23T18:39:45.958 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:45 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:39:45.966 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:44 smithi177 ceph-mon[32308]: pgmap v675: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.8 MiB/s wr, 249 op/s 2024-04-23T18:39:45.974 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:45 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:39:46.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:46 smithi177 ceph-mon[32308]: pgmap v676: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.5 MiB/s wr, 217 op/s 2024-04-23T18:39:46.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:46 smithi188 ceph-mon[32694]: pgmap v676: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.5 MiB/s wr, 217 op/s 2024-04-23T18:39:46.640 INFO:tasks.workunit.client.0.smithi050.stdout: 10 175606 17.51 MB/sec execute 380 sec latency 4853.180 ms 2024-04-23T18:39:46.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:46 smithi050 ceph-mon[29187]: pgmap v676: 97 pgs: 97 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.5 MiB/s wr, 217 op/s 2024-04-23T18:39:47.585 INFO:tasks.workunit.client.0.smithi050.stdout: 10 175970 17.53 MB/sec execute 381 sec latency 5853.209 ms 2024-04-23T18:39:48.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:48 smithi177 ceph-mon[32308]: pgmap v677: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.1 MiB/s wr, 398 op/s 2024-04-23T18:39:48.612 INFO:tasks.workunit.client.0.smithi050.stdout: 10 176378 17.52 MB/sec execute 382 sec latency 6853.239 ms 2024-04-23T18:39:48.619 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:48 smithi188 ceph-mon[32694]: pgmap v677: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.1 MiB/s wr, 398 op/s 2024-04-23T18:39:48.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:48 smithi050 ceph-mon[29187]: pgmap v677: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.1 MiB/s wr, 398 op/s 2024-04-23T18:39:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.071+0000 7f1208ff9640 1 -- 172.21.15.50:0/3468032041 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563eaa810200 con 0x563eaa8ae800 2024-04-23T18:39:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.071+0000 7f1208ff9640 10 client.18108 handle_command_reply: tid=0 2024-04-23T18:39:49.171 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.071+0000 7f122a3dc740 1 client.18108 shutdown 2024-04-23T18:39:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.071+0000 7f122a3dc740 1 -- 172.21.15.50:0/3468032041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa89fb30 msgr2=0x563eaa8b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.071+0000 7f122a3dc740 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa89fb30 0x563eaa8b42d0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f1218000c00 tx=0x7f1218005b50 comp rx=0 tx=0).stop 2024-04-23T18:39:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.072+0000 7f122a3dc740 1 -- 172.21.15.50:0/3468032041 shutdown_connections 2024-04-23T18:39:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.072+0000 7f122a3dc740 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x563eaa8ae800 0x563eaa927620 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:49.211 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.072+0000 7f122a3dc740 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eaa8a3f40 0x563eaa8b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:49.219 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.072+0000 7f122a3dc740 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563eaa8a3620 0x563eaa8b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.072+0000 7f122a3dc740 1 --2- 172.21.15.50:0/3468032041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x563eaa89fb30 0x563eaa8b42d0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.072+0000 7f122a3dc740 1 -- 172.21.15.50:0/3468032041 >> 172.21.15.50:0/3468032041 conn(0x563eaa8abac0 msgr2=0x563eaa89ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.072+0000 7f1208ff9640 0 client.18108 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:39:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.072+0000 7f122a3dc740 1 -- 172.21.15.50:0/3468032041 shutdown_connections 2024-04-23T18:39:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.072+0000 7f122a3dc740 1 -- 172.21.15.50:0/3468032041 wait complete. 2024-04-23T18:39:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.072+0000 7f122a3dc740 20 client.18108 trim_cache size 0 max 16384 2024-04-23T18:39:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:39:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:39:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:39:49.539 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:39:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.073+0000 7f12276cb640 1 -- 172.21.15.50:0/638831050 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f120c0a8250 msgr2=0x7f120c0aa710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:49.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.073+0000 7f12276cb640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f120c0a8250 0x7f120c0aa710 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f121c0099e0 tx=0x7f121c009c10 comp rx=0 tx=0).stop 2024-04-23T18:39:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.073+0000 7f12276cb640 1 -- 172.21.15.50:0/638831050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1220103b00 msgr2=0x7f1220198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:39:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.073+0000 7f12276cb640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1220103b00 0x7f1220198bf0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f121000ea60 tx=0x7f121000ef20 comp rx=0 tx=0).stop 2024-04-23T18:39:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.073+0000 7f12276cb640 1 -- 172.21.15.50:0/638831050 shutdown_connections 2024-04-23T18:39:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.073+0000 7f12276cb640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f120c0a8250 0x7f120c0aa710 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:49.712 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.073+0000 7f12276cb640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f122010c520 0x7f122019cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.073+0000 7f12276cb640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1220103b00 0x7f1220198bf0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.073+0000 7f12276cb640 1 --2- 172.21.15.50:0/638831050 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12201031e0 0x7f12201986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:39:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.073+0000 7f12276cb640 1 -- 172.21.15.50:0/638831050 >> 172.21.15.50:0/638831050 conn(0x7f1220077fd0 msgr2=0x7f12200fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:39:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.073+0000 7f12276cb640 1 -- 172.21.15.50:0/638831050 shutdown_connections 2024-04-23T18:39:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:39:49.073+0000 7f12276cb640 1 -- 172.21.15.50:0/638831050 wait complete. 2024-04-23T18:39:49.920 INFO:tasks.workunit.client.0.smithi050.stdout: 10 176836 17.54 MB/sec execute 383 sec latency 7349.271 ms 2024-04-23T18:39:50.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:50 smithi177 ceph-mon[32308]: pgmap v678: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.2 MiB/s wr, 414 op/s 2024-04-23T18:39:50.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:50 smithi188 ceph-mon[32694]: pgmap v678: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.2 MiB/s wr, 414 op/s 2024-04-23T18:39:50.610 INFO:tasks.workunit.client.0.smithi050.stdout: 10 177319 17.55 MB/sec execute 384 sec latency 7152.295 ms 2024-04-23T18:39:50.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:50 smithi050 ceph-mon[29187]: pgmap v678: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.2 MiB/s wr, 414 op/s 2024-04-23T18:39:51.585 INFO:tasks.workunit.client.0.smithi050.stdout: 10 177669 17.55 MB/sec execute 385 sec latency 4048.695 ms 2024-04-23T18:39:52.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:52 smithi177 ceph-mon[32308]: pgmap v679: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.0 MiB/s wr, 406 op/s 2024-04-23T18:39:52.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:52 smithi188 ceph-mon[32694]: pgmap v679: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.0 MiB/s wr, 406 op/s 2024-04-23T18:39:52.619 INFO:tasks.workunit.client.0.smithi050.stdout: 10 178223 17.60 MB/sec execute 386 sec latency 87.251 ms 2024-04-23T18:39:52.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:52 smithi050 ceph-mon[29187]: pgmap v679: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.0 MiB/s wr, 406 op/s 2024-04-23T18:39:53.585 INFO:tasks.workunit.client.0.smithi050.stdout: 10 178649 17.60 MB/sec execute 387 sec latency 159.724 ms 2024-04-23T18:39:54.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:54 smithi177 ceph-mon[32308]: pgmap v680: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 9.6 MiB/s wr, 578 op/s 2024-04-23T18:39:54.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:54 smithi188 ceph-mon[32694]: pgmap v680: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 9.6 MiB/s wr, 578 op/s 2024-04-23T18:39:54.585 INFO:tasks.workunit.client.0.smithi050.stdout: 10 179129 17.63 MB/sec execute 388 sec latency 123.243 ms 2024-04-23T18:39:54.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:54 smithi050 ceph-mon[29187]: pgmap v680: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 9.6 MiB/s wr, 578 op/s 2024-04-23T18:39:55.585 INFO:tasks.workunit.client.0.smithi050.stdout: 10 179505 17.62 MB/sec execute 389 sec latency 343.160 ms 2024-04-23T18:39:56.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:56 smithi177 ceph-mon[32308]: pgmap v681: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 391 op/s 2024-04-23T18:39:56.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:56 smithi188 ceph-mon[32694]: pgmap v681: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 391 op/s 2024-04-23T18:39:56.587 INFO:tasks.workunit.client.0.smithi050.stdout: 10 180028 17.66 MB/sec execute 390 sec latency 93.634 ms 2024-04-23T18:39:56.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:56 smithi050 ceph-mon[29187]: pgmap v681: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 391 op/s 2024-04-23T18:39:57.586 INFO:tasks.workunit.client.0.smithi050.stdout: 10 180513 17.68 MB/sec execute 391 sec latency 115.402 ms 2024-04-23T18:39:58.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:39:58 smithi177 ceph-mon[32308]: pgmap v682: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.4 MiB/s wr, 490 op/s 2024-04-23T18:39:58.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:39:58 smithi188 ceph-mon[32694]: pgmap v682: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.4 MiB/s wr, 490 op/s 2024-04-23T18:39:58.590 INFO:tasks.workunit.client.0.smithi050.stdout: 10 181065 17.68 MB/sec execute 392 sec latency 86.357 ms 2024-04-23T18:39:58.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:39:58 smithi050 ceph-mon[29187]: pgmap v682: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 8.4 MiB/s wr, 490 op/s 2024-04-23T18:39:59.586 INFO:tasks.workunit.client.0.smithi050.stdout: 10 181649 17.74 MB/sec execute 393 sec latency 82.370 ms 2024-04-23T18:40:00.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:00 smithi177 ceph-mon[32308]: pgmap v683: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.0 MiB/s wr, 368 op/s 2024-04-23T18:40:00.566 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:00 smithi177 ceph-mon[32308]: overall HEALTH_OK 2024-04-23T18:40:00.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:00 smithi188 ceph-mon[32694]: pgmap v683: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.0 MiB/s wr, 368 op/s 2024-04-23T18:40:00.586 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:00 smithi188 ceph-mon[32694]: overall HEALTH_OK 2024-04-23T18:40:00.662 INFO:tasks.workunit.client.0.smithi050.stdout: 10 182192 17.79 MB/sec execute 394 sec latency 90.770 ms 2024-04-23T18:40:00.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:00 smithi050 ceph-mon[29187]: pgmap v683: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.0 MiB/s wr, 368 op/s 2024-04-23T18:40:00.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:00 smithi050 ceph-mon[29187]: overall HEALTH_OK 2024-04-23T18:40:01.586 INFO:tasks.workunit.client.0.smithi050.stdout: 10 182347 17.77 MB/sec execute 395 sec latency 736.875 ms 2024-04-23T18:40:02.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.175+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/2269436792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba9c107290 msgr2=0x7fba9c10ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:02.185 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.175+0000 7fbaa3d6f640 1 --2- 172.21.15.50:0/2269436792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba9c107290 0x7fba9c10ee60 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fba94009ca0 tx=0x7fba9402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:40:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.175+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/2269436792 shutdown_connections 2024-04-23T18:40:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.175+0000 7fbaa3d6f640 1 --2- 172.21.15.50:0/2269436792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba9c107290 0x7fba9c10ee60 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:02.211 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.175+0000 7fbaa3d6f640 1 --2- 172.21.15.50:0/2269436792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba9c102b70 0x7fba9c106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.175+0000 7fbaa3d6f640 1 --2- 172.21.15.50:0/2269436792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba9c1021c0 0x7fba9c1025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:02.228 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.175+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/2269436792 >> 172.21.15.50:0/2269436792 conn(0x7fba9c0fda20 msgr2=0x7fba9c0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.175+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/2269436792 shutdown_connections 2024-04-23T18:40:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.175+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/2269436792 wait complete. 2024-04-23T18:40:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.176+0000 7fbaa3d6f640 1 Processor -- start 2024-04-23T18:40:02.341 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.176+0000 7fbaa3d6f640 1 -- start start 2024-04-23T18:40:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.176+0000 7fbaa3d6f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba9c1021c0 0x7fba9c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:02.396 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.176+0000 7fbaa3d6f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba9c107290 0x7fba9c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.176+0000 7fbaa3d6f640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba9c074cd0 0x7fba9c1a6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.176+0000 7fbaa3d6f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c1121e0 con 0x7fba9c1021c0 2024-04-23T18:40:02.465 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.176+0000 7fbaa3d6f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c112060 con 0x7fba9c107290 2024-04-23T18:40:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.176+0000 7fbaa3d6f640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c112360 con 0x7fba9c074cd0 2024-04-23T18:40:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.176+0000 7fbaa1b0b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba9c1021c0 0x7fba9c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.176+0000 7fbaa1b0b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba9c1021c0 0x7fba9c074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36754/0 (socket says 172.21.15.50:36754) 2024-04-23T18:40:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.176+0000 7fbaa1b0b640 1 -- 172.21.15.50:0/3955553427 learned_addr learned my addr 172.21.15.50:0/3955553427 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.176+0000 7fbaa230c640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba9c074cd0 0x7fba9c1a6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:02.653 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fbaa130a640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba9c107290 0x7fba9c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fbaa1b0b640 1 -- 172.21.15.50:0/3955553427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba9c074cd0 msgr2=0x7fba9c1a6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:02.669 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba9c074cd0 0x7fba9c1a6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:02.767 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fbaa1b0b640 1 -- 172.21.15.50:0/3955553427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba9c107290 msgr2=0x7fba9c074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:02.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba9c107290 0x7fba9c074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fbaa1b0b640 1 -- 172.21.15.50:0/3955553427 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba9c1a6f30 con 0x7fba9c1021c0 2024-04-23T18:40:02.813 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fbaa130a640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba9c107290 0x7fba9c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:40:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba9c1021c0 0x7fba9c074250 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fba9800d950 tx=0x7fba9800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:02.829 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fba92ffd640 1 -- 172.21.15.50:0/3955553427 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba98017030 con 0x7fba9c1021c0 2024-04-23T18:40:02.836 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fba92ffd640 1 -- 172.21.15.50:0/3955553427 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba98004020 con 0x7fba9c1021c0 2024-04-23T18:40:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/3955553427 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba9c1a7220 con 0x7fba9c1021c0 2024-04-23T18:40:02.852 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fba92ffd640 1 -- 172.21.15.50:0/3955553427 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba980047a0 con 0x7fba9c1021c0 2024-04-23T18:40:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.177+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/3955553427 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba9c1a7770 con 0x7fba9c1021c0 2024-04-23T18:40:02.937 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:02 smithi177 ceph-mon[32308]: pgmap v684: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.6 MiB/s wr, 330 op/s 2024-04-23T18:40:02.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:02 smithi188 ceph-mon[32694]: pgmap v684: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.6 MiB/s wr, 330 op/s 2024-04-23T18:40:03.045 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.180+0000 7fba92ffd640 1 -- 172.21.15.50:0/3955553427 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fba98009a30 con 0x7fba9c1021c0 2024-04-23T18:40:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.180+0000 7fbaa6b67740 1 Processor -- start 2024-04-23T18:40:03.061 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa6b67740 1 -- start start 2024-04-23T18:40:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa6b67740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860620 0x55b994860a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa6b67740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b99486b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa6b67740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b99486dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa6b67740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b9947c5970 con 0x55b994860f40 2024-04-23T18:40:03.196 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa6b67740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b9947624a0 con 0x55b99486b800 2024-04-23T18:40:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa6b67740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b9947c1930 con 0x55b994860620 2024-04-23T18:40:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa1b0b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b99486b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa1b0b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b99486b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36756/0 (socket says 172.21.15.50:36756) 2024-04-23T18:40:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa1b0b640 1 -- 172.21.15.50:0/3560856767 learned_addr learned my addr 172.21.15.50:0/3560856767 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:03.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa230c640 1 --2- 172.21.15.50:0/3560856767 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860620 0x55b994860a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:03.238 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fba92ffd640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fba880a3df0 0x7fba880a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa130a640 1 --2- 172.21.15.50:0/3560856767 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b99486dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.181+0000 7fbaa230c640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fba880a3df0 0x7fba880a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:03.383 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.182+0000 7fba92ffd640 1 -- 172.21.15.50:0/3955553427 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7fba980d0200 con 0x7fba9c1021c0 2024-04-23T18:40:03.408 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.182+0000 7fbaa1b0b640 1 -- 172.21.15.50:0/3560856767 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860620 msgr2=0x55b994860a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:03.415 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.182+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/3560856767 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860620 0x55b994860a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.182+0000 7fbaa1b0b640 1 -- 172.21.15.50:0/3560856767 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 msgr2=0x55b99486dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:03.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.182+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/3560856767 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b99486dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:03.436 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.182+0000 7fbaa1b0b640 1 -- 172.21.15.50:0/3560856767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9944d8210 con 0x55b994860f40 2024-04-23T18:40:03.486 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.182+0000 7fbaa230c640 1 --2- 172.21.15.50:0/3560856767 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860620 0x55b994860a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:40:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.182+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/3560856767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b99486b2c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fba9809c120 tx=0x7fba98004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694dd89fc65bf0dd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:03.504 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.182+0000 7fbaa230c640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fba880a3df0 0x7fba880a62b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fba9402f500 tx=0x7fba9402a8c0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.182+0000 7fba87fff640 1 -- 172.21.15.50:0/3560856767 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba980056a0 con 0x55b994860f40 2024-04-23T18:40:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.182+0000 7fba87fff640 1 -- 172.21.15.50:0/3560856767 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba980d3960 con 0x55b994860f40 2024-04-23T18:40:03.529 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.182+0000 7fba87fff640 1 -- 172.21.15.50:0/3560856767 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba980a4e00 con 0x55b994860f40 2024-04-23T18:40:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.183+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3560856767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 msgr2=0x55b99486b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.183+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3560856767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b99486b2c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fba9809c120 tx=0x7fba98004590 comp rx=0 tx=0).stop 2024-04-23T18:40:03.556 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.183+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3560856767 shutdown_connections 2024-04-23T18:40:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.183+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3560856767 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b99486dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.183+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3560856767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b99486b2c0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.183+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3560856767 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860620 0x55b994860a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:03.668 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.183+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3560856767 >> 172.21.15.50:0/3560856767 conn(0x55b99485cb30 msgr2=0x55b99485cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.183+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3560856767 shutdown_connections 2024-04-23T18:40:03.686 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.183+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3560856767 wait complete. 2024-04-23T18:40:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.183+0000 7fbaa6b67740 1 -- start start 2024-04-23T18:40:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.183+0000 7fbaa6b67740 1 Processor -- start 2024-04-23T18:40:03.735 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa6b67740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b994860620 0x55b99486b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:03.744 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa6b67740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b994868550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:03.752 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa6b67740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b99486b800 0x55b994868c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa6b67740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55b99486e3d0 con 0x55b994860f40 2024-04-23T18:40:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa6b67740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b9947600b0 con 0x55b994860620 2024-04-23T18:40:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa6b67740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55b99486e550 con 0x55b99486b800 2024-04-23T18:40:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa6b67740 10 client.0 _ll_register_callbacks cb 0x55b99474dd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:40:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa130a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b994868550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:03.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa130a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b994868550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36766/0 (socket says 172.21.15.50:36766) 2024-04-23T18:40:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa130a640 1 -- 172.21.15.50:0/3644232675 learned_addr learned my addr 172.21.15.50:0/3644232675 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:03.879 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b994860620 0x55b99486b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:03.924 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa230c640 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b99486b800 0x55b994868c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa130a640 1 -- 172.21.15.50:0/3644232675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b99486b800 msgr2=0x55b994868c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:03.944 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa130a640 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b99486b800 0x55b994868c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa130a640 1 -- 172.21.15.50:0/3644232675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b994860620 msgr2=0x55b99486b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:03.961 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa130a640 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b994860620 0x55b99486b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:03.970 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa130a640 1 -- 172.21.15.50:0/3644232675 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9945a4620 con 0x55b994860f40 2024-04-23T18:40:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.184+0000 7fbaa130a640 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b994868550 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fba8c013710 tx=0x7fba8c013b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551ce539e3a84eab server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fba857fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:40:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fba857fa640 1 -- 172.21.15.50:0/3644232675 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8c011e70 con 0x55b994860f40 2024-04-23T18:40:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fba857fa640 1 -- 172.21.15.50:0/3644232675 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7fba8c004020 con 0x55b994860f40 2024-04-23T18:40:04.068 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3644232675 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b9947d3390 con 0x55b994860f40 2024-04-23T18:40:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fba857fa640 1 -- 172.21.15.50:0/3644232675 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fba8c004220 con 0x55b994860f40 2024-04-23T18:40:04.084 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fbaa6b67740 10 client.18120 fetch_fsmap learned FSMap version 64 2024-04-23T18:40:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3644232675 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55b99487a460 con 0x55b994860f40 2024-04-23T18:40:04.101 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fba857fa640 1 -- 172.21.15.50:0/3644232675 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b9947d3390 con 0x55b994860f40 2024-04-23T18:40:04.119 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fba857fa640 1 -- 172.21.15.50:0/3644232675 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7fba8c01a030 con 0x55b994860f40 2024-04-23T18:40:04.190 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fbaa6b67740 10 client.18120 fetch_fsmap finished waiting for FSMap version 64 2024-04-23T18:40:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fbaa6b67740 10 client.18120 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:40:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fbaa6b67740 20 client.18120 populate_metadata read hostname 'smithi050' 2024-04-23T18:40:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b99487b990 0x55b99487dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fbaa6b67740 4 client.18120 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:40:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.185+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3644232675 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9947cd200 con 0x55b99487b990 2024-04-23T18:40:04.295 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.186+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b99487b990 0x55b99487dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.186+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b99487b990 0x55b99487dde0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7225dd935f564478 server_cookie=719afb58a56caddb in_seq=0 out_seq=0 2024-04-23T18:40:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.186+0000 7fba857fa640 10 client.18120 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:40:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fba857fa640 1 -- 172.21.15.50:0/3644232675 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b9947cd200 con 0x55b99487b990 2024-04-23T18:40:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fba857fa640 10 client.18120 handle_command_reply: tid=0 2024-04-23T18:40:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fbaa6b67740 1 client.18120 shutdown 2024-04-23T18:40:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3644232675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 msgr2=0x55b994868550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:04.480 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b994868550 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fba8c013710 tx=0x7fba8c013b10 comp rx=0 tx=0).stop 2024-04-23T18:40:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3644232675 shutdown_connections 2024-04-23T18:40:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b99487b990 0x55b99487dde0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:04.504 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b99486b800 0x55b994868c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b994868550 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3644232675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b994860620 0x55b99486b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fba857fa640 0 client.18120 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:40:04.635 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3644232675 >> 172.21.15.50:0/3644232675 conn(0x55b99485cb30 msgr2=0x55b994862860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3644232675 shutdown_connections 2024-04-23T18:40:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3644232675 wait complete. 2024-04-23T18:40:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.187+0000 7fbaa6b67740 20 client.18120 trim_cache size 0 max 16384 2024-04-23T18:40:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.202+0000 7fbaa6b67740 1 Processor -- start 2024-04-23T18:40:04.726 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.202+0000 7fbaa6b67740 1 -- start start 2024-04-23T18:40:04.734 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.202+0000 7fbaa6b67740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860620 0x55b994877290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:04.742 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.202+0000 7fbaa6b67740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860f40 0x55b99487dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:04.750 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.202+0000 7fbaa6b67740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b9948676b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:04.758 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.202+0000 7fbaa6b67740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c1b27a0 con 0x55b994860620 2024-04-23T18:40:04.769 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.202+0000 7fbaa6b67740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c1b0010 con 0x55b99486b800 2024-04-23T18:40:04.778 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.202+0000 7fbaa6b67740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c1ade80 con 0x55b994860f40 2024-04-23T18:40:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.202+0000 7fbaa1b0b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860620 0x55b994877290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fbaa1b0b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860620 0x55b994877290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36776/0 (socket says 172.21.15.50:36776) 2024-04-23T18:40:04.862 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fbaa1b0b640 1 -- 172.21.15.50:0/3129651741 learned_addr learned my addr 172.21.15.50:0/3129651741 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fbaa230c640 1 --2- 172.21.15.50:0/3129651741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b9948676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fbaa130a640 1 --2- 172.21.15.50:0/3129651741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860f40 0x55b99487dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fbaa1b0b640 1 -- 172.21.15.50:0/3129651741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860f40 msgr2=0x55b99487dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/3129651741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860f40 0x55b99487dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fbaa1b0b640 1 -- 172.21.15.50:0/3129651741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 msgr2=0x55b9948676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/3129651741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b9948676b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:05.121 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fbaa1b0b640 1 -- 172.21.15.50:0/3129651741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b994857090 con 0x55b994860620 2024-04-23T18:40:05.128 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fbaa230c640 1 --2- 172.21.15.50:0/3129651741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b9948676b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:40:05.136 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/3129651741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860620 0x55b994877290 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fba980cd9f0 tx=0x7fba98098560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff07d25538e864f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fba877fe640 1 -- 172.21.15.50:0/3129651741 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba980056a0 con 0x55b994860620 2024-04-23T18:40:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fba877fe640 1 -- 172.21.15.50:0/3129651741 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba9809c190 con 0x55b994860620 2024-04-23T18:40:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.203+0000 7fba877fe640 1 -- 172.21.15.50:0/3129651741 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba980041d0 con 0x55b994860620 2024-04-23T18:40:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.204+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3129651741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860620 msgr2=0x55b994877290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.204+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3129651741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860620 0x55b994877290 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fba980cd9f0 tx=0x7fba98098560 comp rx=0 tx=0).stop 2024-04-23T18:40:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.204+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3129651741 shutdown_connections 2024-04-23T18:40:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.204+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3129651741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b9948676b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:05.316 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.204+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3129651741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860f40 0x55b99487dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:05.366 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.204+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/3129651741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860620 0x55b994877290 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:05.386 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.204+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3129651741 >> 172.21.15.50:0/3129651741 conn(0x55b99485cb30 msgr2=0x55b99485b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:05.433 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.204+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3129651741 shutdown_connections 2024-04-23T18:40:05.473 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.204+0000 7fbaa6b67740 1 -- 172.21.15.50:0/3129651741 wait complete. 2024-04-23T18:40:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.204+0000 7fbaa6b67740 1 -- start start 2024-04-23T18:40:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.204+0000 7fbaa6b67740 1 Processor -- start 2024-04-23T18:40:05.528 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa6b67740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860620 0x55b9948da3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa6b67740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b9948da8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa6b67740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b994870b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa6b67740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c1ae4d0 con 0x55b994860f40 2024-04-23T18:40:05.560 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa6b67740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c1af940 con 0x55b99486b800 2024-04-23T18:40:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa6b67740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c1aeba0 con 0x55b994860620 2024-04-23T18:40:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa6b67740 10 client.0 _ll_register_callbacks cb 0x7fba8c009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:40:05.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa130a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b9948da8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa130a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b9948da8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36782/0 (socket says 172.21.15.50:36782) 2024-04-23T18:40:05.648 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa130a640 1 -- 172.21.15.50:0/1728494851 learned_addr learned my addr 172.21.15.50:0/1728494851 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:05.690 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860620 0x55b9948da3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa230c640 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b994870b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:05.808 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa130a640 1 -- 172.21.15.50:0/1728494851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860620 msgr2=0x55b9948da3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa130a640 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860620 0x55b9948da3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa130a640 1 -- 172.21.15.50:0/1728494851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 msgr2=0x55b994870b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa130a640 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b994870b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa130a640 1 -- 172.21.15.50:0/1728494851 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b994871250 con 0x55b994860f40 2024-04-23T18:40:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860620 0x55b9948da3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:40:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa130a640 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b9948da8e0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x55b994867db0 tx=0x7fba8c0045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd30fd58c1f3f98 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fba857fa640 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2024-04-23T18:40:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fba857fa640 1 -- 172.21.15.50:0/1728494851 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8c004f50 con 0x55b994860f40 2024-04-23T18:40:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fba857fa640 1 -- 172.21.15.50:0/1728494851 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7fba8c0050f0 con 0x55b994860f40 2024-04-23T18:40:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fbaa6b67740 1 -- 172.21.15.50:0/1728494851 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b99485c980 con 0x55b994860f40 2024-04-23T18:40:05.946 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.205+0000 7fba857fa640 1 -- 172.21.15.50:0/1728494851 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fba8c03a050 con 0x55b994860f40 2024-04-23T18:40:05.956 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.206+0000 7fbaa6b67740 10 client.18126 fetch_fsmap learned FSMap version 64 2024-04-23T18:40:05.964 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.206+0000 7fbaa6b67740 1 -- 172.21.15.50:0/1728494851 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55b9948e1b10 con 0x55b994860f40 2024-04-23T18:40:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.206+0000 7fba857fa640 1 -- 172.21.15.50:0/1728494851 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b99485c980 con 0x55b994860f40 2024-04-23T18:40:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.206+0000 7fba857fa640 1 -- 172.21.15.50:0/1728494851 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7fba8c01a050 con 0x55b994860f40 2024-04-23T18:40:06.106 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.206+0000 7fbaa6b67740 10 client.18126 fetch_fsmap finished waiting for FSMap version 64 2024-04-23T18:40:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.206+0000 7fbaa6b67740 10 client.18126 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T18:40:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.206+0000 7fbaa6b67740 20 client.18126 populate_metadata read hostname 'smithi050' 2024-04-23T18:40:06.131 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.206+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b9948e2290 0x55b9948e46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:06.140 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.206+0000 7fbaa6b67740 4 client.18126 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T18:40:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.206+0000 7fbaa6b67740 1 -- 172.21.15.50:0/1728494851 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b9947495f0 con 0x55b9948e2290 2024-04-23T18:40:06.216 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.206+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b9948e2290 0x55b9948e46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.207+0000 7fbaa1b0b640 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b9948e2290 0x55b9948e46b0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2eb74678ca0afd32 server_cookie=d124759d9cbcff0b in_seq=0 out_seq=0 2024-04-23T18:40:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.207+0000 7fba857fa640 10 client.18126 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:40:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.525+0000 7fba857fa640 1 -- 172.21.15.50:0/1728494851 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b9947495f0 con 0x55b9948e2290 2024-04-23T18:40:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.525+0000 7fba857fa640 10 client.18126 handle_command_reply: tid=0 2024-04-23T18:40:06.304 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.525+0000 7fbaa6b67740 1 client.18126 shutdown 2024-04-23T18:40:06.312 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.525+0000 7fbaa6b67740 1 -- 172.21.15.50:0/1728494851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 msgr2=0x55b9948da8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:06.320 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.525+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b9948da8e0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x55b994867db0 tx=0x7fba8c0045d0 comp rx=0 tx=0).stop 2024-04-23T18:40:06.327 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.525+0000 7fbaa6b67740 1 -- 172.21.15.50:0/1728494851 shutdown_connections 2024-04-23T18:40:06.335 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.525+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55b9948e2290 0x55b9948e46b0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:06.400 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.526+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b99486b800 0x55b994870b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.526+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55b994860f40 0x55b9948da8e0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:06.423 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.526+0000 7fbaa6b67740 1 --2- 172.21.15.50:0/1728494851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55b994860620 0x55b9948da3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.526+0000 7fbaa6b67740 1 -- 172.21.15.50:0/1728494851 >> 172.21.15.50:0/1728494851 conn(0x55b99485cb30 msgr2=0x55b99485bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:06.438 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.526+0000 7fba857fa640 0 client.18126 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:40:06.446 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.526+0000 7fbaa6b67740 1 -- 172.21.15.50:0/1728494851 shutdown_connections 2024-04-23T18:40:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.526+0000 7fbaa6b67740 1 -- 172.21.15.50:0/1728494851 wait complete. 2024-04-23T18:40:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.526+0000 7fbaa6b67740 20 client.18126 trim_cache size 0 max 16384 2024-04-23T18:40:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:{ 2024-04-23T18:40:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "message": "", 2024-04-23T18:40:06.549 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout: "return_code": 0 2024-04-23T18:40:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stdout:} 2024-04-23T18:40:06.563 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.527+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/3955553427 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fba880a3df0 msgr2=0x7fba880a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:06.571 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.527+0000 7fbaa3d6f640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fba880a3df0 0x7fba880a62b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fba9402f500 tx=0x7fba9402a8c0 comp rx=0 tx=0).stop 2024-04-23T18:40:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.527+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/3955553427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba9c1021c0 msgr2=0x7fba9c074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.527+0000 7fbaa3d6f640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba9c1021c0 0x7fba9c074250 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fba9800d950 tx=0x7fba9800de10 comp rx=0 tx=0).stop 2024-04-23T18:40:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.527+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/3955553427 shutdown_connections 2024-04-23T18:40:06.682 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.527+0000 7fbaa3d6f640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fba880a3df0 0x7fba880a62b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.527+0000 7fbaa3d6f640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba9c074cd0 0x7fba9c1a6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:06.790 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.527+0000 7fbaa3d6f640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba9c107290 0x7fba9c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.527+0000 7fbaa3d6f640 1 --2- 172.21.15.50:0/3955553427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba9c1021c0 0x7fba9c074250 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:06.803 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.527+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/3955553427 >> 172.21.15.50:0/3955553427 conn(0x7fba9c0fda20 msgr2=0x7fba9c0fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.527+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/3955553427 shutdown_connections 2024-04-23T18:40:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi050.front.sepia.ceph.com.smithi050.stderr:2024-04-23T18:40:02.527+0000 7fbaa3d6f640 1 -- 172.21.15.50:0/3955553427 wait complete. 2024-04-23T18:40:06.887 INFO:tasks.workunit.client.0.smithi050.stdout: 10 182374 17.72 MB/sec execute 396 sec latency 1631.569 ms 2024-04-23T18:40:06.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:02 smithi050 ceph-mon[29187]: pgmap v684: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.6 MiB/s wr, 330 op/s 2024-04-23T18:40:06.904 INFO:tasks.workunit.client.0.smithi050.stdout: 10 182382 17.68 MB/sec execute 397 sec latency 1042.990 ms 2024-04-23T18:40:06.912 INFO:tasks.workunit.client.0.smithi050.stdout: 10 182382 17.63 MB/sec execute 398 sec latency 2043.047 ms 2024-04-23T18:40:06.922 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:04 smithi050 ceph-mon[29187]: pgmap v685: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.8 MiB/s wr, 475 op/s 2024-04-23T18:40:06.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:04 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:40:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T18:40:06.948 INFO:tasks.workunit.client.0.smithi050.stdout:[211383] write failed on handle 13786 (Resource temporarily unavailable) 2024-04-23T18:40:06.958 INFO:tasks.workunit.client.0.smithi050.stdout:Child failed with status 1 2024-04-23T18:40:07.014 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-04-23T18:40:07.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-23T18:40:07.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:05 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:40:07.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:05 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:40:07.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:05 smithi050 ceph-mon[29187]: pgmap v686: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.3 MiB/s wr, 303 op/s 2024-04-23T18:40:07.109 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:05 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:40:07.117 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:05 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:40:07.168 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:05 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:40:07.178 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:04 smithi177 ceph-mon[32308]: pgmap v685: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.8 MiB/s wr, 475 op/s 2024-04-23T18:40:07.186 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:04 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:40:07.194 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:05 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:40:07.204 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:05 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:40:07.212 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:05 smithi177 ceph-mon[32308]: pgmap v686: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.3 MiB/s wr, 303 op/s 2024-04-23T18:40:07.281 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:05 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:40:07.290 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:05 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:40:07.298 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:05 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:40:07.307 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:04 smithi188 ceph-mon[32694]: pgmap v685: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.8 MiB/s wr, 475 op/s 2024-04-23T18:40:07.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:04 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:40:07.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:05 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:40:07.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:05 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:40:07.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:05 smithi188 ceph-mon[32694]: pgmap v686: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.3 MiB/s wr, 303 op/s 2024-04-23T18:40:07.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:05 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:40:07.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:05 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:40:07.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:05 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:40:07.758 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=57e0caf76fdb7ab8b1358588f292d08519163844 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-23T18:40:08.011 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1ff78855679e49a5a20e6e4df305d879 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_57e0caf76fdb7ab8b1358588f292d08519163844/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=57e0caf76fdb7ab8b1358588f292d08519163844 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-23T18:40:08.022 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-23T18:40:08.040 INFO:tasks.quiescer:joining Quiescers 2024-04-23T18:40:08.067 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-04-23T18:40:08.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:40:08.193 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:07 smithi050 ceph-mon[29187]: pgmap v687: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.5 MiB/s wr, 404 op/s 2024-04-23T18:40:08.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.209+0000 7f32edeae640 1 -- 172.21.15.50:0/1237560652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e8101110 msgr2=0x7f32e81014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.209+0000 7f32edeae640 1 --2- 172.21.15.50:0/1237560652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e8101110 0x7f32e81014f0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f32dc009900 tx=0x7f32dc02a410 comp rx=0 tx=0).stop 2024-04-23T18:40:08.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.209+0000 7f32edeae640 1 -- 172.21.15.50:0/1237560652 shutdown_connections 2024-04-23T18:40:08.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.209+0000 7f32edeae640 1 --2- 172.21.15.50:0/1237560652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32e8106190 0x7f32e810dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:08.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.209+0000 7f32edeae640 1 --2- 172.21.15.50:0/1237560652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32e8101ac0 0x7f32e8105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:08.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.209+0000 7f32edeae640 1 --2- 172.21.15.50:0/1237560652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e8101110 0x7f32e81014f0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:08.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.209+0000 7f32edeae640 1 -- 172.21.15.50:0/1237560652 >> 172.21.15.50:0/1237560652 conn(0x7f32e8077fb0 msgr2=0x7f32e80783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:08.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.209+0000 7f32edeae640 1 -- 172.21.15.50:0/1237560652 shutdown_connections 2024-04-23T18:40:08.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.209+0000 7f32edeae640 1 -- 172.21.15.50:0/1237560652 wait complete. 2024-04-23T18:40:08.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32edeae640 1 Processor -- start 2024-04-23T18:40:08.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32edeae640 1 -- start start 2024-04-23T18:40:08.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32edeae640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32e8101110 0x7f32e819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:08.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32edeae640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e8106190 0x7f32e819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:08.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32edeae640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32e81985e0 0x7f32e8198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:08.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32edeae640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81110f0 con 0x7f32e8106190 2024-04-23T18:40:08.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32edeae640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f32e8110f70 con 0x7f32e81985e0 2024-04-23T18:40:08.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32edeae640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f32e8111270 con 0x7f32e8101110 2024-04-23T18:40:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32e6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e8106190 0x7f32e819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32e6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e8106190 0x7f32e819e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36806/0 (socket says 172.21.15.50:36806) 2024-04-23T18:40:08.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32e6ffd640 1 -- 172.21.15.50:0/4034470769 learned_addr learned my addr 172.21.15.50:0/4034470769 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:08.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32e7fff640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32e81985e0 0x7f32e8198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:08.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.210+0000 7f32e77fe640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32e8101110 0x7f32e819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:08.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.211+0000 7f32e6ffd640 1 -- 172.21.15.50:0/4034470769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32e8101110 msgr2=0x7f32e819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:08.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.211+0000 7f32e6ffd640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32e8101110 0x7f32e819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:08.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.211+0000 7f32e6ffd640 1 -- 172.21.15.50:0/4034470769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32e81985e0 msgr2=0x7f32e8198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:08.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.211+0000 7f32e6ffd640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32e81985e0 0x7f32e8198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:08.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.211+0000 7f32e6ffd640 1 -- 172.21.15.50:0/4034470769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e8199100 con 0x7f32e8106190 2024-04-23T18:40:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.211+0000 7f32e6ffd640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e8106190 0x7f32e819e970 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f32d800e9f0 tx=0x7f32d800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:08.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.211+0000 7f32e4ff9640 1 -- 172.21.15.50:0/4034470769 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d800ccc0 con 0x7f32e8106190 2024-04-23T18:40:08.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.211+0000 7f32e4ff9640 1 -- 172.21.15.50:0/4034470769 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32d80054d0 con 0x7f32e8106190 2024-04-23T18:40:08.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.211+0000 7f32edeae640 1 -- 172.21.15.50:0/4034470769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e81993f0 con 0x7f32e8106190 2024-04-23T18:40:08.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.211+0000 7f32e4ff9640 1 -- 172.21.15.50:0/4034470769 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8010430 con 0x7f32e8106190 2024-04-23T18:40:08.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.211+0000 7f32edeae640 1 -- 172.21.15.50:0/4034470769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e81a55f0 con 0x7f32e8106190 2024-04-23T18:40:08.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.212+0000 7f32edeae640 1 -- 172.21.15.50:0/4034470769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32e80691c0 con 0x7f32e8106190 2024-04-23T18:40:08.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.213+0000 7f32e4ff9640 1 -- 172.21.15.50:0/4034470769 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f32d80046c0 con 0x7f32e8106190 2024-04-23T18:40:08.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.214+0000 7f32e4ff9640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f32d00a3d20 0x7f32d00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:08.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.214+0000 7f32e77fe640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f32d00a3d20 0x7f32d00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:08.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.214+0000 7f32e4ff9640 1 -- 172.21.15.50:0/4034470769 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8542+0+0 (secure 0 0 0) 0x7f32d80d0ce0 con 0x7f32e8106190 2024-04-23T18:40:08.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.214+0000 7f32e77fe640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f32d00a3d20 0x7f32d00a61e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f32dc031440 tx=0x7f32dc0095f0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:08.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.218+0000 7f32e4ff9640 1 -- 172.21.15.50:0/4034470769 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f32d809d370 con 0x7f32e8106190 2024-04-23T18:40:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:07 smithi188 ceph-mon[32694]: pgmap v687: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.5 MiB/s wr, 404 op/s 2024-04-23T18:40:08.920 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:07 smithi177 ceph-mon[32308]: pgmap v687: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.5 MiB/s wr, 404 op/s 2024-04-23T18:40:08.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.445+0000 7f32edeae640 1 -- 172.21.15.50:0/4034470769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32e8199b10 con 0x7f32e8106190 2024-04-23T18:40:08.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.446+0000 7f32e4ff9640 1 -- 172.21.15.50:0/4034470769 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+11878 (secure 0 0 0) 0x7f32e8199b10 con 0x7f32e8106190 2024-04-23T18:40:08.998 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:40:09.005 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":64,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:39:43.877572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27194":{"gid":27194,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2754597719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2754597719},{"type":"v1","addr":"172.21.15.177:6837","nonce":2754597719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:40:09.021 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 64 2024-04-23T18:40:09.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.448+0000 7f32edeae640 1 -- 172.21.15.50:0/4034470769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f32d00a3d20 msgr2=0x7f32d00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:09.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.448+0000 7f32edeae640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f32d00a3d20 0x7f32d00a61e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f32dc031440 tx=0x7f32dc0095f0 comp rx=0 tx=0).stop 2024-04-23T18:40:09.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.448+0000 7f32edeae640 1 -- 172.21.15.50:0/4034470769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e8106190 msgr2=0x7f32e819e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:09.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.448+0000 7f32edeae640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e8106190 0x7f32e819e970 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f32d800e9f0 tx=0x7f32d800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:40:09.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.448+0000 7f32edeae640 1 -- 172.21.15.50:0/4034470769 shutdown_connections 2024-04-23T18:40:09.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.448+0000 7f32edeae640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f32d00a3d20 0x7f32d00a61e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:09.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.448+0000 7f32edeae640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32e81985e0 0x7f32e8198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:09.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.448+0000 7f32edeae640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e8106190 0x7f32e819e970 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:09.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.448+0000 7f32edeae640 1 --2- 172.21.15.50:0/4034470769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32e8101110 0x7f32e819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:09.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.448+0000 7f32edeae640 1 -- 172.21.15.50:0/4034470769 >> 172.21.15.50:0/4034470769 conn(0x7f32e8077fb0 msgr2=0x7f32e806a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:09.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.448+0000 7f32edeae640 1 -- 172.21.15.50:0/4034470769 shutdown_connections 2024-04-23T18:40:09.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:08.448+0000 7f32edeae640 1 -- 172.21.15.50:0/4034470769 wait complete. 2024-04-23T18:40:09.290 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24506 quiesce db --cancel --all' 2024-04-23T18:40:09.298 DEBUG:teuthology.orchestra.run.smithi050:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24506 quiesce db --cancel --all 2024-04-23T18:40:09.367 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:08 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4034470769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:40:09.375 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:08 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4034470769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:40:09.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:08 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4034470769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:40:09.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.581+0000 7f23172c2640 1 -- 172.21.15.50:0/1584347234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2310106e80 msgr2=0x7f2310107260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:09.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.581+0000 7f23172c2640 1 --2- 172.21.15.50:0/1584347234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2310106e80 0x7f2310107260 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f230c01acb0 tx=0x7f230c03b920 comp rx=0 tx=0).stop 2024-04-23T18:40:09.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.583+0000 7f23172c2640 1 -- 172.21.15.50:0/1584347234 shutdown_connections 2024-04-23T18:40:09.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.583+0000 7f23172c2640 1 --2- 172.21.15.50:0/1584347234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f231006ad20 0x7f2310105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:09.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.583+0000 7f23172c2640 1 --2- 172.21.15.50:0/1584347234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f231006a2a0 0x7f231006a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:09.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.583+0000 7f23172c2640 1 --2- 172.21.15.50:0/1584347234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2310106e80 0x7f2310107260 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:09.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.583+0000 7f23172c2640 1 -- 172.21.15.50:0/1584347234 >> 172.21.15.50:0/1584347234 conn(0x7f23100769d0 msgr2=0x7f2310076de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:09.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.585+0000 7f23172c2640 1 -- 172.21.15.50:0/1584347234 shutdown_connections 2024-04-23T18:40:09.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.586+0000 7f23172c2640 1 -- 172.21.15.50:0/1584347234 wait complete. 2024-04-23T18:40:09.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.587+0000 7f23172c2640 1 Processor -- start 2024-04-23T18:40:09.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.587+0000 7f23172c2640 1 -- start start 2024-04-23T18:40:09.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.588+0000 7f23172c2640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f231006a2a0 0x7f23100729e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:09.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.588+0000 7f23172c2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f231006ad20 0x7f2310072f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:09.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.588+0000 7f23172c2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2310106e80 0x7f23101a3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:09.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.588+0000 7f23172c2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f231010a010 con 0x7f231006ad20 2024-04-23T18:40:09.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.588+0000 7f23172c2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2310109e90 con 0x7f2310106e80 2024-04-23T18:40:09.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.588+0000 7f23172c2640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f231010a190 con 0x7f231006a2a0 2024-04-23T18:40:09.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.588+0000 7f231505e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f231006a2a0 0x7f23100729e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:09.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.588+0000 7f231505e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f231006a2a0 0x7f23100729e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:57798/0 (socket says 172.21.15.50:57798) 2024-04-23T18:40:09.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.588+0000 7f231505e640 1 -- 172.21.15.50:0/1940726410 learned_addr learned my addr 172.21.15.50:0/1940726410 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:09.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.588+0000 7f231485d640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f231006ad20 0x7f2310072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:09.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.588+0000 7f231585f640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2310106e80 0x7f23101a3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:09.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.589+0000 7f231485d640 1 -- 172.21.15.50:0/1940726410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f231006a2a0 msgr2=0x7f23100729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:09.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.589+0000 7f231485d640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f231006a2a0 0x7f23100729e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:10.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.589+0000 7f231485d640 1 -- 172.21.15.50:0/1940726410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2310106e80 msgr2=0x7f23101a3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:10.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.589+0000 7f231485d640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2310106e80 0x7f23101a3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:10.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.589+0000 7f231485d640 1 -- 172.21.15.50:0/1940726410 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23101a3c90 con 0x7f231006ad20 2024-04-23T18:40:10.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.589+0000 7f231485d640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f231006ad20 0x7f2310072f20 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f230000e9f0 tx=0x7f230000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:10.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.590+0000 7f23067fc640 1 -- 172.21.15.50:0/1940726410 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230000ccc0 con 0x7f231006ad20 2024-04-23T18:40:10.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.590+0000 7f23067fc640 1 -- 172.21.15.50:0/1940726410 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23000108e0 con 0x7f231006ad20 2024-04-23T18:40:10.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.590+0000 7f23067fc640 1 -- 172.21.15.50:0/1940726410 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23000188e0 con 0x7f231006ad20 2024-04-23T18:40:10.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.590+0000 7f23172c2640 1 -- 172.21.15.50:0/1940726410 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23101a3f80 con 0x7f231006ad20 2024-04-23T18:40:10.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.590+0000 7f23172c2640 1 -- 172.21.15.50:0/1940726410 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23101ab850 con 0x7f231006ad20 2024-04-23T18:40:10.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.594+0000 7f2319fd1740 1 Processor -- start 2024-04-23T18:40:10.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.596+0000 7f2319fd1740 1 -- start start 2024-04-23T18:40:10.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.596+0000 7f23067fc640 1 -- 172.21.15.50:0/1940726410 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2300018a80 con 0x7f231006ad20 2024-04-23T18:40:10.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.596+0000 7f2319fd1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 0x55c5b57d5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:10.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.596+0000 7f2319fd1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57dfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:10.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.596+0000 7f2319fd1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57e27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:10.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.596+0000 7f2319fd1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c5b573b130 con 0x55c5b57e0380 2024-04-23T18:40:10.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.596+0000 7f2319fd1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c5b56d7c50 con 0x55c5b57d5360 2024-04-23T18:40:10.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.596+0000 7f2319fd1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c5b57370f0 con 0x55c5b57d5d10 2024-04-23T18:40:10.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.596+0000 7f231485d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57e27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:10.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.596+0000 7f231585f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 0x55c5b57d5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.596+0000 7f231585f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 0x55c5b57d5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.50:55818/0 (socket says 172.21.15.50:55818) 2024-04-23T18:40:10.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.596+0000 7f231585f640 1 -- 172.21.15.50:0/1815662471 learned_addr learned my addr 172.21.15.50:0/1815662471 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:10.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.597+0000 7f231505e640 1 --2- 172.21.15.50:0/1815662471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57dfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:10.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.597+0000 7f231585f640 1 -- 172.21.15.50:0/1815662471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 msgr2=0x55c5b57dfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:10.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.597+0000 7f231585f640 1 --2- 172.21.15.50:0/1815662471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57dfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:10.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.597+0000 7f231585f640 1 -- 172.21.15.50:0/1815662471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 msgr2=0x55c5b57e27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:10.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.597+0000 7f231585f640 1 --2- 172.21.15.50:0/1815662471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57e27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.597+0000 7f231585f640 1 -- 172.21.15.50:0/1815662471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5b56f1fd0 con 0x55c5b57d5360 2024-04-23T18:40:10.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.597+0000 7f231485d640 1 --2- 172.21.15.50:0/1815662471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57e27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:40:10.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.597+0000 7f23067fc640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22fc0a3f20 0x7f22fc0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:10.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.597+0000 7f231585f640 1 --2- 172.21.15.50:0/1815662471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 0x55c5b57d5740 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2308009b90 tx=0x7f23080117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b10cae88abc1d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:10.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.597+0000 7f23067fc640 1 -- 172.21.15.50:0/1940726410 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8586+0+0 (secure 0 0 0) 0x7f23000d0aa0 con 0x7f231006ad20 2024-04-23T18:40:10.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.597+0000 7f22fb7fe640 1 -- 172.21.15.50:0/1815662471 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2308012450 con 0x55c5b57d5360 2024-04-23T18:40:10.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.598+0000 7f22fb7fe640 1 -- 172.21.15.50:0/1815662471 <== mon.2 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2308012aa0 con 0x55c5b57d5360 2024-04-23T18:40:10.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.598+0000 7f22fb7fe640 1 -- 172.21.15.50:0/1815662471 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230800f9e0 con 0x55c5b57d5360 2024-04-23T18:40:10.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.599+0000 7f231505e640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22fc0a3f20 0x7f22fc0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:10.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.599+0000 7f2319fd1740 1 -- 172.21.15.50:0/1815662471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 msgr2=0x55c5b57d5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:10.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.599+0000 7f2319fd1740 1 --2- 172.21.15.50:0/1815662471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 0x55c5b57d5740 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2308009b90 tx=0x7f23080117b0 comp rx=0 tx=0).stop 2024-04-23T18:40:10.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.599+0000 7f2319fd1740 1 -- 172.21.15.50:0/1815662471 shutdown_connections 2024-04-23T18:40:10.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.599+0000 7f2319fd1740 1 --2- 172.21.15.50:0/1815662471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57e27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.599+0000 7f2319fd1740 1 --2- 172.21.15.50:0/1815662471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57dfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.599+0000 7f2319fd1740 1 --2- 172.21.15.50:0/1815662471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 0x55c5b57d5740 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:11.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.599+0000 7f2319fd1740 1 -- 172.21.15.50:0/1815662471 >> 172.21.15.50:0/1815662471 conn(0x55c5b57d1620 msgr2=0x55c5b57d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.599+0000 7f2319fd1740 1 -- 172.21.15.50:0/1815662471 shutdown_connections 2024-04-23T18:40:11.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.600+0000 7f2319fd1740 1 -- 172.21.15.50:0/1815662471 wait complete. 2024-04-23T18:40:11.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.600+0000 7f2319fd1740 1 -- start start 2024-04-23T18:40:11.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.600+0000 7f2319fd1740 1 Processor -- start 2024-04-23T18:40:11.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.600+0000 7f231505e640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22fc0a3f20 0x7f22fc0a63e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f230c005fa0 tx=0x7f230c01a7f0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:11.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f2319fd1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57d5360 0x55c5b57dd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:11.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f2319fd1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57dd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:11.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f2319fd1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57e0380 0x55c5b57ddd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:11.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f2319fd1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x55c5b57e32f0 con 0x55c5b57d5360 2024-04-23T18:40:11.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f2319fd1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c5b56d5640 con 0x55c5b57e0380 2024-04-23T18:40:11.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f2319fd1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c5b57e3470 con 0x55c5b57d5d10 2024-04-23T18:40:11.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f2319fd1740 10 client.0 _ll_register_callbacks cb 0x55c5b56d5030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:40:11.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f231505e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57d5360 0x55c5b57dd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:11.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f231505e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57d5360 0x55c5b57dd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36838/0 (socket says 172.21.15.50:36838) 2024-04-23T18:40:11.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f231505e640 1 -- 172.21.15.50:0/896727545 learned_addr learned my addr 172.21.15.50:0/896727545 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:11.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f231485d640 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57dd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:11.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f231585f640 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57e0380 0x55c5b57ddd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:11.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f231585f640 1 -- 172.21.15.50:0/896727545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 msgr2=0x55c5b57dd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:11.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f231585f640 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57dd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:11.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f231585f640 1 -- 172.21.15.50:0/896727545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57d5360 msgr2=0x55c5b57dd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:11.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f231585f640 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57d5360 0x55c5b57dd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:11.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f231585f640 1 -- 172.21.15.50:0/896727545 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5b5519a20 con 0x55c5b57e0380 2024-04-23T18:40:11.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.601+0000 7f231485d640 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57dd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:40:11.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.602+0000 7f231585f640 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57e0380 0x55c5b57ddd00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f230800f480 tx=0x7f2308005db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98cb6dfb4417e579 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:11.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.602+0000 7f22f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-23T18:40:11.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.602+0000 7f22f8ff9640 1 -- 172.21.15.50:0/896727545 <== mon.2 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2308029020 con 0x55c5b57e0380 2024-04-23T18:40:11.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.602+0000 7f22f8ff9640 1 -- 172.21.15.50:0/896727545 <== mon.2 v2:172.21.15.177:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8586+0+0 (secure 0 0 0) 0x7f2308005250 con 0x55c5b57e0380 2024-04-23T18:40:11.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.603+0000 7f2319fd1740 1 -- 172.21.15.50:0/896727545 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2308012aa0 con 0x55c5b57e0380 2024-04-23T18:40:11.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.604+0000 7f22f8ff9640 1 -- 172.21.15.50:0/896727545 <== mon.2 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f23080128e0 con 0x55c5b57e0380 2024-04-23T18:40:11.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.604+0000 7f2319fd1740 10 client.27620 fetch_fsmap learned FSMap version 64 2024-04-23T18:40:11.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.604+0000 7f2319fd1740 1 -- 172.21.15.50:0/896727545 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55c5b557a1b0 con 0x55c5b57e0380 2024-04-23T18:40:11.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.605+0000 7f22f8ff9640 1 -- 172.21.15.50:0/896727545 <== mon.2 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2308012aa0 con 0x55c5b57e0380 2024-04-23T18:40:11.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.605+0000 7f22f8ff9640 1 -- 172.21.15.50:0/896727545 <== mon.2 v2:172.21.15.177:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7f23080046c0 con 0x55c5b57e0380 2024-04-23T18:40:11.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.605+0000 7f2319fd1740 10 client.27620 fetch_fsmap finished waiting for FSMap version 64 2024-04-23T18:40:11.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.605+0000 7f2319fd1740 10 client.27620 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:40:11.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.605+0000 7f2319fd1740 20 client.27620 populate_metadata read hostname 'smithi050' 2024-04-23T18:40:11.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.605+0000 7f2319fd1740 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c5b57f0c80 0x55c5b57f3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:11.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.605+0000 7f2319fd1740 4 client.27620 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T18:40:12.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.605+0000 7f2319fd1740 1 -- 172.21.15.50:0/896727545 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5b5748b50 con 0x55c5b57f0c80 2024-04-23T18:40:12.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.606+0000 7f231505e640 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c5b57f0c80 0x55c5b57f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:12.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.608+0000 7f231505e640 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c5b57f0c80 0x55c5b57f3120 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=299d7b171d8307a2 server_cookie=a222004ac3fce888 in_seq=0 out_seq=0 2024-04-23T18:40:12.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.608+0000 7f22f8ff9640 10 client.27620 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:40:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.611+0000 7f22f8ff9640 1 -- 172.21.15.50:0/896727545 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c5b5748b50 con 0x55c5b57f0c80 2024-04-23T18:40:12.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.611+0000 7f22f8ff9640 10 client.27620 handle_command_reply: tid=0 2024-04-23T18:40:12.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.611+0000 7f2319fd1740 1 client.27620 shutdown 2024-04-23T18:40:12.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.612+0000 7f2319fd1740 1 -- 172.21.15.50:0/896727545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57e0380 msgr2=0x55c5b57ddd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:12.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.612+0000 7f2319fd1740 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57e0380 0x55c5b57ddd00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f230800f480 tx=0x7f2308005db0 comp rx=0 tx=0).stop 2024-04-23T18:40:12.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.612+0000 7f2319fd1740 1 -- 172.21.15.50:0/896727545 shutdown_connections 2024-04-23T18:40:12.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.612+0000 7f2319fd1740 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c5b57f0c80 0x55c5b57f3120 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:12.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.612+0000 7f2319fd1740 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57e0380 0x55c5b57ddd00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:12.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.612+0000 7f2319fd1740 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57dd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:12.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.612+0000 7f2319fd1740 1 --2- 172.21.15.50:0/896727545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57d5360 0x55c5b57dd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:12.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.612+0000 7f2319fd1740 1 -- 172.21.15.50:0/896727545 >> 172.21.15.50:0/896727545 conn(0x55c5b57d1620 msgr2=0x55c5b57d7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:12.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.612+0000 7f22f8ff9640 0 client.27620 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:40:12.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.613+0000 7f2319fd1740 1 -- 172.21.15.50:0/896727545 shutdown_connections 2024-04-23T18:40:12.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.613+0000 7f2319fd1740 1 -- 172.21.15.50:0/896727545 wait complete. 2024-04-23T18:40:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.613+0000 7f2319fd1740 20 client.27620 trim_cache size 0 max 16384 2024-04-23T18:40:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.647+0000 7f2319fd1740 1 Processor -- start 2024-04-23T18:40:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.648+0000 7f2319fd1740 1 -- start start 2024-04-23T18:40:12.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.648+0000 7f2319fd1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5360 0x55c5b57ec240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:12.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.648+0000 7f2319fd1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5d10 0x55c5b57d0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:12.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.648+0000 7f2319fd1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57f89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:12.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.648+0000 7f2319fd1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f22f40111c0 con 0x55c5b57e0380 2024-04-23T18:40:12.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.648+0000 7f2319fd1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22f400ea30 con 0x55c5b57d5d10 2024-04-23T18:40:12.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.648+0000 7f2319fd1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f22f400c8a0 con 0x55c5b57d5360 2024-04-23T18:40:12.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.649+0000 7f231505e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5360 0x55c5b57ec240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:12.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.649+0000 7f231505e640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5360 0x55c5b57ec240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:57818/0 (socket says 172.21.15.50:57818) 2024-04-23T18:40:12.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.649+0000 7f231505e640 1 -- 172.21.15.50:0/106677360 learned_addr learned my addr 172.21.15.50:0/106677360 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:12.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.649+0000 7f231485d640 1 --2- 172.21.15.50:0/106677360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5d10 0x55c5b57d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:12.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.649+0000 7f231585f640 1 --2- 172.21.15.50:0/106677360 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57f89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:12.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.649+0000 7f231505e640 1 -- 172.21.15.50:0/106677360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5d10 msgr2=0x55c5b57d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:12.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.649+0000 7f231505e640 1 --2- 172.21.15.50:0/106677360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5d10 0x55c5b57d0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:13.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.649+0000 7f231505e640 1 -- 172.21.15.50:0/106677360 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 msgr2=0x55c5b57f89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:13.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.649+0000 7f231505e640 1 --2- 172.21.15.50:0/106677360 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57f89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:13.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.649+0000 7f231505e640 1 -- 172.21.15.50:0/106677360 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5b57d1040 con 0x55c5b57d5360 2024-04-23T18:40:13.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.651+0000 7f231505e640 1 --2- 172.21.15.50:0/106677360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5360 0x55c5b57ec240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f230c005da0 tx=0x7f230c054000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc4c3f7b3289fea9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:13.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.651+0000 7f22faffd640 1 -- 172.21.15.50:0/106677360 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c063340 con 0x55c5b57d5360 2024-04-23T18:40:13.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.651+0000 7f22faffd640 1 -- 172.21.15.50:0/106677360 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f230c041620 con 0x55c5b57d5360 2024-04-23T18:40:13.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.651+0000 7f22faffd640 1 -- 172.21.15.50:0/106677360 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c05a430 con 0x55c5b57d5360 2024-04-23T18:40:13.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.652+0000 7f2319fd1740 1 -- 172.21.15.50:0/106677360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5360 msgr2=0x55c5b57ec240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:13.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.652+0000 7f2319fd1740 1 --2- 172.21.15.50:0/106677360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5360 0x55c5b57ec240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f230c005da0 tx=0x7f230c054000 comp rx=0 tx=0).stop 2024-04-23T18:40:13.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.655+0000 7f2319fd1740 1 -- 172.21.15.50:0/106677360 shutdown_connections 2024-04-23T18:40:13.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.655+0000 7f2319fd1740 1 --2- 172.21.15.50:0/106677360 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57f89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:13.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.655+0000 7f2319fd1740 1 --2- 172.21.15.50:0/106677360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5d10 0x55c5b57d0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:13.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.655+0000 7f2319fd1740 1 --2- 172.21.15.50:0/106677360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5360 0x55c5b57ec240 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:13.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.655+0000 7f2319fd1740 1 -- 172.21.15.50:0/106677360 >> 172.21.15.50:0/106677360 conn(0x55c5b57d1620 msgr2=0x55c5b57dd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:13.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.660+0000 7f2319fd1740 1 -- 172.21.15.50:0/106677360 shutdown_connections 2024-04-23T18:40:13.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.660+0000 7f2319fd1740 1 -- 172.21.15.50:0/106677360 wait complete. 2024-04-23T18:40:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.660+0000 7f2319fd1740 1 -- start start 2024-04-23T18:40:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.660+0000 7f2319fd1740 1 Processor -- start 2024-04-23T18:40:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.661+0000 7f2319fd1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 0x55c5b57da0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.661+0000 7f2319fd1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57da5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:13.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.661+0000 7f2319fd1740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57dab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:13.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.661+0000 7f2319fd1740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f22f400cef0 con 0x55c5b57e0380 2024-04-23T18:40:13.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.661+0000 7f2319fd1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22f400e360 con 0x55c5b57d5360 2024-04-23T18:40:13.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.661+0000 7f2319fd1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f22f400d5c0 con 0x55c5b57d5d10 2024-04-23T18:40:13.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.661+0000 7f2319fd1740 10 client.0 _ll_register_callbacks cb 0x7f23080127a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T18:40:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.662+0000 7f231485d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57da5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:13.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.662+0000 7f231585f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57dab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:13.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.662+0000 7f231485d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57da5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.50:57830/0 (socket says 172.21.15.50:57830) 2024-04-23T18:40:13.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.662+0000 7f231505e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 0x55c5b57da0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:13.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.662+0000 7f231485d640 1 -- 172.21.15.50:0/2519573104 learned_addr learned my addr 172.21.15.50:0/2519573104 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:13.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.662+0000 7f231485d640 1 -- 172.21.15.50:0/2519573104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 msgr2=0x55c5b57da0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:13.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.662+0000 7f231485d640 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 0x55c5b57da0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.662+0000 7f231485d640 1 -- 172.21.15.50:0/2519573104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 msgr2=0x55c5b57dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T18:40:13.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.662+0000 7f231485d640 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57dab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.662+0000 7f231485d640 1 -- 172.21.15.50:0/2519573104 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5b540d310 con 0x55c5b57d5d10 2024-04-23T18:40:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.662+0000 7f231585f640 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57dab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:40:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.662+0000 7f231505e640 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 0x55c5b57da0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:40:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.663+0000 7f231485d640 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57da5e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2300099370 tx=0x7f230009c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f7460e9453c33f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:13.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.663+0000 7f22f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-23T18:40:13.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.664+0000 7f22f8ff9640 1 -- 172.21.15.50:0/2519573104 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230009d0a0 con 0x55c5b57d5d10 2024-04-23T18:40:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.664+0000 7f2319fd1740 1 -- 172.21.15.50:0/2519573104 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f230c041620 con 0x55c5b57d5d10 2024-04-23T18:40:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.664+0000 7f22f8ff9640 1 -- 172.21.15.50:0/2519573104 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8586+0+0 (secure 0 0 0) 0x7f2300099010 con 0x55c5b57d5d10 2024-04-23T18:40:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.665+0000 7f22f8ff9640 1 -- 172.21.15.50:0/2519573104 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f230009c9a0 con 0x55c5b57d5d10 2024-04-23T18:40:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.665+0000 7f2319fd1740 10 client.27649 fetch_fsmap learned FSMap version 64 2024-04-23T18:40:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.665+0000 7f2319fd1740 1 -- 172.21.15.50:0/2519573104 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55c5b5857560 con 0x55c5b57d5d10 2024-04-23T18:40:14.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.665+0000 7f22f8ff9640 1 -- 172.21.15.50:0/2519573104 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f230c041620 con 0x55c5b57d5d10 2024-04-23T18:40:14.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.665+0000 7f22f8ff9640 1 -- 172.21.15.50:0/2519573104 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7f230009bdf0 con 0x55c5b57d5d10 2024-04-23T18:40:14.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.665+0000 7f2319fd1740 10 client.27649 fetch_fsmap finished waiting for FSMap version 64 2024-04-23T18:40:14.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.665+0000 7f2319fd1740 10 client.27649 resolve_mds: validated gid 24506 aka daemon mds.b 2024-04-23T18:40:14.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.665+0000 7f2319fd1740 20 client.27649 populate_metadata read hostname 'smithi050' 2024-04-23T18:40:14.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.665+0000 7f2319fd1740 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c5b5857cf0 0x55c5b585a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:14.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.665+0000 7f2319fd1740 4 client.27649 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-23T18:40:14.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.665+0000 7f2319fd1740 1 -- 172.21.15.50:0/2519573104 --> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55c5b57429c0 con 0x55c5b5857cf0 2024-04-23T18:40:14.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.666+0000 7f231505e640 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c5b5857cf0 0x55c5b585a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.666+0000 7f22f8ff9640 10 client.27649 ms_handle_connect on v2:172.21.15.177:6838/2936205355 2024-04-23T18:40:14.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.666+0000 7f231505e640 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c5b5857cf0 0x55c5b585a110 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84e48361594af0e5 server_cookie=69a6a850524f4bbd in_seq=0 out_seq=0 2024-04-23T18:40:14.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f22f8ff9640 1 -- 172.21.15.50:0/2519573104 <== mds.0 v2:172.21.15.177:6838/2936205355 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x55c5b57429c0 con 0x55c5b5857cf0 2024-04-23T18:40:14.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f22f8ff9640 10 client.27649 handle_command_reply: tid=0 2024-04-23T18:40:14.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f2319fd1740 1 client.27649 shutdown 2024-04-23T18:40:14.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f2319fd1740 1 -- 172.21.15.50:0/2519573104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 msgr2=0x55c5b57da5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:14.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f2319fd1740 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57da5e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2300099370 tx=0x7f230009c7e0 comp rx=0 tx=0).stop 2024-04-23T18:40:14.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f2319fd1740 1 -- 172.21.15.50:0/2519573104 shutdown_connections 2024-04-23T18:40:14.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f2319fd1740 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.177:6838/2936205355,v1:172.21.15.177:6839/2936205355] conn(0x55c5b5857cf0 0x55c5b585a110 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f2319fd1740 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c5b57e0380 0x55c5b57dab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:14.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f2319fd1740 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c5b57d5d10 0x55c5b57da5e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:14.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f2319fd1740 1 --2- 172.21.15.50:0/2519573104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5b57d5360 0x55c5b57da0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:14.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f2319fd1740 1 -- 172.21.15.50:0/2519573104 >> 172.21.15.50:0/2519573104 conn(0x55c5b57d1620 msgr2=0x55c5b57dd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:14.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f22f8ff9640 0 client.27649 ms_handle_reset on v2:172.21.15.177:6838/2936205355 2024-04-23T18:40:14.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f2319fd1740 1 -- 172.21.15.50:0/2519573104 shutdown_connections 2024-04-23T18:40:14.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f2319fd1740 1 -- 172.21.15.50:0/2519573104 wait complete. 2024-04-23T18:40:14.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.667+0000 7f2319fd1740 20 client.27649 trim_cache size 0 max 16384 2024-04-23T18:40:14.624 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:40:14.707 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 64, 2024-04-23T18:40:14.805 INFO:teuthology.orchestra.run.smithi050.stdout: "leader": 24506, 2024-04-23T18:40:14.816 INFO:teuthology.orchestra.run.smithi050.stdout: "set_version": 72, 2024-04-23T18:40:14.826 INFO:teuthology.orchestra.run.smithi050.stdout: "sets": {} 2024-04-23T18:40:14.838 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:40:14.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.668+0000 7f23172c2640 1 -- 172.21.15.50:0/1940726410 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22fc0a3f20 msgr2=0x7f22fc0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:14.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.668+0000 7f23172c2640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22fc0a3f20 0x7f22fc0a63e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f230c005fa0 tx=0x7f230c01a7f0 comp rx=0 tx=0).stop 2024-04-23T18:40:14.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.668+0000 7f23172c2640 1 -- 172.21.15.50:0/1940726410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f231006ad20 msgr2=0x7f2310072f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.668+0000 7f23172c2640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f231006ad20 0x7f2310072f20 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f230000e9f0 tx=0x7f230000eeb0 comp rx=0 tx=0).stop 2024-04-23T18:40:14.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.668+0000 7f23172c2640 1 -- 172.21.15.50:0/1940726410 shutdown_connections 2024-04-23T18:40:14.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.668+0000 7f23172c2640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22fc0a3f20 0x7f22fc0a63e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:14.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.668+0000 7f23172c2640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2310106e80 0x7f23101a3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:14.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.668+0000 7f23172c2640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f231006ad20 0x7f2310072f20 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:15.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.668+0000 7f23172c2640 1 --2- 172.21.15.50:0/1940726410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f231006a2a0 0x7f23100729e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:15.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.668+0000 7f23172c2640 1 -- 172.21.15.50:0/1940726410 >> 172.21.15.50:0/1940726410 conn(0x7f23100769d0 msgr2=0x7f2310103a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:15.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.668+0000 7f23172c2640 1 -- 172.21.15.50:0/1940726410 shutdown_connections 2024-04-23T18:40:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:09.668+0000 7f23172c2640 1 -- 172.21.15.50:0/1940726410 wait complete. 2024-04-23T18:40:15.177 INFO:tasks.quiescer:done joining Quiescers 2024-04-23T18:40:15.226 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-23T18:40:15.249 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi050.front.sepia.ceph.com 2024-04-23T18:40:15.260 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:09 smithi050 ceph-mon[29187]: pgmap v688: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.6 MiB/s wr, 340 op/s 2024-04-23T18:40:15.270 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:09 smithi050 ceph-mon[29187]: osdmap e127: 12 total, 12 up, 12 in 2024-04-23T18:40:15.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:11 smithi050 ceph-mon[29187]: pgmap v690: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.0 MiB/s wr, 337 op/s 2024-04-23T18:40:15.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:13 smithi050 ceph-mon[29187]: pgmap v691: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.4 MiB/s wr, 197 op/s 2024-04-23T18:40:15.320 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:09 smithi188 ceph-mon[32694]: pgmap v688: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.6 MiB/s wr, 340 op/s 2024-04-23T18:40:15.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:09 smithi188 ceph-mon[32694]: osdmap e127: 12 total, 12 up, 12 in 2024-04-23T18:40:15.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:11 smithi188 ceph-mon[32694]: pgmap v690: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.0 MiB/s wr, 337 op/s 2024-04-23T18:40:15.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:13 smithi188 ceph-mon[32694]: pgmap v691: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.4 MiB/s wr, 197 op/s 2024-04-23T18:40:15.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:09 smithi177 ceph-mon[32308]: pgmap v688: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.6 MiB/s wr, 340 op/s 2024-04-23T18:40:15.373 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:09 smithi177 ceph-mon[32308]: osdmap e127: 12 total, 12 up, 12 in 2024-04-23T18:40:15.385 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:11 smithi177 ceph-mon[32308]: pgmap v690: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.0 MiB/s wr, 337 op/s 2024-04-23T18:40:15.401 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:13 smithi177 ceph-mon[32308]: pgmap v691: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.4 MiB/s wr, 197 op/s 2024-04-23T18:40:15.415 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-23T18:40:15.502 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-23T18:40:15.559 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T18:40:15.622 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi050.front.sepia.ceph.com 2024-04-23T18:40:15.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-23T18:40:15.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-23T18:40:15.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-23T18:40:15.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.906+0000 7f147b37d640 1 -- 172.21.15.50:0/2815794174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474106660 msgr2=0x7f1474106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:15.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.906+0000 7f147b37d640 1 --2- 172.21.15.50:0/2815794174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474106660 0x7f1474106a40 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f1470009930 tx=0x7f147002a420 comp rx=0 tx=0).stop 2024-04-23T18:40:15.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.906+0000 7f147b37d640 1 -- 172.21.15.50:0/2815794174 shutdown_connections 2024-04-23T18:40:15.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.906+0000 7f147b37d640 1 --2- 172.21.15.50:0/2815794174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1474100f80 0x7f147410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:16.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.906+0000 7f147b37d640 1 --2- 172.21.15.50:0/2815794174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14741005c0 0x7f1474100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:16.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.906+0000 7f147b37d640 1 --2- 172.21.15.50:0/2815794174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474106660 0x7f1474106a40 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:16.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.906+0000 7f147b37d640 1 -- 172.21.15.50:0/2815794174 >> 172.21.15.50:0/2815794174 conn(0x7f1474077390 msgr2=0x7f14740fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:16.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.906+0000 7f147b37d640 1 -- 172.21.15.50:0/2815794174 shutdown_connections 2024-04-23T18:40:16.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.906+0000 7f147b37d640 1 -- 172.21.15.50:0/2815794174 wait complete. 2024-04-23T18:40:16.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f147b37d640 1 Processor -- start 2024-04-23T18:40:16.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f147b37d640 1 -- start start 2024-04-23T18:40:16.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f147b37d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14741005c0 0x7f1474103e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f147b37d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474100f80 0x7f1474102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:16.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f147b37d640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1474106660 0x7f14741028a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:16.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f147b37d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1474110060 con 0x7f1474100f80 2024-04-23T18:40:16.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f147b37d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f147410fee0 con 0x7f14741005c0 2024-04-23T18:40:16.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f147b37d640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f14741101e0 con 0x7f1474106660 2024-04-23T18:40:16.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f1478918640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474100f80 0x7f1474102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f1478918640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474100f80 0x7f1474102260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51218/0 (socket says 172.21.15.50:51218) 2024-04-23T18:40:16.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f1478918640 1 -- 172.21.15.50:0/3273010395 learned_addr learned my addr 172.21.15.50:0/3273010395 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:16.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f1479119640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14741005c0 0x7f1474103e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:16.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f147991a640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1474106660 0x7f14741028a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:16.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f1478918640 1 -- 172.21.15.50:0/3273010395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1474106660 msgr2=0x7f14741028a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:16.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f1478918640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1474106660 0x7f14741028a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:16.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f1478918640 1 -- 172.21.15.50:0/3273010395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14741005c0 msgr2=0x7f1474103e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:16.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f1478918640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14741005c0 0x7f1474103e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:16.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f1478918640 1 -- 172.21.15.50:0/3273010395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14741ab140 con 0x7f1474100f80 2024-04-23T18:40:16.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.907+0000 7f147991a640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1474106660 0x7f14741028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:40:16.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.908+0000 7f1478918640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474100f80 0x7f1474102260 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f146400ddc0 tx=0x7f146400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:16.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.908+0000 7f146a7fc640 1 -- 172.21.15.50:0/3273010395 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146401b070 con 0x7f1474100f80 2024-04-23T18:40:16.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.908+0000 7f146a7fc640 1 -- 172.21.15.50:0/3273010395 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f146400be40 con 0x7f1474100f80 2024-04-23T18:40:16.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.908+0000 7f147b37d640 1 -- 172.21.15.50:0/3273010395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14741ab370 con 0x7f1474100f80 2024-04-23T18:40:16.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.908+0000 7f146a7fc640 1 -- 172.21.15.50:0/3273010395 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1464016a80 con 0x7f1474100f80 2024-04-23T18:40:16.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.908+0000 7f147b37d640 1 -- 172.21.15.50:0/3273010395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14741ab890 con 0x7f1474100f80 2024-04-23T18:40:16.817 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:15 smithi188 ceph-mon[32694]: pgmap v692: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.4 MiB/s wr, 197 op/s 2024-04-23T18:40:16.828 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:15 smithi177 ceph-mon[32308]: pgmap v692: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.4 MiB/s wr, 197 op/s 2024-04-23T18:40:16.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.909+0000 7f147b37d640 1 -- 172.21.15.50:0/3273010395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1474101d00 con 0x7f1474100f80 2024-04-23T18:40:16.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.910+0000 7f146a7fc640 1 -- 172.21.15.50:0/3273010395 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f146401e070 con 0x7f1474100f80 2024-04-23T18:40:16.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.911+0000 7f146a7fc640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f14600a3dc0 0x7f14600a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:16.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.911+0000 7f1479119640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f14600a3dc0 0x7f14600a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:16.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.911+0000 7f146a7fc640 1 -- 172.21.15.50:0/3273010395 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8586+0+0 (secure 0 0 0) 0x7f14640d08b0 con 0x7f1474100f80 2024-04-23T18:40:16.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.911+0000 7f1479119640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f14600a3dc0 0x7f14600a6280 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1470009900 tx=0x7f1470005a20 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:16.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:15.915+0000 7f146a7fc640 1 -- 172.21.15.50:0/3273010395 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f146409c9d0 con 0x7f1474100f80 2024-04-23T18:40:16.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:15 smithi050 ceph-mon[29187]: pgmap v692: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.4 MiB/s wr, 197 op/s 2024-04-23T18:40:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.127+0000 7f147b37d640 1 -- 172.21.15.50:0/3273010395 --> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f1474062390 con 0x7f14600a3dc0 2024-04-23T18:40:17.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.139+0000 7f146a7fc640 1 -- 172.21.15.50:0/3273010395 <== mgr.25018 v2:172.21.15.50:6800/4210816660 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f1474062390 con 0x7f14600a3dc0 2024-04-23T18:40:17.084 INFO:teuthology.orchestra.run.smithi050.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-23T00:00:00", "created": "2024-04-23T18:18:31", "first": "2024-04-23T18:19:01", "last": "2024-04-23T18:40:00", "last_pruned": "2024-04-23T18:40:02", "created_count": 22, "pruned_count": 16, "active": true} 2024-04-23T18:40:17.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.142+0000 7f147b37d640 1 -- 172.21.15.50:0/3273010395 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f14600a3dc0 msgr2=0x7f14600a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:17.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.142+0000 7f147b37d640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f14600a3dc0 0x7f14600a6280 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1470009900 tx=0x7f1470005a20 comp rx=0 tx=0).stop 2024-04-23T18:40:17.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.142+0000 7f147b37d640 1 -- 172.21.15.50:0/3273010395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474100f80 msgr2=0x7f1474102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:17.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.142+0000 7f147b37d640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474100f80 0x7f1474102260 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f146400ddc0 tx=0x7f146400a620 comp rx=0 tx=0).stop 2024-04-23T18:40:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.142+0000 7f147b37d640 1 -- 172.21.15.50:0/3273010395 shutdown_connections 2024-04-23T18:40:17.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.142+0000 7f147b37d640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f14600a3dc0 0x7f14600a6280 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:17.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.142+0000 7f147b37d640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1474106660 0x7f14741028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:17.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.142+0000 7f147b37d640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1474100f80 0x7f1474102260 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:17.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.142+0000 7f147b37d640 1 --2- 172.21.15.50:0/3273010395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14741005c0 0x7f1474103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:17.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.142+0000 7f147b37d640 1 -- 172.21.15.50:0/3273010395 >> 172.21.15.50:0/3273010395 conn(0x7f1474077390 msgr2=0x7f14740fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:17.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.142+0000 7f147b37d640 1 -- 172.21.15.50:0/3273010395 shutdown_connections 2024-04-23T18:40:17.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:16.142+0000 7f147b37d640 1 -- 172.21.15.50:0/3273010395 wait complete. 2024-04-23T18:40:17.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-23T18:40:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.505+0000 7fbe55bad640 1 -- 172.21.15.50:0/2223223367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe50107330 msgr2=0x7fbe5010eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:17.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.505+0000 7fbe55bad640 1 --2- 172.21.15.50:0/2223223367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe50107330 0x7fbe5010eeb0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fbe48009ca0 tx=0x7fbe4802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:40:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.505+0000 7fbe55bad640 1 -- 172.21.15.50:0/2223223367 shutdown_connections 2024-04-23T18:40:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.505+0000 7fbe55bad640 1 --2- 172.21.15.50:0/2223223367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe50107330 0x7fbe5010eeb0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:17.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.505+0000 7fbe55bad640 1 --2- 172.21.15.50:0/2223223367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe50101070 0x7fbe50106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:17.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.505+0000 7fbe55bad640 1 --2- 172.21.15.50:0/2223223367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe501006c0 0x7fbe50100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:17.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.505+0000 7fbe55bad640 1 -- 172.21.15.50:0/2223223367 >> 172.21.15.50:0/2223223367 conn(0x7fbe50077390 msgr2=0x7fbe500fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.505+0000 7fbe55bad640 1 -- 172.21.15.50:0/2223223367 shutdown_connections 2024-04-23T18:40:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.505+0000 7fbe55bad640 1 -- 172.21.15.50:0/2223223367 wait complete. 2024-04-23T18:40:17.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:17 smithi188 ceph-mon[32694]: from='client.18156 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:40:17.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:17 smithi188 ceph-mon[32694]: osdmap e128: 12 total, 12 up, 12 in 2024-04-23T18:40:17.733 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:17 smithi177 ceph-mon[32308]: from='client.18156 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:40:17.781 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:17 smithi177 ceph-mon[32308]: osdmap e128: 12 total, 12 up, 12 in 2024-04-23T18:40:17.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.506+0000 7fbe55bad640 1 Processor -- start 2024-04-23T18:40:17.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.506+0000 7fbe55bad640 1 -- start start 2024-04-23T18:40:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.506+0000 7fbe55bad640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe501006c0 0x7fbe501987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:17.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.506+0000 7fbe55bad640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe50107330 0x7fbe50198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.506+0000 7fbe55bad640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe5019cf70 0x7fbe5019d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.506+0000 7fbe55bad640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbe501122a0 con 0x7fbe5019cf70 2024-04-23T18:40:17.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.506+0000 7fbe55bad640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbe50112120 con 0x7fbe501006c0 2024-04-23T18:40:18.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.506+0000 7fbe55bad640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fbe50112420 con 0x7fbe50107330 2024-04-23T18:40:18.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.506+0000 7fbe4ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe5019cf70 0x7fbe5019d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:18.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.506+0000 7fbe4ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe5019cf70 0x7fbe5019d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51242/0 (socket says 172.21.15.50:51242) 2024-04-23T18:40:18.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.506+0000 7fbe4ffff640 1 -- 172.21.15.50:0/848820112 learned_addr learned my addr 172.21.15.50:0/848820112 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:18.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.506+0000 7fbe4effd640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe50107330 0x7fbe50198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:18.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe4f7fe640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe501006c0 0x7fbe501987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe4ffff640 1 -- 172.21.15.50:0/848820112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe50107330 msgr2=0x7fbe50198ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:18.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe4ffff640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe50107330 0x7fbe50198ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:18.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe4ffff640 1 -- 172.21.15.50:0/848820112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe501006c0 msgr2=0x7fbe501987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe4ffff640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe501006c0 0x7fbe501987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:18.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe4ffff640 1 -- 172.21.15.50:0/848820112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5019da60 con 0x7fbe5019cf70 2024-04-23T18:40:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe4f7fe640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe501006c0 0x7fbe501987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:40:18.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe4ffff640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe5019cf70 0x7fbe5019d420 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fbe48009430 tx=0x7fbe480319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:18.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe4cff9640 1 -- 172.21.15.50:0/848820112 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe48003c30 con 0x7fbe5019cf70 2024-04-23T18:40:18.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe55bad640 1 -- 172.21.15.50:0/848820112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe5019dcf0 con 0x7fbe5019cf70 2024-04-23T18:40:18.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe4cff9640 1 -- 172.21.15.50:0/848820112 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe48003dd0 con 0x7fbe5019cf70 2024-04-23T18:40:18.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe4cff9640 1 -- 172.21.15.50:0/848820112 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4802faa0 con 0x7fbe5019cf70 2024-04-23T18:40:18.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.507+0000 7fbe55bad640 1 -- 172.21.15.50:0/848820112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe501a5540 con 0x7fbe5019cf70 2024-04-23T18:40:18.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.508+0000 7fbe55bad640 1 -- 172.21.15.50:0/848820112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe5004fce0 con 0x7fbe5019cf70 2024-04-23T18:40:18.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.509+0000 7fbe4cff9640 1 -- 172.21.15.50:0/848820112 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbe4802fca0 con 0x7fbe5019cf70 2024-04-23T18:40:18.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.510+0000 7fbe4cff9640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbe380a3dc0 0x7fbe380a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:18.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.510+0000 7fbe4f7fe640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbe380a3dc0 0x7fbe380a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:18.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.510+0000 7fbe4cff9640 1 -- 172.21.15.50:0/848820112 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8542+0+0 (secure 0 0 0) 0x7fbe480f1d30 con 0x7fbe5019cf70 2024-04-23T18:40:18.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.511+0000 7fbe4f7fe640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbe380a3dc0 0x7fbe380a6280 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbe400058c0 tx=0x7fbe40005850 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.515+0000 7fbe4cff9640 1 -- 172.21.15.50:0/848820112 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe480be470 con 0x7fbe5019cf70 2024-04-23T18:40:18.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:17 smithi050 ceph-mon[29187]: from='client.18156 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:40:18.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:17 smithi050 ceph-mon[29187]: osdmap e128: 12 total, 12 up, 12 in 2024-04-23T18:40:18.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.827+0000 7fbe55bad640 1 -- 172.21.15.50:0/848820112 --> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fbe5010a540 con 0x7fbe380a3dc0 2024-04-23T18:40:18.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.838+0000 7fbe4cff9640 1 -- 172.21.15.50:0/848820112 <== mgr.25018 v2:172.21.15.50:6800/4210816660 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fbe5010a540 con 0x7fbe380a3dc0 2024-04-23T18:40:18.720 INFO:teuthology.orchestra.run.smithi050.stdout:/ 1m 6m3h 2024-04-23T18:40:18.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.840+0000 7fbe55bad640 1 -- 172.21.15.50:0/848820112 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbe380a3dc0 msgr2=0x7fbe380a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:18.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.840+0000 7fbe55bad640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbe380a3dc0 0x7fbe380a6280 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbe400058c0 tx=0x7fbe40005850 comp rx=0 tx=0).stop 2024-04-23T18:40:18.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.840+0000 7fbe55bad640 1 -- 172.21.15.50:0/848820112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe5019cf70 msgr2=0x7fbe5019d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:18.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.840+0000 7fbe55bad640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe5019cf70 0x7fbe5019d420 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fbe48009430 tx=0x7fbe480319f0 comp rx=0 tx=0).stop 2024-04-23T18:40:18.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.840+0000 7fbe55bad640 1 -- 172.21.15.50:0/848820112 shutdown_connections 2024-04-23T18:40:18.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.840+0000 7fbe55bad640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fbe380a3dc0 0x7fbe380a6280 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:18.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.840+0000 7fbe55bad640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe5019cf70 0x7fbe5019d420 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:18.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.840+0000 7fbe55bad640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe50107330 0x7fbe50198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:18.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.840+0000 7fbe55bad640 1 --2- 172.21.15.50:0/848820112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe501006c0 0x7fbe501987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:18.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.840+0000 7fbe55bad640 1 -- 172.21.15.50:0/848820112 >> 172.21.15.50:0/848820112 conn(0x7fbe50077390 msgr2=0x7fbe501059e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.841+0000 7fbe55bad640 1 -- 172.21.15.50:0/848820112 shutdown_connections 2024-04-23T18:40:19.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:17.841+0000 7fbe55bad640 1 -- 172.21.15.50:0/848820112 wait complete. 2024-04-23T18:40:19.035 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-23T18:40:19.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:40:19.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:18 smithi050 ceph-mon[29187]: pgmap v694: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 4.0 MiB/s wr, 128 op/s 2024-04-23T18:40:19.088 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:18 smithi188 ceph-mon[32694]: pgmap v694: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 4.0 MiB/s wr, 128 op/s 2024-04-23T18:40:19.101 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:18 smithi177 ceph-mon[32308]: pgmap v694: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 4.0 MiB/s wr, 128 op/s 2024-04-23T18:40:19.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.206+0000 7f9981319640 1 -- 172.21.15.50:0/617239749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c1031e0 msgr2=0x7f997c1035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:19.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.206+0000 7f9981319640 1 --2- 172.21.15.50:0/617239749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c1031e0 0x7f997c1035c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f996c009860 tx=0x7f996c02a460 comp rx=0 tx=0).stop 2024-04-23T18:40:19.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.206+0000 7f9981319640 1 -- 172.21.15.50:0/617239749 shutdown_connections 2024-04-23T18:40:19.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.206+0000 7f9981319640 1 --2- 172.21.15.50:0/617239749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f997c10c520 0x7f997c10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:19.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.206+0000 7f9981319640 1 --2- 172.21.15.50:0/617239749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f997c103b00 0x7f997c10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:19.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.206+0000 7f9981319640 1 --2- 172.21.15.50:0/617239749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c1031e0 0x7f997c1035c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:19.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.206+0000 7f9981319640 1 -- 172.21.15.50:0/617239749 >> 172.21.15.50:0/617239749 conn(0x7f997c077f50 msgr2=0x7f997c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:19.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.206+0000 7f9981319640 1 -- 172.21.15.50:0/617239749 shutdown_connections 2024-04-23T18:40:19.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.206+0000 7f9981319640 1 -- 172.21.15.50:0/617239749 wait complete. 2024-04-23T18:40:19.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f9981319640 1 Processor -- start 2024-04-23T18:40:19.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f9981319640 1 -- start start 2024-04-23T18:40:19.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f9981319640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c1031e0 0x7f997c100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:19.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f9981319640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f997c103b00 0x7f997c101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:19.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f9981319640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f997c10c520 0x7f997c1018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:19.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f9981319640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f997c112200 con 0x7f997c103b00 2024-04-23T18:40:19.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f9981319640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f997c112080 con 0x7f997c1031e0 2024-04-23T18:40:19.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f9981319640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f997c112380 con 0x7f997c10c520 2024-04-23T18:40:19.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f997a7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f997c103b00 0x7f997c101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:19.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f997a7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f997c103b00 0x7f997c101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51262/0 (socket says 172.21.15.50:51262) 2024-04-23T18:40:19.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f997affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c1031e0 0x7f997c100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:19.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f997a7fc640 1 -- 172.21.15.50:0/1940285394 learned_addr learned my addr 172.21.15.50:0/1940285394 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:19.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f997b7fe640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f997c10c520 0x7f997c1018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:19.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.207+0000 7f997a7fc640 1 -- 172.21.15.50:0/1940285394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f997c10c520 msgr2=0x7f997c1018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:19.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.208+0000 7f997a7fc640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f997c10c520 0x7f997c1018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:19.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.208+0000 7f997a7fc640 1 -- 172.21.15.50:0/1940285394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c1031e0 msgr2=0x7f997c100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:19.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.208+0000 7f997a7fc640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c1031e0 0x7f997c100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:19.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.208+0000 7f997a7fc640 1 -- 172.21.15.50:0/1940285394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997c1a6cc0 con 0x7f997c103b00 2024-04-23T18:40:19.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.208+0000 7f997a7fc640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f997c103b00 0x7f997c101370 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f997000e9c0 tx=0x7f997000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.208+0000 7f9963fff640 1 -- 172.21.15.50:0/1940285394 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997000cde0 con 0x7f997c103b00 2024-04-23T18:40:19.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.208+0000 7f9981319640 1 -- 172.21.15.50:0/1940285394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f997c1a6fb0 con 0x7f997c103b00 2024-04-23T18:40:19.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.208+0000 7f9963fff640 1 -- 172.21.15.50:0/1940285394 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99700035f0 con 0x7f997c103b00 2024-04-23T18:40:19.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.208+0000 7f9963fff640 1 -- 172.21.15.50:0/1940285394 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9970010690 con 0x7f997c103b00 2024-04-23T18:40:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.208+0000 7f9981319640 1 -- 172.21.15.50:0/1940285394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f997c1a74c0 con 0x7f997c103b00 2024-04-23T18:40:19.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.209+0000 7f9981319640 1 -- 172.21.15.50:0/1940285394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f997c068d70 con 0x7f997c103b00 2024-04-23T18:40:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.210+0000 7f9963fff640 1 -- 172.21.15.50:0/1940285394 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9970004000 con 0x7f997c103b00 2024-04-23T18:40:19.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.211+0000 7f9963fff640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99640a3e60 0x7f99640a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:19.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.211+0000 7f9963fff640 1 -- 172.21.15.50:0/1940285394 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8542+0+0 (secure 0 0 0) 0x7f99700d0ae0 con 0x7f997c103b00 2024-04-23T18:40:19.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.211+0000 7f997affd640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99640a3e60 0x7f99640a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.212+0000 7f997affd640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99640a3e60 0x7f99640a6320 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f996c005b60 tx=0x7f996c005af0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:20.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.218+0000 7f9963fff640 1 -- 172.21.15.50:0/1940285394 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9970009510 con 0x7f997c103b00 2024-04-23T18:40:20.040 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:19 smithi188 ceph-mon[32694]: from='client.18162 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:40:20.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:19 smithi177 ceph-mon[32308]: from='client.18162 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:40:20.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.441+0000 7f9981319640 1 -- 172.21.15.50:0/1940285394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f997c1076a0 con 0x7f997c103b00 2024-04-23T18:40:20.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.443+0000 7f9963fff640 1 -- 172.21.15.50:0/1940285394 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) ==== 74+0+23132 (secure 0 0 0) 0x7f997009d110 con 0x7f997c103b00 2024-04-23T18:40:20.092 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:40:20.140 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":128,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:40:16.232694+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":127,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/1479623929":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/3248139242":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6800/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/2484340243":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/444338023":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:40:20.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.445+0000 7f9981319640 1 -- 172.21.15.50:0/1940285394 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99640a3e60 msgr2=0x7f99640a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:20.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.445+0000 7f9981319640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99640a3e60 0x7f99640a6320 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f996c005b60 tx=0x7f996c005af0 comp rx=0 tx=0).stop 2024-04-23T18:40:20.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.445+0000 7f9981319640 1 -- 172.21.15.50:0/1940285394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f997c103b00 msgr2=0x7f997c101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.445+0000 7f9981319640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f997c103b00 0x7f997c101370 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f997000e9c0 tx=0x7f997000ee80 comp rx=0 tx=0).stop 2024-04-23T18:40:20.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.445+0000 7f9981319640 1 -- 172.21.15.50:0/1940285394 shutdown_connections 2024-04-23T18:40:20.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.445+0000 7f9981319640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f99640a3e60 0x7f99640a6320 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:20.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.446+0000 7f9981319640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f997c10c520 0x7f997c1018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:20.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.446+0000 7f9981319640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f997c103b00 0x7f997c101370 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:20.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.446+0000 7f9981319640 1 --2- 172.21.15.50:0/1940285394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f997c1031e0 0x7f997c100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:20.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.446+0000 7f9981319640 1 -- 172.21.15.50:0/1940285394 >> 172.21.15.50:0/1940285394 conn(0x7f997c077f50 msgr2=0x7f997c0ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:20.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.446+0000 7f9981319640 1 -- 172.21.15.50:0/1940285394 shutdown_connections 2024-04-23T18:40:20.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:19.446+0000 7f9981319640 1 -- 172.21.15.50:0/1940285394 wait complete. 2024-04-23T18:40:20.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:19 smithi050 ceph-mon[29187]: from='client.18162 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T18:40:20.464 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:12:02.493647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:16:31.679148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:16:32.269478+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 39, 'snap_epoch': 127, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:40:20.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:40:20.516 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:20 smithi050 ceph-mon[29187]: pgmap v695: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 4.1 MiB/s wr, 124 op/s 2024-04-23T18:40:20.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:20 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1940285394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:40:20.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:20 smithi177 ceph-mon[32308]: pgmap v695: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 4.1 MiB/s wr, 124 op/s 2024-04-23T18:40:20.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:20 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1940285394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:40:20.578 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:20 smithi188 ceph-mon[32694]: pgmap v695: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 4.1 MiB/s wr, 124 op/s 2024-04-23T18:40:20.587 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:20 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1940285394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:40:20.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.617+0000 7f4955587640 1 -- 172.21.15.50:0/2344573930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4950106160 msgr2=0x7f495010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:20.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.617+0000 7f4955587640 1 --2- 172.21.15.50:0/2344573930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4950106160 0x7f495010dd30 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f4948009900 tx=0x7f494802a410 comp rx=0 tx=0).stop 2024-04-23T18:40:20.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.617+0000 7f4955587640 1 -- 172.21.15.50:0/2344573930 shutdown_connections 2024-04-23T18:40:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.617+0000 7f4955587640 1 --2- 172.21.15.50:0/2344573930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4950106160 0x7f495010dd30 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:20.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.617+0000 7f4955587640 1 --2- 172.21.15.50:0/2344573930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4950101a40 0x7f4950105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:20.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.617+0000 7f4955587640 1 --2- 172.21.15.50:0/2344573930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4950101090 0x7f4950101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.617+0000 7f4955587640 1 -- 172.21.15.50:0/2344573930 >> 172.21.15.50:0/2344573930 conn(0x7f4950077f50 msgr2=0x7f4950078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:20.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.617+0000 7f4955587640 1 -- 172.21.15.50:0/2344573930 shutdown_connections 2024-04-23T18:40:20.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.617+0000 7f4955587640 1 -- 172.21.15.50:0/2344573930 wait complete. 2024-04-23T18:40:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.617+0000 7f4955587640 1 Processor -- start 2024-04-23T18:40:20.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.617+0000 7f4955587640 1 -- start start 2024-04-23T18:40:20.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f4955587640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4950101090 0x7f4950199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:20.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f4955587640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4950106160 0x7f4950199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:20.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f4955587640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f495019a050 0x7f49501a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:20.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f4955587640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f49501110d0 con 0x7f4950106160 2024-04-23T18:40:20.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f4955587640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4950110f50 con 0x7f4950101090 2024-04-23T18:40:20.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f4955587640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4950111250 con 0x7f495019a050 2024-04-23T18:40:20.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494e7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4950106160 0x7f4950199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:20.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494e7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4950106160 0x7f4950199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51282/0 (socket says 172.21.15.50:51282) 2024-04-23T18:40:20.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494e7fc640 1 -- 172.21.15.50:0/3266127882 learned_addr learned my addr 172.21.15.50:0/3266127882 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:21.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494effd640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4950101090 0x7f4950199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:21.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494f7fe640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f495019a050 0x7f49501a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:21.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494e7fc640 1 -- 172.21.15.50:0/3266127882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f495019a050 msgr2=0x7f49501a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:21.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494e7fc640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f495019a050 0x7f49501a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:21.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494e7fc640 1 -- 172.21.15.50:0/3266127882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4950101090 msgr2=0x7f4950199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:21.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494e7fc640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4950101090 0x7f4950199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:21.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494e7fc640 1 -- 172.21.15.50:0/3266127882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49501a4dd0 con 0x7f4950106160 2024-04-23T18:40:21.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494f7fe640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f495019a050 0x7f49501a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:40:21.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494effd640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4950101090 0x7f4950199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:40:21.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.618+0000 7f494e7fc640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4950106160 0x7f4950199a70 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f494400d740 tx=0x7f494400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.619+0000 7f4937fff640 1 -- 172.21.15.50:0/3266127882 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4944017070 con 0x7f4950106160 2024-04-23T18:40:21.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.619+0000 7f4937fff640 1 -- 172.21.15.50:0/3266127882 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f494400de50 con 0x7f4950106160 2024-04-23T18:40:21.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.619+0000 7f4955587640 1 -- 172.21.15.50:0/3266127882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49501a50c0 con 0x7f4950106160 2024-04-23T18:40:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.619+0000 7f4937fff640 1 -- 172.21.15.50:0/3266127882 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4944004ce0 con 0x7f4950106160 2024-04-23T18:40:21.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.619+0000 7f4955587640 1 -- 172.21.15.50:0/3266127882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49501a55c0 con 0x7f4950106160 2024-04-23T18:40:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.620+0000 7f4955587640 1 -- 172.21.15.50:0/3266127882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4950068d70 con 0x7f4950106160 2024-04-23T18:40:21.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.621+0000 7f4937fff640 1 -- 172.21.15.50:0/3266127882 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4944004e80 con 0x7f4950106160 2024-04-23T18:40:21.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.621+0000 7f4937fff640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f49380a3d20 0x7f49380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:21.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.622+0000 7f4937fff640 1 -- 172.21.15.50:0/3266127882 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8542+0+0 (secure 0 0 0) 0x7f49440d0240 con 0x7f4950106160 2024-04-23T18:40:21.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.622+0000 7f494effd640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f49380a3d20 0x7f49380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:21.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.622+0000 7f494effd640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f49380a3d20 0x7f49380a61e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4940005fd0 tx=0x7f4940009520 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:21.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.626+0000 7f4937fff640 1 -- 172.21.15.50:0/3266127882 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f494409c870 con 0x7f4950106160 2024-04-23T18:40:21.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.817+0000 7f4955587640 1 -- 172.21.15.50:0/3266127882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4950109390 con 0x7f4950106160 2024-04-23T18:40:21.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.818+0000 7f4937fff640 1 -- 172.21.15.50:0/3266127882 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v128) ==== 93+0+10 (secure 0 0 0) 0x7f494409c170 con 0x7f4950106160 2024-04-23T18:40:21.588 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-23T18:40:21.599 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:21 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3266127882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:40:21.650 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:21 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3266127882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:40:21.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.820+0000 7f4955587640 1 -- 172.21.15.50:0/3266127882 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f49380a3d20 msgr2=0x7f49380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:21.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.820+0000 7f4955587640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f49380a3d20 0x7f49380a61e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4940005fd0 tx=0x7f4940009520 comp rx=0 tx=0).stop 2024-04-23T18:40:21.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.820+0000 7f4955587640 1 -- 172.21.15.50:0/3266127882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4950106160 msgr2=0x7f4950199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:21.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.820+0000 7f4955587640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4950106160 0x7f4950199a70 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f494400d740 tx=0x7f494400dc00 comp rx=0 tx=0).stop 2024-04-23T18:40:21.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.820+0000 7f4955587640 1 -- 172.21.15.50:0/3266127882 shutdown_connections 2024-04-23T18:40:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.820+0000 7f4955587640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f49380a3d20 0x7f49380a61e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:21.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.820+0000 7f4955587640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f495019a050 0x7f49501a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:21.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.820+0000 7f4955587640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4950106160 0x7f4950199a70 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.820+0000 7f4955587640 1 --2- 172.21.15.50:0/3266127882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4950101090 0x7f4950199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:21.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.820+0000 7f4955587640 1 -- 172.21.15.50:0/3266127882 >> 172.21.15.50:0/3266127882 conn(0x7f4950077f50 msgr2=0x7f49500ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:21.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.820+0000 7f4955587640 1 -- 172.21.15.50:0/3266127882 shutdown_connections 2024-04-23T18:40:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:20.820+0000 7f4955587640 1 -- 172.21.15.50:0/3266127882 wait complete. 2024-04-23T18:40:21.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:40:21.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:21 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3266127882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:40:22.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.016+0000 7f7e80b29640 1 -- 172.21.15.50:0/3185323142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e7c10c500 msgr2=0x7f7e7c10e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:22.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.016+0000 7f7e80b29640 1 --2- 172.21.15.50:0/3185323142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e7c10c500 0x7f7e7c10e9a0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009ca0 tx=0x7f7e7402a9e0 comp rx=0 tx=0).stop 2024-04-23T18:40:22.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.016+0000 7f7e80b29640 1 -- 172.21.15.50:0/3185323142 shutdown_connections 2024-04-23T18:40:22.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.016+0000 7f7e80b29640 1 --2- 172.21.15.50:0/3185323142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e7c10c500 0x7f7e7c10e9a0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:22.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.016+0000 7f7e80b29640 1 --2- 172.21.15.50:0/3185323142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e7c103ae0 0x7f7e7c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:22.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.016+0000 7f7e80b29640 1 --2- 172.21.15.50:0/3185323142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e7c1031c0 0x7f7e7c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:22.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.016+0000 7f7e80b29640 1 -- 172.21.15.50:0/3185323142 >> 172.21.15.50:0/3185323142 conn(0x7f7e7c078000 msgr2=0x7f7e7c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.016+0000 7f7e80b29640 1 -- 172.21.15.50:0/3185323142 shutdown_connections 2024-04-23T18:40:22.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.016+0000 7f7e80b29640 1 -- 172.21.15.50:0/3185323142 wait complete. 2024-04-23T18:40:22.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e80b29640 1 Processor -- start 2024-04-23T18:40:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e80b29640 1 -- start start 2024-04-23T18:40:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e80b29640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e7c1031c0 0x7f7e7c100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e80b29640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e7c103ae0 0x7f7e7c101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:22.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e80b29640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e7c10c500 0x7f7e7c101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:22.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e80b29640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c112190 con 0x7f7e7c10c500 2024-04-23T18:40:22.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e80b29640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c112010 con 0x7f7e7c1031c0 2024-04-23T18:40:22.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e80b29640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c112310 con 0x7f7e7c103ae0 2024-04-23T18:40:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e7ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e7c10c500 0x7f7e7c101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:22.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e7a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e7c1031c0 0x7f7e7c100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:22.478 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e7ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e7c10c500 0x7f7e7c101890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56526/0 (socket says 172.21.15.50:56526) 2024-04-23T18:40:22.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e7ad9d640 1 -- 172.21.15.50:0/1539677758 learned_addr learned my addr 172.21.15.50:0/1539677758 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:22.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e79d9b640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e7c103ae0 0x7f7e7c101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e7ad9d640 1 -- 172.21.15.50:0/1539677758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e7c103ae0 msgr2=0x7f7e7c101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:22.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e7ad9d640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e7c103ae0 0x7f7e7c101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:22.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e7ad9d640 1 -- 172.21.15.50:0/1539677758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e7c1031c0 msgr2=0x7f7e7c100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e7ad9d640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e7c1031c0 0x7f7e7c100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:22.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.017+0000 7f7e7ad9d640 1 -- 172.21.15.50:0/1539677758 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c1a6bb0 con 0x7f7e7c10c500 2024-04-23T18:40:22.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.018+0000 7f7e7ad9d640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e7c10c500 0x7f7e7c101890 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f7e74006010 tx=0x7f7e7402fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:22.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.018+0000 7f7e6b7fe640 1 -- 172.21.15.50:0/1539677758 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e74033040 con 0x7f7e7c10c500 2024-04-23T18:40:22.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.018+0000 7f7e6b7fe640 1 -- 172.21.15.50:0/1539677758 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e7402e7a0 con 0x7f7e7c10c500 2024-04-23T18:40:22.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.018+0000 7f7e6b7fe640 1 -- 172.21.15.50:0/1539677758 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e74043720 con 0x7f7e7c10c500 2024-04-23T18:40:22.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.018+0000 7f7e80b29640 1 -- 172.21.15.50:0/1539677758 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7c1a6e40 con 0x7f7e7c10c500 2024-04-23T18:40:22.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.018+0000 7f7e80b29640 1 -- 172.21.15.50:0/1539677758 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7c1a72a0 con 0x7f7e7c10c500 2024-04-23T18:40:22.748 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:22 smithi188 ceph-mon[32694]: pgmap v696: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 4.1 MiB/s wr, 123 op/s 2024-04-23T18:40:22.756 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:22 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1539677758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:40:22.765 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:22 smithi177 ceph-mon[32308]: pgmap v696: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 4.1 MiB/s wr, 123 op/s 2024-04-23T18:40:22.830 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:22 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1539677758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:40:22.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.019+0000 7f7e80b29640 1 -- 172.21.15.50:0/1539677758 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e7c068d70 con 0x7f7e7c10c500 2024-04-23T18:40:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.020+0000 7f7e6b7fe640 1 -- 172.21.15.50:0/1539677758 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7e74036070 con 0x7f7e7c10c500 2024-04-23T18:40:22.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.021+0000 7f7e6b7fe640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e640a3d20 0x7f7e640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:22.895 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.021+0000 7f7e6b7fe640 1 -- 172.21.15.50:0/1539677758 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8542+0+0 (secure 0 0 0) 0x7f7e740f1d50 con 0x7f7e7c10c500 2024-04-23T18:40:22.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.021+0000 7f7e7a59c640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e640a3d20 0x7f7e640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:22.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.022+0000 7f7e7a59c640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e640a3d20 0x7f7e640a61e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00bbe0 tx=0x7f7e6c00b390 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:22.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.025+0000 7f7e6b7fe640 1 -- 172.21.15.50:0/1539677758 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e7403e070 con 0x7f7e7c10c500 2024-04-23T18:40:22.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.211+0000 7f7e80b29640 1 -- 172.21.15.50:0/1539677758 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7e7c02d5c0 con 0x7f7e7c10c500 2024-04-23T18:40:22.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.211+0000 7f7e6b7fe640 1 -- 172.21.15.50:0/1539677758 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v128) ==== 104+0+11 (secure 0 0 0) 0x7f7e740098b0 con 0x7f7e7c10c500 2024-04-23T18:40:22.996 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2024-04-23T18:40:23.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.213+0000 7f7e80b29640 1 -- 172.21.15.50:0/1539677758 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e640a3d20 msgr2=0x7f7e640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:23.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.213+0000 7f7e80b29640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e640a3d20 0x7f7e640a61e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00bbe0 tx=0x7f7e6c00b390 comp rx=0 tx=0).stop 2024-04-23T18:40:23.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.213+0000 7f7e80b29640 1 -- 172.21.15.50:0/1539677758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e7c10c500 msgr2=0x7f7e7c101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:23.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.213+0000 7f7e80b29640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e7c10c500 0x7f7e7c101890 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f7e74006010 tx=0x7f7e7402fbe0 comp rx=0 tx=0).stop 2024-04-23T18:40:23.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.213+0000 7f7e80b29640 1 -- 172.21.15.50:0/1539677758 shutdown_connections 2024-04-23T18:40:23.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.213+0000 7f7e80b29640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e640a3d20 0x7f7e640a61e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:23.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.213+0000 7f7e80b29640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e7c10c500 0x7f7e7c101890 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:23.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.213+0000 7f7e80b29640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e7c103ae0 0x7f7e7c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:23.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.213+0000 7f7e80b29640 1 --2- 172.21.15.50:0/1539677758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e7c1031c0 0x7f7e7c100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:23.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.213+0000 7f7e80b29640 1 -- 172.21.15.50:0/1539677758 >> 172.21.15.50:0/1539677758 conn(0x7f7e7c078000 msgr2=0x7f7e7c0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:23.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.213+0000 7f7e80b29640 1 -- 172.21.15.50:0/1539677758 shutdown_connections 2024-04-23T18:40:23.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:22.213+0000 7f7e80b29640 1 -- 172.21.15.50:0/1539677758 wait complete. 2024-04-23T18:40:23.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:40:23.247 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:22 smithi050 ceph-mon[29187]: pgmap v696: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 4.1 MiB/s wr, 123 op/s 2024-04-23T18:40:23.255 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:22 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1539677758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:40:23.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.376+0000 7f124de05640 1 -- 172.21.15.50:0/1559008987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1248102bc0 msgr2=0x7f1248106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:23.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.376+0000 7f124de05640 1 --2- 172.21.15.50:0/1559008987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1248102bc0 0x7f1248106c00 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f1230009900 tx=0x7f123002a410 comp rx=0 tx=0).stop 2024-04-23T18:40:23.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.376+0000 7f124de05640 1 -- 172.21.15.50:0/1559008987 shutdown_connections 2024-04-23T18:40:23.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.376+0000 7f124de05640 1 --2- 172.21.15.50:0/1559008987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12481072e0 0x7f124810eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.376+0000 7f124de05640 1 --2- 172.21.15.50:0/1559008987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1248102bc0 0x7f1248106c00 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:23.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.376+0000 7f124de05640 1 --2- 172.21.15.50:0/1559008987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1248102210 0x7f12481025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:23.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.376+0000 7f124de05640 1 -- 172.21.15.50:0/1559008987 >> 172.21.15.50:0/1559008987 conn(0x7f12480fdab0 msgr2=0x7f12480ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:23.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.377+0000 7f124de05640 1 -- 172.21.15.50:0/1559008987 shutdown_connections 2024-04-23T18:40:23.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.377+0000 7f124de05640 1 -- 172.21.15.50:0/1559008987 wait complete. 2024-04-23T18:40:23.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.377+0000 7f124de05640 1 Processor -- start 2024-04-23T18:40:23.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.377+0000 7f124de05640 1 -- start start 2024-04-23T18:40:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.377+0000 7f124de05640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1248102210 0x7f124819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:23.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.377+0000 7f124de05640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12481072e0 0x7f124819e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:23.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.377+0000 7f124de05640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1248198670 0x7f1248198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:23.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.377+0000 7f124de05640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1248112220 con 0x7f1248102210 2024-04-23T18:40:23.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.377+0000 7f124de05640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f12481120a0 con 0x7f1248198670 2024-04-23T18:40:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.377+0000 7f124de05640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f12481123a0 con 0x7f12481072e0 2024-04-23T18:40:23.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f12477fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1248102210 0x7f124819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:23.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f12477fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1248102210 0x7f124819e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56544/0 (socket says 172.21.15.50:56544) 2024-04-23T18:40:23.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f12477fe640 1 -- 172.21.15.50:0/1417922044 learned_addr learned my addr 172.21.15.50:0/1417922044 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:23.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f1246ffd640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12481072e0 0x7f124819e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:23.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f1247fff640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1248198670 0x7f1248198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:23.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f12477fe640 1 -- 172.21.15.50:0/1417922044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12481072e0 msgr2=0x7f124819e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:23.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f12477fe640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12481072e0 0x7f124819e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:23.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f12477fe640 1 -- 172.21.15.50:0/1417922044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1248198670 msgr2=0x7f1248198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:23.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f12477fe640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1248198670 0x7f1248198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:23.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f12477fe640 1 -- 172.21.15.50:0/1417922044 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1248199190 con 0x7f1248102210 2024-04-23T18:40:23.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f1247fff640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1248198670 0x7f1248198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:40:23.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f12477fe640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1248102210 0x7f124819e470 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f123800ebc0 tx=0x7f123800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f1244ff9640 1 -- 172.21.15.50:0/1417922044 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1238010630 con 0x7f1248102210 2024-04-23T18:40:23.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f1244ff9640 1 -- 172.21.15.50:0/1417922044 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1238010c80 con 0x7f1248102210 2024-04-23T18:40:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f124de05640 1 -- 172.21.15.50:0/1417922044 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1248199480 con 0x7f1248102210 2024-04-23T18:40:23.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f1244ff9640 1 -- 172.21.15.50:0/1417922044 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1238018b60 con 0x7f1248102210 2024-04-23T18:40:23.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.378+0000 7f124de05640 1 -- 172.21.15.50:0/1417922044 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12481a5680 con 0x7f1248102210 2024-04-23T18:40:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.379+0000 7f124de05640 1 -- 172.21.15.50:0/1417922044 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1248068d70 con 0x7f1248102210 2024-04-23T18:40:24.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.380+0000 7f1244ff9640 1 -- 172.21.15.50:0/1417922044 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1238018d00 con 0x7f1248102210 2024-04-23T18:40:24.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.381+0000 7f1244ff9640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f12340a3d20 0x7f12340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:24.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.381+0000 7f1244ff9640 1 -- 172.21.15.50:0/1417922044 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8542+0+0 (secure 0 0 0) 0x7f12380d0ea0 con 0x7f1248102210 2024-04-23T18:40:24.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.381+0000 7f1246ffd640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f12340a3d20 0x7f12340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:24.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.382+0000 7f1246ffd640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f12340a3d20 0x7f12340a61e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1230000c00 tx=0x7f1230031400 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:24.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.386+0000 7f1244ff9640 1 -- 172.21.15.50:0/1417922044 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f123809d530 con 0x7f1248102210 2024-04-23T18:40:24.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.575+0000 7f124de05640 1 -- 172.21.15.50:0/1417922044 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f124810a540 con 0x7f1248102210 2024-04-23T18:40:24.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.576+0000 7f1244ff9640 1 -- 172.21.15.50:0/1417922044 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v128) ==== 100+0+11 (secure 0 0 0) 0x7f123809ce30 con 0x7f1248102210 2024-04-23T18:40:24.199 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2024-04-23T18:40:24.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.578+0000 7f124de05640 1 -- 172.21.15.50:0/1417922044 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f12340a3d20 msgr2=0x7f12340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:24.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.578+0000 7f124de05640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f12340a3d20 0x7f12340a61e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1230000c00 tx=0x7f1230031400 comp rx=0 tx=0).stop 2024-04-23T18:40:24.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.578+0000 7f124de05640 1 -- 172.21.15.50:0/1417922044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1248102210 msgr2=0x7f124819e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:24.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.578+0000 7f124de05640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1248102210 0x7f124819e470 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f123800ebc0 tx=0x7f123800c6a0 comp rx=0 tx=0).stop 2024-04-23T18:40:24.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.578+0000 7f124de05640 1 -- 172.21.15.50:0/1417922044 shutdown_connections 2024-04-23T18:40:24.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.578+0000 7f124de05640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f12340a3d20 0x7f12340a61e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:24.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.578+0000 7f124de05640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1248198670 0x7f1248198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:24.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.578+0000 7f124de05640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12481072e0 0x7f124819e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:24.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.578+0000 7f124de05640 1 --2- 172.21.15.50:0/1417922044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1248102210 0x7f124819e470 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:24.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.578+0000 7f124de05640 1 -- 172.21.15.50:0/1417922044 >> 172.21.15.50:0/1417922044 conn(0x7f12480fdab0 msgr2=0x7f12480fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:24.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.578+0000 7f124de05640 1 -- 172.21.15.50:0/1417922044 shutdown_connections 2024-04-23T18:40:24.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:23.578+0000 7f124de05640 1 -- 172.21.15.50:0/1417922044 wait complete. 2024-04-23T18:40:24.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-23T18:40:24.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:24 smithi177 ceph-mon[32308]: pgmap v697: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 5.4 MiB/s wr, 174 op/s 2024-04-23T18:40:24.545 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:24 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1417922044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:40:24.636 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:24 smithi188 ceph-mon[32694]: pgmap v697: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 5.4 MiB/s wr, 174 op/s 2024-04-23T18:40:24.646 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:24 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1417922044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:40:24.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.591+0000 7fe5569f7640 1 -- 172.21.15.50:0/3827485261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550106190 msgr2=0x7fe55010dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:24.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.591+0000 7fe5569f7640 1 --2- 172.21.15.50:0/3827485261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550106190 0x7fe55010dd60 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fe548009ca0 tx=0x7fe54802a8c0 comp rx=0 tx=0).stop 2024-04-23T18:40:24.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.591+0000 7fe5569f7640 1 -- 172.21.15.50:0/3827485261 shutdown_connections 2024-04-23T18:40:24.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.591+0000 7fe5569f7640 1 --2- 172.21.15.50:0/3827485261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550106190 0x7fe55010dd60 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.591+0000 7fe5569f7640 1 --2- 172.21.15.50:0/3827485261 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550101ac0 0x7fe550105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:24.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.591+0000 7fe5569f7640 1 --2- 172.21.15.50:0/3827485261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe550101110 0x7fe5501014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:24.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.591+0000 7fe5569f7640 1 -- 172.21.15.50:0/3827485261 >> 172.21.15.50:0/3827485261 conn(0x7fe550077fb0 msgr2=0x7fe5500783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:40:24.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.591+0000 7fe5569f7640 1 -- 172.21.15.50:0/3827485261 shutdown_connections 2024-04-23T18:40:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.591+0000 7fe5569f7640 1 -- 172.21.15.50:0/3827485261 wait complete. 2024-04-23T18:40:24.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.592+0000 7fe5569f7640 1 Processor -- start 2024-04-23T18:40:24.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.592+0000 7fe5569f7640 1 -- start start 2024-04-23T18:40:24.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.592+0000 7fe5569f7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550101110 0x7fe550198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:24.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.592+0000 7fe5569f7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550106190 0x7fe550198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:24.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.592+0000 7fe5569f7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55019cff0 0x7fe55019d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.592+0000 7fe5569f7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501110e0 con 0x7fe550106190 2024-04-23T18:40:24.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.592+0000 7fe5569f7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe550110f60 con 0x7fe550101110 2024-04-23T18:40:24.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.592+0000 7fe5569f7640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe550111260 con 0x7fe55019cff0 2024-04-23T18:40:25.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.592+0000 7fe54f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550106190 0x7fe550198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:25.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.592+0000 7fe54f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550106190 0x7fe550198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56560/0 (socket says 172.21.15.50:56560) 2024-04-23T18:40:25.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.592+0000 7fe54f7fe640 1 -- 172.21.15.50:0/2793906102 learned_addr learned my addr 172.21.15.50:0/2793906102 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:40:25.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe54ffff640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550101110 0x7fe550198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:25.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe554f94640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55019cff0 0x7fe55019d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:25.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe54f7fe640 1 -- 172.21.15.50:0/2793906102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55019cff0 msgr2=0x7fe55019d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:25.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55019cff0 0x7fe55019d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:25.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe54f7fe640 1 -- 172.21.15.50:0/2793906102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550101110 msgr2=0x7fe550198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:40:25.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550101110 0x7fe550198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:40:25.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe54f7fe640 1 -- 172.21.15.50:0/2793906102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55019dae0 con 0x7fe550106190 2024-04-23T18:40:25.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe54ffff640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550101110 0x7fe550198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:40:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe54f7fe640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550106190 0x7fe550198c70 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fe54400d770 tx=0x7fe54400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:25.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe54d7fa640 1 -- 172.21.15.50:0/2793906102 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544017070 con 0x7fe550106190 2024-04-23T18:40:25.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe5569f7640 1 -- 172.21.15.50:0/2793906102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe55019ddd0 con 0x7fe550106190 2024-04-23T18:40:25.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe54d7fa640 1 -- 172.21.15.50:0/2793906102 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe544009c30 con 0x7fe550106190 2024-04-23T18:40:25.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe54d7fa640 1 -- 172.21.15.50:0/2793906102 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54400f620 con 0x7fe550106190 2024-04-23T18:40:25.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.593+0000 7fe5569f7640 1 -- 172.21.15.50:0/2793906102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5501a5610 con 0x7fe550106190 2024-04-23T18:40:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.594+0000 7fe5569f7640 1 -- 172.21.15.50:0/2793906102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5500691c0 con 0x7fe550106190 2024-04-23T18:40:25.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.595+0000 7fe54d7fa640 1 -- 172.21.15.50:0/2793906102 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe54401a070 con 0x7fe550106190 2024-04-23T18:40:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.596+0000 7fe54d7fa640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe5380a3d20 0x7fe5380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:40:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.596+0000 7fe54ffff640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe5380a3d20 0x7fe5380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:40:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.596+0000 7fe54d7fa640 1 -- 172.21.15.50:0/2793906102 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8542+0+0 (secure 0 0 0) 0x7fe5440d0440 con 0x7fe550106190 2024-04-23T18:40:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.597+0000 7fe54ffff640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe5380a3d20 0x7fe5380a61e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe5400058c0 tx=0x7fe540005850 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:40:25.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.601+0000 7fe54d7fa640 1 -- 172.21.15.50:0/2793906102 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe544009de0 con 0x7fe550106190 2024-04-23T18:40:25.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:24 smithi050 ceph-mon[29187]: pgmap v697: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 5.4 MiB/s wr, 174 op/s 2024-04-23T18:40:25.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:24 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1417922044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:40:25.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.840+0000 7fe5569f7640 1 -- 172.21.15.50:0/2793906102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fe550101ac0 con 0x7fe550106190 2024-04-23T18:40:25.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.848+0000 7fe54d7fa640 1 -- 172.21.15.50:0/2793906102 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) ==== 81+0+746060 (secure 0 0 0) 0x7fe54409ca70 con 0x7fe550106190 2024-04-23T18:40:25.454 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:40:25.464 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:40:25.473 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 27, 2024-04-23T18:40:25.481 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:25.534 INFO:teuthology.orchestra.run.smithi050.stdout: "active_gid": 25018, 2024-04-23T18:40:25.549 INFO:teuthology.orchestra.run.smithi050.stdout: "active_name": "x", 2024-04-23T18:40:25.557 INFO:teuthology.orchestra.run.smithi050.stdout: "active_addrs": { 2024-04-23T18:40:25.566 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-23T18:40:25.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:40:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-23T18:40:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6800", 2024-04-23T18:40:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4210816660 2024-04-23T18:40:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:40:25.689 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-23T18:40:25.749 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6801", 2024-04-23T18:40:25.757 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4210816660 2024-04-23T18:40:25.766 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:40:25.774 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-23T18:40:25.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:25.819 INFO:teuthology.orchestra.run.smithi050.stdout: "active_addr": "172.21.15.50:6801/4210816660", 2024-04-23T18:40:25.900 INFO:teuthology.orchestra.run.smithi050.stdout: "active_change": "2024-04-23T18:18:30.504351+0000", 2024-04-23T18:40:25.909 INFO:teuthology.orchestra.run.smithi050.stdout: "active_mgr_features": 4540701547738038271, 2024-04-23T18:40:25.917 INFO:teuthology.orchestra.run.smithi050.stdout: "available": true, 2024-04-23T18:40:25.926 INFO:teuthology.orchestra.run.smithi050.stdout: "standbys": [ 2024-04-23T18:40:25.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:40:25.942 INFO:teuthology.orchestra.run.smithi050.stdout: "gid": 24932, 2024-04-23T18:40:25.950 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "y", 2024-04-23T18:40:25.958 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_features": 4540701547738038271, 2024-04-23T18:40:25.965 INFO:teuthology.orchestra.run.smithi050.stdout: "available_modules": [ 2024-04-23T18:40:25.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:40:26.030 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "alerts", 2024-04-23T18:40:26.087 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:40:26.118 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:40:26.126 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:40:26.135 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:40:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:40:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:40:26.195 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:26.203 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:26.211 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:40:26.219 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:26.226 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:26.233 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:26.241 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T18:40:26.248 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:26.263 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:26.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:26.371 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:40:26.406 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:40:26.413 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:26.420 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:26.428 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:26.435 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:26.516 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:26.524 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:26.531 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:40:26.538 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:40:26.546 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:40:26.553 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:40:26.561 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:40:26.571 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:40:26.579 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:40:26.585 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:40:26.680 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:26.730 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:26.739 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:26.747 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:26.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:26.762 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:40:26.769 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:40:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:26.803 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:26.811 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:26.818 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:26.826 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:26.834 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:26.842 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:26.850 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:26.859 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:26.869 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:26.931 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:26.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:27.052 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:40:27.060 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:40:27.072 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:27.081 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:27.089 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:27.158 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:40:27.167 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:27.175 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:27.184 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:40:27.193 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:40:27.202 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:40:27.213 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:40:27.224 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:40:27.232 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:40:27.241 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:40:27.312 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:40:27.369 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:27.377 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:27.386 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:27.395 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:27.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:27.413 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:40:27.432 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:40:27.442 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:27.451 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:27.463 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:27.472 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:27.480 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:27.490 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:27.499 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:27.508 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:27.517 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:27.583 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:27.669 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:27.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:27.687 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_destination": { 2024-04-23T18:40:27.696 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_destination", 2024-04-23T18:40:27.705 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:27.714 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:27.799 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:27.808 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:27.824 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:27.833 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:27.841 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Email address to send alerts to", 2024-04-23T18:40:27.851 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:27.859 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:27.868 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:27.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:27.941 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_from_name": { 2024-04-23T18:40:28.002 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_from_name", 2024-04-23T18:40:28.011 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:28.019 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:28.028 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:28.036 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "Ceph", 2024-04-23T18:40:28.044 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:28.061 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:28.070 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:28.078 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Email From: name", 2024-04-23T18:40:28.085 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:28.094 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:28.103 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:28.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:28.121 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_host": { 2024-04-23T18:40:28.129 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_host", 2024-04-23T18:40:28.138 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:28.185 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:28.237 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:28.259 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:28.276 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:28.284 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:28.295 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "SMTP server", 2024-04-23T18:40:28.366 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:28.373 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_password": { 2024-04-23T18:40:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_password", 2024-04-23T18:40:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:28.457 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:28.560 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:28.611 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:28.618 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:28.626 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Password to authenticate with", 2024-04-23T18:40:28.633 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:28.640 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:28.647 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:28.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:28.701 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_port": { 2024-04-23T18:40:28.709 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_port", 2024-04-23T18:40:28.715 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:40:28.722 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:28.729 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:28.741 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "465", 2024-04-23T18:40:28.749 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:28.757 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:28.764 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "SMTP port", 2024-04-23T18:40:28.899 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:28.907 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:28.914 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:28.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:28.929 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_sender": { 2024-04-23T18:40:28.936 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_sender", 2024-04-23T18:40:29.023 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:29.030 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:29.037 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:29.054 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:29.065 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:29.074 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:29.083 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "SMTP envelope sender", 2024-04-23T18:40:29.092 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:29.100 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:29.170 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:29.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_ssl": { 2024-04-23T18:40:29.249 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_ssl", 2024-04-23T18:40:29.258 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:29.270 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:29.279 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:29.294 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:40:29.303 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:29.334 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T18:40:29.342 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:29.351 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:29.359 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:29.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:29.376 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_user": { 2024-04-23T18:40:29.460 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_user", 2024-04-23T18:40:29.487 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:29.551 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:29.559 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:29.567 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:29.576 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:29.584 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:29.657 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:29.666 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "User to authenticate as", 2024-04-23T18:40:29.675 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:29.692 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:29.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:40:29.709 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:40:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:40:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "balancer", 2024-04-23T18:40:29.797 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:40:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:40:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:40:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: "active": { 2024-04-23T18:40:29.885 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "active", 2024-04-23T18:40:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:29.902 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:29.928 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:29.937 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:40:29.947 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:29.956 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:29.965 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:29.973 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T18:40:29.982 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:29.992 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:30.000 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:30.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "begin_time": { 2024-04-23T18:40:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "begin_time", 2024-04-23T18:40:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:30.153 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0000", 2024-04-23T18:40:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:30.246 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T18:40:30.252 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:40:30.259 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:30.266 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:30.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:30.279 INFO:teuthology.orchestra.run.smithi050.stdout: "begin_weekday": { 2024-04-23T18:40:30.285 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "begin_weekday", 2024-04-23T18:40:30.292 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:40:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:40:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "0", 2024-04-23T18:40:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "6", 2024-04-23T18:40:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T18:40:30.490 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:40:30.498 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:30.509 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:30.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:30.528 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_compat_max_iterations": { 2024-04-23T18:40:30.537 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crush_compat_max_iterations", 2024-04-23T18:40:30.546 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:40:30.555 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:30.564 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "25", 2024-04-23T18:40:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1", 2024-04-23T18:40:30.692 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "250", 2024-04-23T18:40:30.700 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:30.709 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T18:40:30.717 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:30.725 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:30.798 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:30.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_compat_metrics": { 2024-04-23T18:40:30.828 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crush_compat_metrics", 2024-04-23T18:40:30.837 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T18:40:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:30.912 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:30.982 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:31.036 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T18:40:31.045 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T18:40:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:31.062 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:31.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:31.079 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_compat_step": { 2024-04-23T18:40:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crush_compat_step", 2024-04-23T18:40:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:40:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0.5", 2024-04-23T18:40:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "0.001", 2024-04-23T18:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "0.999", 2024-04-23T18:40:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "aggressiveness of optimization", 2024-04-23T18:40:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T18:40:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:31.312 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:31.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:31.331 INFO:teuthology.orchestra.run.smithi050.stdout: "end_time": { 2024-04-23T18:40:31.339 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "end_time", 2024-04-23T18:40:31.347 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:31.354 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:31.440 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:31.447 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "2359", 2024-04-23T18:40:31.453 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:31.461 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:31.468 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:31.475 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T18:40:31.484 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:40:31.491 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:31.499 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:31.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:31.580 INFO:teuthology.orchestra.run.smithi050.stdout: "end_weekday": { 2024-04-23T18:40:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "end_weekday", 2024-04-23T18:40:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:40:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:40:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "0", 2024-04-23T18:40:31.752 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "6", 2024-04-23T18:40:31.764 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:31.778 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T18:40:31.787 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:40:31.795 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:31.803 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:31.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:31.839 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:40:31.847 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:40:31.856 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:31.864 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:31.933 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:31.967 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:31.978 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:31.987 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:31.996 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:40:32.004 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:40:32.131 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:40:32.142 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:40:32.152 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:40:32.161 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:40:32.170 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:40:32.178 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:40:32.187 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:32.195 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:32.204 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:32.213 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:32.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:40:32.359 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:40:32.369 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:32.378 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:32.428 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:32.438 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:32.465 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:32.475 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:32.484 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:32.497 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:32.507 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:32.516 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:32.524 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:32.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:32.543 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:40:32.552 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:40:32.607 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:32.668 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:32.754 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:32.763 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:40:32.772 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:32.780 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:32.792 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:40:32.865 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:40:32.874 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:40:32.882 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:40:32.891 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:40:32.941 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:40:32.950 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:40:32.959 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:40:32.968 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:32.977 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:32.985 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:33.040 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:33.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:33.143 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:40:33.161 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:40:33.168 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:33.176 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:33.183 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:33.228 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:33.235 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:33.243 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:33.250 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:33.300 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:33.307 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:33.314 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:33.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:33.327 INFO:teuthology.orchestra.run.smithi050.stdout: "min_score": { 2024-04-23T18:40:33.373 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "min_score", 2024-04-23T18:40:33.440 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:40:33.529 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:33.538 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:33.547 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:40:33.554 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:33.562 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:33.638 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:33.682 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T18:40:33.691 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:33.698 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:33.707 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:33.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:33.723 INFO:teuthology.orchestra.run.smithi050.stdout: "mode": { 2024-04-23T18:40:33.732 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mode", 2024-04-23T18:40:33.740 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:33.803 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:33.812 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:33.885 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "upmap", 2024-04-23T18:40:33.945 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:33.962 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:33.971 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:40:34.043 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-compat", 2024-04-23T18:40:34.053 INFO:teuthology.orchestra.run.smithi050.stdout: "none", 2024-04-23T18:40:34.145 INFO:teuthology.orchestra.run.smithi050.stdout: "read", 2024-04-23T18:40:34.154 INFO:teuthology.orchestra.run.smithi050.stdout: "upmap", 2024-04-23T18:40:34.163 INFO:teuthology.orchestra.run.smithi050.stdout: "upmap-read" 2024-04-23T18:40:34.172 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:40:34.180 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Balancer mode", 2024-04-23T18:40:34.188 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:34.197 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:34.205 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:34.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:34.222 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_ids": { 2024-04-23T18:40:34.279 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "pool_ids", 2024-04-23T18:40:34.333 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:34.397 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:34.406 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:34.485 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T18:40:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:40:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:40:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:40:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:34.627 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:40:34.755 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:34.763 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:34.772 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:34.780 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T18:40:34.789 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:34.854 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:34.863 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:34.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:34.878 INFO:teuthology.orchestra.run.smithi050.stdout: "upmap_max_deviation": { 2024-04-23T18:40:34.886 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "upmap_max_deviation", 2024-04-23T18:40:34.894 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:40:34.950 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:34.957 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:34.964 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5", 2024-04-23T18:40:35.024 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1", 2024-04-23T18:40:35.031 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:35.076 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:35.151 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T18:40:35.158 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T18:40:35.165 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:35.172 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:35.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:35.255 INFO:teuthology.orchestra.run.smithi050.stdout: "upmap_max_optimizations": { 2024-04-23T18:40:35.290 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "upmap_max_optimizations", 2024-04-23T18:40:35.298 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:40:35.305 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:35.312 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:35.320 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:40:35.327 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:35.334 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:35.340 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:35.347 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T18:40:35.406 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:35.462 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:35.531 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:40:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:40:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:40:35.573 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephadm", 2024-04-23T18:40:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:40:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:40:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:40:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "agent_down_multiplier": { 2024-04-23T18:40:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "agent_down_multiplier", 2024-04-23T18:40:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:40:35.705 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:35.713 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:35.767 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "3.0", 2024-04-23T18:40:35.775 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:35.829 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:35.897 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:35.972 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T18:40:35.979 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:35.985 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:35.993 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:36.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:36.122 INFO:teuthology.orchestra.run.smithi050.stdout: "agent_refresh_rate": { 2024-04-23T18:40:36.129 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "agent_refresh_rate", 2024-04-23T18:40:36.136 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:40:36.143 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:36.150 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:36.157 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "20", 2024-04-23T18:40:36.163 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:36.170 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:36.241 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:36.265 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T18:40:36.326 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:36.396 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:36.404 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:36.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:36.419 INFO:teuthology.orchestra.run.smithi050.stdout: "agent_starting_port": { 2024-04-23T18:40:36.426 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "agent_starting_port", 2024-04-23T18:40:36.495 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:40:36.502 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:36.510 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:36.517 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "4721", 2024-04-23T18:40:36.524 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:36.588 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:36.595 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:36.601 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T18:40:36.608 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:36.616 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:36.682 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:36.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:36.811 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_ptrace": { 2024-04-23T18:40:36.820 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "allow_ptrace", 2024-04-23T18:40:36.829 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:36.836 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:36.843 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T18:40:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T18:40:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "autotune_interval": { 2024-04-23T18:40:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "autotune_interval", 2024-04-23T18:40:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:40:37.158 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:37.167 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:37.191 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:40:37.200 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:37.220 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:37.323 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T18:40:37.331 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:37.340 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:37.349 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: "autotune_memory_target_ratio": { 2024-04-23T18:40:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T18:40:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:40:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:37.463 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:37.525 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0.7", 2024-04-23T18:40:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T18:40:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:37.692 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:37.703 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:37.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:37.720 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm_log_destination": { 2024-04-23T18:40:37.728 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephadm_log_destination", 2024-04-23T18:40:37.737 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:37.745 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:37.790 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:37.930 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:40:38.003 INFO:teuthology.orchestra.run.smithi050.stdout: "file", 2024-04-23T18:40:38.037 INFO:teuthology.orchestra.run.smithi050.stdout: "file,syslog", 2024-04-23T18:40:38.045 INFO:teuthology.orchestra.run.smithi050.stdout: "syslog" 2024-04-23T18:40:38.053 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:40:38.060 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T18:40:38.068 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:38.160 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:38.169 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:38.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:38.216 INFO:teuthology.orchestra.run.smithi050.stdout: "cgroups_split": { 2024-04-23T18:40:38.224 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cgroups_split", 2024-04-23T18:40:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:38.283 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:40:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T18:40:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:38.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: "config_checks_enabled": { 2024-04-23T18:40:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "config_checks_enabled", 2024-04-23T18:40:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:38.756 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:38.764 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T18:40:38.816 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:38.890 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:38.906 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:38.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:38.920 INFO:teuthology.orchestra.run.smithi050.stdout: "config_dashboard": { 2024-04-23T18:40:38.927 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "config_dashboard", 2024-04-23T18:40:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:40:39.019 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T18:40:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:39.101 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_alertmanager": { 2024-04-23T18:40:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_alertmanager", 2024-04-23T18:40:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T18:40:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Prometheus container image", 2024-04-23T18:40:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_base": { 2024-04-23T18:40:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_base", 2024-04-23T18:40:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:39.574 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:39.584 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:39.595 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T18:40:39.604 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:39.612 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:39.690 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Container image name, without the tag", 2024-04-23T18:40:39.699 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:39.707 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:39.716 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:39.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:39.734 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_elasticsearch": { 2024-04-23T18:40:39.744 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_elasticsearch", 2024-04-23T18:40:39.752 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:39.807 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:39.830 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:39.859 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T18:40:39.924 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:39.934 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:39.945 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:39.954 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "elasticsearch container image", 2024-04-23T18:40:39.964 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:39.972 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:40.041 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:40.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:40.110 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_grafana": { 2024-04-23T18:40:40.119 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_grafana", 2024-04-23T18:40:40.127 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:40.135 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:40.144 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:40.153 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T18:40:40.163 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:40.234 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:40.284 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:40.367 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Prometheus container image", 2024-04-23T18:40:40.386 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:40.435 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:40.440 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:40.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:40.542 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_haproxy": { 2024-04-23T18:40:40.551 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_haproxy", 2024-04-23T18:40:40.563 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:40.573 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:40.582 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:40.592 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T18:40:40.601 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:40.609 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:40.618 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:40.682 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "HAproxy container image", 2024-04-23T18:40:40.762 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:40.798 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:40.881 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:40.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:40.898 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_jaeger_agent": { 2024-04-23T18:40:40.908 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_jaeger_agent", 2024-04-23T18:40:40.917 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:40.978 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:41.060 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:41.069 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T18:40:41.077 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:41.085 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:41.094 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:41.102 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Jaeger agent container image", 2024-04-23T18:40:41.110 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:41.118 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:41.126 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:41.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:41.244 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_jaeger_collector": { 2024-04-23T18:40:41.330 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_jaeger_collector", 2024-04-23T18:40:41.370 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:41.378 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:41.386 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:41.394 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T18:40:41.466 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:41.476 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:41.484 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:41.492 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Jaeger collector container image", 2024-04-23T18:40:41.501 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:41.509 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:41.561 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:41.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:41.578 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_jaeger_query": { 2024-04-23T18:40:41.641 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_jaeger_query", 2024-04-23T18:40:41.658 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:41.716 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:41.792 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:41.800 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T18:40:41.808 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:41.816 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:41.824 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:41.894 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Jaeger query container image", 2024-04-23T18:40:41.911 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:41.918 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:41.926 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:41.941 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_keepalived": { 2024-04-23T18:40:41.947 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_keepalived", 2024-04-23T18:40:41.955 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:41.963 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:42.027 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:42.036 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T18:40:42.074 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:42.135 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:42.143 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:42.150 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Keepalived container image", 2024-04-23T18:40:42.158 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:42.217 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:42.270 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:42.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:42.284 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_loki": { 2024-04-23T18:40:42.292 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_loki", 2024-04-23T18:40:42.299 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:42.306 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:42.312 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:42.319 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T18:40:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:42.390 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:42.451 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Loki container image", 2024-04-23T18:40:42.545 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:42.579 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:42.585 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:42.598 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_node_exporter": { 2024-04-23T18:40:42.647 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_node_exporter", 2024-04-23T18:40:42.655 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:42.662 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:42.669 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:42.675 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T18:40:42.683 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:42.691 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:42.698 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:42.705 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Prometheus container image", 2024-04-23T18:40:42.764 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:42.773 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:42.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:42.887 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_nvmeof": { 2024-04-23T18:40:42.896 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_nvmeof", 2024-04-23T18:40:42.906 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:42.914 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T18:40:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:43.002 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:43.011 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:43.020 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Nvme-of container image", 2024-04-23T18:40:43.029 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:43.038 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:43.047 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:43.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:43.126 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_prometheus": { 2024-04-23T18:40:43.187 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_prometheus", 2024-04-23T18:40:43.256 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:43.265 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:43.274 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:43.282 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T18:40:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Prometheus container image", 2024-04-23T18:40:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_promtail": { 2024-04-23T18:40:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_promtail", 2024-04-23T18:40:43.515 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:43.524 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:43.575 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:43.648 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T18:40:43.656 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:43.665 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:43.674 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Promtail container image", 2024-04-23T18:40:43.746 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:43.755 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:43.763 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:43.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_samba": { 2024-04-23T18:40:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_samba", 2024-04-23T18:40:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:43.927 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T18:40:43.935 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:43.997 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:44.062 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:44.070 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Samba/SMB container image", 2024-04-23T18:40:44.078 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:44.086 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:44.094 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:44.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:44.188 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_snmp_gateway": { 2024-04-23T18:40:44.197 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_snmp_gateway", 2024-04-23T18:40:44.205 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:44.221 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:44.228 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T18:40:44.237 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:44.299 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:44.307 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:44.317 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "SNMP Gateway container image", 2024-04-23T18:40:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: "container_init": { 2024-04-23T18:40:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_init", 2024-04-23T18:40:44.508 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:44.520 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:44.528 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:44.535 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:40:44.543 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:44.550 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:44.556 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:44.563 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T18:40:44.570 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:44.630 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:44.637 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:44.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:44.780 INFO:teuthology.orchestra.run.smithi050.stdout: "daemon_cache_timeout": { 2024-04-23T18:40:44.843 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "daemon_cache_timeout", 2024-04-23T18:40:44.850 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:40:44.857 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:44.864 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:44.930 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:40:44.937 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:44.943 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:44.950 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:44.957 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T18:40:44.965 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:44.971 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:44.978 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:45.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:45.053 INFO:teuthology.orchestra.run.smithi050.stdout: "default_cephadm_command_timeout": { 2024-04-23T18:40:45.063 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T18:40:45.108 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:40:45.223 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:45.232 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:45.240 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "900", 2024-04-23T18:40:45.250 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:45.259 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T18:40:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "default_registry": { 2024-04-23T18:40:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "default_registry", 2024-04-23T18:40:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:45.531 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:45.542 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:45.583 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "docker.io", 2024-04-23T18:40:45.654 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:45.663 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:45.672 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:45.682 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T18:40:45.691 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:45.748 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:45.809 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:45.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:45.825 INFO:teuthology.orchestra.run.smithi050.stdout: "device_cache_timeout": { 2024-04-23T18:40:45.834 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "device_cache_timeout", 2024-04-23T18:40:45.843 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:40:45.852 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:45.862 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:45.870 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "1800", 2024-04-23T18:40:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:46.068 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "seconds to cache device inventory", 2024-04-23T18:40:46.134 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:46.142 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:46.151 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:46.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:46.217 INFO:teuthology.orchestra.run.smithi050.stdout: "device_enhanced_scan": { 2024-04-23T18:40:46.226 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "device_enhanced_scan", 2024-04-23T18:40:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:46.250 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:46.277 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:46.285 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:46.356 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:46.364 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T18:40:46.378 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:46.438 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:46.534 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:46.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:46.550 INFO:teuthology.orchestra.run.smithi050.stdout: "facts_cache_timeout": { 2024-04-23T18:40:46.557 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "facts_cache_timeout", 2024-04-23T18:40:46.564 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:40:46.615 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:46.622 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:46.629 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:40:46.635 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:46.642 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:46.649 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:46.656 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "seconds to cache host facts data", 2024-04-23T18:40:46.664 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:46.741 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:46.750 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:46.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:46.814 INFO:teuthology.orchestra.run.smithi050.stdout: "grafana_dashboards_path": { 2024-04-23T18:40:46.909 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "grafana_dashboards_path", 2024-04-23T18:40:46.918 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:46.927 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:46.935 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:46.945 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T18:40:47.011 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:47.048 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:47.056 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:47.066 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T18:40:47.074 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:47.083 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:47.093 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:47.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:47.110 INFO:teuthology.orchestra.run.smithi050.stdout: "host_check_interval": { 2024-04-23T18:40:47.134 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "host_check_interval", 2024-04-23T18:40:47.142 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:40:47.205 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:47.270 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:47.279 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:40:47.288 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:47.299 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:47.309 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:47.391 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to perform a host check", 2024-04-23T18:40:47.401 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:47.409 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:47.468 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:47.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:47.485 INFO:teuthology.orchestra.run.smithi050.stdout: "hw_monitoring": { 2024-04-23T18:40:47.493 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "hw_monitoring", 2024-04-23T18:40:47.501 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:47.509 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:47.570 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:47.579 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:47.597 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:47.674 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:47.683 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:47.760 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T18:40:47.770 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:47.779 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:47.831 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:47.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:47.850 INFO:teuthology.orchestra.run.smithi050.stdout: "inventory_list_all": { 2024-04-23T18:40:47.858 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "inventory_list_all", 2024-04-23T18:40:47.866 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:47.874 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:47.883 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:47.891 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:47.900 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:47.968 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:47.976 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:48.033 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T18:40:48.095 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:48.103 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:48.110 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:48.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:48.124 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:40:48.186 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:40:48.193 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:48.200 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:48.209 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:48.216 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:48.223 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:48.230 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:48.238 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:40:48.307 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:40:48.314 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:40:48.322 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:40:48.406 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:40:48.531 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:40:48.537 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:40:48.544 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:40:48.552 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:48.561 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:48.636 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:48.643 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:48.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:48.658 INFO:teuthology.orchestra.run.smithi050.stdout: "log_refresh_metadata": { 2024-04-23T18:40:48.665 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_refresh_metadata", 2024-04-23T18:40:48.672 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:48.759 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:48.766 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:48.838 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:48.846 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:48.857 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:48.910 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:48.984 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T18:40:48.992 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:49.000 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:49.007 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:49.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:40:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:40:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:40:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:49.214 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:49.223 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T18:40:49.232 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:49.280 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:49.329 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:49.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:49.415 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:40:49.423 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:40:49.432 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:49.497 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:49.506 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:49.517 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:40:49.526 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:49.534 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:49.543 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:40:49.552 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:40:49.561 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:40:49.568 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:40:49.633 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:40:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:40:49.737 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:40:49.796 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:40:49.807 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:49.815 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:49.823 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:49.832 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:49.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:49.908 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:40:49.917 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:40:49.925 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:49.934 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:49.942 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:49.951 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:49.960 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:49.969 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:50.034 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:50.072 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:50.134 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:50.210 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:50.220 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:50.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:50.238 INFO:teuthology.orchestra.run.smithi050.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T18:40:50.247 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T18:40:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:50.304 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:50.362 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T18:40:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:50.422 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:50.428 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:50.572 INFO:teuthology.orchestra.run.smithi050.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T18:40:50.579 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T18:40:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:50.606 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "*", 2024-04-23T18:40:50.733 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:50.751 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:50.758 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:50.765 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T18:40:50.774 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:50.783 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:50.793 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:50.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:50.857 INFO:teuthology.orchestra.run.smithi050.stdout: "max_count_per_host": { 2024-04-23T18:40:50.866 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_count_per_host", 2024-04-23T18:40:50.902 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:40:50.962 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:51.037 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:40:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:51.110 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:51.120 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "max number of daemons per service per host", 2024-04-23T18:40:51.127 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:51.133 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:51.139 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:51.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:51.153 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd_draining_count": { 2024-04-23T18:40:51.162 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_osd_draining_count", 2024-04-23T18:40:51.169 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:40:51.230 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:51.238 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:51.291 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:40:51.387 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:51.397 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:51.405 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:51.413 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T18:40:51.422 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "migration_current": { 2024-04-23T18:40:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "migration_current", 2024-04-23T18:40:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:40:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:51.637 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:51.647 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:51.656 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:51.689 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:51.763 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "internal - do not modify", 2024-04-23T18:40:51.772 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:51.781 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:51.790 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:51.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:51.864 INFO:teuthology.orchestra.run.smithi050.stdout: "mode": { 2024-04-23T18:40:51.873 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mode", 2024-04-23T18:40:51.882 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:51.891 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:51.900 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:51.910 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "root", 2024-04-23T18:40:51.918 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:51.928 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:51.936 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:40:51.980 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm-package", 2024-04-23T18:40:51.990 INFO:teuthology.orchestra.run.smithi050.stdout: "root" 2024-04-23T18:40:52.043 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:40:52.107 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T18:40:52.116 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:52.124 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:52.132 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:52.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:52.205 INFO:teuthology.orchestra.run.smithi050.stdout: "oob_default_addr": { 2024-04-23T18:40:52.214 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "oob_default_addr", 2024-04-23T18:40:52.223 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:52.231 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:52.315 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:52.324 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "169.254.1.1", 2024-04-23T18:40:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T18:40:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:52.466 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:52.529 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:52.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:52.570 INFO:teuthology.orchestra.run.smithi050.stdout: "prometheus_alerts_path": { 2024-04-23T18:40:52.579 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "prometheus_alerts_path", 2024-04-23T18:40:52.588 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T18:40:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T18:40:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:52.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: "registry_insecure": { 2024-04-23T18:40:52.955 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "registry_insecure", 2024-04-23T18:40:52.977 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:52.986 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:52.994 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:53.003 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:53.063 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:53.074 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:53.084 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:53.092 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T18:40:53.100 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:53.107 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:53.115 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:53.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:53.179 INFO:teuthology.orchestra.run.smithi050.stdout: "registry_password": { 2024-04-23T18:40:53.234 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "registry_password", 2024-04-23T18:40:53.241 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:53.300 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:53.354 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:53.362 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:53.369 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:53.377 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:53.384 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:53.439 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T18:40:53.447 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:53.455 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:53.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:53.481 INFO:teuthology.orchestra.run.smithi050.stdout: "registry_url": { 2024-04-23T18:40:53.490 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "registry_url", 2024-04-23T18:40:53.574 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:53.597 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:53.622 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:53.671 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:53.706 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:53.728 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:53.746 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T18:40:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:54.030 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:54.048 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:54.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:54.139 INFO:teuthology.orchestra.run.smithi050.stdout: "registry_username": { 2024-04-23T18:40:54.198 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "registry_username", 2024-04-23T18:40:54.207 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:54.217 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:54.225 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:54.250 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:54.260 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:54.268 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:54.308 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:54.388 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T18:40:54.396 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:54.407 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:54.415 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:54.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:54.492 INFO:teuthology.orchestra.run.smithi050.stdout: "secure_monitoring_stack": { 2024-04-23T18:40:54.508 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "secure_monitoring_stack", 2024-04-23T18:40:54.516 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:54.524 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:54.534 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:54.541 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:54.550 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:54.559 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:54.632 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:54.639 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T18:40:54.647 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:54.679 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:54.741 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:54.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: "service_discovery_port": { 2024-04-23T18:40:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "service_discovery_port", 2024-04-23T18:40:54.814 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:40:54.885 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:54.892 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:54.901 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8765", 2024-04-23T18:40:54.910 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:54.919 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:54.929 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:54.938 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "cephadm service discovery port", 2024-04-23T18:40:54.945 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:55.013 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:55.061 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:55.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:55.131 INFO:teuthology.orchestra.run.smithi050.stdout: "ssh_config_file": { 2024-04-23T18:40:55.182 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ssh_config_file", 2024-04-23T18:40:55.191 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:55.200 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:55.209 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:55.221 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:55.292 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:55.301 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:55.310 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:55.320 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T18:40:55.329 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:55.339 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:55.350 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:55.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:55.368 INFO:teuthology.orchestra.run.smithi050.stdout: "use_agent": { 2024-04-23T18:40:55.439 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "use_agent", 2024-04-23T18:40:55.499 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:55.572 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:55.583 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:55.591 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:55.671 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:55.687 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T18:40:55.695 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:55.704 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:55.712 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:55.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:55.730 INFO:teuthology.orchestra.run.smithi050.stdout: "use_repo_digest": { 2024-04-23T18:40:55.739 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "use_repo_digest", 2024-04-23T18:40:55.809 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:55.818 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:55.834 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:55.865 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:40:55.932 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:55.942 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:55.950 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:55.959 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T18:40:55.967 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:56.029 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:56.043 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:56.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:56.059 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_on_failed_host_check": { 2024-04-23T18:40:56.068 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_on_failed_host_check", 2024-04-23T18:40:56.076 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:56.086 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:56.094 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:56.159 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:40:56.168 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:56.179 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:56.224 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:56.287 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T18:40:56.295 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:56.304 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:56.313 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:56.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:56.429 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_on_stray_daemons": { 2024-04-23T18:40:56.450 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_on_stray_daemons", 2024-04-23T18:40:56.459 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:56.467 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:56.475 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:56.483 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:40:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T18:40:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:56.699 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:56.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:56.764 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_on_stray_hosts": { 2024-04-23T18:40:56.772 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_on_stray_hosts", 2024-04-23T18:40:56.782 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:56.854 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:56.865 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:40:56.874 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:40:56.883 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:56.894 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:56.903 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T18:40:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:40:57.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:40:57.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:57.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:40:57.244 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crash", 2024-04-23T18:40:57.253 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:40:57.262 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:40:57.333 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:40:57.378 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:40:57.386 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:40:57.395 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:57.404 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:57.412 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:57.421 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:40:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:40:57.518 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:40:57.573 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:40:57.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:40:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:40:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:40:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:40:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:40:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:57.758 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:57.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:57.775 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:40:57.784 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:40:57.792 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:57.801 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:57.936 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:58.033 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:58.120 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:58.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:58.135 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:40:58.143 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:40:58.152 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:40:58.160 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:58.168 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:58.178 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:40:58.251 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:58.281 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:58.288 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:40:58.336 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:40:58.406 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:40:58.414 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:40:58.422 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:40:58.431 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:40:58.439 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:40:58.495 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:40:58.503 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:58.511 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:58.519 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:40:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:40:58.647 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:40:58.656 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:58.666 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:58.725 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:40:58.831 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:59.063 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:59.097 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:59.142 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:40:59.169 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:59.254 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:59.277 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:59.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:59.343 INFO:teuthology.orchestra.run.smithi050.stdout: "retain_interval": { 2024-04-23T18:40:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "retain_interval", 2024-04-23T18:40:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:40:59.451 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "31536000", 2024-04-23T18:40:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:40:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:40:59.614 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:40:59.688 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T18:40:59.722 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:40:59.735 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:40:59.744 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:40:59.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:40:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_recent_interval": { 2024-04-23T18:40:59.868 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_recent_interval", 2024-04-23T18:40:59.877 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:40:59.886 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:40:59.934 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:40:59.943 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "1209600", 2024-04-23T18:40:59.952 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T18:41:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:00.137 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:00.144 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:00.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:00.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:00.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:00.286 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "dashboard", 2024-04-23T18:41:00.295 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:00.303 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:00.311 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:00.320 INFO:teuthology.orchestra.run.smithi050.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T18:41:00.328 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T18:41:00.336 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:00.345 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:00.403 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:00.416 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:41:00.464 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:00.501 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:00.554 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:00.562 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:00.571 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:00.578 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:00.644 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:00.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:00.661 INFO:teuthology.orchestra.run.smithi050.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T18:41:00.668 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T18:41:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:00.687 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:00.726 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:00.735 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:00.802 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:00.812 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:00.821 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:00.879 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:00.890 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:00.952 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:00.961 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:00.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:01.000 INFO:teuthology.orchestra.run.smithi050.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T18:41:01.061 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T18:41:01.070 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:01.092 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:01.101 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:01.111 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:01.120 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:01.129 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:01.139 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:01.147 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:01.179 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:01.188 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:01.233 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:01.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:01.299 INFO:teuthology.orchestra.run.smithi050.stdout: "AUDIT_API_ENABLED": { 2024-04-23T18:41:01.331 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T18:41:01.340 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:01.348 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:01.357 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:01.367 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:01.376 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:01.436 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:01.445 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:01.454 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:01.541 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:01.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:01.564 INFO:teuthology.orchestra.run.smithi050.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T18:41:01.608 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T18:41:01.622 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:01.679 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:01.687 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:01.696 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:01.704 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:01.765 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:01.774 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:01.826 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:01.835 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:01.844 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:01.853 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:01.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:01.868 INFO:teuthology.orchestra.run.smithi050.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T18:41:01.904 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T18:41:01.911 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:01.919 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:01.953 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:01.960 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:02.004 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:02.011 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:02.018 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:02.026 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:02.095 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:02.106 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:02.112 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:02.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:02.126 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T18:41:02.134 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T18:41:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:02.295 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:02.304 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:02.312 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:02.321 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:02.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:02.401 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T18:41:02.410 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T18:41:02.418 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:02.426 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:02.434 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:02.442 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:02.451 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:02.518 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:02.525 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:02.533 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:02.566 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:02.590 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:02.637 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:02.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:02.653 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T18:41:02.688 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T18:41:02.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:02.794 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:02.802 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:02.811 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:02.821 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:02.831 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:02.840 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:02.848 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:02.858 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:02.919 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:02.989 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:03.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:03.042 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T18:41:03.100 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T18:41:03.109 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:03.124 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:03.134 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:03.209 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:03.242 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:03.250 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:03.259 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:03.267 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:03.276 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:03.285 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:03.294 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:03.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:03.363 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T18:41:03.372 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T18:41:03.422 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:03.498 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:03.550 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:03.563 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:03.572 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:03.644 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:03.651 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:03.661 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:03.672 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:03.682 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:03.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:03.700 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T18:41:03.710 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T18:41:03.718 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:03.781 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:03.801 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:03.843 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:03.913 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:03.922 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:03.930 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:03.938 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:03.993 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:04.003 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T18:41:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T18:41:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:04.074 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:04.082 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:04.132 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:04.140 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:04.148 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:04.193 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:04.201 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:04.262 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:04.300 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:04.307 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:04.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T18:41:04.392 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T18:41:04.400 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:04.407 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:04.412 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:04.420 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:04.427 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:04.434 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:04.467 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:04.474 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:04.481 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:04.518 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:04.540 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:04.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:04.602 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T18:41:04.613 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T18:41:04.620 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:04.677 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:04.684 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:04.692 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "admin", 2024-04-23T18:41:04.699 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:04.707 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:04.715 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:04.725 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:04.732 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:04.773 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:04.780 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:04.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:04.836 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T18:41:04.891 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T18:41:04.959 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:04.965 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:04.972 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:04.979 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:05.100 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:05.108 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:05.120 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:05.129 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_API_URL": { 2024-04-23T18:41:05.213 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_API_URL", 2024-04-23T18:41:05.222 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:05.288 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:05.297 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:05.353 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:05.367 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:05.378 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:05.388 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T18:41:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T18:41:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:05.603 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:05.646 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "admin", 2024-04-23T18:41:05.722 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:05.728 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:05.815 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:05.825 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:05.835 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:05.845 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:05.909 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:05.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:05.926 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T18:41:05.953 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T18:41:05.963 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:05.981 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:05.989 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:05.997 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:06.065 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:06.077 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:06.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:06.257 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T18:41:06.263 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T18:41:06.270 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:06.385 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:06.394 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:06.401 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:06.410 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:06.419 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:06.428 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:06.496 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:06.516 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:06.524 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:06.571 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:06.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:06.652 INFO:teuthology.orchestra.run.smithi050.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T18:41:06.661 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T18:41:06.676 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:06.686 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:06.758 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:06.795 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:06.801 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:06.808 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:06.816 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:06.825 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:06.836 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:06.843 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:06.850 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T18:41:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T18:41:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:07.007 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:07.014 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:07.021 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:07.028 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:07.181 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:07.187 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:07.194 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:07.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:07.241 INFO:teuthology.orchestra.run.smithi050.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T18:41:07.294 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T18:41:07.301 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:07.307 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:07.373 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:07.383 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "[]", 2024-04-23T18:41:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:07.417 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:07.426 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:07.507 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:07.582 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:07.588 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:07.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:07.607 INFO:teuthology.orchestra.run.smithi050.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T18:41:07.616 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T18:41:07.625 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:07.634 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:07.708 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:07.717 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "{}", 2024-04-23T18:41:07.725 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:07.756 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:07.784 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:07.792 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:07.801 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:07.810 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:07.819 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:07.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:07.963 INFO:teuthology.orchestra.run.smithi050.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T18:41:07.971 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T18:41:07.980 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:07.989 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:07.997 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:08.005 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:08.013 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:08.021 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:08.072 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:08.116 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:08.182 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:08.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:08.208 INFO:teuthology.orchestra.run.smithi050.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T18:41:08.217 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T18:41:08.284 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:08.384 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:08.424 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:08.432 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:08.441 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:08.448 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:08.456 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:08.465 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:08.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T18:41:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T18:41:08.639 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:08.646 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:08.653 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:08.717 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:08.781 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:08.790 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:08.797 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:08.805 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:08.812 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:08.819 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:08.825 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:08.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:08.900 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T18:41:08.907 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T18:41:08.963 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:08.999 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:09.007 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:09.015 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:09.022 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:09.030 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:09.084 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:09.127 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:09.134 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:09.142 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:09.149 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:09.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:09.163 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T18:41:09.170 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T18:41:09.226 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:09.312 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:09.319 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:09.376 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:09.392 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:09.400 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:09.408 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:09.414 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:09.421 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:09.475 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:09.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:09.495 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T18:41:09.505 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T18:41:09.514 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:09.522 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:09.531 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:09.614 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:09.650 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:09.659 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:09.668 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:09.677 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:09.686 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:09.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:09.750 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T18:41:09.843 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T18:41:09.902 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:09.910 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:09.919 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:09.928 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:09.937 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:09.998 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:10.007 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:10.019 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:10.027 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:10.061 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:10.082 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:10.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:10.101 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T18:41:10.108 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T18:41:10.117 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:10.183 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:10.252 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:10.263 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:10.273 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:10.283 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:10.291 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:10.299 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:10.308 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:10.374 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:10.382 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:10.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:10.425 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T18:41:10.454 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T18:41:10.463 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:10.471 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:10.480 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:10.488 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:10.497 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:10.583 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:10.614 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:10.622 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:10.631 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:10.644 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:10.652 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:10.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:10.680 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T18:41:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T18:41:10.695 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:10.728 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:10.754 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:10.762 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:10.770 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:10.779 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:10.787 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:10.858 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:10.910 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:10.918 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:10.926 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:10.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:10.942 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T18:41:10.949 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T18:41:10.973 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:10.980 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:10.987 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:10.994 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T18:41:11.046 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:11.087 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:11.095 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:11.102 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:11.109 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:11.116 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:11.175 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:11.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:11.235 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T18:41:11.243 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T18:41:11.250 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:11.257 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:11.264 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:11.271 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:41:11.328 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:11.379 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:11.387 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T18:41:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T18:41:11.598 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:11.607 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:11.657 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8", 2024-04-23T18:41:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:11.763 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:11.788 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:11.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:11.832 INFO:teuthology.orchestra.run.smithi050.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T18:41:11.841 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T18:41:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:11.858 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "45", 2024-04-23T18:41:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:11.986 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:11.995 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:12.003 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:12.066 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:12.075 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:12.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:12.095 INFO:teuthology.orchestra.run.smithi050.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T18:41:12.145 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T18:41:12.153 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:12.161 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:12.169 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:12.177 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:12.186 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:12.293 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:12.311 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:12.392 INFO:teuthology.orchestra.run.smithi050.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T18:41:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T18:41:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:12.469 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:12.479 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "admin", 2024-04-23T18:41:12.488 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:12.497 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:12.505 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:12.514 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:12.630 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:12.639 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:12.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:12.658 INFO:teuthology.orchestra.run.smithi050.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T18:41:12.669 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T18:41:12.685 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:12.754 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:12.765 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:12.778 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:12.787 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:12.839 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:12.853 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:12.865 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:12.876 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:12.890 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:12.904 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:12.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T18:41:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T18:41:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:13.114 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:13.123 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:13.134 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:13.143 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:13.203 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:13.211 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:13.218 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:13.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:13.237 INFO:teuthology.orchestra.run.smithi050.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T18:41:13.245 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T18:41:13.299 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:13.390 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:13.399 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:13.408 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:13.416 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:13.423 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:13.433 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:13.467 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:13.488 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:13.496 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:13.507 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:13.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:13.552 INFO:teuthology.orchestra.run.smithi050.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T18:41:13.560 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T18:41:13.568 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:13.578 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:13.590 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:13.649 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:41:13.714 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:13.729 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:13.742 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:13.755 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:13.768 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:13.780 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:13.809 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:13.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:13.832 INFO:teuthology.orchestra.run.smithi050.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T18:41:13.844 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T18:41:13.897 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:41:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:13.961 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:14.042 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:14.092 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:14.104 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:14.119 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:14.133 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:14.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:14.156 INFO:teuthology.orchestra.run.smithi050.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T18:41:14.197 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T18:41:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:14.295 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5", 2024-04-23T18:41:14.306 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:14.320 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:14.330 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:14.341 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:14.353 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:14.416 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:14.489 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:14.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:14.508 INFO:teuthology.orchestra.run.smithi050.stdout: "cross_origin_url": { 2024-04-23T18:41:14.518 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cross_origin_url", 2024-04-23T18:41:14.527 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:14.537 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:14.601 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:14.612 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:14.622 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:14.632 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:14.691 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:14.700 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:14.712 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:14.722 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:14.730 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:14.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:14.810 INFO:teuthology.orchestra.run.smithi050.stdout: "crt_file": { 2024-04-23T18:41:14.893 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crt_file", 2024-04-23T18:41:14.903 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:14.912 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:14.920 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:14.929 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:14.940 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:14.997 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:15.007 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:15.016 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:15.025 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:15.115 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:15.124 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:15.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:15.143 INFO:teuthology.orchestra.run.smithi050.stdout: "debug": { 2024-04-23T18:41:15.226 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "debug", 2024-04-23T18:41:15.240 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:15.317 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:15.414 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:15.422 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:15.433 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:15.439 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:15.450 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:15.511 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Enable/disable debug options", 2024-04-23T18:41:15.520 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:15.530 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:15.539 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:15.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:15.635 INFO:teuthology.orchestra.run.smithi050.stdout: "jwt_token_ttl": { 2024-04-23T18:41:15.648 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "jwt_token_ttl", 2024-04-23T18:41:15.659 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:15.674 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:15.685 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:15.749 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "28800", 2024-04-23T18:41:15.818 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:15.829 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:15.840 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:15.860 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:15.871 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:15.881 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:15.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:15.919 INFO:teuthology.orchestra.run.smithi050.stdout: "key_file": { 2024-04-23T18:41:15.929 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "key_file", 2024-04-23T18:41:15.990 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:16.061 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:16.071 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:16.082 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:16.093 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:16.104 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:16.253 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:16.263 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:16.274 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:16.284 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:16.305 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:16.389 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:16.399 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:16.408 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:16.473 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:16.482 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:16.492 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:16.503 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:16.514 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:16.525 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:16.600 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:16.664 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:16.676 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:16.691 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:16.789 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:16.804 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:16.816 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:16.875 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:16.888 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:16.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:16.915 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:16.956 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:16.967 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:16.979 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:16.990 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:17.002 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:17.018 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:17.090 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:17.165 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:17.205 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:17.215 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:17.230 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:17.243 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:17.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:17.323 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:17.336 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:17.350 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:17.364 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:17.429 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:17.440 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:17.452 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:17.462 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:17.471 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:17.643 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:17.650 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:17.660 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:17.671 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:17.680 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:17.692 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:17.745 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:17.755 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:17.763 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:17.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:17.801 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:17.833 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:17.845 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:17.948 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:18.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:18.144 INFO:teuthology.orchestra.run.smithi050.stdout: "motd": { 2024-04-23T18:41:18.155 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "motd", 2024-04-23T18:41:18.164 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:18.272 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:18.290 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:18.305 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:18.374 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "The message of the day", 2024-04-23T18:41:18.439 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:18.452 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:18.465 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_resolve_ip_addr": { 2024-04-23T18:41:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T18:41:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:18.580 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:18.634 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:18.686 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:18.701 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:18.712 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:18.724 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:18.735 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:18.798 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:18.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:18.898 INFO:teuthology.orchestra.run.smithi050.stdout: "server_addr": { 2024-04-23T18:41:18.907 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_addr", 2024-04-23T18:41:18.915 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:18.924 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:18.981 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "::", 2024-04-23T18:41:18.992 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:19.002 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:19.013 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:19.072 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:19.082 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:19.092 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:19.101 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:19.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:19.121 INFO:teuthology.orchestra.run.smithi050.stdout: "server_port": { 2024-04-23T18:41:19.181 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_port", 2024-04-23T18:41:19.261 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:19.271 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:19.284 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:19.296 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8080", 2024-04-23T18:41:19.305 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:19.317 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:19.402 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:19.413 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:19.427 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:19.488 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:19.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:19.516 INFO:teuthology.orchestra.run.smithi050.stdout: "ssl": { 2024-04-23T18:41:19.530 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ssl", 2024-04-23T18:41:19.542 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:19.624 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:19.636 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:19.650 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:19.691 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:19.702 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:19.712 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:19.726 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:19.802 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:19.822 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:19.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:19.851 INFO:teuthology.orchestra.run.smithi050.stdout: "ssl_server_port": { 2024-04-23T18:41:19.908 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ssl_server_port", 2024-04-23T18:41:19.918 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:19.931 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:19.944 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:19.959 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8443", 2024-04-23T18:41:19.971 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:20.031 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:20.141 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:20.192 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:20.203 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:20.213 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:20.220 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:20.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:20.301 INFO:teuthology.orchestra.run.smithi050.stdout: "standby_behaviour": { 2024-04-23T18:41:20.316 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "standby_behaviour", 2024-04-23T18:41:20.324 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:20.332 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:20.388 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "redirect", 2024-04-23T18:41:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:20.474 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:20.488 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:20.547 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect" 2024-04-23T18:41:20.633 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:20.647 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:20.662 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:20.676 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:20.690 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:20.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:20.771 INFO:teuthology.orchestra.run.smithi050.stdout: "standby_error_status_code": { 2024-04-23T18:41:20.785 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "standby_error_status_code", 2024-04-23T18:41:20.799 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:20.811 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:20.852 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:20.862 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "500", 2024-04-23T18:41:20.871 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "400", 2024-04-23T18:41:20.881 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "599", 2024-04-23T18:41:20.889 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:20.899 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:20.972 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:21.001 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:21.036 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:21.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:21.051 INFO:teuthology.orchestra.run.smithi050.stdout: "url_prefix": { 2024-04-23T18:41:21.061 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "url_prefix", 2024-04-23T18:41:21.069 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:21.123 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:21.131 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:21.139 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:21.147 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:21.208 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:21.216 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:21.226 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:21.235 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:21.245 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:21.254 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:21.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:21.386 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:21.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:21.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:21.414 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "devicehealth", 2024-04-23T18:41:21.426 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:21.442 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:21.494 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:21.504 INFO:teuthology.orchestra.run.smithi050.stdout: "enable_monitoring": { 2024-04-23T18:41:21.517 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "enable_monitoring", 2024-04-23T18:41:21.532 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:21.587 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:21.603 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:21.614 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:21.627 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:21.638 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:21.653 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:21.723 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "monitor device health metrics", 2024-04-23T18:41:21.818 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:21.834 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:21.850 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:21.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:21.877 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:21.887 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:21.968 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:21.981 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:21.994 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:22.064 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:22.081 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:22.096 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:22.109 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:22.168 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:22.231 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:22.321 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:22.331 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:22.340 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:22.350 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:22.359 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:22.368 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:22.440 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:22.503 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:22.515 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:22.562 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:22.573 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:22.584 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:22.594 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:22.672 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:22.756 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:22.768 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:22.783 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:22.794 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:22.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:22.815 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:22.875 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:22.908 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:22.920 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:22.931 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:22.989 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:22.999 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:23.012 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:23.023 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:23.034 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:23.046 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:23.088 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:23.153 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:23.160 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:23.173 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:23.185 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:23.195 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:23.205 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:23.263 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:23.343 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:23.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:23.367 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:23.593 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:23.603 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:23.612 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:23.621 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:23.631 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:23.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:23.705 INFO:teuthology.orchestra.run.smithi050.stdout: "mark_out_threshold": { 2024-04-23T18:41:23.777 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mark_out_threshold", 2024-04-23T18:41:23.787 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:41:23.796 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:23.848 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "2419200", 2024-04-23T18:41:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:23.877 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:23.889 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:23.900 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T18:41:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:24.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:24.084 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": { 2024-04-23T18:41:24.093 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "pool_name", 2024-04-23T18:41:24.103 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:24.169 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:24.181 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:24.195 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "device_health_metrics", 2024-04-23T18:41:24.208 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:24.270 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:24.284 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:24.297 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T18:41:24.310 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:24.325 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:24.335 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:24.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: "retention_period": { 2024-04-23T18:41:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "retention_period", 2024-04-23T18:41:24.556 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:41:24.565 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:24.578 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:24.589 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "15552000", 2024-04-23T18:41:24.599 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:24.640 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:24.650 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:24.661 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how long to retain device health metrics", 2024-04-23T18:41:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: "scrape_frequency": { 2024-04-23T18:41:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "scrape_frequency", 2024-04-23T18:41:24.829 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:41:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "86400", 2024-04-23T18:41:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T18:41:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "self_heal": { 2024-04-23T18:41:25.143 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "self_heal", 2024-04-23T18:41:25.153 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:25.162 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:25.173 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:25.240 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:25.310 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:25.320 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:25.334 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:25.344 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T18:41:25.357 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:25.368 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:25.441 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:25.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:25.458 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:41:25.511 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:41:25.523 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:41:25.532 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:25.539 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:25.546 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:41:25.553 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:25.672 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:25.683 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T18:41:25.693 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:25.703 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:25.718 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:25.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:25.791 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_threshold": { 2024-04-23T18:41:25.799 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_threshold", 2024-04-23T18:41:25.806 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:41:25.819 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:25.851 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:25.879 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "7257600", 2024-04-23T18:41:25.890 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:25.898 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:25.904 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:25.912 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T18:41:25.979 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:26.048 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:26.055 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:26.063 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:26.071 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:26.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:26.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "diskprediction_local", 2024-04-23T18:41:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:26.161 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:26.222 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:26.229 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:26.238 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:26.245 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:26.252 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:26.259 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:26.310 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:26.365 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:26.372 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:26.381 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:26.390 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:26.397 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:26.406 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:26.412 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:26.470 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:26.477 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:26.491 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:26.544 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:26.554 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:26.566 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:26.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:26.587 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:26.598 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:26.678 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:26.749 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:26.761 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:26.769 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:26.782 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:26.806 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:26.845 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:26.857 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:26.867 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:26.876 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:26.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:26.934 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:26.943 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:26.952 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:26.960 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:26.968 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:27.005 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:27.097 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:27.106 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:27.114 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:27.122 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:27.130 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:27.141 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:27.216 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:27.226 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:27.237 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:27.245 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:27.305 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:27.320 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:27.328 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:27.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:27.346 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:27.420 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:27.475 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:27.482 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:27.490 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:27.505 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:27.540 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:27.548 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:27.574 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:27.581 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:27.589 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:27.596 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:27.630 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:27.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:27.645 INFO:teuthology.orchestra.run.smithi050.stdout: "predict_interval": { 2024-04-23T18:41:27.653 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "predict_interval", 2024-04-23T18:41:27.660 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:27.668 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:27.692 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:27.757 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "86400", 2024-04-23T18:41:27.764 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:27.771 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:27.779 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:27.787 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:27.795 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:27.820 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:27.828 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:27.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:27.843 INFO:teuthology.orchestra.run.smithi050.stdout: "predictor_model": { 2024-04-23T18:41:27.871 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "predictor_model", 2024-04-23T18:41:27.880 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:27.891 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:27.902 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:27.913 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "prophetstor", 2024-04-23T18:41:27.922 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:27.987 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:28.084 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:28.098 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:28.106 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:28.114 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:28.122 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:28.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:28.200 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:41:28.207 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:41:28.214 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:28.224 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:28.284 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:28.295 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:41:28.302 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:28.313 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:28.320 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:28.328 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:28.361 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:28.514 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "influx", 2024-04-23T18:41:28.525 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": false, 2024-04-23T18:41:28.534 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "influxdb python module not found", 2024-04-23T18:41:28.541 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:28.606 INFO:teuthology.orchestra.run.smithi050.stdout: "batch_size": { 2024-04-23T18:41:28.616 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "batch_size", 2024-04-23T18:41:28.626 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:28.634 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:28.644 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5000", 2024-04-23T18:41:28.712 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:28.752 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T18:41:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:28.852 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:28.862 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:28.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:28.939 INFO:teuthology.orchestra.run.smithi050.stdout: "database": { 2024-04-23T18:41:28.947 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "database", 2024-04-23T18:41:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:28.984 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:29.021 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:29.029 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "ceph", 2024-04-23T18:41:29.036 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:29.051 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:29.070 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T18:41:29.130 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:29.137 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:29.144 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:29.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:29.161 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": { 2024-04-23T18:41:29.171 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "hostname", 2024-04-23T18:41:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:29.247 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:29.255 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:29.263 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:29.293 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:29.304 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:29.315 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:29.323 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "InfluxDB server hostname", 2024-04-23T18:41:29.330 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:29.340 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:29.403 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:29.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:29.457 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:41:29.466 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:41:29.481 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:41:29.489 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:29.497 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:29.533 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "30", 2024-04-23T18:41:29.541 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "5", 2024-04-23T18:41:29.549 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:29.557 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:29.603 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T18:41:29.623 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:29.670 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:29.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:29.852 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:29.921 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:30.007 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:30.033 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:30.276 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:30.295 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:30.304 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:30.318 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:30.348 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:30.492 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:30.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:30.511 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:30.520 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:30.529 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:30.536 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:30.561 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:30.571 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:30.579 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:30.587 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:30.727 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:30.790 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:30.797 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:30.804 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:30.811 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:30.830 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:30.947 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:30.957 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:30.968 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:30.980 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:30.992 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:31.000 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:31.069 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:31.194 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:31.205 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:31.215 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:31.223 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:31.284 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:31.293 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:31.301 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:31.396 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:31.409 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:31.418 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:31.427 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:31.437 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:31.474 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:31.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: "password": { 2024-04-23T18:41:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "password", 2024-04-23T18:41:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "password of InfluxDB server user", 2024-04-23T18:41:31.758 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:31.769 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:31.829 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:31.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:31.845 INFO:teuthology.orchestra.run.smithi050.stdout: "port": { 2024-04-23T18:41:31.853 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "port", 2024-04-23T18:41:31.929 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:32.006 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:32.015 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:32.022 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8086", 2024-04-23T18:41:32.029 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:32.038 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:32.045 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:32.109 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "InfluxDB server port", 2024-04-23T18:41:32.125 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:32.132 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:32.140 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:32.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:32.207 INFO:teuthology.orchestra.run.smithi050.stdout: "ssl": { 2024-04-23T18:41:32.215 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ssl", 2024-04-23T18:41:32.225 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:32.238 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:32.250 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "false", 2024-04-23T18:41:32.429 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:32.439 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:32.446 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:32.456 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:41:32.467 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:32.476 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:32.541 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:32.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:32.561 INFO:teuthology.orchestra.run.smithi050.stdout: "threads": { 2024-04-23T18:41:32.568 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "threads", 2024-04-23T18:41:32.627 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:32.633 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:32.641 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:32.648 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5", 2024-04-23T18:41:32.656 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1", 2024-04-23T18:41:32.664 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "32", 2024-04-23T18:41:32.787 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:32.870 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T18:41:32.883 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:32.894 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:32.905 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:32.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:32.956 INFO:teuthology.orchestra.run.smithi050.stdout: "username": { 2024-04-23T18:41:33.025 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "username", 2024-04-23T18:41:33.035 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:33.047 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:33.059 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:33.116 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:33.127 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:33.138 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:33.152 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:33.166 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "username of InfluxDB server user", 2024-04-23T18:41:33.175 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:33.227 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:33.293 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:33.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:33.354 INFO:teuthology.orchestra.run.smithi050.stdout: "verify_ssl": { 2024-04-23T18:41:33.362 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "verify_ssl", 2024-04-23T18:41:33.371 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:33.380 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:33.388 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:33.455 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "true", 2024-04-23T18:41:33.466 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:33.475 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:33.540 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:33.552 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:41:33.565 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:33.575 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:33.648 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:33.659 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:33.733 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:33.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:33.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:33.815 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "insights", 2024-04-23T18:41:33.826 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:33.835 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:33.842 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:33.878 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:33.888 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:33.895 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:33.905 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:33.981 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:33.997 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:34.005 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:34.012 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:34.021 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:34.028 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:34.058 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:34.128 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:34.145 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:34.153 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:34.167 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:34.178 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:34.189 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:34.218 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:34.228 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:34.240 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:34.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:34.299 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:34.335 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:34.350 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:34.364 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:34.374 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:34.382 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:34.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:34.729 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:34.740 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:34.750 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:34.760 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:34.773 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:34.780 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:34.835 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:34.894 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:34.902 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:34.911 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:34.962 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:34.974 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:34.985 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:34.996 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:35.058 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:35.068 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:35.077 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:35.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:35.170 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:35.179 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:35.188 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:35.197 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:35.205 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:35.303 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:35.453 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:35.467 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:35.479 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:35.503 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:35.518 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:35.588 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:35.599 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:35.610 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:35.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:35.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:35.712 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "iostat", 2024-04-23T18:41:35.721 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:35.797 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:35.807 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:35.816 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:35.868 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:35.911 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:35.920 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:35.929 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:35.938 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:35.945 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:35.954 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:36.028 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:36.037 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:36.105 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:36.142 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:36.174 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:36.182 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:36.192 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:36.199 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:36.207 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:36.259 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:36.332 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:36.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:36.386 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:36.393 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:36.457 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:36.464 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:36.473 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:36.481 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:36.490 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:36.553 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:36.561 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:36.569 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:36.578 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:36.585 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:36.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:36.649 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:36.711 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:36.718 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:36.725 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:36.733 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:36.743 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:36.771 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:36.777 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:36.784 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:36.791 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:36.799 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:36.859 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:36.865 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:37.128 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:37.159 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:37.192 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:37.201 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:37.255 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:37.264 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:37.326 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:37.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:37.454 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "k8sevents", 2024-04-23T18:41:37.464 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:37.539 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_event_retention_days": { 2024-04-23T18:41:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ceph_event_retention_days", 2024-04-23T18:41:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:37.662 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "7", 2024-04-23T18:41:37.670 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:37.679 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:37.688 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T18:41:37.919 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:37.937 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:37.946 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:37.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:38.026 INFO:teuthology.orchestra.run.smithi050.stdout: "config_check_secs": { 2024-04-23T18:41:38.034 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "config_check_secs", 2024-04-23T18:41:38.042 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:38.051 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:38.060 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:38.069 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:41:38.125 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "10", 2024-04-23T18:41:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:38.143 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:38.151 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T18:41:38.160 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:38.168 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:38.321 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:38.381 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:38.390 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:38.399 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:38.406 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:38.576 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:38.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:38.769 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:38.776 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:38.783 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:38.791 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:38.798 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:38.852 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:38.867 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:38.874 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:38.882 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:38.893 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:38.904 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:38.992 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:39.547 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:39.576 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:39.585 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:39.593 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:39.603 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:39.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:39.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:39.704 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:39.750 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:39.764 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:39.805 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:39.813 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:39.823 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:39.832 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:39.892 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:39.985 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:39.994 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:40.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:40.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:40.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:40.127 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "localpool", 2024-04-23T18:41:40.135 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:40.144 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:40.153 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:40.202 INFO:teuthology.orchestra.run.smithi050.stdout: "failure_domain": { 2024-04-23T18:41:40.211 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "failure_domain", 2024-04-23T18:41:40.219 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:40.228 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:40.237 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:40.245 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "host", 2024-04-23T18:41:40.325 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:40.427 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:40.450 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:40.456 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "failure domain for any created local pool", 2024-04-23T18:41:40.464 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T18:41:40.526 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:40.534 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:40.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:40.549 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:40.556 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:40.563 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:40.615 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:40.623 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:40.630 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:40.638 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:40.647 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:40.656 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:40.725 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:40.841 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:40.849 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:40.909 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:40.917 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:40.925 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:40.933 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:40.941 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:40.947 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:40.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:41.091 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:41.151 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:41.159 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:41.166 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:41.175 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:41.198 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:41.205 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:41.212 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:41.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:41.227 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:41.235 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:41.268 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:41.277 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:41.284 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:41.292 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:41.300 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:41.308 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:41.371 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:41.451 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:41.461 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:41.470 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:41.480 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:41.539 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:41.548 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:41.557 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:41.566 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:41.574 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:41.584 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:41.593 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:41.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:41.668 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:41.676 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:41.738 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:41.749 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:41.794 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:41.804 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:41.813 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:41.822 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:41.830 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:41.839 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:41.889 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:41.899 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:41.954 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:41.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:41.971 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": { 2024-04-23T18:41:42.028 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "min_size", 2024-04-23T18:41:42.038 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:42.048 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:42.056 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:42.065 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:42.104 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:42.179 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:42.189 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:42.199 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "default min_size for any created local pool", 2024-04-23T18:41:42.208 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T18:41:42.216 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:42.263 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:42.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:42.287 INFO:teuthology.orchestra.run.smithi050.stdout: "num_rep": { 2024-04-23T18:41:42.295 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "num_rep", 2024-04-23T18:41:42.304 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:42.315 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "3", 2024-04-23T18:41:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:42.491 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "default replica count for any created local pool", 2024-04-23T18:41:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:42.615 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:42.624 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:42.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:42.639 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": { 2024-04-23T18:41:42.708 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "pg_num", 2024-04-23T18:41:42.716 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:41:42.725 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:42.732 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:42.740 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "128", 2024-04-23T18:41:42.748 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:42.781 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:42.815 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T18:41:42.823 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:42.830 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:42.914 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:43.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:43.021 INFO:teuthology.orchestra.run.smithi050.stdout: "prefix": { 2024-04-23T18:41:43.030 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "prefix", 2024-04-23T18:41:43.041 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:43.048 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:43.113 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:43.121 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:43.130 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:43.138 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:43.145 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:43.152 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "name prefix for any created local pool", 2024-04-23T18:41:43.211 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:43.294 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:43.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: "subtree": { 2024-04-23T18:41:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "subtree", 2024-04-23T18:41:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "rack", 2024-04-23T18:41:43.493 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:43.500 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:43.563 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:43.588 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T18:41:43.595 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T18:41:43.602 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:43.611 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:43.678 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:43.685 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:43.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:43.711 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mds_autoscaler", 2024-04-23T18:41:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:43.986 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:43.995 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:44.004 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:44.063 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:44.072 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:44.082 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:44.091 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:44.102 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:44.111 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:44.188 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:44.198 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:44.207 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:44.217 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:44.226 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:44.466 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:44.479 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:44.488 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:44.497 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:44.505 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:44.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:44.583 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:44.614 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:44.685 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:44.693 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:44.703 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:44.756 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:44.767 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:44.777 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:44.786 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:44.798 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:44.875 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:44.932 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:44.941 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:44.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:45.002 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:45.013 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:45.075 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:45.095 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:45.113 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:45.132 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:45.226 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:45.243 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:45.259 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:45.267 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:45.414 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:45.421 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:45.428 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:45.449 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:45.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:45.522 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:45.541 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:45.548 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:45.555 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:45.613 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:45.620 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:45.626 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:45.633 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:45.640 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:45.707 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:45.742 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:45.750 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:45.758 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:45.767 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:45.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:45.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:45.810 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mirroring", 2024-04-23T18:41:45.816 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:45.823 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:45.832 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:45.896 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:45.903 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:46.075 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:46.119 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:46.128 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:46.138 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:46.146 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:46.154 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:46.219 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:46.312 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:46.322 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:46.331 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:46.339 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:46.399 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:46.409 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:46.417 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:46.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:46.435 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:46.501 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:46.554 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:46.603 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:46.611 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:46.619 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:46.627 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:46.635 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:46.701 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:46.709 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:46.717 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:46.726 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:46.736 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:46.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:46.797 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:46.806 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:46.814 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:46.822 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:46.912 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:46.921 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:46.965 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:46.974 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:46.982 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:46.991 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:47.057 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:47.067 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:47.075 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:47.084 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:47.091 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:47.099 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:47.140 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:47.196 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:47.203 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:47.210 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:47.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:47.300 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:47.340 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:47.348 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:47.357 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:47.364 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:47.372 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:47.428 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:47.487 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:47.493 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:47.501 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:47.508 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:47.516 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:47.557 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:47.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:47.593 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:47.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:47.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:47.675 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "nfs", 2024-04-23T18:41:47.731 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:47.754 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:47.760 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:47.767 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:47.775 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:47.782 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:47.830 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:47.837 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:47.843 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:47.850 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:47.856 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:47.912 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:47.919 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:47.926 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:47.934 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:47.942 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:48.008 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:48.058 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:48.087 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:48.094 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:48.102 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:48.110 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:48.135 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:48.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:48.153 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:48.161 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:48.172 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:48.180 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:48.189 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:48.263 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:48.271 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:48.282 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:48.292 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:48.379 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:48.431 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:48.441 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:48.450 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:48.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:48.468 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:48.477 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:48.533 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:48.542 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:48.549 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:48.557 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:48.567 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:48.623 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:48.632 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:48.640 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:48.649 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:48.658 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:48.756 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:48.761 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:48.769 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:48.778 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:48.818 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:48.827 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:48.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:48.894 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:48.904 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:48.912 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:48.953 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:48.972 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:48.980 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:48.989 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:48.998 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:49.069 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:49.115 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:49.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:49.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:49.251 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "orchestrator", 2024-04-23T18:41:49.259 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:49.267 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:49.316 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:49.323 INFO:teuthology.orchestra.run.smithi050.stdout: "fail_fs": { 2024-04-23T18:41:49.330 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "fail_fs", 2024-04-23T18:41:49.338 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:49.346 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:49.416 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:49.472 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:49.511 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:49.518 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:49.525 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:49.532 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T18:41:49.592 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:49.600 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:49.607 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:49.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:49.621 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:49.629 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:49.686 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:49.692 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:49.700 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:49.707 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:49.758 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:49.797 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:49.828 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:49.834 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:49.842 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:49.849 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:49.873 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:49.881 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:49.888 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:49.895 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:49.903 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:49.910 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:49.953 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:49.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:49.968 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:49.975 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:49.981 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:50.038 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:50.095 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:50.122 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:50.134 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:50.160 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:50.189 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:50.271 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:50.756 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:51.142 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:51.266 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:51.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:51.281 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:51.342 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:51.349 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:51.358 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:51.365 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:51.372 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:51.613 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:51.623 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:51.632 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:51.702 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:51.766 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:51.773 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:51.782 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:51.792 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:51.831 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:51.837 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:51.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:51.854 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:51.863 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:51.920 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:51.959 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:51.970 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:51.978 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:51.987 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:52.078 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:52.087 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:52.138 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:52.148 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:52.157 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:52.180 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:52.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:52.223 INFO:teuthology.orchestra.run.smithi050.stdout: "orchestrator": { 2024-04-23T18:41:52.231 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "orchestrator", 2024-04-23T18:41:52.239 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:52.248 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:52.257 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm", 2024-04-23T18:41:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "rook", 2024-04-23T18:41:52.446 INFO:teuthology.orchestra.run.smithi050.stdout: "test_orchestrator" 2024-04-23T18:41:52.461 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:52.470 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Orchestrator backend", 2024-04-23T18:41:52.478 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:52.487 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:52.496 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:52.528 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:52.536 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:52.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:52.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:52.565 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "osd_perf_query", 2024-04-23T18:41:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:52.764 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:52.771 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:52.778 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:52.787 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:52.819 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:52.827 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:52.842 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:52.907 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:52.960 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:52.967 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:52.975 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:52.982 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:52.990 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:53.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:53.037 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:53.044 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:53.050 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:53.057 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:53.063 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:53.108 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:53.116 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:53.123 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:53.130 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:53.139 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:53.203 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:53.290 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:53.296 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:53.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:53.309 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:53.316 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:53.323 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:53.370 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:53.378 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:53.385 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:53.453 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:53.468 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:53.478 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:53.487 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:53.540 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:53.642 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:53.651 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:53.662 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:53.672 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:53.681 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:53.689 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:53.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:53.752 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:53.761 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:53.770 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:53.779 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:53.996 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:54.005 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:54.013 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:54.021 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:54.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:54.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:54.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:54.118 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "osd_support", 2024-04-23T18:41:54.127 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:54.137 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:54.147 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:54.199 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:54.238 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:54.286 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:54.293 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:54.301 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:54.387 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:54.429 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:54.438 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:54.446 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:54.455 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:54.529 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:54.538 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:54.546 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:54.553 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:54.561 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:54.627 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:54.633 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:54.641 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:54.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:54.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:54.720 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:54.811 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:54.817 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:54.825 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:54.834 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:54.841 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:54.848 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:54.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:54.943 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:54.949 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:54.957 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:54.963 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:54.969 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:55.051 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:55.058 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:55.070 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:55.077 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:55.083 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:55.099 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:55.107 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:55.113 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:55.120 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:55.127 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:55.193 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:55.235 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:55.249 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:55.257 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:55.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:55.331 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:55.364 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:55.372 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:55.379 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:55.387 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:55.396 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:55.460 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:55.480 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:55.488 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:55.495 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:55.503 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:55.563 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:55.570 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:55.579 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:55.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:55.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:55.678 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "pg_autoscaler", 2024-04-23T18:41:55.762 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:55.780 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:55.788 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:55.799 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:55.808 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:55.816 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:55.875 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:55.887 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:55.895 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:55.903 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:55.915 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:55.984 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:55.991 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:55.998 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:56.005 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:56.089 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:56.153 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:56.160 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:56.167 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:56.174 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:56.181 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:56.233 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:56.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:56.247 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:56.254 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:56.261 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:56.267 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:56.323 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:56.330 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:56.338 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:56.344 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:56.353 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:56.417 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:56.462 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:56.470 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:56.477 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:56.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:56.490 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:56.645 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:56.652 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:56.659 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:56.666 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:56.673 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:56.751 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:56.795 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:56.802 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:56.811 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:56.821 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:56.829 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:56.896 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:41:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:41:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:57.160 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:57.203 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:57.213 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:57.222 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:57.231 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:57.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:57.294 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:41:57.304 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:41:57.313 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:41:57.359 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:57.367 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:57.376 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:41:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:57.581 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:57.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "threshold": { 2024-04-23T18:41:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "threshold", 2024-04-23T18:41:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:41:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "3.0", 2024-04-23T18:41:57.755 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1.0", 2024-04-23T18:41:57.765 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:57.773 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "scaling threshold", 2024-04-23T18:41:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T18:41:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:57.909 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:57.919 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:41:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:41:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "progress", 2024-04-23T18:41:58.079 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:41:58.141 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:41:58.201 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:41:58.207 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_pg_recovery_event": { 2024-04-23T18:41:58.214 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "allow_pg_recovery_event", 2024-04-23T18:41:58.221 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:58.228 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:58.235 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:58.285 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:58.292 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:58.299 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:58.306 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:58.313 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T18:41:58.372 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:58.426 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:58.434 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:58.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:58.497 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": { 2024-04-23T18:41:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "enabled", 2024-04-23T18:41:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:41:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:41:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:58.648 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:58.656 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:58.663 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:58.670 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:58.681 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:58.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:41:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:41:58.849 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:58.856 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:58.925 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:58.932 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:41:58.939 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:58.947 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:58.954 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:58.963 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:58.971 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:59.009 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:59.050 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:59.058 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:59.066 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:59.072 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:59.130 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:59.190 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:59.198 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:59.207 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:59.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:59.224 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:41:59.294 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:41:59.367 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:41:59.376 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:59.384 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:41:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:41:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:59.621 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:41:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:41:59.682 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:41:59.692 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:41:59.700 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:41:59.759 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:41:59.768 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:41:59.776 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:41:59.785 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:41:59.795 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:41:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:41:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:41:59.866 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:41:59.875 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:41:59.885 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:41:59.894 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:41:59.902 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:41:59.950 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:41:59.966 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:41:59.974 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:41:59.984 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:41:59.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:00.125 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:00.134 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:00.145 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:00.153 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:00.162 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:00.283 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:00.317 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:00.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:00.333 INFO:teuthology.orchestra.run.smithi050.stdout: "max_completed_events": { 2024-04-23T18:42:00.341 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_completed_events", 2024-04-23T18:42:00.413 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:00.444 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:00.452 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:00.462 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "50", 2024-04-23T18:42:00.468 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:00.475 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:00.483 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:00.533 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "number of past completed events to remember", 2024-04-23T18:42:00.541 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:00.551 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:00.560 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:00.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:00.643 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:42:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:42:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:42:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:00.741 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5", 2024-04-23T18:42:00.815 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:00.824 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:00.836 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:00.847 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how long the module is going to sleep", 2024-04-23T18:42:00.856 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:00.864 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:00.914 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:00.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:00.984 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:00.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:01.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:01.079 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "prometheus", 2024-04-23T18:42:01.152 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:01.160 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:01.169 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:01.178 INFO:teuthology.orchestra.run.smithi050.stdout: "cache": { 2024-04-23T18:42:01.252 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cache", 2024-04-23T18:42:01.325 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:01.333 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:01.344 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:01.352 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:42:01.360 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:01.370 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:01.435 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:01.442 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:01.449 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:01.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:01.577 INFO:teuthology.orchestra.run.smithi050.stdout: "exclude_perf_counters": { 2024-04-23T18:42:01.584 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "exclude_perf_counters", 2024-04-23T18:42:01.618 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:01.700 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:01.732 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:01.739 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:42:01.746 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:01.754 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:01.763 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:01.770 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T18:42:01.777 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T18:42:01.841 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:01.849 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:01.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:01.864 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:01.871 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:01.939 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:01.997 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:02.004 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:02.012 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:02.019 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:02.088 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:02.096 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:02.107 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:02.113 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:02.121 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:02.128 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:02.135 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:02.290 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:02.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:02.337 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:02.343 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:02.350 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:02.419 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:02.426 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:02.432 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:02.440 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:02.447 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:02.455 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:02.464 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:02.500 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:02.530 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:02.539 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:02.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:02.557 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:02.612 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:02.650 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:02.659 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:02.668 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:02.676 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:02.744 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:02.753 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:02.761 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:02.769 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:02.778 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:02.786 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:02.795 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:02.849 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:02.859 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:02.869 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:02.877 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:02.886 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:02.961 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:03.010 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:03.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:03.028 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:03.038 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:03.064 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:03.073 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:03.081 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:03.090 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:03.099 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:03.113 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:03.121 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:03.171 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:03.180 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:03.188 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:03.197 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:03.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:03.265 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_stats_pools": { 2024-04-23T18:42:03.310 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rbd_stats_pools", 2024-04-23T18:42:03.319 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:03.327 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:03.336 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:03.400 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:03.412 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:03.423 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:03.503 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:03.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:03.520 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T18:42:03.527 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T18:42:03.534 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:03.587 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:03.629 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:03.636 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "300", 2024-04-23T18:42:03.645 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:03.653 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:03.713 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:03.719 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:03.727 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:03.734 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:03.741 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:03.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:03.756 INFO:teuthology.orchestra.run.smithi050.stdout: "scrape_interval": { 2024-04-23T18:42:03.810 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "scrape_interval", 2024-04-23T18:42:03.818 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:42:03.826 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:03.833 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:03.841 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "15.0", 2024-04-23T18:42:03.908 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:03.950 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:03.957 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:03.964 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:03.970 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: "server_addr": { 2024-04-23T18:42:04.064 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_addr", 2024-04-23T18:42:04.071 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:04.077 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:04.116 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:04.124 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "::", 2024-04-23T18:42:04.131 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:04.138 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:04.145 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:04.203 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T18:42:04.251 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:04.258 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:04.265 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:04.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:04.345 INFO:teuthology.orchestra.run.smithi050.stdout: "server_port": { 2024-04-23T18:42:04.353 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_port", 2024-04-23T18:42:04.381 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:04.398 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:04.407 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "9283", 2024-04-23T18:42:04.416 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:04.476 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:04.486 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:04.495 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T18:42:04.504 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:04.514 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:04.631 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:04.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:04.701 INFO:teuthology.orchestra.run.smithi050.stdout: "stale_cache_strategy": { 2024-04-23T18:42:04.713 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "stale_cache_strategy", 2024-04-23T18:42:04.723 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:04.732 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:04.800 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:04.809 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "log", 2024-04-23T18:42:04.818 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:04.888 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:04.898 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:04.907 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:04.952 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:04.967 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:04.976 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:04.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:04.995 INFO:teuthology.orchestra.run.smithi050.stdout: "standby_behaviour": { 2024-04-23T18:42:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "standby_behaviour", 2024-04-23T18:42:05.126 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:05.176 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "default", 2024-04-23T18:42:05.184 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:05.243 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:05.252 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:05.263 INFO:teuthology.orchestra.run.smithi050.stdout: "default", 2024-04-23T18:42:05.271 INFO:teuthology.orchestra.run.smithi050.stdout: "error" 2024-04-23T18:42:05.280 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:05.288 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:05.328 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:05.358 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:05.369 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:05.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:05.387 INFO:teuthology.orchestra.run.smithi050.stdout: "standby_error_status_code": { 2024-04-23T18:42:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "standby_error_status_code", 2024-04-23T18:42:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "500", 2024-04-23T18:42:05.601 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "400", 2024-04-23T18:42:05.609 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "599", 2024-04-23T18:42:05.619 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:05.629 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:05.636 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:05.643 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:05.652 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:05.686 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:05.717 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rbd_support", 2024-04-23T18:42:05.836 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:05.876 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:05.883 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:05.891 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:05.898 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:05.922 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:05.929 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:05.937 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:05.944 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:05.952 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:05.959 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:05.966 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:06.021 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:06.057 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:06.065 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:06.072 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:06.083 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:06.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:06.277 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:06.285 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:06.291 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:06.299 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:06.306 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:06.417 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:06.428 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:06.436 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:06.444 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:06.518 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:06.567 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:06.575 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:06.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:06.592 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:06.601 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:06.658 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:06.668 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:06.680 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:06.689 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:06.699 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:06.708 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:06.751 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:06.777 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:06.786 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:06.795 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:06.804 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:06.860 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:07.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:07.013 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:07.022 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:07.032 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:07.113 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:07.180 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:07.188 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:07.196 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:07.205 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:07.292 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:07.351 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:07.360 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:07.370 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:07.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:07.388 INFO:teuthology.orchestra.run.smithi050.stdout: "max_concurrent_snap_create": { 2024-04-23T18:42:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_concurrent_snap_create", 2024-04-23T18:42:07.464 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:07.473 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:07.481 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:07.516 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:42:07.526 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:07.584 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:07.593 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:07.601 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:07.624 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:07.633 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:07.702 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:07.781 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:07.788 INFO:teuthology.orchestra.run.smithi050.stdout: "mirror_snapshot_schedule": { 2024-04-23T18:42:07.797 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T18:42:07.805 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:07.872 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:07.946 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:07.954 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:07.962 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:07.971 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:07.978 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:07.986 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:08.044 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:08.072 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:08.080 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:08.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:08.095 INFO:teuthology.orchestra.run.smithi050.stdout: "trash_purge_schedule": { 2024-04-23T18:42:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "trash_purge_schedule", 2024-04-23T18:42:08.208 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:08.215 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:08.223 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:08.230 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:08.304 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:08.337 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:08.344 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:08.351 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:08.359 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:08.365 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:08.372 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:08.429 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:08.440 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:08.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:08.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:08.461 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "restful", 2024-04-23T18:42:08.514 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:08.558 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:08.576 INFO:teuthology.orchestra.run.smithi050.stdout: "enable_auth": { 2024-04-23T18:42:08.584 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "enable_auth", 2024-04-23T18:42:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:08.668 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:08.677 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:08.686 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:42:08.694 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:08.704 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:08.712 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:08.767 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:08.777 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:08.785 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:08.793 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:08.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:08.865 INFO:teuthology.orchestra.run.smithi050.stdout: "key_file": { 2024-04-23T18:42:08.908 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "key_file", 2024-04-23T18:42:08.971 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:08.980 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:08.988 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:09.049 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:09.080 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:09.091 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:09.099 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:09.109 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:09.157 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:09.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:09.186 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:09.195 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:09.264 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:09.273 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:09.392 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:09.436 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:09.446 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:09.455 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:09.463 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:09.471 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:09.480 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:09.536 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:09.545 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:09.556 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:09.566 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:09.575 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:09.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:09.774 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:09.784 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:09.793 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:09.803 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:09.865 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:09.874 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:09.881 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:09.890 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:09.898 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:09.909 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:09.918 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:09.975 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:10.020 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:10.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:10.035 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:10.042 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:10.134 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:10.174 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:10.181 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:10.190 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:10.197 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:10.261 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:10.269 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:10.277 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:10.285 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:10.293 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:10.301 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:10.308 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:10.363 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:10.378 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:10.385 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:10.394 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:10.400 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:10.451 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:10.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:10.500 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:10.508 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:10.516 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:10.584 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:10.594 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:10.602 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:10.614 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:10.622 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:10.631 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:10.643 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:10.694 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:10.739 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:10.748 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:10.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:10.766 INFO:teuthology.orchestra.run.smithi050.stdout: "server_addr": { 2024-04-23T18:42:10.834 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_addr", 2024-04-23T18:42:10.874 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:10.890 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:10.913 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:10.952 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:10.983 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:10.990 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:11.011 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:11.030 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:11.040 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:11.048 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:11.057 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:11.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:11.136 INFO:teuthology.orchestra.run.smithi050.stdout: "server_port": { 2024-04-23T18:42:11.147 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_port", 2024-04-23T18:42:11.156 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:11.167 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:11.248 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:11.299 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:11.311 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:11.320 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:11.331 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:11.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:11.556 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rgw", 2024-04-23T18:42:11.565 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:11.574 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:11.582 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:11.794 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:11.810 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:11.820 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:11.829 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:11.877 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:11.986 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:12.032 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:12.041 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:12.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:12.057 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:12.087 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:12.134 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:12.142 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:12.205 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:12.322 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:12.348 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:12.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:12.460 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:12.467 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:12.474 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:12.537 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:12.547 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:12.556 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:12.627 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:12.634 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:12.641 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:12.649 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:12.699 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:12.715 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:12.724 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:12.732 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:12.740 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:12.749 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:12.799 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:12.806 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:12.815 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:12.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:12.852 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:12.859 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:12.867 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:12.931 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:12.939 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:12.946 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:12.954 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:13.122 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:13.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:13.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:13.218 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rook", 2024-04-23T18:42:13.227 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:13.277 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:13.368 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:13.377 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:13.386 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:13.395 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:13.445 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:13.454 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:13.462 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:13.470 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:13.479 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:13.488 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:13.534 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:13.564 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:13.572 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:13.581 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:13.591 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:13.661 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:13.671 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:13.759 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:13.767 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:13.776 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:13.784 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:13.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:13.881 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:13.890 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:13.899 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:13.992 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:14.001 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:14.010 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:14.019 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:14.027 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:14.093 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:14.102 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:14.141 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:14.272 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:14.292 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:14.300 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:14.310 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:14.318 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:14.333 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:14.387 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:14.395 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:14.403 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:14.412 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:14.437 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:14.456 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:14.465 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:14.527 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:14.536 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:14.543 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:14.551 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:14.611 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:14.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:14.627 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:14.636 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:14.642 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:14.661 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:14.726 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:14.750 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:14.759 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:14.770 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:14.778 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:14.845 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:14.853 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:14.908 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:14.915 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:14.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:14.929 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_class": { 2024-04-23T18:42:14.983 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "storage_class", 2024-04-23T18:42:14.990 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:14.997 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:15.005 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:15.011 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "local", 2024-04-23T18:42:15.019 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:15.099 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:15.106 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:15.112 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T18:42:15.119 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:15.192 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:15.199 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:15.207 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:15.269 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:15.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:15.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:15.290 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "selftest", 2024-04-23T18:42:15.348 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:15.403 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:15.416 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:15.426 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:15.433 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:15.444 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:15.498 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:15.513 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:15.523 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:15.532 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:15.603 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:15.681 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:15.690 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:15.742 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:15.751 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:15.759 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:15.768 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:15.829 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:15.839 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:15.847 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:15.855 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:15.864 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:15.874 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:15.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:15.924 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:15.932 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:15.969 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:16.041 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:16.401 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:16.579 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:16.636 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:16.660 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:16.676 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:16.723 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:16.730 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:16.737 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:16.745 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:16.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:16.760 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:16.820 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:16.827 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:16.834 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:16.889 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:16.896 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:16.904 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:16.911 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:16.980 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:16.987 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:16.993 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:17.000 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:17.045 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:17.065 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:17.142 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:17.149 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:17.157 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:17.164 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:17.253 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:17.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:17.338 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:17.345 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:17.351 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:17.357 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:17.405 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:17.411 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:17.418 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:17.426 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:17.584 INFO:teuthology.orchestra.run.smithi050.stdout: "roption1": { 2024-04-23T18:42:17.593 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "roption1", 2024-04-23T18:42:17.602 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:17.663 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:17.672 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:17.680 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:17.690 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:17.748 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:17.759 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:17.770 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:17.779 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:17.833 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:17.842 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:17.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: "roption2": { 2024-04-23T18:42:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "roption2", 2024-04-23T18:42:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:17.930 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:17.938 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:17.947 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "xyz", 2024-04-23T18:42:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:18.110 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:18.119 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:18.151 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:18.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption1": { 2024-04-23T18:42:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption1", 2024-04-23T18:42:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:18.226 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:18.293 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:18.346 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:18.354 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:18.378 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:18.390 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:18.399 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:18.408 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption2": { 2024-04-23T18:42:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption2", 2024-04-23T18:42:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:18.596 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:18.608 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:18.618 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:18.628 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:18.637 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:18.646 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:18.686 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:18.693 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:18.746 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:18.754 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:18.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:18.769 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption3": { 2024-04-23T18:42:18.778 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption3", 2024-04-23T18:42:18.840 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:42:18.848 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:18.872 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:18.880 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:18.948 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:18.955 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:18.963 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:18.970 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:18.978 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:18.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:19.053 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption4": { 2024-04-23T18:42:19.060 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption4", 2024-04-23T18:42:19.068 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:19.075 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:19.083 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:19.139 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:19.172 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:19.178 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:19.186 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:19.235 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:19.242 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:19.248 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:19.255 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:19.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:19.269 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption5": { 2024-04-23T18:42:19.311 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption5", 2024-04-23T18:42:19.329 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:19.377 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:19.399 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:19.410 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:19.454 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:19.488 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:19.497 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:19.506 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:19.551 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:19.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:19.569 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption6": { 2024-04-23T18:42:19.581 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption6", 2024-04-23T18:42:19.590 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:19.599 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:19.629 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:19.639 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:42:19.648 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:19.727 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:19.736 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:19.745 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:19.753 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:19.834 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:19.844 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:19.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:19.905 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption7": { 2024-04-23T18:42:19.967 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption7", 2024-04-23T18:42:19.976 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:19.985 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:19.993 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:20.001 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:20.010 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1", 2024-04-23T18:42:20.072 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "42", 2024-04-23T18:42:20.081 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:20.089 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:20.153 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:20.161 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:20.170 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:20.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:20.229 INFO:teuthology.orchestra.run.smithi050.stdout: "testkey": { 2024-04-23T18:42:20.272 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "testkey", 2024-04-23T18:42:20.280 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:20.289 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:20.340 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:20.349 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:20.357 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:20.366 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:20.375 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:20.383 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:20.415 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:20.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:20.493 INFO:teuthology.orchestra.run.smithi050.stdout: "testlkey": { 2024-04-23T18:42:20.499 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "testlkey", 2024-04-23T18:42:20.506 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:20.612 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:20.663 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:20.670 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:20.677 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:20.727 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:20.736 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:20.747 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:20.754 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:20.761 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:20.769 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:20.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:20.804 INFO:teuthology.orchestra.run.smithi050.stdout: "testnewline": { 2024-04-23T18:42:20.811 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "testnewline", 2024-04-23T18:42:20.860 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:20.867 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:20.875 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:20.882 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:20.959 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:20.996 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:21.004 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:21.011 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:21.091 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:21.099 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:21.106 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:21.114 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:21.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:21.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:21.213 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "snap_schedule", 2024-04-23T18:42:21.241 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:21.250 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:21.259 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:21.268 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_m_granularity": { 2024-04-23T18:42:21.276 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "allow_m_granularity", 2024-04-23T18:42:21.340 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:21.349 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:21.358 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:21.366 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:21.417 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:21.426 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:21.436 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:21.445 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T18:42:21.454 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:21.462 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:21.500 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:21.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:21.517 INFO:teuthology.orchestra.run.smithi050.stdout: "dump_on_update": { 2024-04-23T18:42:21.558 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "dump_on_update", 2024-04-23T18:42:21.568 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:21.577 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:21.586 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:21.649 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:21.685 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:21.694 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:21.703 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:21.746 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "dump database to debug log on update", 2024-04-23T18:42:21.755 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:21.763 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:21.772 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:21.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:21.788 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:21.852 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:21.861 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:21.993 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:22.005 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:22.078 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:22.087 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:22.097 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:22.105 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:22.141 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:22.168 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:22.177 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:22.186 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:22.195 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:22.205 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:22.251 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:22.259 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:22.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:22.335 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:22.343 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:22.351 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:22.362 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:22.441 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:22.492 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:22.503 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:22.511 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:22.519 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:22.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:22.580 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:22.588 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:22.664 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:22.688 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:22.698 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:22.708 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:22.715 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:22.770 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:22.781 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:22.790 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:22.800 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:22.879 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:22.894 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:22.906 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:22.915 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:22.923 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:22.932 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:23.020 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:23.032 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:23.104 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:23.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:23.130 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:23.140 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:23.148 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:23.203 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:23.212 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:23.222 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:23.231 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:23.279 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:23.289 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:23.301 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:23.311 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:23.320 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:23.328 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:23.376 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:23.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:23.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "stats", 2024-04-23T18:42:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:23.559 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:23.569 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:23.578 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:23.629 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:23.637 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:23.671 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:23.682 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:23.690 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:23.701 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:23.735 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:23.744 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:23.796 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:23.806 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:23.815 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:23.824 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:23.834 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:23.908 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:23.915 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:23.922 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:23.929 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:24.078 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:24.111 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:24.164 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:24.171 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:24.180 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:24.190 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:24.200 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:24.210 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:24.278 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:24.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:24.295 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:24.336 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:24.383 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:24.418 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:24.427 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:24.436 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:24.444 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:24.453 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:24.555 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:24.622 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:24.631 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:24.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:24.648 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:24.656 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:24.665 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:24.768 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:24.774 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:24.782 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:24.824 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:24.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:24.848 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:24.857 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:24.865 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:24.876 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:25.090 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:25.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:25.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:25.209 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "status", 2024-04-23T18:42:25.218 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:25.235 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:25.287 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:25.303 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:25.365 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:25.382 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:25.395 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:25.404 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:25.412 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:25.468 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:25.476 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:25.485 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:25.493 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:25.532 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:25.558 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:25.567 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:25.576 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:25.584 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:25.643 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:25.710 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:25.751 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:25.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:25.768 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:25.777 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:25.787 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:25.845 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:25.854 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:25.863 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:25.872 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:25.928 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:25.938 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:25.948 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:25.957 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:25.965 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:25.972 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:26.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:26.074 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:26.081 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:26.090 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:26.098 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:26.107 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:26.114 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:26.195 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:26.254 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:26.261 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:26.269 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:26.277 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:26.286 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:26.294 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:26.330 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:26.361 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:26.400 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:26.408 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:26.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:26.425 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:26.434 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:26.485 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:26.493 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:26.501 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:26.510 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:26.549 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:26.578 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:26.586 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:26.595 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:26.604 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:26.613 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:26.639 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:26.658 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:26.710 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:26.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:26.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:26.732 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "telegraf", 2024-04-23T18:42:26.740 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:26.794 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:26.803 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:26.811 INFO:teuthology.orchestra.run.smithi050.stdout: "address": { 2024-04-23T18:42:26.819 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "address", 2024-04-23T18:42:26.880 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:26.889 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:26.897 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:26.906 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T18:42:26.916 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:26.925 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:26.968 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:26.976 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:27.044 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:27.053 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:27.062 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:27.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:27.085 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:42:27.126 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:42:27.143 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:42:27.152 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:27.160 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:27.217 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "15", 2024-04-23T18:42:27.226 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:27.235 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:27.249 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:27.258 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:27.266 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:27.298 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:27.366 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:27.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:27.384 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:27.393 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:27.401 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:27.409 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:27.452 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:27.460 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:27.469 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:27.479 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:27.531 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:27.549 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:27.558 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:27.566 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:27.575 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:27.586 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:27.649 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:27.670 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:27.692 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:27.701 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:27.710 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:27.719 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:27.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:27.777 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:27.787 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:27.795 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:27.804 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:27.862 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:27.871 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:27.880 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:27.889 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:27.900 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:27.908 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:27.941 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:28.016 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:28.023 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:28.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:28.042 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:28.050 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:28.058 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:28.110 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:28.116 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:28.124 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:28.133 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:28.200 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:28.213 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:28.221 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:28.230 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:28.238 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:28.305 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:28.377 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:28.469 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:28.477 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:28.485 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:28.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:28.576 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:28.584 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:28.594 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:28.682 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:28.689 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:28.697 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:28.706 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:28.714 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:28.722 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:28.758 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:28.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:28.863 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:28.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:28.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:28.939 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "telemetry", 2024-04-23T18:42:28.948 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:29.005 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:29.015 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_basic": { 2024-04-23T18:42:29.024 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_basic", 2024-04-23T18:42:29.032 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:29.053 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:29.176 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:42:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:29.250 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:29.259 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:29.271 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T18:42:29.279 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:29.288 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:29.383 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:29.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:29.434 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_crash": { 2024-04-23T18:42:29.443 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_crash", 2024-04-23T18:42:29.485 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:29.509 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:29.518 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:29.527 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:42:29.535 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:29.544 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:29.593 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:29.630 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T18:42:29.639 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:29.647 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:29.657 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:29.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:29.700 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_device": { 2024-04-23T18:42:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_device", 2024-04-23T18:42:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:29.831 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:42:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T18:42:29.887 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:29.960 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:30.019 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:30.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:30.033 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_ident": { 2024-04-23T18:42:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_ident", 2024-04-23T18:42:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T18:42:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:30.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:30.290 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_perf": { 2024-04-23T18:42:30.351 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_perf", 2024-04-23T18:42:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:30.368 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:30.376 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:30.383 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:30.476 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T18:42:30.493 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:30.566 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: "contact": { 2024-04-23T18:42:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "contact", 2024-04-23T18:42:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:30.725 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:30.735 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:30.788 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:30.798 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:30.807 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:30.815 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:30.867 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:30.900 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:30.910 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:30.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:30.956 INFO:teuthology.orchestra.run.smithi050.stdout: "description": { 2024-04-23T18:42:30.981 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "description", 2024-04-23T18:42:30.989 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:30.998 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:31.007 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:31.017 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:31.125 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:31.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:31.272 INFO:teuthology.orchestra.run.smithi050.stdout: "device_url": { 2024-04-23T18:42:31.281 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "device_url", 2024-04-23T18:42:31.289 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:31.339 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:31.348 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:31.943 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T18:42:32.229 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:32.277 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:32.371 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:32.665 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:32.709 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:32.729 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:32.737 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:32.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:32.776 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": { 2024-04-23T18:42:32.910 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "enabled", 2024-04-23T18:42:32.944 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:32.952 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:32.959 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:33.023 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:33.031 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:33.040 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:33.049 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:33.058 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:33.067 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:33.111 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:33.120 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:33.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:33.201 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:42:33.210 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:42:33.219 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:33.262 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:33.342 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:33.352 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "24", 2024-04-23T18:42:33.360 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "8", 2024-04-23T18:42:33.370 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:33.414 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:33.422 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:33.432 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:33.441 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:33.449 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:33.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:33.498 INFO:teuthology.orchestra.run.smithi050.stdout: "last_opt_revision": { 2024-04-23T18:42:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "last_opt_revision", 2024-04-23T18:42:33.597 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:33.605 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:33.614 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:33.623 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "1", 2024-04-23T18:42:33.632 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:33.687 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:33.696 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:33.707 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:33.716 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:33.797 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:33.807 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:33.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:33.855 INFO:teuthology.orchestra.run.smithi050.stdout: "leaderboard": { 2024-04-23T18:42:33.863 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "leaderboard", 2024-04-23T18:42:33.871 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:33.942 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:33.949 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:34.023 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:34.032 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:34.040 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:34.051 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:34.060 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:34.133 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:34.183 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:34.191 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:34.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:34.245 INFO:teuthology.orchestra.run.smithi050.stdout: "leaderboard_description": { 2024-04-23T18:42:34.257 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "leaderboard_description", 2024-04-23T18:42:34.265 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:34.274 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:34.282 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:34.290 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:34.355 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:34.385 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:34.394 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:34.597 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:34.613 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:34.628 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:34.637 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:34.751 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:34.784 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:34.791 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:34.798 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:34.805 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:34.812 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:34.868 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:34.876 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:34.883 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:34.891 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:34.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:34.957 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:34.991 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:34.998 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:35.005 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:35.011 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:35.058 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:35.115 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:35.122 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:35.129 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:35.137 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:35.144 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:35.151 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:35.231 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:35.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:35.247 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:35.292 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:35.363 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:35.372 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:35.382 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:35.392 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:35.404 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:35.409 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:35.454 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:35.524 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:35.534 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:35.543 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:35.553 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:35.563 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:35.574 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:35.625 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:35.721 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:35.746 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:35.755 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:35.766 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:35.775 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:35.824 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:35.897 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:35.906 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:35.914 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:35.950 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:35.961 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:36.012 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:36.021 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:36.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: "organization": { 2024-04-23T18:42:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "organization", 2024-04-23T18:42:36.107 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:36.116 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:36.125 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:36.134 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:36.144 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:36.190 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:36.261 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:36.287 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:36.295 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:36.303 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:36.312 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:36.387 INFO:teuthology.orchestra.run.smithi050.stdout: "proxy": { 2024-04-23T18:42:36.395 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "proxy", 2024-04-23T18:42:36.404 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:36.411 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:36.472 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:36.479 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:36.518 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:36.524 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:36.531 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:36.539 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:36.582 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:36.589 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:36.662 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:36.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:36.679 INFO:teuthology.orchestra.run.smithi050.stdout: "url": { 2024-04-23T18:42:36.686 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "url", 2024-04-23T18:42:36.693 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:36.790 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T18:42:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test_orchestrator", 2024-04-23T18:42:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:37.165 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:37.172 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:37.179 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:37.189 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:37.249 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:37.259 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:37.268 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:37.277 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:37.288 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:37.298 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:37.457 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:37.465 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:37.474 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:37.484 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:37.493 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:37.662 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:37.671 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:37.681 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:37.690 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:37.698 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:37.740 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:37.751 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:37.759 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:37.769 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:37.779 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:37.788 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:37.890 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:37.955 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:37.962 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:37.971 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:37.979 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:37.987 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:37.996 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:38.063 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:38.074 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:38.082 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:38.091 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:38.100 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:38.109 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:38.118 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:38.189 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:38.198 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:38.385 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:38.393 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:38.400 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:38.406 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:38.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:38.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "volumes", 2024-04-23T18:42:38.602 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:38.716 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:38.722 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:38.729 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:38.736 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:38.746 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:38.799 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:38.806 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:38.813 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:38.838 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:38.879 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:38.903 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:38.910 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:38.917 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:38.923 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:38.931 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:39.044 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:39.108 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:39.116 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:39.446 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:39.504 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:39.546 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:39.555 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:39.564 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:39.574 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:39.583 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:39.649 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:39.662 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:39.671 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:39.697 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:39.708 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:39.789 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:39.798 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:39.808 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:39.816 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:39.876 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:39.907 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:39.916 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:39.927 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:39.935 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:39.943 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:39.990 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:40.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:40.023 INFO:teuthology.orchestra.run.smithi050.stdout: "max_concurrent_clones": { 2024-04-23T18:42:40.032 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_concurrent_clones", 2024-04-23T18:42:40.040 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:40.049 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:40.058 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:40.110 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "4", 2024-04-23T18:42:40.119 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:40.127 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:40.145 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:40.206 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T18:42:40.216 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:40.223 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:40.231 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:40.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:40.250 INFO:teuthology.orchestra.run.smithi050.stdout: "periodic_async_work": { 2024-04-23T18:42:40.289 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "periodic_async_work", 2024-04-23T18:42:40.297 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:40.304 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:40.312 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:40.320 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:40.327 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:40.335 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:40.408 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:40.440 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Periodically check for async work", 2024-04-23T18:42:40.450 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:40.459 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:40.515 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:40.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:40.533 INFO:teuthology.orchestra.run.smithi050.stdout: "snapshot_clone_delay": { 2024-04-23T18:42:40.542 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "snapshot_clone_delay", 2024-04-23T18:42:40.551 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:40.560 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:40.606 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:40.617 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:42:40.625 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:40.633 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:40.642 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:40.650 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T18:42:40.658 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:40.727 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:40.788 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:40.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:40.804 INFO:teuthology.orchestra.run.smithi050.stdout: "snapshot_clone_no_wait": { 2024-04-23T18:42:40.854 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T18:42:40.870 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:40.878 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:40.887 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:40.896 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:42:40.906 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:40.964 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:40.987 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:40.995 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T18:42:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:41.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:41.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:41.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:41.147 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "zabbix", 2024-04-23T18:42:41.208 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:41.245 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:41.254 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:41.263 INFO:teuthology.orchestra.run.smithi050.stdout: "discovery_interval": { 2024-04-23T18:42:41.276 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "discovery_interval", 2024-04-23T18:42:41.288 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:42:41.322 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:41.365 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:41.374 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "100", 2024-04-23T18:42:41.384 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:41.393 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:41.403 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:41.412 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:41.474 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:41.522 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:41.530 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:41.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:41.587 INFO:teuthology.orchestra.run.smithi050.stdout: "identifier": { 2024-04-23T18:42:41.604 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "identifier", 2024-04-23T18:42:41.613 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:41.623 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:41.631 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:41.640 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:41.708 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:41.717 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:41.726 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:41.735 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:41.744 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:41.753 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:41.763 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:41.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:41.836 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:42:41.845 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:42:41.853 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:42:41.925 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:41.936 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:41.945 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:42:41.953 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:41.962 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:41.971 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:42.054 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:42.069 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:42.080 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:42.089 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:42.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:42.105 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:42.114 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:42.174 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:42.181 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:42.188 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:42.195 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:42.258 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:42.302 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:42.310 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:42.317 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:42.332 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:42.419 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:42.426 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:42.434 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:42.445 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:42.452 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:42.460 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:42.467 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:42.536 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:42.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:42.560 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:42.568 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:42.620 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:42.630 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:42.638 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:42.646 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:42.654 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:42.663 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:42.717 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:42.731 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:42.740 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:42.748 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:42.763 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:42.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:42.808 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:42.870 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:42.880 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:42.888 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:42.897 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:43.054 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:43.078 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:43.110 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:43.119 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:43.127 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:43.135 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:43.189 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:43.199 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:43.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:43.217 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:43.272 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:43.280 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:43.298 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:43.315 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:43.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:43.577 INFO:teuthology.orchestra.run.smithi050.stdout: "zabbix_host": { 2024-04-23T18:42:43.585 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "zabbix_host", 2024-04-23T18:42:43.593 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:43.602 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:43.684 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:43.703 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:43.712 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:43.720 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:43.745 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:43.754 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:43.812 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "zabbix_port": { 2024-04-23T18:42:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "zabbix_port", 2024-04-23T18:42:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:43.955 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:43.973 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:43.995 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10051", 2024-04-23T18:42:44.003 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:44.044 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:44.051 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:44.059 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:44.068 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:44.077 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:44.086 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:44.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:44.161 INFO:teuthology.orchestra.run.smithi050.stdout: "zabbix_sender": { 2024-04-23T18:42:44.170 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "zabbix_sender", 2024-04-23T18:42:44.207 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:44.225 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:44.234 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T18:42:44.289 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:44.296 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:44.311 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:44.463 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-23T18:42:44.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:44.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:44.489 INFO:teuthology.orchestra.run.smithi050.stdout: "gid": 25012, 2024-04-23T18:42:44.498 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "z", 2024-04-23T18:42:44.505 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_features": 4540701547738038271, 2024-04-23T18:42:44.512 INFO:teuthology.orchestra.run.smithi050.stdout: "available_modules": [ 2024-04-23T18:42:44.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:44.597 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "alerts", 2024-04-23T18:42:44.606 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:44.614 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:44.687 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:44.736 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:42:44.744 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:42:44.752 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:42:44.761 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:44.769 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:44.810 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:42:44.845 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:44.854 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:44.862 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T18:42:44.879 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:44.886 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:44.939 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:44.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:44.973 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:44.982 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:45.040 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:45.048 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:45.057 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:45.065 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:45.073 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:45.083 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:45.133 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:45.251 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:45.261 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:45.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:45.418 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:45.449 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:45.534 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:45.544 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:45.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:45.563 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:45.610 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:45.618 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:45.627 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:45.636 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:45.644 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:45.654 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:45.729 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:45.737 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:45.744 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:45.782 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:45.790 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:45.798 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:45.807 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:45.877 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:45.884 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:45.891 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:45.897 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:46.086 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:46.095 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:46.103 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:46.112 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:46.123 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:46.132 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:46.144 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:46.218 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:46.226 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:46.303 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_destination": { 2024-04-23T18:42:46.326 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_destination", 2024-04-23T18:42:46.334 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:46.343 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:46.352 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:46.361 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:46.426 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:46.434 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:46.442 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:46.450 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Email address to send alerts to", 2024-04-23T18:42:46.459 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:46.467 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:46.476 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:46.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:46.537 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_from_name": { 2024-04-23T18:42:46.545 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_from_name", 2024-04-23T18:42:46.568 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:46.627 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:46.636 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:46.645 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "Ceph", 2024-04-23T18:42:46.655 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:46.662 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:46.671 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:46.741 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Email From: name", 2024-04-23T18:42:46.755 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:46.764 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:46.776 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:46.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:46.793 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_host": { 2024-04-23T18:42:46.846 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_host", 2024-04-23T18:42:46.854 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:46.921 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:46.943 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:46.952 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:47.014 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:47.023 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:47.031 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:47.040 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "SMTP server", 2024-04-23T18:42:47.051 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:47.061 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:47.132 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:47.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:47.153 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_password": { 2024-04-23T18:42:47.162 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_password", 2024-04-23T18:42:47.171 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:47.182 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:47.190 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:47.197 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:47.232 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:47.240 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:47.249 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:47.311 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Password to authenticate with", 2024-04-23T18:42:47.319 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:47.329 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:47.338 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:47.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:47.356 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_port": { 2024-04-23T18:42:47.415 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_port", 2024-04-23T18:42:47.424 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:47.432 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:47.445 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:47.456 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "465", 2024-04-23T18:42:47.464 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:47.474 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:47.520 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:47.528 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "SMTP port", 2024-04-23T18:42:47.537 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:47.546 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:47.601 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:47.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:47.619 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_sender": { 2024-04-23T18:42:47.628 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_sender", 2024-04-23T18:42:47.636 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:47.644 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:47.704 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:47.719 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:47.726 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:47.733 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:47.740 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:47.748 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "SMTP envelope sender", 2024-04-23T18:42:47.755 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:47.803 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:47.810 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:47.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:47.829 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_ssl": { 2024-04-23T18:42:47.894 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_ssl", 2024-04-23T18:42:47.933 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:47.940 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:47.947 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:47.954 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:42:47.963 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:48.020 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:48.028 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:48.036 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T18:42:48.043 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:48.051 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:48.058 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:48.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:48.112 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_user": { 2024-04-23T18:42:48.121 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_user", 2024-04-23T18:42:48.130 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:48.140 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:48.202 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:48.217 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:48.225 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:48.234 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:48.243 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:48.251 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "User to authenticate as", 2024-04-23T18:42:48.312 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:48.321 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:48.329 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:48.337 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:48.345 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:48.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:48.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:48.432 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "balancer", 2024-04-23T18:42:48.457 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:48.465 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:48.473 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:48.553 INFO:teuthology.orchestra.run.smithi050.stdout: "active": { 2024-04-23T18:42:48.560 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "active", 2024-04-23T18:42:48.568 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:48.576 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:48.584 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:48.592 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:42:48.643 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:48.651 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:48.660 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:48.669 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T18:42:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:48.695 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:48.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:48.779 INFO:teuthology.orchestra.run.smithi050.stdout: "begin_time": { 2024-04-23T18:42:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "begin_time", 2024-04-23T18:42:48.796 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:48.864 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:48.873 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:48.884 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0000", 2024-04-23T18:42:48.892 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:48.902 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:48.912 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:48.990 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T18:42:49.023 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:42:49.032 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:49.043 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:49.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:49.062 INFO:teuthology.orchestra.run.smithi050.stdout: "begin_weekday": { 2024-04-23T18:42:49.070 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "begin_weekday", 2024-04-23T18:42:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:42:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:42:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "0", 2024-04-23T18:42:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "6", 2024-04-23T18:42:49.225 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:49.233 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T18:42:49.267 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:42:49.276 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:49.324 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:49.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:49.366 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_compat_max_iterations": { 2024-04-23T18:42:49.374 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crush_compat_max_iterations", 2024-04-23T18:42:49.383 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:42:49.392 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:49.400 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:49.440 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "25", 2024-04-23T18:42:49.449 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1", 2024-04-23T18:42:49.458 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "250", 2024-04-23T18:42:49.466 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:49.518 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T18:42:49.551 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:49.560 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:49.569 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:49.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:49.587 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_compat_metrics": { 2024-04-23T18:42:49.637 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crush_compat_metrics", 2024-04-23T18:42:49.647 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:49.657 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:49.667 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T18:42:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:49.687 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:49.728 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:49.767 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T18:42:49.775 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T18:42:49.782 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:49.838 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:49.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:49.854 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_compat_step": { 2024-04-23T18:42:49.862 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crush_compat_step", 2024-04-23T18:42:49.870 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:42:49.878 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:49.933 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:49.940 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0.5", 2024-04-23T18:42:49.947 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "0.001", 2024-04-23T18:42:49.954 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "0.999", 2024-04-23T18:42:49.962 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:49.970 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "aggressiveness of optimization", 2024-04-23T18:42:49.977 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T18:42:50.017 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:50.024 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:50.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:50.039 INFO:teuthology.orchestra.run.smithi050.stdout: "end_time": { 2024-04-23T18:42:50.093 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "end_time", 2024-04-23T18:42:50.100 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:50.106 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:50.114 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:50.121 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "2359", 2024-04-23T18:42:50.128 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:50.195 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:50.209 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:50.218 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T18:42:50.225 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:42:50.234 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:50.242 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:50.379 INFO:teuthology.orchestra.run.smithi050.stdout: "end_weekday": { 2024-04-23T18:42:50.387 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "end_weekday", 2024-04-23T18:42:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:42:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:42:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "0", 2024-04-23T18:42:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "6", 2024-04-23T18:42:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T18:42:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:42:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:50.637 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:42:50.741 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:42:50.750 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:50.759 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:50.767 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:50.830 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:50.892 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:50.899 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:50.909 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:50.917 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:50.924 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:50.992 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:51.016 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:51.023 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:51.030 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:51.090 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:51.099 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:51.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:51.115 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:42:51.159 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:42:51.185 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:51.191 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:51.198 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:51.206 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:51.214 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:51.257 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:51.264 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:51.271 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:51.279 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:51.285 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:51.292 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:51.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:51.366 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:42:51.374 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:42:51.381 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:51.389 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:51.443 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:42:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:42:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:42:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:42:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:42:51.613 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:42:51.620 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:42:51.628 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:51.637 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:51.674 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:51.682 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:51.715 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:51.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:51.786 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:42:51.795 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:42:51.804 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:51.812 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:51.821 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:51.832 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:51.936 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:51.946 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:51.955 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:51.964 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:42:51.988 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:51.998 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:52.008 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:52.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:52.085 INFO:teuthology.orchestra.run.smithi050.stdout: "min_score": { 2024-04-23T18:42:52.093 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "min_score", 2024-04-23T18:42:52.102 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:42:52.151 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:52.176 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:52.185 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:42:52.193 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:52.201 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:52.225 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T18:42:52.313 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:52.322 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "mode": { 2024-04-23T18:42:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mode", 2024-04-23T18:42:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:52.468 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:52.478 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "upmap", 2024-04-23T18:42:52.542 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:52.550 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:52.559 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:52.568 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-compat", 2024-04-23T18:42:52.576 INFO:teuthology.orchestra.run.smithi050.stdout: "none", 2024-04-23T18:42:52.584 INFO:teuthology.orchestra.run.smithi050.stdout: "read", 2024-04-23T18:42:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "upmap", 2024-04-23T18:42:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "upmap-read" 2024-04-23T18:42:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Balancer mode", 2024-04-23T18:42:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:52.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:52.806 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_ids": { 2024-04-23T18:42:52.815 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "pool_ids", 2024-04-23T18:42:52.822 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:52.886 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:52.899 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:52.907 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:52.923 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:52.979 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T18:42:53.006 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:53.012 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:53.020 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:53.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:53.035 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:42:53.042 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:42:53.103 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:42:53.110 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:53.117 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:53.124 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:42:53.185 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:53.191 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:53.198 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:53.206 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T18:42:53.213 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:53.294 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:53.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:53.326 INFO:teuthology.orchestra.run.smithi050.stdout: "upmap_max_deviation": { 2024-04-23T18:42:53.333 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "upmap_max_deviation", 2024-04-23T18:42:53.340 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:53.350 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:53.358 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5", 2024-04-23T18:42:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1", 2024-04-23T18:42:53.420 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:53.428 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:53.488 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T18:42:53.496 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T18:42:53.505 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:53.512 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:53.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:53.526 INFO:teuthology.orchestra.run.smithi050.stdout: "upmap_max_optimizations": { 2024-04-23T18:42:53.607 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "upmap_max_optimizations", 2024-04-23T18:42:53.678 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:42:53.686 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:53.693 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:53.700 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:42:53.707 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:53.715 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:53.747 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:53.758 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T18:42:53.767 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:53.777 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:42:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:42:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephadm", 2024-04-23T18:42:53.955 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:42:53.992 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:42:54.001 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:42:54.009 INFO:teuthology.orchestra.run.smithi050.stdout: "agent_down_multiplier": { 2024-04-23T18:42:54.018 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "agent_down_multiplier", 2024-04-23T18:42:54.027 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:42:54.037 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:54.098 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:54.107 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "3.0", 2024-04-23T18:42:54.116 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:54.125 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:54.183 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:54.192 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T18:42:54.201 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:54.210 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:54.219 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:54.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:54.273 INFO:teuthology.orchestra.run.smithi050.stdout: "agent_refresh_rate": { 2024-04-23T18:42:54.306 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "agent_refresh_rate", 2024-04-23T18:42:54.315 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:42:54.325 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:54.335 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:54.343 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "20", 2024-04-23T18:42:54.352 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:54.414 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:54.423 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:54.432 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T18:42:54.441 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:54.498 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:54.507 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:54.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:54.525 INFO:teuthology.orchestra.run.smithi050.stdout: "agent_starting_port": { 2024-04-23T18:42:54.533 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "agent_starting_port", 2024-04-23T18:42:54.542 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:42:54.594 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:54.602 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:54.638 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "4721", 2024-04-23T18:42:54.647 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:54.656 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:54.665 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:54.674 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T18:42:54.705 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:54.723 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:54.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_ptrace": { 2024-04-23T18:42:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "allow_ptrace", 2024-04-23T18:42:54.811 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:54.819 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:54.827 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:54.835 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:54.886 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:54.938 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:54.946 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:54.953 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T18:42:54.960 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T18:42:54.968 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:54.975 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:55.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:55.012 INFO:teuthology.orchestra.run.smithi050.stdout: "autotune_interval": { 2024-04-23T18:42:55.020 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "autotune_interval", 2024-04-23T18:42:55.027 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:42:55.075 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:55.082 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:55.090 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:42:55.097 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:55.105 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:55.112 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:55.176 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T18:42:55.221 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:55.229 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:55.235 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:55.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:55.249 INFO:teuthology.orchestra.run.smithi050.stdout: "autotune_memory_target_ratio": { 2024-04-23T18:42:55.256 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T18:42:55.315 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:42:55.322 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:55.330 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:55.337 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0.7", 2024-04-23T18:42:55.384 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:55.395 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:55.401 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:55.407 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T18:42:55.414 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:55.420 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:55.482 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:55.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:55.532 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm_log_destination": { 2024-04-23T18:42:55.540 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephadm_log_destination", 2024-04-23T18:42:55.548 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:55.554 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:55.561 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:42:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:55.652 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:42:55.717 INFO:teuthology.orchestra.run.smithi050.stdout: "file", 2024-04-23T18:42:55.725 INFO:teuthology.orchestra.run.smithi050.stdout: "file,syslog", 2024-04-23T18:42:55.735 INFO:teuthology.orchestra.run.smithi050.stdout: "syslog" 2024-04-23T18:42:55.744 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:42:55.753 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T18:42:55.823 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:55.864 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:55.891 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:55.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:55.909 INFO:teuthology.orchestra.run.smithi050.stdout: "cgroups_split": { 2024-04-23T18:42:55.918 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cgroups_split", 2024-04-23T18:42:55.927 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:55.936 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:55.997 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:56.011 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:42:56.020 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:56.029 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:56.083 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:56.113 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T18:42:56.120 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:56.128 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:56.137 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:56.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:56.213 INFO:teuthology.orchestra.run.smithi050.stdout: "config_checks_enabled": { 2024-04-23T18:42:56.224 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "config_checks_enabled", 2024-04-23T18:42:56.232 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:56.241 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:56.249 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:56.258 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:42:56.267 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:56.313 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:56.322 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:56.331 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T18:42:56.339 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:56.400 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:56.422 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:56.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:56.439 INFO:teuthology.orchestra.run.smithi050.stdout: "config_dashboard": { 2024-04-23T18:42:56.446 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "config_dashboard", 2024-04-23T18:42:56.455 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:42:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:56.522 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:42:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T18:42:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:56.612 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:56.683 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_alertmanager": { 2024-04-23T18:42:56.738 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_alertmanager", 2024-04-23T18:42:56.746 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:56.754 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:56.763 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:56.771 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T18:42:56.832 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:56.840 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:56.850 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:56.857 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Prometheus container image", 2024-04-23T18:42:56.865 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:56.873 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:56.881 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_base": { 2024-04-23T18:42:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_base", 2024-04-23T18:42:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:42:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T18:42:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Container image name, without the tag", 2024-04-23T18:42:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:57.114 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:57.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:57.128 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_elasticsearch": { 2024-04-23T18:42:57.136 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_elasticsearch", 2024-04-23T18:42:57.173 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:57.180 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:57.187 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:57.195 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T18:42:57.283 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:57.327 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:57.349 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:57.356 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "elasticsearch container image", 2024-04-23T18:42:57.363 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:57.379 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_grafana": { 2024-04-23T18:42:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_grafana", 2024-04-23T18:42:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:57.535 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T18:42:57.551 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:57.560 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:57.569 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:57.633 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Prometheus container image", 2024-04-23T18:42:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_haproxy": { 2024-04-23T18:42:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_haproxy", 2024-04-23T18:42:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:57.763 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:57.773 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T18:42:57.782 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:57.791 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "HAproxy container image", 2024-04-23T18:42:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:57.924 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:57.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:57.967 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_jaeger_agent": { 2024-04-23T18:42:57.976 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_jaeger_agent", 2024-04-23T18:42:57.985 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:58.046 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T18:42:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:58.075 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:58.083 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Jaeger agent container image", 2024-04-23T18:42:58.092 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:58.138 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:58.146 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:58.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:58.163 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_jaeger_collector": { 2024-04-23T18:42:58.224 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_jaeger_collector", 2024-04-23T18:42:58.287 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:58.296 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:58.306 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:58.315 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T18:42:58.324 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:58.388 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:58.396 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:58.404 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Jaeger collector container image", 2024-04-23T18:42:58.413 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:58.421 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:58.430 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:58.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:58.490 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_jaeger_query": { 2024-04-23T18:42:58.498 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_jaeger_query", 2024-04-23T18:42:58.509 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:58.518 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T18:42:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:58.615 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Jaeger query container image", 2024-04-23T18:42:58.659 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:58.677 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:58.685 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:58.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:58.704 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_keepalived": { 2024-04-23T18:42:58.712 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_keepalived", 2024-04-23T18:42:58.721 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:58.756 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:58.766 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T18:42:58.781 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:58.849 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:58.871 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:58.879 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Keepalived container image", 2024-04-23T18:42:58.887 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:58.895 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:58.903 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:58.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:58.977 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_loki": { 2024-04-23T18:42:58.985 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_loki", 2024-04-23T18:42:58.994 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:59.010 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:59.018 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T18:42:59.056 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:59.063 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:59.070 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:59.078 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Loki container image", 2024-04-23T18:42:59.132 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:59.171 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:59.178 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:59.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:59.192 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_node_exporter": { 2024-04-23T18:42:59.200 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_node_exporter", 2024-04-23T18:42:59.262 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:59.269 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:59.278 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:59.286 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T18:42:59.293 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:59.300 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:59.308 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:59.354 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Prometheus container image", 2024-04-23T18:42:59.362 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:59.368 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:59.374 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_nvmeof": { 2024-04-23T18:42:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_nvmeof", 2024-04-23T18:42:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T18:42:59.614 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:59.623 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Nvme-of container image", 2024-04-23T18:42:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:42:59.723 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:42:59.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:42:59.772 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_prometheus": { 2024-04-23T18:42:59.782 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_prometheus", 2024-04-23T18:42:59.845 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:42:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:42:59.864 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:42:59.875 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T18:42:59.885 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:42:59.894 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:42:59.967 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:42:59.976 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Prometheus container image", 2024-04-23T18:42:59.986 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:42:59.995 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:00.004 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_promtail": { 2024-04-23T18:43:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_promtail", 2024-04-23T18:43:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:00.159 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T18:43:00.166 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:00.174 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:00.183 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:00.191 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Promtail container image", 2024-04-23T18:43:00.199 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:00.264 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:00.274 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:00.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:00.291 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_samba": { 2024-04-23T18:43:00.300 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_samba", 2024-04-23T18:43:00.308 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:00.317 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:00.380 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:00.389 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T18:43:00.404 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:00.449 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:00.456 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:00.465 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Samba/SMB container image", 2024-04-23T18:43:00.472 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:00.480 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:00.487 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:00.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:00.547 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_snmp_gateway": { 2024-04-23T18:43:00.555 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_snmp_gateway", 2024-04-23T18:43:00.562 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:00.571 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:00.580 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:00.588 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T18:43:00.620 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:00.643 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:00.650 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:00.658 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "SNMP Gateway container image", 2024-04-23T18:43:00.724 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:00.740 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:00.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:00.757 INFO:teuthology.orchestra.run.smithi050.stdout: "container_init": { 2024-04-23T18:43:00.767 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_init", 2024-04-23T18:43:00.827 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:00.836 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:00.844 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:00.853 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:00.862 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:00.872 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:00.881 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:00.920 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T18:43:00.929 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:00.939 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:00.949 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:00.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:01.009 INFO:teuthology.orchestra.run.smithi050.stdout: "daemon_cache_timeout": { 2024-04-23T18:43:01.018 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "daemon_cache_timeout", 2024-04-23T18:43:01.026 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:43:01.035 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:01.044 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:01.092 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:43:01.101 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:01.110 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:01.119 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:01.128 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T18:43:01.137 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:01.146 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:01.195 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:01.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:01.215 INFO:teuthology.orchestra.run.smithi050.stdout: "default_cephadm_command_timeout": { 2024-04-23T18:43:01.223 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T18:43:01.281 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:01.301 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:01.309 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:01.318 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "900", 2024-04-23T18:43:01.331 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:01.342 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:01.409 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:01.417 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T18:43:01.425 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:01.434 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:01.443 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:01.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: "default_registry": { 2024-04-23T18:43:01.554 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "default_registry", 2024-04-23T18:43:01.564 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:01.573 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:01.581 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:01.634 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "docker.io", 2024-04-23T18:43:01.643 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:01.652 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:01.661 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:01.668 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T18:43:01.675 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:01.784 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:01.795 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:01.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:01.809 INFO:teuthology.orchestra.run.smithi050.stdout: "device_cache_timeout": { 2024-04-23T18:43:01.816 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "device_cache_timeout", 2024-04-23T18:43:01.824 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:43:01.831 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:01.879 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:01.886 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "1800", 2024-04-23T18:43:01.893 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:01.901 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:01.948 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:01.955 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "seconds to cache device inventory", 2024-04-23T18:43:01.962 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:01.969 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:01.976 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:02.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:02.109 INFO:teuthology.orchestra.run.smithi050.stdout: "device_enhanced_scan": { 2024-04-23T18:43:02.116 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "device_enhanced_scan", 2024-04-23T18:43:02.123 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:02.131 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:02.138 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:02.197 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:02.212 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T18:43:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:02.278 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:02.310 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:02.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:02.325 INFO:teuthology.orchestra.run.smithi050.stdout: "facts_cache_timeout": { 2024-04-23T18:43:02.335 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "facts_cache_timeout", 2024-04-23T18:43:02.389 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:43:02.484 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:02.534 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:02.558 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:43:02.566 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:02.573 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:02.581 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:02.589 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "seconds to cache host facts data", 2024-04-23T18:43:02.666 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:02.676 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:02.684 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:02.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:02.775 INFO:teuthology.orchestra.run.smithi050.stdout: "grafana_dashboards_path": { 2024-04-23T18:43:02.783 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "grafana_dashboards_path", 2024-04-23T18:43:02.791 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:02.800 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:02.808 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:02.816 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T18:43:02.875 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:02.893 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:02.902 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:02.911 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T18:43:02.919 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:02.928 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:02.936 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:02.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:03.044 INFO:teuthology.orchestra.run.smithi050.stdout: "host_check_interval": { 2024-04-23T18:43:03.059 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "host_check_interval", 2024-04-23T18:43:03.067 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:43:03.123 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:03.135 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:03.144 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:43:03.152 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:03.161 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:03.170 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:03.225 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to perform a host check", 2024-04-23T18:43:03.252 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:03.261 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:03.271 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:03.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:03.288 INFO:teuthology.orchestra.run.smithi050.stdout: "hw_monitoring": { 2024-04-23T18:43:03.296 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "hw_monitoring", 2024-04-23T18:43:03.347 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:03.375 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:03.383 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:03.392 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T18:43:03.482 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:03.490 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:03.549 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:03.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:03.563 INFO:teuthology.orchestra.run.smithi050.stdout: "inventory_list_all": { 2024-04-23T18:43:03.570 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "inventory_list_all", 2024-04-23T18:43:03.577 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:03.585 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:03.593 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:03.663 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:03.709 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:03.717 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:03.725 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:03.781 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T18:43:03.789 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:03.797 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:03.805 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:03.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:03.822 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:03.891 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:03.899 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:03.907 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:03.915 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:03.923 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:03.931 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:03.939 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:04.003 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:04.098 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:04.127 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:04.134 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:04.143 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:04.151 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:04.218 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:04.279 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:04.286 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:04.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:04.329 INFO:teuthology.orchestra.run.smithi050.stdout: "log_refresh_metadata": { 2024-04-23T18:43:04.338 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_refresh_metadata", 2024-04-23T18:43:04.345 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:04.354 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:04.412 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:04.420 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:04.428 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:04.436 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:04.493 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:04.502 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T18:43:04.510 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:04.519 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:04.527 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:04.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:04.612 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:04.631 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:04.640 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:04.649 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:04.658 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:04.666 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:04.675 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:04.733 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:04.741 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:04.750 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T18:43:04.758 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:04.817 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:04.826 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:04.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:04.843 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:04.852 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:04.906 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:04.966 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:05.000 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:05.009 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:05.017 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:05.042 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:05.105 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:05.113 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:05.121 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:05.132 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:05.207 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:05.215 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:05.223 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:05.232 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:05.240 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:05.304 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:05.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:05.404 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:05.413 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:05.422 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:05.589 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:05.596 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:05.603 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:05.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:05.618 INFO:teuthology.orchestra.run.smithi050.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T18:43:05.626 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T18:43:05.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:05.703 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:05.710 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:05.718 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:05.725 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:05.732 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:05.738 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:05.847 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T18:43:05.854 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:05.861 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:05.867 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:05.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:05.923 INFO:teuthology.orchestra.run.smithi050.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T18:43:05.931 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T18:43:05.938 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:05.945 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:06.017 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:06.086 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "*", 2024-04-23T18:43:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T18:43:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: "max_count_per_host": { 2024-04-23T18:43:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_count_per_host", 2024-04-23T18:43:06.289 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:06.297 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:43:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:06.407 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:06.419 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "max number of daemons per service per host", 2024-04-23T18:43:06.427 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:06.434 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:06.442 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:06.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:06.458 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd_draining_count": { 2024-04-23T18:43:06.505 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_osd_draining_count", 2024-04-23T18:43:06.514 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:06.523 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:06.532 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:06.588 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:43:06.630 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:06.639 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:06.647 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:06.656 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T18:43:06.664 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:06.706 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:06.777 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:06.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:06.795 INFO:teuthology.orchestra.run.smithi050.stdout: "migration_current": { 2024-04-23T18:43:06.804 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "migration_current", 2024-04-23T18:43:06.813 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:06.822 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:06.880 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "internal - do not modify", 2024-04-23T18:43:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:07.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:07.071 INFO:teuthology.orchestra.run.smithi050.stdout: "mode": { 2024-04-23T18:43:07.080 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mode", 2024-04-23T18:43:07.088 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:07.096 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:07.105 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:07.115 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "root", 2024-04-23T18:43:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:07.180 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:07.188 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm-package", 2024-04-23T18:43:07.196 INFO:teuthology.orchestra.run.smithi050.stdout: "root" 2024-04-23T18:43:07.247 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:07.253 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T18:43:07.260 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:07.267 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:07.274 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:07.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:07.316 INFO:teuthology.orchestra.run.smithi050.stdout: "oob_default_addr": { 2024-04-23T18:43:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "oob_default_addr", 2024-04-23T18:43:07.487 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:07.497 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:07.504 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:07.510 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "169.254.1.1", 2024-04-23T18:43:07.516 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:07.621 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:07.630 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:07.637 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T18:43:07.644 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:07.673 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:07.781 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:07.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:07.834 INFO:teuthology.orchestra.run.smithi050.stdout: "prometheus_alerts_path": { 2024-04-23T18:43:07.841 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "prometheus_alerts_path", 2024-04-23T18:43:07.905 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:07.967 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:07.979 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:07.986 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T18:43:07.994 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:08.000 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:08.008 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:08.016 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T18:43:08.090 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:08.103 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:08.109 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:08.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: "registry_insecure": { 2024-04-23T18:43:08.182 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "registry_insecure", 2024-04-23T18:43:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:08.202 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:08.210 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:08.246 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:08.298 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:08.310 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:08.337 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:08.345 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T18:43:08.352 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:08.362 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:08.370 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:08.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:08.465 INFO:teuthology.orchestra.run.smithi050.stdout: "registry_password": { 2024-04-23T18:43:08.473 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "registry_password", 2024-04-23T18:43:08.482 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:08.602 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:08.662 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T18:43:08.732 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:08.741 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:08.750 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:08.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:08.767 INFO:teuthology.orchestra.run.smithi050.stdout: "registry_url": { 2024-04-23T18:43:08.776 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "registry_url", 2024-04-23T18:43:08.829 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:08.839 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:08.848 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:08.856 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:08.905 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:08.918 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:08.927 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:08.937 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T18:43:08.946 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:08.969 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:09.031 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:09.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:09.047 INFO:teuthology.orchestra.run.smithi050.stdout: "registry_username": { 2024-04-23T18:43:09.055 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "registry_username", 2024-04-23T18:43:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:09.080 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:09.138 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:09.146 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:09.153 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:09.162 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:09.230 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T18:43:09.238 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:09.245 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:09.253 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:09.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:09.287 INFO:teuthology.orchestra.run.smithi050.stdout: "secure_monitoring_stack": { 2024-04-23T18:43:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "secure_monitoring_stack", 2024-04-23T18:43:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:09.398 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:09.406 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:09.413 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:09.480 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:09.488 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T18:43:09.497 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:09.505 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:09.564 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:09.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:09.579 INFO:teuthology.orchestra.run.smithi050.stdout: "service_discovery_port": { 2024-04-23T18:43:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "service_discovery_port", 2024-04-23T18:43:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:09.707 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:09.715 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8765", 2024-04-23T18:43:09.722 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:09.727 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:09.734 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:09.740 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "cephadm service discovery port", 2024-04-23T18:43:09.747 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:09.802 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:09.810 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:09.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:09.823 INFO:teuthology.orchestra.run.smithi050.stdout: "ssh_config_file": { 2024-04-23T18:43:09.872 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ssh_config_file", 2024-04-23T18:43:09.882 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:09.891 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:09.934 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:09.940 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:09.967 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:10.024 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:10.038 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:10.046 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T18:43:10.055 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:10.064 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:10.073 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:10.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:10.129 INFO:teuthology.orchestra.run.smithi050.stdout: "use_agent": { 2024-04-23T18:43:10.140 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "use_agent", 2024-04-23T18:43:10.149 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:10.159 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:10.221 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:10.251 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:10.262 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:10.271 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:10.280 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:10.288 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T18:43:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:10.340 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:10.350 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:10.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:10.366 INFO:teuthology.orchestra.run.smithi050.stdout: "use_repo_digest": { 2024-04-23T18:43:10.376 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "use_repo_digest", 2024-04-23T18:43:10.384 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:10.447 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:10.455 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:10.464 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:10.471 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:10.546 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:10.554 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:10.563 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T18:43:10.573 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:10.581 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:10.605 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:10.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:10.678 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_on_failed_host_check": { 2024-04-23T18:43:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_on_failed_host_check", 2024-04-23T18:43:10.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:10.703 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:10.711 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:10.719 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:10.805 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:10.813 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:10.886 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:10.894 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T18:43:10.988 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:10.999 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:11.008 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:11.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:11.025 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_on_stray_daemons": { 2024-04-23T18:43:11.053 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_on_stray_daemons", 2024-04-23T18:43:11.109 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:11.118 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:11.126 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:11.133 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:11.144 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:11.153 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:11.162 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:11.261 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T18:43:11.274 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:11.282 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:11.319 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:11.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:11.376 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_on_stray_hosts": { 2024-04-23T18:43:11.384 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_on_stray_hosts", 2024-04-23T18:43:11.392 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:11.402 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:11.543 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:11.554 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:11.563 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:11.572 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T18:43:11.581 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:11.591 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crash", 2024-04-23T18:43:11.767 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:11.776 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:11.785 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:11.854 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:11.990 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:12.054 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:12.062 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:12.070 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:12.079 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:12.135 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:12.169 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:12.175 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:12.183 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:12.192 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:12.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:12.351 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:12.437 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:12.452 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:12.459 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:12.466 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:12.527 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:12.535 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:12.543 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:12.551 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:12.559 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:12.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:12.631 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:12.640 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:12.649 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:12.657 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:12.665 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:12.674 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:12.683 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:12.757 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:12.766 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:12.776 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:12.785 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:12.851 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:12.861 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:12.870 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:12.878 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:12.887 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:12.955 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:13.154 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:13.162 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:13.169 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:13.177 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:13.237 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:13.245 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:13.253 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:13.262 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:13.269 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:13.310 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:13.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:13.373 INFO:teuthology.orchestra.run.smithi050.stdout: "retain_interval": { 2024-04-23T18:43:13.381 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "retain_interval", 2024-04-23T18:43:13.390 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:43:13.402 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:13.410 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:13.420 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "31536000", 2024-04-23T18:43:13.495 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:13.504 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:13.512 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:13.521 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T18:43:13.587 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:13.595 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:13.603 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:13.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:13.621 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_recent_interval": { 2024-04-23T18:43:13.668 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_recent_interval", 2024-04-23T18:43:13.740 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:43:13.748 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:13.755 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:13.764 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "1209600", 2024-04-23T18:43:13.772 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:13.781 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:13.789 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:13.854 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T18:43:13.862 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:13.872 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:13.881 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:13.950 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:13.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:13.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:13.976 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "dashboard", 2024-04-23T18:43:13.984 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:14.047 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:14.055 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:14.063 INFO:teuthology.orchestra.run.smithi050.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T18:43:14.071 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T18:43:14.078 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:14.085 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:14.092 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:14.201 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:43:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:14.267 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:14.346 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:14.357 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:14.364 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:14.371 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:14.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:14.435 INFO:teuthology.orchestra.run.smithi050.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T18:43:14.489 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T18:43:14.497 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:14.504 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:14.513 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:14.519 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:14.527 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:14.535 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:14.629 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:14.635 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:14.641 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:14.647 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:14.702 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:14.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:14.716 INFO:teuthology.orchestra.run.smithi050.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T18:43:14.723 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T18:43:14.730 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:14.790 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:14.838 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:14.878 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:14.886 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:14.894 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:14.902 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:14.911 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:14.919 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:14.988 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:14.996 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:15.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:15.013 INFO:teuthology.orchestra.run.smithi050.stdout: "AUDIT_API_ENABLED": { 2024-04-23T18:43:15.070 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T18:43:15.078 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:15.087 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:15.097 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:15.127 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:15.134 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:15.191 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:15.200 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:15.209 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:15.220 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:15.230 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:15.238 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:15.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:15.381 INFO:teuthology.orchestra.run.smithi050.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T18:43:15.396 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T18:43:15.404 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:15.410 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:15.463 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:15.473 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:15.483 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:15.491 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:15.499 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:15.507 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:15.588 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:15.597 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:15.605 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:15.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:15.654 INFO:teuthology.orchestra.run.smithi050.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T18:43:15.662 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T18:43:15.670 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:15.767 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:15.775 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:15.783 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:15.794 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:15.863 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:15.874 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:15.882 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:15.888 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:15.895 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:15.959 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:16.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:16.012 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T18:43:16.019 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T18:43:16.027 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:16.036 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:16.043 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:16.049 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:16.119 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:16.127 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:16.134 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T18:43:16.282 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T18:43:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:16.374 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:16.380 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:16.387 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:16.436 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:16.443 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:16.449 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:16.456 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:16.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:16.509 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T18:43:16.516 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T18:43:16.523 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:16.529 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:16.572 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:16.654 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:16.661 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:16.668 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:16.674 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:16.681 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:16.688 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:16.696 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:16.758 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:16.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:16.778 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T18:43:16.786 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T18:43:16.846 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:16.893 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:16.901 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:16.909 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:16.917 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:16.970 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:17.021 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:17.030 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:17.047 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:17.055 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:17.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:17.073 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T18:43:17.180 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T18:43:17.199 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:17.208 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:17.216 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:17.280 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:17.289 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:17.298 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:17.306 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:17.374 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:17.422 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:17.475 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T18:43:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T18:43:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:17.577 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:17.585 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:17.594 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:17.603 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:17.664 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:17.672 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:17.680 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:17.688 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:17.695 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:17.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:17.805 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T18:43:17.813 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T18:43:17.821 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:17.830 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:17.838 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:17.846 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:17.945 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:17.954 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:17.963 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:17.972 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:18.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:18.103 INFO:teuthology.orchestra.run.smithi050.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T18:43:18.111 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T18:43:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:18.229 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:18.246 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:18.254 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:18.261 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:18.270 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:18.277 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:18.342 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:18.349 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:18.356 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:18.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:18.418 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T18:43:18.425 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T18:43:18.431 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:18.438 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:18.445 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "admin", 2024-04-23T18:43:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:18.577 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:18.584 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:18.591 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:18.598 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:18.605 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:18.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:18.679 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T18:43:18.686 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T18:43:18.729 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:18.777 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:18.786 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:18.793 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:18.801 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:18.808 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:18.882 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:18.954 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:18.961 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:18.992 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:18.999 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:19.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:19.011 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_API_URL": { 2024-04-23T18:43:19.017 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_API_URL", 2024-04-23T18:43:19.129 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:19.137 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:19.145 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:19.153 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:19.210 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:19.254 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:19.263 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:19.271 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:19.279 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:19.286 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:19.337 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:19.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.854+0000 7fe5569f7640 1 -- 172.21.15.50:0/2793906102 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe5380a3d20 msgr2=0x7fe5380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:43:19.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.854+0000 7fe5569f7640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe5380a3d20 0x7fe5380a61e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe5400058c0 tx=0x7fe540005850 comp rx=0 tx=0).stop 2024-04-23T18:43:19.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.854+0000 7fe5569f7640 1 -- 172.21.15.50:0/2793906102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550106190 msgr2=0x7fe550198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:43:19.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.854+0000 7fe5569f7640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550106190 0x7fe550198c70 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fe54400d770 tx=0x7fe54400dc30 comp rx=0 tx=0).stop 2024-04-23T18:43:19.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.854+0000 7fe5569f7640 1 -- 172.21.15.50:0/2793906102 shutdown_connections 2024-04-23T18:43:19.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.854+0000 7fe5569f7640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe5380a3d20 0x7fe5380a61e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:43:19.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.854+0000 7fe5569f7640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe55019cff0 0x7fe55019d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:43:19.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.854+0000 7fe5569f7640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550106190 0x7fe550198c70 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:43:19.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.854+0000 7fe5569f7640 1 --2- 172.21.15.50:0/2793906102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe550101110 0x7fe550198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:43:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.854+0000 7fe5569f7640 1 -- 172.21.15.50:0/2793906102 >> 172.21.15.50:0/2793906102 conn(0x7fe550077fb0 msgr2=0x7fe55006a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:43:19.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.854+0000 7fe5569f7640 1 -- 172.21.15.50:0/2793906102 shutdown_connections 2024-04-23T18:43:19.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:40:24.854+0000 7fe5569f7640 1 -- 172.21.15.50:0/2793906102 wait complete. 2024-04-23T18:43:19.586 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-04-23T18:43:19.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:19.605 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T18:43:19.614 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T18:43:19.680 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:19.689 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:19.698 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:19.707 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "admin", 2024-04-23T18:43:19.715 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:19.724 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:19.732 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:19.802 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:19.810 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:19.849 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:19.859 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:19.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:19.919 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T18:43:19.929 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T18:43:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:19.945 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:19.953 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:20.019 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:20.031 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:20.039 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:20.048 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:20.056 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:20.064 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:20.073 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:20.161 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:20.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:20.184 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T18:43:20.192 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T18:43:20.252 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:20.260 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:20.268 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:20.276 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:20.284 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:20.316 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:20.370 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:20.378 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:20.386 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:20.403 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:20.410 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:20.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T18:43:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T18:43:20.475 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:20.482 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:20.489 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:20.543 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:20.553 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:20.560 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:20.568 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:20.576 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:20.604 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:20.653 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:20.694 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:20.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:20.709 INFO:teuthology.orchestra.run.smithi050.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T18:43:20.716 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T18:43:20.722 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:20.729 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:20.805 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:20.815 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:20.822 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:20.828 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:20.888 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:20.895 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:20.903 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:20.912 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:20.922 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:20.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:21.060 INFO:teuthology.orchestra.run.smithi050.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T18:43:21.074 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T18:43:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:21.092 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:21.101 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:21.110 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "[]", 2024-04-23T18:43:21.119 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:21.183 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:21.191 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:21.200 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:21.209 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:21.268 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:21.277 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:21.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:21.295 INFO:teuthology.orchestra.run.smithi050.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T18:43:21.327 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T18:43:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:21.404 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:21.413 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:21.421 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "{}", 2024-04-23T18:43:21.430 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:21.440 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:21.448 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:21.457 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:21.510 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:21.519 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:21.527 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:21.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:21.588 INFO:teuthology.orchestra.run.smithi050.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T18:43:21.601 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T18:43:21.609 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:21.617 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:21.627 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:21.682 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:21.736 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:21.745 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:21.755 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:21.765 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:21.800 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:21.809 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:21.819 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:21.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:21.885 INFO:teuthology.orchestra.run.smithi050.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T18:43:21.892 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T18:43:21.900 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:21.959 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:21.968 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:21.976 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:21.984 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:21.993 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:22.111 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:22.119 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:22.128 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:22.136 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:22.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:22.152 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T18:43:22.159 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T18:43:22.166 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:22.174 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:22.183 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:22.190 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:22.255 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:22.263 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:22.271 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:22.280 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:22.287 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:22.346 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T18:43:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T18:43:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:22.442 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:22.450 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:22.459 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:22.468 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:22.476 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:22.484 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:22.538 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:22.546 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:22.553 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:22.561 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:22.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:22.672 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T18:43:22.720 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T18:43:22.727 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:22.734 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:22.742 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:22.749 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:22.756 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:22.763 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:22.771 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:22.779 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:22.786 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:22.792 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:22.859 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:22.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:22.912 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T18:43:22.920 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T18:43:22.929 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:22.988 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:23.049 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:23.059 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:23.068 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:23.077 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:23.086 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:23.094 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:23.103 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:23.112 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:23.123 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:23.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:23.140 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T18:43:23.193 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T18:43:23.201 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:23.210 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:23.243 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:23.252 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:23.293 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:23.361 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:23.369 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:23.379 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:23.388 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:23.396 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:23.405 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:23.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T18:43:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T18:43:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:23.450 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:23.564 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:23.573 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:23.581 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:23.589 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:23.649 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:23.698 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:23.707 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:23.716 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:23.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:23.733 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T18:43:23.742 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T18:43:23.751 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:23.759 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:23.768 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:23.776 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:23.886 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:23.894 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:23.957 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T18:43:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T18:43:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:24.123 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:24.131 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:24.139 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:24.147 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:24.154 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:24.163 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:24.222 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:24.240 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:24.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:24.256 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T18:43:24.317 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T18:43:24.362 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:24.381 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:24.389 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:24.397 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T18:43:24.406 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:24.413 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:24.422 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:24.430 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:24.439 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:24.450 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T18:43:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T18:43:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:24.549 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:24.612 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:24.660 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:43:24.668 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:24.677 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:24.685 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:24.693 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:24.699 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:24.707 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:24.715 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T18:43:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T18:43:24.796 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:24.803 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:24.810 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:24.818 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8", 2024-04-23T18:43:24.825 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T18:43:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T18:43:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "45", 2024-04-23T18:43:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:25.144 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:25.183 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:25.265 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:25.328 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:25.339 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:25.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:25.355 INFO:teuthology.orchestra.run.smithi050.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T18:43:25.364 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T18:43:25.374 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:25.383 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:25.392 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:25.456 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:25.464 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:25.473 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:25.526 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:25.535 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:25.543 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:25.553 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:25.562 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:25.677 INFO:teuthology.orchestra.run.smithi050.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T18:43:25.685 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T18:43:25.692 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:25.700 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:25.763 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:25.770 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "admin", 2024-04-23T18:43:25.777 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:25.784 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:25.792 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:25.798 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:25.804 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:25.842 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:25.850 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:25.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:25.863 INFO:teuthology.orchestra.run.smithi050.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T18:43:25.870 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T18:43:25.925 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:25.994 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:26.002 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:26.072 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:26.080 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:26.086 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:26.093 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:26.099 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:26.107 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:26.114 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:26.122 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:26.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T18:43:26.190 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T18:43:26.197 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:26.205 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:26.212 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:26.280 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:26.319 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:26.353 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:26.360 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:26.366 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:26.372 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:26.378 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:26.385 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:26.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:26.399 INFO:teuthology.orchestra.run.smithi050.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T18:43:26.405 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T18:43:26.413 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:26.458 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:26.484 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:26.492 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:26.499 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:26.506 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:26.575 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:26.631 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:26.639 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:26.701 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:26.710 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:26.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:26.728 INFO:teuthology.orchestra.run.smithi050.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T18:43:26.737 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T18:43:26.745 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:26.755 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:26.766 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:26.775 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:43:26.816 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:26.829 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:26.840 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:26.852 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:26.860 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:26.919 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:26.928 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:26.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:26.985 INFO:teuthology.orchestra.run.smithi050.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T18:43:26.994 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T18:43:27.004 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:27.012 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:27.021 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:27.030 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:43:27.040 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:27.049 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:27.057 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:27.121 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:27.129 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:27.142 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:27.151 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:27.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:27.248 INFO:teuthology.orchestra.run.smithi050.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T18:43:27.298 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T18:43:27.307 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:27.316 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:27.323 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:27.332 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5", 2024-04-23T18:43:27.341 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:27.349 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:27.357 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:27.366 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:27.375 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:27.383 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:27.437 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:27.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:27.497 INFO:teuthology.orchestra.run.smithi050.stdout: "cross_origin_url": { 2024-04-23T18:43:27.506 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cross_origin_url", 2024-04-23T18:43:27.514 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:27.584 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:27.593 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:27.636 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:27.644 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:27.652 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:27.660 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:27.668 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:27.679 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:27.687 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:27.695 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:27.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:27.712 INFO:teuthology.orchestra.run.smithi050.stdout: "crt_file": { 2024-04-23T18:43:27.779 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crt_file", 2024-04-23T18:43:27.798 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:27.806 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:27.822 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:27.890 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:27.937 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:27.944 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:27.952 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:27.958 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:27.966 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:27.974 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:27.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:27.988 INFO:teuthology.orchestra.run.smithi050.stdout: "debug": { 2024-04-23T18:43:27.996 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "debug", 2024-04-23T18:43:28.055 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:28.062 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:28.103 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:28.113 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:28.119 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:28.126 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:28.132 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:28.195 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Enable/disable debug options", 2024-04-23T18:43:28.247 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:28.255 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:28.261 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:28.277 INFO:teuthology.orchestra.run.smithi050.stdout: "jwt_token_ttl": { 2024-04-23T18:43:28.283 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "jwt_token_ttl", 2024-04-23T18:43:28.289 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:28.302 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:28.310 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:28.362 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "28800", 2024-04-23T18:43:28.372 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:28.451 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:28.470 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:28.517 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:28.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:28.607 INFO:teuthology.orchestra.run.smithi050.stdout: "key_file": { 2024-04-23T18:43:28.616 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "key_file", 2024-04-23T18:43:28.626 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:28.636 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:28.644 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:28.654 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:28.723 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:28.735 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:28.749 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:28.758 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:28.852 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:28.902 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:28.947 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:28.965 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:28.977 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:28.986 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:28.995 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:29.003 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:29.040 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:29.049 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:29.058 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:29.066 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:29.119 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:29.130 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:29.140 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:29.149 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:29.157 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:29.264 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:29.322 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:29.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:29.368 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:29.376 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:29.388 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:29.397 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:29.406 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:29.415 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:29.424 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:29.434 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:29.442 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:29.498 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:29.508 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:29.518 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:29.526 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:29.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:29.595 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:29.647 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:29.689 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:29.704 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:29.715 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:29.730 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:29.853 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:29.900 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:29.986 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:30.030 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:30.048 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:30.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:30.352 INFO:teuthology.orchestra.run.smithi050.stdout: "motd": { 2024-04-23T18:43:30.381 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "motd", 2024-04-23T18:43:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "The message of the day", 2024-04-23T18:43:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:30.508 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:30.518 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:30.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:30.531 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_resolve_ip_addr": { 2024-04-23T18:43:30.538 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T18:43:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:30.710 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:30.721 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:30.735 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:30.744 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:30.760 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:30.770 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:30.784 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:30.793 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:30.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: "server_addr": { 2024-04-23T18:43:30.867 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_addr", 2024-04-23T18:43:30.877 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:30.886 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:30.894 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:30.963 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "::", 2024-04-23T18:43:31.034 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:31.046 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:31.069 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:31.080 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:31.119 INFO:teuthology.orchestra.run.smithi050.stdout: "server_port": { 2024-04-23T18:43:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_port", 2024-04-23T18:43:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:31.199 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:31.208 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:31.218 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8080", 2024-04-23T18:43:31.227 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:31.235 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:31.295 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:31.363 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:31.372 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:31.383 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:31.393 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:31.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:31.411 INFO:teuthology.orchestra.run.smithi050.stdout: "ssl": { 2024-04-23T18:43:31.420 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ssl", 2024-04-23T18:43:31.431 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:31.440 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:31.449 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:31.457 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:31.536 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:31.552 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:31.564 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:31.574 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:31.583 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:31.668 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: "ssl_server_port": { 2024-04-23T18:43:31.746 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ssl_server_port", 2024-04-23T18:43:31.757 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:31.767 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:31.776 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:31.784 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8443", 2024-04-23T18:43:31.792 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:31.803 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:31.815 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:31.871 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:31.881 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:31.891 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:31.901 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:31.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:32.022 INFO:teuthology.orchestra.run.smithi050.stdout: "standby_behaviour": { 2024-04-23T18:43:32.071 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "standby_behaviour", 2024-04-23T18:43:32.077 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:32.084 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:32.092 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:32.101 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "redirect", 2024-04-23T18:43:32.109 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:32.116 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:32.123 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:32.130 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:32.137 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect" 2024-04-23T18:43:32.145 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:32.201 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:32.228 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:32.236 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:32.244 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:32.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: "standby_error_status_code": { 2024-04-23T18:43:32.371 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "standby_error_status_code", 2024-04-23T18:43:32.380 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:32.387 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:32.393 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:32.400 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "500", 2024-04-23T18:43:32.406 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "400", 2024-04-23T18:43:32.413 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "599", 2024-04-23T18:43:32.420 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:32.427 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:32.433 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:32.441 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:32.499 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:32.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:32.578 INFO:teuthology.orchestra.run.smithi050.stdout: "url_prefix": { 2024-04-23T18:43:32.586 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "url_prefix", 2024-04-23T18:43:32.593 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:32.659 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:32.701 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:32.709 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:32.720 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:32.731 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:32.741 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:32.750 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:32.760 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:32.770 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:32.779 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:32.789 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:32.800 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:32.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:32.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:32.929 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "devicehealth", 2024-04-23T18:43:32.943 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:32.956 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:33.024 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:33.083 INFO:teuthology.orchestra.run.smithi050.stdout: "enable_monitoring": { 2024-04-23T18:43:33.094 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "enable_monitoring", 2024-04-23T18:43:33.103 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:33.112 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:33.122 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:33.131 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:33.142 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:33.151 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:33.160 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:33.174 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "monitor device health metrics", 2024-04-23T18:43:33.183 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:33.240 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:33.273 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:33.294 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:33.304 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:33.371 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:33.417 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:33.429 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:33.437 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:33.445 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:33.456 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:33.463 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:33.472 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:33.482 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:33.492 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:33.500 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:33.508 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:33.548 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:33.576 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:33.587 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:33.596 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:33.604 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:33.663 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:33.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:33.715 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:33.726 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:33.737 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:33.745 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:33.756 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:33.764 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:33.771 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:33.779 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:33.788 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:33.797 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:33.843 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:33.895 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:33.907 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:33.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:33.929 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:33.979 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:34.029 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:34.041 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:34.053 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:34.065 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:34.073 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:34.085 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:34.094 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:34.104 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:34.155 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:34.201 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:34.233 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:34.241 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:34.249 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:34.257 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:34.301 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:34.353 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:34.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:34.370 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:34.377 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:34.387 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:34.401 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:34.429 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "mark_out_threshold": { 2024-04-23T18:43:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mark_out_threshold", 2024-04-23T18:43:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:43:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "2419200", 2024-04-23T18:43:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:34.722 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:34.729 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T18:43:34.736 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:34.743 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:34.802 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:34.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:34.846 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": { 2024-04-23T18:43:34.854 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "pool_name", 2024-04-23T18:43:34.861 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:34.926 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:34.986 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:34.997 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "device_health_metrics", 2024-04-23T18:43:35.004 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:35.013 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:35.021 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:35.031 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T18:43:35.040 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:35.049 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:35.058 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:35.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:35.075 INFO:teuthology.orchestra.run.smithi050.stdout: "retention_period": { 2024-04-23T18:43:35.132 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "retention_period", 2024-04-23T18:43:35.225 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:43:35.237 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:35.254 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:35.263 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "15552000", 2024-04-23T18:43:35.324 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:35.369 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:35.378 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:35.393 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how long to retain device health metrics", 2024-04-23T18:43:35.423 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:35.438 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:35.444 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:35.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:35.461 INFO:teuthology.orchestra.run.smithi050.stdout: "scrape_frequency": { 2024-04-23T18:43:35.468 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "scrape_frequency", 2024-04-23T18:43:35.477 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:43:35.483 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "86400", 2024-04-23T18:43:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:35.713 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T18:43:35.762 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:35.772 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:35.779 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:35.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:35.793 INFO:teuthology.orchestra.run.smithi050.stdout: "self_heal": { 2024-04-23T18:43:35.800 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "self_heal", 2024-04-23T18:43:35.808 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:35.816 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:35.825 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:43:35.839 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:35.901 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:35.979 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:35.986 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T18:43:35.994 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:36.001 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:36.080 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:43:36.123 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:43:36.132 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:43:36.141 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:36.152 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:36.160 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:43:36.168 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:36.176 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:36.185 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:36.195 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T18:43:36.276 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:36.299 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:36.312 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:36.334 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_threshold": { 2024-04-23T18:43:36.391 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_threshold", 2024-04-23T18:43:36.445 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:43:36.458 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:36.466 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:36.479 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "7257600", 2024-04-23T18:43:36.489 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:36.500 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:36.509 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:36.520 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T18:43:36.531 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:36.540 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:36.555 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:36.631 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:36.670 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:36.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:36.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:36.704 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "diskprediction_local", 2024-04-23T18:43:36.781 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:36.827 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:36.842 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:36.854 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:36.862 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:37.140 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:37.150 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:37.159 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:37.168 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:37.177 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:37.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:37.196 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:37.204 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:37.221 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:37.228 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:37.275 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:37.316 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:37.324 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:37.331 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:37.339 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:37.450 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:37.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:37.466 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:37.472 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:37.479 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:37.486 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:37.492 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:37.500 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:37.507 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:37.513 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:37.550 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:37.676 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:37.688 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:37.755 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:37.761 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:37.771 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:37.779 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:37.787 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:37.917 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:37.977 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:37.989 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:38.006 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:38.016 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:38.095 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:38.197 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:38.205 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:38.216 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:38.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: "predict_interval": { 2024-04-23T18:43:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "predict_interval", 2024-04-23T18:43:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "86400", 2024-04-23T18:43:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:38.359 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:38.380 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:38.391 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:38.402 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: "predictor_model": { 2024-04-23T18:43:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "predictor_model", 2024-04-23T18:43:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "prophetstor", 2024-04-23T18:43:38.561 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:38.569 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:38.578 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:38.586 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:43:38.724 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:43:38.745 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:38.753 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:38.762 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:38.773 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:43:38.782 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:38.793 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:38.806 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:38.817 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:38.826 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:38.836 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:38.845 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:38.883 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:38.917 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:38.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:38.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:38.958 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "influx", 2024-04-23T18:43:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": false, 2024-04-23T18:43:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "influxdb python module not found", 2024-04-23T18:43:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: "batch_size": { 2024-04-23T18:43:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "batch_size", 2024-04-23T18:43:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:39.105 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5000", 2024-04-23T18:43:39.115 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T18:43:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "database": { 2024-04-23T18:43:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "database", 2024-04-23T18:43:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "ceph", 2024-04-23T18:43:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T18:43:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:39.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:39.507 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": { 2024-04-23T18:43:39.513 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "hostname", 2024-04-23T18:43:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:39.570 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:39.602 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:39.610 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:39.617 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:39.624 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:39.631 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:39.644 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "InfluxDB server hostname", 2024-04-23T18:43:39.652 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:39.659 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:39.667 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:39.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:43:39.733 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:43:39.752 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:43:39.759 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:39.768 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:39.776 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "30", 2024-04-23T18:43:39.822 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "5", 2024-04-23T18:43:39.837 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:39.846 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:39.855 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T18:43:39.864 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:39.872 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:39.880 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:39.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:39.897 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:39.906 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:39.915 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:39.923 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:39.973 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:39.982 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:39.991 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:40.000 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:40.008 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:40.120 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:40.130 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:40.141 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:40.150 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:40.160 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:40.176 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:40.185 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:40.196 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:40.216 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:40.233 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:40.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:40.350 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:40.359 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:40.369 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:40.377 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:40.386 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:40.446 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:40.495 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:40.504 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:40.532 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:40.540 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:40.550 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:40.557 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:40.565 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:40.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:40.579 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:40.587 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:40.594 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:40.633 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:40.640 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:40.648 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:40.655 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:40.682 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:40.731 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:40.772 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:40.814 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:40.822 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:40.838 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:40.845 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:40.852 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:40.860 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:40.868 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:40.875 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:40.882 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:40.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:40.924 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:40.932 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:40.939 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:40.963 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:41.070 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:41.162 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:41.168 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:41.175 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:41.182 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:41.190 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:41.198 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:41.207 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:41.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:41.225 INFO:teuthology.orchestra.run.smithi050.stdout: "password": { 2024-04-23T18:43:41.234 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "password", 2024-04-23T18:43:41.291 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:41.300 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:41.309 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:41.317 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:41.386 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:41.464 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:41.569 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:41.576 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "password of InfluxDB server user", 2024-04-23T18:43:41.584 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:41.592 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:41.603 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:41.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:41.622 INFO:teuthology.orchestra.run.smithi050.stdout: "port": { 2024-04-23T18:43:41.631 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "port", 2024-04-23T18:43:41.642 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:41.651 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:41.660 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:41.727 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8086", 2024-04-23T18:43:41.736 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:41.746 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:41.755 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:41.764 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "InfluxDB server port", 2024-04-23T18:43:41.848 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:41.924 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:41.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:41.953 INFO:teuthology.orchestra.run.smithi050.stdout: "ssl": { 2024-04-23T18:43:41.963 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ssl", 2024-04-23T18:43:41.972 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:41.982 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:42.000 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "false", 2024-04-23T18:43:42.009 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:42.020 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:42.068 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:42.077 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:43:42.086 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:42.095 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:42.148 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:42.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:42.220 INFO:teuthology.orchestra.run.smithi050.stdout: "threads": { 2024-04-23T18:43:42.229 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "threads", 2024-04-23T18:43:42.239 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:42.247 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:42.256 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:42.265 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5", 2024-04-23T18:43:42.275 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1", 2024-04-23T18:43:42.284 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "32", 2024-04-23T18:43:42.293 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:42.302 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T18:43:42.310 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:42.364 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:42.447 INFO:teuthology.orchestra.run.smithi050.stdout: "username": { 2024-04-23T18:43:42.475 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "username", 2024-04-23T18:43:42.531 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:42.578 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:42.605 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:42.614 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:42.623 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:42.632 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "username of InfluxDB server user", 2024-04-23T18:43:42.639 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:42.647 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:42.654 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:42.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:42.725 INFO:teuthology.orchestra.run.smithi050.stdout: "verify_ssl": { 2024-04-23T18:43:42.778 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "verify_ssl", 2024-04-23T18:43:42.785 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:42.792 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:42.800 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:42.889 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "true", 2024-04-23T18:43:42.916 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:42.925 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:43:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:43.051 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "insights", 2024-04-23T18:43:43.097 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:43.105 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:43.113 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:43.154 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:43.231 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:43.265 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:43.276 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:43.301 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:43.319 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:43.327 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:43.336 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:43.344 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:43.354 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:43.519 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:43.587 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:43.608 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:43.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:43.624 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:43.631 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:43.639 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:43.648 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:43.656 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:43.665 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:43.674 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:43.724 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:43.733 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:43.805 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:43.967 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:44.024 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:44.032 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:44.041 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:44.050 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:44.059 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:44.068 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:44.076 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:44.085 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:44.095 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:44.106 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:44.116 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:44.156 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:44.176 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:44.185 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:44.194 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:44.268 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:44.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:44.437 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:44.446 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:44.454 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:44.463 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:44.471 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:44.480 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:44.538 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:44.599 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:44.608 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:44.616 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:44.625 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:44.688 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:44.769 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:44.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:44.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:44.795 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "iostat", 2024-04-23T18:43:44.803 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:44.811 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:44.820 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:44.831 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:44.839 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:44.849 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:44.892 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:44.948 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:44.955 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:44.962 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:44.972 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:44.980 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:45.049 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:45.066 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:45.075 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:45.083 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:45.091 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:45.100 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:45.106 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:45.115 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:45.124 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:45.132 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:45.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:45.239 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:45.248 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:45.256 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:45.264 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:45.517 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:45.527 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:45.535 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:45.547 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:45.555 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:45.564 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:45.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:45.579 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:45.589 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:45.597 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:45.648 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:45.657 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:45.665 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:45.673 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:45.713 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:45.755 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:45.785 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:45.793 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:45.801 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:45.808 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:45.816 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:45.826 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:45.834 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:45.844 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:45.852 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:45.860 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:45.869 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:45.942 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:46.066 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:46.121 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:46.130 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:46.139 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:46.148 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:46.156 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:46.167 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:46.176 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:46.184 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:46.194 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:46.202 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:46.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:46.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:46.314 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "k8sevents", 2024-04-23T18:43:46.322 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:46.333 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:46.408 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:46.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_event_retention_days": { 2024-04-23T18:43:46.471 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ceph_event_retention_days", 2024-04-23T18:43:46.479 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:46.488 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:46.498 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:46.507 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "7", 2024-04-23T18:43:46.515 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:46.524 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:46.584 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:46.592 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T18:43:46.600 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:46.609 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:46.684 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:46.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:46.701 INFO:teuthology.orchestra.run.smithi050.stdout: "config_check_secs": { 2024-04-23T18:43:46.710 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "config_check_secs", 2024-04-23T18:43:46.719 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:46.788 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:46.851 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:46.859 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:43:46.868 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "10", 2024-04-23T18:43:46.877 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:46.885 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:46.894 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T18:43:46.903 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:46.913 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:46.922 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:46.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:46.940 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:46.997 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:47.008 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:47.015 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:47.022 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:47.093 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:47.136 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:47.185 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:47.193 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:47.201 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:47.209 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:47.217 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:47.226 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:47.235 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:47.243 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:47.251 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:47.259 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:47.305 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:47.313 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:47.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:47.330 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:47.376 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:47.385 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:47.445 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:47.454 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:47.463 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:47.470 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:47.478 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:47.487 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:47.495 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:47.503 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:47.511 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:47.518 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:47.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:47.578 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:47.585 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:47.591 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:47.599 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:47.625 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:47.668 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:47.698 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:47.705 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:47.712 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:47.719 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:47.725 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:47.733 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:47.740 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:47.747 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:47.754 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:47.761 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:47.768 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:47.823 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:47.831 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:47.838 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:47.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:47.925 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:47.971 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:48.020 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:48.030 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:48.039 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:48.048 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:48.057 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:48.066 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:48.074 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:48.083 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:48.093 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:48.102 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:48.111 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:48.170 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:48.196 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:48.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:48.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:48.266 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "localpool", 2024-04-23T18:43:48.290 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:48.345 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:48.354 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:48.363 INFO:teuthology.orchestra.run.smithi050.stdout: "failure_domain": { 2024-04-23T18:43:48.371 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "failure_domain", 2024-04-23T18:43:48.379 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:48.387 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:48.397 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:48.407 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "host", 2024-04-23T18:43:48.419 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:48.428 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:48.437 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:48.489 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "failure domain for any created local pool", 2024-04-23T18:43:48.497 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T18:43:48.558 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:48.566 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:48.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:48.751 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:48.760 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:48.792 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:48.800 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:48.808 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:48.817 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:48.826 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:48.834 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:48.843 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:48.854 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:48.927 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:48.936 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:48.945 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:48.953 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:49.012 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:49.070 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:49.174 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:49.270 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:49.319 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:49.325 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:49.332 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:49.403 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:49.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:49.495 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:49.502 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:49.509 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:49.516 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:49.523 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:49.530 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:49.604 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:49.611 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:49.617 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:49.624 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:49.631 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:49.690 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:49.697 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:49.705 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:49.768 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:49.820 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:49.883 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:49.890 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:49.898 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:49.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:49.916 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:49.925 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:49.934 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:49.943 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:49.952 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:49.960 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:49.969 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:50.024 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:50.033 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:50.042 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:50.052 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:50.118 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:50.170 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:50.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": { 2024-04-23T18:43:50.264 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "min_size", 2024-04-23T18:43:50.273 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:50.281 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:50.290 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:50.298 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:50.307 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "default min_size for any created local pool", 2024-04-23T18:43:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T18:43:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:50.422 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:50.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:50.438 INFO:teuthology.orchestra.run.smithi050.stdout: "num_rep": { 2024-04-23T18:43:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "num_rep", 2024-04-23T18:43:50.598 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:50.608 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "3", 2024-04-23T18:43:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:50.662 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "default replica count for any created local pool", 2024-04-23T18:43:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:50.685 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:50.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:50.749 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": { 2024-04-23T18:43:50.789 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "pg_num", 2024-04-23T18:43:50.796 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:43:50.864 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:50.934 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:50.978 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "128", 2024-04-23T18:43:50.985 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:50.992 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:51.008 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T18:43:51.015 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:51.031 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: "prefix": { 2024-04-23T18:43:51.095 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "prefix", 2024-04-23T18:43:51.154 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:51.165 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:51.172 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:51.180 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:51.187 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:51.247 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:51.303 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:51.310 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "name prefix for any created local pool", 2024-04-23T18:43:51.329 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:51.337 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:51.344 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:51.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:51.398 INFO:teuthology.orchestra.run.smithi050.stdout: "subtree": { 2024-04-23T18:43:51.406 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "subtree", 2024-04-23T18:43:51.416 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:51.425 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:51.433 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "rack", 2024-04-23T18:43:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T18:43:51.663 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T18:43:51.711 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:51.728 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:51.739 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:51.747 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:51.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:51.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:51.772 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mds_autoscaler", 2024-04-23T18:43:51.780 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:51.788 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:51.796 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:51.805 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:51.873 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:51.882 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:51.891 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:51.900 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:51.908 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:51.994 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:52.061 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:52.092 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:52.102 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:52.111 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:52.119 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:52.159 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:52.169 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:52.178 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:52.188 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:52.200 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:52.208 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:52.254 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:52.272 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:52.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:52.290 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:52.499 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:52.508 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:52.517 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:52.526 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:52.536 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:52.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:52.587 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:52.761 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:52.771 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:52.779 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:52.789 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:52.866 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:52.873 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:52.886 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:52.939 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:52.946 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:52.953 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:52.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:53.006 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:53.012 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:53.061 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:53.076 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:53.084 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:53.090 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:53.098 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:53.154 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:53.163 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:53.170 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:53.177 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:53.186 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:53.193 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:53.237 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:53.253 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:53.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:53.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:53.295 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mirroring", 2024-04-23T18:43:53.302 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:53.361 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:53.396 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:53.420 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:53.427 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:53.434 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:53.491 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:53.498 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:53.505 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:53.557 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:53.571 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:53.579 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:53.587 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:53.653 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:53.661 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:53.748 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:53.757 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:53.766 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:53.774 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:53.783 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:53.976 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:53.985 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:53.996 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:54.067 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:54.112 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:54.178 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:54.187 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:54.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:54.204 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:54.212 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:54.287 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:54.296 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:54.306 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:54.329 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:54.338 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:54.376 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:54.413 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:54.421 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:54.430 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:54.439 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:54.476 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:54.511 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:54.520 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:54.530 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:54.539 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:54.547 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:54.623 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:54.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:54.640 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:54.647 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:54.656 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:54.664 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:54.714 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:54.724 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:54.732 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:54.741 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:54.749 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:54.810 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:54.878 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:54.888 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:54.897 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:54.904 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:54.912 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:54.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:54.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:54.987 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "nfs", 2024-04-23T18:43:54.994 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:55.002 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:55.009 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:55.070 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:55.079 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:55.087 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:55.096 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:55.162 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:55.178 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:55.218 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:55.225 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:55.233 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:55.240 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:55.250 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:55.315 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:55.321 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:55.329 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:55.337 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:55.345 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:55.352 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:55.405 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:55.437 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:55.446 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:55.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:55.495 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:55.586 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:55.595 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:55.605 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:55.694 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:55.713 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:55.722 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:55.731 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:55.739 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:55.748 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:55.812 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:55.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:55.830 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:55.838 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:55.901 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:55.909 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:55.993 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:56.002 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:56.011 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:56.020 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:56.027 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:56.077 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:56.085 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:56.094 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:56.103 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:56.113 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:56.120 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:56.193 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:56.200 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:56.208 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:56.261 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:56.270 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:56.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:56.385 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:56.392 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:56.400 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:56.407 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:56.413 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:56.485 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:56.662 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:56.670 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:56.677 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:56.683 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:56.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:56.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:56.801 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "orchestrator", 2024-04-23T18:43:56.824 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:56.831 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:56.838 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:56.844 INFO:teuthology.orchestra.run.smithi050.stdout: "fail_fs": { 2024-04-23T18:43:56.897 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "fail_fs", 2024-04-23T18:43:56.904 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:43:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T18:43:57.118 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:57.125 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:57.187 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:57.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:57.258 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:57.265 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:57.273 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:57.280 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:57.327 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:57.333 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:57.340 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:57.393 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:57.401 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:57.522 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:57.532 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:57.541 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:43:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:43:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:57.908 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:57.978 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:58.012 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:58.085 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:58.134 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:58.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:58.151 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:43:58.159 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:43:58.166 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:58.257 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:58.265 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:58.274 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:43:58.285 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:58.294 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:58.353 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:58.413 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:58.455 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:58.465 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:58.473 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:58.482 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:58.640 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:58.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:43:58.757 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:43:58.771 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:43:58.778 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:58.787 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:43:58.851 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:58.858 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:58.865 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:43:58.902 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:43:58.909 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:58.962 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:58.980 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:58.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:58.994 INFO:teuthology.orchestra.run.smithi050.stdout: "orchestrator": { 2024-04-23T18:43:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "orchestrator", 2024-04-23T18:43:59.121 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:59.128 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:59.135 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:59.142 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:59.149 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:59.206 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:59.253 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:59.261 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm", 2024-04-23T18:43:59.268 INFO:teuthology.orchestra.run.smithi050.stdout: "rook", 2024-04-23T18:43:59.275 INFO:teuthology.orchestra.run.smithi050.stdout: "test_orchestrator" 2024-04-23T18:43:59.350 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:43:59.391 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Orchestrator backend", 2024-04-23T18:43:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:43:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:43:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:43:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:43:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:43:59.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:43:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "osd_perf_query", 2024-04-23T18:43:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:43:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:43:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:43:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:43:59.728 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:43:59.737 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:43:59.745 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:43:59.754 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:43:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:43:59.884 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:43:59.893 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:43:59.901 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:43:59.911 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:43:59.920 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:43:59.966 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:43:59.975 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:43:59.983 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:43:59.991 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:43:59.999 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:00.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:00.187 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:00.195 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:00.272 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:00.281 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:00.290 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:00.299 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:00.308 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:00.373 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:00.380 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:00.389 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:00.413 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:00.418 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:00.462 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:00.517 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:00.525 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:00.531 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:00.538 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:00.603 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:00.667 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:00.774 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:00.782 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:00.789 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:00.796 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:00.803 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:00.859 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:00.910 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:00.917 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:00.930 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:01.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:01.028 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:01.085 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:01.092 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:01.099 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:01.106 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:01.112 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:01.170 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:01.177 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:01.184 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:01.191 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:01.198 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:01.231 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:01.285 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:01.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:01.322 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:01.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:01.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:01.366 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "osd_support", 2024-04-23T18:44:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:01.572 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:01.583 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:01.594 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:01.604 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:01.664 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:01.677 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:01.687 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:01.697 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:01.706 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:01.716 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:01.768 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:01.784 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:01.797 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:01.810 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:01.871 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:01.883 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:01.972 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:01.984 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:01.993 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:02.003 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:02.015 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:02.075 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:02.290 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:02.302 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:02.312 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:02.325 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:02.387 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:02.397 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:02.464 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:02.500 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:02.513 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:02.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:02.538 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:02.587 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:02.598 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:02.607 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:02.614 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:02.626 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:02.680 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:02.740 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:02.752 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:02.763 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:02.776 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:02.843 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:02.862 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:02.933 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:02.943 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:02.951 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:02.958 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:02.966 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:03.040 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:03.048 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:03.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:03.066 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:03.074 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:03.127 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:03.181 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:03.196 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:03.202 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:03.210 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:03.273 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:03.283 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:03.340 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:03.350 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:03.359 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:03.366 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:03.375 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:03.416 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:03.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:03.507 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "pg_autoscaler", 2024-04-23T18:44:03.519 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:03.574 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:03.626 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:03.639 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:03.648 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:03.658 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:03.722 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:03.738 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:03.808 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:03.821 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:03.835 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:03.862 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:03.947 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:03.962 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:03.976 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:03.985 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:03.994 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:04.126 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:04.150 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:04.164 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:04.175 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:04.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:04.248 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:04.319 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:04.328 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:04.338 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:04.349 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:04.359 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:04.438 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:04.445 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:04.454 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:04.463 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:04.472 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:04.551 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:04.618 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:04.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:04.633 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:04.642 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:04.652 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:04.711 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:04.774 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:04.784 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:04.794 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:04.802 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:04.810 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:04.866 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:04.885 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:04.895 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:04.906 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:04.966 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:05.017 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:05.133 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:05.202 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:05.212 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:05.223 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:05.234 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:05.244 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:05.308 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:05.396 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:05.416 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:05.425 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:05.589 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:44:05.604 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:44:05.687 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:44:05.704 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:05.714 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:05.724 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:44:05.734 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:05.817 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:05.829 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:05.837 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:05.846 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:05.855 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:05.908 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:05.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:05.989 INFO:teuthology.orchestra.run.smithi050.stdout: "threshold": { 2024-04-23T18:44:05.998 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "threshold", 2024-04-23T18:44:06.010 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:44:06.022 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "3.0", 2024-04-23T18:44:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1.0", 2024-04-23T18:44:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "scaling threshold", 2024-04-23T18:44:06.277 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T18:44:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:06.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:06.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:06.504 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "progress", 2024-04-23T18:44:06.513 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:06.522 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:06.595 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:06.644 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_pg_recovery_event": { 2024-04-23T18:44:06.709 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "allow_pg_recovery_event", 2024-04-23T18:44:06.718 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:06.726 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:06.735 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:06.744 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:06.836 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:06.843 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:06.853 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:06.862 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T18:44:06.871 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:06.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": { 2024-04-23T18:44:07.005 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "enabled", 2024-04-23T18:44:07.076 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:07.085 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:44:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:07.224 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:07.239 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:07.248 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:07.259 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:07.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:07.307 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:07.368 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:07.382 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:07.404 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:07.483 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:07.496 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:07.549 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:07.565 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:07.581 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:07.595 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:07.609 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:07.670 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:07.682 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:07.694 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:07.703 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:07.713 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:07.722 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:07.772 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:07.787 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:07.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:07.817 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:07.890 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:07.906 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:07.967 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:07.979 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:07.988 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:07.999 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:08.007 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:08.065 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:08.074 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:08.084 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:08.098 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:08.110 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:08.226 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:08.287 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:08.297 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:08.306 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:08.316 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:08.389 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:08.577 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:08.598 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:08.685 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:08.693 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:08.704 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:08.712 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:08.732 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:08.785 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:08.795 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:08.803 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:08.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:08.876 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:08.923 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:09.029 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:09.047 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:09.054 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:09.074 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:09.150 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:09.163 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:09.174 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:09.186 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:09.197 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:09.254 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:09.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:09.325 INFO:teuthology.orchestra.run.smithi050.stdout: "max_completed_events": { 2024-04-23T18:44:09.333 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_completed_events", 2024-04-23T18:44:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:44:09.401 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:09.408 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:09.460 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "50", 2024-04-23T18:44:09.469 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:09.477 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:09.485 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:09.493 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "number of past completed events to remember", 2024-04-23T18:44:09.582 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:44:09.675 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:44:09.724 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:44:09.733 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:09.745 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:09.755 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5", 2024-04-23T18:44:09.814 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:09.849 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:09.902 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:09.909 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how long the module is going to sleep", 2024-04-23T18:44:09.920 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:09.934 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:09.944 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:10.003 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:10.014 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:10.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:10.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:10.092 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "prometheus", 2024-04-23T18:44:10.147 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:10.191 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:10.202 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:10.213 INFO:teuthology.orchestra.run.smithi050.stdout: "cache": { 2024-04-23T18:44:10.221 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cache", 2024-04-23T18:44:10.270 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:10.342 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:10.443 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:44:10.453 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:10.462 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:10.472 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:10.554 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:10.564 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:10.573 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:10.582 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:10.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:10.599 INFO:teuthology.orchestra.run.smithi050.stdout: "exclude_perf_counters": { 2024-04-23T18:44:10.671 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "exclude_perf_counters", 2024-04-23T18:44:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:10.694 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:10.704 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:10.761 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:44:10.817 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:10.872 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:10.885 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:10.898 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T18:44:10.910 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T18:44:10.922 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:10.973 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:10.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:10.991 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:11.005 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:11.015 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:11.023 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:11.081 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:11.094 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:11.107 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:11.115 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:11.124 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:11.186 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:11.231 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:11.239 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:11.251 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:11.259 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:11.270 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:11.336 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:11.346 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:11.359 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:11.371 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:11.383 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:11.544 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:11.615 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:11.768 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:11.830 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:11.839 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:11.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:11.859 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:11.868 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:11.987 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:11.997 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:12.007 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:12.016 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:12.088 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:12.148 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:12.369 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:12.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:12.472 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:12.482 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:12.493 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:12.562 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:12.610 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:12.664 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:12.671 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:12.682 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:12.691 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:12.703 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:12.768 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:12.775 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:12.784 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:12.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:12.805 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_stats_pools": { 2024-04-23T18:44:12.855 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rbd_stats_pools", 2024-04-23T18:44:12.913 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:12.922 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:12.931 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:12.939 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:13.118 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:13.136 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:13.171 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:13.178 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:13.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:13.255 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T18:44:13.267 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T18:44:13.277 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:44:13.289 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:13.344 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:13.422 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "300", 2024-04-23T18:44:13.432 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:13.445 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:13.459 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:13.502 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:13.550 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:13.609 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:13.619 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:13.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:13.642 INFO:teuthology.orchestra.run.smithi050.stdout: "scrape_interval": { 2024-04-23T18:44:13.651 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "scrape_interval", 2024-04-23T18:44:13.716 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:44:13.722 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:13.732 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:13.740 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "15.0", 2024-04-23T18:44:13.749 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:13.757 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:13.803 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:13.812 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:13.821 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:13.829 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:13.900 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:13.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:14.018 INFO:teuthology.orchestra.run.smithi050.stdout: "server_addr": { 2024-04-23T18:44:14.026 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_addr", 2024-04-23T18:44:14.035 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:14.044 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:14.054 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:14.135 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "::", 2024-04-23T18:44:14.144 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:14.155 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:14.164 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:14.173 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T18:44:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:14.278 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:14.287 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:14.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:14.305 INFO:teuthology.orchestra.run.smithi050.stdout: "server_port": { 2024-04-23T18:44:14.376 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_port", 2024-04-23T18:44:14.439 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:44:14.505 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:14.514 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:14.523 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "9283", 2024-04-23T18:44:14.534 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:14.543 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:14.612 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:14.621 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T18:44:14.630 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:14.639 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:14.648 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:14.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:14.754 INFO:teuthology.orchestra.run.smithi050.stdout: "stale_cache_strategy": { 2024-04-23T18:44:14.763 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "stale_cache_strategy", 2024-04-23T18:44:14.772 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:14.780 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:14.855 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "log", 2024-04-23T18:44:14.926 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:14.936 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:14.943 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:14.950 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:15.024 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:15.031 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:15.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:15.048 INFO:teuthology.orchestra.run.smithi050.stdout: "standby_behaviour": { 2024-04-23T18:44:15.055 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "standby_behaviour", 2024-04-23T18:44:15.105 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:15.148 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:15.155 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:15.162 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "default", 2024-04-23T18:44:15.169 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:15.238 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:15.309 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:15.363 INFO:teuthology.orchestra.run.smithi050.stdout: "default", 2024-04-23T18:44:15.370 INFO:teuthology.orchestra.run.smithi050.stdout: "error" 2024-04-23T18:44:15.377 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:15.385 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:15.414 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:15.481 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:15.490 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:15.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:15.508 INFO:teuthology.orchestra.run.smithi050.stdout: "standby_error_status_code": { 2024-04-23T18:44:15.516 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "standby_error_status_code", 2024-04-23T18:44:15.580 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:44:15.639 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:15.648 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:15.658 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "500", 2024-04-23T18:44:15.669 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "400", 2024-04-23T18:44:15.739 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "599", 2024-04-23T18:44:15.782 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:15.829 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:15.839 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:15.847 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:15.856 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:15.865 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:15.945 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:15.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:15.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:15.976 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rbd_support", 2024-04-23T18:44:15.985 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:15.995 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:16.052 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:16.061 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:16.070 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:16.080 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:16.088 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:16.213 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:16.223 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:16.232 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:16.241 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:16.249 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:16.300 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:16.308 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:16.317 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:16.405 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:16.454 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:16.463 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:16.472 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:16.481 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:16.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:16.632 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:16.688 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:16.695 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:16.709 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:16.715 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:16.759 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:16.766 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:16.773 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:16.781 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:16.788 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:16.843 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:16.894 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:16.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:16.909 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:16.917 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:16.982 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:17.024 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:17.081 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:17.095 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:17.103 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:17.110 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:17.117 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:17.167 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:17.175 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:17.182 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:17.190 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:17.198 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:17.231 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:17.303 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:17.310 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:17.318 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:17.325 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:17.332 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:17.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:17.437 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:17.445 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:17.454 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:17.462 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:17.469 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:17.612 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:17.653 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:17.660 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:17.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:17.677 INFO:teuthology.orchestra.run.smithi050.stdout: "max_concurrent_snap_create": { 2024-04-23T18:44:17.735 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_concurrent_snap_create", 2024-04-23T18:44:17.796 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:44:17.846 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:44:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:17.949 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:17.958 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:17.967 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:17.976 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:17.985 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:18.119 INFO:teuthology.orchestra.run.smithi050.stdout: "mirror_snapshot_schedule": { 2024-04-23T18:44:18.133 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T18:44:18.142 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:18.153 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:18.265 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:18.329 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:18.338 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:18.347 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:18.356 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:18.365 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:18.441 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:18.450 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:18.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:18.468 INFO:teuthology.orchestra.run.smithi050.stdout: "trash_purge_schedule": { 2024-04-23T18:44:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "trash_purge_schedule", 2024-04-23T18:44:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:18.589 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:18.600 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:18.608 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:18.616 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:18.624 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:18.669 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:18.720 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:18.730 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:18.739 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:18.749 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:18.761 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:18.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:18.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:18.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:18.845 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "restful", 2024-04-23T18:44:18.853 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:18.862 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:18.913 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:18.926 INFO:teuthology.orchestra.run.smithi050.stdout: "enable_auth": { 2024-04-23T18:44:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "enable_auth", 2024-04-23T18:44:18.943 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:19.008 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:19.048 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:19.095 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:44:19.103 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:19.109 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:19.116 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:19.125 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:19.182 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:19.190 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:19.197 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:19.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:19.213 INFO:teuthology.orchestra.run.smithi050.stdout: "key_file": { 2024-04-23T18:44:19.267 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "key_file", 2024-04-23T18:44:19.323 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:19.330 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:19.337 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:19.344 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:19.351 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:19.414 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:19.470 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:19.477 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:19.484 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:19.494 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:19.500 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:19.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:19.573 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:19.580 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:19.587 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:19.594 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:19.639 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:19.678 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:19.685 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:19.693 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:19.701 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:19.762 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:19.806 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:19.862 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:19.869 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:19.878 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:19.886 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:19.895 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:20.001 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:20.010 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:20.017 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:20.026 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:20.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:20.094 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:20.144 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:20.154 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:20.162 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:20.171 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:20.181 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:20.249 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:20.312 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:20.337 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:20.346 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:20.355 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:20.363 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:20.372 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:20.526 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:20.569 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:20.579 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:20.587 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:20.596 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:20.666 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:20.709 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:20.761 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:20.769 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:20.778 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:20.786 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:20.795 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:20.885 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:20.894 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:20.902 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:20.910 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:20.919 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:20.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:21.024 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:21.032 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:21.050 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:21.059 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:21.128 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:21.191 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:21.201 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:21.211 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:21.220 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:21.228 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:21.293 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:21.301 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:21.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:21.318 INFO:teuthology.orchestra.run.smithi050.stdout: "server_addr": { 2024-04-23T18:44:21.326 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_addr", 2024-04-23T18:44:21.375 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:21.422 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:21.431 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:21.438 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:21.446 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:21.506 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:21.560 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:21.623 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:21.630 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:21.638 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:21.645 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:21.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:21.734 INFO:teuthology.orchestra.run.smithi050.stdout: "server_port": { 2024-04-23T18:44:21.743 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_port", 2024-04-23T18:44:21.750 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:21.757 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:21.764 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:21.807 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:21.852 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:21.859 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:21.866 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:21.872 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:21.992 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:22.061 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:22.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:22.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:22.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:22.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:22.159 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rgw", 2024-04-23T18:44:22.166 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:22.173 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:22.181 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:22.188 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:22.244 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:22.302 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:22.311 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:22.318 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:22.326 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:22.478 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:22.485 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:22.495 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:22.503 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:22.513 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:22.583 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:22.593 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:22.601 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:22.611 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:22.621 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:22.630 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:22.685 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:22.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:22.704 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:22.711 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:22.767 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:22.835 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:22.911 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:22.920 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:22.929 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:22.938 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:22.948 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:23.025 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:23.034 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:23.044 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:23.052 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:23.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:23.070 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:23.124 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:23.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:23.144 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:23.154 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:23.162 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:23.239 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:23.290 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:23.300 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:23.310 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:23.319 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:23.329 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:23.379 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:23.389 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:23.398 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:23.407 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:23.415 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:23.480 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:23.489 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:23.576 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:23.639 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:23.694 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:23.704 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:23.712 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:23.720 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:23.729 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:23.789 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:23.796 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:23.803 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:23.819 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:23.927 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:23.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:23.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:23.952 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rook", 2024-04-23T18:44:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:24.035 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:24.094 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:24.101 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:24.108 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:24.115 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:24.123 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:24.214 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:24.223 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:24.238 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:24.245 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:24.295 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:24.351 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:24.359 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:24.366 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:24.373 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:24.436 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:24.552 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:24.558 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:24.565 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:24.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:24.637 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:24.645 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:24.653 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:24.660 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:24.667 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:24.785 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:24.792 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:24.800 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:24.806 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:24.814 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:24.878 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:25.167 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:25.202 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:25.232 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:25.242 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:25.252 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:25.318 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:25.369 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:25.379 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:25.388 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:25.398 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:25.407 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:25.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:25.482 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:25.491 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:25.499 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:25.508 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:25.569 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:25.620 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:25.637 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:25.711 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:25.777 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:25.786 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:25.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:25.804 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_class": { 2024-04-23T18:44:25.813 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "storage_class", 2024-04-23T18:44:25.868 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:25.878 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:25.890 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:25.899 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "local", 2024-04-23T18:44:25.908 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:25.917 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:25.977 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:25.986 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T18:44:25.996 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:26.005 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:26.063 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:26.113 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:26.160 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "selftest", 2024-04-23T18:44:26.192 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:26.249 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:26.266 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:26.274 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:26.282 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:26.289 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:26.349 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:26.356 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:26.363 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:26.370 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:26.436 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:26.486 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:26.535 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:26.544 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:26.551 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:26.560 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:26.567 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:26.642 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:26.649 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:26.660 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:26.667 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:26.674 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:26.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:26.794 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:26.801 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:26.809 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:26.817 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:26.884 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:26.963 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:27.018 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:27.068 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:27.076 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:27.084 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:27.091 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:27.099 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:27.106 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:27.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:27.121 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:27.129 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:27.183 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:27.228 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:27.238 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:27.247 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:27.256 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:27.304 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:27.379 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:27.435 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:27.497 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:27.506 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:27.521 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:27.531 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:27.540 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:27.549 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:27.558 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:27.567 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:27.578 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:27.620 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:27.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:27.684 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:27.698 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:27.745 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:27.754 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:27.802 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:27.851 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:27.912 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:27.920 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:27.931 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:27.939 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:27.948 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:27.957 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:27.967 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:27.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:27.984 INFO:teuthology.orchestra.run.smithi050.stdout: "roption1": { 2024-04-23T18:44:27.993 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "roption1", 2024-04-23T18:44:28.055 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:28.064 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:28.073 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:28.126 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:28.135 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:28.221 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:28.271 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:28.375 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:28.385 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:28.394 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:28.402 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "roption2": { 2024-04-23T18:44:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "roption2", 2024-04-23T18:44:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:28.497 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:28.545 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "xyz", 2024-04-23T18:44:28.556 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:28.564 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:28.631 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:28.639 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:28.699 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:28.760 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:28.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:28.859 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption1": { 2024-04-23T18:44:28.868 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption1", 2024-04-23T18:44:28.875 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:28.884 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:28.892 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:28.900 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:28.908 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:28.961 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:29.013 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:29.021 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:29.028 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:29.090 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:29.097 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:29.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:29.221 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption2": { 2024-04-23T18:44:29.288 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption2", 2024-04-23T18:44:29.296 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:44:29.304 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:29.319 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:29.326 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:29.333 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:29.341 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:29.480 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:29.551 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:29.609 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:29.671 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:29.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:29.688 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption3": { 2024-04-23T18:44:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption3", 2024-04-23T18:44:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:44:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:29.917 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:29.948 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:29.957 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:29.965 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:29.973 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:29.981 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:29.991 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:29.999 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:30.007 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:30.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption4": { 2024-04-23T18:44:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption4", 2024-04-23T18:44:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:30.315 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:30.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption5": { 2024-04-23T18:44:30.476 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption5", 2024-04-23T18:44:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:30.544 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:30.552 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:30.560 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:30.569 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:30.708 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:30.762 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:30.835 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:30.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption6": { 2024-04-23T18:44:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption6", 2024-04-23T18:44:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:30.887 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:30.948 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:44:30.991 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:30.999 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:31.009 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:31.067 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:31.125 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:31.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:31.248 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption7": { 2024-04-23T18:44:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption7", 2024-04-23T18:44:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:44:31.273 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:31.281 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:31.289 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:31.296 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1", 2024-04-23T18:44:31.304 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "42", 2024-04-23T18:44:31.313 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:31.372 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:31.405 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:31.413 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:31.437 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:31.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:31.508 INFO:teuthology.orchestra.run.smithi050.stdout: "testkey": { 2024-04-23T18:44:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "testkey", 2024-04-23T18:44:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:31.736 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:31.744 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:31.752 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:31.761 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:31.770 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:31.779 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:31.828 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:31.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:31.846 INFO:teuthology.orchestra.run.smithi050.stdout: "testlkey": { 2024-04-23T18:44:31.855 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "testlkey", 2024-04-23T18:44:31.914 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:31.954 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:32.003 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:32.054 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:32.064 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:32.073 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:32.083 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:32.092 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:32.102 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:32.113 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:32.122 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:32.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:32.178 INFO:teuthology.orchestra.run.smithi050.stdout: "testnewline": { 2024-04-23T18:44:32.219 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "testnewline", 2024-04-23T18:44:32.229 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:32.238 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:32.267 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:32.276 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:32.388 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:32.481 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:32.489 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:32.499 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:32.508 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:32.516 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:32.525 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:32.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:32.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:32.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:32.560 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "snap_schedule", 2024-04-23T18:44:32.621 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:32.630 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:32.639 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:32.648 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_m_granularity": { 2024-04-23T18:44:32.718 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "allow_m_granularity", 2024-04-23T18:44:32.800 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:32.918 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:32.927 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:32.936 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:32.944 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:32.953 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:32.962 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T18:44:32.970 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:32.978 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:32.987 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:32.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:33.083 INFO:teuthology.orchestra.run.smithi050.stdout: "dump_on_update": { 2024-04-23T18:44:33.091 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "dump_on_update", 2024-04-23T18:44:33.098 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:33.155 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:33.164 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:33.173 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:33.258 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:33.337 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:33.345 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:33.354 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "dump database to debug log on update", 2024-04-23T18:44:33.363 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:33.372 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:33.380 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:33.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:33.396 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:33.403 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:33.461 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:33.516 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:33.524 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:33.533 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:33.594 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:33.605 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:33.704 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:33.766 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:33.857 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:33.865 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:33.873 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:33.883 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:33.891 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:33.898 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:33.908 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:33.918 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:33.926 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:33.983 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:34.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:34.033 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:34.042 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:34.109 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:34.171 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:34.225 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:34.288 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:34.295 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:34.303 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:34.313 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:34.322 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:34.332 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:34.340 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:34.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:34.358 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:34.366 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:34.612 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:34.720 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:34.731 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:34.740 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:34.749 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:34.763 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:34.772 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:34.821 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:34.868 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:34.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:34.885 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:34.895 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:34.961 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:34.994 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:35.032 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:35.084 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:35.093 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:35.105 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:35.138 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:35.147 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:35.157 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:35.165 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:35.174 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:35.182 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:35.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:35.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:35.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:35.279 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "stats", 2024-04-23T18:44:35.331 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:35.340 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:35.370 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:35.408 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:35.508 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:35.515 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:35.523 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:35.530 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:35.552 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:35.568 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:35.625 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:35.709 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:35.777 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:35.787 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:35.839 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:35.849 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:35.922 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:35.931 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:35.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:35.948 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:35.956 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:35.964 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:35.971 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:35.979 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:35.987 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:36.103 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:36.112 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:36.119 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:36.184 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:36.194 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:36.242 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:36.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:36.363 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:36.371 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:36.387 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:36.396 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:36.406 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:36.414 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:36.424 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:36.432 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:36.483 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:36.532 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:36.540 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:36.550 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:36.614 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:36.622 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:36.662 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:36.668 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:36.752 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:36.761 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:36.772 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:36.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:36.789 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:36.799 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:36.809 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:36.819 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:36.828 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:36.838 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:37.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:37.156 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:37.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:37.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:37.184 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "status", 2024-04-23T18:44:37.194 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:37.203 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:37.222 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:37.261 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:37.309 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:37.318 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:37.329 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:37.339 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:37.444 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:37.491 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:37.543 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:37.552 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:37.684 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:37.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:37.701 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:37.777 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:37.930 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:37.940 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:37.948 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:37.957 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:37.965 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:37.973 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:37.982 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:37.990 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:37.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:38.046 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:38.101 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:38.118 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:38.162 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:38.170 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:38.217 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:38.379 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:38.386 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:38.395 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:38.403 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:38.572 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:38.581 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:38.749 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:38.758 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:38.766 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:38.777 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:38.786 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:38.805 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:38.814 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:38.823 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:38.881 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:38.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:38.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:38.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:38.975 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "telegraf", 2024-04-23T18:44:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:39.109 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "address": { 2024-04-23T18:44:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "address", 2024-04-23T18:44:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T18:44:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:44:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:44:39.515 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:44:39.586 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:39.594 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:39.603 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "15", 2024-04-23T18:44:39.612 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:39.621 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:39.632 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:39.640 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:39.648 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:39.656 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:39.665 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:39.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:39.767 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:39.775 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:39.784 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:39.831 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:39.915 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:39.976 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:40.037 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:40.047 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:40.055 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:40.071 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:40.080 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:40.088 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:40.096 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:40.104 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:40.112 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:40.167 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:40.174 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:40.182 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:40.253 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:40.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:40.302 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:40.310 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:40.370 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:40.389 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:40.398 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:40.406 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:40.414 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:40.421 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:40.429 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:40.438 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:40.446 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:40.502 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:40.558 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:40.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:40.574 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:40.628 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:40.636 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:40.675 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:40.777 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:40.854 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:40.864 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:40.873 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:40.882 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:40.891 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:40.900 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:40.909 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:40.918 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:40.927 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:40.991 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:41.124 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:41.134 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:41.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:41.247 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:41.307 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:41.318 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:41.327 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:41.335 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:41.344 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:41.361 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:41.370 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:41.378 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:41.386 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:41.448 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:41.457 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:41.466 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:41.514 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:41.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:41.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:41.605 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "telemetry", 2024-04-23T18:44:41.671 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:41.680 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:41.689 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:41.699 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_basic": { 2024-04-23T18:44:41.708 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_basic", 2024-04-23T18:44:41.716 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:41.725 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:41.734 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:41.743 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:44:41.750 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:41.804 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:41.813 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:41.822 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T18:44:41.830 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:41.881 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:41.909 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:41.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:42.021 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_crash": { 2024-04-23T18:44:42.028 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_crash", 2024-04-23T18:44:42.038 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:42.048 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:42.055 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:42.093 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:44:42.102 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:42.109 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:42.117 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:42.150 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T18:44:42.191 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:42.200 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:42.209 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:42.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:42.275 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_device": { 2024-04-23T18:44:42.310 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_device", 2024-04-23T18:44:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:42.411 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:42.419 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:42.427 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:44:42.436 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:42.445 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:42.452 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:42.460 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T18:44:42.467 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:42.474 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:42.482 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:42.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:42.549 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_ident": { 2024-04-23T18:44:42.557 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_ident", 2024-04-23T18:44:42.614 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:42.623 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:42.659 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:42.723 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:42.820 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:42.827 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:42.835 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:42.844 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T18:44:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:42.858 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:42.865 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:42.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:42.880 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_perf": { 2024-04-23T18:44:42.887 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_perf", 2024-04-23T18:44:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:43.009 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:43.042 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:43.089 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:43.153 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T18:44:43.161 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:43.170 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:43.181 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:43.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:43.200 INFO:teuthology.orchestra.run.smithi050.stdout: "contact": { 2024-04-23T18:44:43.209 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "contact", 2024-04-23T18:44:43.219 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:43.228 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:43.281 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:43.341 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:43.351 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:43.360 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:43.524 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:43.578 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:43.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:43.596 INFO:teuthology.orchestra.run.smithi050.stdout: "description": { 2024-04-23T18:44:43.604 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "description", 2024-04-23T18:44:43.613 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:43.622 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:43.632 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:43.641 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:43.650 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:43.705 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:43.749 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:43.759 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:43.768 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:43.808 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:43.932 INFO:teuthology.orchestra.run.smithi050.stdout: "device_url": { 2024-04-23T18:44:43.988 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "device_url", 2024-04-23T18:44:43.996 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:44.005 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:44.014 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:44.023 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T18:44:44.033 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:44.042 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:44.053 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:44.062 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:44.070 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:44.117 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:44.125 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:44.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:44.181 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": { 2024-04-23T18:44:44.190 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "enabled", 2024-04-23T18:44:44.199 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:44.241 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:44.302 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:44.310 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:44.318 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:44.325 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:44.466 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:44:44.474 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:44:44.482 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:44:44.546 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:44.553 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:44.625 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "24", 2024-04-23T18:44:44.680 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "8", 2024-04-23T18:44:44.756 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:44.763 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:44.770 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:44.777 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:44.785 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:44.792 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:44.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:44.812 INFO:teuthology.orchestra.run.smithi050.stdout: "last_opt_revision": { 2024-04-23T18:44:44.821 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "last_opt_revision", 2024-04-23T18:44:44.860 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:44:44.899 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:44.906 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:44.912 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "1", 2024-04-23T18:44:44.976 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:44.984 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:44.992 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:45.038 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:45.135 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:45.164 INFO:teuthology.orchestra.run.smithi050.stdout: "leaderboard": { 2024-04-23T18:44:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "leaderboard", 2024-04-23T18:44:45.180 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:45.197 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:45.258 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:45.515 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:45.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:45.597 INFO:teuthology.orchestra.run.smithi050.stdout: "leaderboard_description": { 2024-04-23T18:44:45.607 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "leaderboard_description", 2024-04-23T18:44:45.618 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:45.628 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:45.637 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:45.648 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:45.658 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:45.667 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:45.675 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:45.747 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:45.758 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:45.864 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:45.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:46.048 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:46.173 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:46.185 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:46.195 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:46.203 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:46.212 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:46.222 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:46.251 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:46.318 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:46.326 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:46.335 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:46.344 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:46.410 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:46.457 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:46.515 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:46.558 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:46.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:46.575 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:46.584 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:46.592 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:46.601 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:46.609 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:46.617 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:46.625 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:46.633 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:46.698 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:46.710 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:46.718 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:46.726 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:46.782 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:46.887 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:46.950 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:46.957 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:46.964 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:46.972 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:46.980 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:46.990 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:46.997 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:47.005 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:47.011 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:47.071 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:47.107 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:47.115 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:47.122 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:47.129 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:47.196 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:47.240 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:47.281 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:47.336 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:47.343 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:47.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:47.357 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:47.364 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:47.371 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:47.377 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:47.384 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:47.390 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:47.450 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:47.482 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:47.490 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:47.497 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:47.567 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:47.575 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:47.617 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:47.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:47.722 INFO:teuthology.orchestra.run.smithi050.stdout: "organization": { 2024-04-23T18:44:47.730 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "organization", 2024-04-23T18:44:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:47.745 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:47.754 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:47.763 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:47.778 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:47.788 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:47.797 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:47.855 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:47.891 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:47.901 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:47.911 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:47.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:47.976 INFO:teuthology.orchestra.run.smithi050.stdout: "proxy": { 2024-04-23T18:44:48.015 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "proxy", 2024-04-23T18:44:48.056 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:48.117 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:48.127 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:48.137 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:48.148 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:48.158 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:48.167 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:48.176 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:48.185 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:48.194 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:48.203 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:48.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:48.274 INFO:teuthology.orchestra.run.smithi050.stdout: "url": { 2024-04-23T18:44:48.285 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "url", 2024-04-23T18:44:48.294 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:48.357 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:48.390 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:48.446 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T18:44:48.509 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:48.519 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:48.528 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:48.536 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:48.545 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:48.554 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:48.563 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:48.572 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:48.582 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:48.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:48.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:48.666 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test_orchestrator", 2024-04-23T18:44:48.675 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:48.757 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:48.829 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:48.888 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:48.897 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:48.914 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:48.921 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:48.929 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:48.938 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:48.946 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:48.955 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:48.963 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:48.996 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:49.031 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:49.041 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:49.050 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:49.130 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:49.229 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:49.275 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:49.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:49.290 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:49.298 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:49.304 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:49.312 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:49.319 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:49.327 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:49.334 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:49.393 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:49.433 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:49.440 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:49.450 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:49.505 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:49.513 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:49.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:49.628 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:49.707 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:49.714 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:49.722 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:49.729 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:49.736 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:49.744 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:49.751 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:49.758 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:49.765 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:49.833 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:49.889 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:49.898 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:49.905 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:49.976 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:49.984 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:50.032 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:50.096 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:50.151 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:50.159 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:50.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:50.177 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:50.186 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:50.195 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:50.204 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:50.213 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:50.223 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:50.280 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:50.331 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:50.423 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:50.545 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:50.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "volumes", 2024-04-23T18:44:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:50.649 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:50.668 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:50.678 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:50.688 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:50.748 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:50.758 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:50.768 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:50.819 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:50.833 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:50.881 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:50.937 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:50.993 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:51.006 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:51.018 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:51.078 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:51.091 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:51.150 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:51.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:51.212 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:51.224 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:51.271 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:51.282 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:51.323 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:51.394 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:51.442 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:51.454 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:51.630 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:51.638 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:51.648 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:51.656 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:51.707 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:51.748 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:51.804 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:51.857 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:51.868 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:51.884 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:51.896 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:51.911 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:51.925 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:51.934 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:51.947 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:51.961 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:52.009 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:52.057 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:52.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:52.078 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:52.130 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:52.139 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:52.192 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:52.244 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:52.283 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:52.295 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:52.310 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:52.320 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "max_concurrent_clones": { 2024-04-23T18:44:52.463 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_concurrent_clones", 2024-04-23T18:44:52.474 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:44:52.502 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:52.537 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:52.549 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "4", 2024-04-23T18:44:52.601 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T18:44:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:52.771 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:52.785 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:52.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:52.815 INFO:teuthology.orchestra.run.smithi050.stdout: "periodic_async_work": { 2024-04-23T18:44:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "periodic_async_work", 2024-04-23T18:44:52.842 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:52.877 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:52.938 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:52.950 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:52.965 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:52.974 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:53.038 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:53.095 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Periodically check for async work", 2024-04-23T18:44:53.144 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:53.200 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:53.210 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:53.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:53.232 INFO:teuthology.orchestra.run.smithi050.stdout: "snapshot_clone_delay": { 2024-04-23T18:44:53.244 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "snapshot_clone_delay", 2024-04-23T18:44:53.254 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:44:53.263 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:53.273 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:53.284 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:44:53.333 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:53.372 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:53.381 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T18:44:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:53.444 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:53.476 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:53.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:53.568 INFO:teuthology.orchestra.run.smithi050.stdout: "snapshot_clone_no_wait": { 2024-04-23T18:44:53.578 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T18:44:53.587 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:53.597 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:53.607 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:53.616 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:44:53.626 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:53.637 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:53.646 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:53.656 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T18:44:53.707 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:53.716 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:53.727 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:53.798 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:53.975 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "zabbix", 2024-04-23T18:44:53.984 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:53.998 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:54.012 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:54.023 INFO:teuthology.orchestra.run.smithi050.stdout: "discovery_interval": { 2024-04-23T18:44:54.034 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "discovery_interval", 2024-04-23T18:44:54.045 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:44:54.059 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:54.070 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:54.134 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "100", 2024-04-23T18:44:54.196 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:54.208 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:54.221 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:54.268 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:54.278 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:54.318 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:54.361 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:54.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:54.419 INFO:teuthology.orchestra.run.smithi050.stdout: "identifier": { 2024-04-23T18:44:54.428 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "identifier", 2024-04-23T18:44:54.438 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:54.447 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:54.458 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:54.477 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:54.487 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:54.540 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:54.580 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:54.596 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:54.641 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:54.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:54.696 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:44:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:44:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:44:54.820 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:54.830 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:54.841 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:44:54.854 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:54.867 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:54.880 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:54.890 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:54.898 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:54.953 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:54.989 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:54.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:55.011 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:55.020 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:55.089 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:55.119 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:55.156 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:55.216 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:55.230 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:55.240 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:55.250 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:55.261 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:55.271 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:55.281 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:55.291 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:55.303 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:55.358 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:55.405 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:55.415 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:55.425 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:55.493 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:55.504 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:55.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:55.666 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:55.675 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:55.685 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:55.695 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:55.708 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:55.718 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:55.729 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:55.738 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:55.748 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:55.805 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:55.862 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:55.877 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:55.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:55.966 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:56.029 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:56.072 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:56.116 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:56.160 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:56.173 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:56.184 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:56.193 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:56.205 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:56.217 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:56.228 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:56.241 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:56.269 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:56.321 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:56.329 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:56.340 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:56.413 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:56.423 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:56.463 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:56.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:44:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:44:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:56.666 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:56.713 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:56.724 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:56.733 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:56.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:56.812 INFO:teuthology.orchestra.run.smithi050.stdout: "zabbix_host": { 2024-04-23T18:44:56.861 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "zabbix_host", 2024-04-23T18:44:56.897 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:57.112 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:57.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:57.165 INFO:teuthology.orchestra.run.smithi050.stdout: "zabbix_port": { 2024-04-23T18:44:57.180 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "zabbix_port", 2024-04-23T18:44:57.235 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:44:57.247 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:57.307 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:57.347 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10051", 2024-04-23T18:44:57.359 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:57.370 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:57.380 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:57.394 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:57.406 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:57.417 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:57.427 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:57.539 INFO:teuthology.orchestra.run.smithi050.stdout: "zabbix_sender": { 2024-04-23T18:44:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "zabbix_sender", 2024-04-23T18:44:57.629 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:44:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T18:44:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:57.779 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-23T18:44:57.904 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:44:57.967 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:58.013 INFO:teuthology.orchestra.run.smithi050.stdout: "modules": [ 2024-04-23T18:44:58.022 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm", 2024-04-23T18:44:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: "dashboard", 2024-04-23T18:44:58.117 INFO:teuthology.orchestra.run.smithi050.stdout: "iostat", 2024-04-23T18:44:58.126 INFO:teuthology.orchestra.run.smithi050.stdout: "nfs", 2024-04-23T18:44:58.162 INFO:teuthology.orchestra.run.smithi050.stdout: "restful", 2024-04-23T18:44:58.205 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_schedule" 2024-04-23T18:44:58.217 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:58.228 INFO:teuthology.orchestra.run.smithi050.stdout: "available_modules": [ 2024-04-23T18:44:58.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:44:58.252 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "alerts", 2024-04-23T18:44:58.268 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:44:58.282 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:44:58.296 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:44:58.308 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:44:58.362 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:44:58.421 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:44:58.479 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:58.487 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:58.496 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:44:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:58.613 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:58.684 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T18:44:58.694 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:58.702 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:58.710 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:58.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:58.729 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:44:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:44:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:58.755 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:58.765 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:58.818 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:44:58.896 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:58.906 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:58.957 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:44:59.019 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:44:59.033 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:44:59.106 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:44:59.160 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:44:59.171 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:44:59.182 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:44:59.241 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:44:59.255 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:59.269 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:59.282 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:59.291 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:59.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:59.312 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:44:59.365 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:44:59.421 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:44:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:44:59.540 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:59.614 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:44:59.689 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:44:59.701 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:44:59.710 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:44:59.719 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:44:59.730 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:44:59.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:44:59.749 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:44:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:44:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:44:59.819 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:44:59.878 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:44:59.929 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:44:59.938 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:44:59.952 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:45:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:45:00.133 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:45:00.144 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:45:00.161 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:45:00.170 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:45:00.181 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:00.191 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:00.200 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:00.213 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:00.221 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:00.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:00.302 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:45:00.344 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:45:00.353 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:00.362 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:00.423 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:00.430 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:00.506 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:00.538 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:00.546 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:00.554 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:00.561 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:00.568 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:00.575 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:00.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:00.588 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_destination": { 2024-04-23T18:45:00.596 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_destination", 2024-04-23T18:45:00.603 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:00.633 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:00.694 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:00.741 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:00.792 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Email address to send alerts to", 2024-04-23T18:45:00.857 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:00.866 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:00.878 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:00.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:00.894 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_from_name": { 2024-04-23T18:45:00.903 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_from_name", 2024-04-23T18:45:00.912 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:00.922 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:00.968 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:00.976 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "Ceph", 2024-04-23T18:45:01.029 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:01.078 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:01.087 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:01.095 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Email From: name", 2024-04-23T18:45:01.144 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:01.155 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:01.203 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:01.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:01.289 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_host": { 2024-04-23T18:45:01.296 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_host", 2024-04-23T18:45:01.303 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:01.312 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:01.321 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:01.333 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:01.342 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:01.351 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:01.361 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:01.408 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "SMTP server", 2024-04-23T18:45:01.453 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:01.462 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:01.469 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_password": { 2024-04-23T18:45:01.509 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_password", 2024-04-23T18:45:01.625 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:01.633 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:01.642 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:01.650 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:01.659 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:01.667 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:01.675 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:01.684 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Password to authenticate with", 2024-04-23T18:45:01.691 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:01.700 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:01.747 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:01.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:01.844 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_port": { 2024-04-23T18:45:01.852 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_port", 2024-04-23T18:45:01.916 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:01.925 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:01.969 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:02.012 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "465", 2024-04-23T18:45:02.019 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:02.029 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:02.038 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:02.047 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "SMTP port", 2024-04-23T18:45:02.056 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:02.110 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:02.120 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:02.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:02.137 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_sender": { 2024-04-23T18:45:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_sender", 2024-04-23T18:45:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:02.280 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:02.289 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:02.320 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:02.377 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:02.385 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "SMTP envelope sender", 2024-04-23T18:45:02.394 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:02.406 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:02.414 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:02.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:02.432 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_ssl": { 2024-04-23T18:45:02.441 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_ssl", 2024-04-23T18:45:02.450 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:02.459 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:02.507 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:02.553 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:02.589 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:02.598 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:02.653 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:02.662 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T18:45:02.703 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:02.762 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:02.769 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:02.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:02.783 INFO:teuthology.orchestra.run.smithi050.stdout: "smtp_user": { 2024-04-23T18:45:02.790 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "smtp_user", 2024-04-23T18:45:02.797 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:02.804 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:02.812 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:02.819 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:02.864 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:02.920 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:02.977 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:02.986 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "User to authenticate as", 2024-04-23T18:45:02.993 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:03.061 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:03.067 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:03.094 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:45:03.161 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:45:03.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:03.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:45:03.184 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "balancer", 2024-04-23T18:45:03.191 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:45:03.199 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:45:03.207 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:45:03.214 INFO:teuthology.orchestra.run.smithi050.stdout: "active": { 2024-04-23T18:45:03.221 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "active", 2024-04-23T18:45:03.236 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:03.292 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:03.328 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:03.364 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:03.373 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T18:45:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:03.518 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:03.527 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:03.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:03.544 INFO:teuthology.orchestra.run.smithi050.stdout: "begin_time": { 2024-04-23T18:45:03.552 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "begin_time", 2024-04-23T18:45:03.561 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:03.570 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:03.597 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:03.607 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0000", 2024-04-23T18:45:03.616 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:03.675 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:03.685 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:03.694 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T18:45:03.726 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:45:03.736 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:03.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:03.864 INFO:teuthology.orchestra.run.smithi050.stdout: "begin_weekday": { 2024-04-23T18:45:03.873 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "begin_weekday", 2024-04-23T18:45:03.882 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:45:03.889 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:03.897 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:03.905 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:45:03.914 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "0", 2024-04-23T18:45:03.922 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "6", 2024-04-23T18:45:03.931 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:03.994 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T18:45:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:45:04.061 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:04.068 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:04.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:04.173 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_compat_max_iterations": { 2024-04-23T18:45:04.216 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crush_compat_max_iterations", 2024-04-23T18:45:04.277 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:45:04.285 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:04.293 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:04.301 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "25", 2024-04-23T18:45:04.309 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1", 2024-04-23T18:45:04.317 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "250", 2024-04-23T18:45:04.324 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:04.333 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T18:45:04.340 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:04.348 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:04.356 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:04.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:04.436 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_compat_metrics": { 2024-04-23T18:45:04.443 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crush_compat_metrics", 2024-04-23T18:45:04.512 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:04.519 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:04.557 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:04.610 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T18:45:04.621 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:04.628 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:04.682 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:04.689 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T18:45:04.697 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T18:45:04.703 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:04.710 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:04.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:04.724 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_compat_step": { 2024-04-23T18:45:04.788 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crush_compat_step", 2024-04-23T18:45:04.836 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:45:04.842 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:04.848 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:04.855 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0.5", 2024-04-23T18:45:04.923 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "0.001", 2024-04-23T18:45:04.965 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "0.999", 2024-04-23T18:45:05.011 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:05.022 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "aggressiveness of optimization", 2024-04-23T18:45:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T18:45:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: "end_time": { 2024-04-23T18:45:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "end_time", 2024-04-23T18:45:05.086 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:05.094 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:05.217 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "2359", 2024-04-23T18:45:05.226 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:05.235 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:05.309 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:05.317 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T18:45:05.369 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T18:45:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "end_weekday": { 2024-04-23T18:45:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "end_weekday", 2024-04-23T18:45:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:45:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:05.512 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:45:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "0", 2024-04-23T18:45:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "6", 2024-04-23T18:45:05.602 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:05.609 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T18:45:05.616 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T18:45:05.682 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:05.688 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:05.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:45:05.796 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:45:05.805 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:05.813 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:05.820 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:05.828 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:05.836 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:05.844 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:05.853 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:05.862 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:45:05.921 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:45:05.961 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:45:05.970 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:45:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:45:06.038 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:45:06.047 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:45:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:45:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:06.220 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:06.290 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:06.430 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:06.439 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:06.477 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:06.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:06.588 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:45:06.598 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:45:06.606 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:06.614 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:06.621 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:06.629 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:45:06.638 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:06.646 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:06.704 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:06.713 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:45:06.759 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:45:06.767 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:45:06.776 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:45:06.828 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:45:06.837 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:45:06.879 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:07.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:07.009 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:45:07.016 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:45:07.023 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:07.029 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:07.085 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:07.093 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:07.185 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:07.192 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:07.259 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:07.267 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:07.294 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:07.347 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:07.360 INFO:teuthology.orchestra.run.smithi050.stdout: "min_score": { 2024-04-23T18:45:07.366 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "min_score", 2024-04-23T18:45:07.373 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:45:07.380 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:07.386 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:45:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:07.514 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:07.522 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:07.586 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T18:45:07.593 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:07.600 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:07.674 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:07.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:07.776 INFO:teuthology.orchestra.run.smithi050.stdout: "mode": { 2024-04-23T18:45:07.853 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mode", 2024-04-23T18:45:07.862 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:07.871 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:07.878 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:07.887 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "upmap", 2024-04-23T18:45:07.895 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:07.903 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:07.912 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:07.921 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-compat", 2024-04-23T18:45:07.929 INFO:teuthology.orchestra.run.smithi050.stdout: "none", 2024-04-23T18:45:07.980 INFO:teuthology.orchestra.run.smithi050.stdout: "read", 2024-04-23T18:45:08.042 INFO:teuthology.orchestra.run.smithi050.stdout: "upmap", 2024-04-23T18:45:08.051 INFO:teuthology.orchestra.run.smithi050.stdout: "upmap-read" 2024-04-23T18:45:08.060 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:08.134 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Balancer mode", 2024-04-23T18:45:08.143 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:08.227 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:08.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:08.246 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_ids": { 2024-04-23T18:45:08.255 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "pool_ids", 2024-04-23T18:45:08.264 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:08.272 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:08.281 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:08.291 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:08.300 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:08.349 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:08.357 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:08.402 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T18:45:08.410 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:08.418 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:08.488 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:45:08.601 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:45:08.615 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:45:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:08.636 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:08.644 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:45:08.652 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:08.660 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:08.668 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:08.676 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T18:45:08.732 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:08.757 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:08.803 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:08.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:08.819 INFO:teuthology.orchestra.run.smithi050.stdout: "upmap_max_deviation": { 2024-04-23T18:45:08.888 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "upmap_max_deviation", 2024-04-23T18:45:08.896 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:08.937 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:09.010 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:09.025 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5", 2024-04-23T18:45:09.033 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1", 2024-04-23T18:45:09.042 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:09.049 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:09.056 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T18:45:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T18:45:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:09.080 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:09.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:09.143 INFO:teuthology.orchestra.run.smithi050.stdout: "upmap_max_optimizations": { 2024-04-23T18:45:09.217 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "upmap_max_optimizations", 2024-04-23T18:45:09.225 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:45:09.233 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:09.295 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:09.304 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:45:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:09.399 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:09.407 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:09.416 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T18:45:09.424 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:09.433 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:09.444 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:09.452 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:45:09.461 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:45:09.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:09.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:45:09.540 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephadm", 2024-04-23T18:45:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:45:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:45:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:45:09.683 INFO:teuthology.orchestra.run.smithi050.stdout: "agent_down_multiplier": { 2024-04-23T18:45:09.692 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "agent_down_multiplier", 2024-04-23T18:45:09.744 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:45:09.808 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:09.817 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:09.826 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "3.0", 2024-04-23T18:45:09.834 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:09.843 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:09.852 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:09.861 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T18:45:09.868 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:09.875 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:09.929 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:09.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:09.976 INFO:teuthology.orchestra.run.smithi050.stdout: "agent_refresh_rate": { 2024-04-23T18:45:09.998 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "agent_refresh_rate", 2024-04-23T18:45:10.064 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:45:10.073 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:10.080 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:10.110 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "20", 2024-04-23T18:45:10.210 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:10.265 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:10.271 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:10.279 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T18:45:10.286 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:10.292 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:10.299 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:10.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:10.313 INFO:teuthology.orchestra.run.smithi050.stdout: "agent_starting_port": { 2024-04-23T18:45:10.383 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "agent_starting_port", 2024-04-23T18:45:10.390 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:10.502 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:10.555 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:10.597 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "4721", 2024-04-23T18:45:10.604 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:10.611 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:10.731 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:10.770 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T18:45:10.790 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:10.799 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:10.808 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:10.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:10.826 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_ptrace": { 2024-04-23T18:45:10.834 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "allow_ptrace", 2024-04-23T18:45:10.843 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:10.851 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:10.930 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:10.940 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:11.022 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:11.037 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:11.046 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:11.106 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T18:45:11.114 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T18:45:11.212 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:11.235 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:11.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:11.341 INFO:teuthology.orchestra.run.smithi050.stdout: "autotune_interval": { 2024-04-23T18:45:11.353 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "autotune_interval", 2024-04-23T18:45:11.362 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:45:11.370 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:11.378 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:11.386 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:45:11.395 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:11.555 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T18:45:11.563 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: "autotune_memory_target_ratio": { 2024-04-23T18:45:11.789 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T18:45:11.806 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:45:11.814 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:11.821 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:11.830 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0.7", 2024-04-23T18:45:11.839 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:11.847 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:11.855 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:11.863 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T18:45:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:12.009 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:12.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:12.092 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm_log_destination": { 2024-04-23T18:45:12.101 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephadm_log_destination", 2024-04-23T18:45:12.108 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:12.143 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: "file", 2024-04-23T18:45:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: "file,syslog", 2024-04-23T18:45:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "syslog" 2024-04-23T18:45:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T18:45:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:12.451 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:12.457 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:12.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:12.532 INFO:teuthology.orchestra.run.smithi050.stdout: "cgroups_split": { 2024-04-23T18:45:12.541 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cgroups_split", 2024-04-23T18:45:12.576 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:12.617 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:12.660 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:12.668 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:12.679 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:12.687 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:12.695 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:12.704 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T18:45:12.713 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:12.721 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:12.730 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:12.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:12.842 INFO:teuthology.orchestra.run.smithi050.stdout: "config_checks_enabled": { 2024-04-23T18:45:12.851 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "config_checks_enabled", 2024-04-23T18:45:12.914 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:12.951 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:12.960 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:13.102 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T18:45:13.118 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:13.126 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:13.135 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:13.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: "config_dashboard": { 2024-04-23T18:45:13.206 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "config_dashboard", 2024-04-23T18:45:13.219 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:13.259 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:13.266 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:13.330 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:13.337 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:13.344 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:13.434 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T18:45:13.441 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:13.448 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:13.454 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:13.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:13.526 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_alertmanager": { 2024-04-23T18:45:13.533 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_alertmanager", 2024-04-23T18:45:13.541 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:13.550 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:13.558 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:13.610 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T18:45:13.657 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:13.665 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:13.673 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:13.738 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Prometheus container image", 2024-04-23T18:45:13.779 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:13.844 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:13.855 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:13.949 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_base": { 2024-04-23T18:45:13.957 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_base", 2024-04-23T18:45:13.965 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:13.973 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:13.982 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:13.991 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T18:45:14.000 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:14.008 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:14.060 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:14.108 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Container image name, without the tag", 2024-04-23T18:45:14.117 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:14.125 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:14.197 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:14.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_elasticsearch": { 2024-04-23T18:45:14.291 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_elasticsearch", 2024-04-23T18:45:14.298 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:14.305 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:14.314 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:14.322 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T18:45:14.358 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:14.376 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:14.382 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:14.390 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "elasticsearch container image", 2024-04-23T18:45:14.398 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:14.423 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:14.475 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:14.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:14.510 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_grafana": { 2024-04-23T18:45:14.518 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_grafana", 2024-04-23T18:45:14.525 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:14.609 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:14.668 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:14.676 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T18:45:14.683 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:14.691 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:14.699 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:14.707 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Prometheus container image", 2024-04-23T18:45:14.715 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:14.723 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:14.730 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:14.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_haproxy": { 2024-04-23T18:45:14.858 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_haproxy", 2024-04-23T18:45:14.865 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:14.941 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:14.949 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T18:45:15.038 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:15.099 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:15.108 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:15.117 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "HAproxy container image", 2024-04-23T18:45:15.125 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:15.133 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:15.140 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:15.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:15.156 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_jaeger_agent": { 2024-04-23T18:45:15.167 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_jaeger_agent", 2024-04-23T18:45:15.223 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:15.230 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:15.283 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:15.292 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T18:45:15.376 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:15.386 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:15.398 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:15.433 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Jaeger agent container image", 2024-04-23T18:45:15.500 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:15.507 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:15.583 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:15.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:15.600 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_jaeger_collector": { 2024-04-23T18:45:15.608 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_jaeger_collector", 2024-04-23T18:45:15.617 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:15.625 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:15.633 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:15.641 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T18:45:15.686 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:15.740 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:15.747 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:15.798 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Jaeger collector container image", 2024-04-23T18:45:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:15.859 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:15.867 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:15.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:15.950 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_jaeger_query": { 2024-04-23T18:45:15.958 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_jaeger_query", 2024-04-23T18:45:15.965 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:15.972 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:15.980 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:15.988 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T18:45:15.995 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:16.004 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:16.012 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:16.071 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Jaeger query container image", 2024-04-23T18:45:16.116 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:16.127 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_keepalived": { 2024-04-23T18:45:16.287 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_keepalived", 2024-04-23T18:45:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T18:45:16.377 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:16.385 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:16.393 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:16.401 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Keepalived container image", 2024-04-23T18:45:16.409 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:16.445 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:16.455 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:16.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:16.516 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_loki": { 2024-04-23T18:45:16.589 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_loki", 2024-04-23T18:45:16.629 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:16.638 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:16.687 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:16.740 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T18:45:16.749 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:16.758 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:16.766 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:16.774 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Loki container image", 2024-04-23T18:45:16.781 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:16.790 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:16.799 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:16.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:16.820 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_node_exporter": { 2024-04-23T18:45:16.864 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_node_exporter", 2024-04-23T18:45:16.976 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:16.990 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:17.061 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:17.071 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T18:45:17.081 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:17.159 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:17.172 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:17.293 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Prometheus container image", 2024-04-23T18:45:17.301 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:17.308 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:17.316 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:17.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:17.334 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_nvmeof": { 2024-04-23T18:45:17.341 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_nvmeof", 2024-04-23T18:45:17.347 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:17.414 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:17.422 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:17.473 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T18:45:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:17.577 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:17.588 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:17.597 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Nvme-of container image", 2024-04-23T18:45:17.626 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:17.636 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:17.686 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:17.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:17.700 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_prometheus": { 2024-04-23T18:45:17.710 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_prometheus", 2024-04-23T18:45:17.719 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:17.778 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:17.788 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T18:45:17.844 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:17.916 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:17.922 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Prometheus container image", 2024-04-23T18:45:17.984 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_promtail": { 2024-04-23T18:45:18.156 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_promtail", 2024-04-23T18:45:18.164 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:18.174 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T18:45:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:18.202 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:18.254 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Promtail container image", 2024-04-23T18:45:18.263 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:18.308 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:18.315 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:18.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:18.383 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_samba": { 2024-04-23T18:45:18.420 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_samba", 2024-04-23T18:45:18.461 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T18:45:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Samba/SMB container image", 2024-04-23T18:45:18.577 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:18.583 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:18.590 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:18.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:18.690 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image_snmp_gateway": { 2024-04-23T18:45:18.697 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_image_snmp_gateway", 2024-04-23T18:45:18.765 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:18.774 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:18.783 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:18.791 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T18:45:18.852 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:18.857 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:18.865 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:18.872 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "SNMP Gateway container image", 2024-04-23T18:45:18.880 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:18.888 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:18.895 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:18.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:18.911 INFO:teuthology.orchestra.run.smithi050.stdout: "container_init": { 2024-04-23T18:45:18.969 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "container_init", 2024-04-23T18:45:18.997 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:19.067 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:19.076 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:19.130 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:19.139 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:19.147 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:19.206 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:19.305 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T18:45:19.314 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:19.322 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:19.332 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:19.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:19.350 INFO:teuthology.orchestra.run.smithi050.stdout: "daemon_cache_timeout": { 2024-04-23T18:45:19.361 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "daemon_cache_timeout", 2024-04-23T18:45:19.370 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:45:19.378 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:19.449 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:45:19.499 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:19.563 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:19.622 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:19.633 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T18:45:19.642 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:19.685 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:19.726 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:19.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:19.742 INFO:teuthology.orchestra.run.smithi050.stdout: "default_cephadm_command_timeout": { 2024-04-23T18:45:19.750 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T18:45:19.759 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:19.768 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:19.777 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:19.785 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "900", 2024-04-23T18:45:19.797 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:19.805 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:19.862 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:19.923 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T18:45:19.932 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:19.978 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:19.987 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:19.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:20.067 INFO:teuthology.orchestra.run.smithi050.stdout: "default_registry": { 2024-04-23T18:45:20.124 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "default_registry", 2024-04-23T18:45:20.133 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:20.140 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:20.147 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:20.155 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "docker.io", 2024-04-23T18:45:20.162 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:20.169 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:20.176 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:20.185 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T18:45:20.248 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:20.255 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:20.298 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:20.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:20.368 INFO:teuthology.orchestra.run.smithi050.stdout: "device_cache_timeout": { 2024-04-23T18:45:20.392 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "device_cache_timeout", 2024-04-23T18:45:20.399 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:45:20.405 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:20.484 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:20.492 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "1800", 2024-04-23T18:45:20.500 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:20.508 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:20.516 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:20.523 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "seconds to cache device inventory", 2024-04-23T18:45:20.531 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:20.537 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:20.546 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:20.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:20.613 INFO:teuthology.orchestra.run.smithi050.stdout: "device_enhanced_scan": { 2024-04-23T18:45:20.687 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "device_enhanced_scan", 2024-04-23T18:45:20.715 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:20.783 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:20.792 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:20.801 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:20.869 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:21.012 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T18:45:21.023 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:21.031 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:21.038 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:21.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:21.051 INFO:teuthology.orchestra.run.smithi050.stdout: "facts_cache_timeout": { 2024-04-23T18:45:21.059 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "facts_cache_timeout", 2024-04-23T18:45:21.067 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:45:21.119 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:21.172 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:21.182 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:45:21.249 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:21.258 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:21.266 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:21.312 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "seconds to cache host facts data", 2024-04-23T18:45:21.379 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:21.393 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:21.401 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:21.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:21.417 INFO:teuthology.orchestra.run.smithi050.stdout: "grafana_dashboards_path": { 2024-04-23T18:45:21.425 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "grafana_dashboards_path", 2024-04-23T18:45:21.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:21.441 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:21.449 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:21.458 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T18:45:21.518 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:21.572 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:21.578 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:21.623 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T18:45:21.669 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:21.677 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:21.707 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:21.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:21.768 INFO:teuthology.orchestra.run.smithi050.stdout: "host_check_interval": { 2024-04-23T18:45:21.777 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "host_check_interval", 2024-04-23T18:45:21.785 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:45:21.792 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:21.802 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:21.809 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:45:21.817 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:21.825 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:21.833 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:21.889 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to perform a host check", 2024-04-23T18:45:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: "hw_monitoring": { 2024-04-23T18:45:22.073 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "hw_monitoring", 2024-04-23T18:45:22.126 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:22.136 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:22.145 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:22.155 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:22.163 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:22.171 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:22.180 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:22.188 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T18:45:22.198 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:22.257 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:22.267 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:22.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:22.315 INFO:teuthology.orchestra.run.smithi050.stdout: "inventory_list_all": { 2024-04-23T18:45:22.378 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "inventory_list_all", 2024-04-23T18:45:22.437 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:22.445 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:22.507 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:22.565 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:22.574 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:22.583 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:22.590 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:22.597 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T18:45:22.605 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:22.612 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:22.619 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:22.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:22.671 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:45:22.678 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:45:22.748 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:22.773 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:22.781 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:22.852 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:22.859 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:22.866 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:22.934 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:22.941 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:45:22.948 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:45:22.955 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:45:22.961 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:45:22.970 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:45:22.979 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:45:22.987 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:23.039 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:23.046 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:23.103 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:23.139 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:23.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:23.218 INFO:teuthology.orchestra.run.smithi050.stdout: "log_refresh_metadata": { 2024-04-23T18:45:23.228 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_refresh_metadata", 2024-04-23T18:45:23.237 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:23.293 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:23.380 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:23.386 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:23.393 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:23.401 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:23.411 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:23.419 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T18:45:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:23.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:45:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:45:23.594 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:23.659 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:23.677 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:23.718 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:23.780 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:23.790 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:23.799 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T18:45:23.807 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:23.909 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:23.920 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:23.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:23.935 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:45:23.941 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:45:23.949 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:24.073 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:45:24.124 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:24.182 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:24.190 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:24.223 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:45:24.296 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:45:24.305 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:45:24.313 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:45:24.322 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:45:24.331 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:45:24.339 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:24.348 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:24.357 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:24.367 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:24.375 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:24.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:45:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:45:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:24.550 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:24.558 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:24.589 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:24.648 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:24.654 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:24.707 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T18:45:24.822 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T18:45:24.830 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:24.872 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:24.881 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:24.888 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T18:45:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T18:45:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T18:45:25.164 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:25.173 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:25.205 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:25.273 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "*", 2024-04-23T18:45:25.314 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:25.323 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:25.405 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:25.411 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T18:45:25.418 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:25.464 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:25.513 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:25.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:25.558 INFO:teuthology.orchestra.run.smithi050.stdout: "max_count_per_host": { 2024-04-23T18:45:25.567 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_count_per_host", 2024-04-23T18:45:25.577 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:25.594 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:45:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:25.657 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:25.664 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:25.721 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "max number of daemons per service per host", 2024-04-23T18:45:25.731 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:25.739 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:25.807 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:25.857 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd_draining_count": { 2024-04-23T18:45:25.920 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_osd_draining_count", 2024-04-23T18:45:25.929 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:25.937 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:25.947 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:25.958 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:45:25.967 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:25.977 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:25.986 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:25.993 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T18:45:26.046 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:26.054 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:26.122 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:26.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:26.139 INFO:teuthology.orchestra.run.smithi050.stdout: "migration_current": { 2024-04-23T18:45:26.205 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "migration_current", 2024-04-23T18:45:26.215 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:26.238 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:26.287 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:26.293 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:26.300 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:26.307 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:26.314 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:26.321 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "internal - do not modify", 2024-04-23T18:45:26.328 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:26.335 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:26.342 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:26.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:26.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mode": { 2024-04-23T18:45:26.433 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mode", 2024-04-23T18:45:26.472 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:26.516 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:26.524 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:26.533 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "root", 2024-04-23T18:45:26.557 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:26.564 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:26.612 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:26.621 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm-package", 2024-04-23T18:45:26.630 INFO:teuthology.orchestra.run.smithi050.stdout: "root" 2024-04-23T18:45:26.710 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:26.719 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T18:45:26.727 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:26.735 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:26.744 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:26.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:26.815 INFO:teuthology.orchestra.run.smithi050.stdout: "oob_default_addr": { 2024-04-23T18:45:26.863 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "oob_default_addr", 2024-04-23T18:45:26.872 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:26.940 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:26.969 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:26.978 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "169.254.1.1", 2024-04-23T18:45:27.010 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:27.018 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:27.122 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:27.132 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T18:45:27.143 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:27.151 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:27.161 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:27.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:27.207 INFO:teuthology.orchestra.run.smithi050.stdout: "prometheus_alerts_path": { 2024-04-23T18:45:27.215 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "prometheus_alerts_path", 2024-04-23T18:45:27.224 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:27.281 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:27.321 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:27.330 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T18:45:27.337 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:27.386 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:27.399 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:27.407 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T18:45:27.453 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:27.564 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:27.576 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:27.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:27.593 INFO:teuthology.orchestra.run.smithi050.stdout: "registry_insecure": { 2024-04-23T18:45:27.601 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "registry_insecure", 2024-04-23T18:45:27.610 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:27.620 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:27.629 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:27.685 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:27.693 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:27.745 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:27.754 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T18:45:27.826 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:27.835 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:27.891 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:27.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:27.963 INFO:teuthology.orchestra.run.smithi050.stdout: "registry_password": { 2024-04-23T18:45:27.971 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "registry_password", 2024-04-23T18:45:27.978 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:27.987 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:27.998 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:28.005 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:28.012 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:28.019 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:28.072 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:28.081 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T18:45:28.162 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:28.169 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:28.176 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:28.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:28.240 INFO:teuthology.orchestra.run.smithi050.stdout: "registry_url": { 2024-04-23T18:45:28.286 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "registry_url", 2024-04-23T18:45:28.345 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:28.352 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:28.360 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:28.468 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:28.474 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T18:45:28.483 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:28.535 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:28.544 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:28.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:28.597 INFO:teuthology.orchestra.run.smithi050.stdout: "registry_username": { 2024-04-23T18:45:28.667 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "registry_username", 2024-04-23T18:45:28.729 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:28.738 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:28.768 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:28.869 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:28.878 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:28.887 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T18:45:28.896 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:28.904 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:28.913 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:28.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:28.950 INFO:teuthology.orchestra.run.smithi050.stdout: "secure_monitoring_stack": { 2024-04-23T18:45:28.959 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "secure_monitoring_stack", 2024-04-23T18:45:29.029 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:29.078 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:29.086 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:29.132 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:29.141 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:29.183 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:29.246 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T18:45:29.254 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:29.263 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:29.271 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:29.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:29.337 INFO:teuthology.orchestra.run.smithi050.stdout: "service_discovery_port": { 2024-04-23T18:45:29.345 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "service_discovery_port", 2024-04-23T18:45:29.352 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:29.360 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:29.413 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:29.484 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8765", 2024-04-23T18:45:29.492 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:29.553 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:29.560 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:29.568 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "cephadm service discovery port", 2024-04-23T18:45:29.626 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:29.713 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:29.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: "ssh_config_file": { 2024-04-23T18:45:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ssh_config_file", 2024-04-23T18:45:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:29.834 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:29.904 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T18:45:29.967 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:30.024 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:30.032 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "use_agent": { 2024-04-23T18:45:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "use_agent", 2024-04-23T18:45:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:30.170 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:30.178 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:30.192 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T18:45:30.245 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:30.299 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:30.307 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:30.369 INFO:teuthology.orchestra.run.smithi050.stdout: "use_repo_digest": { 2024-04-23T18:45:30.377 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "use_repo_digest", 2024-04-23T18:45:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:30.490 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:30.498 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:30.507 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T18:45:30.516 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:30.524 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:30.579 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_on_failed_host_check": { 2024-04-23T18:45:30.747 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_on_failed_host_check", 2024-04-23T18:45:30.755 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:30.764 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:30.772 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:30.806 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:30.921 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:30.929 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:30.936 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T18:45:30.943 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:30.949 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:30.956 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:30.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:30.969 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_on_stray_daemons": { 2024-04-23T18:45:31.009 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_on_stray_daemons", 2024-04-23T18:45:31.016 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:31.062 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:31.119 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:31.226 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:31.286 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T18:45:31.294 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:31.301 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:31.309 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:31.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:31.324 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_on_stray_hosts": { 2024-04-23T18:45:31.332 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_on_stray_hosts", 2024-04-23T18:45:31.340 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:31.349 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:31.356 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:31.403 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:31.445 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:31.456 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:31.519 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:31.528 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T18:45:31.537 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:45:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:45:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:45:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crash", 2024-04-23T18:45:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:45:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:45:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:45:31.769 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:45:31.780 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:45:31.846 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:31.916 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:31.928 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:32.015 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:32.028 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:32.093 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:32.155 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:32.168 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:45:32.182 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:45:32.195 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:45:32.206 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:45:32.217 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:45:32.228 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:45:32.239 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:32.250 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:32.260 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:32.354 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:32.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:32.401 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:45:32.430 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:45:32.440 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:32.472 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:32.482 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:32.492 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:32.544 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:32.553 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:32.563 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:32.571 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:32.578 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:32.586 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:32.595 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:32.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:32.713 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:45:32.762 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:45:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:32.869 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:45:32.906 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:32.979 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:32.986 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:32.993 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:45:33.004 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:45:33.017 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:45:33.030 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:45:33.039 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:45:33.053 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:45:33.065 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:33.131 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:33.143 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:33.203 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:33.278 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:33.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:33.306 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:45:33.319 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:45:33.344 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:33.404 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:33.415 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:33.426 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:33.435 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:33.445 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:33.457 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:33.469 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:33.480 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:33.539 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:33.595 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:33.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:33.645 INFO:teuthology.orchestra.run.smithi050.stdout: "retain_interval": { 2024-04-23T18:45:33.718 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "retain_interval", 2024-04-23T18:45:33.731 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:45:33.742 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:33.751 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:33.775 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "31536000", 2024-04-23T18:45:33.827 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:33.836 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:33.846 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:33.859 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T18:45:33.870 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:33.881 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:33.892 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:33.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:33.913 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_recent_interval": { 2024-04-23T18:45:33.968 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_recent_interval", 2024-04-23T18:45:33.981 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:45:34.045 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:34.132 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "1209600", 2024-04-23T18:45:34.143 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:34.154 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:34.180 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:34.254 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T18:45:34.265 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:34.279 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:34.354 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:34.365 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:45:34.378 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:45:34.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:34.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:45:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "dashboard", 2024-04-23T18:45:34.471 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:45:34.482 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:45:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:45:34.599 INFO:teuthology.orchestra.run.smithi050.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T18:45:34.608 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T18:45:34.618 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:34.627 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:34.779 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:45:34.788 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:34.798 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:34.809 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:34.897 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:34.908 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:34.920 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:34.931 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:34.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:35.010 INFO:teuthology.orchestra.run.smithi050.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T18:45:35.019 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T18:45:35.069 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:35.139 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:35.148 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:35.163 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:35.178 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:35.204 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:35.260 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:35.271 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:35.283 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:35.299 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:35.317 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:35.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:35.344 INFO:teuthology.orchestra.run.smithi050.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T18:45:35.356 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T18:45:35.370 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:35.422 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:35.429 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:35.478 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:35.569 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:35.583 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:35.608 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:35.712 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:35.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:35.731 INFO:teuthology.orchestra.run.smithi050.stdout: "AUDIT_API_ENABLED": { 2024-04-23T18:45:35.741 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T18:45:35.752 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:35.759 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:35.769 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:35.776 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:35.830 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:35.837 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:35.961 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:35.972 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:35.982 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:35.991 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:36.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T18:45:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T18:45:36.105 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:36.114 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:36.124 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:36.134 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:36.175 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:36.182 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:36.195 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:36.251 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:36.260 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:36.311 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:36.386 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:36.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:36.435 INFO:teuthology.orchestra.run.smithi050.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T18:45:36.447 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T18:45:36.471 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:36.526 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:36.532 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:36.546 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:36.561 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:36.575 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:36.587 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:36.603 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:36.616 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:36.629 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:36.681 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:36.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:36.752 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T18:45:36.817 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T18:45:36.827 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:36.836 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:36.844 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T18:45:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T18:45:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:37.152 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:37.223 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:37.233 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:37.242 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:37.249 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:37.283 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T18:45:37.433 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T18:45:37.479 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:37.490 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:37.683 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:37.691 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:37.746 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:37.760 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T18:45:37.886 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T18:45:37.897 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:37.910 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:37.921 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:37.930 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:37.940 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:37.988 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:37.998 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:38.066 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:38.131 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:38.144 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:38.155 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:38.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:38.204 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T18:45:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T18:45:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:38.402 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T18:45:38.560 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T18:45:38.574 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:38.659 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:38.723 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:38.736 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:38.749 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:38.803 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:38.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:38.867 INFO:teuthology.orchestra.run.smithi050.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T18:45:38.911 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T18:45:38.971 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T18:45:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T18:45:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:39.517 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:39.549 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:39.561 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:39.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:39.580 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T18:45:39.591 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T18:45:39.601 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:39.612 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:39.623 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:39.660 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "admin", 2024-04-23T18:45:39.671 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:39.727 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:39.783 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:39.812 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:39.821 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:39.831 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:39.857 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:39.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:39.917 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T18:45:39.931 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T18:45:39.944 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:39.956 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:39.969 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:39.978 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:39.992 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:40.007 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:40.075 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:40.123 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:40.190 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:40.203 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:40.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:40.231 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_API_URL": { 2024-04-23T18:45:40.269 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_API_URL", 2024-04-23T18:45:40.350 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:40.404 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:40.413 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:40.427 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:40.440 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:40.453 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:40.466 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:40.478 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:40.489 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:40.553 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:40.566 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:40.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:40.683 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T18:45:40.693 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T18:45:40.706 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:40.715 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:40.745 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:40.803 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "admin", 2024-04-23T18:45:40.816 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:40.838 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:40.886 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:40.893 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:40.903 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:40.913 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:40.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:40.936 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T18:45:40.994 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T18:45:41.088 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:41.167 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:41.179 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:41.191 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:41.205 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:41.243 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:41.348 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:41.356 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:41.371 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:41.383 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:41.401 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:41.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:41.427 INFO:teuthology.orchestra.run.smithi050.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T18:45:41.442 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T18:45:41.453 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:41.531 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:41.543 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:41.584 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:41.657 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:41.667 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:41.673 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:41.683 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:41.744 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:41.808 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:41.821 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:41.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:41.846 INFO:teuthology.orchestra.run.smithi050.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T18:45:41.862 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T18:45:41.869 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:41.880 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:41.887 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:41.894 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:41.952 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:41.960 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:42.001 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:42.072 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:42.079 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:42.085 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:42.091 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:42.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:42.124 INFO:teuthology.orchestra.run.smithi050.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T18:45:42.201 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T18:45:42.228 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:42.236 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:42.244 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:42.251 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:42.258 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:42.265 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:42.273 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:42.279 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:42.335 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:42.444 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:42.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:42.462 INFO:teuthology.orchestra.run.smithi050.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T18:45:42.474 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T18:45:42.503 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:42.568 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:42.577 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:42.585 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "[]", 2024-04-23T18:45:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:42.604 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:42.612 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:42.621 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:42.686 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:42.695 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:42.703 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:42.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:42.792 INFO:teuthology.orchestra.run.smithi050.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T18:45:42.861 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T18:45:42.920 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:42.928 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "{}", 2024-04-23T18:45:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:43.078 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:43.161 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:43.169 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:43.177 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:43.185 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:43.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:43.202 INFO:teuthology.orchestra.run.smithi050.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T18:45:43.262 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T18:45:43.269 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:43.325 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:43.534 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:43.542 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:43.553 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:43.560 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:43.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:43.574 INFO:teuthology.orchestra.run.smithi050.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T18:45:43.581 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T18:45:43.588 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:43.594 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:43.650 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:43.656 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:43.764 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:43.770 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:43.777 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:43.783 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:43.839 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T18:45:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T18:45:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:43.928 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:43.993 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:44.001 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:44.010 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:44.019 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:44.066 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:44.117 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:44.143 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:44.152 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:44.194 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:44.250 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T18:45:44.258 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T18:45:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:44.486 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:44.494 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:44.534 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:44.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:44.661 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T18:45:44.669 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T18:45:44.677 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:44.718 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:44.775 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:44.781 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:44.787 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:44.850 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:44.857 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:44.864 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:44.871 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:44.878 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:44.886 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:44.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:44.949 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T18:45:44.992 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T18:45:45.056 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:45.064 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:45.073 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:45.082 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:45.124 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:45.133 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:45.254 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:45.263 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T18:45:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T18:45:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:45.506 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:45.515 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:45.549 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:45.617 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:45.625 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:45.633 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:45.707 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:45.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:45.723 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T18:45:45.730 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T18:45:45.737 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:45.745 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:45.752 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:45.796 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:45.825 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:45.886 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:45.894 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:45.901 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:46.034 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:46.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:46.084 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T18:45:46.091 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T18:45:46.098 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:46.105 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:46.112 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:46.119 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:46.126 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:46.132 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:46.158 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:46.191 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:46.200 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:46.277 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:46.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:46.316 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T18:45:46.392 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T18:45:46.401 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:46.410 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:46.418 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:46.427 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:46.477 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:46.488 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:46.499 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:46.508 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:46.564 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:46.573 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:46.608 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:46.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:46.687 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T18:45:46.696 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T18:45:46.704 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:46.732 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:46.800 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:46.808 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T18:45:46.822 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:46.831 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:46.839 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:46.846 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:46.852 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:46.859 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:46.867 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:46.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:46.935 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T18:45:46.966 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T18:45:47.029 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:47.036 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:47.045 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:47.054 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:45:47.081 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:47.141 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:47.150 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:47.158 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:47.167 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:47.180 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:47.187 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:47.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:47.205 INFO:teuthology.orchestra.run.smithi050.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T18:45:47.213 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T18:45:47.221 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:47.282 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:47.325 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:47.378 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8", 2024-04-23T18:45:47.385 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:47.394 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:47.403 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:47.431 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:47.481 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:47.488 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:47.495 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:47.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:47.512 INFO:teuthology.orchestra.run.smithi050.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T18:45:47.519 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T18:45:47.527 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:47.536 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:47.603 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:47.662 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "45", 2024-04-23T18:45:47.672 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:47.708 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:47.715 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:47.781 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:47.809 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:47.816 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:47.847 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:47.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:47.917 INFO:teuthology.orchestra.run.smithi050.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T18:45:47.924 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T18:45:47.930 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:47.938 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:47.946 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:47.953 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:47.962 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:48.019 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:48.076 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:48.082 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:48.116 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:48.124 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:48.193 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:48.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:48.207 INFO:teuthology.orchestra.run.smithi050.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T18:45:48.244 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T18:45:48.308 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:48.316 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:48.322 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:48.329 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "admin", 2024-04-23T18:45:48.335 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:48.341 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:48.347 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:48.353 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:48.359 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:48.415 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:48.421 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:48.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:48.519 INFO:teuthology.orchestra.run.smithi050.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T18:45:48.553 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T18:45:48.559 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:48.566 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:48.622 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:48.744 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:48.752 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:48.760 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:48.770 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:48.778 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:48.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:48.864 INFO:teuthology.orchestra.run.smithi050.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T18:45:48.874 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T18:45:48.899 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:48.958 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:48.967 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:48.977 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:48.985 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:48.993 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:49.056 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T18:45:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T18:45:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:49.219 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:49.278 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:49.307 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:49.373 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:49.427 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:49.439 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:49.447 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:49.505 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:49.557 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:49.566 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:49.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:49.582 INFO:teuthology.orchestra.run.smithi050.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T18:45:49.635 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T18:45:49.647 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:49.656 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:49.664 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:45:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:49.742 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:49.774 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:49.781 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:49.849 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:49.859 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:49.868 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:49.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:49.999 INFO:teuthology.orchestra.run.smithi050.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T18:45:50.007 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T18:45:50.073 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:50.083 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:50.091 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:50.100 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:45:50.109 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:50.118 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:50.126 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:50.186 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:50.194 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:50.232 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:50.305 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T18:45:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T18:45:50.364 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:50.432 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:50.439 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5", 2024-04-23T18:45:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:50.527 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:50.543 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: "cross_origin_url": { 2024-04-23T18:45:50.652 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cross_origin_url", 2024-04-23T18:45:50.719 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:50.727 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:50.734 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:50.763 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:50.849 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:50.856 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:50.862 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:50.873 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:50.926 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:50.934 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:50.941 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:50.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:50.955 INFO:teuthology.orchestra.run.smithi050.stdout: "crt_file": { 2024-04-23T18:45:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "crt_file", 2024-04-23T18:45:51.012 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:51.048 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:51.116 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:51.135 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:51.144 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:51.151 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:51.180 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:51.232 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:51.239 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:51.246 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:51.253 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:51.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:51.266 INFO:teuthology.orchestra.run.smithi050.stdout: "debug": { 2024-04-23T18:45:51.274 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "debug", 2024-04-23T18:45:51.327 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:51.333 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:51.340 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:51.395 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:51.423 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:51.431 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Enable/disable debug options", 2024-04-23T18:45:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:51.648 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:51.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:51.666 INFO:teuthology.orchestra.run.smithi050.stdout: "jwt_token_ttl": { 2024-04-23T18:45:51.675 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "jwt_token_ttl", 2024-04-23T18:45:51.684 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:51.693 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:51.702 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:51.710 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "28800", 2024-04-23T18:45:51.719 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:51.727 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:51.785 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:51.806 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:51.870 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:51.879 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:51.889 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:51.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:51.905 INFO:teuthology.orchestra.run.smithi050.stdout: "key_file": { 2024-04-23T18:45:51.973 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "key_file", 2024-04-23T18:45:51.986 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:52.055 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:52.065 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:52.073 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:52.081 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:52.089 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:52.098 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:52.105 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:52.156 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:52.165 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:52.198 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:52.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:52.278 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:45:52.286 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:45:52.294 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:52.440 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:52.448 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:52.502 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:52.513 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:45:52.522 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:45:52.533 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:45:52.545 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:45:52.593 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:45:52.604 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:45:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:52.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:52.890 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:45:52.902 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:45:52.912 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:52.921 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:52.928 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:52.935 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:52.943 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:53.001 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:53.012 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:53.065 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:53.078 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:53.106 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:53.173 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:53.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:53.188 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:45:53.195 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:45:53.230 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:53.290 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:53.299 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:53.376 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:45:53.383 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:53.396 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:45:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:45:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:45:53.421 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:45:53.470 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:45:53.506 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:45:53.571 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:53.659 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:53.673 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:53.681 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:53.687 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:53.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:53.780 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:45:53.793 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:45:53.800 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:53.807 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:54.001 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:54.035 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:54.046 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:54.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:54.086 INFO:teuthology.orchestra.run.smithi050.stdout: "motd": { 2024-04-23T18:45:54.130 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "motd", 2024-04-23T18:45:54.198 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:54.226 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:54.235 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:54.244 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:54.251 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:54.260 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:54.269 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:54.279 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "The message of the day", 2024-04-23T18:45:54.288 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:54.296 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:54.350 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:54.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:54.450 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_resolve_ip_addr": { 2024-04-23T18:45:54.496 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T18:45:54.506 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:54.517 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:54.555 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:54.565 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:54.576 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:54.631 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:54.641 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:54.651 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:54.666 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:54.679 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:54.689 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:54.839 INFO:teuthology.orchestra.run.smithi050.stdout: "server_addr": { 2024-04-23T18:45:54.850 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_addr", 2024-04-23T18:45:54.889 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:54.949 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:55.017 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:55.026 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "::", 2024-04-23T18:45:55.040 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:55.070 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:55.156 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:55.225 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:55.235 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:55.251 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:55.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:55.266 INFO:teuthology.orchestra.run.smithi050.stdout: "server_port": { 2024-04-23T18:45:55.274 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_port", 2024-04-23T18:45:55.282 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:55.291 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:55.344 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:55.426 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8080", 2024-04-23T18:45:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:55.488 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:55.498 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:55.509 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:55.546 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:55.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:55.682 INFO:teuthology.orchestra.run.smithi050.stdout: "ssl": { 2024-04-23T18:45:55.692 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ssl", 2024-04-23T18:45:55.703 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:55.712 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:55.719 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:55.726 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:55.780 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:55.791 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:55.814 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:55.880 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:55.917 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:55.925 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:55.932 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:55.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:56.039 INFO:teuthology.orchestra.run.smithi050.stdout: "ssl_server_port": { 2024-04-23T18:45:56.045 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ssl_server_port", 2024-04-23T18:45:56.052 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:56.117 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:56.125 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:56.132 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8443", 2024-04-23T18:45:56.139 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:56.145 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:56.152 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:56.159 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:56.195 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:56.230 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:56.294 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:56.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:56.341 INFO:teuthology.orchestra.run.smithi050.stdout: "standby_behaviour": { 2024-04-23T18:45:56.351 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "standby_behaviour", 2024-04-23T18:45:56.388 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "redirect", 2024-04-23T18:45:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:45:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect" 2024-04-23T18:45:56.652 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:56.661 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:56.719 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:56.727 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:56.813 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:56.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:56.839 INFO:teuthology.orchestra.run.smithi050.stdout: "standby_error_status_code": { 2024-04-23T18:45:56.848 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "standby_error_status_code", 2024-04-23T18:45:56.857 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:45:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "500", 2024-04-23T18:45:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "400", 2024-04-23T18:45:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "599", 2024-04-23T18:45:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:57.130 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:57.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:57.190 INFO:teuthology.orchestra.run.smithi050.stdout: "url_prefix": { 2024-04-23T18:45:57.251 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "url_prefix", 2024-04-23T18:45:57.259 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:57.268 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:57.275 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:45:57.339 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:57.387 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:57.395 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:57.403 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:57.410 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:57.418 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:57.426 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:45:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:45:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:57.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:45:57.559 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "devicehealth", 2024-04-23T18:45:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:45:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:45:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:45:57.612 INFO:teuthology.orchestra.run.smithi050.stdout: "enable_monitoring": { 2024-04-23T18:45:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "enable_monitoring", 2024-04-23T18:45:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:57.702 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:45:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "monitor device health metrics", 2024-04-23T18:45:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:57.773 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:57.780 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:57.921 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:45:57.928 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:45:57.937 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:57.943 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:57.969 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:45:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:45:58.074 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:45:58.081 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:45:58.088 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:45:58.096 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:45:58.154 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:45:58.201 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:58.240 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:58.303 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:58.310 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:58.319 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:58.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:58.373 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:45:58.420 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:45:58.429 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:58.439 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:58.448 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:58.509 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:58.519 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:58.538 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:58.548 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:58.657 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:58.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:58.728 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:45:58.792 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:45:58.801 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:58.811 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:58.871 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:58.895 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:45:58.904 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:58.913 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:58.922 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:45:58.930 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:45:58.939 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:45:58.948 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:45:58.957 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:45:58.965 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:45:59.009 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:45:59.081 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:45:59.093 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:59.152 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:59.161 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:59.169 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:59.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:59.314 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:45:59.322 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:45:59.333 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:45:59.343 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:59.351 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:59.359 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:45:59.370 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:59.381 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:59.388 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:45:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: "mark_out_threshold": { 2024-04-23T18:45:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mark_out_threshold", 2024-04-23T18:45:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:45:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "2419200", 2024-04-23T18:45:59.540 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:59.548 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:59.557 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:59.570 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T18:45:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:59.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:59.621 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": { 2024-04-23T18:45:59.630 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "pool_name", 2024-04-23T18:45:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:45:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "device_health_metrics", 2024-04-23T18:45:59.719 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:59.729 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:59.738 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:59.747 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T18:45:59.756 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:59.764 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:59.772 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:59.781 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:59.790 INFO:teuthology.orchestra.run.smithi050.stdout: "retention_period": { 2024-04-23T18:45:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "retention_period", 2024-04-23T18:45:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:45:59.816 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "15552000", 2024-04-23T18:45:59.845 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:45:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:45:59.864 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:45:59.911 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how long to retain device health metrics", 2024-04-23T18:45:59.920 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:45:59.929 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:45:59.938 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:45:59.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:45:59.955 INFO:teuthology.orchestra.run.smithi050.stdout: "scrape_frequency": { 2024-04-23T18:45:59.962 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "scrape_frequency", 2024-04-23T18:45:59.972 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:45:59.980 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:45:59.989 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:45:59.998 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "86400", 2024-04-23T18:46:00.006 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T18:46:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:00.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:00.127 INFO:teuthology.orchestra.run.smithi050.stdout: "self_heal": { 2024-04-23T18:46:00.137 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "self_heal", 2024-04-23T18:46:00.150 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:00.159 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:00.168 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:00.177 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:46:00.187 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:00.200 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:00.214 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:00.228 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T18:46:00.242 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:00.252 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:00.263 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:00.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:00.282 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:46:00.336 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:46:00.344 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:46:00.354 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:00.363 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:00.372 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:46:00.383 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:00.395 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:00.406 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:00.413 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T18:46:00.422 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:00.430 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:00.437 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:00.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:00.452 INFO:teuthology.orchestra.run.smithi050.stdout: "warn_threshold": { 2024-04-23T18:46:00.460 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "warn_threshold", 2024-04-23T18:46:00.467 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:46:00.474 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:00.533 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:00.543 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "7257600", 2024-04-23T18:46:00.550 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:00.557 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:00.565 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:00.572 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T18:46:00.579 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:00.587 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:00.594 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:00.603 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:00.610 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:00.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:00.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:00.631 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "diskprediction_local", 2024-04-23T18:46:00.638 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:00.647 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:00.653 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:00.696 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:00.734 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:00.744 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:00.752 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:00.759 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:00.765 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:00.774 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:00.781 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:00.791 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:00.799 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:00.809 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:00.819 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:00.870 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:00.878 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:00.893 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:00.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:00.913 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:00.933 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:00.943 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:00.957 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:00.966 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:00.977 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:00.990 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:01.002 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:01.014 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:01.025 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:01.035 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:01.047 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:01.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:01.104 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:01.117 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:01.129 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:01.139 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:01.148 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:01.159 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:01.171 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:01.180 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:01.189 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:01.201 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:01.210 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:01.220 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:01.229 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:01.238 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:01.247 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:01.256 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:01.298 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:01.325 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:01.336 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:01.345 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:01.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:01.362 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:01.371 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:01.380 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:01.388 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:01.396 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:01.407 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:01.417 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:01.426 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:01.434 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:01.442 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:01.451 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:01.459 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:01.519 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:01.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:01.536 INFO:teuthology.orchestra.run.smithi050.stdout: "predict_interval": { 2024-04-23T18:46:01.545 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "predict_interval", 2024-04-23T18:46:01.554 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:01.563 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:01.571 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:01.580 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "86400", 2024-04-23T18:46:01.588 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:01.596 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:01.604 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:01.613 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:01.622 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:01.630 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:01.640 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:01.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:01.658 INFO:teuthology.orchestra.run.smithi050.stdout: "predictor_model": { 2024-04-23T18:46:01.711 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "predictor_model", 2024-04-23T18:46:01.721 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:01.729 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:01.739 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:01.748 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "prophetstor", 2024-04-23T18:46:01.758 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:01.766 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:01.773 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:01.781 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:01.791 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:01.801 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:01.809 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:01.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:01.827 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:46:01.836 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:46:01.845 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:01.854 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:01.911 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:01.920 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "600", 2024-04-23T18:46:01.928 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:01.936 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:01.947 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:01.955 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:01.965 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:01.973 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:01.981 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:01.990 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:01.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:02.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:02.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:02.026 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "influx", 2024-04-23T18:46:02.034 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": false, 2024-04-23T18:46:02.044 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "influxdb python module not found", 2024-04-23T18:46:02.052 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:02.103 INFO:teuthology.orchestra.run.smithi050.stdout: "batch_size": { 2024-04-23T18:46:02.111 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "batch_size", 2024-04-23T18:46:02.118 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:02.124 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:02.131 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:02.138 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5000", 2024-04-23T18:46:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T18:46:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:02.184 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:02.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "database": { 2024-04-23T18:46:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "database", 2024-04-23T18:46:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:02.281 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:02.289 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "ceph", 2024-04-23T18:46:02.296 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:02.302 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:02.310 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:02.317 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T18:46:02.325 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:02.332 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:02.341 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:02.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:02.357 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": { 2024-04-23T18:46:02.366 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "hostname", 2024-04-23T18:46:02.375 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:02.384 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:02.392 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:02.401 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:02.461 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:02.470 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:02.479 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:02.488 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "InfluxDB server hostname", 2024-04-23T18:46:02.497 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:02.506 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:02.515 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:02.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:02.533 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:46:02.541 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:46:02.549 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:46:02.558 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:02.566 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:02.575 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "30", 2024-04-23T18:46:02.584 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "5", 2024-04-23T18:46:02.592 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:02.601 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:02.652 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T18:46:02.661 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:02.670 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:02.680 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:02.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:02.698 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:02.706 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:02.717 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:02.733 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:02.742 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:02.751 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:02.759 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:02.768 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:02.777 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:02.786 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:02.795 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:02.803 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:02.855 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:02.864 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:02.872 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:02.882 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:02.891 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:02.907 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:02.916 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:02.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:02.935 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:02.943 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:02.953 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:02.962 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:02.970 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:02.988 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:02.997 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:03.040 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:03.049 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:03.057 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:03.065 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:03.073 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:03.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:03.087 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:03.094 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:03.101 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:03.108 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:03.115 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:03.123 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:03.130 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:03.139 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:03.146 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:03.154 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:03.161 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:03.202 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:03.219 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:03.228 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:03.235 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:03.243 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:03.250 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:03.257 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:03.265 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:03.273 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:03.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:03.287 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:03.296 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:03.303 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:03.309 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:03.315 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:03.322 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:03.330 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:03.386 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:03.393 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:03.399 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:03.406 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:03.413 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:03.420 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: "password": { 2024-04-23T18:46:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "password", 2024-04-23T18:46:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:03.476 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:03.483 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:03.490 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:03.497 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "password of InfluxDB server user", 2024-04-23T18:46:03.552 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:03.560 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:03.566 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:03.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:03.580 INFO:teuthology.orchestra.run.smithi050.stdout: "port": { 2024-04-23T18:46:03.588 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "port", 2024-04-23T18:46:03.595 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:03.602 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:03.609 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:03.616 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "8086", 2024-04-23T18:46:03.624 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:03.631 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:03.638 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:03.645 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "InfluxDB server port", 2024-04-23T18:46:03.653 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:03.661 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:03.670 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:03.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:03.728 INFO:teuthology.orchestra.run.smithi050.stdout: "ssl": { 2024-04-23T18:46:03.737 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ssl", 2024-04-23T18:46:03.746 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:03.754 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:03.775 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "false", 2024-04-23T18:46:03.784 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:03.793 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:03.801 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:03.809 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:46:03.818 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:03.827 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:03.836 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:03.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: "threads": { 2024-04-23T18:46:03.865 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "threads", 2024-04-23T18:46:03.911 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:03.938 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:03.947 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:03.956 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5", 2024-04-23T18:46:03.965 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1", 2024-04-23T18:46:03.974 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "32", 2024-04-23T18:46:03.991 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:03.999 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T18:46:04.008 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "username": { 2024-04-23T18:46:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "username", 2024-04-23T18:46:04.062 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:04.070 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:04.078 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:04.145 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:04.153 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:04.161 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:04.169 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:04.178 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "username of InfluxDB server user", 2024-04-23T18:46:04.186 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:04.195 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:04.203 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:04.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:04.219 INFO:teuthology.orchestra.run.smithi050.stdout: "verify_ssl": { 2024-04-23T18:46:04.228 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "verify_ssl", 2024-04-23T18:46:04.236 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:04.245 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:04.253 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:04.261 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "true", 2024-04-23T18:46:04.270 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:04.279 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:04.319 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:04.339 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T18:46:04.347 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:04.356 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:04.364 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:04.372 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:04.380 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:04.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:04.404 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "insights", 2024-04-23T18:46:04.413 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:04.421 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:04.429 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:04.436 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:04.442 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:04.449 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:04.457 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:04.519 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:04.537 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:04.544 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:04.552 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:04.559 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:04.566 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:04.573 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:04.581 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:04.589 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:04.597 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:04.604 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:04.613 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:04.620 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:04.627 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:04.633 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:04.640 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:04.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:04.703 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:04.712 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:04.718 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:04.725 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:04.732 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:04.739 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:04.746 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:04.754 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:04.761 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:04.768 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:04.775 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:04.783 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:04.790 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:04.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:04.804 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:04.813 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:04.821 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:04.885 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:04.893 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:04.901 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:04.908 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:04.915 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:04.922 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:04.931 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:04.937 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:04.944 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:04.952 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:04.961 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:04.970 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:04.979 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:04.990 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:04.999 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:05.008 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:05.016 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:05.070 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:05.089 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:05.098 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:05.106 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:05.114 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:05.125 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:05.133 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:05.175 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:05.184 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:05.193 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:05.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:05.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:05.278 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "iostat", 2024-04-23T18:46:05.286 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:05.295 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:05.304 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:05.314 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:05.322 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:05.332 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:05.341 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:05.349 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:05.357 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:05.365 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:05.373 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:05.381 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:05.390 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:05.405 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:05.414 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:05.547 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:05.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:05.597 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:05.605 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:05.612 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:05.621 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:05.629 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:05.638 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:05.646 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:05.676 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:05.727 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:05.735 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:05.744 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:05.798 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:05.806 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:05.814 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:05.821 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:05.827 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:05.834 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:05.841 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:05.873 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:05.927 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:05.935 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:05.941 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:05.947 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:05.954 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:05.961 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:05.967 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:05.981 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:05.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:05.996 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:06.003 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:06.010 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:06.020 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:06.029 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:06.036 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:06.087 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "k8sevents", 2024-04-23T18:46:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:06.255 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:06.266 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_event_retention_days": { 2024-04-23T18:46:06.275 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "ceph_event_retention_days", 2024-04-23T18:46:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:06.375 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:06.382 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "7", 2024-04-23T18:46:06.390 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:06.399 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:06.421 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:06.430 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T18:46:06.438 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:06.446 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:06.455 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:06.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:06.471 INFO:teuthology.orchestra.run.smithi050.stdout: "config_check_secs": { 2024-04-23T18:46:06.480 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "config_check_secs", 2024-04-23T18:46:06.489 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:06.498 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:06.508 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:06.516 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:46:06.568 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "10", 2024-04-23T18:46:06.575 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:06.583 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:06.591 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T18:46:06.599 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:06.616 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:06.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:06.635 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:06.644 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:06.652 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:06.661 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:06.670 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:06.679 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:06.688 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:06.696 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:06.729 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:06.835 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:06.844 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:06.855 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:06.863 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:06.873 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:06.882 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:07.009 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:07.065 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:07.074 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:07.083 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:07.092 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:07.100 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:07.108 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:07.117 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:07.186 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:07.195 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:07.224 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:07.276 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:07.285 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:07.294 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:07.303 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:07.311 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:07.320 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:07.329 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:07.337 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:07.345 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:07.353 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:07.362 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:07.370 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:07.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:07.386 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:07.403 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:07.468 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:07.477 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:07.486 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:07.495 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:07.503 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:07.512 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:07.520 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:07.529 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:07.537 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:07.546 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:07.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:07.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:07.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:07.577 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "localpool", 2024-04-23T18:46:07.585 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:07.593 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:07.626 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:07.684 INFO:teuthology.orchestra.run.smithi050.stdout: "failure_domain": { 2024-04-23T18:46:07.693 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "failure_domain", 2024-04-23T18:46:07.702 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:07.712 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:07.720 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:07.728 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "host", 2024-04-23T18:46:07.734 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:07.741 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:07.748 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:07.756 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "failure domain for any created local pool", 2024-04-23T18:46:07.764 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T18:46:07.772 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:07.781 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:07.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:07.796 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:07.803 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:07.831 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:07.901 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:07.908 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:07.914 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:07.921 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:07.928 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:07.935 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:07.942 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:07.949 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:07.957 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:07.964 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:07.989 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:07.996 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:08.004 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:08.010 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:08.017 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:08.025 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:08.062 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:08.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:08.125 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:08.133 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:08.139 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:08.160 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:08.181 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:08.207 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:08.216 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:08.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:08.233 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:08.257 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:08.309 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:08.318 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:08.327 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:08.335 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:08.343 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:08.351 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:08.378 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:08.386 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:08.395 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:08.405 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:08.413 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:08.422 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:08.430 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:08.441 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:08.450 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:08.459 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:08.489 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:08.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:08.571 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:08.597 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:08.606 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:08.642 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:08.651 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:08.660 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:08.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:08.691 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": { 2024-04-23T18:46:08.749 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "min_size", 2024-04-23T18:46:08.757 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:08.766 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:08.774 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:08.783 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:08.791 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:08.800 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:08.808 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:08.817 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "default min_size for any created local pool", 2024-04-23T18:46:08.825 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T18:46:08.834 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:08.842 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:08.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:08.858 INFO:teuthology.orchestra.run.smithi050.stdout: "num_rep": { 2024-04-23T18:46:08.866 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "num_rep", 2024-04-23T18:46:08.875 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:08.916 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:08.992 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:09.003 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "3", 2024-04-23T18:46:09.010 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:09.018 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:09.026 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:09.034 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "default replica count for any created local pool", 2024-04-23T18:46:09.041 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:09.049 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:09.056 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:09.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": { 2024-04-23T18:46:09.077 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "pg_num", 2024-04-23T18:46:09.083 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:09.089 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:09.097 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:09.105 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "128", 2024-04-23T18:46:09.129 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:09.184 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:09.191 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T18:46:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:09.217 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:09.225 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:09.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:09.239 INFO:teuthology.orchestra.run.smithi050.stdout: "prefix": { 2024-04-23T18:46:09.246 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "prefix", 2024-04-23T18:46:09.253 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:09.259 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:09.267 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:09.274 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:09.281 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:09.288 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:09.295 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:09.333 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "name prefix for any created local pool", 2024-04-23T18:46:09.393 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:09.400 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:09.408 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:09.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:09.423 INFO:teuthology.orchestra.run.smithi050.stdout: "subtree": { 2024-04-23T18:46:09.431 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "subtree", 2024-04-23T18:46:09.438 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:09.445 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:09.453 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:09.460 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "rack", 2024-04-23T18:46:09.467 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:09.475 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:09.483 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:09.489 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T18:46:09.495 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T18:46:09.502 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:09.548 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:09.649 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mds_autoscaler", 2024-04-23T18:46:09.656 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:09.666 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:09.675 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:09.683 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:09.693 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:09.702 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:09.713 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:09.723 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:09.733 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:09.742 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:09.751 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:09.789 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:09.834 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:09.844 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:09.853 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:09.862 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:09.870 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:09.879 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:09.887 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:09.895 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:09.903 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:09.912 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:09.920 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:09.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:09.937 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:09.946 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:09.955 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:09.963 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:10.000 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:10.059 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:10.086 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:10.095 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:10.104 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:10.112 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:10.123 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:10.131 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:10.140 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:10.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:10.158 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:10.167 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:10.175 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:10.183 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:10.191 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:10.201 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:10.209 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:10.248 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:10.321 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:10.330 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:10.341 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:10.349 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:10.358 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:10.366 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:10.383 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:10.392 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:10.400 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:10.408 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:10.415 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:10.423 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:10.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:10.439 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:10.446 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:10.454 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:10.487 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:10.542 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:10.551 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:10.559 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:10.566 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:10.573 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:10.581 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:10.587 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:10.594 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:10.601 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:10.608 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:10.615 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:10.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:10.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:10.638 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mirroring", 2024-04-23T18:46:10.645 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:10.652 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:10.658 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:10.709 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:10.716 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:10.722 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:10.729 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:10.737 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:10.744 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:10.750 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:10.758 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:10.765 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:10.771 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:10.778 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:10.785 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:10.793 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:10.801 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:10.810 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:10.819 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:10.864 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:10.921 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:10.929 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:10.938 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:10.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:10.952 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:10.959 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:10.968 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:10.977 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:10.986 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:10.995 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:11.002 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:11.010 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:11.019 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:11.034 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:11.043 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:11.052 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:11.078 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:11.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:11.138 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:11.146 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:11.153 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:11.162 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:11.170 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:11.181 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:11.190 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:11.198 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:11.207 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:11.216 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:11.224 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:11.234 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:11.243 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:11.253 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:11.261 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:11.270 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:11.321 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:11.330 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:11.338 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:11.347 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:11.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:11.363 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:11.371 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:11.380 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:11.388 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:11.579 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:11.588 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:11.596 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:11.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:11.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:11.620 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "nfs", 2024-04-23T18:46:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:11.860 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:11.869 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:11.878 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:11.989 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:12.037 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:12.112 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:12.119 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:12.126 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:12.135 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:12.152 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:12.160 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:12.170 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:12.178 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:12.186 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:12.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:12.202 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:12.209 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:12.347 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:12.360 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:12.367 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:12.380 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:12.387 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:12.431 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:12.438 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:12.475 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:12.533 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:12.540 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:12.547 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:12.553 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:12.561 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:12.568 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:12.577 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:12.584 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:12.591 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:12.598 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:12.606 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:12.615 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:12.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:12.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:12.640 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "orchestrator", 2024-04-23T18:46:12.650 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:12.683 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:12.740 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:12.748 INFO:teuthology.orchestra.run.smithi050.stdout: "fail_fs": { 2024-04-23T18:46:12.756 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "fail_fs", 2024-04-23T18:46:12.763 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:12.771 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:12.780 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:12.787 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:12.795 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:12.804 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:12.813 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:12.821 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T18:46:12.831 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:12.840 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:12.849 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:12.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:12.867 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:12.894 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:12.949 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:12.958 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:12.967 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:13.028 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:13.185 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:13.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:13.201 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:13.210 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:13.218 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:13.227 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:13.235 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:13.245 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:13.254 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:13.261 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:13.270 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:13.279 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:13.288 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:13.295 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:13.303 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:13.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:13.347 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:13.404 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:13.412 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:13.421 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:13.429 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:13.440 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:13.448 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:13.457 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:13.466 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:13.475 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:13.483 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:13.492 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:13.501 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:13.510 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:13.519 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:13.528 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:13.535 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:13.568 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:13.616 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:13.624 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:13.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:13.640 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:13.647 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:13.654 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:13.661 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:13.669 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:13.677 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:13.683 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:13.690 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:13.697 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:13.703 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:13.709 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:13.714 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:13.724 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:13.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:13.812 INFO:teuthology.orchestra.run.smithi050.stdout: "orchestrator": { 2024-04-23T18:46:13.822 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "orchestrator", 2024-04-23T18:46:13.828 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:13.837 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:13.846 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:13.855 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:13.863 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:13.872 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:13.880 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:13.889 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm", 2024-04-23T18:46:13.898 INFO:teuthology.orchestra.run.smithi050.stdout: "rook", 2024-04-23T18:46:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "test_orchestrator" 2024-04-23T18:46:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Orchestrator backend", 2024-04-23T18:46:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:13.948 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:14.012 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:14.021 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:14.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:14.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:14.044 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "osd_perf_query", 2024-04-23T18:46:14.053 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:14.061 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:14.071 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:14.080 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:14.089 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:14.099 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:14.108 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:14.116 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:14.124 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:14.131 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:14.140 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:14.170 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:14.247 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:14.255 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:14.263 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:14.271 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:14.280 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:14.289 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:14.297 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:14.307 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:14.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:14.324 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:14.333 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:14.359 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:14.368 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:14.397 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:14.441 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:14.450 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:14.459 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:14.468 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:14.477 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:14.487 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:14.495 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:14.504 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:14.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:14.521 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:14.530 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:14.539 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:14.548 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:14.558 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:14.566 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:14.574 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:14.611 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:14.665 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:14.674 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:14.683 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:14.691 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:14.700 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:14.709 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:14.717 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:14.726 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:14.735 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:14.743 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:14.753 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:14.762 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:14.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:14.779 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:14.806 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:14.857 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:14.866 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:14.875 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:14.883 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:14.891 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:14.900 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:14.908 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:14.916 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:14.924 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:14.933 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:14.941 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:14.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:14.966 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "osd_support", 2024-04-23T18:46:14.975 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:14.983 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:15.013 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:15.077 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:15.086 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:15.096 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:15.105 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:15.114 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:15.123 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:15.133 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:15.142 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:15.151 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:15.160 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:15.168 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:15.177 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:15.186 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:15.194 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:15.203 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:15.212 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:15.246 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:15.303 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:15.310 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:15.317 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:15.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:15.336 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:15.344 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:15.354 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:15.361 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:15.371 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:15.378 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:15.385 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:15.402 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:15.410 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:15.419 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:15.427 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:15.454 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:15.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:15.516 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:15.526 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:15.535 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:15.543 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:15.552 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:15.561 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:15.570 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:15.579 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:15.587 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:15.595 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:15.603 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:15.611 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:15.618 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:15.625 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:15.632 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:15.659 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:15.712 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:15.720 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:15.727 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:15.734 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:15.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:15.747 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:15.755 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:15.762 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:15.769 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:15.776 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:15.783 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:15.789 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:15.796 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:15.802 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:15.812 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:15.819 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:15.921 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:15.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:15.938 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:15.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:15.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:15.962 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "pg_autoscaler", 2024-04-23T18:46:15.971 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:15.979 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:15.988 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:15.997 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:16.005 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:16.014 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:16.023 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:16.032 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:16.040 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:16.049 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:16.094 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:16.225 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:16.234 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:16.244 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:16.253 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:16.261 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:16.270 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:16.279 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:16.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:16.296 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:16.304 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:16.313 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:16.399 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:16.423 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:16.431 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:16.440 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:16.449 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:16.458 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:16.467 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:16.476 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:16.485 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:16.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:16.501 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:16.510 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:16.519 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:16.527 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:16.536 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:16.544 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:16.587 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:16.636 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:16.654 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:16.664 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:16.672 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:16.681 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:16.689 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:16.699 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:16.716 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:16.725 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:16.734 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:16.742 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:16.751 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:16.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:16.766 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:16.774 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:16.798 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:16.854 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:16.862 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:16.869 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:16.876 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:16.883 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:16.891 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:16.900 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:16.906 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:16.913 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:16.920 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:16.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:16.934 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:46:16.941 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:46:16.948 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:46:16.954 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:16.961 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:16.997 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:46:17.048 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:17.056 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:17.064 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:17.085 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:17.092 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:17.099 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:17.105 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:17.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:17.118 INFO:teuthology.orchestra.run.smithi050.stdout: "threshold": { 2024-04-23T18:46:17.125 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "threshold", 2024-04-23T18:46:17.132 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:46:17.140 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:17.146 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:17.152 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "3.0", 2024-04-23T18:46:17.159 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1.0", 2024-04-23T18:46:17.165 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:17.242 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:17.319 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "scaling threshold", 2024-04-23T18:46:17.327 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T18:46:17.334 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:17.342 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:17.349 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:17.356 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:17.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:17.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:17.377 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "progress", 2024-04-23T18:46:17.384 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:17.393 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:17.401 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:17.410 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_pg_recovery_event": { 2024-04-23T18:46:17.418 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "allow_pg_recovery_event", 2024-04-23T18:46:17.427 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:17.434 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:17.470 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T18:46:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:17.576 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:17.585 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:17.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:17.603 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": { 2024-04-23T18:46:17.611 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "enabled", 2024-04-23T18:46:17.621 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:17.630 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:17.638 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:17.646 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:46:17.655 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:17.690 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:17.736 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:17.749 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:17.757 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:17.766 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:17.775 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:17.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:17.793 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:17.801 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:17.809 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:17.818 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:17.827 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:17.836 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:17.846 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:17.912 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:17.967 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:17.976 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:17.985 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:18.094 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:18.127 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:18.215 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:18.232 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:18.247 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:18.255 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:18.263 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:18.270 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:18.276 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:18.283 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:18.290 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:18.297 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:18.304 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:18.339 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:18.394 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:18.401 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:18.418 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:18.425 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:18.431 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:18.439 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:18.446 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:18.452 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:18.459 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:18.465 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:18.596 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:18.602 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:18.608 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:18.615 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:18.621 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:18.627 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:18.636 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:18.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:18.653 INFO:teuthology.orchestra.run.smithi050.stdout: "max_completed_events": { 2024-04-23T18:46:18.662 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_completed_events", 2024-04-23T18:46:18.671 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:18.679 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:18.688 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:18.696 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "50", 2024-04-23T18:46:18.704 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:18.713 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:18.775 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:18.836 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "number of past completed events to remember", 2024-04-23T18:46:18.852 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:18.858 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:18.867 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:18.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:18.885 INFO:teuthology.orchestra.run.smithi050.stdout: "sleep_interval": { 2024-04-23T18:46:18.893 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "sleep_interval", 2024-04-23T18:46:18.902 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:46:18.910 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:18.918 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:18.925 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "5", 2024-04-23T18:46:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:18.941 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:18.948 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:18.956 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "how long the module is going to sleep", 2024-04-23T18:46:18.963 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:18.988 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:19.039 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:19.052 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:19.061 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:19.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:19.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:19.086 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "prometheus", 2024-04-23T18:46:19.095 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:19.103 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:19.112 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:19.121 INFO:teuthology.orchestra.run.smithi050.stdout: "cache": { 2024-04-23T18:46:19.129 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cache", 2024-04-23T18:46:19.142 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:19.150 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:19.158 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:19.166 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:46:19.174 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:19.182 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:19.239 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:19.248 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:19.275 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:19.284 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:19.293 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:19.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:19.310 INFO:teuthology.orchestra.run.smithi050.stdout: "exclude_perf_counters": { 2024-04-23T18:46:19.318 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "exclude_perf_counters", 2024-04-23T18:46:19.326 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:19.335 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:19.343 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:19.351 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:46:19.360 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:19.377 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T18:46:19.466 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T18:46:19.536 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:19.551 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:19.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:19.569 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:19.581 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:19.588 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:19.596 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:19.607 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:19.615 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:19.623 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:19.632 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:19.642 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:19.652 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:19.660 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:19.669 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:19.678 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:19.704 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:19.765 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:19.774 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:19.783 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:19.792 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:19.800 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:19.812 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:19.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:19.826 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:19.832 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:19.839 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:19.846 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:19.854 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:19.861 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:19.867 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:19.876 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:19.885 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:20.041 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:20.050 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:20.057 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:20.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:20.074 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:20.083 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:20.091 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:20.100 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:20.110 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:20.118 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:20.126 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:20.133 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:20.142 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:20.152 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:20.162 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:20.171 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:20.180 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:20.231 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:20.239 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:20.248 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:20.257 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:20.266 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:20.275 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:20.283 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:20.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:20.299 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:20.308 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:20.317 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:20.326 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:20.334 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:20.343 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:20.351 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:20.360 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:20.387 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:20.477 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:20.486 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:20.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:20.504 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_stats_pools": { 2024-04-23T18:46:20.513 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rbd_stats_pools", 2024-04-23T18:46:20.522 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:20.530 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:20.538 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:20.546 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:20.555 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:20.563 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:20.571 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:20.580 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:20.590 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:20.627 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:20.684 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:20.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:20.702 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T18:46:20.710 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T18:46:20.719 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:20.727 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:20.735 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:20.743 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "300", 2024-04-23T18:46:20.751 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:20.759 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:20.768 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:20.777 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:20.785 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:20.793 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:20.802 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:20.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:20.847 INFO:teuthology.orchestra.run.smithi050.stdout: "scrape_interval": { 2024-04-23T18:46:20.925 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "scrape_interval", 2024-04-23T18:46:20.934 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:46:20.942 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:20.950 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:20.958 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "15.0", 2024-04-23T18:46:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:20.975 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:20.984 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:20.991 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:20.998 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:21.005 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:21.013 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:21.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:21.026 INFO:teuthology.orchestra.run.smithi050.stdout: "server_addr": { 2024-04-23T18:46:21.033 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_addr", 2024-04-23T18:46:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:21.078 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:21.136 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:21.143 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "::", 2024-04-23T18:46:21.153 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:21.162 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:21.170 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:21.178 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T18:46:21.187 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:21.195 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:21.204 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:21.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:21.220 INFO:teuthology.orchestra.run.smithi050.stdout: "server_port": { 2024-04-23T18:46:21.227 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_port", 2024-04-23T18:46:21.235 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:21.242 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:21.250 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:21.258 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "9283", 2024-04-23T18:46:21.302 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:21.378 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:21.388 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:21.396 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T18:46:21.404 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:21.413 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:21.421 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:21.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:21.438 INFO:teuthology.orchestra.run.smithi050.stdout: "stale_cache_strategy": { 2024-04-23T18:46:21.446 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "stale_cache_strategy", 2024-04-23T18:46:21.454 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:21.462 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:21.470 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:21.478 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "log", 2024-04-23T18:46:21.485 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:21.493 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:21.502 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:21.605 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:21.613 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:21.623 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:21.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:21.643 INFO:teuthology.orchestra.run.smithi050.stdout: "standby_behaviour": { 2024-04-23T18:46:21.651 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "standby_behaviour", 2024-04-23T18:46:21.659 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:21.668 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:21.676 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:21.685 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "default", 2024-04-23T18:46:21.693 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:21.701 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:21.711 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:21.719 INFO:teuthology.orchestra.run.smithi050.stdout: "default", 2024-04-23T18:46:21.728 INFO:teuthology.orchestra.run.smithi050.stdout: "error" 2024-04-23T18:46:21.736 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:21.742 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:21.804 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:21.811 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:21.821 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:21.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:21.838 INFO:teuthology.orchestra.run.smithi050.stdout: "standby_error_status_code": { 2024-04-23T18:46:21.848 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "standby_error_status_code", 2024-04-23T18:46:21.857 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:21.866 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:21.875 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:21.883 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "500", 2024-04-23T18:46:21.892 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "400", 2024-04-23T18:46:21.900 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "599", 2024-04-23T18:46:21.909 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:21.919 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:21.982 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:22.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:22.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:22.077 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rbd_support", 2024-04-23T18:46:22.086 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:22.095 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:22.104 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:22.113 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:22.122 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:22.131 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:22.139 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:22.147 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:22.158 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:22.166 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:22.175 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:22.201 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:22.251 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:22.326 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:22.333 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:22.342 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:22.351 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:22.361 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:22.369 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:22.378 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:22.438 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:22.448 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:22.481 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:22.539 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:22.548 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:22.564 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:22.573 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:22.582 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:22.590 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:22.599 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:22.608 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:22.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:22.627 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:22.635 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:22.643 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:22.650 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:22.658 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:22.665 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:22.672 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:22.703 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:22.814 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:22.821 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:22.831 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:22.839 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:22.848 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:22.858 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:22.866 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:22.873 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:22.880 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:22.887 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:22.894 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:22.902 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:22.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:22.917 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:22.924 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:22.932 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:22.966 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:23.022 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:23.029 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:23.037 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:23.044 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:23.051 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:23.057 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:23.064 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:23.071 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:23.079 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:23.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:23.093 INFO:teuthology.orchestra.run.smithi050.stdout: "max_concurrent_snap_create": { 2024-04-23T18:46:23.104 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_concurrent_snap_create", 2024-04-23T18:46:23.112 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:23.120 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:23.128 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:23.139 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10", 2024-04-23T18:46:23.165 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:23.234 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:23.243 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:23.250 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:23.257 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:23.264 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:23.272 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:23.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:23.286 INFO:teuthology.orchestra.run.smithi050.stdout: "mirror_snapshot_schedule": { 2024-04-23T18:46:23.294 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T18:46:23.301 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:23.308 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:23.320 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:23.327 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:23.336 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:23.344 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:23.352 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:23.359 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:23.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:23.467 INFO:teuthology.orchestra.run.smithi050.stdout: "trash_purge_schedule": { 2024-04-23T18:46:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "trash_purge_schedule", 2024-04-23T18:46:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:23.528 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:23.567 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:23.577 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:23.586 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:23.625 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:23.697 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:23.711 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:23.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:23.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:23.743 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "restful", 2024-04-23T18:46:23.756 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:23.767 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:23.776 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:23.787 INFO:teuthology.orchestra.run.smithi050.stdout: "enable_auth": { 2024-04-23T18:46:23.797 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "enable_auth", 2024-04-23T18:46:23.806 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:23.818 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:23.832 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:23.843 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:46:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:23.888 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:23.949 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:23.961 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:23.971 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "key_file": { 2024-04-23T18:46:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "key_file", 2024-04-23T18:46:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:24.075 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:24.085 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:24.095 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:24.136 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:24.203 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:24.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:24.221 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:24.238 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:24.247 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:24.255 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:24.263 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:24.274 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:24.283 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:24.292 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:24.302 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:24.310 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:24.318 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:24.328 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:24.338 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:24.366 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:24.421 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:24.446 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:24.550 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:24.563 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:24.573 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:24.583 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:24.613 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:24.686 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:24.697 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:24.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:24.770 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:24.781 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:24.790 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:24.799 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:24.809 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:24.819 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:24.829 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:24.837 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:24.845 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:24.876 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:25.098 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:25.162 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:25.170 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:25.178 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:25.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:25.194 INFO:teuthology.orchestra.run.smithi050.stdout: "server_addr": { 2024-04-23T18:46:25.204 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_addr", 2024-04-23T18:46:25.212 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:25.224 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:25.232 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:25.242 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:25.254 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:25.262 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:25.273 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:25.282 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:25.291 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:25.302 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:25.328 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:25.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:25.390 INFO:teuthology.orchestra.run.smithi050.stdout: "server_port": { 2024-04-23T18:46:25.401 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "server_port", 2024-04-23T18:46:25.411 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:25.421 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:25.432 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:25.442 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:25.455 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:25.464 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:25.473 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:25.484 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:25.493 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:25.502 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:25.511 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:25.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:25.533 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:25.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:25.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:25.652 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rgw", 2024-04-23T18:46:25.662 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:25.672 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:25.685 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:25.693 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:25.703 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:25.712 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:25.734 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:25.743 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:25.751 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:25.761 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:25.772 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:25.782 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:25.790 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:25.896 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:25.912 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:25.922 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:25.930 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:25.938 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:25.949 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:25.958 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:25.967 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:25.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:25.987 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:25.996 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:26.004 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:26.012 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:26.021 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:26.030 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:26.037 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:26.077 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:26.159 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:26.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:26.211 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:26.222 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:26.233 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:26.244 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:26.252 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:26.262 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:26.273 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:26.282 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:26.291 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:26.300 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:26.336 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:26.404 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:26.418 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:26.430 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:26.439 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:26.446 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:26.457 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:26.465 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:26.471 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:26.479 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:26.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:26.498 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:26.506 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:26.516 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:26.526 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:26.532 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:26.543 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:26.578 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:26.667 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:26.675 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:26.683 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:26.691 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:26.698 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:26.705 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:26.712 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:26.720 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:26.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:26.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:26.746 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rook", 2024-04-23T18:46:26.754 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:26.761 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:26.769 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:26.776 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:26.783 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:26.816 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:26.875 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:26.883 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:26.891 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:26.898 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:26.906 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:26.915 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:26.924 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:26.935 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:26.945 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:26.973 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:26.982 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:26.991 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:27.001 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:27.011 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:27.019 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:27.028 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:27.037 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:27.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:27.141 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:27.153 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:27.163 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:27.172 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:27.183 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:27.192 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:27.199 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:27.209 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:27.222 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:27.235 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:27.246 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:27.255 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:27.265 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:27.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:27.283 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:27.344 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:27.426 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:27.438 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:27.447 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:27.455 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:27.465 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:27.476 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:27.483 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:27.490 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:27.497 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:27.504 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:27.512 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:27.521 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:27.529 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:27.536 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:27.543 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:27.552 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:27.591 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:27.648 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:27.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:27.666 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:27.673 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:27.681 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:27.690 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:27.698 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:27.707 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:27.714 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:27.722 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:27.731 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:27.737 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:27.745 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:27.753 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:27.760 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:27.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:27.804 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_class": { 2024-04-23T18:46:27.873 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "storage_class", 2024-04-23T18:46:27.886 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:27.896 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:27.906 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:27.916 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "local", 2024-04-23T18:46:27.925 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:27.933 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:27.941 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:27.950 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T18:46:27.964 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:27.975 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:27.984 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:27.996 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:28.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:28.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:28.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:28.053 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "selftest", 2024-04-23T18:46:28.108 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:28.116 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:28.126 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:28.138 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:28.145 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:28.153 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:28.163 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:28.178 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:28.186 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:28.194 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:28.203 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:28.212 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:28.221 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:28.229 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:28.236 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:28.244 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:28.281 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:28.365 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:28.376 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:28.385 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:28.394 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:28.403 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:28.474 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:28.483 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:28.492 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:28.501 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:28.511 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:28.545 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:28.623 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:28.632 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:28.643 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:28.660 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:28.669 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:28.678 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:28.688 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:28.698 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:28.709 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:28.718 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:28.731 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:28.741 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:28.751 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:28.759 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:28.771 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:28.874 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:28.885 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:28.895 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:28.906 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:28.916 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:28.926 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:28.935 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:28.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:28.959 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:28.975 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:28.984 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:28.998 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:29.011 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:29.021 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:29.030 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:29.051 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:29.107 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:29.126 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:29.138 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:29.147 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:29.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:29.173 INFO:teuthology.orchestra.run.smithi050.stdout: "roption1": { 2024-04-23T18:46:29.182 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "roption1", 2024-04-23T18:46:29.191 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:29.199 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:29.210 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:29.219 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:29.228 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:29.251 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:29.258 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:29.266 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:29.275 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:29.310 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:29.368 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:29.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:29.388 INFO:teuthology.orchestra.run.smithi050.stdout: "roption2": { 2024-04-23T18:46:29.395 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "roption2", 2024-04-23T18:46:29.406 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:29.413 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:29.423 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:29.430 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "xyz", 2024-04-23T18:46:29.440 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:29.449 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:29.458 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:29.466 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:29.474 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:29.481 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:29.488 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:29.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:29.532 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption1": { 2024-04-23T18:46:29.582 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption1", 2024-04-23T18:46:29.591 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:29.598 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:29.607 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:29.615 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:29.623 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:29.631 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:29.643 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:29.652 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:29.660 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:29.672 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:29.680 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:29.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:29.700 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption2": { 2024-04-23T18:46:29.710 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption2", 2024-04-23T18:46:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:29.888 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:29.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:29.914 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption3": { 2024-04-23T18:46:29.921 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption3", 2024-04-23T18:46:29.929 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "float", 2024-04-23T18:46:29.937 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:29.947 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:29.957 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:29.994 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption4": { 2024-04-23T18:46:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption4", 2024-04-23T18:46:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:30.194 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:30.235 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:30.293 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:30.312 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:30.321 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:30.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:30.339 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption5": { 2024-04-23T18:46:30.350 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption5", 2024-04-23T18:46:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:30.369 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:30.379 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:30.388 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:30.433 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:30.443 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:30.490 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:30.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:30.585 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption6": { 2024-04-23T18:46:30.597 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption6", 2024-04-23T18:46:30.610 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:46:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:30.712 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:30.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:30.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rwoption7": { 2024-04-23T18:46:30.795 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rwoption7", 2024-04-23T18:46:30.886 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:30.895 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:30.903 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:30.910 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:30.917 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "1", 2024-04-23T18:46:30.925 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "42", 2024-04-23T18:46:30.932 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:30.939 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:30.946 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:30.960 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:30.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:30.974 INFO:teuthology.orchestra.run.smithi050.stdout: "testkey": { 2024-04-23T18:46:30.981 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "testkey", 2024-04-23T18:46:30.988 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:30.995 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:31.031 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:31.118 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:31.125 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: "testlkey": { 2024-04-23T18:46:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "testlkey", 2024-04-23T18:46:31.184 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:31.192 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:31.201 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:31.208 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:31.215 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:31.331 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:31.342 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:31.351 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:31.363 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:31.376 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:31.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:31.398 INFO:teuthology.orchestra.run.smithi050.stdout: "testnewline": { 2024-04-23T18:46:31.405 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "testnewline", 2024-04-23T18:46:31.413 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:31.421 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:31.429 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:31.438 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:31.446 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:31.461 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:31.516 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:31.525 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:31.569 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "snap_schedule", 2024-04-23T18:46:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_m_granularity": { 2024-04-23T18:46:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "allow_m_granularity", 2024-04-23T18:46:31.743 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:31.756 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:31.765 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:31.774 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:31.782 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:31.856 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:31.869 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:31.879 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T18:46:31.888 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:31.899 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:31.908 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:31.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:31.928 INFO:teuthology.orchestra.run.smithi050.stdout: "dump_on_update": { 2024-04-23T18:46:31.937 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "dump_on_update", 2024-04-23T18:46:31.947 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:31.955 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:31.964 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:31.972 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:31.980 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:31.989 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:31.998 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:32.027 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "dump database to debug log on update", 2024-04-23T18:46:32.095 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:32.107 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:32.116 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:32.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:32.135 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:32.144 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:32.154 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:32.163 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:32.172 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:32.181 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:32.190 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:32.205 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:32.213 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:32.223 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:32.233 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:32.242 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:32.273 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:32.361 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:32.371 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:32.381 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:32.389 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:32.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:32.407 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:32.420 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:32.431 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:32.448 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:32.458 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:32.473 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:32.486 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:32.497 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:32.528 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:32.611 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:32.620 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:32.635 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:32.644 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:32.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:32.665 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:32.674 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:32.684 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:32.694 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:32.704 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:32.711 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:32.719 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:32.726 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:32.735 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:32.742 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:32.751 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:32.783 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:32.862 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:32.869 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:32.878 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:32.888 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:32.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:32.904 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:32.913 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:32.921 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:32.929 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:32.936 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:32.944 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:32.951 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:32.962 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:33.023 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:33.057 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:33.063 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:33.072 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:33.087 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:33.094 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:33.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:33.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:33.122 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "stats", 2024-04-23T18:46:33.131 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:33.143 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:33.152 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:33.161 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:33.171 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:33.179 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:33.213 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:33.271 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:33.278 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:33.298 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:33.312 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:33.319 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:33.327 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:33.337 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:33.344 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:33.354 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:33.363 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:33.372 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:33.380 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:33.393 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:33.402 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:33.418 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:33.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:33.523 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:33.535 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:33.547 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:33.557 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:33.566 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:33.576 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:33.586 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:33.597 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:33.606 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:33.619 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:33.629 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:33.639 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:33.648 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:33.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:33.668 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:33.677 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:33.720 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:33.775 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:33.784 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:33.794 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:33.803 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:33.812 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:33.831 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:33.839 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:33.850 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:33.861 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:33.869 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:33.878 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:33.887 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:33.895 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:33.904 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:33.914 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:33.950 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:34.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:34.015 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:34.023 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:34.032 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:34.041 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:34.050 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:34.059 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:34.070 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:34.079 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:34.087 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:34.098 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:34.107 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:34.125 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:34.134 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:34.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:34.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:34.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:34.234 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "status", 2024-04-23T18:46:34.243 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:34.256 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:34.266 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:34.276 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:34.285 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:34.294 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:34.306 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:34.315 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:34.324 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:34.334 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:34.343 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:34.353 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:34.362 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:34.371 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:34.404 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:34.464 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:34.473 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:34.483 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:34.600 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:34.635 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:34.718 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:34.727 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:34.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:34.744 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:34.754 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:34.762 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:34.770 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:34.778 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:34.785 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:34.792 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:34.799 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:34.807 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:34.814 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:34.849 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:34.898 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:34.918 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:34.926 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:34.934 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:34.941 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:34.949 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:34.957 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:34.965 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:34.972 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:34.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:34.986 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:34.993 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:35.000 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:35.007 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:35.013 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:35.022 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:35.054 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:35.122 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:35.132 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:35.140 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:35.148 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:35.155 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:35.163 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:35.172 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:35.181 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:35.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:35.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:35.210 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "telegraf", 2024-04-23T18:46:35.219 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:35.228 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:35.237 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:35.247 INFO:teuthology.orchestra.run.smithi050.stdout: "address": { 2024-04-23T18:46:35.262 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "address", 2024-04-23T18:46:35.300 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:35.341 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:35.365 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:35.374 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T18:46:35.383 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:35.393 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:35.409 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:35.418 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:35.426 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:35.438 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:35.447 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:35.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:35.465 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:46:35.473 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:46:35.482 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:46:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:35.501 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:35.540 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "15", 2024-04-23T18:46:35.596 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:35.605 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:35.614 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:35.624 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:35.707 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:35.717 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:35.726 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:35.766 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:35.828 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:35.839 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:35.847 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:35.855 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:35.864 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:35.890 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:35.898 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:35.909 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:35.919 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:35.931 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:35.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:35.948 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:35.957 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:35.966 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:35.974 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:36.005 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:36.081 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:36.107 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:36.113 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:36.121 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:36.128 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:36.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:36.145 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:36.154 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:36.162 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:36.171 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:36.178 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:36.186 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:36.193 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:36.236 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:36.333 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:36.340 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:36.348 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:36.357 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:36.366 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:36.374 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:36.391 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:36.400 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:36.409 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:36.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:36.427 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:36.436 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:36.445 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:36.483 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:36.533 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:36.543 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:36.552 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:36.561 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:36.571 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:36.580 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:36.589 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:36.598 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:36.606 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:36.616 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:36.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:36.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:36.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:36.653 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "telemetry", 2024-04-23T18:46:36.661 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:36.668 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:36.696 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:36.738 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_basic": { 2024-04-23T18:46:36.764 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_basic", 2024-04-23T18:46:36.772 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:36.781 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:36.790 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:36.799 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:46:36.808 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:36.818 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:36.827 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:36.835 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T18:46:36.845 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:36.854 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:36.863 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_crash": { 2024-04-23T18:46:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_crash", 2024-04-23T18:46:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:46:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T18:46:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_device": { 2024-04-23T18:46:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_device", 2024-04-23T18:46:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:37.132 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:37.165 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:46:37.220 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:37.230 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:37.240 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:37.249 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T18:46:37.260 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:37.270 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:37.279 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:37.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:37.297 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_ident": { 2024-04-23T18:46:37.307 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_ident", 2024-04-23T18:46:37.315 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:37.324 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:37.333 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:37.342 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:37.468 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T18:46:37.477 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:37.486 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:37.495 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:37.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:37.513 INFO:teuthology.orchestra.run.smithi050.stdout: "channel_perf": { 2024-04-23T18:46:37.522 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "channel_perf", 2024-04-23T18:46:37.531 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:37.540 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:37.549 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T18:46:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:37.697 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:37.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:37.716 INFO:teuthology.orchestra.run.smithi050.stdout: "contact": { 2024-04-23T18:46:37.724 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "contact", 2024-04-23T18:46:37.732 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:37.740 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:37.748 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:37.756 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:37.763 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:37.771 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:37.779 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:37.791 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:37.806 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: "description": { 2024-04-23T18:46:37.915 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "description", 2024-04-23T18:46:37.924 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:37.932 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:37.941 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:37.950 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:37.971 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:37.979 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:37.989 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:37.998 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:38.007 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:38.015 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:38.023 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:38.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:38.039 INFO:teuthology.orchestra.run.smithi050.stdout: "device_url": { 2024-04-23T18:46:38.047 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "device_url", 2024-04-23T18:46:38.055 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:38.085 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:38.129 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:38.138 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T18:46:38.147 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:38.155 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:38.164 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:38.173 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:38.182 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:38.191 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:38.200 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:38.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:38.214 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": { 2024-04-23T18:46:38.221 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "enabled", 2024-04-23T18:46:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:38.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:38.382 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:46:38.390 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:46:38.397 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:38.406 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "24", 2024-04-23T18:46:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "8", 2024-04-23T18:46:38.439 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:38.562 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:38.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:38.580 INFO:teuthology.orchestra.run.smithi050.stdout: "last_opt_revision": { 2024-04-23T18:46:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "last_opt_revision", 2024-04-23T18:46:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "1", 2024-04-23T18:46:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:38.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:38.771 INFO:teuthology.orchestra.run.smithi050.stdout: "leaderboard": { 2024-04-23T18:46:38.780 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "leaderboard", 2024-04-23T18:46:38.789 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:38.798 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:38.807 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:38.820 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:38.829 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:38.841 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:38.849 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:38.858 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:38.866 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:38.874 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:38.883 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:38.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:38.899 INFO:teuthology.orchestra.run.smithi050.stdout: "leaderboard_description": { 2024-04-23T18:46:38.908 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "leaderboard_description", 2024-04-23T18:46:38.961 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:38.969 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:39.020 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:39.498 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:39.505 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:39.535 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:39.579 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:39.587 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:39.594 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:39.602 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:39.609 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:39.616 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:39.623 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:39.630 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:39.637 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:39.645 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:39.652 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:39.658 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:39.665 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:39.671 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:39.678 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:39.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:39.762 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:39.770 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:39.777 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:39.785 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:39.792 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:39.799 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:39.824 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:39.832 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:39.838 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:39.848 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:39.854 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:39.864 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:39.872 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:39.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:39.887 INFO:teuthology.orchestra.run.smithi050.stdout: "organization": { 2024-04-23T18:46:39.896 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "organization", 2024-04-23T18:46:39.904 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:39.965 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:39.974 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:39.983 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:39.991 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:40.000 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:40.008 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:40.019 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:40.027 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:40.037 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:40.046 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:40.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: "proxy": { 2024-04-23T18:46:40.072 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "proxy", 2024-04-23T18:46:40.081 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:40.090 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:40.098 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:40.134 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:40.204 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:40.213 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:40.222 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:40.232 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:40.241 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:40.250 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:40.258 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:40.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:40.276 INFO:teuthology.orchestra.run.smithi050.stdout: "url": { 2024-04-23T18:46:40.284 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "url", 2024-04-23T18:46:40.293 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:40.302 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:40.310 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:40.318 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T18:46:40.326 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:40.335 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:40.374 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:40.401 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:40.456 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:40.466 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:40.473 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:40.482 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:40.491 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:40.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:40.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:40.518 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test_orchestrator", 2024-04-23T18:46:40.527 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:40.535 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:40.543 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:40.552 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:40.561 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:40.568 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:40.577 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:40.611 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:40.667 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:40.676 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:40.685 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:40.694 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:40.701 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:40.709 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:40.718 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:40.726 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:40.734 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:40.743 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:40.749 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:40.756 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:40.762 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:40.771 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:40.780 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:40.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:40.863 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:40.877 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:40.887 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:40.897 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:40.907 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:40.916 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:40.925 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:40.934 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:40.943 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:40.952 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:40.961 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:40.969 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:40.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:40.984 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:40.991 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:41.065 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:41.073 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:41.081 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:41.090 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:41.099 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:41.108 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:41.116 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:41.125 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:41.134 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:41.141 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:41.149 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:41.158 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:41.168 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:41.177 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:41.186 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:41.195 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:41.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:41.279 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:41.287 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:41.296 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:41.305 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:41.316 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:41.324 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:41.334 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:41.342 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:41.349 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:41.357 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:41.364 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:41.370 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:41.377 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:41.384 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:41.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:41.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:41.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:41.497 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "volumes", 2024-04-23T18:46:41.506 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:41.514 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:41.523 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:41.531 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:41.540 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:41.548 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:41.555 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:41.564 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:41.572 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:41.579 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:41.586 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:41.592 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:41.601 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:41.609 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:41.617 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:41.670 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:41.752 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:41.761 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:41.769 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:41.778 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:41.786 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:41.795 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:41.803 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:41.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:41.820 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:41.828 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:41.838 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:41.853 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:41.862 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:41.872 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:41.883 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:41.892 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:41.935 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:41.995 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:42.006 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:42.015 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:42.024 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:42.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:42.041 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:42.049 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:42.057 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:42.066 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:42.075 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:42.084 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:42.092 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:42.101 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:42.110 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:42.119 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:42.128 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:42.167 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:42.218 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:42.226 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:42.237 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:42.246 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:42.255 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:42.264 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:42.273 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:42.282 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:42.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:42.301 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:42.310 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:42.320 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:42.333 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:42.437 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:42.447 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:42.456 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:42.464 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:42.472 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:42.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:42.489 INFO:teuthology.orchestra.run.smithi050.stdout: "max_concurrent_clones": { 2024-04-23T18:46:42.499 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "max_concurrent_clones", 2024-04-23T18:46:42.509 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:42.518 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:42.526 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:42.534 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "4", 2024-04-23T18:46:42.543 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:42.553 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:42.562 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:42.570 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T18:46:42.577 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:42.638 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:42.645 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:42.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:42.662 INFO:teuthology.orchestra.run.smithi050.stdout: "periodic_async_work": { 2024-04-23T18:46:42.669 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "periodic_async_work", 2024-04-23T18:46:42.686 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:42.693 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:42.699 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:42.706 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:42.713 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:42.720 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:42.727 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:42.734 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Periodically check for async work", 2024-04-23T18:46:42.740 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:42.746 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:42.753 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:42.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:42.855 INFO:teuthology.orchestra.run.smithi050.stdout: "snapshot_clone_delay": { 2024-04-23T18:46:42.862 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "snapshot_clone_delay", 2024-04-23T18:46:42.870 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:42.877 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:42.883 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:42.889 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "0", 2024-04-23T18:46:42.895 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:42.905 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:42.913 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:42.920 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T18:46:42.927 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "snapshot_clone_no_wait": { 2024-04-23T18:46:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T18:46:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:43.043 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:43.051 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:43.060 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "True", 2024-04-23T18:46:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:43.076 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:43.086 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:43.095 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T18:46:43.103 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:43.112 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:43.120 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:43.129 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:43.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:43.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:43.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:43.164 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "zabbix", 2024-04-23T18:46:43.173 INFO:teuthology.orchestra.run.smithi050.stdout: "can_run": true, 2024-04-23T18:46:43.210 INFO:teuthology.orchestra.run.smithi050.stdout: "error_string": "", 2024-04-23T18:46:43.263 INFO:teuthology.orchestra.run.smithi050.stdout: "module_options": { 2024-04-23T18:46:43.271 INFO:teuthology.orchestra.run.smithi050.stdout: "discovery_interval": { 2024-04-23T18:46:43.280 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "discovery_interval", 2024-04-23T18:46:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "uint", 2024-04-23T18:46:43.300 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:43.320 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "100", 2024-04-23T18:46:43.329 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:43.338 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:43.347 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:43.356 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "identifier": { 2024-04-23T18:46:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "identifier", 2024-04-23T18:46:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:43.496 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:43.504 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:43.513 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:43.521 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:43.530 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:43.539 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:43.547 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:43.555 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:43.562 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:43.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:43.579 INFO:teuthology.orchestra.run.smithi050.stdout: "interval": { 2024-04-23T18:46:43.587 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "interval", 2024-04-23T18:46:43.595 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "secs", 2024-04-23T18:46:43.603 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:43.645 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:43.736 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "60", 2024-04-23T18:46:43.745 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:43.755 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:43.764 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:43.772 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:43.781 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:43.790 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:43.798 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:43.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: "log_level": { 2024-04-23T18:46:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_level", 2024-04-23T18:46:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:43.968 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:43.976 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:43.984 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:44.009 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:44.016 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:44.023 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:44.032 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:44.038 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:44.045 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:44.053 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:44.060 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:44.067 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:44.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:44.082 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster": { 2024-04-23T18:46:44.089 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster", 2024-04-23T18:46:44.096 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:44.130 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:44.189 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:44.197 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:44.206 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:44.221 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:44.235 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:44.243 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:44.250 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:44.259 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:44.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:44.276 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_cluster_level": { 2024-04-23T18:46:44.285 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_cluster_level", 2024-04-23T18:46:44.294 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:44.313 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:44.322 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "info", 2024-04-23T18:46:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [ 2024-04-23T18:46:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: "", 2024-04-23T18:46:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: "critical", 2024-04-23T18:46:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: "debug", 2024-04-23T18:46:44.444 INFO:teuthology.orchestra.run.smithi050.stdout: "error", 2024-04-23T18:46:44.453 INFO:teuthology.orchestra.run.smithi050.stdout: "info", 2024-04-23T18:46:44.463 INFO:teuthology.orchestra.run.smithi050.stdout: "warning" 2024-04-23T18:46:44.471 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:44.480 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:44.488 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:44.499 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:44.508 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:44.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:44.526 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": { 2024-04-23T18:46:44.534 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "log_to_file", 2024-04-23T18:46:44.543 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "bool", 2024-04-23T18:46:44.588 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:44.597 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-23T18:46:44.606 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "False", 2024-04-23T18:46:44.615 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:44.623 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:44.634 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:44.642 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:44.650 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:44.659 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:44.668 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:44.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:44.684 INFO:teuthology.orchestra.run.smithi050.stdout: "zabbix_host": { 2024-04-23T18:46:44.692 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "zabbix_host", 2024-04-23T18:46:44.700 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:44.709 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:44.717 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:44.748 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "", 2024-04-23T18:46:44.793 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:44.802 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:44.810 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:44.819 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:44.827 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:44.836 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:44.845 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:44.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:44.861 INFO:teuthology.orchestra.run.smithi050.stdout: "zabbix_port": { 2024-04-23T18:46:44.872 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "zabbix_port", 2024-04-23T18:46:44.880 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "int", 2024-04-23T18:46:44.889 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:44.898 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:44.909 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "10051", 2024-04-23T18:46:44.917 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:44.927 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:44.959 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:45.021 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:45.029 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:45.036 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:45.043 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:45.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:45.057 INFO:teuthology.orchestra.run.smithi050.stdout: "zabbix_sender": { 2024-04-23T18:46:45.063 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "zabbix_sender", 2024-04-23T18:46:45.069 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "str", 2024-04-23T18:46:45.077 INFO:teuthology.orchestra.run.smithi050.stdout: "level": "advanced", 2024-04-23T18:46:45.084 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 0, 2024-04-23T18:46:45.091 INFO:teuthology.orchestra.run.smithi050.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T18:46:45.099 INFO:teuthology.orchestra.run.smithi050.stdout: "min": "", 2024-04-23T18:46:45.106 INFO:teuthology.orchestra.run.smithi050.stdout: "max": "", 2024-04-23T18:46:45.113 INFO:teuthology.orchestra.run.smithi050.stdout: "enum_allowed": [], 2024-04-23T18:46:45.120 INFO:teuthology.orchestra.run.smithi050.stdout: "desc": "", 2024-04-23T18:46:45.129 INFO:teuthology.orchestra.run.smithi050.stdout: "long_desc": "", 2024-04-23T18:46:45.166 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2024-04-23T18:46:45.218 INFO:teuthology.orchestra.run.smithi050.stdout: "see_also": [] 2024-04-23T18:46:45.224 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:45.231 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:45.238 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:45.246 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:45.253 INFO:teuthology.orchestra.run.smithi050.stdout: "services": { 2024-04-23T18:46:45.260 INFO:teuthology.orchestra.run.smithi050.stdout: "dashboard": "https://172.21.15.50:8443/" 2024-04-23T18:46:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "always_on_modules": { 2024-04-23T18:46:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus": [ 2024-04-23T18:46:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "balancer", 2024-04-23T18:46:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "crash", 2024-04-23T18:46:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "devicehealth", 2024-04-23T18:46:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "orchestrator", 2024-04-23T18:46:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscaler", 2024-04-23T18:46:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "progress", 2024-04-23T18:46:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_support", 2024-04-23T18:46:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: "status", 2024-04-23T18:46:45.403 INFO:teuthology.orchestra.run.smithi050.stdout: "telemetry", 2024-04-23T18:46:45.413 INFO:teuthology.orchestra.run.smithi050.stdout: "volumes" 2024-04-23T18:46:45.420 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:45.428 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific": [ 2024-04-23T18:46:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "balancer", 2024-04-23T18:46:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: "crash", 2024-04-23T18:46:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: "devicehealth", 2024-04-23T18:46:45.454 INFO:teuthology.orchestra.run.smithi050.stdout: "orchestrator", 2024-04-23T18:46:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscaler", 2024-04-23T18:46:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: "progress", 2024-04-23T18:46:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_support", 2024-04-23T18:46:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "status", 2024-04-23T18:46:45.490 INFO:teuthology.orchestra.run.smithi050.stdout: "telemetry", 2024-04-23T18:46:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: "volumes" 2024-04-23T18:46:45.505 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:45.535 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy": [ 2024-04-23T18:46:45.634 INFO:teuthology.orchestra.run.smithi050.stdout: "balancer", 2024-04-23T18:46:45.642 INFO:teuthology.orchestra.run.smithi050.stdout: "crash", 2024-04-23T18:46:45.651 INFO:teuthology.orchestra.run.smithi050.stdout: "devicehealth", 2024-04-23T18:46:45.660 INFO:teuthology.orchestra.run.smithi050.stdout: "orchestrator", 2024-04-23T18:46:45.670 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscaler", 2024-04-23T18:46:45.679 INFO:teuthology.orchestra.run.smithi050.stdout: "progress", 2024-04-23T18:46:45.687 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_support", 2024-04-23T18:46:45.697 INFO:teuthology.orchestra.run.smithi050.stdout: "status", 2024-04-23T18:46:45.706 INFO:teuthology.orchestra.run.smithi050.stdout: "telemetry", 2024-04-23T18:46:45.715 INFO:teuthology.orchestra.run.smithi050.stdout: "volumes" 2024-04-23T18:46:45.723 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "reef": [ 2024-04-23T18:46:45.742 INFO:teuthology.orchestra.run.smithi050.stdout: "balancer", 2024-04-23T18:46:45.750 INFO:teuthology.orchestra.run.smithi050.stdout: "crash", 2024-04-23T18:46:45.759 INFO:teuthology.orchestra.run.smithi050.stdout: "devicehealth", 2024-04-23T18:46:45.767 INFO:teuthology.orchestra.run.smithi050.stdout: "orchestrator", 2024-04-23T18:46:45.778 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscaler", 2024-04-23T18:46:45.844 INFO:teuthology.orchestra.run.smithi050.stdout: "progress", 2024-04-23T18:46:45.852 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_support", 2024-04-23T18:46:45.907 INFO:teuthology.orchestra.run.smithi050.stdout: "status", 2024-04-23T18:46:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: "telemetry", 2024-04-23T18:46:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: "volumes" 2024-04-23T18:46:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-23T18:46:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: "squid": [ 2024-04-23T18:46:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: "balancer", 2024-04-23T18:46:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: "crash", 2024-04-23T18:46:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: "devicehealth", 2024-04-23T18:46:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: "orchestrator", 2024-04-23T18:46:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscaler", 2024-04-23T18:46:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "progress", 2024-04-23T18:46:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_support", 2024-04-23T18:46:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: "status", 2024-04-23T18:46:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: "telemetry", 2024-04-23T18:46:46.075 INFO:teuthology.orchestra.run.smithi050.stdout: "volumes" 2024-04-23T18:46:46.149 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-23T18:46:46.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:46.166 INFO:teuthology.orchestra.run.smithi050.stdout: "last_failure_osd_epoch": 90, 2024-04-23T18:46:46.175 INFO:teuthology.orchestra.run.smithi050.stdout: "active_clients": [ 2024-04-23T18:46:46.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:46.195 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "devicehealth", 2024-04-23T18:46:46.204 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-23T18:46:46.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:46.222 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-23T18:46:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:0", 2024-04-23T18:46:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3229855169 2024-04-23T18:46:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-23T18:46:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:46.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:46.282 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "libcephsqlite", 2024-04-23T18:46:46.310 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-23T18:46:46.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:46.380 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-23T18:46:46.389 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:0", 2024-04-23T18:46:46.398 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 550178861 2024-04-23T18:46:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:46.414 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-23T18:46:46.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:46.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:46.440 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "rbd_support", 2024-04-23T18:46:46.449 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-23T18:46:46.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:46.465 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-23T18:46:46.475 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:0", 2024-04-23T18:46:46.487 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3061711256 2024-04-23T18:46:46.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:46.504 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-23T18:46:46.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:46.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:46.613 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "snap_schedule", 2024-04-23T18:46:46.621 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-23T18:46:46.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:46.637 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-23T18:46:46.644 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:0", 2024-04-23T18:46:46.652 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 397138844 2024-04-23T18:46:46.658 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:46.664 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-23T18:46:46.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:46.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:46.686 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "snap_schedule", 2024-04-23T18:46:46.693 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-23T18:46:46.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:46.707 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-23T18:46:46.714 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:0", 2024-04-23T18:46:46.744 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 465314410 2024-04-23T18:46:46.817 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:46.824 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-23T18:46:46.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:46.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:46.849 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "volumes", 2024-04-23T18:46:46.857 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-23T18:46:46.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-23T18:46:46.871 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-23T18:46:46.878 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:0", 2024-04-23T18:46:46.886 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3658453227 2024-04-23T18:46:46.895 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:46.902 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-23T18:46:46.912 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:46.921 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-23T18:46:46.930 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:46:46.946 DEBUG:tasks.check_counter:Getting stats from a 2024-04-23T18:46:46.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-23T18:46:47.089 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:25 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2793906102' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T18:46:47.096 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:26 smithi188 ceph-mon[32694]: pgmap v698: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 5.4 MiB/s wr, 174 op/s 2024-04-23T18:46:47.105 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:28 smithi188 ceph-mon[32694]: pgmap v699: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 6.4 MiB/s wr, 214 op/s 2024-04-23T18:46:47.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:30 smithi188 ceph-mon[32694]: pgmap v700: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 4.3 MiB/s wr, 148 op/s 2024-04-23T18:46:47.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:32 smithi188 ceph-mon[32694]: pgmap v701: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 3.6 MiB/s wr, 131 op/s 2024-04-23T18:46:47.132 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:34 smithi188 ceph-mon[32694]: pgmap v702: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 5.3 MiB/s wr, 192 op/s 2024-04-23T18:46:47.142 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:36 smithi188 ceph-mon[32694]: pgmap v703: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 3.2 MiB/s wr, 121 op/s 2024-04-23T18:46:47.151 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:38 smithi188 ceph-mon[32694]: pgmap v704: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 4.5 MiB/s wr, 153 op/s 2024-04-23T18:46:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:40 smithi188 ceph-mon[32694]: pgmap v705: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 3.8 MiB/s wr, 117 op/s 2024-04-23T18:46:47.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:42 smithi188 ceph-mon[32694]: pgmap v706: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 3.4 MiB/s wr, 101 op/s 2024-04-23T18:46:47.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:44 smithi188 ceph-mon[32694]: pgmap v707: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 3.5 MiB/s wr, 109 op/s 2024-04-23T18:46:47.189 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:46 smithi188 ceph-mon[32694]: pgmap v708: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 1.8 MiB/s wr, 48 op/s 2024-04-23T18:46:47.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:48 smithi188 ceph-mon[32694]: pgmap v709: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.8 MiB/s wr, 53 op/s 2024-04-23T18:46:47.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:50 smithi188 ceph-mon[32694]: pgmap v710: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 523 KiB/s wr, 22 op/s 2024-04-23T18:46:47.220 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:52 smithi188 ceph-mon[32694]: pgmap v711: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 61 KiB/s wr, 12 op/s 2024-04-23T18:46:47.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:54 smithi188 ceph-mon[32694]: pgmap v712: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 82 KiB/s wr, 17 op/s 2024-04-23T18:46:47.279 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:56 smithi188 ceph-mon[32694]: pgmap v713: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 10 op/s 2024-04-23T18:46:47.380 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:40:58 smithi188 ceph-mon[32694]: pgmap v714: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 14 op/s 2024-04-23T18:46:47.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:00 smithi188 ceph-mon[32694]: pgmap v715: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 9 op/s 2024-04-23T18:46:47.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:02 smithi188 ceph-mon[32694]: pgmap v716: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 9 op/s 2024-04-23T18:46:47.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:04 smithi188 ceph-mon[32694]: pgmap v717: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 96 KiB/s wr, 30 op/s 2024-04-23T18:46:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:06 smithi188 ceph-mon[32694]: pgmap v718: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 75 KiB/s wr, 25 op/s 2024-04-23T18:46:47.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:06 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:47.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:06 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:47.448 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:06 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:47.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:06 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:47.468 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:06 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:47.477 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:07 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:47.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:07 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:47.494 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:07 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:47.503 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:08 smithi188 ceph-mon[32694]: pgmap v719: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 121 KiB/s wr, 29 op/s 2024-04-23T18:46:47.514 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:10 smithi188 ceph-mon[32694]: pgmap v720: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 106 KiB/s wr, 24 op/s 2024-04-23T18:46:47.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:10 smithi188 ceph-mon[32694]: osdmap e129: 12 total, 12 up, 12 in 2024-04-23T18:46:47.576 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:12 smithi188 ceph-mon[32694]: pgmap v722: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 127 KiB/s wr, 29 op/s 2024-04-23T18:46:47.641 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:14 smithi188 ceph-mon[32694]: pgmap v723: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 179 KiB/s wr, 13 op/s 2024-04-23T18:46:47.657 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:16 smithi188 ceph-mon[32694]: pgmap v724: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 179 KiB/s wr, 13 op/s 2024-04-23T18:46:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:16 smithi188 ceph-mon[32694]: osdmap e130: 12 total, 12 up, 12 in 2024-04-23T18:46:47.673 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:18 smithi188 ceph-mon[32694]: pgmap v726: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 175 KiB/s wr, 17 op/s 2024-04-23T18:46:47.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:20 smithi188 ceph-mon[32694]: pgmap v727: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 147 KiB/s wr, 14 op/s 2024-04-23T18:46:47.700 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:22 smithi188 ceph-mon[32694]: pgmap v728: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 140 KiB/s wr, 13 op/s 2024-04-23T18:46:47.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:24 smithi188 ceph-mon[32694]: pgmap v729: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 10 op/s 2024-04-23T18:46:47.718 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:26 smithi188 ceph-mon[32694]: pgmap v730: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 10 op/s 2024-04-23T18:46:47.726 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:28 smithi188 ceph-mon[32694]: pgmap v731: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 12 op/s 2024-04-23T18:46:47.734 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:30 smithi188 ceph-mon[32694]: pgmap v732: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-04-23T18:46:47.742 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:32 smithi188 ceph-mon[32694]: pgmap v733: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-04-23T18:46:47.751 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:34 smithi188 ceph-mon[32694]: pgmap v734: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-23T18:46:47.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:36 smithi188 ceph-mon[32694]: pgmap v735: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 KiB/s wr, 6 op/s 2024-04-23T18:46:47.773 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:38 smithi188 ceph-mon[32694]: pgmap v736: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2024-04-23T18:46:47.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:40 smithi188 ceph-mon[32694]: pgmap v737: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2024-04-23T18:46:47.790 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:42 smithi188 ceph-mon[32694]: pgmap v738: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2024-04-23T18:46:47.837 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:44 smithi188 ceph-mon[32694]: pgmap v739: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-23T18:46:47.895 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:46 smithi188 ceph-mon[32694]: pgmap v740: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.2 KiB/s wr, 5 op/s 2024-04-23T18:46:47.906 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:48 smithi188 ceph-mon[32694]: pgmap v741: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-23T18:46:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:50 smithi188 ceph-mon[32694]: pgmap v742: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 5 op/s 2024-04-23T18:46:47.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:52 smithi188 ceph-mon[32694]: pgmap v743: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 5 op/s 2024-04-23T18:46:47.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:54 smithi188 ceph-mon[32694]: pgmap v744: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 KiB/s wr, 8 op/s 2024-04-23T18:46:47.942 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:56 smithi188 ceph-mon[32694]: pgmap v745: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-04-23T18:46:47.951 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:41:58 smithi188 ceph-mon[32694]: pgmap v746: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 4.5 KiB/s wr, 7 op/s 2024-04-23T18:46:47.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:00 smithi188 ceph-mon[32694]: pgmap v747: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-04-23T18:46:47.969 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:02 smithi188 ceph-mon[32694]: pgmap v748: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-04-23T18:46:47.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:04 smithi188 ceph-mon[32694]: pgmap v749: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 53 KiB/s wr, 25 op/s 2024-04-23T18:46:47.986 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:05 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:46:47.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:05 smithi188 ceph-mon[32694]: pgmap v750: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 48 KiB/s wr, 22 op/s 2024-04-23T18:46:48.002 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:06 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:48.011 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:07 smithi188 ceph-mon[32694]: pgmap v751: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 48 KiB/s wr, 25 op/s 2024-04-23T18:46:48.020 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:07 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:48.031 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:07 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:48.041 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:07 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:48.104 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:09 smithi188 ceph-mon[32694]: pgmap v752: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 48 KiB/s wr, 22 op/s 2024-04-23T18:46:48.111 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:09 smithi188 ceph-mon[32694]: osdmap e131: 12 total, 12 up, 12 in 2024-04-23T18:46:48.119 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:11 smithi188 ceph-mon[32694]: pgmap v754: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 58 KiB/s wr, 27 op/s 2024-04-23T18:46:48.129 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:13 smithi188 ceph-mon[32694]: pgmap v755: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-23T18:46:48.138 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:14 smithi188 ceph-mon[32694]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:46:48.147 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:15 smithi188 ceph-mon[32694]: pgmap v756: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-23T18:46:48.156 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:17 smithi188 ceph-mon[32694]: osdmap e132: 12 total, 12 up, 12 in 2024-04-23T18:46:48.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:18 smithi188 ceph-mon[32694]: pgmap v758: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-04-23T18:46:48.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:20 smithi188 ceph-mon[32694]: pgmap v759: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-23T18:46:48.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:22 smithi188 ceph-mon[32694]: pgmap v760: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-23T18:46:48.188 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:24 smithi188 ceph-mon[32694]: pgmap v761: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-23T18:46:48.196 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:26 smithi188 ceph-mon[32694]: pgmap v762: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-23T18:46:48.203 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:28 smithi188 ceph-mon[32694]: pgmap v763: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-04-23T18:46:48.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:30 smithi188 ceph-mon[32694]: pgmap v764: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-04-23T18:46:48.219 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:32 smithi188 ceph-mon[32694]: pgmap v765: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-04-23T18:46:48.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:34 smithi188 ceph-mon[32694]: pgmap v766: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-04-23T18:46:48.265 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:36 smithi188 ceph-mon[32694]: pgmap v767: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-23T18:46:48.324 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:38 smithi188 ceph-mon[32694]: pgmap v768: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 7 op/s 2024-04-23T18:46:48.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:40 smithi188 ceph-mon[32694]: pgmap v769: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-23T18:46:48.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:42 smithi188 ceph-mon[32694]: pgmap v770: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-23T18:46:48.350 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:44 smithi188 ceph-mon[32694]: pgmap v771: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 7 op/s 2024-04-23T18:46:48.359 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:46 smithi188 ceph-mon[32694]: pgmap v772: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.367 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:48 smithi188 ceph-mon[32694]: pgmap v773: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:48.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:50 smithi188 ceph-mon[32694]: pgmap v774: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.382 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:52 smithi188 ceph-mon[32694]: pgmap v775: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.389 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:54 smithi188 ceph-mon[32694]: pgmap v776: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:48.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:56 smithi188 ceph-mon[32694]: pgmap v777: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:42:58 smithi188 ceph-mon[32694]: pgmap v778: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:48.410 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:00 smithi188 ceph-mon[32694]: pgmap v779: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:02 smithi188 ceph-mon[32694]: pgmap v780: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:04 smithi188 ceph-mon[32694]: pgmap v781: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:48.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:06 smithi188 ceph-mon[32694]: pgmap v782: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:48.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:08 smithi188 ceph-mon[32694]: pgmap v783: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:48.487 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:08 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:48.598 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:08 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:48.607 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:08 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:48.616 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:08 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:48.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:10 smithi188 ceph-mon[32694]: pgmap v784: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:48.634 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:10 smithi188 ceph-mon[32694]: osdmap e133: 12 total, 12 up, 12 in 2024-04-23T18:46:48.646 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:12 smithi188 ceph-mon[32694]: pgmap v786: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-23T18:46:48.656 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:14 smithi188 ceph-mon[32694]: pgmap v787: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:16 smithi188 ceph-mon[32694]: pgmap v788: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:48.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:16 smithi188 ceph-mon[32694]: osdmap e134: 12 total, 12 up, 12 in 2024-04-23T18:46:48.690 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:18 smithi188 ceph-mon[32694]: pgmap v790: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:46:48.699 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:20 smithi188 ceph-mon[32694]: pgmap v791: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2024-04-23T18:46:48.706 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:22 smithi188 ceph-mon[32694]: pgmap v792: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:48.715 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:24 smithi188 ceph-mon[32694]: pgmap v793: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:48.724 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:26 smithi188 ceph-mon[32694]: pgmap v794: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:48.731 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:28 smithi188 ceph-mon[32694]: pgmap v795: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:46:48.739 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:30 smithi188 ceph-mon[32694]: pgmap v796: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.764 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:32 smithi188 ceph-mon[32694]: pgmap v797: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.833 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:34 smithi188 ceph-mon[32694]: pgmap v798: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:48.842 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:36 smithi188 ceph-mon[32694]: pgmap v799: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:38 smithi188 ceph-mon[32694]: pgmap v800: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:48.860 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:40 smithi188 ceph-mon[32694]: pgmap v801: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.869 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:42 smithi188 ceph-mon[32694]: pgmap v802: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.878 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:44 smithi188 ceph-mon[32694]: pgmap v803: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:48.887 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:46 smithi188 ceph-mon[32694]: pgmap v804: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.897 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:48 smithi188 ceph-mon[32694]: pgmap v805: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:48.906 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:50 smithi188 ceph-mon[32694]: pgmap v806: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:52 smithi188 ceph-mon[32694]: pgmap v807: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.924 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:54 smithi188 ceph-mon[32694]: pgmap v808: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:48.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:56 smithi188 ceph-mon[32694]: pgmap v809: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.940 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:43:58 smithi188 ceph-mon[32694]: pgmap v810: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:48.949 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:00 smithi188 ceph-mon[32694]: pgmap v811: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.958 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:02 smithi188 ceph-mon[32694]: pgmap v812: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:48.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:04 smithi188 ceph-mon[32694]: pgmap v813: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:49.004 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:06 smithi188 ceph-mon[32694]: pgmap v814: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:49.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:08 smithi188 ceph-mon[32694]: pgmap v815: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:49.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:08 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:49.080 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:09 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:49.089 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:09 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:49.098 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:09 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:49.106 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:10 smithi188 ceph-mon[32694]: pgmap v816: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:49.116 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:10 smithi188 ceph-mon[32694]: osdmap e135: 12 total, 12 up, 12 in 2024-04-23T18:46:49.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:12 smithi188 ceph-mon[32694]: pgmap v818: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-23T18:46:49.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:14 smithi188 ceph-mon[32694]: pgmap v819: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:49.143 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:15 smithi188 ceph-mon[32694]: pgmap v820: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:49.152 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:17 smithi188 ceph-mon[32694]: osdmap e136: 12 total, 12 up, 12 in 2024-04-23T18:46:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:18 smithi188 ceph-mon[32694]: pgmap v822: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:46:49.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:20 smithi188 ceph-mon[32694]: pgmap v823: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2024-04-23T18:46:49.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:22 smithi188 ceph-mon[32694]: pgmap v824: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:49.190 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:24 smithi188 ceph-mon[32694]: pgmap v825: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:49.199 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:26 smithi188 ceph-mon[32694]: pgmap v826: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:49.234 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:28 smithi188 ceph-mon[32694]: pgmap v827: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:46:49.295 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:30 smithi188 ceph-mon[32694]: pgmap v828: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.303 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:32 smithi188 ceph-mon[32694]: pgmap v829: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.312 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:34 smithi188 ceph-mon[32694]: pgmap v830: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:49.321 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:36 smithi188 ceph-mon[32694]: pgmap v831: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.330 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:38 smithi188 ceph-mon[32694]: pgmap v832: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:49.339 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:40 smithi188 ceph-mon[32694]: pgmap v833: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:42 smithi188 ceph-mon[32694]: pgmap v834: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.356 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:44 smithi188 ceph-mon[32694]: pgmap v835: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:49.366 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:46 smithi188 ceph-mon[32694]: pgmap v836: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.377 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:48 smithi188 ceph-mon[32694]: pgmap v837: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:49.386 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:50 smithi188 ceph-mon[32694]: pgmap v838: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.394 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:52 smithi188 ceph-mon[32694]: pgmap v839: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:54 smithi188 ceph-mon[32694]: pgmap v840: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:56 smithi188 ceph-mon[32694]: pgmap v841: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:44:58 smithi188 ceph-mon[32694]: pgmap v842: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:49.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:00 smithi188 ceph-mon[32694]: pgmap v843: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:02 smithi188 ceph-mon[32694]: pgmap v844: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.536 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:04 smithi188 ceph-mon[32694]: pgmap v845: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:49.546 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:06 smithi188 ceph-mon[32694]: pgmap v846: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:49.555 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:08 smithi188 ceph-mon[32694]: pgmap v847: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:49.563 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:09 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:49.571 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:10 smithi188 ceph-mon[32694]: pgmap v848: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:49.579 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:10 smithi188 ceph-mon[32694]: osdmap e137: 12 total, 12 up, 12 in 2024-04-23T18:46:49.588 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:12 smithi188 ceph-mon[32694]: pgmap v850: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-23T18:46:49.596 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:14 smithi188 ceph-mon[32694]: pgmap v851: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:49.604 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:14 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:49.613 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:14 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:49.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:14 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:49.630 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:14 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:49.639 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:14 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:49.648 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:16 smithi188 ceph-mon[32694]: pgmap v852: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:49.658 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:16 smithi188 ceph-mon[32694]: osdmap e138: 12 total, 12 up, 12 in 2024-04-23T18:46:49.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:18 smithi188 ceph-mon[32694]: pgmap v854: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:46:49.709 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:20 smithi188 ceph-mon[32694]: pgmap v855: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2024-04-23T18:46:49.791 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:22 smithi188 ceph-mon[32694]: pgmap v856: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:49.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:24 smithi188 ceph-mon[32694]: pgmap v857: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:49.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:26 smithi188 ceph-mon[32694]: pgmap v858: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:49.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:28 smithi188 ceph-mon[32694]: pgmap v859: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:46:49.822 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:30 smithi188 ceph-mon[32694]: pgmap v860: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:32 smithi188 ceph-mon[32694]: pgmap v861: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:34 smithi188 ceph-mon[32694]: pgmap v862: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:49.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:36 smithi188 ceph-mon[32694]: pgmap v863: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.857 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:38 smithi188 ceph-mon[32694]: pgmap v864: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:49.863 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:40 smithi188 ceph-mon[32694]: pgmap v865: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.871 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:42 smithi188 ceph-mon[32694]: pgmap v866: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.878 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:44 smithi188 ceph-mon[32694]: pgmap v867: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:49.885 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:46 smithi188 ceph-mon[32694]: pgmap v868: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.893 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:48 smithi188 ceph-mon[32694]: pgmap v869: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:49.900 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:50 smithi188 ceph-mon[32694]: pgmap v870: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.908 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:52 smithi188 ceph-mon[32694]: pgmap v871: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:49.951 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:54 smithi188 ceph-mon[32694]: pgmap v872: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:50.023 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:56 smithi188 ceph-mon[32694]: pgmap v873: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:50.030 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:45:57 smithi188 ceph-mon[32694]: pgmap v874: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:50.036 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:00 smithi188 ceph-mon[32694]: pgmap v875: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:50.044 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:02 smithi188 ceph-mon[32694]: pgmap v876: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:50.050 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:04 smithi188 ceph-mon[32694]: pgmap v877: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:46:50.057 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:06 smithi188 ceph-mon[32694]: pgmap v878: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:46:50.065 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:08 smithi188 ceph-mon[32694]: pgmap v879: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:46:50.072 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:10 smithi188 ceph-mon[32694]: pgmap v880: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:46:50.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:10 smithi188 ceph-mon[32694]: osdmap e139: 12 total, 12 up, 12 in 2024-04-23T18:46:50.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:12 smithi188 ceph-mon[32694]: pgmap v882: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-23T18:46:50.099 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:14 smithi188 ceph-mon[32694]: pgmap v883: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:50.105 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:14 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:50.112 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:16 smithi188 ceph-mon[32694]: pgmap v884: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:50.121 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:17 smithi188 ceph-mon[32694]: osdmap e140: 12 total, 12 up, 12 in 2024-04-23T18:46:50.129 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:18 smithi188 ceph-mon[32694]: pgmap v886: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:46:50.136 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:19 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:50.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:19 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:50.232 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:19 smithi188 ceph-mon[32694]: pgmap v887: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-04-23T18:46:50.240 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:19 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:50.248 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:19 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:50.256 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:19 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:50.264 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:19 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:50.271 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:19 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:50.279 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:22 smithi188 ceph-mon[32694]: pgmap v888: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:50.290 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:24 smithi188 ceph-mon[32694]: pgmap v889: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:50.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:26 smithi188 ceph-mon[32694]: pgmap v890: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:50.306 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:28 smithi188 ceph-mon[32694]: pgmap v891: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:46:50.313 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:30 smithi188 ceph-mon[32694]: pgmap v892: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:50.320 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:32 smithi188 ceph-mon[32694]: pgmap v893: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:50.328 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:34 smithi188 ceph-mon[32694]: pgmap v894: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:50.335 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:36 smithi188 ceph-mon[32694]: pgmap v895: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:50.343 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:38 smithi188 ceph-mon[32694]: pgmap v896: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:50.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:40 smithi188 ceph-mon[32694]: pgmap v897: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:50.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:42 smithi188 ceph-mon[32694]: pgmap v898: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:50.453 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:44 smithi188 ceph-mon[32694]: pgmap v899: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:50.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:46 smithi188 ceph-mon[32694]: pgmap v900: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:50.467 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:25 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2793906102' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T18:46:50.476 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:26 smithi177 ceph-mon[32308]: pgmap v698: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 5.4 MiB/s wr, 174 op/s 2024-04-23T18:46:50.485 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:28 smithi177 ceph-mon[32308]: pgmap v699: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 6.4 MiB/s wr, 214 op/s 2024-04-23T18:46:50.495 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:30 smithi177 ceph-mon[32308]: pgmap v700: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 4.3 MiB/s wr, 148 op/s 2024-04-23T18:46:50.505 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:32 smithi177 ceph-mon[32308]: pgmap v701: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 3.6 MiB/s wr, 131 op/s 2024-04-23T18:46:50.514 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:34 smithi177 ceph-mon[32308]: pgmap v702: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 5.3 MiB/s wr, 192 op/s 2024-04-23T18:46:50.523 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:36 smithi177 ceph-mon[32308]: pgmap v703: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 3.2 MiB/s wr, 121 op/s 2024-04-23T18:46:50.532 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:38 smithi177 ceph-mon[32308]: pgmap v704: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 4.5 MiB/s wr, 153 op/s 2024-04-23T18:46:50.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:40 smithi177 ceph-mon[32308]: pgmap v705: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 3.8 MiB/s wr, 117 op/s 2024-04-23T18:46:50.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:42 smithi177 ceph-mon[32308]: pgmap v706: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 3.4 MiB/s wr, 101 op/s 2024-04-23T18:46:50.559 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:44 smithi177 ceph-mon[32308]: pgmap v707: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 3.5 MiB/s wr, 109 op/s 2024-04-23T18:46:50.568 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:46 smithi177 ceph-mon[32308]: pgmap v708: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 1.8 MiB/s wr, 48 op/s 2024-04-23T18:46:50.577 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:48 smithi177 ceph-mon[32308]: pgmap v709: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.8 MiB/s wr, 53 op/s 2024-04-23T18:46:50.586 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:50 smithi177 ceph-mon[32308]: pgmap v710: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 523 KiB/s wr, 22 op/s 2024-04-23T18:46:50.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:52 smithi177 ceph-mon[32308]: pgmap v711: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 61 KiB/s wr, 12 op/s 2024-04-23T18:46:50.669 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:54 smithi177 ceph-mon[32308]: pgmap v712: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 82 KiB/s wr, 17 op/s 2024-04-23T18:46:50.690 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:56 smithi177 ceph-mon[32308]: pgmap v713: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 10 op/s 2024-04-23T18:46:50.698 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:40:58 smithi177 ceph-mon[32308]: pgmap v714: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 14 op/s 2024-04-23T18:46:50.708 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:00 smithi177 ceph-mon[32308]: pgmap v715: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 9 op/s 2024-04-23T18:46:50.716 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:02 smithi177 ceph-mon[32308]: pgmap v716: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 9 op/s 2024-04-23T18:46:50.729 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:04 smithi177 ceph-mon[32308]: pgmap v717: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 96 KiB/s wr, 30 op/s 2024-04-23T18:46:50.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:06 smithi177 ceph-mon[32308]: pgmap v718: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 75 KiB/s wr, 25 op/s 2024-04-23T18:46:50.749 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:06 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:50.758 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:06 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:50.766 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:06 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:50.775 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:06 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:50.784 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:06 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:50.793 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:07 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:50.802 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:07 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:50.812 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:07 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:50.821 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:08 smithi177 ceph-mon[32308]: pgmap v719: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 121 KiB/s wr, 29 op/s 2024-04-23T18:46:50.829 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:10 smithi177 ceph-mon[32308]: pgmap v720: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 106 KiB/s wr, 24 op/s 2024-04-23T18:46:50.886 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:10 smithi177 ceph-mon[32308]: osdmap e129: 12 total, 12 up, 12 in 2024-04-23T18:46:50.895 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:12 smithi177 ceph-mon[32308]: pgmap v722: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 127 KiB/s wr, 29 op/s 2024-04-23T18:46:50.904 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:14 smithi177 ceph-mon[32308]: pgmap v723: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 179 KiB/s wr, 13 op/s 2024-04-23T18:46:50.914 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:16 smithi177 ceph-mon[32308]: pgmap v724: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 179 KiB/s wr, 13 op/s 2024-04-23T18:46:50.922 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:16 smithi177 ceph-mon[32308]: osdmap e130: 12 total, 12 up, 12 in 2024-04-23T18:46:50.930 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:18 smithi177 ceph-mon[32308]: pgmap v726: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 175 KiB/s wr, 17 op/s 2024-04-23T18:46:50.939 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:20 smithi177 ceph-mon[32308]: pgmap v727: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 147 KiB/s wr, 14 op/s 2024-04-23T18:46:50.947 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:22 smithi177 ceph-mon[32308]: pgmap v728: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 140 KiB/s wr, 13 op/s 2024-04-23T18:46:50.955 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:24 smithi177 ceph-mon[32308]: pgmap v729: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 10 op/s 2024-04-23T18:46:50.963 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:26 smithi177 ceph-mon[32308]: pgmap v730: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 10 op/s 2024-04-23T18:46:50.972 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:28 smithi177 ceph-mon[32308]: pgmap v731: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 12 op/s 2024-04-23T18:46:50.980 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:30 smithi177 ceph-mon[32308]: pgmap v732: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-04-23T18:46:50.988 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:32 smithi177 ceph-mon[32308]: pgmap v733: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-04-23T18:46:50.996 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:34 smithi177 ceph-mon[32308]: pgmap v734: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-23T18:46:51.005 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:36 smithi177 ceph-mon[32308]: pgmap v735: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 KiB/s wr, 6 op/s 2024-04-23T18:46:51.014 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:38 smithi177 ceph-mon[32308]: pgmap v736: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2024-04-23T18:46:51.081 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:40 smithi177 ceph-mon[32308]: pgmap v737: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2024-04-23T18:46:51.132 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:42 smithi177 ceph-mon[32308]: pgmap v738: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2024-04-23T18:46:51.141 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:44 smithi177 ceph-mon[32308]: pgmap v739: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-23T18:46:51.150 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:46 smithi177 ceph-mon[32308]: pgmap v740: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.2 KiB/s wr, 5 op/s 2024-04-23T18:46:51.160 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:48 smithi177 ceph-mon[32308]: pgmap v741: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-23T18:46:51.169 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:50 smithi177 ceph-mon[32308]: pgmap v742: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 5 op/s 2024-04-23T18:46:51.176 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:52 smithi177 ceph-mon[32308]: pgmap v743: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 5 op/s 2024-04-23T18:46:51.185 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:54 smithi177 ceph-mon[32308]: pgmap v744: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 KiB/s wr, 8 op/s 2024-04-23T18:46:51.194 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:56 smithi177 ceph-mon[32308]: pgmap v745: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-04-23T18:46:51.202 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:41:58 smithi177 ceph-mon[32308]: pgmap v746: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 4.5 KiB/s wr, 7 op/s 2024-04-23T18:46:51.211 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:00 smithi177 ceph-mon[32308]: pgmap v747: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-04-23T18:46:51.219 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:02 smithi177 ceph-mon[32308]: pgmap v748: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-04-23T18:46:51.228 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:04 smithi177 ceph-mon[32308]: pgmap v749: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 53 KiB/s wr, 25 op/s 2024-04-23T18:46:51.237 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:05 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:46:51.245 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:05 smithi177 ceph-mon[32308]: pgmap v750: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 48 KiB/s wr, 22 op/s 2024-04-23T18:46:51.253 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:06 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:51.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:07 smithi177 ceph-mon[32308]: pgmap v751: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 48 KiB/s wr, 25 op/s 2024-04-23T18:46:51.292 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:07 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:51.364 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:07 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:51.370 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:07 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:51.376 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:09 smithi177 ceph-mon[32308]: pgmap v752: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 48 KiB/s wr, 22 op/s 2024-04-23T18:46:51.382 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:09 smithi177 ceph-mon[32308]: osdmap e131: 12 total, 12 up, 12 in 2024-04-23T18:46:51.389 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:11 smithi177 ceph-mon[32308]: pgmap v754: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 58 KiB/s wr, 27 op/s 2024-04-23T18:46:51.395 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:13 smithi177 ceph-mon[32308]: pgmap v755: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-23T18:46:51.402 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:14 smithi177 ceph-mon[32308]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:46:51.408 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:15 smithi177 ceph-mon[32308]: pgmap v756: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-23T18:46:51.416 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:17 smithi177 ceph-mon[32308]: osdmap e132: 12 total, 12 up, 12 in 2024-04-23T18:46:51.425 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:18 smithi177 ceph-mon[32308]: pgmap v758: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-04-23T18:46:51.432 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:20 smithi177 ceph-mon[32308]: pgmap v759: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-23T18:46:51.439 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:22 smithi177 ceph-mon[32308]: pgmap v760: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-23T18:46:51.445 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:24 smithi177 ceph-mon[32308]: pgmap v761: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-23T18:46:51.452 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:26 smithi177 ceph-mon[32308]: pgmap v762: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-23T18:46:51.458 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:28 smithi177 ceph-mon[32308]: pgmap v763: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-04-23T18:46:51.465 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:30 smithi177 ceph-mon[32308]: pgmap v764: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-04-23T18:46:51.495 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:32 smithi177 ceph-mon[32308]: pgmap v765: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-04-23T18:46:51.561 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:34 smithi177 ceph-mon[32308]: pgmap v766: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-04-23T18:46:51.569 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:36 smithi177 ceph-mon[32308]: pgmap v767: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-23T18:46:51.578 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:38 smithi177 ceph-mon[32308]: pgmap v768: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 7 op/s 2024-04-23T18:46:51.585 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:40 smithi177 ceph-mon[32308]: pgmap v769: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-23T18:46:51.592 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:42 smithi177 ceph-mon[32308]: pgmap v770: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-23T18:46:51.598 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:44 smithi177 ceph-mon[32308]: pgmap v771: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 7 op/s 2024-04-23T18:46:51.605 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:46 smithi177 ceph-mon[32308]: pgmap v772: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:48 smithi177 ceph-mon[32308]: pgmap v773: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:51.618 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:50 smithi177 ceph-mon[32308]: pgmap v774: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:51.626 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:52 smithi177 ceph-mon[32308]: pgmap v775: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:51.633 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:54 smithi177 ceph-mon[32308]: pgmap v776: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:51.665 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:56 smithi177 ceph-mon[32308]: pgmap v777: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:51.674 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:42:58 smithi177 ceph-mon[32308]: pgmap v778: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:51.680 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:00 smithi177 ceph-mon[32308]: pgmap v779: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:51.687 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:02 smithi177 ceph-mon[32308]: pgmap v780: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:51.695 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:04 smithi177 ceph-mon[32308]: pgmap v781: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:51.704 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:06 smithi177 ceph-mon[32308]: pgmap v782: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:51.773 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:08 smithi177 ceph-mon[32308]: pgmap v783: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:51.783 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:08 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:51.791 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:08 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:51.800 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:08 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:51.808 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:08 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:51.815 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:10 smithi177 ceph-mon[32308]: pgmap v784: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:51.822 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:10 smithi177 ceph-mon[32308]: osdmap e133: 12 total, 12 up, 12 in 2024-04-23T18:46:51.837 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:12 smithi177 ceph-mon[32308]: pgmap v786: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-23T18:46:51.845 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:14 smithi177 ceph-mon[32308]: pgmap v787: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:51.852 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:16 smithi177 ceph-mon[32308]: pgmap v788: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:51.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:16 smithi177 ceph-mon[32308]: osdmap e134: 12 total, 12 up, 12 in 2024-04-23T18:46:51.871 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:18 smithi177 ceph-mon[32308]: pgmap v790: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:46:51.879 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:20 smithi177 ceph-mon[32308]: pgmap v791: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2024-04-23T18:46:51.888 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:22 smithi177 ceph-mon[32308]: pgmap v792: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:51.898 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:24 smithi177 ceph-mon[32308]: pgmap v793: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:51.907 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:26 smithi177 ceph-mon[32308]: pgmap v794: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:51.944 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:28 smithi177 ceph-mon[32308]: pgmap v795: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:46:52.025 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:30 smithi177 ceph-mon[32308]: pgmap v796: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.034 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:32 smithi177 ceph-mon[32308]: pgmap v797: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.042 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:34 smithi177 ceph-mon[32308]: pgmap v798: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.051 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:36 smithi177 ceph-mon[32308]: pgmap v799: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.059 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:38 smithi177 ceph-mon[32308]: pgmap v800: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.069 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:40 smithi177 ceph-mon[32308]: pgmap v801: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.077 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:42 smithi177 ceph-mon[32308]: pgmap v802: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.086 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:44 smithi177 ceph-mon[32308]: pgmap v803: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.095 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:46 smithi177 ceph-mon[32308]: pgmap v804: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.103 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:48 smithi177 ceph-mon[32308]: pgmap v805: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:50 smithi177 ceph-mon[32308]: pgmap v806: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.120 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:52 smithi177 ceph-mon[32308]: pgmap v807: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.128 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:54 smithi177 ceph-mon[32308]: pgmap v808: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.139 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:56 smithi177 ceph-mon[32308]: pgmap v809: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.148 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:43:58 smithi177 ceph-mon[32308]: pgmap v810: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.156 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:00 smithi177 ceph-mon[32308]: pgmap v811: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.178 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:02 smithi177 ceph-mon[32308]: pgmap v812: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:04 smithi177 ceph-mon[32308]: pgmap v813: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:52.265 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:06 smithi177 ceph-mon[32308]: pgmap v814: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:52.276 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:08 smithi177 ceph-mon[32308]: pgmap v815: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:52.286 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:08 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:52.295 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:09 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:52.304 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:09 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:52.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:09 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:52.323 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:10 smithi177 ceph-mon[32308]: pgmap v816: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:52.331 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:10 smithi177 ceph-mon[32308]: osdmap e135: 12 total, 12 up, 12 in 2024-04-23T18:46:52.340 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:12 smithi177 ceph-mon[32308]: pgmap v818: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-23T18:46:52.350 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:14 smithi177 ceph-mon[32308]: pgmap v819: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:15 smithi177 ceph-mon[32308]: pgmap v820: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:52.370 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:17 smithi177 ceph-mon[32308]: osdmap e136: 12 total, 12 up, 12 in 2024-04-23T18:46:52.380 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:18 smithi177 ceph-mon[32308]: pgmap v822: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:46:52.389 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:20 smithi177 ceph-mon[32308]: pgmap v823: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2024-04-23T18:46:52.398 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:22 smithi177 ceph-mon[32308]: pgmap v824: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:52.432 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:24 smithi177 ceph-mon[32308]: pgmap v825: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:52.502 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:26 smithi177 ceph-mon[32308]: pgmap v826: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:52.511 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:28 smithi177 ceph-mon[32308]: pgmap v827: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:46:52.520 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:30 smithi177 ceph-mon[32308]: pgmap v828: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.529 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:32 smithi177 ceph-mon[32308]: pgmap v829: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:34 smithi177 ceph-mon[32308]: pgmap v830: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:36 smithi177 ceph-mon[32308]: pgmap v831: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.555 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:38 smithi177 ceph-mon[32308]: pgmap v832: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.563 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:40 smithi177 ceph-mon[32308]: pgmap v833: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.571 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:42 smithi177 ceph-mon[32308]: pgmap v834: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.580 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:44 smithi177 ceph-mon[32308]: pgmap v835: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.589 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:46 smithi177 ceph-mon[32308]: pgmap v836: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.596 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:48 smithi177 ceph-mon[32308]: pgmap v837: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.605 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:50 smithi177 ceph-mon[32308]: pgmap v838: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:52 smithi177 ceph-mon[32308]: pgmap v839: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.618 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:54 smithi177 ceph-mon[32308]: pgmap v840: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.625 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:56 smithi177 ceph-mon[32308]: pgmap v841: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.663 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:44:58 smithi177 ceph-mon[32308]: pgmap v842: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.701 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:00 smithi177 ceph-mon[32308]: pgmap v843: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.709 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:02 smithi177 ceph-mon[32308]: pgmap v844: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.717 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:04 smithi177 ceph-mon[32308]: pgmap v845: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:52.724 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:06 smithi177 ceph-mon[32308]: pgmap v846: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:52.730 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:08 smithi177 ceph-mon[32308]: pgmap v847: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:52.737 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:09 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:52.744 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:10 smithi177 ceph-mon[32308]: pgmap v848: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:52.751 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:10 smithi177 ceph-mon[32308]: osdmap e137: 12 total, 12 up, 12 in 2024-04-23T18:46:52.759 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:12 smithi177 ceph-mon[32308]: pgmap v850: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-23T18:46:52.766 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:14 smithi177 ceph-mon[32308]: pgmap v851: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:52.773 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:14 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:52.779 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:14 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:52.785 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:14 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:52.791 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:14 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:52.799 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:14 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:52.806 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:16 smithi177 ceph-mon[32308]: pgmap v852: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:52.843 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:16 smithi177 ceph-mon[32308]: osdmap e138: 12 total, 12 up, 12 in 2024-04-23T18:46:52.891 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:18 smithi177 ceph-mon[32308]: pgmap v854: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:46:52.898 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:20 smithi177 ceph-mon[32308]: pgmap v855: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2024-04-23T18:46:52.904 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:22 smithi177 ceph-mon[32308]: pgmap v856: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:52.913 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:24 smithi177 ceph-mon[32308]: pgmap v857: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:52.922 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:26 smithi177 ceph-mon[32308]: pgmap v858: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:52.931 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:28 smithi177 ceph-mon[32308]: pgmap v859: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:46:52.940 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:30 smithi177 ceph-mon[32308]: pgmap v860: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.950 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:32 smithi177 ceph-mon[32308]: pgmap v861: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.959 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:34 smithi177 ceph-mon[32308]: pgmap v862: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.966 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:36 smithi177 ceph-mon[32308]: pgmap v863: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.974 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:38 smithi177 ceph-mon[32308]: pgmap v864: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:52.983 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:40 smithi177 ceph-mon[32308]: pgmap v865: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.990 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:42 smithi177 ceph-mon[32308]: pgmap v866: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:52.999 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:44 smithi177 ceph-mon[32308]: pgmap v867: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:53.007 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:46 smithi177 ceph-mon[32308]: pgmap v868: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:53.016 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:48 smithi177 ceph-mon[32308]: pgmap v869: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:53.041 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:50 smithi177 ceph-mon[32308]: pgmap v870: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:53.116 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:52 smithi177 ceph-mon[32308]: pgmap v871: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:53.126 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:54 smithi177 ceph-mon[32308]: pgmap v872: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:53.135 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:56 smithi177 ceph-mon[32308]: pgmap v873: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:53.144 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:45:57 smithi177 ceph-mon[32308]: pgmap v874: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:53.152 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:00 smithi177 ceph-mon[32308]: pgmap v875: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:53.161 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:02 smithi177 ceph-mon[32308]: pgmap v876: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:53.170 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:04 smithi177 ceph-mon[32308]: pgmap v877: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:46:53.179 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:06 smithi177 ceph-mon[32308]: pgmap v878: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:46:53.189 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:08 smithi177 ceph-mon[32308]: pgmap v879: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:46:53.197 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:10 smithi177 ceph-mon[32308]: pgmap v880: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:46:53.206 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:10 smithi177 ceph-mon[32308]: osdmap e139: 12 total, 12 up, 12 in 2024-04-23T18:46:53.215 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:12 smithi177 ceph-mon[32308]: pgmap v882: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-23T18:46:53.224 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:14 smithi177 ceph-mon[32308]: pgmap v883: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:53.234 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:14 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:53.242 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:16 smithi177 ceph-mon[32308]: pgmap v884: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:53.251 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:17 smithi177 ceph-mon[32308]: osdmap e140: 12 total, 12 up, 12 in 2024-04-23T18:46:53.292 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:18 smithi177 ceph-mon[32308]: pgmap v886: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:46:53.337 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:19 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:53.345 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:19 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:53.354 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:19 smithi177 ceph-mon[32308]: pgmap v887: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-04-23T18:46:53.364 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:19 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:53.374 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:19 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:53.383 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:19 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:53.392 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:19 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:53.401 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:19 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:53.410 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:22 smithi177 ceph-mon[32308]: pgmap v888: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:53.419 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:24 smithi177 ceph-mon[32308]: pgmap v889: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:53.428 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:26 smithi177 ceph-mon[32308]: pgmap v890: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:53.437 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:28 smithi177 ceph-mon[32308]: pgmap v891: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:46:53.454 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:30 smithi177 ceph-mon[32308]: pgmap v892: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:53.463 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:32 smithi177 ceph-mon[32308]: pgmap v893: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:53.472 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:34 smithi177 ceph-mon[32308]: pgmap v894: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:53.481 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:36 smithi177 ceph-mon[32308]: pgmap v895: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:53.515 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:38 smithi177 ceph-mon[32308]: pgmap v896: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:53.573 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:40 smithi177 ceph-mon[32308]: pgmap v897: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:53.582 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:42 smithi177 ceph-mon[32308]: pgmap v898: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:53.590 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:44 smithi177 ceph-mon[32308]: pgmap v899: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:53.600 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:46 smithi177 ceph-mon[32308]: pgmap v900: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:53.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:25 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2793906102' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T18:46:53.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:26 smithi050 ceph-mon[29187]: pgmap v698: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 5.4 MiB/s wr, 174 op/s 2024-04-23T18:46:53.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:28 smithi050 ceph-mon[29187]: pgmap v699: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 6.4 MiB/s wr, 214 op/s 2024-04-23T18:46:53.647 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:30 smithi050 ceph-mon[29187]: pgmap v700: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 4.3 MiB/s wr, 148 op/s 2024-04-23T18:46:53.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:32 smithi050 ceph-mon[29187]: pgmap v701: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 3.6 MiB/s wr, 131 op/s 2024-04-23T18:46:53.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:34 smithi050 ceph-mon[29187]: pgmap v702: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 5.3 MiB/s wr, 192 op/s 2024-04-23T18:46:53.675 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:36 smithi050 ceph-mon[29187]: pgmap v703: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 3.2 MiB/s wr, 121 op/s 2024-04-23T18:46:53.684 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:38 smithi050 ceph-mon[29187]: pgmap v704: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 4.5 MiB/s wr, 153 op/s 2024-04-23T18:46:53.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:40 smithi050 ceph-mon[29187]: pgmap v705: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 3.8 MiB/s wr, 117 op/s 2024-04-23T18:46:53.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:42 smithi050 ceph-mon[29187]: pgmap v706: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 3.4 MiB/s wr, 101 op/s 2024-04-23T18:46:53.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:44 smithi050 ceph-mon[29187]: pgmap v707: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 3.5 MiB/s wr, 109 op/s 2024-04-23T18:46:53.721 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:46 smithi050 ceph-mon[29187]: pgmap v708: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 1.8 MiB/s wr, 48 op/s 2024-04-23T18:46:53.753 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:48 smithi050 ceph-mon[29187]: pgmap v709: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.8 MiB/s wr, 53 op/s 2024-04-23T18:46:53.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:50 smithi050 ceph-mon[29187]: pgmap v710: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 523 KiB/s wr, 22 op/s 2024-04-23T18:46:53.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:52 smithi050 ceph-mon[29187]: pgmap v711: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 61 KiB/s wr, 12 op/s 2024-04-23T18:46:53.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:54 smithi050 ceph-mon[29187]: pgmap v712: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 82 KiB/s wr, 17 op/s 2024-04-23T18:46:53.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:56 smithi050 ceph-mon[29187]: pgmap v713: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 10 op/s 2024-04-23T18:46:53.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:40:58 smithi050 ceph-mon[29187]: pgmap v714: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 14 op/s 2024-04-23T18:46:53.939 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:00 smithi050 ceph-mon[29187]: pgmap v715: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 9 op/s 2024-04-23T18:46:53.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:02 smithi050 ceph-mon[29187]: pgmap v716: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 9 op/s 2024-04-23T18:46:53.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:04 smithi050 ceph-mon[29187]: pgmap v717: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 96 KiB/s wr, 30 op/s 2024-04-23T18:46:53.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:06 smithi050 ceph-mon[29187]: pgmap v718: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 75 KiB/s wr, 25 op/s 2024-04-23T18:46:53.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:06 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:53.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:06 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:53.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:06 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:53.999 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:06 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:54.008 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:06 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:54.015 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:07 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:54.022 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:07 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:54.088 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:07 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:54.178 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:08 smithi050 ceph-mon[29187]: pgmap v719: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 121 KiB/s wr, 29 op/s 2024-04-23T18:46:54.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:10 smithi050 ceph-mon[29187]: pgmap v720: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 106 KiB/s wr, 24 op/s 2024-04-23T18:46:54.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:10 smithi050 ceph-mon[29187]: osdmap e129: 12 total, 12 up, 12 in 2024-04-23T18:46:54.204 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:12 smithi050 ceph-mon[29187]: pgmap v722: 97 pgs: 97 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 127 KiB/s wr, 29 op/s 2024-04-23T18:46:54.212 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:14 smithi050 ceph-mon[29187]: pgmap v723: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 179 KiB/s wr, 13 op/s 2024-04-23T18:46:54.222 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:16 smithi050 ceph-mon[29187]: pgmap v724: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 179 KiB/s wr, 13 op/s 2024-04-23T18:46:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:16 smithi050 ceph-mon[29187]: osdmap e130: 12 total, 12 up, 12 in 2024-04-23T18:46:54.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:18 smithi050 ceph-mon[29187]: pgmap v726: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 175 KiB/s wr, 17 op/s 2024-04-23T18:46:54.246 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:20 smithi050 ceph-mon[29187]: pgmap v727: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 147 KiB/s wr, 14 op/s 2024-04-23T18:46:54.254 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:22 smithi050 ceph-mon[29187]: pgmap v728: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 140 KiB/s wr, 13 op/s 2024-04-23T18:46:54.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:24 smithi050 ceph-mon[29187]: pgmap v729: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 10 op/s 2024-04-23T18:46:54.272 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:26 smithi050 ceph-mon[29187]: pgmap v730: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 10 op/s 2024-04-23T18:46:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:28 smithi050 ceph-mon[29187]: pgmap v731: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 12 op/s 2024-04-23T18:46:54.292 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:30 smithi050 ceph-mon[29187]: pgmap v732: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-04-23T18:46:54.302 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:32 smithi050 ceph-mon[29187]: pgmap v733: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-04-23T18:46:54.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:34 smithi050 ceph-mon[29187]: pgmap v734: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-23T18:46:54.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:36 smithi050 ceph-mon[29187]: pgmap v735: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 KiB/s wr, 6 op/s 2024-04-23T18:46:54.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:38 smithi050 ceph-mon[29187]: pgmap v736: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2024-04-23T18:46:54.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:40 smithi050 ceph-mon[29187]: pgmap v737: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2024-04-23T18:46:54.421 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:42 smithi050 ceph-mon[29187]: pgmap v738: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2024-04-23T18:46:54.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:44 smithi050 ceph-mon[29187]: pgmap v739: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-23T18:46:54.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:46 smithi050 ceph-mon[29187]: pgmap v740: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.2 KiB/s wr, 5 op/s 2024-04-23T18:46:54.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:48 smithi050 ceph-mon[29187]: pgmap v741: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-23T18:46:54.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:50 smithi050 ceph-mon[29187]: pgmap v742: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 5 op/s 2024-04-23T18:46:54.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:52 smithi050 ceph-mon[29187]: pgmap v743: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.1 KiB/s wr, 5 op/s 2024-04-23T18:46:54.471 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:54 smithi050 ceph-mon[29187]: pgmap v744: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 KiB/s wr, 8 op/s 2024-04-23T18:46:54.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:56 smithi050 ceph-mon[29187]: pgmap v745: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-04-23T18:46:54.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:41:58 smithi050 ceph-mon[29187]: pgmap v746: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 4.5 KiB/s wr, 7 op/s 2024-04-23T18:46:54.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:00 smithi050 ceph-mon[29187]: pgmap v747: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-04-23T18:46:54.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:02 smithi050 ceph-mon[29187]: pgmap v748: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-04-23T18:46:54.513 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:04 smithi050 ceph-mon[29187]: pgmap v749: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 53 KiB/s wr, 25 op/s 2024-04-23T18:46:54.522 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:05 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:46:54.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:05 smithi050 ceph-mon[29187]: pgmap v750: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 48 KiB/s wr, 22 op/s 2024-04-23T18:46:54.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:06 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:54.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:07 smithi050 ceph-mon[29187]: pgmap v751: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 48 KiB/s wr, 25 op/s 2024-04-23T18:46:54.621 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:07 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:54.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:07 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:54.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:07 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:54.646 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:09 smithi050 ceph-mon[29187]: pgmap v752: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 48 KiB/s wr, 22 op/s 2024-04-23T18:46:54.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:09 smithi050 ceph-mon[29187]: osdmap e131: 12 total, 12 up, 12 in 2024-04-23T18:46:54.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:11 smithi050 ceph-mon[29187]: pgmap v754: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 58 KiB/s wr, 27 op/s 2024-04-23T18:46:54.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:13 smithi050 ceph-mon[29187]: pgmap v755: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-23T18:46:54.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:14 smithi050 ceph-mon[29187]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T18:46:54.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:15 smithi050 ceph-mon[29187]: pgmap v756: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-23T18:46:54.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:17 smithi050 ceph-mon[29187]: osdmap e132: 12 total, 12 up, 12 in 2024-04-23T18:46:54.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:18 smithi050 ceph-mon[29187]: pgmap v758: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-04-23T18:46:54.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:20 smithi050 ceph-mon[29187]: pgmap v759: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-23T18:46:54.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:22 smithi050 ceph-mon[29187]: pgmap v760: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-23T18:46:54.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:24 smithi050 ceph-mon[29187]: pgmap v761: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-23T18:46:54.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:26 smithi050 ceph-mon[29187]: pgmap v762: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-23T18:46:54.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:28 smithi050 ceph-mon[29187]: pgmap v763: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-04-23T18:46:54.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:30 smithi050 ceph-mon[29187]: pgmap v764: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-04-23T18:46:54.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:32 smithi050 ceph-mon[29187]: pgmap v765: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-04-23T18:46:54.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:34 smithi050 ceph-mon[29187]: pgmap v766: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-04-23T18:46:54.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:36 smithi050 ceph-mon[29187]: pgmap v767: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-23T18:46:54.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:38 smithi050 ceph-mon[29187]: pgmap v768: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 7 op/s 2024-04-23T18:46:54.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:40 smithi050 ceph-mon[29187]: pgmap v769: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-23T18:46:54.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:42 smithi050 ceph-mon[29187]: pgmap v770: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-23T18:46:54.940 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:44 smithi050 ceph-mon[29187]: pgmap v771: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 7 op/s 2024-04-23T18:46:54.949 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:46 smithi050 ceph-mon[29187]: pgmap v772: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:54.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:48 smithi050 ceph-mon[29187]: pgmap v773: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:54.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:50 smithi050 ceph-mon[29187]: pgmap v774: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:54.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:52 smithi050 ceph-mon[29187]: pgmap v775: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:54.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:54 smithi050 ceph-mon[29187]: pgmap v776: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:54.996 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:56 smithi050 ceph-mon[29187]: pgmap v777: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.005 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:42:58 smithi050 ceph-mon[29187]: pgmap v778: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:55.015 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:00 smithi050 ceph-mon[29187]: pgmap v779: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.043 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:02 smithi050 ceph-mon[29187]: pgmap v780: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.101 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:04 smithi050 ceph-mon[29187]: pgmap v781: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:55.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:06 smithi050 ceph-mon[29187]: pgmap v782: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:55.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:08 smithi050 ceph-mon[29187]: pgmap v783: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:55.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:08 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:55.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:08 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:55.147 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:08 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:55.156 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:08 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:55.164 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:10 smithi050 ceph-mon[29187]: pgmap v784: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:55.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:10 smithi050 ceph-mon[29187]: osdmap e133: 12 total, 12 up, 12 in 2024-04-23T18:46:55.183 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:12 smithi050 ceph-mon[29187]: pgmap v786: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-23T18:46:55.192 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:14 smithi050 ceph-mon[29187]: pgmap v787: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:55.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:16 smithi050 ceph-mon[29187]: pgmap v788: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:55.211 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:16 smithi050 ceph-mon[29187]: osdmap e134: 12 total, 12 up, 12 in 2024-04-23T18:46:55.220 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:18 smithi050 ceph-mon[29187]: pgmap v790: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:46:55.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:20 smithi050 ceph-mon[29187]: pgmap v791: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2024-04-23T18:46:55.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:22 smithi050 ceph-mon[29187]: pgmap v792: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:55.266 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:24 smithi050 ceph-mon[29187]: pgmap v793: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:55.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:26 smithi050 ceph-mon[29187]: pgmap v794: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:55.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:28 smithi050 ceph-mon[29187]: pgmap v795: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:46:55.343 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:30 smithi050 ceph-mon[29187]: pgmap v796: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:32 smithi050 ceph-mon[29187]: pgmap v797: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.361 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:34 smithi050 ceph-mon[29187]: pgmap v798: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:55.370 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:36 smithi050 ceph-mon[29187]: pgmap v799: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:38 smithi050 ceph-mon[29187]: pgmap v800: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:55.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:40 smithi050 ceph-mon[29187]: pgmap v801: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.396 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:42 smithi050 ceph-mon[29187]: pgmap v802: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:44 smithi050 ceph-mon[29187]: pgmap v803: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:55.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:46 smithi050 ceph-mon[29187]: pgmap v804: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.425 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:48 smithi050 ceph-mon[29187]: pgmap v805: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:55.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:50 smithi050 ceph-mon[29187]: pgmap v806: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:52 smithi050 ceph-mon[29187]: pgmap v807: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.451 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:54 smithi050 ceph-mon[29187]: pgmap v808: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:55.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:56 smithi050 ceph-mon[29187]: pgmap v809: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:43:58 smithi050 ceph-mon[29187]: pgmap v810: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:55.550 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:00 smithi050 ceph-mon[29187]: pgmap v811: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:02 smithi050 ceph-mon[29187]: pgmap v812: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:04 smithi050 ceph-mon[29187]: pgmap v813: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:55.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:06 smithi050 ceph-mon[29187]: pgmap v814: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:55.589 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:08 smithi050 ceph-mon[29187]: pgmap v815: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:55.598 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:08 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:55.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:09 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:55.615 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:09 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:55.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:09 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:55.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:10 smithi050 ceph-mon[29187]: pgmap v816: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:55.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:10 smithi050 ceph-mon[29187]: osdmap e135: 12 total, 12 up, 12 in 2024-04-23T18:46:55.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:12 smithi050 ceph-mon[29187]: pgmap v818: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-23T18:46:55.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:14 smithi050 ceph-mon[29187]: pgmap v819: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:55.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:15 smithi050 ceph-mon[29187]: pgmap v820: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:55.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:17 smithi050 ceph-mon[29187]: osdmap e136: 12 total, 12 up, 12 in 2024-04-23T18:46:55.687 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:18 smithi050 ceph-mon[29187]: pgmap v822: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:46:55.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:20 smithi050 ceph-mon[29187]: pgmap v823: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 215 B/s wr, 6 op/s 2024-04-23T18:46:55.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:22 smithi050 ceph-mon[29187]: pgmap v824: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:55.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:24 smithi050 ceph-mon[29187]: pgmap v825: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:55.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:26 smithi050 ceph-mon[29187]: pgmap v826: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:55.838 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:28 smithi050 ceph-mon[29187]: pgmap v827: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:46:55.845 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:30 smithi050 ceph-mon[29187]: pgmap v828: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:32 smithi050 ceph-mon[29187]: pgmap v829: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:34 smithi050 ceph-mon[29187]: pgmap v830: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:55.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:36 smithi050 ceph-mon[29187]: pgmap v831: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:38 smithi050 ceph-mon[29187]: pgmap v832: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:55.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:40 smithi050 ceph-mon[29187]: pgmap v833: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:42 smithi050 ceph-mon[29187]: pgmap v834: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:44 smithi050 ceph-mon[29187]: pgmap v835: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:55.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:46 smithi050 ceph-mon[29187]: pgmap v836: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.920 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:48 smithi050 ceph-mon[29187]: pgmap v837: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:55.929 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:50 smithi050 ceph-mon[29187]: pgmap v838: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.936 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:52 smithi050 ceph-mon[29187]: pgmap v839: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:55.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:54 smithi050 ceph-mon[29187]: pgmap v840: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.015 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:56 smithi050 ceph-mon[29187]: pgmap v841: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:44:58 smithi050 ceph-mon[29187]: pgmap v842: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.034 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:00 smithi050 ceph-mon[29187]: pgmap v843: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.043 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:02 smithi050 ceph-mon[29187]: pgmap v844: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.051 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:04 smithi050 ceph-mon[29187]: pgmap v845: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:56.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:06 smithi050 ceph-mon[29187]: pgmap v846: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:56.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:08 smithi050 ceph-mon[29187]: pgmap v847: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-23T18:46:56.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:09 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:56.087 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:10 smithi050 ceph-mon[29187]: pgmap v848: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-23T18:46:56.095 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:10 smithi050 ceph-mon[29187]: osdmap e137: 12 total, 12 up, 12 in 2024-04-23T18:46:56.102 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:12 smithi050 ceph-mon[29187]: pgmap v850: 97 pgs: 97 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-23T18:46:56.109 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:14 smithi050 ceph-mon[29187]: pgmap v851: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:56.116 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:14 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:56.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:14 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:56.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:14 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:56.137 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:14 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:56.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:14 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:56.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:16 smithi050 ceph-mon[29187]: pgmap v852: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:56.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:16 smithi050 ceph-mon[29187]: osdmap e138: 12 total, 12 up, 12 in 2024-04-23T18:46:56.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:18 smithi050 ceph-mon[29187]: pgmap v854: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:46:56.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:20 smithi050 ceph-mon[29187]: pgmap v855: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 214 B/s wr, 6 op/s 2024-04-23T18:46:56.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:22 smithi050 ceph-mon[29187]: pgmap v856: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:56.270 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:24 smithi050 ceph-mon[29187]: pgmap v857: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:56.277 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:26 smithi050 ceph-mon[29187]: pgmap v858: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:56.284 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:28 smithi050 ceph-mon[29187]: pgmap v859: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:46:56.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:30 smithi050 ceph-mon[29187]: pgmap v860: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.298 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:32 smithi050 ceph-mon[29187]: pgmap v861: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:34 smithi050 ceph-mon[29187]: pgmap v862: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:36 smithi050 ceph-mon[29187]: pgmap v863: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:38 smithi050 ceph-mon[29187]: pgmap v864: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:40 smithi050 ceph-mon[29187]: pgmap v865: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:42 smithi050 ceph-mon[29187]: pgmap v866: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.337 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:44 smithi050 ceph-mon[29187]: pgmap v867: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.362 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:46 smithi050 ceph-mon[29187]: pgmap v868: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:48 smithi050 ceph-mon[29187]: pgmap v869: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:50 smithi050 ceph-mon[29187]: pgmap v870: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:52 smithi050 ceph-mon[29187]: pgmap v871: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:54 smithi050 ceph-mon[29187]: pgmap v872: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:56 smithi050 ceph-mon[29187]: pgmap v873: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.469 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:45:57 smithi050 ceph-mon[29187]: pgmap v874: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.476 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:00 smithi050 ceph-mon[29187]: pgmap v875: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:02 smithi050 ceph-mon[29187]: pgmap v876: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:04 smithi050 ceph-mon[29187]: pgmap v877: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:46:56.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:06 smithi050 ceph-mon[29187]: pgmap v878: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:46:56.504 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:08 smithi050 ceph-mon[29187]: pgmap v879: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:46:56.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:10 smithi050 ceph-mon[29187]: pgmap v880: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:46:56.518 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:10 smithi050 ceph-mon[29187]: osdmap e139: 12 total, 12 up, 12 in 2024-04-23T18:46:56.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:12 smithi050 ceph-mon[29187]: pgmap v882: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-23T18:46:56.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:14 smithi050 ceph-mon[29187]: pgmap v883: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:56.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:14 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:46:56.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:16 smithi050 ceph-mon[29187]: pgmap v884: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:56.646 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:17 smithi050 ceph-mon[29187]: osdmap e140: 12 total, 12 up, 12 in 2024-04-23T18:46:56.656 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:18 smithi050 ceph-mon[29187]: pgmap v886: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:46:56.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:19 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:56.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:19 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:56.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:19 smithi050 ceph-mon[29187]: pgmap v887: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-04-23T18:46:56.691 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:19 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:56.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:19 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:56.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:19 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:46:56.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:19 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:46:56.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:19 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:46:56.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:22 smithi050 ceph-mon[29187]: pgmap v888: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:46:56.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:24 smithi050 ceph-mon[29187]: pgmap v889: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:56.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:26 smithi050 ceph-mon[29187]: pgmap v890: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:46:56.770 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:28 smithi050 ceph-mon[29187]: pgmap v891: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:46:56.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:30 smithi050 ceph-mon[29187]: pgmap v892: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:32 smithi050 ceph-mon[29187]: pgmap v893: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:34 smithi050 ceph-mon[29187]: pgmap v894: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:36 smithi050 ceph-mon[29187]: pgmap v895: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:38 smithi050 ceph-mon[29187]: pgmap v896: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:40 smithi050 ceph-mon[29187]: pgmap v897: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:42 smithi050 ceph-mon[29187]: pgmap v898: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.916 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:44 smithi050 ceph-mon[29187]: pgmap v899: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:46 smithi050 ceph-mon[29187]: pgmap v900: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:46:56.932 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:48 smithi050 ceph-mon[29187]: pgmap v901: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:50 smithi050 ceph-mon[29187]: pgmap v902: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-23T18:46:56.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:52 smithi050 ceph-mon[29187]: pgmap v903: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-23T18:46:56.960 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:48 smithi177 ceph-mon[32308]: pgmap v901: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.970 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:50 smithi177 ceph-mon[32308]: pgmap v902: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-23T18:46:56.980 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:52 smithi177 ceph-mon[32308]: pgmap v903: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-23T18:46:56.988 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:54 smithi177 ceph-mon[32308]: pgmap v904: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:56.997 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:56 smithi177 ceph-mon[32308]: pgmap v905: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-23T18:46:57.005 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:48 smithi188 ceph-mon[32694]: pgmap v901: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:57.014 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:50 smithi188 ceph-mon[32694]: pgmap v902: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-23T18:46:57.047 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:52 smithi188 ceph-mon[32694]: pgmap v903: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-23T18:46:57.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:54 smithi188 ceph-mon[32694]: pgmap v904: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:57.099 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:56 smithi188 ceph-mon[32694]: pgmap v905: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-23T18:46:57.110 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:46:57.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:54 smithi050 ceph-mon[29187]: pgmap v904: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:57.128 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:46:57.136 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:46:57.146 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 0, 2024-04-23T18:46:57.154 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 2, 2024-04-23T18:46:57.164 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 0, 2024-04-23T18:46:57.173 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 0, 2024-04-23T18:46:57.181 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 4, 2024-04-23T18:46:57.191 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 0, 2024-04-23T18:46:57.201 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.001551160, 2024-04-23T18:46:57.208 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T18:46:57.216 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.000350447, 2024-04-23T18:46:57.224 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:46:57.232 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:46:57.286 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:46:57.294 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:46:57.304 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:46:57.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:57.322 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:46:57.332 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:46:57.341 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:46:57.351 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:46:57.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:57.365 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:46:57.372 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:46:57.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:57.387 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:46:57.394 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 7, 2024-04-23T18:46:57.402 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 373, 2024-04-23T18:46:57.409 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 3472, 2024-04-23T18:46:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 300128, 2024-04-23T18:46:57.515 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 6, 2024-04-23T18:46:57.528 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 3, 2024-04-23T18:46:57.544 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.027938016, 2024-04-23T18:46:57.553 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.020683975, 2024-04-23T18:46:57.560 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.002679508, 2024-04-23T18:46:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:46:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:46:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 371, 2024-04-23T18:46:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.021863896, 2024-04-23T18:46:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000058932 2024-04-23T18:46:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:57.612 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:46:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:46:57.627 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:46:57.634 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:46:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-04-23T18:46:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 300128 2024-04-23T18:46:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:46:57.755 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 497, 2024-04-23T18:46:57.765 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 485, 2024-04-23T18:46:57.772 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 1807392, 2024-04-23T18:46:57.778 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 300288, 2024-04-23T18:46:57.785 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 2, 2024-04-23T18:46:57.793 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 1, 2024-04-23T18:46:57.801 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.078763061, 2024-04-23T18:46:57.808 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.029143199, 2024-04-23T18:46:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.045182811, 2024-04-23T18:46:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.010075979, 2024-04-23T18:46:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:46:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 484, 2024-04-23T18:46:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.022677509, 2024-04-23T18:46:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000046854 2024-04-23T18:46:57.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:57.963 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:46:57.971 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:46:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:46:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:46:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 1807392, 2024-04-23T18:46:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 300288 2024-04-23T18:46:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: "cct": { 2024-04-23T18:46:58.043 INFO:teuthology.orchestra.run.smithi050.stdout: "total_workers": 0, 2024-04-23T18:46:58.051 INFO:teuthology.orchestra.run.smithi050.stdout: "unhealthy_workers": 0 2024-04-23T18:46:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "mempool": { 2024-04-23T18:46:58.078 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:46:58.086 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_items": 0, 2024-04-23T18:46:58.121 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:46:58.187 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:46:58.197 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:46:58.205 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:46:58.214 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:46:58.224 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:46:58.233 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:46:58.243 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:46:58.252 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:46:58.261 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:46:58.271 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:46:58.279 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:46:58.314 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:46:58.324 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_items": 0, 2024-04-23T18:46:58.333 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:46:58.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_items": 0, 2024-04-23T18:46:58.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:46:58.359 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:46:58.412 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:46:58.423 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:46:58.433 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:46:58.443 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:46:58.451 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:46:58.459 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_items": 0, 2024-04-23T18:46:58.468 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:46:58.477 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:46:58.487 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:46:58.495 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_items": 0, 2024-04-23T18:46:58.503 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_bytes": 0, 2024-04-23T18:46:58.512 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_items": 0, 2024-04-23T18:46:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:46:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:46:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:46:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:46:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_bytes": 148754, 2024-04-23T18:46:58.646 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_items": 31, 2024-04-23T18:46:58.655 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:46:58.665 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_items": 0, 2024-04-23T18:46:58.674 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bytes": 0, 2024-04-23T18:46:58.683 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_items": 0, 2024-04-23T18:46:58.692 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:46:58.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_items": 0, 2024-04-23T18:46:58.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:46:58.720 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_items": 0, 2024-04-23T18:46:58.729 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_bytes": 0, 2024-04-23T18:46:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_items": 0, 2024-04-23T18:46:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:46:58.758 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:46:58.766 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_bytes": 0, 2024-04-23T18:46:58.779 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_items": 0, 2024-04-23T18:46:58.789 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_bytes": 928, 2024-04-23T18:46:58.828 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_items": 2, 2024-04-23T18:46:58.883 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_bytes": 0, 2024-04-23T18:46:58.891 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_items": 0, 2024-04-23T18:46:58.900 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_bytes": 0, 2024-04-23T18:46:58.910 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_items": 0 2024-04-23T18:46:58.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:46:58.930 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:46:58.938 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-04-23T18:46:58.945 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 104857600, 2024-04-23T18:46:58.952 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-04-23T18:46:58.960 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 500, 2024-04-23T18:46:58.966 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 1737175, 2024-04-23T18:46:58.973 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-04-23T18:46:58.980 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 500, 2024-04-23T18:46:58.986 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-04-23T18:46:58.993 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-04-23T18:46:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 500, 2024-04-23T18:46:59.007 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 1737175, 2024-04-23T18:46:59.065 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-04-23T18:46:59.073 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:46:59.081 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:46:59.090 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:46:59.097 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:59.103 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:46:59.110 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:46:59.122 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:56 smithi050 ceph-mon[29187]: pgmap v905: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-23T18:46:59.130 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-23T18:46:59.138 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-23T18:46:59.145 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-04-23T18:46:59.154 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-04-23T18:46:59.162 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-23T18:46:59.171 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-23T18:46:59.180 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-23T18:46:59.188 DEBUG:tasks.check_counter:Getting stats from d 2024-04-23T18:46:59.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-23T18:46:59.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:46:58 smithi050 ceph-mon[29187]: pgmap v906: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:59.342 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:46:58 smithi177 ceph-mon[32308]: pgmap v906: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:59.351 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:46:58 smithi188 ceph-mon[32694]: pgmap v906: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:46:59.605 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:47:00.327 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:00 smithi188 ceph-mon[32694]: pgmap v907: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:00.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:00 smithi050 ceph-mon[29187]: pgmap v907: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:00.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:00 smithi177 ceph-mon[32308]: pgmap v907: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:01.231 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:47:01.238 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:47:01.246 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 4, 2024-04-23T18:47:01.254 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 4, 2024-04-23T18:47:01.265 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 2992, 2024-04-23T18:47:01.274 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 448, 2024-04-23T18:47:01.285 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 4, 2024-04-23T18:47:01.294 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 0, 2024-04-23T18:47:01.303 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.001622286, 2024-04-23T18:47:01.349 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.000040010, 2024-04-23T18:47:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.000564928, 2024-04-23T18:47:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:47:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:01.486 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 3, 2024-04-23T18:47:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000759371, 2024-04-23T18:47:01.527 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000253123 2024-04-23T18:47:01.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:01.544 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:01.552 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:01.560 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:01.568 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:01.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:01.584 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-23T18:47:01.592 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T18:47:01.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:01.607 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:47:01.616 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 503, 2024-04-23T18:47:01.677 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 491, 2024-04-23T18:47:01.686 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 1809024, 2024-04-23T18:47:01.695 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 304128, 2024-04-23T18:47:01.703 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 2, 2024-04-23T18:47:01.712 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 1, 2024-04-23T18:47:01.747 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.081908794, 2024-04-23T18:47:01.755 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.031464393, 2024-04-23T18:47:01.763 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.045018226, 2024-04-23T18:47:01.771 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.010150932, 2024-04-23T18:47:01.779 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:01.789 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 490, 2024-04-23T18:47:01.798 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.022503498, 2024-04-23T18:47:01.806 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000045925 2024-04-23T18:47:01.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:01.823 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:01.832 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:01.863 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:01.923 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:01.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:01.940 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 1809024, 2024-04-23T18:47:01.948 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 304128 2024-04-23T18:47:01.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:01.990 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:47:01.999 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 3, 2024-04-23T18:47:02.008 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 376, 2024-04-23T18:47:02.015 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 480, 2024-04-23T18:47:02.023 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 300560, 2024-04-23T18:47:02.033 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 6, 2024-04-23T18:47:02.041 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 3, 2024-04-23T18:47:02.050 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.026892968, 2024-04-23T18:47:02.059 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.019910013, 2024-04-23T18:47:02.066 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.002330001, 2024-04-23T18:47:02.074 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:47:02.083 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 373, 2024-04-23T18:47:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.019157304, 2024-04-23T18:47:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000051360 2024-04-23T18:47:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:02.183 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:02.211 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-23T18:47:02.245 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 300560 2024-04-23T18:47:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: "cct": { 2024-04-23T18:47:02.271 INFO:teuthology.orchestra.run.smithi050.stdout: "total_workers": 0, 2024-04-23T18:47:02.280 INFO:teuthology.orchestra.run.smithi050.stdout: "unhealthy_workers": 0 2024-04-23T18:47:02.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:02.320 INFO:teuthology.orchestra.run.smithi050.stdout: "mempool": { 2024-04-23T18:47:02.404 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:47:02.413 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_items": 0, 2024-04-23T18:47:02.423 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:47:02.431 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:47:02.440 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:47:02.463 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:47:02.471 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:47:02.478 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:47:02.486 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:47:02.494 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:47:02.501 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:47:02.507 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:47:02.515 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:47:02.521 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:47:02.529 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:47:02.536 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_items": 0, 2024-04-23T18:47:02.567 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:47:02.630 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_items": 0, 2024-04-23T18:47:02.637 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:47:02.645 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:47:02.652 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:47:02.660 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:47:02.688 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:47:02.696 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:47:02.703 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:47:02.710 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_items": 0, 2024-04-23T18:47:02.717 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:47:02.724 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:47:02.732 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:47:02.739 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_items": 0, 2024-04-23T18:47:02.747 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_bytes": 0, 2024-04-23T18:47:02.754 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_items": 0, 2024-04-23T18:47:02.761 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:47:02.806 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:47:02.870 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:47:02.880 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:47:02.889 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_bytes": 148714, 2024-04-23T18:47:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_items": 30, 2024-04-23T18:47:02.907 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:47:02.930 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_items": 0, 2024-04-23T18:47:02.937 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bytes": 0, 2024-04-23T18:47:02.946 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_items": 0, 2024-04-23T18:47:02.955 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:47:02.963 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_items": 0, 2024-04-23T18:47:02.973 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:47:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_items": 0, 2024-04-23T18:47:02.991 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_bytes": 0, 2024-04-23T18:47:03.002 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_items": 0, 2024-04-23T18:47:03.013 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:47:03.022 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:47:03.055 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_bytes": 0, 2024-04-23T18:47:03.123 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_items": 0, 2024-04-23T18:47:03.132 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_bytes": 928, 2024-04-23T18:47:03.140 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_items": 2, 2024-04-23T18:47:03.149 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_bytes": 0, 2024-04-23T18:47:03.158 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_items": 0, 2024-04-23T18:47:03.187 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_bytes": 0, 2024-04-23T18:47:03.196 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_items": 0 2024-04-23T18:47:03.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:03.214 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:47:03.222 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-04-23T18:47:03.230 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 104857600, 2024-04-23T18:47:03.240 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-04-23T18:47:03.249 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 506, 2024-04-23T18:47:03.257 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 1737955, 2024-04-23T18:47:03.266 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:03.275 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 506, 2024-04-23T18:47:03.313 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-04-23T18:47:03.364 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-04-23T18:47:03.374 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 506, 2024-04-23T18:47:03.383 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 1737955, 2024-04-23T18:47:03.390 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-04-23T18:47:03.399 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:47:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:47:03.465 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:47:03.473 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:02 smithi177 ceph-mon[32308]: pgmap v908: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:03.485 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:02 smithi188 ceph-mon[32694]: pgmap v908: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:03.494 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-23T18:47:03.502 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-23T18:47:03.511 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-23T18:47:03.519 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-23T18:47:03.551 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-23T18:47:03.623 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-23T18:47:03.635 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-23T18:47:03.644 DEBUG:tasks.check_counter:Getting stats from g 2024-04-23T18:47:03.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-23T18:47:03.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:02 smithi050 ceph-mon[29187]: pgmap v908: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:03.959 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:47:04.327 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:04 smithi188 ceph-mon[32694]: pgmap v909: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:47:04.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:04 smithi050 ceph-mon[29187]: pgmap v909: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:47:04.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:04 smithi177 ceph-mon[32308]: pgmap v909: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:47:05.598 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:47:05.606 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:47:05.615 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 4, 2024-04-23T18:47:05.622 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 4, 2024-04-23T18:47:05.632 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 2992, 2024-04-23T18:47:05.642 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 448, 2024-04-23T18:47:05.653 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 4, 2024-04-23T18:47:05.722 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 0, 2024-04-23T18:47:05.796 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.001962892, 2024-04-23T18:47:05.806 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.000037016, 2024-04-23T18:47:05.815 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.000879132, 2024-04-23T18:47:05.824 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:47:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 3, 2024-04-23T18:47:05.879 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000888824, 2024-04-23T18:47:05.888 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000296274 2024-04-23T18:47:05.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:05.907 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:05.916 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:05.924 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:05.933 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:05.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:05.950 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-23T18:47:05.961 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T18:47:05.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:06.052 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:47:06.061 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 500, 2024-04-23T18:47:06.074 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 488, 2024-04-23T18:47:06.083 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 1808208, 2024-04-23T18:47:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 302208, 2024-04-23T18:47:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 2, 2024-04-23T18:47:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 1, 2024-04-23T18:47:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.079271227, 2024-04-23T18:47:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.029441159, 2024-04-23T18:47:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.046342719, 2024-04-23T18:47:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.010003155, 2024-04-23T18:47:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 487, 2024-04-23T18:47:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.021189781, 2024-04-23T18:47:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000043510 2024-04-23T18:47:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:06.277 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:06.376 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 1808208, 2024-04-23T18:47:06.412 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 302208 2024-04-23T18:47:06.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:06.429 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:47:06.438 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 3, 2024-04-23T18:47:06.447 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 373, 2024-04-23T18:47:06.455 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 480, 2024-04-23T18:47:06.465 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 300032, 2024-04-23T18:47:06.475 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 6, 2024-04-23T18:47:06.484 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 3, 2024-04-23T18:47:06.494 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.027567482, 2024-04-23T18:47:06.502 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.020913383, 2024-04-23T18:47:06.511 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.002186044, 2024-04-23T18:47:06.585 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:47:06.591 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:06.599 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 370, 2024-04-23T18:47:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.021102002, 2024-04-23T18:47:06.616 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000057032 2024-04-23T18:47:06.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:06.655 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:06.663 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:06.672 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:06.681 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:06.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:06.699 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-23T18:47:06.707 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 300032 2024-04-23T18:47:06.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:06.722 INFO:teuthology.orchestra.run.smithi050.stdout: "cct": { 2024-04-23T18:47:06.729 INFO:teuthology.orchestra.run.smithi050.stdout: "total_workers": 0, 2024-04-23T18:47:06.769 INFO:teuthology.orchestra.run.smithi050.stdout: "unhealthy_workers": 0 2024-04-23T18:47:06.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:06.840 INFO:teuthology.orchestra.run.smithi050.stdout: "mempool": { 2024-04-23T18:47:06.850 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:47:06.859 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_items": 0, 2024-04-23T18:47:06.867 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:47:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:47:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:47:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:47:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:47:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:47:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:47:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:47:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:47:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:47:06.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:47:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:47:07.009 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:47:07.099 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_items": 0, 2024-04-23T18:47:07.108 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:47:07.116 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_items": 0, 2024-04-23T18:47:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:47:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:47:07.179 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:47:07.186 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:47:07.194 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:47:07.201 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:47:07.208 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:47:07.216 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_items": 0, 2024-04-23T18:47:07.222 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:47:07.230 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:47:07.237 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:47:07.243 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_items": 0, 2024-04-23T18:47:07.251 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_bytes": 0, 2024-04-23T18:47:07.258 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_items": 0, 2024-04-23T18:47:07.313 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:47:07.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:47:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:47:07.361 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:47:07.370 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_bytes": 148794, 2024-04-23T18:47:07.403 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_items": 32, 2024-04-23T18:47:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:47:07.418 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_items": 0, 2024-04-23T18:47:07.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bytes": 0, 2024-04-23T18:47:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_items": 0, 2024-04-23T18:47:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:47:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_items": 0, 2024-04-23T18:47:07.461 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:47:07.470 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_items": 0, 2024-04-23T18:47:07.479 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_bytes": 0, 2024-04-23T18:47:07.490 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_items": 0, 2024-04-23T18:47:07.527 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:47:07.589 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:47:07.599 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_bytes": 0, 2024-04-23T18:47:07.607 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_items": 0, 2024-04-23T18:47:07.616 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_bytes": 928, 2024-04-23T18:47:07.624 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_items": 2, 2024-04-23T18:47:07.633 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_bytes": 0, 2024-04-23T18:47:07.670 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_items": 0, 2024-04-23T18:47:07.680 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_bytes": 0, 2024-04-23T18:47:07.691 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_items": 0 2024-04-23T18:47:07.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:07.709 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:47:07.718 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-04-23T18:47:07.727 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 104857600, 2024-04-23T18:47:07.742 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-04-23T18:47:07.752 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 503, 2024-04-23T18:47:07.761 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 1737565, 2024-04-23T18:47:07.790 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:07.846 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 503, 2024-04-23T18:47:07.862 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-04-23T18:47:07.868 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-04-23T18:47:07.876 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 503, 2024-04-23T18:47:07.885 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 1737565, 2024-04-23T18:47:07.911 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-04-23T18:47:07.920 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:07.928 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:07.938 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:07.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:47:07.955 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:47:07.964 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:47:07.973 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:06 smithi177 ceph-mon[32308]: pgmap v910: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:47:07.982 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:06 smithi188 ceph-mon[32694]: pgmap v910: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:47:07.992 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-23T18:47:08.000 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-23T18:47:08.030 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-04-23T18:47:08.106 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-04-23T18:47:08.114 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-23T18:47:08.184 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-23T18:47:08.203 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-23T18:47:08.247 DEBUG:tasks.check_counter:Getting stats from j 2024-04-23T18:47:08.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-23T18:47:08.261 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:06 smithi050 ceph-mon[29187]: pgmap v910: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:47:08.327 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:08 smithi188 ceph-mon[32694]: pgmap v911: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:47:08.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:08 smithi050 ceph-mon[29187]: pgmap v911: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:47:08.532 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T18:47:08.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:08 smithi177 ceph-mon[32308]: pgmap v911: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:47:10.215 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-23T18:47:10.224 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:47:10.234 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 0, 2024-04-23T18:47:10.243 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 2, 2024-04-23T18:47:10.251 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 0, 2024-04-23T18:47:10.289 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 0, 2024-04-23T18:47:10.315 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 4, 2024-04-23T18:47:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 0, 2024-04-23T18:47:10.346 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.001402417, 2024-04-23T18:47:10.359 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T18:47:10.415 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.000279374, 2024-04-23T18:47:10.423 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:47:10.432 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:10.440 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:10.449 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:10.458 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:10.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:10.483 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:10.492 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:10.521 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:10.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:10.624 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:47:10.634 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:47:10.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:10.652 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:47:10.660 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 506, 2024-04-23T18:47:10.698 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 494, 2024-04-23T18:47:10.707 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 1809840, 2024-04-23T18:47:10.718 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 306048, 2024-04-23T18:47:10.727 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 2, 2024-04-23T18:47:10.736 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 1, 2024-04-23T18:47:10.745 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.084874034, 2024-04-23T18:47:10.754 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.031775320, 2024-04-23T18:47:10.762 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.048725628, 2024-04-23T18:47:10.771 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.010185897, 2024-04-23T18:47:10.779 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:10.787 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 493, 2024-04-23T18:47:10.850 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.021625522, 2024-04-23T18:47:10.917 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000043865 2024-04-23T18:47:10.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:10.933 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:10.942 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:10.950 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:10.991 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:11.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:11.008 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 1809840, 2024-04-23T18:47:11.017 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 306048 2024-04-23T18:47:11.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:11.034 INFO:teuthology.orchestra.run.smithi050.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:47:11.043 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_messages": 7, 2024-04-23T18:47:11.050 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages": 380, 2024-04-23T18:47:11.058 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_bytes": 3472, 2024-04-23T18:47:11.065 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_bytes": 301360, 2024-04-23T18:47:11.074 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_created_connections": 6, 2024-04-23T18:47:11.107 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_active_connections": 3, 2024-04-23T18:47:11.173 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_total_time": 0.028787389, 2024-04-23T18:47:11.183 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_send_time": 0.021604208, 2024-04-23T18:47:11.193 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_recv_time": 0.002491837, 2024-04-23T18:47:11.201 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:47:11.210 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:11.244 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 378, 2024-04-23T18:47:11.251 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.019683095, 2024-04-23T18:47:11.259 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000052071 2024-04-23T18:47:11.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:11.274 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:11.282 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:11.289 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:11.296 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:11.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:11.312 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-04-23T18:47:11.318 INFO:teuthology.orchestra.run.smithi050.stdout: "msgr_send_encrypted_bytes": 301360 2024-04-23T18:47:11.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "cct": { 2024-04-23T18:47:11.458 INFO:teuthology.orchestra.run.smithi050.stdout: "total_workers": 0, 2024-04-23T18:47:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "unhealthy_workers": 0 2024-04-23T18:47:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: "mempool": { 2024-04-23T18:47:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:47:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: "bloom_filter_items": 0, 2024-04-23T18:47:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:47:11.540 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:47:11.549 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:47:11.556 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:47:11.564 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:47:11.571 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:47:11.579 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:47:11.586 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:47:11.593 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:47:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:47:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:47:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:47:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:47:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_items": 0, 2024-04-23T18:47:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:47:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blob_items": 0, 2024-04-23T18:47:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:47:11.770 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:47:11.779 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:47:11.788 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:47:11.798 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:47:11.806 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:47:11.815 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:47:11.825 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_txc_items": 0, 2024-04-23T18:47:11.834 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:47:11.843 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:47:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:47:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_writing_items": 0, 2024-04-23T18:47:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_bytes": 0, 2024-04-23T18:47:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_items": 0, 2024-04-23T18:47:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:47:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:47:12.019 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:47:12.028 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:47:12.038 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_bytes": 148731, 2024-04-23T18:47:12.046 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_anon_items": 31, 2024-04-23T18:47:12.055 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:47:12.063 INFO:teuthology.orchestra.run.smithi050.stdout: "buffer_meta_items": 0, 2024-04-23T18:47:12.072 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bytes": 0, 2024-04-23T18:47:12.080 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_items": 0, 2024-04-23T18:47:12.089 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:47:12.110 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mapbl_items": 0, 2024-04-23T18:47:12.119 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:47:12.155 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pglog_items": 0, 2024-04-23T18:47:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_bytes": 0, 2024-04-23T18:47:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_items": 0, 2024-04-23T18:47:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:47:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:47:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_bytes": 0, 2024-04-23T18:47:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "pgmap_items": 0, 2024-04-23T18:47:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_bytes": 928, 2024-04-23T18:47:12.327 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_co_items": 2, 2024-04-23T18:47:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_bytes": 0, 2024-04-23T18:47:12.347 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_1_items": 0, 2024-04-23T18:47:12.356 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_bytes": 0, 2024-04-23T18:47:12.366 INFO:teuthology.orchestra.run.smithi050.stdout: "unittest_2_items": 0 2024-04-23T18:47:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-23T18:47:12.384 INFO:teuthology.orchestra.run.smithi050.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:47:12.392 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 0, 2024-04-23T18:47:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 104857600, 2024-04-23T18:47:12.450 INFO:teuthology.orchestra.run.smithi050.stdout: "get_started": 0, 2024-04-23T18:47:12.503 INFO:teuthology.orchestra.run.smithi050.stdout: "get": 509, 2024-04-23T18:47:12.512 INFO:teuthology.orchestra.run.smithi050.stdout: "get_sum": 1738345, 2024-04-23T18:47:12.521 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:12.530 INFO:teuthology.orchestra.run.smithi050.stdout: "get_or_fail_success": 509, 2024-04-23T18:47:12.539 INFO:teuthology.orchestra.run.smithi050.stdout: "take": 0, 2024-04-23T18:47:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: "take_sum": 0, 2024-04-23T18:47:12.582 INFO:teuthology.orchestra.run.smithi050.stdout: "put": 509, 2024-04-23T18:47:12.591 INFO:teuthology.orchestra.run.smithi050.stdout: "put_sum": 1738345, 2024-04-23T18:47:12.600 INFO:teuthology.orchestra.run.smithi050.stdout: "wait": { 2024-04-23T18:47:12.608 INFO:teuthology.orchestra.run.smithi050.stdout: "avgcount": 0, 2024-04-23T18:47:12.616 INFO:teuthology.orchestra.run.smithi050.stdout: "sum": 0.000000000, 2024-04-23T18:47:12.623 INFO:teuthology.orchestra.run.smithi050.stdout: "avgtime": 0.000000000 2024-04-23T18:47:12.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:47:12.636 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-23T18:47:12.644 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-23T18:47:12.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:10 smithi050 ceph-mon[29187]: pgmap v912: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:47:12.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:10 smithi050 ceph-mon[29187]: osdmap e141: 12 total, 12 up, 12 in 2024-04-23T18:47:12.743 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:10 smithi188 ceph-mon[32694]: pgmap v912: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:47:12.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:10 smithi188 ceph-mon[32694]: osdmap e141: 12 total, 12 up, 12 in 2024-04-23T18:47:12.757 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:12 smithi188 ceph-mon[32694]: pgmap v914: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-23T18:47:12.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:10 smithi177 ceph-mon[32308]: pgmap v912: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:47:12.771 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:10 smithi177 ceph-mon[32308]: osdmap e141: 12 total, 12 up, 12 in 2024-04-23T18:47:12.852 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:12 smithi177 ceph-mon[32308]: pgmap v914: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-23T18:47:12.861 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-23T18:47:12.870 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-23T18:47:12.877 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-23T18:47:12.883 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-23T18:47:12.889 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-23T18:47:12.896 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-23T18:47:12.902 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-23T18:47:12.918 DEBUG:tasks.check_counter:Getting stats from b 2024-04-23T18:47:12.926 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-23T18:47:12.937 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:12 smithi050 ceph-mon[29187]: pgmap v914: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-23T18:47:13.268 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:47:14.333 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:14 smithi177 ceph-mon[32308]: pgmap v915: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:47:14.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:14 smithi050 ceph-mon[29187]: pgmap v915: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:47:14.577 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:14 smithi188 ceph-mon[32694]: pgmap v915: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:47:14.976 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-23T18:47:14.983 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:47:15.032 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 290323, 2024-04-23T18:47:15.038 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 348413, 2024-04-23T18:47:15.046 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 77763023, 2024-04-23T18:47:15.055 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 491288774, 2024-04-23T18:47:15.063 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 208, 2024-04-23T18:47:15.071 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-04-23T18:47:15.080 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 16.655795320, 2024-04-23T18:47:15.089 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 7.212855117, 2024-04-23T18:47:15.098 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 8.917783204, 2024-04-23T18:47:15.107 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 1.556651604, 2024-04-23T18:47:15.117 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:15.126 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 348412, 2024-04-23T18:47:15.201 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 26.159567255, 2024-04-23T18:47:15.209 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000075082 2024-04-23T18:47:15.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:15.227 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:15.235 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 523699, 2024-04-23T18:47:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.070723989, 2024-04-23T18:47:15.274 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000135 2024-04-23T18:47:15.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:15.290 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 447584, 2024-04-23T18:47:15.299 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 155136 2024-04-23T18:47:15.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:15.316 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:47:15.325 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 41813, 2024-04-23T18:47:15.331 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 41969, 2024-04-23T18:47:15.338 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 16386632, 2024-04-23T18:47:15.345 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 213725295, 2024-04-23T18:47:15.370 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 77, 2024-04-23T18:47:15.449 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 7, 2024-04-23T18:47:15.458 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 3.143848179, 2024-04-23T18:47:15.466 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 1.130360354, 2024-04-23T18:47:15.475 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.706835105, 2024-04-23T18:47:15.483 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 1.106789504, 2024-04-23T18:47:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 41967, 2024-04-23T18:47:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 24.277673385, 2024-04-23T18:47:15.549 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000578494 2024-04-23T18:47:15.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:15.562 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:15.569 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 353, 2024-04-23T18:47:15.576 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000047043, 2024-04-23T18:47:15.584 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000133 2024-04-23T18:47:15.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:15.598 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 2174192, 2024-04-23T18:47:15.634 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 1072464 2024-04-23T18:47:15.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:15.717 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:47:15.724 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 711888, 2024-04-23T18:47:15.730 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 892697, 2024-04-23T18:47:15.737 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 205705691, 2024-04-23T18:47:15.776 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 653929431, 2024-04-23T18:47:15.783 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 20, 2024-04-23T18:47:15.790 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 6, 2024-04-23T18:47:15.798 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 33.336726871, 2024-04-23T18:47:15.805 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 14.555360100, 2024-04-23T18:47:15.813 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 21.720544092, 2024-04-23T18:47:15.820 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 1.141599385, 2024-04-23T18:47:15.827 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:15.834 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 892695, 2024-04-23T18:47:15.841 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 39.855463574, 2024-04-23T18:47:15.849 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000044646 2024-04-23T18:47:15.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:15.924 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:15.931 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1452803, 2024-04-23T18:47:15.940 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.199952848, 2024-04-23T18:47:15.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000137 2024-04-23T18:47:15.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:16.000 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 514224, 2024-04-23T18:47:16.010 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 289648 2024-04-23T18:47:16.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:16.027 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-23T18:47:16.036 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-23T18:47:16.047 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-23T18:47:16.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:16.064 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-23T18:47:16.072 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-23T18:47:16.082 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-23T18:47:16.090 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 127024, 2024-04-23T18:47:16.119 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 124.737813821, 2024-04-23T18:47:16.182 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000982001 2024-04-23T18:47:16.190 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:16.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:16.207 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-23T18:47:16.216 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-23T18:47:16.251 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-23T18:47:16.260 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 54769, 2024-04-23T18:47:16.267 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 7.201132023, 2024-04-23T18:47:16.275 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000131481 2024-04-23T18:47:16.284 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:16.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:16.301 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-23T18:47:16.310 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 283924, 2024-04-23T18:47:16.319 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 246913, 2024-04-23T18:47:16.329 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-23T18:47:16.338 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 246913, 2024-04-23T18:47:16.367 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 1688.511126791, 2024-04-23T18:47:16.429 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.006838486 2024-04-23T18:47:16.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:16.447 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-23T18:47:16.455 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 37011, 2024-04-23T18:47:16.462 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 39, 2024-04-23T18:47:16.501 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-04-23T18:47:16.511 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 4099, 2024-04-23T18:47:16.521 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 21, 2024-04-23T18:47:16.527 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 20, 2024-04-23T18:47:16.535 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 2916, 2024-04-23T18:47:16.544 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 1436, 2024-04-23T18:47:16.552 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 1260, 2024-04-23T18:47:16.561 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 220, 2024-04-23T18:47:16.570 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 865, 2024-04-23T18:47:16.579 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 3031, 2024-04-23T18:47:16.587 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 147, 2024-04-23T18:47:16.623 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 148, 2024-04-23T18:47:16.684 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 33, 2024-04-23T18:47:16.692 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 439290, 2024-04-23T18:47:16.701 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 251212, 2024-04-23T18:47:16.710 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 36969, 2024-04-23T18:47:16.719 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 170, 2024-04-23T18:47:16.759 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 16, 2024-04-23T18:47:16.769 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-23T18:47:16.778 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 3269, 2024-04-23T18:47:16.786 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 93, 2024-04-23T18:47:16.795 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-23T18:47:16.803 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 36, 2024-04-23T18:47:16.812 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 2200, 2024-04-23T18:47:16.820 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 30, 2024-04-23T18:47:16.826 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 1167, 2024-04-23T18:47:16.833 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-23T18:47:16.842 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:47:16.868 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-23T18:47:16.930 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 774, 2024-04-23T18:47:16.937 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 231532028, 2024-04-23T18:47:16.945 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 6, 2024-04-23T18:47:16.953 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:47:16.961 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-23T18:47:17.001 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:47:17.007 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:47:17.015 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:47:17.022 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:47:17.028 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:47:17.035 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-23T18:47:17.043 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 21, 2024-04-23T18:47:17.067 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 47570, 2024-04-23T18:47:17.074 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 122093, 2024-04-23T18:47:17.081 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:47:17.089 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:47:17.133 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:47:17.199 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 86910, 2024-04-23T18:47:17.209 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 20806, 2024-04-23T18:47:17.218 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 10632, 2024-04-23T18:47:17.227 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 162347 2024-04-23T18:47:17.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:17.275 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-23T18:47:17.285 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 562, 2024-04-23T18:47:17.298 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 244, 2024-04-23T18:47:17.306 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 158, 2024-04-23T18:47:17.315 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 262, 2024-04-23T18:47:17.324 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 1535, 2024-04-23T18:47:17.332 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-23T18:47:17.344 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-23T18:47:17.353 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:47:17.362 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 32025, 2024-04-23T18:47:17.369 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 32025, 2024-04-23T18:47:17.404 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-23T18:47:17.479 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-23T18:47:17.488 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-23T18:47:17.496 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:47:17.505 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:47:17.514 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-23T18:47:17.547 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-23T18:47:17.558 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 18, 2024-04-23T18:47:17.568 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 2842, 2024-04-23T18:47:17.576 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:47:17.586 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 39, 2024-04-23T18:47:17.595 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-23T18:47:17.603 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 41, 2024-04-23T18:47:17.612 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-23T18:47:17.622 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-23T18:47:17.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:17.640 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-23T18:47:17.667 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-23T18:47:17.759 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 178997, 2024-04-23T18:47:17.765 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 177451, 2024-04-23T18:47:17.774 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 177451, 2024-04-23T18:47:17.782 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 1546, 2024-04-23T18:47:17.791 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-23T18:47:17.833 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 4737, 2024-04-23T18:47:17.842 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 335, 2024-04-23T18:47:17.850 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 333, 2024-04-23T18:47:17.859 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 333, 2024-04-23T18:47:17.867 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-04-23T18:47:17.876 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 1, 2024-04-23T18:47:17.884 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-23T18:47:17.891 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 8, 2024-04-23T18:47:17.898 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 680655713, 2024-04-23T18:47:17.905 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 686637953, 2024-04-23T18:47:17.912 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 0, 2024-04-23T18:47:17.941 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-23T18:47:18.035 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 66145, 2024-04-23T18:47:18.044 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 326.732996138, 2024-04-23T18:47:18.053 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.004939647 2024-04-23T18:47:18.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:18.070 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 0 2024-04-23T18:47:18.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:18.130 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-23T18:47:18.137 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 1953, 2024-04-23T18:47:18.145 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 36167, 2024-04-23T18:47:18.153 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 34214, 2024-04-23T18:47:18.161 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 296, 2024-04-23T18:47:18.169 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 682, 2024-04-23T18:47:18.176 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 386, 2024-04-23T18:47:18.183 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 2916, 2024-04-23T18:47:18.190 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 74575, 2024-04-23T18:47:18.197 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 71659, 2024-04-23T18:47:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 148, 2024-04-23T18:47:18.293 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 36147, 2024-04-23T18:47:18.304 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 35999, 2024-04-23T18:47:18.309 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 106904, 2024-04-23T18:47:18.317 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 231800 2024-04-23T18:47:18.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:18.358 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-23T18:47:18.366 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 409770, 2024-04-23T18:47:18.372 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 36805, 2024-04-23T18:47:18.377 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 283924, 2024-04-23T18:47:18.384 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 3196, 2024-04-23T18:47:18.390 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 77069, 2024-04-23T18:47:18.395 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-23T18:47:18.403 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 32760, 2024-04-23T18:47:18.410 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 653.108960898, 2024-04-23T18:47:18.416 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.019936170 2024-04-23T18:47:18.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:18.457 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-23T18:47:18.533 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 30191, 2024-04-23T18:47:18.541 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 161.121218275, 2024-04-23T18:47:18.548 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.005336730 2024-04-23T18:47:18.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:18.562 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-23T18:47:18.606 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:18.613 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:18.621 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:18.629 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:18.636 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-23T18:47:18.644 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:18.651 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:18.659 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:18.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:18.675 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-23T18:47:18.684 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 67707, 2024-04-23T18:47:18.693 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 171.960719261, 2024-04-23T18:47:18.767 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.002539777 2024-04-23T18:47:18.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:18.782 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-23T18:47:18.791 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:18.800 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:18.833 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:18.842 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:18.849 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-23T18:47:18.857 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2, 2024-04-23T18:47:18.865 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000389627, 2024-04-23T18:47:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000194813 2024-04-23T18:47:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-23T18:47:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.007 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-23T18:47:19.016 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:19.025 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:19.032 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:19.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.083 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:16 smithi188 ceph-mon[32694]: pgmap v916: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:47:19.092 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:17 smithi188 ceph-mon[32694]: osdmap e142: 12 total, 12 up, 12 in 2024-04-23T18:47:19.100 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:18 smithi188 ceph-mon[32694]: pgmap v918: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:47:19.109 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:16 smithi050 ceph-mon[29187]: pgmap v916: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:47:19.118 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:17 smithi050 ceph-mon[29187]: osdmap e142: 12 total, 12 up, 12 in 2024-04-23T18:47:19.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:18 smithi050 ceph-mon[29187]: pgmap v918: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:47:19.139 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-23T18:47:19.148 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:19.157 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:19.166 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:19.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.215 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-23T18:47:19.275 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 29, 2024-04-23T18:47:19.284 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.006930034, 2024-04-23T18:47:19.293 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000238966 2024-04-23T18:47:19.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.309 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-23T18:47:19.358 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 119, 2024-04-23T18:47:19.367 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.828008080, 2024-04-23T18:47:19.376 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.006958051 2024-04-23T18:47:19.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.396 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-23T18:47:19.405 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:19.414 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:19.423 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-23T18:47:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 29, 2024-04-23T18:47:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 3.798171919, 2024-04-23T18:47:19.535 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.130971445 2024-04-23T18:47:19.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.565 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-23T18:47:19.573 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 10, 2024-04-23T18:47:19.584 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.027268250, 2024-04-23T18:47:19.622 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.002726825 2024-04-23T18:47:19.628 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.636 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-23T18:47:19.644 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 77999, 2024-04-23T18:47:19.653 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 309.436474692, 2024-04-23T18:47:19.662 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.003967185 2024-04-23T18:47:19.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.679 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-23T18:47:19.687 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 5244, 2024-04-23T18:47:19.698 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 205.105806560, 2024-04-23T18:47:19.705 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.039112472 2024-04-23T18:47:19.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.803 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-23T18:47:19.809 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:19.816 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:19.824 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:19.832 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.904 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-23T18:47:19.911 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:19.919 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:19.928 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:19.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.945 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-23T18:47:19.953 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1, 2024-04-23T18:47:19.963 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.001167157, 2024-04-23T18:47:19.971 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.001167157 2024-04-23T18:47:19.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:19.990 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-23T18:47:20.030 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 23, 2024-04-23T18:47:20.095 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 2.856823397, 2024-04-23T18:47:20.104 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.124209712 2024-04-23T18:47:20.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:20.122 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-23T18:47:20.132 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:20.163 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:20.169 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:20.177 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:20.186 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-23T18:47:20.194 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 47, 2024-04-23T18:47:20.203 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.522025523, 2024-04-23T18:47:20.211 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.011106926 2024-04-23T18:47:20.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:20.228 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-23T18:47:20.236 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:20.245 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:20.252 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:20.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:20.328 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-23T18:47:20.337 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 720, 2024-04-23T18:47:20.346 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 1.652925655, 2024-04-23T18:47:20.354 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.002295730 2024-04-23T18:47:20.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:20.398 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-23T18:47:20.407 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:20.413 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:20.420 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:20.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:20.434 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-23T18:47:20.441 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 7, 2024-04-23T18:47:20.447 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.017269322, 2024-04-23T18:47:20.454 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.002467046 2024-04-23T18:47:20.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:20.507 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-23T18:47:20.565 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:20.578 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:20.584 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:20.592 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:20.601 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-23T18:47:20.630 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 32025, 2024-04-23T18:47:20.638 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 178.066968141, 2024-04-23T18:47:20.647 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.005560248 2024-04-23T18:47:20.656 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:20.665 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:47:20.674 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:47:20.682 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-23T18:47:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:20.716 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:20.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-23T18:47:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 2, 2024-04-23T18:47:20.846 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 6, 2024-04-23T18:47:20.854 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 4, 2024-04-23T18:47:20.863 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 2, 2024-04-23T18:47:20.896 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-23T18:47:20.905 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 173, 2024-04-23T18:47:20.914 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 86, 2024-04-23T18:47:20.922 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 1229, 2024-04-23T18:47:20.931 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-23T18:47:20.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:20.949 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-23T18:47:20.957 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 600, 2024-04-23T18:47:20.965 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 600, 2024-04-23T18:47:20.974 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:47:20.983 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:47:21.027 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:47:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:47:21.092 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:47:21.100 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:47:21.109 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:47:21.118 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:47:21.171 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:47:21.180 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:47:21.190 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:47:21.199 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:47:21.208 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:47:21.217 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-23T18:47:21.226 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:47:21.235 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-23T18:47:21.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:47:21.253 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:47:21.262 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:47:21.271 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:47:21.326 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:47:21.334 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:47:21.343 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:47:21.350 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-23T18:47:21.357 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:47:21.391 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:47:21.399 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:47:21.407 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-23T18:47:21.415 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-23T18:47:21.426 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-23T18:47:21.435 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:47:21.443 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:47:21.452 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:47:21.459 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:47:21.466 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 351192, 2024-04-23T18:47:21.473 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 1173, 2024-04-23T18:47:21.502 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 264, 2024-04-23T18:47:21.560 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 3, 2024-04-23T18:47:21.569 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-23T18:47:21.577 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-23T18:47:21.586 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:47:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-23T18:47:21.638 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:47:21.645 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-23T18:47:21.651 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 8248, 2024-04-23T18:47:21.658 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 172, 2024-04-23T18:47:21.663 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:47:21.670 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:47:21.677 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-23T18:47:21.684 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-23T18:47:21.691 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 7606283, 2024-04-23T18:47:21.698 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 62949, 2024-04-23T18:47:21.705 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-23T18:47:21.755 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-23T18:47:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-23T18:47:21.831 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-23T18:47:21.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:21.848 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-23T18:47:21.854 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-23T18:47:21.860 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-23T18:47:21.866 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 122506, 2024-04-23T18:47:21.873 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 716977341, 2024-04-23T18:47:21.924 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 37, 2024-04-23T18:47:21.931 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 122506, 2024-04-23T18:47:21.938 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-23T18:47:21.967 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 122506, 2024-04-23T18:47:21.974 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 1929.812561596, 2024-04-23T18:47:21.981 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.015752800 2024-04-23T18:47:21.988 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:21.995 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-23T18:47:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-23T18:47:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 122506, 2024-04-23T18:47:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 163180 2024-04-23T18:47:22.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:22.098 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 122506, 2024-04-23T18:47:22.105 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 1146, 2024-04-23T18:47:22.148 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 121360, 2024-04-23T18:47:22.157 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-23T18:47:22.164 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-23T18:47:22.170 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 3781, 2024-04-23T18:47:22.178 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 14946, 2024-04-23T18:47:22.187 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 1107, 2024-04-23T18:47:22.196 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 67252, 2024-04-23T18:47:22.205 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 986, 2024-04-23T18:47:22.214 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-23T18:47:22.222 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-23T18:47:22.231 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-04-23T18:47:22.239 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 21, 2024-04-23T18:47:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 32985, 2024-04-23T18:47:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-23T18:47:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-23T18:47:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-23T18:47:22.327 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 31, 2024-04-23T18:47:22.371 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 29892, 2024-04-23T18:47:22.380 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:47:22.388 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-23T18:47:22.397 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:47:22.412 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-23T18:47:22.420 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-23T18:47:22.429 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-23T18:47:22.437 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:47:22.446 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-23T18:47:22.457 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:47:22.466 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-23T18:47:22.498 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-23T18:47:22.550 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-23T18:47:22.560 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-23T18:47:22.568 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-23T18:47:22.577 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-23T18:47:22.586 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-23T18:47:22.632 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-23T18:47:22.640 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-23T18:47:22.649 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-23T18:47:22.658 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-23T18:47:22.668 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-23T18:47:22.676 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-23T18:47:22.685 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-23T18:47:22.693 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-23T18:47:22.702 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-23T18:47:22.711 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-23T18:47:22.719 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 141, 2024-04-23T18:47:22.771 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-23T18:47:22.860 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 55, 2024-04-23T18:47:22.866 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 12, 2024-04-23T18:47:22.876 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 12, 2024-04-23T18:47:22.884 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-04-23T18:47:22.892 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-23T18:47:22.900 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 9417, 2024-04-23T18:47:22.946 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 78, 2024-04-23T18:47:22.956 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 2684 2024-04-23T18:47:22.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:22.972 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-23T18:47:22.981 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 1, 2024-04-23T18:47:22.990 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:47:22.997 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-23T18:47:23.007 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-23T18:47:23.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:23.023 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-23T18:47:23.059 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 4106 2024-04-23T18:47:23.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:23.124 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-23T18:47:23.131 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-23T18:47:23.140 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 10, 2024-04-23T18:47:23.148 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-23T18:47:23.182 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 10, 2024-04-23T18:47:23.189 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 32025, 2024-04-23T18:47:23.196 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 32025, 2024-04-23T18:47:23.203 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-23T18:47:23.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:23.216 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:47:23.223 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:23.230 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-23T18:47:23.237 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:23.245 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 1044020, 2024-04-23T18:47:23.252 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 205745581, 2024-04-23T18:47:23.282 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 1044020, 2024-04-23T18:47:23.351 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-23T18:47:23.358 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-23T18:47:23.365 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 1044020, 2024-04-23T18:47:23.372 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 205745581, 2024-04-23T18:47:23.412 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:23.419 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:23.428 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:23.435 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:23.443 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:23.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:23.462 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-23T18:47:23.471 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:23.481 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-23T18:47:23.491 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:23.501 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-23T18:47:23.535 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-23T18:47:23.592 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:23.599 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-23T18:47:23.608 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 122506, 2024-04-23T18:47:23.617 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 720216418, 2024-04-23T18:47:23.626 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 122506, 2024-04-23T18:47:23.665 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 720216418, 2024-04-23T18:47:23.674 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:23.681 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:23.689 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:23.697 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:23.704 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:23.711 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:23.719 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-23T18:47:23.728 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:23.738 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-23T18:47:23.746 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:23.799 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-23T18:47:23.867 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-23T18:47:23.877 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:23.885 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-23T18:47:23.897 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 122506, 2024-04-23T18:47:23.906 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 122506, 2024-04-23T18:47:23.935 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 122506, 2024-04-23T18:47:23.945 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 122506, 2024-04-23T18:47:23.953 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:23.962 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:23.970 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:23.979 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:23.988 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:23.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:24.005 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:47:24.013 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:24.022 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-23T18:47:24.054 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:24.117 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 32025, 2024-04-23T18:47:24.126 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 3239701, 2024-04-23T18:47:24.135 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:24.144 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 32025, 2024-04-23T18:47:24.153 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-23T18:47:24.203 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-23T18:47:24.210 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 1107, 2024-04-23T18:47:24.219 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 3239701, 2024-04-23T18:47:24.227 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:24.235 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:24.244 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:24.253 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:24.262 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:24.270 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:24.279 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x559217932500": { 2024-04-23T18:47:24.287 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:24.294 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-23T18:47:24.383 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:24.419 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 178997, 2024-04-23T18:47:24.428 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 682443649, 2024-04-23T18:47:24.437 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:24.445 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 178997, 2024-04-23T18:47:24.495 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-23T18:47:24.505 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-23T18:47:24.513 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 66145, 2024-04-23T18:47:24.522 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 682443649, 2024-04-23T18:47:24.551 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:24.560 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:24.568 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:24.576 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:24.584 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:24.594 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:24.602 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-23T18:47:24.632 INFO:tasks.check_counter:Daemon mds.b mds.exported=36 2024-04-23T18:47:24.691 INFO:tasks.check_counter:Daemon mds.b mds.imported=30 2024-04-23T18:47:24.701 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=562 2024-04-23T18:47:24.714 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=244 2024-04-23T18:47:24.723 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-23T18:47:24.731 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=29 2024-04-23T18:47:24.795 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=23 2024-04-23T18:47:24.803 DEBUG:tasks.check_counter:Getting stats from e 2024-04-23T18:47:24.809 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-23T18:47:24.816 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:16 smithi177 ceph-mon[32308]: pgmap v916: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:47:24.823 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:17 smithi177 ceph-mon[32308]: osdmap e142: 12 total, 12 up, 12 in 2024-04-23T18:47:24.832 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:18 smithi177 ceph-mon[32308]: pgmap v918: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:47:24.841 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:20 smithi177 ceph-mon[32308]: pgmap v919: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 207 B/s wr, 6 op/s 2024-04-23T18:47:24.851 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:20 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:47:24.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:20 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:47:24.869 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:20 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:47:24.878 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:20 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:47:24.911 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:22 smithi177 ceph-mon[32308]: pgmap v920: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:47:24.967 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:24 smithi177 ceph-mon[32308]: pgmap v921: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2024-04-23T18:47:24.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:20 smithi188 ceph-mon[32694]: pgmap v919: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 207 B/s wr, 6 op/s 2024-04-23T18:47:24.983 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:20 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:47:24.990 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:20 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:47:24.998 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:20 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:47:25.032 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:20 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:47:25.038 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:22 smithi188 ceph-mon[32694]: pgmap v920: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:47:25.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:24 smithi188 ceph-mon[32694]: pgmap v921: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2024-04-23T18:47:25.054 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:20 smithi050 ceph-mon[29187]: pgmap v919: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 207 B/s wr, 6 op/s 2024-04-23T18:47:25.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:20 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:47:25.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:20 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:47:25.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:20 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:47:25.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:20 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:47:25.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:22 smithi050 ceph-mon[29187]: pgmap v920: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:47:25.097 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:24 smithi050 ceph-mon[29187]: pgmap v921: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2024-04-23T18:47:25.223 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:47:26.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:26 smithi177 ceph-mon[32308]: pgmap v922: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2024-04-23T18:47:26.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:26 smithi050 ceph-mon[29187]: pgmap v922: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2024-04-23T18:47:26.827 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:26 smithi188 ceph-mon[32694]: pgmap v922: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 5 op/s 2024-04-23T18:47:26.904 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-23T18:47:26.910 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:47:26.919 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 2516, 2024-04-23T18:47:26.948 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 2518, 2024-04-23T18:47:26.957 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 237749013, 2024-04-23T18:47:26.966 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 800152, 2024-04-23T18:47:26.977 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 8, 2024-04-23T18:47:26.985 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-04-23T18:47:26.993 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.302033193, 2024-04-23T18:47:27.003 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.058912894, 2024-04-23T18:47:27.012 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.164669906, 2024-04-23T18:47:27.021 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.061885875, 2024-04-23T18:47:27.030 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:27.039 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2516, 2024-04-23T18:47:27.082 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.091067751, 2024-04-23T18:47:27.151 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000036195 2024-04-23T18:47:27.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:27.169 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:27.177 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:27.186 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:27.223 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:27.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:27.239 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:47:27.250 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:47:27.259 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:27.267 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:47:27.275 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 1556, 2024-04-23T18:47:27.283 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 1934, 2024-04-23T18:47:27.293 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 235285608, 2024-04-23T18:47:27.302 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 1419443, 2024-04-23T18:47:27.310 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 10, 2024-04-23T18:47:27.343 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 7, 2024-04-23T18:47:27.391 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.257689370, 2024-04-23T18:47:27.400 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.042477715, 2024-04-23T18:47:27.407 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.162659163, 2024-04-23T18:47:27.416 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.038924247, 2024-04-23T18:47:27.424 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:27.456 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1931, 2024-04-23T18:47:27.465 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.056331575, 2024-04-23T18:47:27.474 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000029172 2024-04-23T18:47:27.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:27.490 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:27.499 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:27.508 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:27.515 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:27.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:27.531 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-23T18:47:27.539 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 925584 2024-04-23T18:47:27.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:27.620 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:47:27.627 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 3763, 2024-04-23T18:47:27.634 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 3703, 2024-04-23T18:47:27.643 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 225323091, 2024-04-23T18:47:27.650 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 1318720, 2024-04-23T18:47:27.681 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 5, 2024-04-23T18:47:27.689 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-04-23T18:47:27.695 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.484535015, 2024-04-23T18:47:27.703 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.111511772, 2024-04-23T18:47:27.711 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.253318339, 2024-04-23T18:47:27.721 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.103971657, 2024-04-23T18:47:27.729 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:27.738 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3703, 2024-04-23T18:47:27.746 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.134715722, 2024-04-23T18:47:27.754 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000036380 2024-04-23T18:47:27.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:27.787 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:27.833 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:27.843 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:27.850 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:27.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:27.867 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 2058480, 2024-04-23T18:47:27.906 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 318928 2024-04-23T18:47:27.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:27.924 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-23T18:47:27.933 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-23T18:47:27.942 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-23T18:47:27.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:27.961 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-23T18:47:27.970 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-23T18:47:27.979 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-23T18:47:27.992 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 11624, 2024-04-23T18:47:28.001 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.298128415, 2024-04-23T18:47:28.034 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000025647 2024-04-23T18:47:28.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:28.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:28.097 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-23T18:47:28.105 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-23T18:47:28.113 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-23T18:47:28.121 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:28.156 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:28.164 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:28.172 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:28.180 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:28.189 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-23T18:47:28.196 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 0, 2024-04-23T18:47:28.203 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 0, 2024-04-23T18:47:28.209 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-23T18:47:28.216 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:28.223 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:28.256 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:28.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:28.319 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-23T18:47:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-04-23T18:47:28.336 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-04-23T18:47:28.344 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-04-23T18:47:28.389 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 0, 2024-04-23T18:47:28.398 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-04-23T18:47:28.407 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-04-23T18:47:28.415 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 28037, 2024-04-23T18:47:28.424 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 5517, 2024-04-23T18:47:28.433 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 22520, 2024-04-23T18:47:28.442 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-04-23T18:47:28.451 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 20156, 2024-04-23T18:47:28.459 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 0, 2024-04-23T18:47:28.467 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 0, 2024-04-23T18:47:28.475 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 0, 2024-04-23T18:47:28.512 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 21, 2024-04-23T18:47:28.566 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 0, 2024-04-23T18:47:28.580 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 0, 2024-04-23T18:47:28.590 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-04-23T18:47:28.597 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 0, 2024-04-23T18:47:28.603 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:47:28.665 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-23T18:47:28.676 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-04-23T18:47:28.685 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-04-23T18:47:28.694 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-23T18:47:28.702 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-04-23T18:47:28.710 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-04-23T18:47:28.717 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-04-23T18:47:28.724 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-04-23T18:47:28.730 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-23T18:47:28.737 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:47:28.744 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-23T18:47:28.792 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 0, 2024-04-23T18:47:28.867 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 0, 2024-04-23T18:47:28.876 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 0, 2024-04-23T18:47:28.885 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:47:28.894 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-23T18:47:28.903 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:47:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:47:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:47:28.968 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:47:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:47:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-23T18:47:28.994 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-04-23T18:47:29.002 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T18:47:29.011 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-04-23T18:47:29.020 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:47:29.028 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:47:29.036 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:47:29.043 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-04-23T18:47:29.134 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-04-23T18:47:29.144 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-04-23T18:47:29.153 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-04-23T18:47:29.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:29.171 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-23T18:47:29.180 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-04-23T18:47:29.189 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 0, 2024-04-23T18:47:29.290 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 0, 2024-04-23T18:47:29.299 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 0, 2024-04-23T18:47:29.306 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-04-23T18:47:29.313 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-23T18:47:29.320 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-23T18:47:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:47:29.333 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-04-23T18:47:29.340 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-04-23T18:47:29.347 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-23T18:47:29.377 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-23T18:47:29.432 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-23T18:47:29.439 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:47:29.446 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:47:29.452 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-23T18:47:29.459 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-23T18:47:29.510 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 0, 2024-04-23T18:47:29.517 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 0, 2024-04-23T18:47:29.525 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:47:29.532 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-04-23T18:47:29.539 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-23T18:47:29.545 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-04-23T18:47:29.552 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-23T18:47:29.559 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-23T18:47:29.566 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:29.574 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-23T18:47:29.582 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-23T18:47:29.617 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 0, 2024-04-23T18:47:29.671 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-04-23T18:47:29.678 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-04-23T18:47:29.685 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 150036, 2024-04-23T18:47:29.692 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-23T18:47:29.699 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-04-23T18:47:29.752 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 0, 2024-04-23T18:47:29.758 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-04-23T18:47:29.766 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-04-23T18:47:29.773 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 143, 2024-04-23T18:47:29.779 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 14, 2024-04-23T18:47:29.786 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-23T18:47:29.793 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-04-23T18:47:29.801 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 150863246, 2024-04-23T18:47:29.810 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 698442440, 2024-04-23T18:47:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 698441866, 2024-04-23T18:47:29.826 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-23T18:47:29.834 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:29.897 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:29.906 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:29.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:29.922 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 150036 2024-04-23T18:47:29.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:29.993 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-23T18:47:30.002 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 2353, 2024-04-23T18:47:30.009 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 28601, 2024-04-23T18:47:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 26248, 2024-04-23T18:47:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 126, 2024-04-23T18:47:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 287, 2024-04-23T18:47:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 161, 2024-04-23T18:47:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 28037, 2024-04-23T18:47:30.059 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 28934, 2024-04-23T18:47:30.068 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 897, 2024-04-23T18:47:30.078 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 0, 2024-04-23T18:47:30.110 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 0, 2024-04-23T18:47:30.161 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-04-23T18:47:30.170 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 88144, 2024-04-23T18:47:30.179 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 215416 2024-04-23T18:47:30.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:30.197 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-23T18:47:30.243 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 0, 2024-04-23T18:47:30.252 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-04-23T18:47:30.261 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 0, 2024-04-23T18:47:30.270 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 0, 2024-04-23T18:47:30.278 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-04-23T18:47:30.286 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-23T18:47:30.295 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:30.303 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:30.312 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:30.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:30.327 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-23T18:47:30.336 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:30.382 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:30.391 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:30.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:30.407 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-23T18:47:30.416 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:30.468 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:30.475 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:30.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:30.491 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-23T18:47:30.500 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:30.508 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:30.517 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:30.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:30.533 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-23T18:47:30.551 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:30.559 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:30.596 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:30.657 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:30.664 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-23T18:47:30.672 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:30.678 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:30.686 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:30.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:30.742 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-23T18:47:30.749 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:30.755 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:30.763 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:30.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:30.776 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-23T18:47:30.784 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:30.791 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:30.799 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:30.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:30.842 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-23T18:47:30.898 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:30.908 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:30.917 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:30.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:30.932 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-23T18:47:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:30.987 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:30.994 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:31.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:31.013 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-23T18:47:31.020 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:31.028 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:31.035 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:31.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:28 smithi188 ceph-mon[32694]: pgmap v923: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:47:31.049 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:30 smithi188 ceph-mon[32694]: pgmap v924: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:31.056 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:28 smithi050 ceph-mon[29187]: pgmap v923: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:47:31.089 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:30 smithi050 ceph-mon[29187]: pgmap v924: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:31.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:31.156 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-23T18:47:31.163 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:31.170 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:31.177 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:31.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:31.233 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-23T18:47:31.239 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:31.245 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:31.253 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:31.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:31.268 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-23T18:47:31.274 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:31.280 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:31.289 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:31.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:31.360 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-23T18:47:31.452 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:31.460 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:31.467 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:31.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:31.485 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-23T18:47:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:31.543 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:31.550 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:31.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:31.570 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-23T18:47:31.580 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:31.588 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:31.597 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:31.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:31.616 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-23T18:47:31.625 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:31.657 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:31.723 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:31.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:31.741 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-23T18:47:31.749 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:31.757 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:31.765 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:31.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:31.811 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-23T18:47:31.821 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:31.831 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:31.841 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:31.849 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:31.858 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-23T18:47:31.867 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:31.875 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:31.885 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:31.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:31.971 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-23T18:47:31.981 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:31.990 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:31.998 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:32.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:32.051 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-23T18:47:32.059 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:32.067 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:32.075 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:32.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:32.091 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-23T18:47:32.100 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:32.108 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:32.118 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:32.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:32.135 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-23T18:47:32.168 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:32.225 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:32.265 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:32.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:32.282 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-23T18:47:32.290 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:32.334 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:32.344 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:32.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:32.361 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-23T18:47:32.371 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:32.379 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:32.389 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:32.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:32.404 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-23T18:47:32.411 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:32.418 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:32.448 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:32.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:32.506 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-23T18:47:32.513 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:32.521 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:32.527 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:32.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:32.542 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:47:32.604 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:47:32.612 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-23T18:47:32.619 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:32.627 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:32.635 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:32.642 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:32.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:32.657 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-23T18:47:32.663 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 2, 2024-04-23T18:47:32.696 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 5, 2024-04-23T18:47:32.740 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 3, 2024-04-23T18:47:32.748 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 2, 2024-04-23T18:47:32.755 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-23T18:47:32.762 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 0, 2024-04-23T18:47:32.769 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 0, 2024-04-23T18:47:32.826 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 5190, 2024-04-23T18:47:32.833 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-23T18:47:32.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:32.846 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-23T18:47:32.853 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:47:32.861 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-23T18:47:32.869 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:47:32.876 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:47:32.883 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:47:32.890 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:47:32.896 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:47:32.925 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:47:32.973 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:47:32.990 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:47:32.997 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:47:33.005 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:47:33.012 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:47:33.059 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:47:33.068 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:47:33.074 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-23T18:47:33.081 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:47:33.089 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-23T18:47:33.097 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:47:33.106 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:47:33.114 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:47:33.123 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:47:33.132 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:47:33.141 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:47:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:47:33.214 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-23T18:47:33.223 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:47:33.231 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:47:33.240 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:47:33.250 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-23T18:47:33.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-23T18:47:33.315 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-23T18:47:33.324 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:47:33.334 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:47:33.343 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:47:33.351 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:47:33.361 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 344155, 2024-04-23T18:47:33.370 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 2870, 2024-04-23T18:47:33.379 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:47:33.387 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-23T18:47:33.396 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-23T18:47:33.435 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-23T18:47:33.496 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:47:33.504 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-23T18:47:33.513 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:47:33.521 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-23T18:47:33.530 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 8128, 2024-04-23T18:47:33.592 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 169, 2024-04-23T18:47:33.604 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:47:33.613 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:47:33.622 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-23T18:47:33.631 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-23T18:47:33.639 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 25879097, 2024-04-23T18:47:33.647 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 174288, 2024-04-23T18:47:33.657 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-23T18:47:33.666 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-23T18:47:33.674 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-23T18:47:33.683 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-23T18:47:33.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:33.756 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-23T18:47:33.765 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-23T18:47:33.773 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-23T18:47:33.781 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 7213, 2024-04-23T18:47:33.790 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 21, 2024-04-23T18:47:33.854 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 3, 2024-04-23T18:47:33.862 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 7213, 2024-04-23T18:47:33.871 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-23T18:47:33.880 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 7213, 2024-04-23T18:47:33.890 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 7.675790921, 2024-04-23T18:47:33.897 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.001064160 2024-04-23T18:47:33.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:33.913 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-23T18:47:33.922 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-23T18:47:33.931 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 7213, 2024-04-23T18:47:33.940 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 7214 2024-04-23T18:47:33.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:34.044 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 7213, 2024-04-23T18:47:34.053 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 7213, 2024-04-23T18:47:34.059 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 0, 2024-04-23T18:47:34.069 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-23T18:47:34.075 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-23T18:47:34.117 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 4358, 2024-04-23T18:47:34.124 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 0, 2024-04-23T18:47:34.134 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 2853, 2024-04-23T18:47:34.139 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 0, 2024-04-23T18:47:34.145 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 0, 2024-04-23T18:47:34.151 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-23T18:47:34.158 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-23T18:47:34.165 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-04-23T18:47:34.173 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-04-23T18:47:34.182 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 0, 2024-04-23T18:47:34.191 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-23T18:47:34.199 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-23T18:47:34.257 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-23T18:47:34.267 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 1, 2024-04-23T18:47:34.275 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 0, 2024-04-23T18:47:34.285 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:47:34.303 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-23T18:47:34.362 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:47:34.383 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-23T18:47:34.389 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-23T18:47:34.398 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-23T18:47:34.406 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:47:34.415 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-23T18:47:34.423 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:47:34.432 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-23T18:47:34.440 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-23T18:47:34.449 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-23T18:47:34.458 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-23T18:47:34.467 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-23T18:47:34.520 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-23T18:47:34.529 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-23T18:47:34.538 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-23T18:47:34.547 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-23T18:47:34.556 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-23T18:47:34.582 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-23T18:47:34.590 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-23T18:47:34.597 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-23T18:47:34.604 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-23T18:47:34.612 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-23T18:47:34.621 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-23T18:47:34.629 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-23T18:47:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 142, 2024-04-23T18:47:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-23T18:47:34.655 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 55, 2024-04-23T18:47:34.664 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 11, 2024-04-23T18:47:34.691 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 11, 2024-04-23T18:47:34.760 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-04-23T18:47:34.767 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-23T18:47:34.774 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 0, 2024-04-23T18:47:34.782 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 2, 2024-04-23T18:47:34.791 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 0 2024-04-23T18:47:34.858 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:34.864 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-23T18:47:34.871 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 0, 2024-04-23T18:47:34.880 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:47:34.888 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-23T18:47:34.896 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-23T18:47:34.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:34.914 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-23T18:47:34.923 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-04-23T18:47:34.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:34.940 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-23T18:47:34.972 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-23T18:47:35.055 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T18:47:35.062 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-23T18:47:35.077 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-04-23T18:47:35.086 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-04-23T18:47:35.095 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-04-23T18:47:35.147 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-23T18:47:35.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:35.163 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:47:35.172 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:35.180 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-23T18:47:35.190 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:35.199 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 7831, 2024-04-23T18:47:35.209 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 697618697, 2024-04-23T18:47:35.222 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:35.230 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 7831, 2024-04-23T18:47:35.239 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-23T18:47:35.247 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-23T18:47:35.295 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 7831, 2024-04-23T18:47:35.327 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 697618697, 2024-04-23T18:47:35.337 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:35.346 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:35.354 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:35.403 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:35.411 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:35.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:35.429 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-23T18:47:35.436 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:35.445 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-23T18:47:35.455 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:35.464 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-23T18:47:35.473 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-23T18:47:35.482 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:35.491 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-23T18:47:35.542 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 7213, 2024-04-23T18:47:35.591 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 694309294, 2024-04-23T18:47:35.600 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 7213, 2024-04-23T18:47:35.609 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 694309294, 2024-04-23T18:47:35.617 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:35.626 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:35.635 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:35.684 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:35.694 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:35.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:35.712 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-23T18:47:35.721 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:35.730 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-23T18:47:35.739 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:35.748 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-23T18:47:35.756 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-23T18:47:35.764 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:35.786 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-23T18:47:35.842 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 7213, 2024-04-23T18:47:35.849 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 7213, 2024-04-23T18:47:35.856 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 7213, 2024-04-23T18:47:35.863 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 7213, 2024-04-23T18:47:35.870 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:35.920 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:35.927 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:35.935 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:35.941 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:35.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:35.956 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:47:35.963 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:35.970 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-23T18:47:35.980 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:35.989 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-23T18:47:36.016 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-23T18:47:36.023 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:36.072 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-23T18:47:36.079 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-23T18:47:36.089 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-23T18:47:36.098 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-23T18:47:36.106 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-23T18:47:36.167 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:36.173 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:36.182 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:36.190 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:36.197 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:36.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:36.210 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x55d7251506e0": { 2024-04-23T18:47:36.218 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:36.225 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-23T18:47:36.233 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:36.281 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-23T18:47:36.288 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-23T18:47:36.363 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:36.370 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-23T18:47:36.378 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-23T18:47:36.387 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-23T18:47:36.394 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-23T18:47:36.433 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-23T18:47:36.443 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:36.452 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:36.461 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:36.470 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:36.478 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:36.486 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:36.493 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-23T18:47:36.502 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-23T18:47:36.509 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-23T18:47:36.548 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-04-23T18:47:36.557 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-04-23T18:47:36.605 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-23T18:47:36.615 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:47:36.624 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:47:36.633 DEBUG:tasks.check_counter:Getting stats from h 2024-04-23T18:47:36.642 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-23T18:47:36.696 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:28 smithi177 ceph-mon[32308]: pgmap v923: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:47:36.703 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:30 smithi177 ceph-mon[32308]: pgmap v924: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:36.712 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:32 smithi177 ceph-mon[32308]: pgmap v925: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:36.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:34 smithi177 ceph-mon[32308]: pgmap v926: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:36.733 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:32 smithi188 ceph-mon[32694]: pgmap v925: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:36.742 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:34 smithi188 ceph-mon[32694]: pgmap v926: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:36.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:32 smithi050 ceph-mon[29187]: pgmap v925: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:36.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:34 smithi050 ceph-mon[29187]: pgmap v926: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:36.766 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:36 smithi050 ceph-mon[29187]: pgmap v927: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:36.776 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:36 smithi177 ceph-mon[32308]: pgmap v927: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:36.827 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:36 smithi188 ceph-mon[32694]: pgmap v927: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:36.982 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:47:38.419 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:38 smithi177 ceph-mon[32308]: pgmap v928: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:38.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:38 smithi050 ceph-mon[29187]: pgmap v928: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:38.825 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-23T18:47:38.834 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:47:38.842 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 951, 2024-04-23T18:47:38.968 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 951, 2024-04-23T18:47:38.978 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 46440956, 2024-04-23T18:47:38.987 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 301613, 2024-04-23T18:47:38.995 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 8, 2024-04-23T18:47:39.004 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-04-23T18:47:39.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.112147309, 2024-04-23T18:47:39.021 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.033716152, 2024-04-23T18:47:39.030 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.043275744, 2024-04-23T18:47:39.038 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.024983708, 2024-04-23T18:47:39.046 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:39.055 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 950, 2024-04-23T18:47:39.084 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.045557855, 2024-04-23T18:47:39.160 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000047955 2024-04-23T18:47:39.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:39.197 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:39.205 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:39.213 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:39.269 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:39.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:39.287 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-23T18:47:39.294 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T18:47:39.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:39.309 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:47:39.316 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 1353, 2024-04-23T18:47:39.323 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 1346, 2024-04-23T18:47:39.330 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 45139227, 2024-04-23T18:47:39.338 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 478059, 2024-04-23T18:47:39.376 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 5, 2024-04-23T18:47:39.383 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-04-23T18:47:39.446 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.124720057, 2024-04-23T18:47:39.452 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.030066567, 2024-04-23T18:47:39.463 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.054610685, 2024-04-23T18:47:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.035092145, 2024-04-23T18:47:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1345, 2024-04-23T18:47:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.044658748, 2024-04-23T18:47:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000033203 2024-04-23T18:47:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:39.562 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:39.569 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:39.576 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:39.583 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:39.590 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 247888, 2024-04-23T18:47:39.643 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 117072 2024-04-23T18:47:39.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:39.701 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:47:39.723 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 274, 2024-04-23T18:47:39.732 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 412, 2024-04-23T18:47:39.740 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 42013917, 2024-04-23T18:47:39.749 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 418287, 2024-04-23T18:47:39.818 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 6, 2024-04-23T18:47:39.825 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 5, 2024-04-23T18:47:39.832 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.051926153, 2024-04-23T18:47:39.841 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.010101574, 2024-04-23T18:47:39.850 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.032715996, 2024-04-23T18:47:39.859 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.006401069, 2024-04-23T18:47:39.903 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:39.911 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 410, 2024-04-23T18:47:39.920 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.020590362, 2024-04-23T18:47:39.929 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000050220 2024-04-23T18:47:39.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:39.996 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:40.039 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:40.049 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:40.058 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:40.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:40.075 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-23T18:47:40.116 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 331472 2024-04-23T18:47:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:40.132 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-23T18:47:40.141 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-23T18:47:40.150 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-23T18:47:40.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:40.167 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-23T18:47:40.175 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-23T18:47:40.183 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-23T18:47:40.191 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3902, 2024-04-23T18:47:40.217 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.144527862, 2024-04-23T18:47:40.226 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000037039 2024-04-23T18:47:40.292 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:40.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:40.309 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-23T18:47:40.318 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-23T18:47:40.326 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-23T18:47:40.374 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:40.384 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:40.393 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:40.404 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:40.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:40.422 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-23T18:47:40.431 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 0, 2024-04-23T18:47:40.440 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 0, 2024-04-23T18:47:40.449 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-23T18:47:40.458 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:40.489 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:40.497 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:40.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:40.570 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-23T18:47:40.579 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-04-23T18:47:40.587 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-04-23T18:47:40.595 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-04-23T18:47:40.657 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 0, 2024-04-23T18:47:40.666 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-04-23T18:47:40.674 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-04-23T18:47:40.681 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 6801, 2024-04-23T18:47:40.692 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 4195, 2024-04-23T18:47:40.699 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 2606, 2024-04-23T18:47:40.706 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-04-23T18:47:40.713 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 846, 2024-04-23T18:47:40.721 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 0, 2024-04-23T18:47:40.728 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 0, 2024-04-23T18:47:40.761 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 0, 2024-04-23T18:47:40.768 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 31, 2024-04-23T18:47:40.822 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 0, 2024-04-23T18:47:40.829 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 0, 2024-04-23T18:47:40.836 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-04-23T18:47:40.843 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 0, 2024-04-23T18:47:40.853 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:47:40.899 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-23T18:47:40.907 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-04-23T18:47:40.915 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-04-23T18:47:40.923 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-23T18:47:40.931 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-04-23T18:47:40.939 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-04-23T18:47:40.948 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-04-23T18:47:40.958 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-04-23T18:47:40.966 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-23T18:47:40.975 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:47:41.008 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-23T18:47:41.017 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 774, 2024-04-23T18:47:41.083 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 231532028, 2024-04-23T18:47:41.091 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 6, 2024-04-23T18:47:41.100 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:47:41.109 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-23T18:47:41.118 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:47:41.165 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:47:41.174 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:47:41.182 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:47:41.190 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:47:41.198 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-23T18:47:41.207 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-04-23T18:47:41.216 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T18:47:41.225 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-04-23T18:47:41.231 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:47:41.238 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:47:41.262 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:47:41.269 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-04-23T18:47:41.320 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-04-23T18:47:41.327 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-04-23T18:47:41.335 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-04-23T18:47:41.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:41.353 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-23T18:47:41.407 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-04-23T18:47:41.413 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 0, 2024-04-23T18:47:41.424 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 0, 2024-04-23T18:47:41.433 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 0, 2024-04-23T18:47:41.442 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-04-23T18:47:41.449 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-23T18:47:41.458 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-23T18:47:41.469 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:47:41.478 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-04-23T18:47:41.487 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-04-23T18:47:41.518 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-23T18:47:41.528 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-23T18:47:41.583 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-23T18:47:41.590 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:47:41.599 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:47:41.608 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-23T18:47:41.617 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-23T18:47:41.625 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 0, 2024-04-23T18:47:41.665 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 0, 2024-04-23T18:47:41.677 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:47:41.686 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-04-23T18:47:41.695 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-23T18:47:41.703 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-04-23T18:47:41.712 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-23T18:47:41.722 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-23T18:47:41.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:41.738 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-23T18:47:41.773 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-23T18:47:41.781 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 0, 2024-04-23T18:47:41.841 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-04-23T18:47:41.851 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-04-23T18:47:41.859 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 35844, 2024-04-23T18:47:41.867 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-23T18:47:41.878 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-04-23T18:47:41.932 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 0, 2024-04-23T18:47:41.941 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-04-23T18:47:41.950 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-04-23T18:47:41.959 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-04-23T18:47:41.967 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 17, 2024-04-23T18:47:41.976 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-23T18:47:41.985 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-04-23T18:47:41.993 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 680655713, 2024-04-23T18:47:42.001 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 686637953, 2024-04-23T18:47:42.011 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 686637841, 2024-04-23T18:47:42.019 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-23T18:47:42.053 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:42.114 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 35844 2024-04-23T18:47:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:42.193 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-23T18:47:42.201 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 1004, 2024-04-23T18:47:42.210 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 6683, 2024-04-23T18:47:42.218 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 5679, 2024-04-23T18:47:42.227 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 94, 2024-04-23T18:47:42.236 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 124, 2024-04-23T18:47:42.244 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 30, 2024-04-23T18:47:42.251 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 6801, 2024-04-23T18:47:42.260 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 6835, 2024-04-23T18:47:42.268 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 34, 2024-04-23T18:47:42.291 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 0, 2024-04-23T18:47:42.300 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 0, 2024-04-23T18:47:42.355 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-04-23T18:47:42.362 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 60692, 2024-04-23T18:47:42.369 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 215392 2024-04-23T18:47:42.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:42.384 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-23T18:47:42.392 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 0, 2024-04-23T18:47:42.432 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-04-23T18:47:42.479 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 0, 2024-04-23T18:47:42.486 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 0, 2024-04-23T18:47:42.493 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-04-23T18:47:42.502 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-23T18:47:42.511 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:42.531 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:42.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:42.572 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-23T18:47:42.580 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:42.628 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:42.635 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:42.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:42.649 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-23T18:47:42.655 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:42.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:42.728 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-23T18:47:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:42.745 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-23T18:47:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:42.810 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:42.817 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:42.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:42.876 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-23T18:47:42.883 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:42.889 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:42.897 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:42.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:42.967 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-23T18:47:42.974 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:42.981 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:42.988 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:42.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:43.001 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-23T18:47:43.009 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:43.016 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:43.026 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:43.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:43.066 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-23T18:47:43.146 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:43.165 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:43.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:43.183 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-23T18:47:43.248 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:43.267 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:43.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-23T18:47:43.294 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:43.311 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:43.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:43.330 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:40 smithi050 ceph-mon[29187]: pgmap v929: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:43.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:42 smithi050 ceph-mon[29187]: pgmap v930: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:43.370 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:38 smithi188 ceph-mon[32694]: pgmap v928: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:43.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:40 smithi188 ceph-mon[32694]: pgmap v929: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:43.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:42 smithi188 ceph-mon[32694]: pgmap v930: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:43.447 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-23T18:47:43.458 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:43.467 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:43.523 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:43.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:43.541 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-23T18:47:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:43.559 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:43.567 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:43.576 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:43.585 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-23T18:47:43.594 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:43.603 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:43.634 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:43.643 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:43.703 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-23T18:47:43.712 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:43.720 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:43.729 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:43.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:43.789 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-23T18:47:43.795 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:43.802 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:43.812 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:43.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:43.826 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-23T18:47:43.834 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:43.841 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:43.849 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:43.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:43.918 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-23T18:47:43.926 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:44.030 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:44.051 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:44.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:44.066 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-23T18:47:44.073 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:44.180 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:44.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:44.207 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-23T18:47:44.214 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:44.222 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:44.229 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:44.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:44.245 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-23T18:47:44.253 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:44.317 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:44.326 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:44.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:44.402 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-23T18:47:44.411 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:44.422 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:44.431 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:44.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:44.500 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-23T18:47:44.509 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:44.520 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:44.531 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:44.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:44.550 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-23T18:47:44.559 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:44.570 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:44.579 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:44.622 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:44.631 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-23T18:47:44.694 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:44.703 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:44.712 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:44.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:44.729 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-23T18:47:44.789 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:44.799 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:44.810 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:44.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:44.828 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-23T18:47:44.837 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:44.847 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:44.855 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:44.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:44.870 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-23T18:47:44.898 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:44.906 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:44.994 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:45.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:45.011 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-23T18:47:45.019 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:45.028 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:45.065 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:45.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:45.083 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:47:45.092 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:47:45.101 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-23T18:47:45.108 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:45.114 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:45.123 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:45.131 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:45.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:45.170 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-23T18:47:45.177 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 2, 2024-04-23T18:47:45.247 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 2, 2024-04-23T18:47:45.255 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 0, 2024-04-23T18:47:45.263 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 0, 2024-04-23T18:47:45.270 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-23T18:47:45.277 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 0, 2024-04-23T18:47:45.370 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 0, 2024-04-23T18:47:45.378 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 2898, 2024-04-23T18:47:45.385 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-23T18:47:45.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:45.408 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-23T18:47:45.413 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:47:45.419 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-23T18:47:45.425 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:47:45.432 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:47:45.437 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:47:45.442 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:47:45.476 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:47:45.540 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:47:45.547 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:47:45.556 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:47:45.563 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:47:45.571 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:47:45.635 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:47:45.642 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:47:45.648 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:47:45.655 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-23T18:47:45.662 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:47:45.671 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-23T18:47:45.679 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:47:45.688 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:47:45.698 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:47:45.710 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:47:45.747 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:47:45.756 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:47:45.819 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:47:45.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-23T18:47:45.837 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:47:45.846 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:47:45.855 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:47:45.865 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-23T18:47:45.914 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-23T18:47:45.922 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-23T18:47:45.930 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:47:45.939 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:47:45.947 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:47:45.956 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:47:45.965 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 278149, 2024-04-23T18:47:45.974 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 1358, 2024-04-23T18:47:45.981 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:47:46.010 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-23T18:47:46.019 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-23T18:47:46.099 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-23T18:47:46.107 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:47:46.116 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-23T18:47:46.125 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:47:46.133 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-23T18:47:46.172 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 8128, 2024-04-23T18:47:46.181 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 169, 2024-04-23T18:47:46.190 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:47:46.199 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:47:46.209 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-23T18:47:46.218 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-23T18:47:46.226 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 7525931, 2024-04-23T18:47:46.235 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 55787, 2024-04-23T18:47:46.245 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-23T18:47:46.254 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-23T18:47:46.298 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-23T18:47:46.307 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-23T18:47:46.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:46.385 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-23T18:47:46.395 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-23T18:47:46.404 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-23T18:47:46.415 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 2355, 2024-04-23T18:47:46.463 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 26, 2024-04-23T18:47:46.471 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 4, 2024-04-23T18:47:46.480 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 2355, 2024-04-23T18:47:46.489 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-23T18:47:46.498 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2355, 2024-04-23T18:47:46.506 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 2.182941033, 2024-04-23T18:47:46.515 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000926938 2024-04-23T18:47:46.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:46.533 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-23T18:47:46.542 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-23T18:47:46.579 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2355, 2024-04-23T18:47:46.587 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 2356 2024-04-23T18:47:46.663 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:46.669 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 2355, 2024-04-23T18:47:46.678 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 2355, 2024-04-23T18:47:46.687 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 0, 2024-04-23T18:47:46.698 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-23T18:47:46.747 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-23T18:47:46.756 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 1406, 2024-04-23T18:47:46.764 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 0, 2024-04-23T18:47:46.775 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 946, 2024-04-23T18:47:46.783 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 0, 2024-04-23T18:47:46.792 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 0, 2024-04-23T18:47:46.798 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-23T18:47:46.805 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-23T18:47:46.812 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-04-23T18:47:46.820 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-04-23T18:47:46.850 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 0, 2024-04-23T18:47:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-23T18:47:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-23T18:47:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-23T18:47:46.933 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 2, 2024-04-23T18:47:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 0, 2024-04-23T18:47:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:47:46.997 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-23T18:47:47.006 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:47:47.014 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-23T18:47:47.022 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-23T18:47:47.031 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-23T18:47:47.039 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:47:47.049 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-23T18:47:47.058 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:47:47.066 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-23T18:47:47.075 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-23T18:47:47.106 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-23T18:47:47.113 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-23T18:47:47.174 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-23T18:47:47.183 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-23T18:47:47.191 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-23T18:47:47.203 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-23T18:47:47.213 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-23T18:47:47.263 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-23T18:47:47.273 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-23T18:47:47.282 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-23T18:47:47.291 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-23T18:47:47.299 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-23T18:47:47.308 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-23T18:47:47.317 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-23T18:47:47.325 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-23T18:47:47.334 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 142, 2024-04-23T18:47:47.345 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-23T18:47:47.387 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 22, 2024-04-23T18:47:47.395 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 10, 2024-04-23T18:47:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 10, 2024-04-23T18:47:47.486 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-04-23T18:47:47.494 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-23T18:47:47.502 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 0, 2024-04-23T18:47:47.511 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 2, 2024-04-23T18:47:47.555 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 0 2024-04-23T18:47:47.565 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:47.572 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-23T18:47:47.581 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 0, 2024-04-23T18:47:47.589 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:47:47.595 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-23T18:47:47.602 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-23T18:47:47.609 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:47.616 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-23T18:47:47.624 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-04-23T18:47:47.631 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:47.639 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-23T18:47:47.700 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-23T18:47:47.709 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T18:47:47.719 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-23T18:47:47.727 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-04-23T18:47:47.736 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-04-23T18:47:47.797 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-04-23T18:47:47.802 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-23T18:47:47.810 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:47.844 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:47:47.855 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:47.863 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-23T18:47:47.872 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:47.880 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 2574, 2024-04-23T18:47:47.889 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 133348328, 2024-04-23T18:47:47.897 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:47.936 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 2574, 2024-04-23T18:47:47.947 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-23T18:47:48.032 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-23T18:47:48.041 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2574, 2024-04-23T18:47:48.050 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 133348328, 2024-04-23T18:47:48.060 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:48.072 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:48.120 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:48.128 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:48.135 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:48.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:48.150 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-23T18:47:48.157 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:48.165 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-23T18:47:48.171 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:48.178 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-23T18:47:48.185 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-23T18:47:48.216 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:48.223 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-23T18:47:48.285 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 2355, 2024-04-23T18:47:48.320 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 132683891, 2024-04-23T18:47:48.330 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2355, 2024-04-23T18:47:48.340 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 132683891, 2024-04-23T18:47:48.349 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:48.388 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:48.399 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:48.407 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:48.415 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:48.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:48.431 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-23T18:47:48.440 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:48.449 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-23T18:47:48.459 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:48.468 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-23T18:47:48.507 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-23T18:47:48.515 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:48.568 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-23T18:47:48.575 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 2355, 2024-04-23T18:47:48.583 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 2355, 2024-04-23T18:47:48.594 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2355, 2024-04-23T18:47:48.602 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 2355, 2024-04-23T18:47:48.638 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:48.647 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:48.655 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:48.664 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:48.671 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:48.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:48.688 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:47:48.696 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:48.703 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-23T18:47:48.711 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:48.744 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-23T18:47:48.753 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-23T18:47:48.818 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:48.825 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-23T18:47:48.833 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-23T18:47:48.840 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-23T18:47:48.848 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-23T18:47:48.896 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-23T18:47:48.904 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:48.910 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:48.918 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:48.926 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:48.935 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:48.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:48.953 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x55e8cb5646e0": { 2024-04-23T18:47:48.961 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:48.970 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-23T18:47:49.002 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:49.010 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-23T18:47:49.076 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-23T18:47:49.085 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-23T18:47:49.102 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-23T18:47:49.111 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-23T18:47:49.146 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-23T18:47:49.156 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-23T18:47:49.165 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:49.175 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:49.184 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:49.194 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:49.201 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:49.210 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:49.219 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-23T18:47:49.228 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-23T18:47:49.259 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-23T18:47:49.269 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-04-23T18:47:49.334 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-04-23T18:47:49.343 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-23T18:47:49.352 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:47:49.361 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:47:49.369 DEBUG:tasks.check_counter:Getting stats from k 2024-04-23T18:47:49.421 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-23T18:47:49.428 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:40 smithi177 ceph-mon[32308]: pgmap v929: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:49.437 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:42 smithi177 ceph-mon[32308]: pgmap v930: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:49.446 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:44 smithi177 ceph-mon[32308]: pgmap v931: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:49.455 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:46 smithi177 ceph-mon[32308]: pgmap v932: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:49.464 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:48 smithi177 ceph-mon[32308]: pgmap v933: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:49.473 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:44 smithi050 ceph-mon[29187]: pgmap v931: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:46 smithi050 ceph-mon[29187]: pgmap v932: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:49.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:48 smithi050 ceph-mon[29187]: pgmap v933: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:49.500 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:44 smithi188 ceph-mon[32694]: pgmap v931: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:49.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:46 smithi188 ceph-mon[32694]: pgmap v932: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:49.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:48 smithi188 ceph-mon[32694]: pgmap v933: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:49.718 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T18:47:50.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:50 smithi177 ceph-mon[32308]: pgmap v934: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:50.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:50 smithi050 ceph-mon[29187]: pgmap v934: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:50.827 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:50 smithi188 ceph-mon[32694]: pgmap v934: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:51.473 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-23T18:47:51.487 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:47:51.493 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 521, 2024-04-23T18:47:51.502 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 507, 2024-04-23T18:47:51.511 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 1815824, 2024-04-23T18:47:51.519 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 313536, 2024-04-23T18:47:51.528 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-04-23T18:47:51.536 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 0, 2024-04-23T18:47:51.544 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.082159768, 2024-04-23T18:47:51.553 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.028853663, 2024-04-23T18:47:51.563 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.046876953, 2024-04-23T18:47:51.605 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.011687907, 2024-04-23T18:47:51.613 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:51.682 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 507, 2024-04-23T18:47:51.722 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.022481790, 2024-04-23T18:47:51.733 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000044342 2024-04-23T18:47:51.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:51.749 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:51.795 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:51.804 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:51.812 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:51.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:51.828 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 1815824, 2024-04-23T18:47:51.837 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 313536 2024-04-23T18:47:51.846 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:51.855 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:47:51.864 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 3, 2024-04-23T18:47:51.872 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 387, 2024-04-23T18:47:51.900 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 480, 2024-04-23T18:47:51.908 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 302496, 2024-04-23T18:47:51.966 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 6, 2024-04-23T18:47:51.973 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-04-23T18:47:51.982 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.022749741, 2024-04-23T18:47:52.007 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.014821477, 2024-04-23T18:47:52.015 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.001760644, 2024-04-23T18:47:52.053 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:47:52.062 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:52.069 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 384, 2024-04-23T18:47:52.078 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.029811003, 2024-04-23T18:47:52.086 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000077632 2024-04-23T18:47:52.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:52.102 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:52.109 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:52.124 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:52.132 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:52.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:52.171 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-23T18:47:52.231 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 302496 2024-04-23T18:47:52.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:52.248 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:47:52.256 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 0, 2024-04-23T18:47:52.262 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 2, 2024-04-23T18:47:52.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 0, 2024-04-23T18:47:52.314 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 0, 2024-04-23T18:47:52.320 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 2, 2024-04-23T18:47:52.327 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 1, 2024-04-23T18:47:52.333 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.000256231, 2024-04-23T18:47:52.341 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T18:47:52.347 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.000172701, 2024-04-23T18:47:52.355 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:47:52.361 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:52.368 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:52.402 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:52.411 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:52.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:52.476 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:52.485 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:52.493 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:52.501 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:52.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:52.558 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:47:52.566 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:47:52.575 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:52.584 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-23T18:47:52.591 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 0, 2024-04-23T18:47:52.599 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-23T18:47:52.607 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:52.616 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-23T18:47:52.624 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:47:52.632 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-23T18:47:52.670 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:47:52.731 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:47:52.738 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:47:52.746 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:47:52.754 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:47:52.762 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:47:52.815 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:47:52.821 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:47:52.830 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:47:52.840 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:47:52.848 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:47:52.856 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:47:52.864 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:47:52.873 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-23T18:47:52.880 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:47:52.886 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-23T18:47:52.894 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:47:52.926 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:47:52.977 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:47:52.984 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:47:52.991 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:47:52.998 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:47:53.004 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:47:53.048 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-23T18:47:53.057 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:47:53.066 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:47:53.075 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:47:53.084 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-23T18:47:53.093 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-23T18:47:53.102 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-23T18:47:53.111 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:47:53.120 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:47:53.129 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:47:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:47:53.170 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 148704, 2024-04-23T18:47:53.217 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 29, 2024-04-23T18:47:53.242 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:47:53.250 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-23T18:47:53.260 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-23T18:47:53.268 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-23T18:47:53.315 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:47:53.324 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-23T18:47:53.333 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:47:53.343 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-23T18:47:53.353 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 0, 2024-04-23T18:47:53.360 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 0, 2024-04-23T18:47:53.368 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:47:53.375 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:47:53.382 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-23T18:47:53.390 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-23T18:47:53.434 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 928, 2024-04-23T18:47:53.443 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 2, 2024-04-23T18:47:53.543 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-23T18:47:53.551 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-23T18:47:53.560 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-23T18:47:53.569 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-23T18:47:53.578 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-23T18:47:53.623 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:47:53.631 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-23T18:47:53.640 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-23T18:47:53.648 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-23T18:47:53.657 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 520, 2024-04-23T18:47:53.666 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 1739775, 2024-04-23T18:47:53.675 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-23T18:47:53.684 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 520, 2024-04-23T18:47:53.691 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-23T18:47:53.700 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-23T18:47:53.745 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 520, 2024-04-23T18:47:53.753 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 1739775, 2024-04-23T18:47:53.813 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-23T18:47:53.822 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-23T18:47:53.831 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-23T18:47:53.840 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-23T18:47:53.849 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:53.895 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-23T18:47:53.905 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-23T18:47:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:52 smithi188 ceph-mon[32694]: pgmap v935: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:53.923 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:52 smithi050 ceph-mon[29187]: pgmap v935: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:53.932 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-23T18:47:53.939 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-23T18:47:53.945 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-04-23T18:47:53.953 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-04-23T18:47:53.960 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-23T18:47:53.968 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-23T18:47:53.976 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-23T18:47:54.000 DEBUG:tasks.check_counter:Getting stats from c 2024-04-23T18:47:54.058 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-23T18:47:54.065 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:52 smithi177 ceph-mon[32308]: pgmap v935: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:47:54.376 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:47:54.499 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:54 smithi188 ceph-mon[32694]: pgmap v936: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:54.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:54 smithi050 ceph-mon[29187]: pgmap v936: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:54.785 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:54 smithi177 ceph-mon[32308]: pgmap v936: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:47:56.197 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T18:47:56.206 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:47:56.215 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 446208, 2024-04-23T18:47:56.223 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 383960, 2024-04-23T18:47:56.232 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 138799521, 2024-04-23T18:47:56.242 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 231841972, 2024-04-23T18:47:56.250 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 9, 2024-04-23T18:47:56.259 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 4, 2024-04-23T18:47:56.268 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 22.307283513, 2024-04-23T18:47:56.323 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 9.613596442, 2024-04-23T18:47:56.331 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 15.662640271, 2024-04-23T18:47:56.426 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.746508500, 2024-04-23T18:47:56.436 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:56.445 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 383964, 2024-04-23T18:47:56.454 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 19.515133394, 2024-04-23T18:47:56.463 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000050825 2024-04-23T18:47:56.552 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:56.558 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:56.567 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 795809, 2024-04-23T18:47:56.575 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.120554081, 2024-04-23T18:47:56.584 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000151 2024-04-23T18:47:56.592 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:56.601 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 2071040, 2024-04-23T18:47:56.609 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 337360 2024-04-23T18:47:56.617 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:56.625 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:47:56.634 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 34719, 2024-04-23T18:47:56.663 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 34754, 2024-04-23T18:47:56.718 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 8783106, 2024-04-23T18:47:56.726 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 403795534, 2024-04-23T18:47:56.735 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 17, 2024-04-23T18:47:56.743 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 9, 2024-04-23T18:47:56.752 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 4.672935145, 2024-04-23T18:47:56.797 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 2.299253589, 2024-04-23T18:47:56.806 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.633107350, 2024-04-23T18:47:56.815 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 1.537591590, 2024-04-23T18:47:56.823 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:56.832 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 34751, 2024-04-23T18:47:56.841 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 2.804256570, 2024-04-23T18:47:56.850 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000080695 2024-04-23T18:47:56.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:56.864 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:56.871 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 35, 2024-04-23T18:47:56.901 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000004663, 2024-04-23T18:47:56.908 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000133 2024-04-23T18:47:56.966 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:56.973 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 4064, 2024-04-23T18:47:56.982 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 131792 2024-04-23T18:47:56.989 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:56.996 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:47:57.038 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 344640, 2024-04-23T18:47:57.046 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 477299, 2024-04-23T18:47:57.054 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 105363199, 2024-04-23T18:47:57.061 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 553800372, 2024-04-23T18:47:57.069 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 12, 2024-04-23T18:47:57.076 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 7, 2024-04-23T18:47:57.084 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 20.288165810, 2024-04-23T18:47:57.091 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 8.593153663, 2024-04-23T18:47:57.097 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 13.383473380, 2024-04-23T18:47:57.104 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 1.171345527, 2024-04-23T18:47:57.146 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:47:57.153 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 477293, 2024-04-23T18:47:57.207 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 28.598049733, 2024-04-23T18:47:57.214 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000059917 2024-04-23T18:47:57.221 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:57.227 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:47:57.233 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 711333, 2024-04-23T18:47:57.277 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.135693758, 2024-04-23T18:47:57.286 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000190 2024-04-23T18:47:57.293 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:57.299 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 1744, 2024-04-23T18:47:57.307 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 721776 2024-04-23T18:47:57.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:57.321 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2024-04-23T18:47:57.328 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2024-04-23T18:47:57.336 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2024-04-23T18:47:57.343 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:57.376 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2024-04-23T18:47:57.383 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-23T18:47:57.443 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-23T18:47:57.450 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 65202, 2024-04-23T18:47:57.456 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 113.137489645, 2024-04-23T18:47:57.465 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001735184 2024-04-23T18:47:57.474 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:47:57.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:57.531 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2024-04-23T18:47:57.539 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-23T18:47:57.546 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-23T18:47:57.553 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 41822, 2024-04-23T18:47:57.561 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 10.192640648, 2024-04-23T18:47:57.568 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000243714 2024-04-23T18:47:57.576 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:47:57.584 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:57.593 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2024-04-23T18:47:57.623 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 232101, 2024-04-23T18:47:57.631 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 197819, 2024-04-23T18:47:57.698 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2024-04-23T18:47:57.710 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 197819, 2024-04-23T18:47:57.719 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 1651.833081853, 2024-04-23T18:47:57.728 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.008350224 2024-04-23T18:47:57.738 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:57.777 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2024-04-23T18:47:57.785 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 34282, 2024-04-23T18:47:57.793 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 17, 2024-04-23T18:47:57.804 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2024-04-23T18:47:57.812 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 233, 2024-04-23T18:47:57.819 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 21, 2024-04-23T18:47:57.828 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 21, 2024-04-23T18:47:57.835 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 4675, 2024-04-23T18:47:57.843 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 2342, 2024-04-23T18:47:57.852 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 2318, 2024-04-23T18:47:57.876 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 15, 2024-04-23T18:47:57.884 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 1329, 2024-04-23T18:47:57.951 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 1193, 2024-04-23T18:47:57.961 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 147, 2024-04-23T18:47:57.971 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 147, 2024-04-23T18:47:57.980 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 33, 2024-04-23T18:47:57.989 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 358961, 2024-04-23T18:47:57.998 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 187944, 2024-04-23T18:47:58.006 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 34254, 2024-04-23T18:47:58.013 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 173, 2024-04-23T18:47:58.021 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:47:58.072 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2024-04-23T18:47:58.080 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 3018, 2024-04-23T18:47:58.089 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 0, 2024-04-23T18:47:58.099 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 0, 2024-04-23T18:47:58.108 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 24, 2024-04-23T18:47:58.117 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 890, 2024-04-23T18:47:58.147 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 20, 2024-04-23T18:47:58.155 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 1115, 2024-04-23T18:47:58.220 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2024-04-23T18:47:58.228 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:47:58.238 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2024-04-23T18:47:58.246 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 774, 2024-04-23T18:47:58.255 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 231532028, 2024-04-23T18:47:58.288 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 6, 2024-04-23T18:47:58.294 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:47:58.302 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2024-04-23T18:47:58.311 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:47:58.319 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:47:58.329 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:47:58.338 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:47:58.347 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:47:58.356 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 0, 2024-04-23T18:47:58.365 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 22, 2024-04-23T18:47:58.399 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 26287, 2024-04-23T18:47:58.405 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 110876, 2024-04-23T18:47:58.465 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:47:58.473 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:47:58.480 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:47:58.489 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 69596, 2024-04-23T18:47:58.497 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 22698, 2024-04-23T18:47:58.538 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 11039, 2024-04-23T18:47:58.547 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 150366 2024-04-23T18:47:58.556 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:58.566 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2024-04-23T18:47:58.574 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 247, 2024-04-23T18:47:58.583 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 1053, 2024-04-23T18:47:58.592 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 584, 2024-04-23T18:47:58.601 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 1018, 2024-04-23T18:47:58.610 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 680, 2024-04-23T18:47:58.619 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 0, 2024-04-23T18:47:58.627 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2024-04-23T18:47:58.657 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:47:58.725 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 25452, 2024-04-23T18:47:58.734 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 25452, 2024-04-23T18:47:58.743 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2024-04-23T18:47:58.752 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2024-04-23T18:47:58.761 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2024-04-23T18:47:58.813 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:47:58.833 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:47:58.842 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2024-04-23T18:47:58.851 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2024-04-23T18:47:58.859 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_path": 18, 2024-04-23T18:47:58.867 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_inode": 3204, 2024-04-23T18:47:58.874 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:47:58.882 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 28, 2024-04-23T18:47:58.890 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2024-04-23T18:47:58.899 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 42, 2024-04-23T18:47:58.942 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2024-04-23T18:47:58.951 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2024-04-23T18:47:59.019 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:59.026 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2024-04-23T18:47:59.034 INFO:teuthology.orchestra.run.smithi188.stdout: "evlrg": 0, 2024-04-23T18:47:59.043 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 150036, 2024-04-23T18:47:59.052 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 71494, 2024-04-23T18:47:59.104 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 61451, 2024-04-23T18:47:59.112 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 88585, 2024-04-23T18:47:59.119 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2024-04-23T18:47:59.128 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 10043, 2024-04-23T18:47:59.136 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 224, 2024-04-23T18:47:59.145 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 96, 2024-04-23T18:47:59.152 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 81, 2024-04-23T18:47:59.159 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 143, 2024-04-23T18:47:59.167 INFO:teuthology.orchestra.run.smithi188.stdout: "segmjr": 8, 2024-04-23T18:47:59.174 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2024-04-23T18:47:59.202 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 15, 2024-04-23T18:47:59.211 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 150863246, 2024-04-23T18:47:59.278 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 698442440, 2024-04-23T18:47:59.295 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 0, 2024-04-23T18:47:59.305 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2024-04-23T18:47:59.314 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 36928, 2024-04-23T18:47:59.324 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 256.380515738, 2024-04-23T18:47:59.332 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.006942713 2024-04-23T18:47:59.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:59.463 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 0 2024-04-23T18:47:59.472 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:59.481 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2024-04-23T18:47:59.490 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 2463, 2024-04-23T18:47:59.498 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 28851, 2024-04-23T18:47:59.506 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 26388, 2024-04-23T18:47:59.515 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 285, 2024-04-23T18:47:59.524 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 949, 2024-04-23T18:47:59.567 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 664, 2024-04-23T18:47:59.575 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 4675, 2024-04-23T18:47:59.679 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 54338, 2024-04-23T18:47:59.687 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 49663, 2024-04-23T18:47:59.696 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 147, 2024-04-23T18:47:59.705 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 30445, 2024-04-23T18:47:59.713 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 30298, 2024-04-23T18:47:59.762 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 93996, 2024-04-23T18:47:59.771 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 223608 2024-04-23T18:47:59.779 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:47:59.790 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2024-04-23T18:47:59.799 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 319056, 2024-04-23T18:47:59.810 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 2521, 2024-04-23T18:47:59.819 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 232101, 2024-04-23T18:47:59.829 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 3716, 2024-04-23T18:47:59.838 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 14508, 2024-04-23T18:47:59.847 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2024-04-23T18:47:59.890 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 25469, 2024-04-23T18:47:59.899 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 449.070119397, 2024-04-23T18:47:59.989 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.017632027 2024-04-23T18:47:59.998 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.007 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2024-04-23T18:48:00.015 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 7418, 2024-04-23T18:48:00.023 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 103.114019074, 2024-04-23T18:48:00.062 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.013900514 2024-04-23T18:48:00.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.080 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2024-04-23T18:48:00.088 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:00.097 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:00.106 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:00.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.123 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2024-04-23T18:48:00.134 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:00.142 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:00.151 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:00.187 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.275 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2024-04-23T18:48:00.285 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 55832, 2024-04-23T18:48:00.294 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 194.821795161, 2024-04-23T18:48:00.305 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.003489428 2024-04-23T18:48:00.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.346 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2024-04-23T18:48:00.357 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:00.366 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:00.375 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:00.384 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.391 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2024-04-23T18:48:00.398 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:00.410 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:00.419 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:00.427 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.457 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2024-04-23T18:48:00.464 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:00.514 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:00.522 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:00.530 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.538 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2024-04-23T18:48:00.547 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:00.587 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:00.593 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:00.601 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.608 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2024-04-23T18:48:00.617 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:56 smithi177 ceph-mon[32308]: pgmap v937: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:00.624 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:47:58 smithi177 ceph-mon[32308]: pgmap v938: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:48:00.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:56 smithi050 ceph-mon[29187]: pgmap v937: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:00.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:47:58 smithi050 ceph-mon[29187]: pgmap v938: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:48:00.646 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:00.652 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:00.678 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:00.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.746 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2024-04-23T18:48:00.754 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:00.761 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:00.768 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:00.775 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.781 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2024-04-23T18:48:00.788 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:00.837 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:00.842 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:00.849 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.855 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2024-04-23T18:48:00.862 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:00.871 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:00.879 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:00.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:00.894 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2024-04-23T18:48:00.900 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:00.973 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:01.001 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:01.011 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.019 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2024-04-23T18:48:01.028 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:01.062 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:01.071 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:01.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.090 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2024-04-23T18:48:01.099 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 77877, 2024-04-23T18:48:01.108 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 333.309695765, 2024-04-23T18:48:01.116 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.004279950 2024-04-23T18:48:01.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.133 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2024-04-23T18:48:01.142 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 4918, 2024-04-23T18:48:01.183 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 77.853932579, 2024-04-23T18:48:01.192 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.015830405 2024-04-23T18:48:01.248 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.257 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2024-04-23T18:48:01.266 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:01.276 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:01.287 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:01.321 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.330 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2024-04-23T18:48:01.339 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:01.348 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:01.359 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:01.367 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.376 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2024-04-23T18:48:01.386 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:01.395 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:01.403 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:01.453 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.460 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2024-04-23T18:48:01.516 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:01.525 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:01.534 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:01.542 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.552 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2024-04-23T18:48:01.620 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:01.627 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:01.636 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:01.644 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.652 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2024-04-23T18:48:01.661 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 113, 2024-04-23T18:48:01.670 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 1.619115153, 2024-04-23T18:48:01.678 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.014328452 2024-04-23T18:48:01.687 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.696 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2024-04-23T18:48:01.725 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:01.734 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:01.791 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:01.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.811 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2024-04-23T18:48:01.819 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 740, 2024-04-23T18:48:01.828 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 2.750974999, 2024-04-23T18:48:01.870 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.003717533 2024-04-23T18:48:01.878 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.885 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2024-04-23T18:48:01.893 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:01.900 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:01.907 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:01.915 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:01.922 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2024-04-23T18:48:01.930 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:01.937 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:01.988 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:01.996 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:02.048 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2024-04-23T18:48:02.055 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:02.065 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:02.072 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:02.078 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:02.120 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2024-04-23T18:48:02.126 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 25452, 2024-04-23T18:48:02.134 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 489.293429725, 2024-04-23T18:48:02.141 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.019224164 2024-04-23T18:48:02.149 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:02.157 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:48:02.164 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:48:02.170 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2024-04-23T18:48:02.177 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:02.185 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:02.219 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:02.226 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:02.289 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:02.298 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2024-04-23T18:48:02.307 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 2, 2024-04-23T18:48:02.316 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 5, 2024-04-23T18:48:02.325 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 3, 2024-04-23T18:48:02.335 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 2, 2024-04-23T18:48:02.388 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2024-04-23T18:48:02.399 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 21608, 2024-04-23T18:48:02.411 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 10804, 2024-04-23T18:48:02.420 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 1270, 2024-04-23T18:48:02.427 INFO:teuthology.orchestra.run.smithi188.stdout: "mdthresh_evicted": 0 2024-04-23T18:48:02.434 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:02.441 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2024-04-23T18:48:02.448 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:48:02.454 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2024-04-23T18:48:02.489 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:48:02.500 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:48:02.564 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:48:02.573 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:48:02.581 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:48:02.590 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:48:02.599 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:48:02.628 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:48:02.636 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:48:02.643 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:48:02.650 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:48:02.659 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:48:02.670 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:48:02.679 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2024-04-23T18:48:02.690 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:48:02.701 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2024-04-23T18:48:02.709 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:48:02.744 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:48:02.753 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:48:02.811 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:48:02.819 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:48:02.827 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:48:02.835 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:48:02.845 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2024-04-23T18:48:02.896 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:48:02.906 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:48:02.916 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:48:02.925 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2024-04-23T18:48:02.935 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2024-04-23T18:48:02.944 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2024-04-23T18:48:02.954 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:48:02.964 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:48:02.974 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:48:02.984 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:48:03.007 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 334549, 2024-04-23T18:48:03.018 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 1139, 2024-04-23T18:48:03.075 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 88, 2024-04-23T18:48:03.083 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 1, 2024-04-23T18:48:03.092 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2024-04-23T18:48:03.103 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2024-04-23T18:48:03.112 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:48:03.145 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2024-04-23T18:48:03.153 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:48:03.161 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2024-04-23T18:48:03.170 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 8128, 2024-04-23T18:48:03.178 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 169, 2024-04-23T18:48:03.188 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:48:03.198 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:48:03.206 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2024-04-23T18:48:03.214 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2024-04-23T18:48:03.223 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 9932680, 2024-04-23T18:48:03.261 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 86785, 2024-04-23T18:48:03.270 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2024-04-23T18:48:03.341 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2024-04-23T18:48:03.350 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2024-04-23T18:48:03.358 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2024-04-23T18:48:03.366 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:03.376 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2024-04-23T18:48:03.384 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2024-04-23T18:48:03.419 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2024-04-23T18:48:03.428 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 66525, 2024-04-23T18:48:03.436 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 702326133, 2024-04-23T18:48:03.444 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 27, 2024-04-23T18:48:03.451 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 66525, 2024-04-23T18:48:03.462 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2024-04-23T18:48:03.470 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 66525, 2024-04-23T18:48:03.479 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 418.738703760, 2024-04-23T18:48:03.488 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.006294456 2024-04-23T18:48:03.525 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:03.531 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2024-04-23T18:48:03.588 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2024-04-23T18:48:03.598 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 66525, 2024-04-23T18:48:03.607 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 68225 2024-04-23T18:48:03.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:03.625 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 66525, 2024-04-23T18:48:03.661 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 1057, 2024-04-23T18:48:03.670 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 65468, 2024-04-23T18:48:03.678 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2024-04-23T18:48:03.687 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2024-04-23T18:48:03.696 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 32, 2024-04-23T18:48:03.705 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 21, 2024-04-23T18:48:03.713 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 1040, 2024-04-23T18:48:03.721 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 37968, 2024-04-23T18:48:03.727 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 592, 2024-04-23T18:48:03.735 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2024-04-23T18:48:03.743 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2024-04-23T18:48:03.782 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2024-04-23T18:48:03.838 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 21, 2024-04-23T18:48:03.847 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 25775, 2024-04-23T18:48:03.856 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2024-04-23T18:48:03.865 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2024-04-23T18:48:03.874 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2024-04-23T18:48:03.911 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 0, 2024-04-23T18:48:03.920 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 42, 2024-04-23T18:48:03.929 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:48:03.951 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2024-04-23T18:48:03.958 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:48:03.965 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2024-04-23T18:48:03.972 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2024-04-23T18:48:03.979 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2024-04-23T18:48:03.987 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:48:03.995 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2024-04-23T18:48:04.037 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:48:04.043 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2024-04-23T18:48:04.105 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2024-04-23T18:48:04.113 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2024-04-23T18:48:04.120 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2024-04-23T18:48:04.128 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2024-04-23T18:48:04.136 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2024-04-23T18:48:04.170 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2024-04-23T18:48:04.178 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2024-04-23T18:48:04.186 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2024-04-23T18:48:04.194 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2024-04-23T18:48:04.201 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2024-04-23T18:48:04.209 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2024-04-23T18:48:04.217 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2024-04-23T18:48:04.225 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2024-04-23T18:48:04.232 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2024-04-23T18:48:04.240 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2024-04-23T18:48:04.274 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2024-04-23T18:48:04.282 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 142, 2024-04-23T18:48:04.338 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2024-04-23T18:48:04.348 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 55, 2024-04-23T18:48:04.356 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 12, 2024-04-23T18:48:04.365 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 12, 2024-04-23T18:48:04.374 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 0, 2024-04-23T18:48:04.419 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2024-04-23T18:48:04.441 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 2066, 2024-04-23T18:48:04.448 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 34, 2024-04-23T18:48:04.457 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 634 2024-04-23T18:48:04.466 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:04.475 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2024-04-23T18:48:04.484 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 1, 2024-04-23T18:48:04.493 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:48:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2024-04-23T18:48:04.511 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2024-04-23T18:48:04.535 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:04.544 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2024-04-23T18:48:04.618 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 3825 2024-04-23T18:48:04.626 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:04.635 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2024-04-23T18:48:04.644 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2024-04-23T18:48:04.652 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 10, 2024-04-23T18:48:04.711 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2024-04-23T18:48:04.721 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 10, 2024-04-23T18:48:04.729 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed_ops": 25452, 2024-04-23T18:48:04.738 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 25452, 2024-04-23T18:48:04.747 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2024-04-23T18:48:04.756 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:04.766 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:48:04.775 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:04.784 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-23T18:48:04.795 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:04.803 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 825563, 2024-04-23T18:48:04.833 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 178608142, 2024-04-23T18:48:04.898 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:04.907 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 825563, 2024-04-23T18:48:04.916 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-23T18:48:04.926 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-23T18:48:04.934 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 825563, 2024-04-23T18:48:04.966 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 178608142, 2024-04-23T18:48:04.975 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:04.983 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:04.992 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:05.000 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:05.008 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:05.016 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:05.025 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2024-04-23T18:48:05.033 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:05.043 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-23T18:48:05.076 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:05.083 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-23T18:48:05.144 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-23T18:48:05.153 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:05.162 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-23T18:48:05.171 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 66525, 2024-04-23T18:48:05.180 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 704902801, 2024-04-23T18:48:05.227 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 66525, 2024-04-23T18:48:05.236 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 704902801, 2024-04-23T18:48:05.246 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:05.255 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:05.263 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:05.271 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:05.279 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:05.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:05.293 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2024-04-23T18:48:05.304 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:05.312 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2024-04-23T18:48:05.350 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:05.409 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-23T18:48:05.417 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-23T18:48:05.426 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:05.434 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-23T18:48:05.441 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 66525, 2024-04-23T18:48:05.477 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 66525, 2024-04-23T18:48:05.484 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 66525, 2024-04-23T18:48:05.490 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 66525, 2024-04-23T18:48:05.496 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:05.502 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:05.509 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:05.517 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:05.524 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:05.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:05.543 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:48:05.579 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:05.586 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-23T18:48:05.654 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:05.661 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 25452, 2024-04-23T18:48:05.669 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 2576940, 2024-04-23T18:48:05.684 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:05.723 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 25452, 2024-04-23T18:48:05.784 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-23T18:48:05.790 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-23T18:48:05.797 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 1040, 2024-04-23T18:48:05.804 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 2576940, 2024-04-23T18:48:05.813 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:05.820 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:05.827 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:05.836 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:05.845 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:05.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:05.885 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x55ab045fa500": { 2024-04-23T18:48:05.894 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:05.948 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-23T18:48:05.957 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:05.967 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 150036, 2024-04-23T18:48:05.978 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 694248136, 2024-04-23T18:48:05.987 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:06.061 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 150036, 2024-04-23T18:48:06.072 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-23T18:48:06.079 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-23T18:48:06.088 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 36928, 2024-04-23T18:48:06.096 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 694248136, 2024-04-23T18:48:06.105 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:06.113 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:06.122 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:06.131 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:06.140 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:06.148 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:06.175 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T18:48:06.240 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:56 smithi188 ceph-mon[32694]: pgmap v937: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:06.250 INFO:tasks.check_counter:Daemon mds.c mds.exported=24 2024-04-23T18:48:06.261 INFO:tasks.check_counter:Daemon mds.c mds.imported=20 2024-04-23T18:48:06.270 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=247 2024-04-23T18:48:06.279 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=1053 2024-04-23T18:48:06.319 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-23T18:48:06.328 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:48:06.338 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:48:06.347 DEBUG:tasks.check_counter:Getting stats from f 2024-04-23T18:48:06.356 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-23T18:48:06.365 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:47:58 smithi188 ceph-mon[32694]: pgmap v938: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:48:06.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:00 smithi188 ceph-mon[32694]: pgmap v939: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:06.384 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:02 smithi188 ceph-mon[32694]: pgmap v940: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:06.392 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:04 smithi188 ceph-mon[32694]: pgmap v941: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:48:06.402 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:00 smithi177 ceph-mon[32308]: pgmap v939: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:06.429 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:02 smithi177 ceph-mon[32308]: pgmap v940: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:06.437 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:04 smithi177 ceph-mon[32308]: pgmap v941: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:48:06.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:00 smithi050 ceph-mon[29187]: pgmap v939: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:06.513 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:02 smithi050 ceph-mon[29187]: pgmap v940: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:06.522 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:04 smithi050 ceph-mon[29187]: pgmap v941: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:48:06.654 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:48:06.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:06 smithi050 ceph-mon[29187]: pgmap v942: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:48:06.776 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:06 smithi188 ceph-mon[32694]: pgmap v942: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:48:06.785 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:06 smithi177 ceph-mon[32308]: pgmap v942: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:48:08.395 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T18:48:08.405 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:48:08.412 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 4709, 2024-04-23T18:48:08.419 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 4710, 2024-04-23T18:48:08.427 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 333832307, 2024-04-23T18:48:08.434 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 1497151, 2024-04-23T18:48:08.442 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 8, 2024-04-23T18:48:08.449 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 4, 2024-04-23T18:48:08.506 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.544202737, 2024-04-23T18:48:08.514 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.124098007, 2024-04-23T18:48:08.603 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.252537797, 2024-04-23T18:48:08.611 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.133440489, 2024-04-23T18:48:08.619 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:48:08.627 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 4708, 2024-04-23T18:48:08.636 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.164718047, 2024-04-23T18:48:08.668 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000034986 2024-04-23T18:48:08.677 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:08.685 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:48:08.694 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:08.701 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:08.709 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:08.720 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:08.728 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T18:48:08.736 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T18:48:08.743 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:08.775 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:48:08.784 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 1370, 2024-04-23T18:48:08.851 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 1756, 2024-04-23T18:48:08.861 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 247842560, 2024-04-23T18:48:08.871 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 1374688, 2024-04-23T18:48:08.884 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 10, 2024-04-23T18:48:08.893 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 7, 2024-04-23T18:48:08.935 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.261431710, 2024-04-23T18:48:08.947 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.040534480, 2024-04-23T18:48:08.956 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.174894215, 2024-04-23T18:48:08.969 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.033281401, 2024-04-23T18:48:08.979 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:48:08.989 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1753, 2024-04-23T18:48:08.998 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.066618631, 2024-04-23T18:48:09.008 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000038002 2024-04-23T18:48:09.021 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:09.029 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:48:09.068 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:09.078 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:09.119 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:09.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:09.143 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-23T18:48:09.155 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 939920 2024-04-23T18:48:09.167 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:09.210 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:48:09.223 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 1998, 2024-04-23T18:48:09.232 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 1938, 2024-04-23T18:48:09.242 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 275101673, 2024-04-23T18:48:09.253 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 760670, 2024-04-23T18:48:09.264 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 5, 2024-04-23T18:48:09.275 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 4, 2024-04-23T18:48:09.286 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.328199606, 2024-04-23T18:48:09.297 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.059132018, 2024-04-23T18:48:09.309 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.225909660, 2024-04-23T18:48:09.357 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.044001202, 2024-04-23T18:48:09.366 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:48:09.428 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1938, 2024-04-23T18:48:09.436 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.076111563, 2024-04-23T18:48:09.445 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000039273 2024-04-23T18:48:09.454 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:09.462 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:48:09.518 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:09.524 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:09.534 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:09.543 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:09.552 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 2061200, 2024-04-23T18:48:09.562 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 325328 2024-04-23T18:48:09.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:09.578 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2024-04-23T18:48:09.589 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2024-04-23T18:48:09.597 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2024-04-23T18:48:09.606 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:09.639 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2024-04-23T18:48:09.690 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-23T18:48:09.697 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-23T18:48:09.705 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 11947, 2024-04-23T18:48:09.712 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.341409952, 2024-04-23T18:48:09.720 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000028577 2024-04-23T18:48:09.755 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:09.762 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:09.770 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2024-04-23T18:48:09.777 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-23T18:48:09.784 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-23T18:48:09.794 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:09.801 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:09.808 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:09.815 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:09.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:09.860 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2024-04-23T18:48:09.870 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 0, 2024-04-23T18:48:09.936 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 0, 2024-04-23T18:48:09.947 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2024-04-23T18:48:09.955 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:09.965 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:09.974 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:10.009 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:10.019 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2024-04-23T18:48:10.027 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 0, 2024-04-23T18:48:10.034 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 0, 2024-04-23T18:48:10.045 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2024-04-23T18:48:10.053 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 0, 2024-04-23T18:48:10.064 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 0, 2024-04-23T18:48:10.071 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 0, 2024-04-23T18:48:10.080 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 35413, 2024-04-23T18:48:10.090 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 12945, 2024-04-23T18:48:10.123 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 22468, 2024-04-23T18:48:10.131 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 0, 2024-04-23T18:48:10.186 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 16921, 2024-04-23T18:48:10.194 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 0, 2024-04-23T18:48:10.203 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 0, 2024-04-23T18:48:10.213 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 0, 2024-04-23T18:48:10.221 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 12, 2024-04-23T18:48:10.268 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 0, 2024-04-23T18:48:10.278 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 0, 2024-04-23T18:48:10.287 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 0, 2024-04-23T18:48:10.295 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 0, 2024-04-23T18:48:10.305 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 0, 2024-04-23T18:48:10.313 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2024-04-23T18:48:10.320 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 0, 2024-04-23T18:48:10.329 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 0, 2024-04-23T18:48:10.335 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 0, 2024-04-23T18:48:10.344 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 0, 2024-04-23T18:48:10.379 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 0, 2024-04-23T18:48:10.386 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 0, 2024-04-23T18:48:10.436 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 0, 2024-04-23T18:48:10.444 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2024-04-23T18:48:10.453 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:48:10.462 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2024-04-23T18:48:10.471 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 0, 2024-04-23T18:48:10.505 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 0, 2024-04-23T18:48:10.514 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 0, 2024-04-23T18:48:10.523 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:48:10.532 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2024-04-23T18:48:10.541 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:48:10.550 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:48:10.559 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:48:10.569 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:48:10.596 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:48:10.605 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 0, 2024-04-23T18:48:10.634 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 0, 2024-04-23T18:48:10.644 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T18:48:10.692 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 0, 2024-04-23T18:48:10.701 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:48:10.711 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T18:48:10.721 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:48:10.732 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 0, 2024-04-23T18:48:10.771 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 0, 2024-04-23T18:48:10.781 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 0, 2024-04-23T18:48:10.790 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 0 2024-04-23T18:48:10.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:10.807 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2024-04-23T18:48:10.818 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 0, 2024-04-23T18:48:10.829 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 0, 2024-04-23T18:48:10.839 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 0, 2024-04-23T18:48:10.851 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 0, 2024-04-23T18:48:10.860 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 0, 2024-04-23T18:48:10.898 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 0, 2024-04-23T18:48:10.910 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2024-04-23T18:48:10.973 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:48:10.991 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 0, 2024-04-23T18:48:11.006 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 0, 2024-04-23T18:48:11.018 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2024-04-23T18:48:11.028 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2024-04-23T18:48:11.093 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2024-04-23T18:48:11.104 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:48:11.113 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:48:11.123 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2024-04-23T18:48:11.132 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2024-04-23T18:48:11.142 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_path": 0, 2024-04-23T18:48:11.151 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_inode": 0, 2024-04-23T18:48:11.163 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:48:11.173 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 0, 2024-04-23T18:48:11.189 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2024-04-23T18:48:11.229 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 0, 2024-04-23T18:48:11.240 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2024-04-23T18:48:11.311 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2024-04-23T18:48:11.320 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:11.328 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2024-04-23T18:48:11.337 INFO:teuthology.orchestra.run.smithi188.stdout: "evlrg": 0, 2024-04-23T18:48:11.344 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 0, 2024-04-23T18:48:11.401 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 0, 2024-04-23T18:48:11.409 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 0, 2024-04-23T18:48:11.418 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 184226, 2024-04-23T18:48:11.427 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2024-04-23T18:48:11.434 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 0, 2024-04-23T18:48:11.442 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 0, 2024-04-23T18:48:11.450 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 0, 2024-04-23T18:48:11.459 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 0, 2024-04-23T18:48:11.466 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 135, 2024-04-23T18:48:11.473 INFO:teuthology.orchestra.run.smithi188.stdout: "segmjr": 16, 2024-04-23T18:48:11.518 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2024-04-23T18:48:11.530 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 0, 2024-04-23T18:48:11.590 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 315062108, 2024-04-23T18:48:11.603 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 856694689, 2024-04-23T18:48:11.613 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 856694115, 2024-04-23T18:48:11.623 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2024-04-23T18:48:11.633 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:11.684 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:11.697 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:11.705 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:11.715 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 184226 2024-04-23T18:48:11.723 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:11.731 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2024-04-23T18:48:11.742 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 2709, 2024-04-23T18:48:11.751 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 35428, 2024-04-23T18:48:11.761 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 32719, 2024-04-23T18:48:11.770 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 130, 2024-04-23T18:48:11.823 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 265, 2024-04-23T18:48:11.833 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 135, 2024-04-23T18:48:11.886 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 35413, 2024-04-23T18:48:11.896 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 35722, 2024-04-23T18:48:11.910 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 309, 2024-04-23T18:48:11.918 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 0, 2024-04-23T18:48:11.926 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 0, 2024-04-23T18:48:11.967 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 0, 2024-04-23T18:48:11.975 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 92204, 2024-04-23T18:48:11.984 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 215416 2024-04-23T18:48:11.992 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:12.002 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2024-04-23T18:48:12.011 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 0, 2024-04-23T18:48:12.018 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 0, 2024-04-23T18:48:12.025 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 0, 2024-04-23T18:48:12.032 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 0, 2024-04-23T18:48:12.040 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 0, 2024-04-23T18:48:12.071 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2024-04-23T18:48:12.079 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:12.136 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:12.146 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:12.153 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:12.161 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2024-04-23T18:48:12.169 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:12.209 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:12.217 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:12.225 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:12.231 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2024-04-23T18:48:12.238 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:12.245 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:12.253 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:12.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:12.266 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2024-04-23T18:48:12.273 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:12.302 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:12.310 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:12.371 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:12.379 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2024-04-23T18:48:12.388 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:12.397 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:12.407 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:12.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:12.451 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2024-04-23T18:48:12.460 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:12.469 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:12.478 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:12.491 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:12.503 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2024-04-23T18:48:12.513 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:12.523 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:12.532 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:12.582 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:12.595 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2024-04-23T18:48:12.696 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:12.707 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:12.717 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:12.727 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:12.737 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2024-04-23T18:48:12.779 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:12.791 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:12.802 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:12.811 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:12.820 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2024-04-23T18:48:12.830 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:12.842 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:12.855 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:12.864 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:12.874 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2024-04-23T18:48:12.905 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:12.918 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:12.969 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:12.980 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:12.989 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2024-04-23T18:48:12.999 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.010 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.050 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:13.061 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:13.074 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2024-04-23T18:48:13.083 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.092 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.102 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:13.110 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:13.119 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2024-04-23T18:48:13.128 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.137 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.169 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:13.177 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:13.269 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2024-04-23T18:48:13.275 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.283 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.290 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:13.298 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:13.342 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2024-04-23T18:48:13.350 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.357 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.365 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:13.373 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:13.385 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2024-04-23T18:48:13.392 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.399 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.406 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:13.412 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:13.449 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2024-04-23T18:48:13.457 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.512 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.522 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:13.531 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:13.540 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2024-04-23T18:48:13.548 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.608 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.618 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:13.629 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:13.636 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2024-04-23T18:48:13.647 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.656 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.665 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:13.674 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:13.681 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2024-04-23T18:48:13.689 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.747 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.755 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:13.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:13.845 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2024-04-23T18:48:13.853 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.861 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.869 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:13.914 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:13.923 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2024-04-23T18:48:13.931 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.938 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.948 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:13.955 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:13.962 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2024-04-23T18:48:13.970 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:13.976 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:13.983 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:14.010 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:14.018 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2024-04-23T18:48:14.093 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:14.102 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:14.109 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:14.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:14.125 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2024-04-23T18:48:14.132 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:14.179 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:14.186 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:14.194 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:14.203 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2024-04-23T18:48:14.212 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:14.219 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:14.228 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:14.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:14.246 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2024-04-23T18:48:14.274 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:14.283 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:14.331 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:14.341 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:14.351 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2024-04-23T18:48:14.361 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:14.370 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:14.414 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:14.425 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:14.439 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:48:14.450 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:48:14.463 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2024-04-23T18:48:14.473 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:14.484 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:14.493 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:14.500 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:14.509 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:14.555 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2024-04-23T18:48:14.564 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 2, 2024-04-23T18:48:14.662 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 5, 2024-04-23T18:48:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 3, 2024-04-23T18:48:14.685 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 2, 2024-04-23T18:48:14.697 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2024-04-23T18:48:14.709 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 0, 2024-04-23T18:48:14.785 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 0, 2024-04-23T18:48:14.798 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 5254, 2024-04-23T18:48:14.811 INFO:teuthology.orchestra.run.smithi188.stdout: "mdthresh_evicted": 0 2024-04-23T18:48:14.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:14.834 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2024-04-23T18:48:14.842 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:48:14.853 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2024-04-23T18:48:14.861 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:48:14.871 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:48:14.880 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:48:14.935 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:48:14.944 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:48:15.055 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:48:15.062 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:48:15.071 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:48:15.079 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:48:15.088 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:48:15.130 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:48:15.139 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:48:15.149 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:48:15.158 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2024-04-23T18:48:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:48:15.173 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2024-04-23T18:48:15.179 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:48:15.186 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:48:15.193 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:48:15.200 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:48:15.238 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:48:15.245 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:48:15.303 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:48:15.311 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2024-04-23T18:48:15.318 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:48:15.325 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:48:15.332 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:48:15.366 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2024-04-23T18:48:15.372 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2024-04-23T18:48:15.379 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2024-04-23T18:48:15.387 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:48:15.394 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:48:15.403 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:48:15.410 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:48:15.418 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 359039, 2024-04-23T18:48:15.427 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 3273, 2024-04-23T18:48:15.435 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:48:15.472 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 0, 2024-04-23T18:48:15.482 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2024-04-23T18:48:15.580 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2024-04-23T18:48:15.593 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:48:15.607 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2024-04-23T18:48:15.615 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:48:15.622 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2024-04-23T18:48:15.663 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 8128, 2024-04-23T18:48:15.672 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 169, 2024-04-23T18:48:15.684 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:48:15.694 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:48:15.703 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2024-04-23T18:48:15.711 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2024-04-23T18:48:15.719 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 31955963, 2024-04-23T18:48:15.728 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 211657, 2024-04-23T18:48:15.738 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2024-04-23T18:48:15.747 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2024-04-23T18:48:15.788 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2024-04-23T18:48:15.795 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2024-04-23T18:48:15.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:15.869 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2024-04-23T18:48:15.881 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2024-04-23T18:48:15.891 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2024-04-23T18:48:15.899 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 7444, 2024-04-23T18:48:15.945 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 21, 2024-04-23T18:48:15.953 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 3, 2024-04-23T18:48:15.960 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 7444, 2024-04-23T18:48:15.968 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2024-04-23T18:48:15.974 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 7444, 2024-04-23T18:48:15.981 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 7.975860506, 2024-04-23T18:48:15.989 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001071448 2024-04-23T18:48:15.997 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:16.006 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2024-04-23T18:48:16.014 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2024-04-23T18:48:16.043 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 7444, 2024-04-23T18:48:16.050 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 7445 2024-04-23T18:48:16.121 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:16.131 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 7444, 2024-04-23T18:48:16.149 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 7444, 2024-04-23T18:48:16.158 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 0, 2024-04-23T18:48:16.167 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2024-04-23T18:48:16.205 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2024-04-23T18:48:16.213 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 4489, 2024-04-23T18:48:16.223 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 0, 2024-04-23T18:48:16.235 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 2953, 2024-04-23T18:48:16.243 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 0, 2024-04-23T18:48:16.252 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 0, 2024-04-23T18:48:16.262 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2024-04-23T18:48:16.274 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2024-04-23T18:48:16.284 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2024-04-23T18:48:16.294 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 0, 2024-04-23T18:48:16.324 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 0, 2024-04-23T18:48:16.335 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2024-04-23T18:48:16.384 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2024-04-23T18:48:16.396 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2024-04-23T18:48:16.405 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 1, 2024-04-23T18:48:16.414 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 0, 2024-04-23T18:48:16.423 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:48:16.463 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2024-04-23T18:48:16.476 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:48:16.486 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2024-04-23T18:48:16.495 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2024-04-23T18:48:16.505 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2024-04-23T18:48:16.517 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:48:16.527 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2024-04-23T18:48:16.536 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:48:16.546 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2024-04-23T18:48:16.555 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2024-04-23T18:48:16.588 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2024-04-23T18:48:16.597 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2024-04-23T18:48:16.652 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2024-04-23T18:48:16.660 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2024-04-23T18:48:16.669 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2024-04-23T18:48:16.681 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2024-04-23T18:48:16.690 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2024-04-23T18:48:16.721 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2024-04-23T18:48:16.730 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2024-04-23T18:48:16.739 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2024-04-23T18:48:16.748 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2024-04-23T18:48:16.757 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2024-04-23T18:48:16.766 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2024-04-23T18:48:16.775 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2024-04-23T18:48:16.787 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2024-04-23T18:48:16.796 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 142, 2024-04-23T18:48:16.805 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2024-04-23T18:48:16.835 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 55, 2024-04-23T18:48:16.844 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 11, 2024-04-23T18:48:16.901 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 11, 2024-04-23T18:48:16.907 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 0, 2024-04-23T18:48:16.918 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2024-04-23T18:48:16.926 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 0, 2024-04-23T18:48:16.933 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 2, 2024-04-23T18:48:16.971 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 0 2024-04-23T18:48:16.977 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:16.983 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2024-04-23T18:48:16.990 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 0, 2024-04-23T18:48:16.997 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:48:17.005 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2024-04-23T18:48:17.011 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2024-04-23T18:48:17.018 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:17.027 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2024-04-23T18:48:17.037 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 0 2024-04-23T18:48:17.075 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:17.084 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2024-04-23T18:48:17.145 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2024-04-23T18:48:17.153 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T18:48:17.160 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2024-04-23T18:48:17.170 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 0, 2024-04-23T18:48:17.179 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed_ops": 0, 2024-04-23T18:48:17.213 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 0, 2024-04-23T18:48:17.226 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2024-04-23T18:48:17.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:17.247 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:48:17.257 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:17.266 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-23T18:48:17.275 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:17.286 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 8073, 2024-04-23T18:48:17.297 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 856015225, 2024-04-23T18:48:17.307 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 8073, 2024-04-23T18:48:17.346 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-23T18:48:17.392 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-23T18:48:17.400 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 8073, 2024-04-23T18:48:17.410 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 856015225, 2024-04-23T18:48:17.421 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:17.430 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:17.471 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:17.479 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:17.488 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:17.498 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:17.505 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2024-04-23T18:48:17.512 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:17.520 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-23T18:48:17.531 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:17.539 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-23T18:48:17.549 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-23T18:48:17.584 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:17.591 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-23T18:48:17.651 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 7444, 2024-04-23T18:48:17.661 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 852661024, 2024-04-23T18:48:17.669 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 7444, 2024-04-23T18:48:17.677 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 852661024, 2024-04-23T18:48:17.685 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:17.719 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:17.727 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:17.735 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:17.745 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:17.755 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:17.765 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2024-04-23T18:48:17.775 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:17.785 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2024-04-23T18:48:17.795 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:17.804 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-23T18:48:17.839 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-23T18:48:17.849 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:17.918 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-23T18:48:17.927 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 7444, 2024-04-23T18:48:17.938 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 7444, 2024-04-23T18:48:17.946 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 7444, 2024-04-23T18:48:17.957 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 7444, 2024-04-23T18:48:17.994 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:18.003 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:18.013 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:18.026 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:18.041 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:18.053 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:18.067 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:48:18.075 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:18.085 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-23T18:48:18.098 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:18.170 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-23T18:48:18.179 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-23T18:48:18.267 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:18.288 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-23T18:48:18.297 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-23T18:48:18.307 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-23T18:48:18.319 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 0, 2024-04-23T18:48:18.354 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 0, 2024-04-23T18:48:18.363 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:18.373 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:18.382 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:18.391 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:18.400 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:18.409 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:18.422 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x56191080e6e0": { 2024-04-23T18:48:18.431 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:18.446 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-23T18:48:18.475 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:18.484 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-23T18:48:18.575 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-23T18:48:18.586 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:18.594 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-23T18:48:18.601 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-23T18:48:18.610 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-23T18:48:18.654 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 0, 2024-04-23T18:48:18.663 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 0, 2024-04-23T18:48:18.672 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:18.684 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:18.692 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:18.701 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:18.710 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:18.718 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:18.727 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T18:48:18.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:08 smithi177 ceph-mon[32308]: pgmap v943: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:48:18.780 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:10 smithi177 ceph-mon[32308]: pgmap v944: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:48:18.789 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:10 smithi177 ceph-mon[32308]: osdmap e143: 12 total, 12 up, 12 in 2024-04-23T18:48:18.911 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:12 smithi177 ceph-mon[32308]: pgmap v946: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-23T18:48:18.928 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:08 smithi050 ceph-mon[29187]: pgmap v943: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:48:18.936 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:10 smithi050 ceph-mon[29187]: pgmap v944: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:48:18.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:10 smithi050 ceph-mon[29187]: osdmap e143: 12 total, 12 up, 12 in 2024-04-23T18:48:18.959 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:12 smithi050 ceph-mon[29187]: pgmap v946: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-23T18:48:18.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:08 smithi188 ceph-mon[32694]: pgmap v943: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-23T18:48:19.010 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-23T18:48:19.019 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-23T18:48:19.029 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-23T18:48:19.039 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-04-23T18:48:19.047 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-23T18:48:19.061 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:48:19.071 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:48:19.080 DEBUG:tasks.check_counter:Getting stats from i 2024-04-23T18:48:19.091 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-23T18:48:19.120 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:10 smithi188 ceph-mon[32694]: pgmap v944: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-23T18:48:19.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:10 smithi188 ceph-mon[32694]: osdmap e143: 12 total, 12 up, 12 in 2024-04-23T18:48:19.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:12 smithi188 ceph-mon[32694]: pgmap v946: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-23T18:48:19.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:14 smithi188 ceph-mon[32694]: pgmap v947: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:48:19.217 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:16 smithi188 ceph-mon[32694]: pgmap v948: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:48:19.225 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:16 smithi188 ceph-mon[32694]: osdmap e144: 12 total, 12 up, 12 in 2024-04-23T18:48:19.232 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:18 smithi188 ceph-mon[32694]: pgmap v950: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:48:19.266 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:14 smithi177 ceph-mon[32308]: pgmap v947: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:48:19.274 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:16 smithi177 ceph-mon[32308]: pgmap v948: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:48:19.283 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:16 smithi177 ceph-mon[32308]: osdmap e144: 12 total, 12 up, 12 in 2024-04-23T18:48:19.294 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:18 smithi177 ceph-mon[32308]: pgmap v950: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:48:19.302 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:14 smithi050 ceph-mon[29187]: pgmap v947: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:48:19.313 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:16 smithi050 ceph-mon[29187]: pgmap v948: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:48:19.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:16 smithi050 ceph-mon[29187]: osdmap e144: 12 total, 12 up, 12 in 2024-04-23T18:48:19.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:18 smithi050 ceph-mon[29187]: pgmap v950: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T18:48:19.510 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:48:20.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:20 smithi188 ceph-mon[32694]: pgmap v951: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2024-04-23T18:48:20.691 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:20 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:48:20.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:20 smithi050 ceph-mon[29187]: pgmap v951: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2024-04-23T18:48:20.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:20 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:48:20.818 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:20 smithi177 ceph-mon[32308]: pgmap v951: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2024-04-23T18:48:20.827 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:20 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T18:48:21.177 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T18:48:21.235 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:48:21.245 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 170050, 2024-04-23T18:48:21.254 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 173785, 2024-04-23T18:48:21.262 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 46615114, 2024-04-23T18:48:21.271 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 391609820, 2024-04-23T18:48:21.280 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 14, 2024-04-23T18:48:21.288 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 5, 2024-04-23T18:48:21.299 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 12.103057258, 2024-04-23T18:48:21.307 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 5.680043708, 2024-04-23T18:48:21.316 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 6.067673711, 2024-04-23T18:48:21.348 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 1.270459606, 2024-04-23T18:48:21.357 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:48:21.417 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 173781, 2024-04-23T18:48:21.425 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 10.230759260, 2024-04-23T18:48:21.432 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000058871 2024-04-23T18:48:21.442 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:21.451 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:48:21.498 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 270473, 2024-04-23T18:48:21.504 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.051571207, 2024-04-23T18:48:21.513 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000190 2024-04-23T18:48:21.521 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:21.530 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-04-23T18:48:21.539 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 730032 2024-04-23T18:48:21.548 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:21.559 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:48:21.567 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 690341, 2024-04-23T18:48:21.576 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 827750, 2024-04-23T18:48:21.628 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 210300662, 2024-04-23T18:48:21.637 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 661885409, 2024-04-23T18:48:21.736 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 7, 2024-04-23T18:48:21.748 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 5, 2024-04-23T18:48:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 35.110104971, 2024-04-23T18:48:21.765 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 14.858249052, 2024-04-23T18:48:21.774 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 24.426312278, 2024-04-23T18:48:21.811 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 1.188994516, 2024-04-23T18:48:21.818 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:48:21.825 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 827753, 2024-04-23T18:48:21.834 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 41.188238875, 2024-04-23T18:48:21.842 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000049759 2024-04-23T18:48:21.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:21.859 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:48:21.868 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1400387, 2024-04-23T18:48:21.877 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.218102573, 2024-04-23T18:48:21.886 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000155 2024-04-23T18:48:21.893 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:21.900 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 2073616, 2024-04-23T18:48:21.949 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 331888 2024-04-23T18:48:21.957 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:21.964 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:48:21.973 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 34117, 2024-04-23T18:48:21.980 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 34203, 2024-04-23T18:48:22.020 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 11856966, 2024-04-23T18:48:22.027 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 335482449, 2024-04-23T18:48:22.034 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 17, 2024-04-23T18:48:22.040 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 11, 2024-04-23T18:48:22.047 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 3.896690336, 2024-04-23T18:48:22.054 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 1.683883510, 2024-04-23T18:48:22.061 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.601459060, 2024-04-23T18:48:22.067 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 1.425639556, 2024-04-23T18:48:22.074 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:48:22.081 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 34199, 2024-04-23T18:48:22.137 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 2.431507484, 2024-04-23T18:48:22.143 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000071098 2024-04-23T18:48:22.216 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:22.223 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:48:22.230 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 124, 2024-04-23T18:48:22.238 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000023108, 2024-04-23T18:48:22.245 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000186 2024-04-23T18:48:22.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:22.296 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 10352, 2024-04-23T18:48:22.305 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 151312 2024-04-23T18:48:22.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:22.323 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2024-04-23T18:48:22.332 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2024-04-23T18:48:22.340 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2024-04-23T18:48:22.349 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:22.358 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2024-04-23T18:48:22.367 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-23T18:48:22.399 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-23T18:48:22.407 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 93807, 2024-04-23T18:48:22.517 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 125.917610206, 2024-04-23T18:48:22.526 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001342305 2024-04-23T18:48:22.535 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:22.544 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:22.553 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2024-04-23T18:48:22.594 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-23T18:48:22.603 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-23T18:48:22.612 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 54416, 2024-04-23T18:48:22.621 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 9.938904138, 2024-04-23T18:48:22.658 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000182646 2024-04-23T18:48:22.667 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:22.676 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2024-04-23T18:48:22.693 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 258788, 2024-04-23T18:48:22.701 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 225189, 2024-04-23T18:48:22.710 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2024-04-23T18:48:22.742 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 225189, 2024-04-23T18:48:22.808 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 1772.662647106, 2024-04-23T18:48:22.816 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.007871888 2024-04-23T18:48:22.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:22.834 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2024-04-23T18:48:22.843 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 33597, 2024-04-23T18:48:22.878 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 4, 2024-04-23T18:48:22.886 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2024-04-23T18:48:22.895 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 228, 2024-04-23T18:48:22.903 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 20, 2024-04-23T18:48:22.911 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 19, 2024-04-23T18:48:22.920 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 4745, 2024-04-23T18:48:22.928 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 2398, 2024-04-23T18:48:22.936 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 2337, 2024-04-23T18:48:22.947 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 10, 2024-04-23T18:48:22.955 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 1297, 2024-04-23T18:48:22.984 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 604, 2024-04-23T18:48:22.993 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 86, 2024-04-23T18:48:23.075 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 86, 2024-04-23T18:48:23.084 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 17, 2024-04-23T18:48:23.093 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 458561, 2024-04-23T18:48:23.102 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 239746, 2024-04-23T18:48:23.110 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 33586, 2024-04-23T18:48:23.154 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 259, 2024-04-23T18:48:23.162 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 1, 2024-04-23T18:48:23.170 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2024-04-23T18:48:23.179 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 6814, 2024-04-23T18:48:23.187 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 0, 2024-04-23T18:48:23.195 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 0, 2024-04-23T18:48:23.203 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 8, 2024-04-23T18:48:23.210 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 298, 2024-04-23T18:48:23.216 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 18, 2024-04-23T18:48:23.223 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 1106, 2024-04-23T18:48:23.248 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2024-04-23T18:48:23.256 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 0, 2024-04-23T18:48:23.320 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2024-04-23T18:48:23.327 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 774, 2024-04-23T18:48:23.334 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 231532028, 2024-04-23T18:48:23.341 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 6, 2024-04-23T18:48:23.348 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T18:48:23.393 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2024-04-23T18:48:23.401 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T18:48:23.408 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2024-04-23T18:48:23.415 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 0, 2024-04-23T18:48:23.422 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T18:48:23.429 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T18:48:23.437 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 0, 2024-04-23T18:48:23.443 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 23, 2024-04-23T18:48:23.451 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 17136, 2024-04-23T18:48:23.457 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 126429, 2024-04-23T18:48:23.487 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T18:48:23.494 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 1, 2024-04-23T18:48:23.580 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T18:48:23.587 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 69798, 2024-04-23T18:48:23.594 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 31044, 2024-04-23T18:48:23.600 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 13108, 2024-04-23T18:48:23.607 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 180581 2024-04-23T18:48:23.645 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:23.657 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2024-04-23T18:48:23.664 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 394, 2024-04-23T18:48:23.671 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 1406, 2024-04-23T18:48:23.681 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 758, 2024-04-23T18:48:23.688 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 1463, 2024-04-23T18:48:23.695 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 528, 2024-04-23T18:48:23.702 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 0, 2024-04-23T18:48:23.709 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2024-04-23T18:48:23.716 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2024-04-23T18:48:23.751 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 32414, 2024-04-23T18:48:23.760 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 32414, 2024-04-23T18:48:23.825 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2024-04-23T18:48:23.833 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2024-04-23T18:48:23.841 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2024-04-23T18:48:23.849 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2024-04-23T18:48:23.856 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2024-04-23T18:48:23.912 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2024-04-23T18:48:23.920 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2024-04-23T18:48:23.929 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_path": 18, 2024-04-23T18:48:23.940 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_inode": 2948, 2024-04-23T18:48:23.949 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T18:48:23.959 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 9, 2024-04-23T18:48:23.968 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2024-04-23T18:48:23.977 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 39, 2024-04-23T18:48:23.985 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2024-04-23T18:48:23.994 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2024-04-23T18:48:24.031 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:24.040 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2024-04-23T18:48:24.111 INFO:teuthology.orchestra.run.smithi188.stdout: "evlrg": 0, 2024-04-23T18:48:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 184226, 2024-04-23T18:48:24.130 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 115356, 2024-04-23T18:48:24.140 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 110611, 2024-04-23T18:48:24.149 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 73615, 2024-04-23T18:48:24.192 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2024-04-23T18:48:24.201 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 4745, 2024-04-23T18:48:24.209 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 279, 2024-04-23T18:48:24.217 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 151, 2024-04-23T18:48:24.225 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 144, 2024-04-23T18:48:24.234 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 135, 2024-04-23T18:48:24.242 INFO:teuthology.orchestra.run.smithi188.stdout: "segmjr": 6, 2024-04-23T18:48:24.250 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2024-04-23T18:48:24.258 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 7, 2024-04-23T18:48:24.266 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 315062108, 2024-04-23T18:48:24.295 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 856694689, 2024-04-23T18:48:24.303 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 0, 2024-04-23T18:48:24.357 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2024-04-23T18:48:24.366 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 50995, 2024-04-23T18:48:24.374 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 265.890775552, 2024-04-23T18:48:24.383 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.005214055 2024-04-23T18:48:24.391 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:24.428 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 0 2024-04-23T18:48:24.437 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:24.448 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2024-04-23T18:48:24.455 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 2793, 2024-04-23T18:48:24.463 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 35642, 2024-04-23T18:48:24.469 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 32849, 2024-04-23T18:48:24.479 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 269, 2024-04-23T18:48:24.486 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 1083, 2024-04-23T18:48:24.494 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 814, 2024-04-23T18:48:24.502 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 4745, 2024-04-23T18:48:24.533 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 60907, 2024-04-23T18:48:24.541 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 56162, 2024-04-23T18:48:24.590 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 86, 2024-04-23T18:48:24.597 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 36012, 2024-04-23T18:48:24.605 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 35926, 2024-04-23T18:48:24.612 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 91856, 2024-04-23T18:48:24.619 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 223608 2024-04-23T18:48:24.684 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:24.699 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2024-04-23T18:48:24.705 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 396168, 2024-04-23T18:48:24.712 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 3022, 2024-04-23T18:48:24.718 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 258788, 2024-04-23T18:48:24.726 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 6198, 2024-04-23T18:48:24.733 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 32872, 2024-04-23T18:48:24.739 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2024-04-23T18:48:24.746 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 32436, 2024-04-23T18:48:24.753 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 586.538177512, 2024-04-23T18:48:24.778 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.018082938 2024-04-23T18:48:24.786 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:24.832 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2024-04-23T18:48:24.839 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 8540, 2024-04-23T18:48:24.846 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 111.753132778, 2024-04-23T18:48:24.853 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.013085846 2024-04-23T18:48:24.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:24.866 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2024-04-23T18:48:24.873 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:24.962 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:24.969 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:24.975 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:24.982 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2024-04-23T18:48:24.988 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:24.996 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:25.002 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:25.008 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.034 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2024-04-23T18:48:25.041 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 66734, 2024-04-23T18:48:25.098 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 184.850901428, 2024-04-23T18:48:25.105 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.002769965 2024-04-23T18:48:25.114 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.123 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2024-04-23T18:48:25.138 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:25.178 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:25.186 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:25.195 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.204 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2024-04-23T18:48:25.212 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:25.221 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:25.230 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:25.239 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.248 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2024-04-23T18:48:25.257 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:25.290 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:25.298 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:25.368 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.376 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2024-04-23T18:48:25.384 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:25.393 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:25.408 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:25.450 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.459 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2024-04-23T18:48:25.469 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:25.477 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:25.487 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:25.496 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.505 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2024-04-23T18:48:25.514 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:25.523 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:25.532 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:25.560 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.569 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2024-04-23T18:48:25.606 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:25.631 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:25.639 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:25.648 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.655 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2024-04-23T18:48:25.692 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:25.701 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:25.710 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:25.718 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.727 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2024-04-23T18:48:25.735 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:25.745 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:25.754 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:25.763 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.771 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2024-04-23T18:48:25.808 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:25.818 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:25.883 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:25.891 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.899 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2024-04-23T18:48:25.907 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 74715, 2024-04-23T18:48:25.915 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 321.284966891, 2024-04-23T18:48:25.923 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.004300140 2024-04-23T18:48:25.974 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:25.983 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2024-04-23T18:48:25.990 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 8760, 2024-04-23T18:48:25.998 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 145.739349999, 2024-04-23T18:48:26.007 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.016636912 2024-04-23T18:48:26.015 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.027 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2024-04-23T18:48:26.035 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:26.043 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:26.072 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:26.079 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.137 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2024-04-23T18:48:26.144 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:26.150 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:26.157 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:26.163 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.215 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2024-04-23T18:48:26.222 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:26.229 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:26.238 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:26.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.251 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2024-04-23T18:48:26.258 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:26.264 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:26.270 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:26.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.318 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2024-04-23T18:48:26.325 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:26.373 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:26.380 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:26.388 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.395 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2024-04-23T18:48:26.403 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 142, 2024-04-23T18:48:26.439 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 44.607840507, 2024-04-23T18:48:26.448 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.314139721 2024-04-23T18:48:26.457 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.466 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2024-04-23T18:48:26.475 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:26.484 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:26.493 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:26.502 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.509 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2024-04-23T18:48:26.517 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1448, 2024-04-23T18:48:26.548 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 2.087894030, 2024-04-23T18:48:26.556 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001441915 2024-04-23T18:48:26.615 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.624 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2024-04-23T18:48:26.632 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:26.641 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:26.650 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:26.683 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.692 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2024-04-23T18:48:26.700 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:26.709 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:26.719 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:26.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.736 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2024-04-23T18:48:26.745 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:26.752 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:26.760 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:26.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.800 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2024-04-23T18:48:26.861 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 32414, 2024-04-23T18:48:26.869 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 375.800383961, 2024-04-23T18:48:26.877 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.011593767 2024-04-23T18:48:26.886 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:26.895 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2024-04-23T18:48:26.949 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2024-04-23T18:48:26.958 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2024-04-23T18:48:26.968 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:26.977 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:26.986 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:26.995 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:27.004 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:27.014 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2024-04-23T18:48:27.022 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 2, 2024-04-23T18:48:27.031 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 5, 2024-04-23T18:48:27.063 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 3, 2024-04-23T18:48:27.072 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 2, 2024-04-23T18:48:27.123 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2024-04-23T18:48:27.132 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 22183, 2024-04-23T18:48:27.142 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 11091, 2024-04-23T18:48:27.151 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 1266, 2024-04-23T18:48:27.159 INFO:teuthology.orchestra.run.smithi188.stdout: "mdthresh_evicted": 0 2024-04-23T18:48:27.191 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:27.200 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2024-04-23T18:48:27.209 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:48:27.218 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2024-04-23T18:48:27.227 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:48:27.238 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:48:27.247 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:48:27.254 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:48:27.263 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:48:27.271 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:48:27.302 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:48:27.310 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:48:27.364 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:48:27.373 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:48:27.381 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:48:27.390 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:48:27.398 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:48:27.457 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2024-04-23T18:48:27.463 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:48:27.471 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2024-04-23T18:48:27.479 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:48:27.487 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:48:27.495 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:48:27.505 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:48:27.513 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:48:27.522 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:48:27.531 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:48:27.580 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2024-04-23T18:48:27.589 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:48:27.630 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:48:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:48:27.672 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2024-04-23T18:48:27.681 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2024-04-23T18:48:27.691 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2024-04-23T18:48:27.725 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:48:27.733 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:48:27.742 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:48:27.751 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:48:27.758 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 335245, 2024-04-23T18:48:27.766 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 1365, 2024-04-23T18:48:27.774 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 88, 2024-04-23T18:48:27.782 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 1, 2024-04-23T18:48:27.790 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2024-04-23T18:48:27.797 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2024-04-23T18:48:27.826 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:48:27.835 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2024-04-23T18:48:27.922 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:48:27.931 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2024-04-23T18:48:27.940 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 8128, 2024-04-23T18:48:27.949 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 169, 2024-04-23T18:48:27.958 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:48:27.991 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:48:27.998 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2024-04-23T18:48:28.007 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2024-04-23T18:48:28.016 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 10711035, 2024-04-23T18:48:28.024 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 90501, 2024-04-23T18:48:28.033 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2024-04-23T18:48:28.042 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2024-04-23T18:48:28.050 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2024-04-23T18:48:28.058 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2024-04-23T18:48:28.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:28.098 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2024-04-23T18:48:28.107 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2024-04-23T18:48:28.161 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2024-04-23T18:48:28.168 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 87919, 2024-04-23T18:48:28.175 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 861564147, 2024-04-23T18:48:28.182 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 32, 2024-04-23T18:48:28.189 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 87919, 2024-04-23T18:48:28.197 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2024-04-23T18:48:28.266 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 87919, 2024-04-23T18:48:28.274 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 487.648471293, 2024-04-23T18:48:28.281 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.005546565 2024-04-23T18:48:28.287 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:28.294 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2024-04-23T18:48:28.301 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2024-04-23T18:48:28.309 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 87919, 2024-04-23T18:48:28.315 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 89654 2024-04-23T18:48:28.322 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:28.340 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 87919, 2024-04-23T18:48:28.347 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 1179, 2024-04-23T18:48:28.461 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 86740, 2024-04-23T18:48:28.468 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2024-04-23T18:48:28.478 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2024-04-23T18:48:28.484 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 38, 2024-04-23T18:48:28.491 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 23, 2024-04-23T18:48:28.524 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 1175, 2024-04-23T18:48:28.531 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 52170, 2024-04-23T18:48:28.538 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 612, 2024-04-23T18:48:28.545 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2024-04-23T18:48:28.553 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2024-04-23T18:48:28.560 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2024-04-23T18:48:28.566 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 20, 2024-04-23T18:48:28.575 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 32809, 2024-04-23T18:48:28.583 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2024-04-23T18:48:28.592 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2024-04-23T18:48:28.629 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2024-04-23T18:48:28.636 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 1, 2024-04-23T18:48:28.696 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 46, 2024-04-23T18:48:28.705 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2024-04-23T18:48:28.714 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2024-04-23T18:48:28.723 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2024-04-23T18:48:28.732 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2024-04-23T18:48:28.765 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2024-04-23T18:48:28.774 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2024-04-23T18:48:28.782 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T18:48:28.791 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2024-04-23T18:48:28.800 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2024-04-23T18:48:28.809 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2024-04-23T18:48:28.818 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2024-04-23T18:48:28.826 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2024-04-23T18:48:28.836 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2024-04-23T18:48:28.845 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2024-04-23T18:48:28.876 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2024-04-23T18:48:28.884 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2024-04-23T18:48:28.947 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2024-04-23T18:48:28.956 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2024-04-23T18:48:28.965 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2024-04-23T18:48:28.974 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2024-04-23T18:48:28.982 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2024-04-23T18:48:29.022 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2024-04-23T18:48:29.031 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2024-04-23T18:48:29.041 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2024-04-23T18:48:29.048 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2024-04-23T18:48:29.057 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2024-04-23T18:48:29.065 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 144, 2024-04-23T18:48:29.074 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2024-04-23T18:48:29.083 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 57, 2024-04-23T18:48:29.091 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 12, 2024-04-23T18:48:29.099 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 12, 2024-04-23T18:48:29.133 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 0, 2024-04-23T18:48:29.142 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2024-04-23T18:48:29.211 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 2087, 2024-04-23T18:48:29.219 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 8, 2024-04-23T18:48:29.228 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 665 2024-04-23T18:48:29.237 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:29.246 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2024-04-23T18:48:29.282 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 1, 2024-04-23T18:48:29.290 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2024-04-23T18:48:29.299 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2024-04-23T18:48:29.308 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2024-04-23T18:48:29.317 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:29.326 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2024-04-23T18:48:29.334 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 3628 2024-04-23T18:48:29.344 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:29.352 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2024-04-23T18:48:29.362 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2024-04-23T18:48:29.405 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 10, 2024-04-23T18:48:29.414 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2024-04-23T18:48:29.473 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 10, 2024-04-23T18:48:29.495 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed_ops": 32414, 2024-04-23T18:48:29.502 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 32414, 2024-04-23T18:48:29.510 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2024-04-23T18:48:29.517 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:29.558 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:48:29.565 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:29.572 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-23T18:48:29.579 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:29.587 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 894504, 2024-04-23T18:48:29.594 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 188229432, 2024-04-23T18:48:29.601 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:29.609 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 894504, 2024-04-23T18:48:29.616 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-23T18:48:29.623 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-23T18:48:29.652 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 894504, 2024-04-23T18:48:29.660 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 188229432, 2024-04-23T18:48:29.740 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:29.746 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:29.753 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:29.760 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:29.767 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:29.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:29.806 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2024-04-23T18:48:29.813 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:29.820 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-23T18:48:29.826 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:29.834 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-23T18:48:29.841 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-23T18:48:29.849 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:29.856 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-23T18:48:29.863 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 87919, 2024-04-23T18:48:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 864842761, 2024-04-23T18:48:29.906 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 87919, 2024-04-23T18:48:29.987 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 864842761, 2024-04-23T18:48:29.996 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:30.006 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:30.016 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:30.025 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:30.057 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:30.066 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:30.075 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2024-04-23T18:48:30.084 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:30.093 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2024-04-23T18:48:30.129 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:30.139 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-23T18:48:30.147 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-23T18:48:30.156 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:30.165 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-23T18:48:30.198 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 87919, 2024-04-23T18:48:30.207 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 87919, 2024-04-23T18:48:30.263 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 87919, 2024-04-23T18:48:30.271 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 87919, 2024-04-23T18:48:30.280 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:30.286 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:30.295 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:30.303 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:30.341 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:30.351 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:30.359 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2024-04-23T18:48:30.367 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:30.375 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-23T18:48:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:30.392 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 32414, 2024-04-23T18:48:30.402 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 3278678, 2024-04-23T18:48:30.412 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:30.445 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 32414, 2024-04-23T18:48:30.454 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-23T18:48:30.507 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-23T18:48:30.517 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 1175, 2024-04-23T18:48:30.527 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 3278678, 2024-04-23T18:48:30.537 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:30.545 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:30.581 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:30.590 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:30.599 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:30.608 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:30.616 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x5620e43f6500": { 2024-04-23T18:48:30.628 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:30.637 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-23T18:48:30.646 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:30.655 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 184226, 2024-04-23T18:48:30.664 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 852500385, 2024-04-23T18:48:30.700 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:30.708 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 184226, 2024-04-23T18:48:30.763 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-23T18:48:30.772 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-23T18:48:30.780 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 50995, 2024-04-23T18:48:30.789 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 852500385, 2024-04-23T18:48:30.798 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:30.832 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:30.840 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:30.849 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:30.858 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:30.867 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:30.876 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T18:48:30.885 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:21 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:48:30.894 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:21 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:48:30.903 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:21 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:48:30.911 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:22 smithi177 ceph-mon[32308]: pgmap v952: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:48:30.938 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:24 smithi177 ceph-mon[32308]: pgmap v953: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:48:30.948 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:21 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:48:30.997 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:21 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:48:31.005 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:21 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:48:31.014 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:22 smithi050 ceph-mon[29187]: pgmap v952: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:48:31.023 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:24 smithi050 ceph-mon[29187]: pgmap v953: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:48:31.034 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:21 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T18:48:31.082 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:21 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T18:48:31.093 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:21 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' 2024-04-23T18:48:31.103 INFO:tasks.check_counter:Daemon mds.i mds.exported=8 2024-04-23T18:48:31.109 INFO:tasks.check_counter:Daemon mds.i mds.imported=18 2024-04-23T18:48:31.117 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=394 2024-04-23T18:48:31.124 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=1406 2024-04-23T18:48:31.131 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-23T18:48:31.138 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-23T18:48:31.145 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T18:48:31.153 DEBUG:tasks.check_counter:Getting stats from l 2024-04-23T18:48:31.187 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 5d677974-019c-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-23T18:48:31.199 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:22 smithi188 ceph-mon[32694]: pgmap v952: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-23T18:48:31.272 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:24 smithi188 ceph-mon[32694]: pgmap v953: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:48:31.279 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:26 smithi188 ceph-mon[32694]: pgmap v954: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:48:31.286 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:28 smithi188 ceph-mon[32694]: pgmap v955: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:48:31.293 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:30 smithi188 ceph-mon[32694]: pgmap v956: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:31.303 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:26 smithi177 ceph-mon[32308]: pgmap v954: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:48:31.338 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:28 smithi177 ceph-mon[32308]: pgmap v955: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:48:31.345 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:30 smithi177 ceph-mon[32308]: pgmap v956: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:31.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:26 smithi050 ceph-mon[29187]: pgmap v954: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-23T18:48:31.362 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:28 smithi050 ceph-mon[29187]: pgmap v955: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 96 B/s wr, 8 op/s 2024-04-23T18:48:31.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:30 smithi050 ceph-mon[29187]: pgmap v956: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:31.554 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T18:48:32.738 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:32 smithi188 ceph-mon[32694]: pgmap v957: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:32.785 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:32 smithi177 ceph-mon[32308]: pgmap v957: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:32.942 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:32 smithi050 ceph-mon[29187]: pgmap v957: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:33.284 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-23T18:48:33.293 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T18:48:33.304 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 528, 2024-04-23T18:48:33.372 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 516, 2024-04-23T18:48:33.381 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 1815824, 2024-04-23T18:48:33.392 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 320128, 2024-04-23T18:48:33.401 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 4, 2024-04-23T18:48:33.411 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 0, 2024-04-23T18:48:33.501 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.083019317, 2024-04-23T18:48:33.507 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.028330499, 2024-04-23T18:48:33.516 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.047777021, 2024-04-23T18:48:33.526 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.012313971, 2024-04-23T18:48:33.535 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:48:33.544 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 515, 2024-04-23T18:48:33.554 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.022771521, 2024-04-23T18:48:33.562 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000044216 2024-04-23T18:48:33.570 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:33.578 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:48:33.618 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:33.629 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:33.691 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:33.698 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:33.710 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 1815824, 2024-04-23T18:48:33.719 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 320128 2024-04-23T18:48:33.728 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:33.772 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T18:48:33.781 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 4, 2024-04-23T18:48:33.790 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 397, 2024-04-23T18:48:33.799 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 640, 2024-04-23T18:48:33.807 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 382560, 2024-04-23T18:48:33.815 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 7, 2024-04-23T18:48:33.823 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 3, 2024-04-23T18:48:33.834 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.021538478, 2024-04-23T18:48:33.844 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.015479184, 2024-04-23T18:48:33.853 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.001927693, 2024-04-23T18:48:33.900 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:48:33.908 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:48:33.958 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 394, 2024-04-23T18:48:33.970 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.029561171, 2024-04-23T18:48:33.979 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000075028 2024-04-23T18:48:33.988 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:33.997 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:48:34.006 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:34.014 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:34.022 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:34.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:34.074 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 640, 2024-04-23T18:48:34.082 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 382560 2024-04-23T18:48:34.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:34.100 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T18:48:34.109 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 4, 2024-04-23T18:48:34.118 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 4, 2024-04-23T18:48:34.147 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 2992, 2024-04-23T18:48:34.157 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 448, 2024-04-23T18:48:34.218 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 2, 2024-04-23T18:48:34.226 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 1, 2024-04-23T18:48:34.234 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.000616628, 2024-04-23T18:48:34.242 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.000032243, 2024-04-23T18:48:34.249 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.000593584, 2024-04-23T18:48:34.289 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T18:48:34.297 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T18:48:34.305 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 3, 2024-04-23T18:48:34.314 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000678996, 2024-04-23T18:48:34.322 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000226332 2024-04-23T18:48:34.330 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:34.338 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-23T18:48:34.345 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:34.354 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:34.363 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:34.372 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:34.382 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-23T18:48:34.431 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T18:48:34.440 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:34.448 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2024-04-23T18:48:34.458 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 0, 2024-04-23T18:48:34.467 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2024-04-23T18:48:34.524 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:34.533 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2024-04-23T18:48:34.543 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2024-04-23T18:48:34.552 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2024-04-23T18:48:34.559 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T18:48:34.566 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2024-04-23T18:48:34.574 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T18:48:34.581 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2024-04-23T18:48:34.588 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T18:48:34.597 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T18:48:34.633 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T18:48:34.642 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T18:48:34.689 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T18:48:34.699 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2024-04-23T18:48:34.708 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T18:48:34.717 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T18:48:34.727 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2024-04-23T18:48:34.771 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2024-04-23T18:48:34.781 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2024-04-23T18:48:34.790 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2024-04-23T18:48:34.799 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T18:48:34.807 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T18:48:34.814 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T18:48:34.824 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T18:48:34.833 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T18:48:34.841 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2024-04-23T18:48:34.850 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2024-04-23T18:48:34.889 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2024-04-23T18:48:34.898 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T18:48:34.960 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T18:48:34.967 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2024-04-23T18:48:34.976 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2024-04-23T18:48:34.985 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2024-04-23T18:48:34.994 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2024-04-23T18:48:35.029 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T18:48:35.038 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2024-04-23T18:48:35.047 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T18:48:35.056 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2024-04-23T18:48:35.066 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 148721, 2024-04-23T18:48:35.077 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 30, 2024-04-23T18:48:35.085 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 0, 2024-04-23T18:48:35.095 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 0, 2024-04-23T18:48:35.104 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2024-04-23T18:48:35.113 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2024-04-23T18:48:35.122 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2024-04-23T18:48:35.161 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2024-04-23T18:48:35.220 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2024-04-23T18:48:35.228 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2024-04-23T18:48:35.237 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 0, 2024-04-23T18:48:35.245 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 0, 2024-04-23T18:48:35.254 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T18:48:35.296 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2024-04-23T18:48:35.306 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2024-04-23T18:48:35.315 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2024-04-23T18:48:35.324 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 928, 2024-04-23T18:48:35.333 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 2, 2024-04-23T18:48:35.343 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2024-04-23T18:48:35.351 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2024-04-23T18:48:35.360 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2024-04-23T18:48:35.367 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2024-04-23T18:48:35.376 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-23T18:48:35.408 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T18:48:35.418 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-23T18:48:35.478 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-23T18:48:35.486 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-23T18:48:35.494 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 532, 2024-04-23T18:48:35.503 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 1741232, 2024-04-23T18:48:35.512 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-23T18:48:35.570 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 532, 2024-04-23T18:48:35.581 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-23T18:48:35.591 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-23T18:48:35.599 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 532, 2024-04-23T18:48:35.608 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 1741232, 2024-04-23T18:48:35.617 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-23T18:48:35.627 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-23T18:48:35.636 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-23T18:48:35.647 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-23T18:48:35.656 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:35.688 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-23T18:48:35.698 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-23T18:48:35.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:34 smithi050 ceph-mon[29187]: pgmap v958: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:48:35.761 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:34 smithi177 ceph-mon[32308]: pgmap v958: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:48:35.772 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-23T18:48:35.782 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-23T18:48:35.791 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-04-23T18:48:35.821 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-04-23T18:48:35.828 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-23T18:48:35.835 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-23T18:48:35.842 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-23T18:48:35.850 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-23T18:48:35.866 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-23T18:48:35.876 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:48:35.884 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T18:48:35.893 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:34 smithi188 ceph-mon[32694]: pgmap v958: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:48:35.923 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-23T18:48:35.957 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T18:48:35.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:48:36.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.151+0000 7fabd29af640 1 -- 172.21.15.50:0/805594671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc104d50 msgr2=0x7fabcc108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:36.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.151+0000 7fabd29af640 1 --2- 172.21.15.50:0/805594671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc104d50 0x7fabcc108d90 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009900 tx=0x7fabc002a410 comp rx=0 tx=0).stop 2024-04-23T18:48:36.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.151+0000 7fabd29af640 1 -- 172.21.15.50:0/805594671 shutdown_connections 2024-04-23T18:48:36.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.151+0000 7fabd29af640 1 --2- 172.21.15.50:0/805594671 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabcc109470 0x7fabcc111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.151+0000 7fabd29af640 1 --2- 172.21.15.50:0/805594671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc104d50 0x7fabcc108d90 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:36.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.151+0000 7fabd29af640 1 --2- 172.21.15.50:0/805594671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc1043a0 0x7fabcc104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:36.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.151+0000 7fabd29af640 1 -- 172.21.15.50:0/805594671 >> 172.21.15.50:0/805594671 conn(0x7fabcc0ffc00 msgr2=0x7fabcc102020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:36.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.151+0000 7fabd29af640 1 -- 172.21.15.50:0/805594671 shutdown_connections 2024-04-23T18:48:36.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.151+0000 7fabd29af640 1 -- 172.21.15.50:0/805594671 wait complete. 2024-04-23T18:48:36.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabd29af640 1 Processor -- start 2024-04-23T18:48:36.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabd29af640 1 -- start start 2024-04-23T18:48:36.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabd29af640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabcc1043a0 0x7fabcc1a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:36.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabd29af640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc109470 0x7fabcc1a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabd29af640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc19a800 0x7fabcc19acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:36.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabd29af640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc1143b0 con 0x7fabcc109470 2024-04-23T18:48:36.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabd29af640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc114230 con 0x7fabcc19a800 2024-04-23T18:48:36.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabd29af640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc114530 con 0x7fabcc1043a0 2024-04-23T18:48:36.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabcb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc109470 0x7fabcc1a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:36.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabcb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc109470 0x7fabcc1a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44680/0 (socket says 172.21.15.50:44680) 2024-04-23T18:48:36.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabcb7fe640 1 -- 172.21.15.50:0/2494367081 learned_addr learned my addr 172.21.15.50:0/2494367081 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:48:36.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabcbfff640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabcc1043a0 0x7fabcc1a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:36.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabd0f4c640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc19a800 0x7fabcc19acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:36.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabcb7fe640 1 -- 172.21.15.50:0/2494367081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabcc1043a0 msgr2=0x7fabcc1a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:36.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabcb7fe640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabcc1043a0 0x7fabcc1a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:36.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabcb7fe640 1 -- 172.21.15.50:0/2494367081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc19a800 msgr2=0x7fabcc19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:36.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabcb7fe640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc19a800 0x7fabcc19acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:36.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.152+0000 7fabcb7fe640 1 -- 172.21.15.50:0/2494367081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabcc19b390 con 0x7fabcc109470 2024-04-23T18:48:36.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.153+0000 7fabcb7fe640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc109470 0x7fabcc1a0b80 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fabc0006010 tx=0x7fabc0004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:36.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.153+0000 7fabc97fa640 1 -- 172.21.15.50:0/2494367081 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc003f030 con 0x7fabcc109470 2024-04-23T18:48:36.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.153+0000 7fabc97fa640 1 -- 172.21.15.50:0/2494367081 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabc0003aa0 con 0x7fabcc109470 2024-04-23T18:48:36.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.153+0000 7fabd29af640 1 -- 172.21.15.50:0/2494367081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabcc19b620 con 0x7fabcc109470 2024-04-23T18:48:36.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.153+0000 7fabc97fa640 1 -- 172.21.15.50:0/2494367081 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc0035070 con 0x7fabcc109470 2024-04-23T18:48:36.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.153+0000 7fabd29af640 1 -- 172.21.15.50:0/2494367081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabcc1a77b0 con 0x7fabcc109470 2024-04-23T18:48:36.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.154+0000 7fabd29af640 1 -- 172.21.15.50:0/2494367081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabcc105870 con 0x7fabcc109470 2024-04-23T18:48:36.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.155+0000 7fabc97fa640 1 -- 172.21.15.50:0/2494367081 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fabc0042030 con 0x7fabcc109470 2024-04-23T18:48:36.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.156+0000 7fabc97fa640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fabb40a3da0 0x7fabb40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:36.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.156+0000 7fabcbfff640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fabb40a3da0 0x7fabb40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:36.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.156+0000 7fabc97fa640 1 -- 172.21.15.50:0/2494367081 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 8542+0+0 (secure 0 0 0) 0x7fabc00f0af0 con 0x7fabcc109470 2024-04-23T18:48:36.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.156+0000 7fabcbfff640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fabb40a3da0 0x7fabb40a6260 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00bbe0 tx=0x7fabbc00b390 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:36.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.161+0000 7fabc97fa640 1 -- 172.21.15.50:0/2494367081 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fabc00bcb30 con 0x7fabcc109470 2024-04-23T18:48:36.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.348+0000 7fabd29af640 1 -- 172.21.15.50:0/2494367081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fabcc10c6a0 con 0x7fabcc109470 2024-04-23T18:48:36.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.349+0000 7fabc97fa640 1 -- 172.21.15.50:0/2494367081 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) ==== 74+0+23132 (secure 0 0 0) 0x7fabc00c0e30 con 0x7fabcc109470 2024-04-23T18:48:36.739 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:48:36.748 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":144,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:48:16.345905+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":55,"snap_epoch":143,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/1479623929":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/3248139242":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6800/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/2484340243":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/444338023":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":42,"length":1},{"begin":55,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:48:36.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.351+0000 7fabd29af640 1 -- 172.21.15.50:0/2494367081 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fabb40a3da0 msgr2=0x7fabb40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:36.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.351+0000 7fabd29af640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fabb40a3da0 0x7fabb40a6260 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00bbe0 tx=0x7fabbc00b390 comp rx=0 tx=0).stop 2024-04-23T18:48:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.351+0000 7fabd29af640 1 -- 172.21.15.50:0/2494367081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc109470 msgr2=0x7fabcc1a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:36.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.351+0000 7fabd29af640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc109470 0x7fabcc1a0b80 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fabc0006010 tx=0x7fabc0004060 comp rx=0 tx=0).stop 2024-04-23T18:48:36.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.351+0000 7fabd29af640 1 -- 172.21.15.50:0/2494367081 shutdown_connections 2024-04-23T18:48:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.351+0000 7fabd29af640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fabb40a3da0 0x7fabb40a6260 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:36.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.351+0000 7fabd29af640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabcc19a800 0x7fabcc19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:36.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.351+0000 7fabd29af640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc109470 0x7fabcc1a0b80 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:36.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.351+0000 7fabd29af640 1 --2- 172.21.15.50:0/2494367081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabcc1043a0 0x7fabcc1a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:36.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.352+0000 7fabd29af640 1 -- 172.21.15.50:0/2494367081 >> 172.21.15.50:0/2494367081 conn(0x7fabcc0ffc00 msgr2=0x7fabcc101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:36.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.352+0000 7fabd29af640 1 -- 172.21.15.50:0/2494367081 shutdown_connections 2024-04-23T18:48:36.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:36.352+0000 7fabd29af640 1 -- 172.21.15.50:0/2494367081 wait complete. 2024-04-23T18:48:37.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T18:12:02.493647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T18:16:31.679148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T18:16:32.269478+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 55, 'snap_epoch': 143, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T18:48:37.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T18:48:37.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:36 smithi050 ceph-mon[29187]: pgmap v959: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:37.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:36 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2494367081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:48:37.097 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:36 smithi188 ceph-mon[32694]: pgmap v959: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:37.105 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:36 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2494367081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:48:37.114 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:36 smithi177 ceph-mon[32308]: pgmap v959: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:37.123 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:36 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2494367081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:48:37.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.222+0000 7f693c88a640 1 -- 172.21.15.50:0/701196131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6938102b40 msgr2=0x7f6938106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:37.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.222+0000 7f693c88a640 1 --2- 172.21.15.50:0/701196131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6938102b40 0x7f6938106b80 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f6920009900 tx=0x7f692002a410 comp rx=0 tx=0).stop 2024-04-23T18:48:37.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.222+0000 7f693c88a640 1 -- 172.21.15.50:0/701196131 shutdown_connections 2024-04-23T18:48:37.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.222+0000 7f693c88a640 1 --2- 172.21.15.50:0/701196131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6938107260 0x7f693810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:37.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.222+0000 7f693c88a640 1 --2- 172.21.15.50:0/701196131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6938102b40 0x7f6938106b80 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:37.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.222+0000 7f693c88a640 1 --2- 172.21.15.50:0/701196131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6938102190 0x7f6938102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:37.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.222+0000 7f693c88a640 1 -- 172.21.15.50:0/701196131 >> 172.21.15.50:0/701196131 conn(0x7f69380fd990 msgr2=0x7f69380ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:37.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.222+0000 7f693c88a640 1 -- 172.21.15.50:0/701196131 shutdown_connections 2024-04-23T18:48:37.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.222+0000 7f693c88a640 1 -- 172.21.15.50:0/701196131 wait complete. 2024-04-23T18:48:37.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.223+0000 7f693c88a640 1 Processor -- start 2024-04-23T18:48:37.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.223+0000 7f693c88a640 1 -- start start 2024-04-23T18:48:37.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.223+0000 7f693c88a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6938102190 0x7f69381987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:37.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.223+0000 7f693c88a640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6938107260 0x7f6938198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:37.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.223+0000 7f693c88a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f693819d020 0x7f693819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:37.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.223+0000 7f693c88a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6938112180 con 0x7f6938102190 2024-04-23T18:48:37.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.223+0000 7f693c88a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6938112000 con 0x7f693819d020 2024-04-23T18:48:37.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.223+0000 7f693c88a640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6938112300 con 0x7f6938107260 2024-04-23T18:48:37.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f693659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6938102190 0x7f69381987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:37.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f693659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6938102190 0x7f69381987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44704/0 (socket says 172.21.15.50:44704) 2024-04-23T18:48:37.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f693659c640 1 -- 172.21.15.50:0/471050025 learned_addr learned my addr 172.21.15.50:0/471050025 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:48:37.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f692dd9b640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6938107260 0x7f6938198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:37.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f6936d9d640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f693819d020 0x7f693819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f693659c640 1 -- 172.21.15.50:0/471050025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6938107260 msgr2=0x7f6938198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:37.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f693659c640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6938107260 0x7f6938198cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f693659c640 1 -- 172.21.15.50:0/471050025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f693819d020 msgr2=0x7f693819d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f693659c640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f693819d020 0x7f693819d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:37.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f693659c640 1 -- 172.21.15.50:0/471050025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693819db10 con 0x7f6938102190 2024-04-23T18:48:37.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f6936d9d640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f693819d020 0x7f693819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:48:37.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f693659c640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6938102190 0x7f69381987b0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f692800d950 tx=0x7f692800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:37.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f692ffff640 1 -- 172.21.15.50:0/471050025 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6928017070 con 0x7f6938102190 2024-04-23T18:48:37.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f693c88a640 1 -- 172.21.15.50:0/471050025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f693819de00 con 0x7f6938102190 2024-04-23T18:48:37.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f692ffff640 1 -- 172.21.15.50:0/471050025 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6928009c30 con 0x7f6938102190 2024-04-23T18:48:37.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f692ffff640 1 -- 172.21.15.50:0/471050025 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692800f620 con 0x7f6938102190 2024-04-23T18:48:37.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.224+0000 7f693c88a640 1 -- 172.21.15.50:0/471050025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69381a5610 con 0x7f6938102190 2024-04-23T18:48:37.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.225+0000 7f693c88a640 1 -- 172.21.15.50:0/471050025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69380691c0 con 0x7f6938102190 2024-04-23T18:48:37.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.227+0000 7f692ffff640 1 -- 172.21.15.50:0/471050025 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6928010460 con 0x7f6938102190 2024-04-23T18:48:37.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.227+0000 7f692ffff640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f69240a3d20 0x7f69240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:37.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.228+0000 7f692dd9b640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f69240a3d20 0x7f69240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:37.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.228+0000 7f692ffff640 1 -- 172.21.15.50:0/471050025 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 8542+0+0 (secure 0 0 0) 0x7f69280d0450 con 0x7f6938102190 2024-04-23T18:48:37.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.228+0000 7f692dd9b640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f69240a3d20 0x7f69240a61e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6920000c00 tx=0x7f6920031400 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:37.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.231+0000 7f692ffff640 1 -- 172.21.15.50:0/471050025 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f692809cac0 con 0x7f6938102190 2024-04-23T18:48:37.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.416+0000 7f693c88a640 1 -- 172.21.15.50:0/471050025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f693810a4c0 con 0x7f6938102190 2024-04-23T18:48:37.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.416+0000 7f692ffff640 1 -- 172.21.15.50:0/471050025 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v144) ==== 93+0+10 (secure 0 0 0) 0x7f6928009de0 con 0x7f6938102190 2024-04-23T18:48:38.029 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-23T18:48:38.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.418+0000 7f693c88a640 1 -- 172.21.15.50:0/471050025 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f69240a3d20 msgr2=0x7f69240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:38.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.418+0000 7f693c88a640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f69240a3d20 0x7f69240a61e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6920000c00 tx=0x7f6920031400 comp rx=0 tx=0).stop 2024-04-23T18:48:38.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.418+0000 7f693c88a640 1 -- 172.21.15.50:0/471050025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6938102190 msgr2=0x7f69381987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:38.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.418+0000 7f693c88a640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6938102190 0x7f69381987b0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f692800d950 tx=0x7f692800de10 comp rx=0 tx=0).stop 2024-04-23T18:48:38.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.418+0000 7f693c88a640 1 -- 172.21.15.50:0/471050025 shutdown_connections 2024-04-23T18:48:38.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.418+0000 7f693c88a640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f69240a3d20 0x7f69240a61e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:38.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.418+0000 7f693c88a640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f693819d020 0x7f693819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:38.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.418+0000 7f693c88a640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6938107260 0x7f6938198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:38.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.418+0000 7f693c88a640 1 --2- 172.21.15.50:0/471050025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6938102190 0x7f69381987b0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:38.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.418+0000 7f693c88a640 1 -- 172.21.15.50:0/471050025 >> 172.21.15.50:0/471050025 conn(0x7f69380fd990 msgr2=0x7f69380fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:38.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.418+0000 7f693c88a640 1 -- 172.21.15.50:0/471050025 shutdown_connections 2024-04-23T18:48:38.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:37.418+0000 7f693c88a640 1 -- 172.21.15.50:0/471050025 wait complete. 2024-04-23T18:48:38.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T18:48:38.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:37 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/471050025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:48:38.234 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:37 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/471050025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:48:38.242 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:37 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/471050025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T18:48:38.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.361+0000 7fdd160a7640 1 -- 172.21.15.50:0/2273426609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd10106150 msgr2=0x7fdd1010dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:38.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.361+0000 7fdd160a7640 1 --2- 172.21.15.50:0/2273426609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd10106150 0x7fdd1010dd20 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009900 tx=0x7fdd0802a410 comp rx=0 tx=0).stop 2024-04-23T18:48:38.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.361+0000 7fdd160a7640 1 -- 172.21.15.50:0/2273426609 shutdown_connections 2024-04-23T18:48:38.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.361+0000 7fdd160a7640 1 --2- 172.21.15.50:0/2273426609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd10106150 0x7fdd1010dd20 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:38.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.361+0000 7fdd160a7640 1 --2- 172.21.15.50:0/2273426609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd10101a30 0x7fdd10105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:38.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.361+0000 7fdd160a7640 1 --2- 172.21.15.50:0/2273426609 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd10101080 0x7fdd10101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:38.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.361+0000 7fdd160a7640 1 -- 172.21.15.50:0/2273426609 >> 172.21.15.50:0/2273426609 conn(0x7fdd10078000 msgr2=0x7fdd10078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:38.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.361+0000 7fdd160a7640 1 -- 172.21.15.50:0/2273426609 shutdown_connections 2024-04-23T18:48:38.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.361+0000 7fdd160a7640 1 -- 172.21.15.50:0/2273426609 wait complete. 2024-04-23T18:48:38.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.361+0000 7fdd160a7640 1 Processor -- start 2024-04-23T18:48:38.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd160a7640 1 -- start start 2024-04-23T18:48:38.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd160a7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd10101080 0x7fdd101a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd160a7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd10106150 0x7fdd101a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd160a7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd1019a6c0 0x7fdd1019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:38.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd160a7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdd101110c0 con 0x7fdd10101080 2024-04-23T18:48:38.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd160a7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd10110f40 con 0x7fdd1019a6c0 2024-04-23T18:48:38.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd160a7640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdd10111240 con 0x7fdd10106150 2024-04-23T18:48:38.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd10101080 0x7fdd101a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:38.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd10101080 0x7fdd101a0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44728/0 (socket says 172.21.15.50:44728) 2024-04-23T18:48:38.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0f7fe640 1 -- 172.21.15.50:0/2342839324 learned_addr learned my addr 172.21.15.50:0/2342839324 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:48:38.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0ffff640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd1019a6c0 0x7fdd1019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:38.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0f7fe640 1 -- 172.21.15.50:0/2342839324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd10106150 msgr2=0x7fdd101a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:38.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0effd640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd10106150 0x7fdd101a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:38.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0f7fe640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd10106150 0x7fdd101a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0f7fe640 1 -- 172.21.15.50:0/2342839324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd1019a6c0 msgr2=0x7fdd1019ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:38.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0f7fe640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd1019a6c0 0x7fdd1019ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:38.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0f7fe640 1 -- 172.21.15.50:0/2342839324 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1019b2c0 con 0x7fdd10101080 2024-04-23T18:48:38.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0ffff640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd1019a6c0 0x7fdd1019ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:48:38.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0effd640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd10106150 0x7fdd101a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:48:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.362+0000 7fdd0f7fe640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd10101080 0x7fdd101a0600 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000eb90 tx=0x7fdd0000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:38.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.363+0000 7fdd0cff9640 1 -- 172.21.15.50:0/2342839324 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd00004d20 con 0x7fdd10101080 2024-04-23T18:48:38.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.363+0000 7fdd160a7640 1 -- 172.21.15.50:0/2342839324 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd101a73a0 con 0x7fdd10101080 2024-04-23T18:48:38.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.363+0000 7fdd0cff9640 1 -- 172.21.15.50:0/2342839324 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd0000ee90 con 0x7fdd10101080 2024-04-23T18:48:38.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.363+0000 7fdd0cff9640 1 -- 172.21.15.50:0/2342839324 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd00010670 con 0x7fdd10101080 2024-04-23T18:48:38.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.363+0000 7fdd160a7640 1 -- 172.21.15.50:0/2342839324 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd101a77a0 con 0x7fdd10101080 2024-04-23T18:48:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.364+0000 7fdd160a7640 1 -- 172.21.15.50:0/2342839324 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd10102550 con 0x7fdd10101080 2024-04-23T18:48:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.366+0000 7fdd0cff9640 1 -- 172.21.15.50:0/2342839324 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdd00010930 con 0x7fdd10101080 2024-04-23T18:48:38.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.366+0000 7fdd0cff9640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdcfc0a3d20 0x7fdcfc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.366+0000 7fdd0effd640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdcfc0a3d20 0x7fdcfc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:38.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.366+0000 7fdd0cff9640 1 -- 172.21.15.50:0/2342839324 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 8542+0+0 (secure 0 0 0) 0x7fdd000d1120 con 0x7fdd10101080 2024-04-23T18:48:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.367+0000 7fdd0effd640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdcfc0a3d20 0x7fdcfc0a61e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8005fd0 tx=0x7fdcf80094d0 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:38.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.370+0000 7fdd0cff9640 1 -- 172.21.15.50:0/2342839324 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd00053ff0 con 0x7fdd10101080 2024-04-23T18:48:38.958 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:38 smithi188 ceph-mon[32694]: pgmap v960: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:48:38.967 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:38 smithi177 ceph-mon[32308]: pgmap v960: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:48:38.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.557+0000 7fdd160a7640 1 -- 172.21.15.50:0/2342839324 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdd10068d70 con 0x7fdd10101080 2024-04-23T18:48:38.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.557+0000 7fdd0cff9640 1 -- 172.21.15.50:0/2342839324 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v144) ==== 104+0+11 (secure 0 0 0) 0x7fdd0001f340 con 0x7fdd10101080 2024-04-23T18:48:39.052 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2024-04-23T18:48:39.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.559+0000 7fdd160a7640 1 -- 172.21.15.50:0/2342839324 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdcfc0a3d20 msgr2=0x7fdcfc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:39.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.559+0000 7fdd160a7640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdcfc0a3d20 0x7fdcfc0a61e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8005fd0 tx=0x7fdcf80094d0 comp rx=0 tx=0).stop 2024-04-23T18:48:39.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.559+0000 7fdd160a7640 1 -- 172.21.15.50:0/2342839324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd10101080 msgr2=0x7fdd101a0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:39.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.559+0000 7fdd160a7640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd10101080 0x7fdd101a0600 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000eb90 tx=0x7fdd0000c6a0 comp rx=0 tx=0).stop 2024-04-23T18:48:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.559+0000 7fdd160a7640 1 -- 172.21.15.50:0/2342839324 shutdown_connections 2024-04-23T18:48:39.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.559+0000 7fdd160a7640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fdcfc0a3d20 0x7fdcfc0a61e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.559+0000 7fdd160a7640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd1019a6c0 0x7fdd1019ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:39.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.559+0000 7fdd160a7640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd10106150 0x7fdd101a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:39.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.559+0000 7fdd160a7640 1 --2- 172.21.15.50:0/2342839324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd10101080 0x7fdd101a0600 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:39.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.559+0000 7fdd160a7640 1 -- 172.21.15.50:0/2342839324 >> 172.21.15.50:0/2342839324 conn(0x7fdd10078000 msgr2=0x7fdd10100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:39.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.559+0000 7fdd160a7640 1 -- 172.21.15.50:0/2342839324 shutdown_connections 2024-04-23T18:48:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:38.559+0000 7fdd160a7640 1 -- 172.21.15.50:0/2342839324 wait complete. 2024-04-23T18:48:39.188 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:38 smithi050 ceph-mon[29187]: pgmap v960: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:48:39.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T18:48:39.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.364+0000 7f459dd22640 1 -- 172.21.15.50:0/604479580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f459806a5d0 msgr2=0x7f459806a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.364+0000 7f459dd22640 1 --2- 172.21.15.50:0/604479580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f459806a5d0 0x7f459806a9b0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f4588009900 tx=0x7f458802a410 comp rx=0 tx=0).stop 2024-04-23T18:48:39.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.364+0000 7f459dd22640 1 -- 172.21.15.50:0/604479580 shutdown_connections 2024-04-23T18:48:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.364+0000 7f459dd22640 1 --2- 172.21.15.50:0/604479580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f459810e960 0x7f4598110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:39.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.364+0000 7f459dd22640 1 --2- 172.21.15.50:0/604479580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f459806aef0 0x7f459810e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:39.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.364+0000 7f459dd22640 1 --2- 172.21.15.50:0/604479580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f459806a5d0 0x7f459806a9b0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:39.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.364+0000 7f459dd22640 1 -- 172.21.15.50:0/604479580 >> 172.21.15.50:0/604479580 conn(0x7f4598077eb0 msgr2=0x7f45980782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:39.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.364+0000 7f459dd22640 1 -- 172.21.15.50:0/604479580 shutdown_connections 2024-04-23T18:48:39.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.364+0000 7f459dd22640 1 -- 172.21.15.50:0/604479580 wait complete. 2024-04-23T18:48:39.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.365+0000 7f459dd22640 1 Processor -- start 2024-04-23T18:48:39.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.365+0000 7f459dd22640 1 -- start start 2024-04-23T18:48:39.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.365+0000 7f459dd22640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f459806a5d0 0x7f45981a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:39.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.365+0000 7f459dd22640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f459806aef0 0x7f45981a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:39.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.365+0000 7f459dd22640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f459810e960 0x7f459819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:39.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.365+0000 7f459dd22640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f45981143e0 con 0x7f459806a5d0 2024-04-23T18:48:39.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.365+0000 7f459dd22640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4598114260 con 0x7f459810e960 2024-04-23T18:48:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.365+0000 7f459dd22640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4598114560 con 0x7f459806aef0 2024-04-23T18:48:39.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f45977fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f459806a5d0 0x7f45981a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:39.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f45977fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f459806a5d0 0x7f45981a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44752/0 (socket says 172.21.15.50:44752) 2024-04-23T18:48:39.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f45977fe640 1 -- 172.21.15.50:0/2636559860 learned_addr learned my addr 172.21.15.50:0/2636559860 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:48:39.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f4597fff640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f459810e960 0x7f459819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:39.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f4596ffd640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f459806aef0 0x7f45981a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:39.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f45977fe640 1 -- 172.21.15.50:0/2636559860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f459806aef0 msgr2=0x7f45981a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:39.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f45977fe640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f459806aef0 0x7f45981a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:39.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f45977fe640 1 -- 172.21.15.50:0/2636559860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f459810e960 msgr2=0x7f459819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:39.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f45977fe640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f459810e960 0x7f459819a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:39.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f45977fe640 1 -- 172.21.15.50:0/2636559860 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459819b080 con 0x7f459806a5d0 2024-04-23T18:48:39.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f45977fe640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f459806a5d0 0x7f45981a06b0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f4588005940 tx=0x7f45880040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:39.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f4594ff9640 1 -- 172.21.15.50:0/2636559860 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458803f030 con 0x7f459806a5d0 2024-04-23T18:48:39.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f4594ff9640 1 -- 172.21.15.50:0/2636559860 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45880044c0 con 0x7f459806a5d0 2024-04-23T18:48:39.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f459dd22640 1 -- 172.21.15.50:0/2636559860 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f459819b310 con 0x7f459806a5d0 2024-04-23T18:48:39.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f4594ff9640 1 -- 172.21.15.50:0/2636559860 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4588035070 con 0x7f459806a5d0 2024-04-23T18:48:39.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.366+0000 7f459dd22640 1 -- 172.21.15.50:0/2636559860 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45981a7450 con 0x7f459806a5d0 2024-04-23T18:48:39.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:39 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2342839324' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:48:39.903 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:39 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2342839324' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:48:39.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.367+0000 7f459dd22640 1 -- 172.21.15.50:0/2636559860 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4598101460 con 0x7f459806a5d0 2024-04-23T18:48:39.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.368+0000 7f4594ff9640 1 -- 172.21.15.50:0/2636559860 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4588042030 con 0x7f459806a5d0 2024-04-23T18:48:39.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.369+0000 7f4594ff9640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f45800a3d20 0x7f45800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:39.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.369+0000 7f4596ffd640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f45800a3d20 0x7f45800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:39.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.369+0000 7f4594ff9640 1 -- 172.21.15.50:0/2636559860 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 8542+0+0 (secure 0 0 0) 0x7f45880f0e60 con 0x7f459806a5d0 2024-04-23T18:48:39.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.370+0000 7f4596ffd640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f45800a3d20 0x7f45800a61e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f459819bfc0 tx=0x7f458c005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:40.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.373+0000 7f4594ff9640 1 -- 172.21.15.50:0/2636559860 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45880bd490 con 0x7f459806a5d0 2024-04-23T18:48:40.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.562+0000 7f459dd22640 1 -- 172.21.15.50:0/2636559860 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4598109af0 con 0x7f459806a5d0 2024-04-23T18:48:40.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.562+0000 7f4594ff9640 1 -- 172.21.15.50:0/2636559860 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v144) ==== 100+0+11 (secure 0 0 0) 0x7f45880bcd90 con 0x7f459806a5d0 2024-04-23T18:48:40.059 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2024-04-23T18:48:40.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.564+0000 7f459dd22640 1 -- 172.21.15.50:0/2636559860 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f45800a3d20 msgr2=0x7f45800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:40.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.564+0000 7f459dd22640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f45800a3d20 0x7f45800a61e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f459819bfc0 tx=0x7f458c005f90 comp rx=0 tx=0).stop 2024-04-23T18:48:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.564+0000 7f459dd22640 1 -- 172.21.15.50:0/2636559860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f459806a5d0 msgr2=0x7f45981a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:40.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.564+0000 7f459dd22640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f459806a5d0 0x7f45981a06b0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f4588005940 tx=0x7f45880040c0 comp rx=0 tx=0).stop 2024-04-23T18:48:40.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.564+0000 7f459dd22640 1 -- 172.21.15.50:0/2636559860 shutdown_connections 2024-04-23T18:48:40.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.564+0000 7f459dd22640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f45800a3d20 0x7f45800a61e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:40.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.564+0000 7f459dd22640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f459810e960 0x7f459819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:40.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.564+0000 7f459dd22640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f459806aef0 0x7f45981a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:40.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.564+0000 7f459dd22640 1 --2- 172.21.15.50:0/2636559860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f459806a5d0 0x7f45981a06b0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:40.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.564+0000 7f459dd22640 1 -- 172.21.15.50:0/2636559860 >> 172.21.15.50:0/2636559860 conn(0x7f4598077eb0 msgr2=0x7f459810feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:40.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.564+0000 7f459dd22640 1 -- 172.21.15.50:0/2636559860 shutdown_connections 2024-04-23T18:48:40.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:39.564+0000 7f459dd22640 1 -- 172.21.15.50:0/2636559860 wait complete. 2024-04-23T18:48:40.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:39 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2342839324' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T18:48:40.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-23T18:48:40.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.417+0000 7f06397f7640 1 -- 172.21.15.50:0/696094725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0634069350 msgr2=0x7f0634069730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:40.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.417+0000 7f06397f7640 1 --2- 172.21.15.50:0/696094725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0634069350 0x7f0634069730 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f0624009900 tx=0x7f062402a410 comp rx=0 tx=0).stop 2024-04-23T18:48:40.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.417+0000 7f06397f7640 1 -- 172.21.15.50:0/696094725 shutdown_connections 2024-04-23T18:48:40.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.417+0000 7f06397f7640 1 --2- 172.21.15.50:0/696094725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0634107240 0x7f063410ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:40.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.417+0000 7f06397f7640 1 --2- 172.21.15.50:0/696094725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0634069d00 0x7f0634106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:40.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.417+0000 7f06397f7640 1 --2- 172.21.15.50:0/696094725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0634069350 0x7f0634069730 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:40.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.417+0000 7f06397f7640 1 -- 172.21.15.50:0/696094725 >> 172.21.15.50:0/696094725 conn(0x7f0634077cb0 msgr2=0x7f06340780c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:40.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.417+0000 7f06397f7640 1 -- 172.21.15.50:0/696094725 shutdown_connections 2024-04-23T18:48:40.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.417+0000 7f06397f7640 1 -- 172.21.15.50:0/696094725 wait complete. 2024-04-23T18:48:40.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06397f7640 1 Processor -- start 2024-04-23T18:48:40.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06397f7640 1 -- start start 2024-04-23T18:48:40.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06397f7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0634069350 0x7f06341986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:40.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06397f7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0634107240 0x7f0634198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:40.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06397f7640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f063419cf80 0x7f063419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:40.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06397f7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f06341121a0 con 0x7f0634107240 2024-04-23T18:48:40.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06397f7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0634112020 con 0x7f0634069350 2024-04-23T18:48:40.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06397f7640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0634112320 con 0x7f063419cf80 2024-04-23T18:48:40.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06327fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0634107240 0x7f0634198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:40.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06327fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0634107240 0x7f0634198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44774/0 (socket says 172.21.15.50:44774) 2024-04-23T18:48:40.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06327fc640 1 -- 172.21.15.50:0/2970857957 learned_addr learned my addr 172.21.15.50:0/2970857957 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:48:40.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f0632ffd640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0634069350 0x7f06341986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:40.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06337fe640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f063419cf80 0x7f063419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:40.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06327fc640 1 -- 172.21.15.50:0/2970857957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f063419cf80 msgr2=0x7f063419d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:40.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06327fc640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f063419cf80 0x7f063419d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:40.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06327fc640 1 -- 172.21.15.50:0/2970857957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0634069350 msgr2=0x7f06341986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:40.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06327fc640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0634069350 0x7f06341986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:40.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.418+0000 7f06327fc640 1 -- 172.21.15.50:0/2970857957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f063419da70 con 0x7f0634107240 2024-04-23T18:48:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.419+0000 7f06327fc640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0634107240 0x7f0634198c00 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f062800e9f0 tx=0x7f062800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:40.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.419+0000 7f061bfff640 1 -- 172.21.15.50:0/2970857957 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062800ccc0 con 0x7f0634107240 2024-04-23T18:48:40.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.419+0000 7f061bfff640 1 -- 172.21.15.50:0/2970857957 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06280054d0 con 0x7f0634107240 2024-04-23T18:48:40.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.419+0000 7f06397f7640 1 -- 172.21.15.50:0/2970857957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f063419dd60 con 0x7f0634107240 2024-04-23T18:48:40.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.419+0000 7f061bfff640 1 -- 172.21.15.50:0/2970857957 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0628010430 con 0x7f0634107240 2024-04-23T18:48:40.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.419+0000 7f06397f7640 1 -- 172.21.15.50:0/2970857957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06341a5550 con 0x7f0634107240 2024-04-23T18:48:40.955 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:40 smithi188 ceph-mon[32694]: pgmap v961: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:40.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:40 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2636559860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:48:40.972 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:40 smithi177 ceph-mon[32308]: pgmap v961: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:40.979 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:40 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2636559860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:48:40.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.420+0000 7f06397f7640 1 -- 172.21.15.50:0/2970857957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0634068d70 con 0x7f0634107240 2024-04-23T18:48:40.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.421+0000 7f061bfff640 1 -- 172.21.15.50:0/2970857957 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f06280046c0 con 0x7f0634107240 2024-04-23T18:48:41.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.422+0000 7f061bfff640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f061c0a3da0 0x7f061c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:41.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.422+0000 7f0632ffd640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f061c0a3da0 0x7f061c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:41.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.422+0000 7f061bfff640 1 -- 172.21.15.50:0/2970857957 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 8542+0+0 (secure 0 0 0) 0x7f06280d0e20 con 0x7f0634107240 2024-04-23T18:48:41.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.423+0000 7f0632ffd640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f061c0a3da0 0x7f061c0a6260 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0624031440 tx=0x7f0624005b50 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:41.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.426+0000 7f061bfff640 1 -- 172.21.15.50:0/2970857957 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0628053ff0 con 0x7f0634107240 2024-04-23T18:48:41.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.626+0000 7f06397f7640 1 -- 172.21.15.50:0/2970857957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0634199a10 con 0x7f0634107240 2024-04-23T18:48:41.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.626+0000 7f061bfff640 1 -- 172.21.15.50:0/2970857957 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v144) ==== 81+0+941 (secure 0 0 0) 0x7f0634199a10 con 0x7f0634107240 2024-04-23T18:48:41.136 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:0/1479623929 2024-04-24T18:18:30.504131+0000 2024-04-23T18:48:41.144 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:0/3248139242 2024-04-24T18:18:30.504131+0000 2024-04-23T18:48:41.153 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:6800/1004018784 2024-04-24T18:18:30.504131+0000 2024-04-23T18:48:41.161 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:0/2484340243 2024-04-24T18:18:30.504131+0000 2024-04-23T18:48:41.169 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:6801/103327972 2024-04-24T18:09:34.403529+0000 2024-04-23T18:48:41.177 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:6800/103327972 2024-04-24T18:09:34.403529+0000 2024-04-23T18:48:41.220 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:0/1510580864 2024-04-24T18:09:34.403529+0000 2024-04-23T18:48:41.227 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:0/444338023 2024-04-24T18:18:30.504131+0000 2024-04-23T18:48:41.235 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:6801/1004018784 2024-04-24T18:18:30.504131+0000 2024-04-23T18:48:41.242 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:0/587528851 2024-04-24T18:09:07.548503+0000 2024-04-23T18:48:41.250 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:0/688464416 2024-04-24T18:09:07.548503+0000 2024-04-23T18:48:41.259 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:6800/1300452320 2024-04-24T18:09:07.548503+0000 2024-04-23T18:48:41.266 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:0/1448760865 2024-04-24T18:09:34.403529+0000 2024-04-23T18:48:41.273 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:6801/1300452320 2024-04-24T18:09:07.548503+0000 2024-04-23T18:48:41.305 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:0/3023139440 2024-04-24T18:09:07.548503+0000 2024-04-23T18:48:41.312 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50:0/2673950234 2024-04-24T18:09:34.403529+0000 2024-04-23T18:48:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:listed 16 entries 2024-04-23T18:48:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.628+0000 7f06397f7640 1 -- 172.21.15.50:0/2970857957 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f061c0a3da0 msgr2=0x7f061c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.628+0000 7f06397f7640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f061c0a3da0 0x7f061c0a6260 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0624031440 tx=0x7f0624005b50 comp rx=0 tx=0).stop 2024-04-23T18:48:41.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.628+0000 7f06397f7640 1 -- 172.21.15.50:0/2970857957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0634107240 msgr2=0x7f0634198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:41.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.628+0000 7f06397f7640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0634107240 0x7f0634198c00 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f062800e9f0 tx=0x7f062800eeb0 comp rx=0 tx=0).stop 2024-04-23T18:48:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.628+0000 7f06397f7640 1 -- 172.21.15.50:0/2970857957 shutdown_connections 2024-04-23T18:48:41.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.628+0000 7f06397f7640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f061c0a3da0 0x7f061c0a6260 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.628+0000 7f06397f7640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f063419cf80 0x7f063419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:41.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.628+0000 7f06397f7640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0634107240 0x7f0634198c00 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.628+0000 7f06397f7640 1 --2- 172.21.15.50:0/2970857957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0634069350 0x7f06341986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.628+0000 7f06397f7640 1 -- 172.21.15.50:0/2970857957 >> 172.21.15.50:0/2970857957 conn(0x7f0634077cb0 msgr2=0x7f06340ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.628+0000 7f06397f7640 1 -- 172.21.15.50:0/2970857957 shutdown_connections 2024-04-23T18:48:41.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:40.628+0000 7f06397f7640 1 -- 172.21.15.50:0/2970857957 wait complete. 2024-04-23T18:48:41.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:40 smithi050 ceph-mon[29187]: pgmap v961: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:41.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:40 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2636559860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T18:48:41.545 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-23T18:48:41.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:48:41.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-23T18:48:41.785 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:41 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2970857957' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T18:48:41.827 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:41 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2970857957' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T18:48:41.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:41 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2970857957' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T18:48:41.849 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2024-04-23T18:48:41.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:48:41.894 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-23T18:48:41.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:48:41.974 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2024-04-23T18:48:42.046 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-23T18:48:42.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:48:42.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-23T18:48:42.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T18:48:42.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link delete ceph-brx 2024-04-23T18:48:42.240 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-23T18:48:42.280 INFO:tasks.cephadm:Teardown begin 2024-04-23T18:48:42.290 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-23T18:48:42.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:48:42.478 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.481+0000 7fe97b5ba640 1 -- 172.21.15.50:0/936871073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9741010a0 msgr2=0x7fe974101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:42.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.481+0000 7fe97b5ba640 1 --2- 172.21.15.50:0/936871073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9741010a0 0x7fe974101480 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe970009860 tx=0x7fe97002a340 comp rx=0 tx=0).stop 2024-04-23T18:48:42.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.481+0000 7fe97b5ba640 1 -- 172.21.15.50:0/936871073 shutdown_connections 2024-04-23T18:48:42.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.481+0000 7fe97b5ba640 1 --2- 172.21.15.50:0/936871073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe974106170 0x7fe97410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:42.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.481+0000 7fe97b5ba640 1 --2- 172.21.15.50:0/936871073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe974101a50 0x7fe974105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:42.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.481+0000 7fe97b5ba640 1 --2- 172.21.15.50:0/936871073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9741010a0 0x7fe974101480 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:42.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.481+0000 7fe97b5ba640 1 -- 172.21.15.50:0/936871073 >> 172.21.15.50:0/936871073 conn(0x7fe974077f50 msgr2=0x7fe974078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:42.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.481+0000 7fe97b5ba640 1 -- 172.21.15.50:0/936871073 shutdown_connections 2024-04-23T18:48:42.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.481+0000 7fe97b5ba640 1 -- 172.21.15.50:0/936871073 wait complete. 2024-04-23T18:48:42.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe97b5ba640 1 Processor -- start 2024-04-23T18:48:42.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe97b5ba640 1 -- start start 2024-04-23T18:48:42.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe97b5ba640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9741010a0 0x7fe974198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:42.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe97b5ba640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe974106170 0x7fe974198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:42.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe97b5ba640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe97419cff0 0x7fe97419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:42.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe97b5ba640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe9741110b0 con 0x7fe974106170 2024-04-23T18:48:42.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe97b5ba640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe974110f30 con 0x7fe97419cff0 2024-04-23T18:48:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe97b5ba640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe974111230 con 0x7fe9741010a0 2024-04-23T18:48:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe978b55640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe974106170 0x7fe974198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe978b55640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe974106170 0x7fe974198cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37802/0 (socket says 172.21.15.50:37802) 2024-04-23T18:48:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe978b55640 1 -- 172.21.15.50:0/1508726513 learned_addr learned my addr 172.21.15.50:0/1508726513 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:48:42.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe979356640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9741010a0 0x7fe974198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:42.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe979b57640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe97419cff0 0x7fe97419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:42.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe978b55640 1 -- 172.21.15.50:0/1508726513 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9741010a0 msgr2=0x7fe974198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:42.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe978b55640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9741010a0 0x7fe974198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:42.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe978b55640 1 -- 172.21.15.50:0/1508726513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe97419cff0 msgr2=0x7fe97419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:42.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe978b55640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe97419cff0 0x7fe97419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:42.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.482+0000 7fe978b55640 1 -- 172.21.15.50:0/1508726513 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97419dae0 con 0x7fe974106170 2024-04-23T18:48:42.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.483+0000 7fe978b55640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe974106170 0x7fe974198cc0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fe96400e9a0 tx=0x7fe96400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:42.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.483+0000 7fe96a7fc640 1 -- 172.21.15.50:0/1508726513 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96400cd80 con 0x7fe974106170 2024-04-23T18:48:42.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.483+0000 7fe97b5ba640 1 -- 172.21.15.50:0/1508726513 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97419ddd0 con 0x7fe974106170 2024-04-23T18:48:42.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.483+0000 7fe96a7fc640 1 -- 172.21.15.50:0/1508726513 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9640035f0 con 0x7fe974106170 2024-04-23T18:48:42.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.483+0000 7fe96a7fc640 1 -- 172.21.15.50:0/1508726513 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe964010690 con 0x7fe974106170 2024-04-23T18:48:42.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.483+0000 7fe97b5ba640 1 -- 172.21.15.50:0/1508726513 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9741a5610 con 0x7fe974106170 2024-04-23T18:48:43.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.484+0000 7fe97b5ba640 1 -- 172.21.15.50:0/1508726513 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe974068d70 con 0x7fe974106170 2024-04-23T18:48:43.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.485+0000 7fe96a7fc640 1 -- 172.21.15.50:0/1508726513 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe964003ac0 con 0x7fe974106170 2024-04-23T18:48:43.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.486+0000 7fe96a7fc640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe9600a3d20 0x7fe9600a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:43.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.486+0000 7fe979356640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe9600a3d20 0x7fe9600a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:43.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.486+0000 7fe96a7fc640 1 -- 172.21.15.50:0/1508726513 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 8542+0+0 (secure 0 0 0) 0x7fe9640d0ce0 con 0x7fe974106170 2024-04-23T18:48:43.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.486+0000 7fe979356640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe9600a3d20 0x7fe9600a61e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe970009830 tx=0x7fe970030340 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:43.106 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:42 smithi188 ceph-mon[32694]: pgmap v962: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:43.115 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:42 smithi177 ceph-mon[32308]: pgmap v962: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:43.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.490+0000 7fe96a7fc640 1 -- 172.21.15.50:0/1508726513 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe96409d3a0 con 0x7fe974106170 2024-04-23T18:48:43.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.721+0000 7fe97b5ba640 1 -- 172.21.15.50:0/1508726513 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe974199a80 con 0x7fe974106170 2024-04-23T18:48:43.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.722+0000 7fe96a7fc640 1 -- 172.21.15.50:0/1508726513 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+11875 (secure 0 0 0) 0x7fe974199a80 con 0x7fe974106170 2024-04-23T18:48:43.154 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:48:43.163 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":66,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:42:13.881738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27194":{"gid":27194,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2754597719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2754597719},{"type":"v1","addr":"172.21.15.177:6837","nonce":2754597719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:48:43.173 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 66 2024-04-23T18:48:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.724+0000 7fe97b5ba640 1 -- 172.21.15.50:0/1508726513 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe9600a3d20 msgr2=0x7fe9600a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:43.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.724+0000 7fe97b5ba640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe9600a3d20 0x7fe9600a61e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe970009830 tx=0x7fe970030340 comp rx=0 tx=0).stop 2024-04-23T18:48:43.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.724+0000 7fe97b5ba640 1 -- 172.21.15.50:0/1508726513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe974106170 msgr2=0x7fe974198cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:43.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.724+0000 7fe97b5ba640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe974106170 0x7fe974198cc0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fe96400e9a0 tx=0x7fe96400eda0 comp rx=0 tx=0).stop 2024-04-23T18:48:43.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.724+0000 7fe97b5ba640 1 -- 172.21.15.50:0/1508726513 shutdown_connections 2024-04-23T18:48:43.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.724+0000 7fe97b5ba640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7fe9600a3d20 0x7fe9600a61e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:43.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.724+0000 7fe97b5ba640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe97419cff0 0x7fe97419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:43.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.724+0000 7fe97b5ba640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe974106170 0x7fe974198cc0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:43.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.724+0000 7fe97b5ba640 1 --2- 172.21.15.50:0/1508726513 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9741010a0 0x7fe974198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:43.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.724+0000 7fe97b5ba640 1 -- 172.21.15.50:0/1508726513 >> 172.21.15.50:0/1508726513 conn(0x7fe974077f50 msgr2=0x7fe9740694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:43.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.724+0000 7fe97b5ba640 1 -- 172.21.15.50:0/1508726513 shutdown_connections 2024-04-23T18:48:43.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:42.724+0000 7fe97b5ba640 1 -- 172.21.15.50:0/1508726513 wait complete. 2024-04-23T18:48:43.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:42 smithi050 ceph-mon[29187]: pgmap v962: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:43.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T18:48:43.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.576+0000 7f7255551640 1 -- 172.21.15.50:0/2992228180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7250103b50 msgr2=0x7f725010c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:43.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.576+0000 7f7255551640 1 --2- 172.21.15.50:0/2992228180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7250103b50 0x7f725010c030 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f7238009900 tx=0x7f723802a480 comp rx=0 tx=0).stop 2024-04-23T18:48:43.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.576+0000 7f7255551640 1 -- 172.21.15.50:0/2992228180 shutdown_connections 2024-04-23T18:48:43.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.576+0000 7f7255551640 1 --2- 172.21.15.50:0/2992228180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f725010c570 0x7f725010e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:43.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.576+0000 7f7255551640 1 --2- 172.21.15.50:0/2992228180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7250103b50 0x7f725010c030 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:43.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.576+0000 7f7255551640 1 --2- 172.21.15.50:0/2992228180 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7250103230 0x7f7250103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:43.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.576+0000 7f7255551640 1 -- 172.21.15.50:0/2992228180 >> 172.21.15.50:0/2992228180 conn(0x7f7250077fb0 msgr2=0x7f72500783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:43.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.576+0000 7f7255551640 1 -- 172.21.15.50:0/2992228180 shutdown_connections 2024-04-23T18:48:43.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.576+0000 7f7255551640 1 -- 172.21.15.50:0/2992228180 wait complete. 2024-04-23T18:48:43.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.577+0000 7f7255551640 1 Processor -- start 2024-04-23T18:48:43.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.577+0000 7f7255551640 1 -- start start 2024-04-23T18:48:43.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.577+0000 7f7255551640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250103230 0x7f72501986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:43.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.577+0000 7f7255551640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7250103b50 0x7f7250198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:43.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.577+0000 7f7255551640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f725010c570 0x7f725019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:43.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.577+0000 7f7255551640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7250112200 con 0x7f7250103b50 2024-04-23T18:48:43.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.577+0000 7f7255551640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7250112080 con 0x7f7250103230 2024-04-23T18:48:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.577+0000 7f7255551640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7250112380 con 0x7f725010c570 2024-04-23T18:48:43.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.577+0000 7f72467fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7250103b50 0x7f7250198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:43.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.577+0000 7f72467fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7250103b50 0x7f7250198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37820/0 (socket says 172.21.15.50:37820) 2024-04-23T18:48:43.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.577+0000 7f72467fc640 1 -- 172.21.15.50:0/73817757 learned_addr learned my addr 172.21.15.50:0/73817757 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:48:43.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f724f7fe640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f725010c570 0x7f725019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:43.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f724effd640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250103230 0x7f72501986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:43.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f72467fc640 1 -- 172.21.15.50:0/73817757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f725010c570 msgr2=0x7f725019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:43.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f72467fc640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f725010c570 0x7f725019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:43.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f72467fc640 1 -- 172.21.15.50:0/73817757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250103230 msgr2=0x7f72501986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:43.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f72467fc640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250103230 0x7f72501986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:43.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f72467fc640 1 -- 172.21.15.50:0/73817757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725019d6b0 con 0x7f7250103b50 2024-04-23T18:48:44.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f724effd640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250103230 0x7f72501986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:48:44.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f72467fc640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7250103b50 0x7f7250198c00 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f7238005ab0 tx=0x7f7238003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f724cff9640 1 -- 172.21.15.50:0/73817757 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723803f070 con 0x7f7250103b50 2024-04-23T18:48:44.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f724cff9640 1 -- 172.21.15.50:0/73817757 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7238031d60 con 0x7f7250103b50 2024-04-23T18:48:44.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f7255551640 1 -- 172.21.15.50:0/73817757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f725019d940 con 0x7f7250103b50 2024-04-23T18:48:44.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f724cff9640 1 -- 172.21.15.50:0/73817757 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238035030 con 0x7f7250103b50 2024-04-23T18:48:44.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.578+0000 7f7255551640 1 -- 172.21.15.50:0/73817757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72501a5230 con 0x7f7250103b50 2024-04-23T18:48:44.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.579+0000 7f7255551640 1 -- 172.21.15.50:0/73817757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72500691c0 con 0x7f7250103b50 2024-04-23T18:48:44.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.580+0000 7f724cff9640 1 -- 172.21.15.50:0/73817757 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7238042070 con 0x7f7250103b50 2024-04-23T18:48:44.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.581+0000 7f724cff9640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f723c0a3d20 0x7f723c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:44.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.581+0000 7f724cff9640 1 -- 172.21.15.50:0/73817757 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 8542+0+0 (secure 0 0 0) 0x7f72380f10e0 con 0x7f7250103b50 2024-04-23T18:48:44.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.581+0000 7f724effd640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f723c0a3d20 0x7f723c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.581+0000 7f724effd640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f723c0a3d20 0x7f723c0a61e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f724000baa0 tx=0x7f7240005d90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:44.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.586+0000 7f724cff9640 1 -- 172.21.15.50:0/73817757 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f72380bd810 con 0x7f7250103b50 2024-04-23T18:48:44.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:43 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/1508726513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:48:44.222 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:43 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/1508726513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:48:44.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.812+0000 7f7255551640 1 -- 172.21.15.50:0/73817757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7250107720 con 0x7f7250103b50 2024-04-23T18:48:44.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.812+0000 7f724cff9640 1 -- 172.21.15.50:0/73817757 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+11875 (secure 0 0 0) 0x7f72380bd110 con 0x7f7250103b50 2024-04-23T18:48:44.271 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:48:44.280 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/878281924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":878281924},{"type":"v1","addr":"172.21.15.50:6835","nonce":878281924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/1480144576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":1480144576},{"type":"v1","addr":"172.21.15.50:6837","nonce":1480144576}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2730935833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2730935833},{"type":"v1","addr":"172.21.15.50:6839","nonce":2730935833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6841/3198542689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6840","nonce":3198542689},{"type":"v1","addr":"172.21.15.50:6841","nonce":3198542689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/878222815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":878222815},{"type":"v1","addr":"172.21.15.177:6833","nonce":878222815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3630824399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3630824399},{"type":"v1","addr":"172.21.15.188:6833","nonce":3630824399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":66,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T18:16:33.316468+0000","modified":"2024-04-23T18:42:13.881738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24440":{"gid":24440,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/475975073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":475975073},{"type":"v1","addr":"172.21.15.177:6835","nonce":475975073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/1248812622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1248812622},{"type":"v1","addr":"172.21.15.188:6835","nonce":1248812622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.188:6837/2190889779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":2190889779},{"type":"v1","addr":"172.21.15.188:6837","nonce":2190889779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.188:6839/2202634301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":2202634301},{"type":"v1","addr":"172.21.15.188:6839","nonce":2202634301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6839/2936205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":2936205355},{"type":"v1","addr":"172.21.15.177:6839","nonce":2936205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27194":{"gid":27194,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2754597719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2754597719},{"type":"v1","addr":"172.21.15.177:6837","nonce":2754597719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24475,24506,24499]},"id":1}]} 2024-04-23T18:48:44.290 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 66 2024-04-23T18:48:44.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.815+0000 7f7255551640 1 -- 172.21.15.50:0/73817757 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f723c0a3d20 msgr2=0x7f723c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:44.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.815+0000 7f7255551640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f723c0a3d20 0x7f723c0a61e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f724000baa0 tx=0x7f7240005d90 comp rx=0 tx=0).stop 2024-04-23T18:48:44.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.815+0000 7f7255551640 1 -- 172.21.15.50:0/73817757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7250103b50 msgr2=0x7f7250198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:44.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.815+0000 7f7255551640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7250103b50 0x7f7250198c00 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f7238005ab0 tx=0x7f7238003cd0 comp rx=0 tx=0).stop 2024-04-23T18:48:44.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.815+0000 7f7255551640 1 -- 172.21.15.50:0/73817757 shutdown_connections 2024-04-23T18:48:44.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.815+0000 7f7255551640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f723c0a3d20 0x7f723c0a61e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:44.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.815+0000 7f7255551640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f725010c570 0x7f725019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:44.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.815+0000 7f7255551640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7250103b50 0x7f7250198c00 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:44.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.815+0000 7f7255551640 1 --2- 172.21.15.50:0/73817757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250103230 0x7f72501986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:44.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.815+0000 7f7255551640 1 -- 172.21.15.50:0/73817757 >> 172.21.15.50:0/73817757 conn(0x7f7250077fb0 msgr2=0x7f72500fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.815+0000 7f7255551640 1 -- 172.21.15.50:0/73817757 shutdown_connections 2024-04-23T18:48:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:43.815+0000 7f7255551640 1 -- 172.21.15.50:0/73817757 wait complete. 2024-04-23T18:48:44.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:43 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/1508726513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:48:44.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T18:48:44.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.684+0000 7f456e7db640 1 -- 172.21.15.50:0/190751535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4568107280 msgr2=0x7f456810ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:44.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.684+0000 7f456e7db640 1 --2- 172.21.15.50:0/190751535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4568107280 0x7f456810ee50 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f4560009ca0 tx=0x7f456002a8c0 comp rx=0 tx=0).stop 2024-04-23T18:48:44.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.684+0000 7f456e7db640 1 -- 172.21.15.50:0/190751535 shutdown_connections 2024-04-23T18:48:44.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.684+0000 7f456e7db640 1 --2- 172.21.15.50:0/190751535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4568107280 0x7f456810ee50 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:44.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.684+0000 7f456e7db640 1 --2- 172.21.15.50:0/190751535 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4568102bf0 0x7f4568106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:44.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.684+0000 7f456e7db640 1 --2- 172.21.15.50:0/190751535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4568102240 0x7f4568102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:44.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.684+0000 7f456e7db640 1 -- 172.21.15.50:0/190751535 >> 172.21.15.50:0/190751535 conn(0x7f45680790e0 msgr2=0x7f45680794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:44.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.684+0000 7f456e7db640 1 -- 172.21.15.50:0/190751535 shutdown_connections 2024-04-23T18:48:44.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.684+0000 7f456e7db640 1 -- 172.21.15.50:0/190751535 wait complete. 2024-04-23T18:48:44.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.685+0000 7f456e7db640 1 Processor -- start 2024-04-23T18:48:44.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.685+0000 7f456e7db640 1 -- start start 2024-04-23T18:48:44.804 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:44 smithi177 ceph-mon[32308]: pgmap v963: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:48:44.864 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:44 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/73817757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:48:44.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:44 smithi188 ceph-mon[32694]: pgmap v963: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:48:44.882 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:44 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/73817757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:48:44.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.685+0000 7f456e7db640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4568102240 0x7f45681986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:44.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.685+0000 7f456e7db640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4568107280 0x7f4568198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:44.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.685+0000 7f456e7db640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f456819cfa0 0x7f456819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:44.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.685+0000 7f456e7db640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f45681121e0 con 0x7f4568107280 2024-04-23T18:48:44.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.685+0000 7f456e7db640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4568112060 con 0x7f456819cfa0 2024-04-23T18:48:44.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.685+0000 7f456e7db640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4568112360 con 0x7f4568102240 2024-04-23T18:48:44.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.685+0000 7f45677fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4568107280 0x7f4568198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:44.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.685+0000 7f45677fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4568107280 0x7f4568198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37834/0 (socket says 172.21.15.50:37834) 2024-04-23T18:48:44.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.685+0000 7f45677fe640 1 -- 172.21.15.50:0/659850509 learned_addr learned my addr 172.21.15.50:0/659850509 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:48:44.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.685+0000 7f4567fff640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4568102240 0x7f45681986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:44.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f456cd78640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f456819cfa0 0x7f456819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f45677fe640 1 -- 172.21.15.50:0/659850509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4568102240 msgr2=0x7f45681986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f45677fe640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4568102240 0x7f45681986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f45677fe640 1 -- 172.21.15.50:0/659850509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f456819cfa0 msgr2=0x7f456819d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:45.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f45677fe640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f456819cfa0 0x7f456819d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:45.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f45677fe640 1 -- 172.21.15.50:0/659850509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f456819da90 con 0x7f4568107280 2024-04-23T18:48:45.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f456cd78640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f456819cfa0 0x7f456819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:48:45.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f45677fe640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4568107280 0x7f4568198c20 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f455c00d950 tx=0x7f455c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:45.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f45657fa640 1 -- 172.21.15.50:0/659850509 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455c017070 con 0x7f4568107280 2024-04-23T18:48:45.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f45657fa640 1 -- 172.21.15.50:0/659850509 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f455c009c30 con 0x7f4568107280 2024-04-23T18:48:45.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f456e7db640 1 -- 172.21.15.50:0/659850509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f456819dd80 con 0x7f4568107280 2024-04-23T18:48:45.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f45657fa640 1 -- 172.21.15.50:0/659850509 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455c0047a0 con 0x7f4568107280 2024-04-23T18:48:45.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.686+0000 7f456e7db640 1 -- 172.21.15.50:0/659850509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45681a55c0 con 0x7f4568107280 2024-04-23T18:48:45.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.687+0000 7f456e7db640 1 -- 172.21.15.50:0/659850509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45680691c0 con 0x7f4568107280 2024-04-23T18:48:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.688+0000 7f45657fa640 1 -- 172.21.15.50:0/659850509 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f455c004020 con 0x7f4568107280 2024-04-23T18:48:45.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.689+0000 7f45657fa640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f45500a3da0 0x7f45500a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:45.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.689+0000 7f45657fa640 1 -- 172.21.15.50:0/659850509 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 8542+0+0 (secure 0 0 0) 0x7f455c0d03d0 con 0x7f4568107280 2024-04-23T18:48:45.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.689+0000 7f4567fff640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f45500a3da0 0x7f45500a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:45.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.690+0000 7f4567fff640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f45500a3da0 0x7f45500a6260 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4558009bb0 tx=0x7f4558005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:45.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.694+0000 7f45657fa640 1 -- 172.21.15.50:0/659850509 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f455c020020 con 0x7f4568107280 2024-04-23T18:48:45.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.881+0000 7f456e7db640 1 -- 172.21.15.50:0/659850509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4568102bf0 con 0x7f4568107280 2024-04-23T18:48:45.313 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.882+0000 7f45657fa640 1 -- 172.21.15.50:0/659850509 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) ==== 74+0+23132 (secure 0 0 0) 0x7f455c09ca00 con 0x7f4568107280 2024-04-23T18:48:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-23T18:48:45.405 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":144,"fsid":"5d677974-019c-11ef-bc93-c7b262605968","created":"2024-04-23T18:08:37.569945+0000","modified":"2024-04-23T18:48:16.345905+0000","last_up_change":"2024-04-23T18:15:45.903571+0000","last_in_change":"2024-04-23T18:15:26.540285+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T18:12:02.493647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T18:16:31.679148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T18:16:32.269478+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":55,"snap_epoch":143,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3cb7067-1f57-4e79-a884-bfae449d5508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6803","nonce":4125602948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6805","nonce":4125602948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6809","nonce":4125602948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":4125602948},{"type":"v1","addr":"172.21.15.50:6807","nonce":4125602948}]},"public_addr":"172.21.15.50:6803/4125602948","cluster_addr":"172.21.15.50:6805/4125602948","heartbeat_back_addr":"172.21.15.50:6809/4125602948","heartbeat_front_addr":"172.21.15.50:6807/4125602948","state":["exists","up"]},{"osd":1,"uuid":"1b334ea9-1d7a-49b5-92bc-01416987b3ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6801","nonce":3726952435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6803","nonce":3726952435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6807","nonce":3726952435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3726952435},{"type":"v1","addr":"172.21.15.177:6805","nonce":3726952435}]},"public_addr":"172.21.15.177:6801/3726952435","cluster_addr":"172.21.15.177:6803/3726952435","heartbeat_back_addr":"172.21.15.177:6807/3726952435","heartbeat_front_addr":"172.21.15.177:6805/3726952435","state":["exists","up"]},{"osd":2,"uuid":"3a46cc56-b2f7-431d-87c3-5160a3db721d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6801","nonce":2182491369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6803","nonce":2182491369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6807","nonce":2182491369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2182491369},{"type":"v1","addr":"172.21.15.188:6805","nonce":2182491369}]},"public_addr":"172.21.15.188:6801/2182491369","cluster_addr":"172.21.15.188:6803/2182491369","heartbeat_back_addr":"172.21.15.188:6807/2182491369","heartbeat_front_addr":"172.21.15.188:6805/2182491369","state":["exists","up"]},{"osd":3,"uuid":"02828390-a4c5-468a-8425-092eb1513bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6811","nonce":3121149303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6813","nonce":3121149303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6817","nonce":3121149303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3121149303},{"type":"v1","addr":"172.21.15.50:6815","nonce":3121149303}]},"public_addr":"172.21.15.50:6811/3121149303","cluster_addr":"172.21.15.50:6813/3121149303","heartbeat_back_addr":"172.21.15.50:6817/3121149303","heartbeat_front_addr":"172.21.15.50:6815/3121149303","state":["exists","up"]},{"osd":4,"uuid":"88f4c3e4-72bc-459f-a052-6a41dd0a0e73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6809","nonce":1367742686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6811","nonce":1367742686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6815","nonce":1367742686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1367742686},{"type":"v1","addr":"172.21.15.177:6813","nonce":1367742686}]},"public_addr":"172.21.15.177:6809/1367742686","cluster_addr":"172.21.15.177:6811/1367742686","heartbeat_back_addr":"172.21.15.177:6815/1367742686","heartbeat_front_addr":"172.21.15.177:6813/1367742686","state":["exists","up"]},{"osd":5,"uuid":"1fee2f23-6eec-465e-a935-970aa8ded943","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6809","nonce":56520768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6811","nonce":56520768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6815","nonce":56520768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":56520768},{"type":"v1","addr":"172.21.15.188:6813","nonce":56520768}]},"public_addr":"172.21.15.188:6809/56520768","cluster_addr":"172.21.15.188:6811/56520768","heartbeat_back_addr":"172.21.15.188:6815/56520768","heartbeat_front_addr":"172.21.15.188:6813/56520768","state":["exists","up"]},{"osd":6,"uuid":"4a24ed7a-b3d4-4cb0-8d06-5b356a8f1b5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6819","nonce":4290265415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6821","nonce":4290265415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6825","nonce":4290265415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":4290265415},{"type":"v1","addr":"172.21.15.50:6823","nonce":4290265415}]},"public_addr":"172.21.15.50:6819/4290265415","cluster_addr":"172.21.15.50:6821/4290265415","heartbeat_back_addr":"172.21.15.50:6825/4290265415","heartbeat_front_addr":"172.21.15.50:6823/4290265415","state":["exists","up"]},{"osd":7,"uuid":"b6d24ee9-5db8-439b-907e-c5f2c7c4658a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6817","nonce":1787147913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6819","nonce":1787147913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6823","nonce":1787147913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1787147913},{"type":"v1","addr":"172.21.15.177:6821","nonce":1787147913}]},"public_addr":"172.21.15.177:6817/1787147913","cluster_addr":"172.21.15.177:6819/1787147913","heartbeat_back_addr":"172.21.15.177:6823/1787147913","heartbeat_front_addr":"172.21.15.177:6821/1787147913","state":["exists","up"]},{"osd":8,"uuid":"cb3e8deb-e2b5-479b-aed3-5decbd2b9044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6817","nonce":4982360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6819","nonce":4982360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6823","nonce":4982360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":4982360},{"type":"v1","addr":"172.21.15.188:6821","nonce":4982360}]},"public_addr":"172.21.15.188:6817/4982360","cluster_addr":"172.21.15.188:6819/4982360","heartbeat_back_addr":"172.21.15.188:6823/4982360","heartbeat_front_addr":"172.21.15.188:6821/4982360","state":["exists","up"]},{"osd":9,"uuid":"46facc79-031d-4199-a0c0-e551f82182fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6827","nonce":1497125791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6829","nonce":1497125791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6833","nonce":1497125791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1497125791},{"type":"v1","addr":"172.21.15.50:6831","nonce":1497125791}]},"public_addr":"172.21.15.50:6827/1497125791","cluster_addr":"172.21.15.50:6829/1497125791","heartbeat_back_addr":"172.21.15.50:6833/1497125791","heartbeat_front_addr":"172.21.15.50:6831/1497125791","state":["exists","up"]},{"osd":10,"uuid":"b7af2e9f-fa4b-43d7-9567-013393024176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6825","nonce":1988770803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6827","nonce":1988770803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6831","nonce":1988770803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1988770803},{"type":"v1","addr":"172.21.15.177:6829","nonce":1988770803}]},"public_addr":"172.21.15.177:6825/1988770803","cluster_addr":"172.21.15.177:6827/1988770803","heartbeat_back_addr":"172.21.15.177:6831/1988770803","heartbeat_front_addr":"172.21.15.177:6829/1988770803","state":["exists","up"]},{"osd":11,"uuid":"40d6c591-e063-4b2f-ae02-58b45f6935cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6825","nonce":854484853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6827","nonce":854484853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6831","nonce":854484853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":854484853},{"type":"v1","addr":"172.21.15.188:6829","nonce":854484853}]},"public_addr":"172.21.15.188:6825/854484853","cluster_addr":"172.21.15.188:6827/854484853","heartbeat_back_addr":"172.21.15.188:6831/854484853","heartbeat_front_addr":"172.21.15.188:6829/854484853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:10.601909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:34.851183+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:11:59.312392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:23.982541+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:12:48.760433+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:13.102845+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:13:38.399002+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:03.377442+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:28.301421+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:14:53.643417+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:19.096171+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T18:15:44.650938+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/1479623929":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/3248139242":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6800/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/2484340243":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6800/103327972":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/1510580864":"2024-04-24T18:09:34.403529+0000","172.21.15.50:0/444338023":"2024-04-24T18:18:30.504131+0000","172.21.15.50:6801/1004018784":"2024-04-24T18:18:30.504131+0000","172.21.15.50:0/587528851":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/688464416":"2024-04-24T18:09:07.548503+0000","172.21.15.50:6800/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/1448760865":"2024-04-24T18:09:34.403529+0000","172.21.15.50:6801/1300452320":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/3023139440":"2024-04-24T18:09:07.548503+0000","172.21.15.50:0/2673950234":"2024-04-24T18:09:34.403529+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":42,"length":1},{"begin":55,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T18:48:45.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.884+0000 7f456e7db640 1 -- 172.21.15.50:0/659850509 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f45500a3da0 msgr2=0x7f45500a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:45.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.884+0000 7f456e7db640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f45500a3da0 0x7f45500a6260 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4558009bb0 tx=0x7f4558005f90 comp rx=0 tx=0).stop 2024-04-23T18:48:45.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.884+0000 7f456e7db640 1 -- 172.21.15.50:0/659850509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4568107280 msgr2=0x7f4568198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:45.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.884+0000 7f456e7db640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4568107280 0x7f4568198c20 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f455c00d950 tx=0x7f455c00de10 comp rx=0 tx=0).stop 2024-04-23T18:48:45.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.885+0000 7f456e7db640 1 -- 172.21.15.50:0/659850509 shutdown_connections 2024-04-23T18:48:45.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.885+0000 7f456e7db640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f45500a3da0 0x7f45500a6260 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:45.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.885+0000 7f456e7db640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f456819cfa0 0x7f456819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:45.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.885+0000 7f456e7db640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4568107280 0x7f4568198c20 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:45.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.885+0000 7f456e7db640 1 --2- 172.21.15.50:0/659850509 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4568102240 0x7f45681986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:45.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.885+0000 7f456e7db640 1 -- 172.21.15.50:0/659850509 >> 172.21.15.50:0/659850509 conn(0x7f45680790e0 msgr2=0x7f45681001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:45.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.885+0000 7f456e7db640 1 -- 172.21.15.50:0/659850509 shutdown_connections 2024-04-23T18:48:45.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:44.885+0000 7f456e7db640 1 -- 172.21.15.50:0/659850509 wait complete. 2024-04-23T18:48:45.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:44 smithi050 ceph-mon[29187]: pgmap v963: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T18:48:45.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:44 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/73817757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T18:48:45.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-23T18:48:45.888 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:45 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/659850509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:48:45.895 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:45 smithi188 ceph-mon[32694]: pgmap v964: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:45.903 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:45 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/659850509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:48:45.912 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:45 smithi177 ceph-mon[32308]: pgmap v964: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:45.946 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:45 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/659850509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T18:48:45.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:45 smithi050 ceph-mon[29187]: pgmap v964: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T18:48:46.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.070+0000 7f23009ef640 1 -- 172.21.15.50:0/2111215080 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22fc10c600 msgr2=0x7f22fc10ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:46.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.070+0000 7f23009ef640 1 --2- 172.21.15.50:0/2111215080 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22fc10c600 0x7f22fc10ea50 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009ca0 tx=0x7f22f402a950 comp rx=0 tx=0).stop 2024-04-23T18:48:46.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.070+0000 7f23009ef640 1 -- 172.21.15.50:0/2111215080 shutdown_connections 2024-04-23T18:48:46.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.070+0000 7f23009ef640 1 --2- 172.21.15.50:0/2111215080 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22fc10c600 0x7f22fc10ea50 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:46.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.070+0000 7f23009ef640 1 --2- 172.21.15.50:0/2111215080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22fc103b50 0x7f22fc10c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:46.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.070+0000 7f23009ef640 1 --2- 172.21.15.50:0/2111215080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22fc103230 0x7f22fc103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:46.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.070+0000 7f23009ef640 1 -- 172.21.15.50:0/2111215080 >> 172.21.15.50:0/2111215080 conn(0x7f22fc077f80 msgr2=0x7f22fc078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:46.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.071+0000 7f23009ef640 1 -- 172.21.15.50:0/2111215080 shutdown_connections 2024-04-23T18:48:46.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.071+0000 7f23009ef640 1 -- 172.21.15.50:0/2111215080 wait complete. 2024-04-23T18:48:46.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.071+0000 7f23009ef640 1 Processor -- start 2024-04-23T18:48:46.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.071+0000 7f23009ef640 1 -- start start 2024-04-23T18:48:46.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f23009ef640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22fc103230 0x7f22fc100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:46.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f23009ef640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22fc103b50 0x7f22fc1013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:46.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f23009ef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22fc10c600 0x7f22fc101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:46.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f23009ef640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc112210 con 0x7f22fc103b50 2024-04-23T18:48:46.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f23009ef640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc112090 con 0x7f22fc10c600 2024-04-23T18:48:46.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f23009ef640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc112390 con 0x7f22fc103230 2024-04-23T18:48:46.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22f3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22fc103b50 0x7f22fc1013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:46.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22f3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22fc103b50 0x7f22fc1013c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37860/0 (socket says 172.21.15.50:37860) 2024-04-23T18:48:46.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22f3fff640 1 -- 172.21.15.50:0/505092894 learned_addr learned my addr 172.21.15.50:0/505092894 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:48:46.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22fad9d640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22fc10c600 0x7f22fc101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:46.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22fa59c640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22fc103230 0x7f22fc100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:46.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22f3fff640 1 -- 172.21.15.50:0/505092894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22fc103230 msgr2=0x7f22fc100e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:46.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22f3fff640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22fc103230 0x7f22fc100e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:46.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22f3fff640 1 -- 172.21.15.50:0/505092894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22fc10c600 msgr2=0x7f22fc101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:46.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22f3fff640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22fc10c600 0x7f22fc101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:46.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22f3fff640 1 -- 172.21.15.50:0/505092894 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22fc1a6d10 con 0x7f22fc103b50 2024-04-23T18:48:46.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22fa59c640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22fc103230 0x7f22fc100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T18:48:46.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22f3fff640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22fc103b50 0x7f22fc1013c0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f22e400d950 tx=0x7f22e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:46.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22f37fe640 1 -- 172.21.15.50:0/505092894 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e4017070 con 0x7f22fc103b50 2024-04-23T18:48:46.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22f37fe640 1 -- 172.21.15.50:0/505092894 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22e4009c30 con 0x7f22fc103b50 2024-04-23T18:48:46.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f23009ef640 1 -- 172.21.15.50:0/505092894 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22fc1a6fa0 con 0x7f22fc103b50 2024-04-23T18:48:46.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.072+0000 7f22f37fe640 1 -- 172.21.15.50:0/505092894 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e400f620 con 0x7f22fc103b50 2024-04-23T18:48:46.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.073+0000 7f23009ef640 1 -- 172.21.15.50:0/505092894 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22fc1a74c0 con 0x7f22fc103b50 2024-04-23T18:48:46.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.074+0000 7f23009ef640 1 -- 172.21.15.50:0/505092894 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22fc06add0 con 0x7f22fc103b50 2024-04-23T18:48:46.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.075+0000 7f22f37fe640 1 -- 172.21.15.50:0/505092894 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f22e4010460 con 0x7f22fc103b50 2024-04-23T18:48:46.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.075+0000 7f22f37fe640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22e80a3d20 0x7f22e80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:46.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.076+0000 7f22fa59c640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22e80a3d20 0x7f22e80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:46.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.076+0000 7f22f37fe640 1 -- 172.21.15.50:0/505092894 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 8542+0+0 (secure 0 0 0) 0x7f22e40d0480 con 0x7f22fc103b50 2024-04-23T18:48:46.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.076+0000 7f22fa59c640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22e80a3d20 0x7f22e80a61e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009b50 tx=0x7f22ec005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:46.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.080+0000 7f22f37fe640 1 -- 172.21.15.50:0/505092894 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f22e409cb10 con 0x7f22fc103b50 2024-04-23T18:48:46.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.308+0000 7f23009ef640 1 -- 172.21.15.50:0/505092894 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f22fc1076f0 con 0x7f22fc103b50 2024-04-23T18:48:46.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.311+0000 7f22f37fe640 1 -- 172.21.15.50:0/505092894 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) ==== 161+0+0 (secure 0 0 0) 0x7f22e4009de0 con 0x7f22fc103b50 2024-04-23T18:48:46.679 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-23T18:48:46.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.313+0000 7f23009ef640 1 -- 172.21.15.50:0/505092894 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22e80a3d20 msgr2=0x7f22e80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:46.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.313+0000 7f23009ef640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22e80a3d20 0x7f22e80a61e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009b50 tx=0x7f22ec005f90 comp rx=0 tx=0).stop 2024-04-23T18:48:46.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.313+0000 7f23009ef640 1 -- 172.21.15.50:0/505092894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22fc103b50 msgr2=0x7f22fc1013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:46.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.313+0000 7f23009ef640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22fc103b50 0x7f22fc1013c0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f22e400d950 tx=0x7f22e400de10 comp rx=0 tx=0).stop 2024-04-23T18:48:46.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.314+0000 7f23009ef640 1 -- 172.21.15.50:0/505092894 shutdown_connections 2024-04-23T18:48:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.314+0000 7f23009ef640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f22e80a3d20 0x7f22e80a61e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:46.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.314+0000 7f23009ef640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22fc10c600 0x7f22fc101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:46.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.314+0000 7f23009ef640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22fc103b50 0x7f22fc1013c0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:46.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.314+0000 7f23009ef640 1 --2- 172.21.15.50:0/505092894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22fc103230 0x7f22fc100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:46.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.314+0000 7f23009ef640 1 -- 172.21.15.50:0/505092894 >> 172.21.15.50:0/505092894 conn(0x7f22fc077f80 msgr2=0x7f22fc0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:46.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.314+0000 7f23009ef640 1 -- 172.21.15.50:0/505092894 shutdown_connections 2024-04-23T18:48:46.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:46.314+0000 7f23009ef640 1 -- 172.21.15.50:0/505092894 wait complete. 2024-04-23T18:48:46.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-23T18:48:46.915 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-c[63700]: ignoring --setuser ceph since I am not root 2024-04-23T18:48:46.924 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-c[63700]: ignoring --setgroup ceph since I am not root 2024-04-23T18:48:46.952 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-mds[63704]: -- 172.21.15.188:0/790121461 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559f81701a40 con 0x559f816d4800 2024-04-23T18:48:46.960 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-c[63700]: starting mds.c at 2024-04-23T18:48:47.034 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/505092894' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T18:48:47.044 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-mon[32308]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T18:48:47.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-mon[32308]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T18:48:47.062 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/505092894' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T18:48:47.069 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-mon[32308]: osdmap e145: 12 total, 12 up, 12 in 2024-04-23T18:48:47.103 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-mon[32308]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-23T18:48:47.113 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:46 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a[29183]: 2024-04-23T18:48:46.309+0000 7f932c8cb700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T18:48:47.122 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:46 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/505092894' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T18:48:47.140 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:46 smithi050 ceph-mon[29187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T18:48:47.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:46 smithi050 ceph-mon[29187]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T18:48:47.162 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:46 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/505092894' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T18:48:47.176 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:46 smithi050 ceph-mon[29187]: osdmap e145: 12 total, 12 up, 12 in 2024-04-23T18:48:47.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:46 smithi050 ceph-mon[29187]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-23T18:48:47.196 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-i[64020]: ignoring --setuser ceph since I am not root 2024-04-23T18:48:47.206 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-i[64020]: ignoring --setgroup ceph since I am not root 2024-04-23T18:48:47.240 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-mds[64024]: -- 172.21.15.188:0/418289131 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563bf6f5da40 con 0x563bf6f30800 2024-04-23T18:48:47.247 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-i[64020]: starting mds.i at 2024-04-23T18:48:47.300 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: ignoring --setuser ceph since I am not root 2024-04-23T18:48:47.309 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: ignoring --setgroup ceph since I am not root 2024-04-23T18:48:47.316 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-mds[66745]: -- 172.21.15.177:0/260898337 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a1aa321a40 con 0x55a1aa2f4800 2024-04-23T18:48:47.324 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: starting mds.h at 2024-04-23T18:48:47.332 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/505092894' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T18:48:47.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-mon[32694]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T18:48:47.382 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-mon[32694]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T18:48:47.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/505092894' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T18:48:47.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-mon[32694]: osdmap e145: 12 total, 12 up, 12 in 2024-04-23T18:48:47.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-mon[32694]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-23T18:48:47.412 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b[63024]: ignoring --setuser ceph since I am not root 2024-04-23T18:48:47.420 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b[63024]: ignoring --setgroup ceph since I am not root 2024-04-23T18:48:47.428 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-mds[63028]: -- 172.21.15.177:0/18162895 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a457b0ba40 con 0x55a457adec00 2024-04-23T18:48:47.436 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b[63024]: starting mds.b at 2024-04-23T18:48:47.445 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f[63391]: ignoring --setuser ceph since I am not root 2024-04-23T18:48:47.479 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f[63391]: ignoring --setgroup ceph since I am not root 2024-04-23T18:48:47.487 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-mds[63395]: -- 172.21.15.188:0/1587258040 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55959aacba40 con 0x55959aa9ec00 2024-04-23T18:48:47.544 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f[63391]: starting mds.f at 2024-04-23T18:48:47.553 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:48:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f[63391]: 2024-04-23T18:48:46.399+0000 7f8a4bcb4700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6835/0: (98) Address already in use 2024-04-23T18:48:47.562 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e[62392]: ignoring --setuser ceph since I am not root 2024-04-23T18:48:47.574 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e[62392]: ignoring --setgroup ceph since I am not root 2024-04-23T18:48:47.582 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-mds[62396]: -- 172.21.15.177:0/1373717065 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5600feebba40 con 0x5600fee8f000 2024-04-23T18:48:47.603 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:48:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e[62392]: starting mds.e at 2024-04-23T18:48:47.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.324+0000 7ff18c164640 1 -- 172.21.15.50:0/1244337004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841061e0 msgr2=0x7ff18410ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:47.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.324+0000 7ff18c164640 1 --2- 172.21.15.50:0/1244337004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841061e0 0x7ff18410ddb0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009ca0 tx=0x7ff17c02a8c0 comp rx=0 tx=0).stop 2024-04-23T18:48:47.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.324+0000 7ff18c164640 1 -- 172.21.15.50:0/1244337004 shutdown_connections 2024-04-23T18:48:47.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.324+0000 7ff18c164640 1 --2- 172.21.15.50:0/1244337004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841061e0 0x7ff18410ddb0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:47.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.324+0000 7ff18c164640 1 --2- 172.21.15.50:0/1244337004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184101ac0 0x7ff184105b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:47.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.324+0000 7ff18c164640 1 --2- 172.21.15.50:0/1244337004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff184101110 0x7ff1841014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:47.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.324+0000 7ff18c164640 1 -- 172.21.15.50:0/1244337004 >> 172.21.15.50:0/1244337004 conn(0x7ff184077f80 msgr2=0x7ff184078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:47.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.324+0000 7ff18c164640 1 -- 172.21.15.50:0/1244337004 shutdown_connections 2024-04-23T18:48:47.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.324+0000 7ff18c164640 1 -- 172.21.15.50:0/1244337004 wait complete. 2024-04-23T18:48:47.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.324+0000 7ff18c164640 1 Processor -- start 2024-04-23T18:48:47.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.324+0000 7ff18c164640 1 -- start start 2024-04-23T18:48:47.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18c164640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184101110 0x7ff18419e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:47.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18c164640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1841061e0 0x7ff18419e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:47.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18c164640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841985e0 0x7ff184198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:47.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18c164640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff184111150 con 0x7ff1841985e0 2024-04-23T18:48:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18c164640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff184110fd0 con 0x7ff184101110 2024-04-23T18:48:47.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18c164640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff1841112d0 con 0x7ff1841061e0 2024-04-23T18:48:47.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18a701640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841985e0 0x7ff184198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18a701640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841985e0 0x7ff184198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37874/0 (socket says 172.21.15.50:37874) 2024-04-23T18:48:47.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18a701640 1 -- 172.21.15.50:0/4220408576 learned_addr learned my addr 172.21.15.50:0/4220408576 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:48:47.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff1896ff640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1841061e0 0x7ff18419e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:47.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff189f00640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184101110 0x7ff18419e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:47.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18a701640 1 -- 172.21.15.50:0/4220408576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1841061e0 msgr2=0x7ff18419e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:47.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18a701640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1841061e0 0x7ff18419e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:47.896 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18a701640 1 -- 172.21.15.50:0/4220408576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184101110 msgr2=0x7ff18419e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:47.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18a701640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184101110 0x7ff18419e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:47.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18a701640 1 -- 172.21.15.50:0/4220408576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff184199100 con 0x7ff1841985e0 2024-04-23T18:48:47.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff189f00640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184101110 0x7ff18419e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T18:48:48.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff18a701640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841985e0 0x7ff184198a90 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff17c005cc0 tx=0x7ff17c031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:48.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff17affd640 1 -- 172.21.15.50:0/4220408576 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c003b70 con 0x7ff1841985e0 2024-04-23T18:48:48.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.325+0000 7ff17affd640 1 -- 172.21.15.50:0/4220408576 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff17c0318f0 con 0x7ff1841985e0 2024-04-23T18:48:48.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.326+0000 7ff18c164640 1 -- 172.21.15.50:0/4220408576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff184199390 con 0x7ff1841985e0 2024-04-23T18:48:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.326+0000 7ff17affd640 1 -- 172.21.15.50:0/4220408576 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c02f950 con 0x7ff1841985e0 2024-04-23T18:48:48.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.326+0000 7ff18c164640 1 -- 172.21.15.50:0/4220408576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1841a5630 con 0x7ff1841985e0 2024-04-23T18:48:48.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.327+0000 7ff18c164640 1 -- 172.21.15.50:0/4220408576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1840684e0 con 0x7ff1841985e0 2024-04-23T18:48:48.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.328+0000 7ff17affd640 1 -- 172.21.15.50:0/4220408576 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff17c02fb10 con 0x7ff1841985e0 2024-04-23T18:48:48.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.328+0000 7ff17affd640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff1700a3da0 0x7ff1700a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:48.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.329+0000 7ff17affd640 1 -- 172.21.15.50:0/4220408576 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 8756+0+0 (secure 0 0 0) 0x7ff17c0f1b50 con 0x7ff1841985e0 2024-04-23T18:48:48.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.329+0000 7ff189f00640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff1700a3da0 0x7ff1700a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:48.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.329+0000 7ff189f00640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff1700a3da0 0x7ff1700a6260 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff180009c20 tx=0x7ff180005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:48.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.333+0000 7ff17affd640 1 -- 172.21.15.50:0/4220408576 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff17c0be1b0 con 0x7ff1841985e0 2024-04-23T18:48:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:47.559+0000 7ff18c164640 1 -- 172.21.15.50:0/4220408576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff18410a420 con 0x7ff1841985e0 2024-04-23T18:48:48.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.495+0000 7ff17affd640 1 -- 172.21.15.50:0/4220408576 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v69) ==== 104+0+0 (secure 0 0 0) 0x7ff17c0de0f0 con 0x7ff1841985e0 2024-04-23T18:48:48.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.498+0000 7ff18c164640 1 -- 172.21.15.50:0/4220408576 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff1700a3da0 msgr2=0x7ff1700a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:48.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.498+0000 7ff18c164640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff1700a3da0 0x7ff1700a6260 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff180009c20 tx=0x7ff180005f90 comp rx=0 tx=0).stop 2024-04-23T18:48:48.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.498+0000 7ff18c164640 1 -- 172.21.15.50:0/4220408576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841985e0 msgr2=0x7ff184198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.498+0000 7ff18c164640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841985e0 0x7ff184198a90 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff17c005cc0 tx=0x7ff17c031700 comp rx=0 tx=0).stop 2024-04-23T18:48:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.498+0000 7ff18c164640 1 -- 172.21.15.50:0/4220408576 shutdown_connections 2024-04-23T18:48:48.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.498+0000 7ff18c164640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7ff1700a3da0 0x7ff1700a6260 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.498+0000 7ff18c164640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841985e0 0x7ff184198a90 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.498+0000 7ff18c164640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1841061e0 0x7ff18419e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:48.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.498+0000 7ff18c164640 1 --2- 172.21.15.50:0/4220408576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff184101110 0x7ff18419e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.498+0000 7ff18c164640 1 -- 172.21.15.50:0/4220408576 >> 172.21.15.50:0/4220408576 conn(0x7ff184077f80 msgr2=0x7ff18406a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:48.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.498+0000 7ff18c164640 1 -- 172.21.15.50:0/4220408576 shutdown_connections 2024-04-23T18:48:48.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:48.498+0000 7ff18c164640 1 -- 172.21.15.50:0/4220408576 wait complete. 2024-04-23T18:48:48.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-23T18:48:48.695 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: pgmap v966: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-23T18:48:48.702 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6835/4231634718,v1:172.21.15.177:6838/4231634718] up:boot 2024-04-23T18:48:48.709 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6834/1283448256,v1:172.21.15.177:6837/1283448256] up:boot 2024-04-23T18:48:48.716 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.177:6836/3029152065,v1:172.21.15.177:6839/3029152065] up:boot 2024-04-23T18:48:48.748 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.188:6834/3960668816,v1:172.21.15.188:6837/3960668816] up:boot 2024-04-23T18:48:48.756 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.188:6835/1145985046,v1:172.21.15.188:6838/1145985046] up:boot 2024-04-23T18:48:48.808 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: mds.? [v2:172.21.15.188:6836/450211997,v1:172.21.15.188:6839/450211997] up:boot 2024-04-23T18:48:48.826 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-23T18:48:48.838 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:48:48.847 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:48:48.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:48:48.886 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:48:48.897 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:48:48.905 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:48:48.917 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:48 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4220408576' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:48:48.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: pgmap v966: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-23T18:48:48.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6835/4231634718,v1:172.21.15.177:6838/4231634718] up:boot 2024-04-23T18:48:48.941 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6834/1283448256,v1:172.21.15.177:6837/1283448256] up:boot 2024-04-23T18:48:48.950 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.177:6836/3029152065,v1:172.21.15.177:6839/3029152065] up:boot 2024-04-23T18:48:48.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.188:6834/3960668816,v1:172.21.15.188:6837/3960668816] up:boot 2024-04-23T18:48:48.969 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.188:6835/1145985046,v1:172.21.15.188:6838/1145985046] up:boot 2024-04-23T18:48:49.004 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: mds.? [v2:172.21.15.188:6836/450211997,v1:172.21.15.188:6839/450211997] up:boot 2024-04-23T18:48:49.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-23T18:48:49.077 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:48:49.087 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:48:49.099 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:48:49.109 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:48:49.118 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:48:49.145 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:48:49.154 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:48 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4220408576' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:48:49.164 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: pgmap v966: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-23T18:48:49.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6835/4231634718,v1:172.21.15.177:6838/4231634718] up:boot 2024-04-23T18:48:49.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6834/1283448256,v1:172.21.15.177:6837/1283448256] up:boot 2024-04-23T18:48:49.205 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.177:6836/3029152065,v1:172.21.15.177:6839/3029152065] up:boot 2024-04-23T18:48:49.217 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.188:6834/3960668816,v1:172.21.15.188:6837/3960668816] up:boot 2024-04-23T18:48:49.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.188:6835/1145985046,v1:172.21.15.188:6838/1145985046] up:boot 2024-04-23T18:48:49.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: mds.? [v2:172.21.15.188:6836/450211997,v1:172.21.15.188:6839/450211997] up:boot 2024-04-23T18:48:49.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-23T18:48:49.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T18:48:49.275 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T18:48:49.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T18:48:49.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T18:48:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T18:48:49.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: from='mgr.25018 172.21.15.50:0/338768952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T18:48:49.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:48 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4220408576' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T18:48:49.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.495+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/950291472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e74102190 msgr2=0x7f7e74102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:49.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.495+0000 7f7e7a2e1640 1 --2- 172.21.15.50:0/950291472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e74102190 0x7f7e74102570 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009900 tx=0x7f7e6402a410 comp rx=0 tx=0).stop 2024-04-23T18:48:49.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.495+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/950291472 shutdown_connections 2024-04-23T18:48:49.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.495+0000 7f7e7a2e1640 1 --2- 172.21.15.50:0/950291472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e74107260 0x7f7e7410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:49.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.495+0000 7f7e7a2e1640 1 --2- 172.21.15.50:0/950291472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e74102b40 0x7f7e74106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:49.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.495+0000 7f7e7a2e1640 1 --2- 172.21.15.50:0/950291472 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e74102190 0x7f7e74102570 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:49.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.495+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/950291472 >> 172.21.15.50:0/950291472 conn(0x7f7e740fd990 msgr2=0x7f7e740ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:49.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.495+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/950291472 shutdown_connections 2024-04-23T18:48:49.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.496+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/950291472 wait complete. 2024-04-23T18:48:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.496+0000 7f7e7a2e1640 1 Processor -- start 2024-04-23T18:48:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.496+0000 7f7e7a2e1640 1 -- start start 2024-04-23T18:48:49.611 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:49 smithi188 ceph-mon[32694]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T18:48:49.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:49 smithi188 ceph-mon[32694]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T18:48:49.700 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:49 smithi188 ceph-mon[32694]: Cluster is now healthy 2024-04-23T18:48:49.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:49 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/4220408576' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:48:49.719 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:49 smithi188 ceph-mon[32694]: fsmap 12 up:standby 2024-04-23T18:48:49.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e7a2e1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e74102190 0x7f7e74074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:49.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e7a2e1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e74107260 0x7f7e74074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:49.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e7a2e1640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e74074cd0 0x7f7e741a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:49.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e7a2e1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741121d0 con 0x7f7e74102190 2024-04-23T18:48:49.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e7a2e1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7e74112050 con 0x7f7e74107260 2024-04-23T18:48:49.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e7a2e1640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7e74112350 con 0x7f7e74074cd0 2024-04-23T18:48:49.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e74102190 0x7f7e74074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:49.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e74102190 0x7f7e74074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37888/0 (socket says 172.21.15.50:37888) 2024-04-23T18:48:49.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e73fff640 1 -- 172.21.15.50:0/2794472991 learned_addr learned my addr 172.21.15.50:0/2794472991 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:48:49.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e737fe640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e74107260 0x7f7e74074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e7887e640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e74074cd0 0x7f7e741a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:49.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e73fff640 1 -- 172.21.15.50:0/2794472991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e74074cd0 msgr2=0x7f7e741a6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:49.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e73fff640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e74074cd0 0x7f7e741a6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:49.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e73fff640 1 -- 172.21.15.50:0/2794472991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e74107260 msgr2=0x7f7e74074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:49.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e73fff640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e74107260 0x7f7e74074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:49.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e73fff640 1 -- 172.21.15.50:0/2794472991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e741a7020 con 0x7f7e74102190 2024-04-23T18:48:49.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.497+0000 7f7e73fff640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e74102190 0x7f7e74074250 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f7e64005940 tx=0x7f7e64004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:49.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.499+0000 7f7e717fa640 1 -- 172.21.15.50:0/2794472991 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6403f030 con 0x7f7e74102190 2024-04-23T18:48:50.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.499+0000 7f7e717fa640 1 -- 172.21.15.50:0/2794472991 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e64003aa0 con 0x7f7e74102190 2024-04-23T18:48:50.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.499+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/2794472991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e741a72b0 con 0x7f7e74102190 2024-04-23T18:48:50.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.499+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/2794472991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e741a77f0 con 0x7f7e74102190 2024-04-23T18:48:50.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.499+0000 7f7e717fa640 1 -- 172.21.15.50:0/2794472991 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e64035070 con 0x7f7e74102190 2024-04-23T18:48:50.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.500+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/2794472991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e740691c0 con 0x7f7e74102190 2024-04-23T18:48:50.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.501+0000 7f7e717fa640 1 -- 172.21.15.50:0/2794472991 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7e64042030 con 0x7f7e74102190 2024-04-23T18:48:50.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.502+0000 7f7e717fa640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e5c0a3da0 0x7f7e5c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:50.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.502+0000 7f7e737fe640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e5c0a3da0 0x7f7e5c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:50.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.502+0000 7f7e717fa640 1 -- 172.21.15.50:0/2794472991 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 8756+0+0 (secure 0 0 0) 0x7f7e640f0d60 con 0x7f7e74102190 2024-04-23T18:48:50.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.502+0000 7f7e737fe640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e5c0a3da0 0x7f7e5c0a6260 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7e74075c90 tx=0x7f7e6800b390 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:50.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.506+0000 7f7e717fa640 1 -- 172.21.15.50:0/2794472991 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e640bd310 con 0x7f7e74102190 2024-04-23T18:48:50.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:49.692+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/2794472991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7e7410a490 con 0x7f7e74102190 2024-04-23T18:48:50.190 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:49 smithi050 ceph-mon[29187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T18:48:50.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:49 smithi050 ceph-mon[29187]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T18:48:50.207 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:49 smithi050 ceph-mon[29187]: Cluster is now healthy 2024-04-23T18:48:50.217 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:49 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/4220408576' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:48:50.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:49 smithi050 ceph-mon[29187]: fsmap 12 up:standby 2024-04-23T18:48:50.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:49 smithi177 ceph-mon[32308]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T18:48:50.267 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:49 smithi177 ceph-mon[32308]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T18:48:50.275 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:49 smithi177 ceph-mon[32308]: Cluster is now healthy 2024-04-23T18:48:50.285 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:49 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/4220408576' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T18:48:50.296 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:49 smithi177 ceph-mon[32308]: fsmap 12 up:standby 2024-04-23T18:48:50.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.498+0000 7f7e717fa640 1 -- 172.21.15.50:0/2794472991 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v146) ==== 181+0+0 (secure 0 0 0) 0x7f7e640bcc10 con 0x7f7e74102190 2024-04-23T18:48:50.507 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2024-04-23T18:48:50.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.501+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/2794472991 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e5c0a3da0 msgr2=0x7f7e5c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:50.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.501+0000 7f7e7a2e1640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e5c0a3da0 0x7f7e5c0a6260 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7e74075c90 tx=0x7f7e6800b390 comp rx=0 tx=0).stop 2024-04-23T18:48:50.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.501+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/2794472991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e74102190 msgr2=0x7f7e74074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:50.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.501+0000 7f7e7a2e1640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e74102190 0x7f7e74074250 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f7e64005940 tx=0x7f7e64004270 comp rx=0 tx=0).stop 2024-04-23T18:48:50.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.501+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/2794472991 shutdown_connections 2024-04-23T18:48:50.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.501+0000 7f7e7a2e1640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f7e5c0a3da0 0x7f7e5c0a6260 secure :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7e74075c90 tx=0x7f7e6800b390 comp rx=0 tx=0).stop 2024-04-23T18:48:50.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.501+0000 7f7e7a2e1640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e74074cd0 0x7f7e741a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:50.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.501+0000 7f7e7a2e1640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e74107260 0x7f7e74074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:50.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.501+0000 7f7e7a2e1640 1 --2- 172.21.15.50:0/2794472991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e74102190 0x7f7e74074250 secure :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f7e64005940 tx=0x7f7e64004270 comp rx=0 tx=0).stop 2024-04-23T18:48:50.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.501+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/2794472991 >> 172.21.15.50:0/2794472991 conn(0x7f7e740fd990 msgr2=0x7f7e740fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:50.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.502+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/2794472991 shutdown_connections 2024-04-23T18:48:50.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.502+0000 7f7e7a2e1640 1 -- 172.21.15.50:0/2794472991 wait complete. 2024-04-23T18:48:50.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-23T18:48:50.815 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:50 smithi177 ceph-mon[32308]: pgmap v967: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-23T18:48:50.824 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:50 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2794472991' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:48:50.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:50 smithi188 ceph-mon[32694]: pgmap v967: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-23T18:48:50.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:50 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2794472991' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:48:50.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:50 smithi050 ceph-mon[29187]: pgmap v967: 97 pgs: 97 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-23T18:48:50.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:50 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2794472991' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:48:50.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.980+0000 7f71292e1640 1 -- 172.21.15.50:0/411278838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712410c510 msgr2=0x7f712410e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:51.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.980+0000 7f71292e1640 1 --2- 172.21.15.50:0/411278838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712410c510 0x7f712410e9b0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f711c009ca0 tx=0x7f711c02a9e0 comp rx=0 tx=0).stop 2024-04-23T18:48:51.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.980+0000 7f71292e1640 1 -- 172.21.15.50:0/411278838 shutdown_connections 2024-04-23T18:48:51.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.980+0000 7f71292e1640 1 --2- 172.21.15.50:0/411278838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712410c510 0x7f712410e9b0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:51.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.980+0000 7f71292e1640 1 --2- 172.21.15.50:0/411278838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7124103af0 0x7f712410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:51.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.980+0000 7f71292e1640 1 --2- 172.21.15.50:0/411278838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f71241031d0 0x7f71241035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:51.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.980+0000 7f71292e1640 1 -- 172.21.15.50:0/411278838 >> 172.21.15.50:0/411278838 conn(0x7f7124077f50 msgr2=0x7f7124078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:51.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.981+0000 7f71292e1640 1 -- 172.21.15.50:0/411278838 shutdown_connections 2024-04-23T18:48:51.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.981+0000 7f71292e1640 1 -- 172.21.15.50:0/411278838 wait complete. 2024-04-23T18:48:51.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.981+0000 7f71292e1640 1 Processor -- start 2024-04-23T18:48:51.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.981+0000 7f71292e1640 1 -- start start 2024-04-23T18:48:51.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.981+0000 7f71292e1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71241031d0 0x7f71241986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:51.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.981+0000 7f71292e1640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7124103af0 0x7f7124198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:51.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.981+0000 7f71292e1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f712410c510 0x7f712419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:51.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.981+0000 7f71292e1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7124112180 con 0x7f71241031d0 2024-04-23T18:48:51.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.981+0000 7f71292e1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7124112000 con 0x7f712410c510 2024-04-23T18:48:51.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.981+0000 7f71292e1640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7124112300 con 0x7f7124103af0 2024-04-23T18:48:51.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f7122ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71241031d0 0x7f71241986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:51.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f7122ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71241031d0 0x7f71241986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37896/0 (socket says 172.21.15.50:37896) 2024-04-23T18:48:51.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f7122ffd640 1 -- 172.21.15.50:0/3713096673 learned_addr learned my addr 172.21.15.50:0/3713096673 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-23T18:48:51.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f71237fe640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f712410c510 0x7f712419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:51.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f71227fc640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7124103af0 0x7f7124198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:51.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f7122ffd640 1 -- 172.21.15.50:0/3713096673 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7124103af0 msgr2=0x7f7124198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:51.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f7122ffd640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7124103af0 0x7f7124198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:51.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f7122ffd640 1 -- 172.21.15.50:0/3713096673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f712410c510 msgr2=0x7f712419cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:51.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f7122ffd640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f712410c510 0x7f712419cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f7122ffd640 1 -- 172.21.15.50:0/3713096673 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f712419d6d0 con 0x7f71241031d0 2024-04-23T18:48:51.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f71237fe640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f712410c510 0x7f712419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T18:48:51.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f7122ffd640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71241031d0 0x7f71241986e0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f711400e9f0 tx=0x7f711400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:51.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f710bfff640 1 -- 172.21.15.50:0/3713096673 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711400cd80 con 0x7f71241031d0 2024-04-23T18:48:51.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f710bfff640 1 -- 172.21.15.50:0/3713096673 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71140108e0 con 0x7f71241031d0 2024-04-23T18:48:51.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f71292e1640 1 -- 172.21.15.50:0/3713096673 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f712419d9c0 con 0x7f71241031d0 2024-04-23T18:48:51.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f710bfff640 1 -- 172.21.15.50:0/3713096673 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7114018980 con 0x7f71241031d0 2024-04-23T18:48:51.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.982+0000 7f71292e1640 1 -- 172.21.15.50:0/3713096673 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71241a5250 con 0x7f71241031d0 2024-04-23T18:48:51.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.983+0000 7f71292e1640 1 -- 172.21.15.50:0/3713096673 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7124068d70 con 0x7f71241031d0 2024-04-23T18:48:51.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.985+0000 7f710bfff640 1 -- 172.21.15.50:0/3713096673 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7114010430 con 0x7f71241031d0 2024-04-23T18:48:51.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.986+0000 7f710bfff640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f710c0a3d20 0x7f710c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T18:48:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.986+0000 7f71227fc640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f710c0a3d20 0x7f710c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T18:48:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.986+0000 7f710bfff640 1 -- 172.21.15.50:0/3713096673 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 8202+0+0 (secure 0 0 0) 0x7f71140d09e0 con 0x7f71241031d0 2024-04-23T18:48:51.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.987+0000 7f71227fc640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f710c0a3d20 0x7f710c0a61e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f712410e340 tx=0x7f7118005f90 comp rx=0 tx=0).ready entity=mgr.25018 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T18:48:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:50.991+0000 7f710bfff640 1 -- 172.21.15.50:0/3713096673 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f711409cb10 con 0x7f71241031d0 2024-04-23T18:48:51.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.181+0000 7f71292e1640 1 -- 172.21.15.50:0/3713096673 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7124199a30 con 0x7f71241031d0 2024-04-23T18:48:51.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.505+0000 7f710bfff640 1 -- 172.21.15.50:0/3713096673 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v147) ==== 169+0+0 (secure 0 0 0) 0x7f71140a0e10 con 0x7f71241031d0 2024-04-23T18:48:51.761 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2024-04-23T18:48:51.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.508+0000 7f71292e1640 1 -- 172.21.15.50:0/3713096673 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f710c0a3d20 msgr2=0x7f710c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:51.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.508+0000 7f71292e1640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f710c0a3d20 0x7f710c0a61e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f712410e340 tx=0x7f7118005f90 comp rx=0 tx=0).stop 2024-04-23T18:48:51.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.508+0000 7f71292e1640 1 -- 172.21.15.50:0/3713096673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71241031d0 msgr2=0x7f71241986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T18:48:51.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.508+0000 7f71292e1640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71241031d0 0x7f71241986e0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f711400e9f0 tx=0x7f711400eeb0 comp rx=0 tx=0).stop 2024-04-23T18:48:51.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.510+0000 7f71292e1640 1 -- 172.21.15.50:0/3713096673 shutdown_connections 2024-04-23T18:48:51.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.510+0000 7f71292e1640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.50:6800/4210816660,v1:172.21.15.50:6801/4210816660] conn(0x7f710c0a3d20 0x7f710c0a61e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:51.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.510+0000 7f71292e1640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f712410c510 0x7f712419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:51.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.510+0000 7f71292e1640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7124103af0 0x7f7124198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:51.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.510+0000 7f71292e1640 1 --2- 172.21.15.50:0/3713096673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71241031d0 0x7f71241986e0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T18:48:51.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.510+0000 7f71292e1640 1 -- 172.21.15.50:0/3713096673 >> 172.21.15.50:0/3713096673 conn(0x7f7124077f50 msgr2=0x7f71240fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T18:48:51.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.510+0000 7f71292e1640 1 -- 172.21.15.50:0/3713096673 shutdown_connections 2024-04-23T18:48:51.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-23T18:48:51.511+0000 7f71292e1640 1 -- 172.21.15.50:0/3713096673 wait complete. 2024-04-23T18:48:52.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:48:52.021 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:51 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/2794472991' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T18:48:52.029 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:51 smithi188 ceph-mon[32694]: osdmap e146: 12 total, 12 up, 12 in 2024-04-23T18:48:52.052 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:51 smithi188 ceph-mon[32694]: pgmap v969: 65 pgs: 65 active+clean; 221 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:48:52.061 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:51 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3713096673' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:48:52.070 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:51 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/2794472991' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T18:48:52.079 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:51 smithi177 ceph-mon[32308]: osdmap e146: 12 total, 12 up, 12 in 2024-04-23T18:48:52.086 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:51 smithi177 ceph-mon[32308]: pgmap v969: 65 pgs: 65 active+clean; 221 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:48:52.095 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:51 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3713096673' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:48:52.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:51 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/2794472991' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T18:48:52.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:51 smithi050 ceph-mon[29187]: osdmap e146: 12 total, 12 up, 12 in 2024-04-23T18:48:52.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:51 smithi050 ceph-mon[29187]: pgmap v969: 65 pgs: 65 active+clean; 221 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T18:48:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:51 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3713096673' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T18:48:52.192 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:48:52.245 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:48:52.298 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-23T18:48:52.305 DEBUG:teuthology.orchestra.run.smithi050:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T18:48:52.330 DEBUG:teuthology.orchestra.run.smithi177:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T18:48:52.354 DEBUG:teuthology.orchestra.run.smithi188:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T18:48:52.378 INFO:tasks.cephadm:Stopping all daemons... 2024-04-23T18:48:52.419 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-23T18:48:52.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.a 2024-04-23T18:48:52.577 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:52 smithi188 ceph-mon[32694]: from='client.? 172.21.15.50:0/3713096673' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T18:48:52.584 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:52 smithi188 ceph-mon[32694]: osdmap e147: 12 total, 12 up, 12 in 2024-04-23T18:48:52.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:52 smithi050 systemd[1]: Stopping Ceph mon.a for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:48:52.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:52 smithi050 ceph-mon[29187]: from='client.? 172.21.15.50:0/3713096673' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T18:48:52.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:52 smithi050 ceph-mon[29187]: osdmap e147: 12 total, 12 up, 12 in 2024-04-23T18:48:52.792 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:52 smithi177 ceph-mon[32308]: from='client.? 172.21.15.50:0/3713096673' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T18:48:52.799 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:52 smithi177 ceph-mon[32308]: osdmap e147: 12 total, 12 up, 12 in 2024-04-23T18:48:53.192 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a[29183]: 2024-04-23T18:48:52.789+0000 7f93320d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:48:53.203 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a[29183]: 2024-04-23T18:48:52.789+0000 7f93320d6700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-23T18:48:53.211 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:53 smithi050 podman[100147]: 2024-04-23 18:48:53.003345503 +0000 UTC m=+0.330877980 container died b09dd59349eaaeb88989a44baf823c9759e8afb28e129f28a624fcc728688dbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:48:53.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:53 smithi050 podman[100147]: 2024-04-23 18:48:53.481175696 +0000 UTC m=+0.808708187 container cleanup b09dd59349eaaeb88989a44baf823c9759e8afb28e129f28a624fcc728688dbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-23T18:48:53.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 23 18:48:53 smithi050 bash[100147]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-a 2024-04-23T18:48:53.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.a.service' 2024-04-23T18:48:53.927 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:48:53.936 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-23T18:48:53.972 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-23T18:48:53.980 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.b 2024-04-23T18:48:54.285 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:54 smithi177 systemd[1]: Stopping Ceph mon.b for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:48:54.785 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:54 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-b[32304]: 2024-04-23T18:48:54.534+0000 7f6cf23dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:48:54.792 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:54 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-b[32304]: 2024-04-23T18:48:54.534+0000 7f6cf23dd700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-23T18:48:54.800 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:54 smithi177 podman[76548]: 2024-04-23 18:48:54.565356665 +0000 UTC m=+0.126067715 container died c0b85f066c10d17d60cab01a451916338ab37fd6276fb0bb6e6a30b386b83268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-23T18:48:55.243 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:54 smithi177 podman[76548]: 2024-04-23 18:48:54.951824386 +0000 UTC m=+0.512535440 container cleanup c0b85f066c10d17d60cab01a451916338ab37fd6276fb0bb6e6a30b386b83268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-04-23T18:48:55.262 INFO:journalctl@ceph.mon.b.smithi177.stdout:Apr 23 18:48:54 smithi177 bash[76548]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-b 2024-04-23T18:48:55.354 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.b.service' 2024-04-23T18:48:55.406 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:48:55.433 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-23T18:48:55.440 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-23T18:48:55.539 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.c 2024-04-23T18:48:55.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:55 smithi188 systemd[1]: Stopping Ceph mon.c for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:48:56.326 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:55 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-c[32690]: 2024-04-23T18:48:55.925+0000 7f50c9009700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:48:56.333 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:55 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-c[32690]: 2024-04-23T18:48:55.925+0000 7f50c9009700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-23T18:48:56.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:56 smithi188 podman[76268]: 2024-04-23 18:48:56.037378198 +0000 UTC m=+0.235607784 container died df051151edb949b3c073340ab59c88066970ae1322fb0a8190157a8d1d716997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:48:56.577 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:56 smithi188 podman[76268]: 2024-04-23 18:48:56.435380589 +0000 UTC m=+0.633610177 container cleanup df051151edb949b3c073340ab59c88066970ae1322fb0a8190157a8d1d716997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-23T18:48:56.584 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:56 smithi188 bash[76268]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-c 2024-04-23T18:48:56.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 23 18:48:56 smithi188 podman[76282]: 2024-04-23 18:48:56.720340766 +0000 UTC m=+0.677611993 container remove df051151edb949b3c073340ab59c88066970ae1322fb0a8190157a8d1d716997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:48:56.988 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mon.c.service' 2024-04-23T18:48:57.044 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:48:57.051 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-23T18:48:57.058 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-23T18:48:57.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.x 2024-04-23T18:48:57.430 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:48:57 smithi050 systemd[1]: Stopping Ceph mgr.x for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:48:57.694 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:48:57 smithi050 podman[100364]: 2024-04-23 18:48:57.441990826 +0000 UTC m=+0.160518764 container died 6cdd679662e2945e6caca10a6a1930bc5d391b218a03f36f075a6014369916a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-23T18:48:58.303 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:48:58 smithi050 podman[100364]: 2024-04-23 18:48:58.04445674 +0000 UTC m=+0.762984672 container cleanup 6cdd679662e2945e6caca10a6a1930bc5d391b218a03f36f075a6014369916a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:48:58.310 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Apr 23 18:48:58 smithi050 bash[100364]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-x 2024-04-23T18:48:58.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-23T18:48:58.449 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:48:58.458 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-23T18:48:58.467 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-23T18:48:58.475 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.y 2024-04-23T18:48:58.785 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:48:58 smithi177 systemd[1]: Stopping Ceph mgr.y for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:48:59.123 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:48:58 smithi177 podman[76669]: 2024-04-23 18:48:58.844279492 +0000 UTC m=+0.141728055 container died 0f041e72f6eecf5a5511d1d4b228b21719b0e407848854282f703c6e3098ddfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-23T18:48:59.456 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:48:59 smithi177 podman[76669]: 2024-04-23 18:48:59.22356684 +0000 UTC m=+0.521015398 container cleanup 0f041e72f6eecf5a5511d1d4b228b21719b0e407848854282f703c6e3098ddfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:48:59.463 INFO:journalctl@ceph.mgr.y.smithi177.stdout:Apr 23 18:48:59 smithi177 bash[76669]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-y 2024-04-23T18:48:59.568 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-23T18:48:59.609 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:48:59.616 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-23T18:48:59.625 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-23T18:48:59.634 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.z 2024-04-23T18:48:59.994 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:48:59 smithi188 systemd[1]: Stopping Ceph mgr.z for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:49:00.327 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:49:00 smithi188 podman[76388]: 2024-04-23 18:49:00.003741541 +0000 UTC m=+0.140253861 container died 315dbf4c4ee4fdd584a4e4432871e5008e152a1afe3b14c4da4e4aa040fbb8f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-23T18:49:00.766 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:49:00 smithi188 podman[76388]: 2024-04-23 18:49:00.447882463 +0000 UTC m=+0.584394782 container cleanup 315dbf4c4ee4fdd584a4e4432871e5008e152a1afe3b14c4da4e4aa040fbb8f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2) 2024-04-23T18:49:00.815 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 23 18:49:00 smithi188 bash[76388]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mgr-z 2024-04-23T18:49:00.922 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-23T18:49:00.963 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:49:00.969 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-23T18:49:00.978 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-23T18:49:01.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.0 2024-04-23T18:49:01.339 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:01 smithi050 systemd[1]: Stopping Ceph osd.0 for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:49:01.692 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0[41799]: 2024-04-23T18:49:01.348+0000 7fc019942700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:49:01.746 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0[41799]: 2024-04-23T18:49:01.348+0000 7fc019942700 -1 osd.0 147 *** Got signal Terminated *** 2024-04-23T18:49:01.755 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0[41799]: 2024-04-23T18:49:01.348+0000 7fc019942700 -1 osd.0 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:49:07.348 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:06 smithi050 podman[100485]: 2024-04-23 18:49:06.953620077 +0000 UTC m=+5.713233568 container died 186e4caf556a5c5dc4c0d272dde93efdc0365513635d7d5df85de891d3268b96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:49:07.692 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:07 smithi050 podman[100485]: 2024-04-23 18:49:07.457302104 +0000 UTC m=+6.216915606 container cleanup 186e4caf556a5c5dc4c0d272dde93efdc0365513635d7d5df85de891d3268b96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-23T18:49:07.699 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:07 smithi050 bash[100485]: ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0 2024-04-23T18:49:08.024 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:07 smithi050 podman[100502]: 2024-04-23 18:49:07.708994292 +0000 UTC m=+0.756776012 container remove 186e4caf556a5c5dc4c0d272dde93efdc0365513635d7d5df85de891d3268b96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:49:08.319 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:08 smithi050 podman[100594]: 2024-04-23 18:49:08.034699101 +0000 UTC m=+0.115314454 container create 1eef0d2cc229151f9cf2e8aa84acbfac78f01bd3d40feac59df7078c90235d36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-23T18:49:08.326 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:08 smithi050 podman[100594]: 2024-04-23 18:49:07.937895314 +0000 UTC m=+0.018510686 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:49:08.333 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:08 smithi050 podman[100594]: 2024-04-23 18:49:08.253755405 +0000 UTC m=+0.334370755 container init 1eef0d2cc229151f9cf2e8aa84acbfac78f01bd3d40feac59df7078c90235d36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:49:08.341 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:08 smithi050 podman[100594]: 2024-04-23 18:49:08.256895828 +0000 UTC m=+0.337511174 container start 1eef0d2cc229151f9cf2e8aa84acbfac78f01bd3d40feac59df7078c90235d36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-23T18:49:08.347 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:08 smithi050 podman[100594]: 2024-04-23 18:49:08.32867142 +0000 UTC m=+0.409286776 container attach 1eef0d2cc229151f9cf2e8aa84acbfac78f01bd3d40feac59df7078c90235d36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-23T18:49:08.692 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 23 18:49:08 smithi050 podman[100594]: 2024-04-23 18:49:08.560045177 +0000 UTC m=+0.640660537 container died 1eef0d2cc229151f9cf2e8aa84acbfac78f01bd3d40feac59df7078c90235d36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0) 2024-04-23T18:49:09.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.0.service' 2024-04-23T18:49:09.316 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:49:09.324 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-23T18:49:09.335 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-23T18:49:09.345 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.1 2024-04-23T18:49:09.718 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:09 smithi177 systemd[1]: Stopping Ceph osd.1 for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:49:10.035 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:09 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1[36954]: 2024-04-23T18:49:09.726+0000 7fd723e48700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:49:10.115 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:09 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1[36954]: 2024-04-23T18:49:09.726+0000 7fd723e48700 -1 osd.1 147 *** Got signal Terminated *** 2024-04-23T18:49:10.129 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:09 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1[36954]: 2024-04-23T18:49:09.726+0000 7fd723e48700 -1 osd.1 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:49:15.585 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:15 smithi177 podman[76792]: 2024-04-23 18:49:15.302654812 +0000 UTC m=+5.748131573 container died c84cc7089d298a1a69a6971c8d1ff7344de919d9d9dfadaaf9640dbbf0185ac2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-23T18:49:15.952 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:15 smithi177 podman[76792]: 2024-04-23 18:49:15.685431839 +0000 UTC m=+6.130908592 container cleanup c84cc7089d298a1a69a6971c8d1ff7344de919d9d9dfadaaf9640dbbf0185ac2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:49:15.962 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:15 smithi177 bash[76792]: ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1 2024-04-23T18:49:16.250 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:15 smithi177 podman[76810]: 2024-04-23 18:49:15.961815918 +0000 UTC m=+0.653709261 container remove c84cc7089d298a1a69a6971c8d1ff7344de919d9d9dfadaaf9640dbbf0185ac2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-23T18:49:16.514 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:16 smithi177 podman[76900]: 2024-04-23 18:49:16.163161346 +0000 UTC m=+0.018021109 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:49:16.521 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:16 smithi177 podman[76900]: 2024-04-23 18:49:16.262571742 +0000 UTC m=+0.117431539 container create b2aad5237689229eb2ef4584466549e269f97385ae914da15cb50d2aa731f6ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-23T18:49:16.528 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:16 smithi177 podman[76900]: 2024-04-23 18:49:16.523846412 +0000 UTC m=+0.378706167 container init b2aad5237689229eb2ef4584466549e269f97385ae914da15cb50d2aa731f6ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-23T18:49:16.785 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:16 smithi177 podman[76900]: 2024-04-23 18:49:16.527232584 +0000 UTC m=+0.382092349 container start b2aad5237689229eb2ef4584466549e269f97385ae914da15cb50d2aa731f6ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:49:16.793 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:16 smithi177 podman[76900]: 2024-04-23 18:49:16.606970384 +0000 UTC m=+0.461830140 container attach b2aad5237689229eb2ef4584466549e269f97385ae914da15cb50d2aa731f6ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T18:49:17.134 INFO:journalctl@ceph.osd.1.smithi177.stdout:Apr 23 18:49:16 smithi177 podman[76900]: 2024-04-23 18:49:16.82509066 +0000 UTC m=+0.679950418 container died b2aad5237689229eb2ef4584466549e269f97385ae914da15cb50d2aa731f6ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True) 2024-04-23T18:49:17.441 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.1.service' 2024-04-23T18:49:17.488 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:49:17.513 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-23T18:49:17.521 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-23T18:49:17.530 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.2 2024-04-23T18:49:17.827 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:17 smithi188 systemd[1]: Stopping Ceph osd.2 for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:49:18.327 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:17 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2[37874]: 2024-04-23T18:49:17.925+0000 7f4d0e259700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:49:18.344 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:17 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2[37874]: 2024-04-23T18:49:17.925+0000 7f4d0e259700 -1 osd.2 147 *** Got signal Terminated *** 2024-04-23T18:49:18.469 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:17 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2[37874]: 2024-04-23T18:49:17.925+0000 7f4d0e259700 -1 osd.2 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:49:23.811 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:23 smithi188 podman[76517]: 2024-04-23 18:49:23.521744977 +0000 UTC m=+5.780100769 container died a2f6df90dae5591ffe0ad06d10a5f8446902a40e3460e4c52f40290cbafcadef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-23T18:49:24.077 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:23 smithi188 podman[76517]: 2024-04-23 18:49:23.902360412 +0000 UTC m=+6.160716199 container cleanup a2f6df90dae5591ffe0ad06d10a5f8446902a40e3460e4c52f40290cbafcadef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-23T18:49:24.085 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:23 smithi188 bash[76517]: ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2 2024-04-23T18:49:24.512 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:24 smithi188 podman[76546]: 2024-04-23 18:49:24.213002715 +0000 UTC m=+0.683364321 container remove a2f6df90dae5591ffe0ad06d10a5f8446902a40e3460e4c52f40290cbafcadef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-23T18:49:24.807 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:24 smithi188 podman[76634]: 2024-04-23 18:49:24.522005307 +0000 UTC m=+0.111801524 container create c6e491cd62cdb0b6bdafe523c044b445c81231d05e89ac527c76ca8d4dc8991b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:49:24.814 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:24 smithi188 podman[76634]: 2024-04-23 18:49:24.428322718 +0000 UTC m=+0.018118936 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:49:24.822 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:24 smithi188 podman[76634]: 2024-04-23 18:49:24.73341634 +0000 UTC m=+0.323212558 container init c6e491cd62cdb0b6bdafe523c044b445c81231d05e89ac527c76ca8d4dc8991b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-23T18:49:24.830 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:24 smithi188 podman[76634]: 2024-04-23 18:49:24.736826047 +0000 UTC m=+0.326622272 container start c6e491cd62cdb0b6bdafe523c044b445c81231d05e89ac527c76ca8d4dc8991b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-23T18:49:24.837 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:24 smithi188 podman[76634]: 2024-04-23 18:49:24.816749576 +0000 UTC m=+0.406545803 container attach c6e491cd62cdb0b6bdafe523c044b445c81231d05e89ac527c76ca8d4dc8991b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:49:25.077 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 23 18:49:25 smithi188 podman[76634]: 2024-04-23 18:49:25.064620395 +0000 UTC m=+0.654416614 container died c6e491cd62cdb0b6bdafe523c044b445c81231d05e89ac527c76ca8d4dc8991b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-23T18:49:25.794 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.2.service' 2024-04-23T18:49:25.855 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:49:25.884 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-23T18:49:25.893 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-23T18:49:25.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.3 2024-04-23T18:49:26.192 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:25 smithi050 systemd[1]: Stopping Ceph osd.3 for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:49:26.473 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:26 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3[48756]: 2024-04-23T18:49:26.221+0000 7fa5fff3e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:49:26.480 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:26 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3[48756]: 2024-04-23T18:49:26.221+0000 7fa5fff3e700 -1 osd.3 147 *** Got signal Terminated *** 2024-04-23T18:49:26.489 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:26 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3[48756]: 2024-04-23T18:49:26.221+0000 7fa5fff3e700 -1 osd.3 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:49:27.942 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:27 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3[48756]: 2024-04-23T18:49:27.640+0000 7fa5f852a700 -1 osd.3 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.992087+0000 front 2024-04-23T18:49:02.992127+0000 (oldest deadline 2024-04-23T18:49:26.992769+0000) 2024-04-23T18:49:28.577 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:28.429+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:28.597 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:28.249+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:28.942 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:28 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3[48756]: 2024-04-23T18:49:28.640+0000 7fa5f852a700 -1 osd.3 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.992087+0000 front 2024-04-23T18:49:02.992127+0000 (oldest deadline 2024-04-23T18:49:26.992769+0000) 2024-04-23T18:49:28.986 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:28 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:28.771+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:29.442 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:29 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:29.154+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:29.469 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:29.214+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:29.785 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:29 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:29.365+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:29.827 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:29.477+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:29.942 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:29 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3[48756]: 2024-04-23T18:49:29.676+0000 7fa5f852a700 -1 osd.3 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.992087+0000 front 2024-04-23T18:49:02.992127+0000 (oldest deadline 2024-04-23T18:49:26.992769+0000) 2024-04-23T18:49:29.949 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:29 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:29.764+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:30.442 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:30.199+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:30.473 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:30 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:30.215+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:30.785 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:30 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:30.322+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:30.827 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:30 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:30.481+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:30.942 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3[48756]: 2024-04-23T18:49:30.699+0000 7fa5f852a700 -1 osd.3 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.992087+0000 front 2024-04-23T18:49:02.992127+0000 (oldest deadline 2024-04-23T18:49:26.992769+0000) 2024-04-23T18:49:30.978 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:30 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:30.804+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:31.535 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:31 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:31.281+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:31.577 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:31 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:31.212+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:31.588 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:31 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:31.451+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:31.692 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:31 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:31.215+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:32.151 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:31 smithi050 podman[100701]: 2024-04-23 18:49:31.789191255 +0000 UTC m=+5.671897219 container died 5a43518c9d040457fb15d072c6a1e682671231d55933e51de8c13ad8e071539c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:49:32.200 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:31 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:31.832+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:32.442 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:32 smithi050 podman[100701]: 2024-04-23 18:49:32.259447663 +0000 UTC m=+6.142153617 container cleanup 5a43518c9d040457fb15d072c6a1e682671231d55933e51de8c13ad8e071539c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:49:32.450 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:32 smithi050 bash[100701]: ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3 2024-04-23T18:49:32.459 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:32 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:32.177+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:32.535 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:32 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:32.292+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:32.577 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:32 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:32.252+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:32.588 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:32 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:32.502+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:32.780 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:32 smithi050 podman[100730]: 2024-04-23 18:49:32.535944713 +0000 UTC m=+0.744221952 container remove 5a43518c9d040457fb15d072c6a1e682671231d55933e51de8c13ad8e071539c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-23T18:49:33.164 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:32 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:32.789+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:33.183 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:32 smithi050 podman[100821]: 2024-04-23 18:49:32.739263066 +0000 UTC m=+0.018452115 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:49:33.191 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:32 smithi050 podman[100821]: 2024-04-23 18:49:32.853273869 +0000 UTC m=+0.132462893 container create 10ead1f687b3a8c27c6017dbb18528d58d590fdc0325c131efbb618c167aa8d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-23T18:49:33.442 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:33 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:33.173+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:33.450 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:33 smithi050 podman[100821]: 2024-04-23 18:49:33.256802098 +0000 UTC m=+0.535991116 container init 10ead1f687b3a8c27c6017dbb18528d58d590fdc0325c131efbb618c167aa8d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:49:33.542 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:33 smithi050 podman[100821]: 2024-04-23 18:49:33.260357922 +0000 UTC m=+0.539546940 container start 10ead1f687b3a8c27c6017dbb18528d58d590fdc0325c131efbb618c167aa8d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:49:33.551 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:33 smithi050 podman[100821]: 2024-04-23 18:49:33.314232915 +0000 UTC m=+0.593421938 container attach 10ead1f687b3a8c27c6017dbb18528d58d590fdc0325c131efbb618c167aa8d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-23T18:49:33.561 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:33 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:33.288+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:33.586 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:33 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:33.231+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:33.614 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:33 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:33.476+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:33.824 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:33 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:33.772+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:33.832 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:33 smithi050 podman[100821]: 2024-04-23 18:49:33.563778274 +0000 UTC m=+0.842967303 container died 10ead1f687b3a8c27c6017dbb18528d58d590fdc0325c131efbb618c167aa8d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-23T18:49:34.082 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 23 18:49:34 smithi050 podman[100847]: 2024-04-23 18:49:34.075953362 +0000 UTC m=+0.506836466 container remove 10ead1f687b3a8c27c6017dbb18528d58d590fdc0325c131efbb618c167aa8d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-23T18:49:34.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.3.service' 2024-04-23T18:49:34.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:49:34.150 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-23T18:49:34.157 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-23T18:49:34.165 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.4 2024-04-23T18:49:34.442 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:34 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:34.194+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:34.475 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:34 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:34.188+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:34.552 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:34 smithi177 systemd[1]: Stopping Ceph osd.4 for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:49:34.565 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:34 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:34.278+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:34.827 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:34 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:34.454+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:35.035 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:34 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4[43498]: 2024-04-23T18:49:34.545+0000 7f1ba5765700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:49:35.042 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:34 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4[43498]: 2024-04-23T18:49:34.545+0000 7f1ba5765700 -1 osd.4 147 *** Got signal Terminated *** 2024-04-23T18:49:35.096 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:34 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4[43498]: 2024-04-23T18:49:34.545+0000 7f1ba5765700 -1 osd.4 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:49:35.192 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:34 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:34.805+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:35.461 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:35 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:35.235+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:35.535 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:35 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:35.248+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:35.692 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:35 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:35.222+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:35.827 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:35 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:35.470+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:36.192 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:35 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:35.844+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:36.199 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:35 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:35.844+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:36.452 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:36 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:36.207+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:36.578 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:36 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:36.442+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:36.585 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:36 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:36.282+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:36.594 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:36 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:36.282+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:36.706 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:36 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:36.232+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:36.785 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:36 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:36.460+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:37.192 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:36 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:36.856+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:37.199 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:36 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:36.856+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:37.535 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:37 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:37.195+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:37.550 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:37 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:37.418+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:37.577 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:37 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:37.464+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:37.585 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:37 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:37.464+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.010716+0000 front 2024-04-23T18:49:13.010879+0000 (oldest deadline 2024-04-23T18:49:36.510743+0000) 2024-04-23T18:49:37.593 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:37 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:37.288+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:37.601 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:37 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:37.288+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:37.692 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:37 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:37.205+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:37.702 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:37 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:37.205+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:38.192 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:37 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:37.846+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:38.199 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:37 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:37.846+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:38.222 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:37 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4[43498]: 2024-04-23T18:49:37.840+0000 7f1b9dd51700 -1 osd.4 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:12.792919+0000 front 2024-04-23T18:49:12.792980+0000 (oldest deadline 2024-04-23T18:49:37.493084+0000) 2024-04-23T18:49:38.278 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:37 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:37.887+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:38.535 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:38 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:38.463+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:38.544 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:38 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:38.230+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:38.577 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:38 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:38.287+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:38.634 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:38 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:38.287+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:38.642 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:38 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:38.439+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:38.651 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:38 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:38.439+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.010716+0000 front 2024-04-23T18:49:13.010879+0000 (oldest deadline 2024-04-23T18:49:36.510743+0000) 2024-04-23T18:49:38.692 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:38 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:38.248+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:38.725 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:38 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:38.248+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:39.192 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:38 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:38.850+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:39.200 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:38 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:38.850+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:39.218 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:38 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4[43498]: 2024-04-23T18:49:38.817+0000 7f1b9dd51700 -1 osd.4 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:12.792919+0000 front 2024-04-23T18:49:12.792980+0000 (oldest deadline 2024-04-23T18:49:37.493084+0000) 2024-04-23T18:49:39.246 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:38 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:38.861+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:39.535 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:39 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:39.436+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:39.543 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:39 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:39.208+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:39.577 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:39 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:39.407+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:39.587 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:39 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:39.407+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.010716+0000 front 2024-04-23T18:49:13.010879+0000 (oldest deadline 2024-04-23T18:49:36.510743+0000) 2024-04-23T18:49:39.596 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:39 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:39.254+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:39.675 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:39 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:39.254+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:39.778 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:39 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:39.246+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:39.787 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:39 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:39.246+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:40.077 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:39 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:39.818+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:40.192 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:39 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:39.821+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:40.199 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:39 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:39.821+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:40.363 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:40 smithi177 podman[77018]: 2024-04-23 18:49:40.092379283 +0000 UTC m=+5.666697822 container died be28b11658c5b93bea8092cea597a9c87179bbf617245b39cb337a7598690e7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:49:40.372 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:40 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:40.178+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:40.577 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:40 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:40.218+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:40.586 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:40 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:40.218+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:40.595 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:40 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:40.423+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:40.604 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:40 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:40.423+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.010716+0000 front 2024-04-23T18:49:13.010879+0000 (oldest deadline 2024-04-23T18:49:36.510743+0000) 2024-04-23T18:49:40.692 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:40 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:40.242+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:40.699 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:40 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:40.242+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:40.746 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:40 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:40.411+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:40.754 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:40 smithi177 podman[77018]: 2024-04-23 18:49:40.479823713 +0000 UTC m=+6.054142259 container cleanup be28b11658c5b93bea8092cea597a9c87179bbf617245b39cb337a7598690e7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-23T18:49:40.776 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:40 smithi177 bash[77018]: ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4 2024-04-23T18:49:41.035 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:40 smithi177 podman[77036]: 2024-04-23 18:49:40.755822885 +0000 UTC m=+0.665085389 container remove be28b11658c5b93bea8092cea597a9c87179bbf617245b39cb337a7598690e7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-23T18:49:41.163 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:40 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:40.843+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:41.192 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:40 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:40.806+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:41.242 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:40 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:40.806+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:41.357 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:41 smithi177 podman[77127]: 2024-04-23 18:49:41.005069831 +0000 UTC m=+0.017752997 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:49:41.364 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:41 smithi177 podman[77127]: 2024-04-23 18:49:41.123467574 +0000 UTC m=+0.136150716 container create e4267da79391b1801a16ae85a94cbe11d2b7853e9c6f1b3c2b027b7ff80953fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:49:41.376 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:41 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:41.197+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:41.413 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:41 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:41.197+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:41.420 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:41 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:41.172+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:41.427 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:41 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:41.172+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:41.650 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:41 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:41.397+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:41.657 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:41 smithi177 podman[77127]: 2024-04-23 18:49:41.367264717 +0000 UTC m=+0.379947858 container init e4267da79391b1801a16ae85a94cbe11d2b7853e9c6f1b3c2b027b7ff80953fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-23T18:49:41.665 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:41 smithi177 podman[77127]: 2024-04-23 18:49:41.370406035 +0000 UTC m=+0.383089183 container start e4267da79391b1801a16ae85a94cbe11d2b7853e9c6f1b3c2b027b7ff80953fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:49:41.700 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:41 smithi177 podman[77127]: 2024-04-23 18:49:41.450644436 +0000 UTC m=+0.463327579 container attach e4267da79391b1801a16ae85a94cbe11d2b7853e9c6f1b3c2b027b7ff80953fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:49:41.708 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:41 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:41.264+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:41.714 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:41 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:41.264+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:41.827 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:41 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:41.429+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:41.833 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:41 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:41.429+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.010716+0000 front 2024-04-23T18:49:13.010879+0000 (oldest deadline 2024-04-23T18:49:36.510743+0000) 2024-04-23T18:49:41.943 INFO:journalctl@ceph.osd.4.smithi177.stdout:Apr 23 18:49:41 smithi177 podman[77127]: 2024-04-23 18:49:41.660334899 +0000 UTC m=+0.673018064 container died e4267da79391b1801a16ae85a94cbe11d2b7853e9c6f1b3c2b027b7ff80953fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True) 2024-04-23T18:49:42.192 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:41 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:41.834+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:42.202 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:41 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:41.834+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:42.211 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:41 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:41.886+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:42.245 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.4.service' 2024-04-23T18:49:42.302 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:49:42.309 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-23T18:49:42.317 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-23T18:49:42.326 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.5 2024-04-23T18:49:42.536 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:42 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:42.378+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:42.620 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:42 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:42.242+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:42.631 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:42 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:42.242+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:42.640 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:42 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:42.218+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:42.649 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:42 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:42.218+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:42.659 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:42 smithi188 systemd[1]: Stopping Ceph osd.5 for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:49:42.714 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:42 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:42.459+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:42.723 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:42 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:42.459+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.010716+0000 front 2024-04-23T18:49:13.010879+0000 (oldest deadline 2024-04-23T18:49:36.510743+0000) 2024-04-23T18:49:42.733 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:42 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:42.229+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:42.744 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:42 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:42.229+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:43.077 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:42 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:42.893+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:43.087 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:42 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:42.698+0000 7fdcf5458700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:49:43.096 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:42 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:42.698+0000 7fdcf5458700 -1 osd.5 147 *** Got signal Terminated *** 2024-04-23T18:49:43.106 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:42 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:42.698+0000 7fdcf5458700 -1 osd.5 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:49:43.192 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:42 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:42.858+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:43.200 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:42 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:42.858+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:43.536 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:43 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:43.404+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:43.543 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:43 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:43.258+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:43.552 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:43 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:43.258+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:43.578 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:43 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:43.433+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:43.587 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:43 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:43.433+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.010716+0000 front 2024-04-23T18:49:13.010879+0000 (oldest deadline 2024-04-23T18:49:36.510743+0000) 2024-04-23T18:49:43.597 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:43 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:43.187+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:43.605 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:43 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:43.187+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:43.692 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:43 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:43.202+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:43.702 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:43 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:43.202+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:44.160 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:43 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:43.815+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:44.171 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:43 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:43.815+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:44.226 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:43 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:43.850+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:44.421 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:44 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:44.174+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:44.429 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:44 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:44.174+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:44.460 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:44 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:44.174+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:44.499 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:44 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:44.169+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:44.508 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:44 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:44.169+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:44.785 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:44 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:44.377+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:44.795 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:44 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:44.308+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:44.803 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:44 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:44.308+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:44.814 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:44 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:44.308+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:44.827 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:44 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:44.429+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:44.833 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:44 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:44.429+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.010716+0000 front 2024-04-23T18:49:13.010879+0000 (oldest deadline 2024-04-23T18:49:36.510743+0000) 2024-04-23T18:49:45.156 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:44 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:44.878+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:45.179 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:44 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:44.776+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:45.187 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:44 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:44.776+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:45.442 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:45 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:45.188+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:45.450 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:45 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:45.188+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:45.578 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:45 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:45.388+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:45.643 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:45 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:45.388+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.010716+0000 front 2024-04-23T18:49:13.010879+0000 (oldest deadline 2024-04-23T18:49:36.510743+0000) 2024-04-23T18:49:45.677 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:45 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:45.388+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.512101+0000 front 2024-04-23T18:49:20.512115+0000 (oldest deadline 2024-04-23T18:49:45.212267+0000) 2024-04-23T18:49:45.735 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:45 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:45.164+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:45.783 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:45 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:45.164+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:45.790 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:45 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:45.164+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:45.798 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:45 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:45.427+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:45.805 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:45 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:45.332+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:45.811 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:45 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:45.332+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:45.818 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:45 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:45.332+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:46.116 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:45 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:45.863+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:46.131 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:45 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:45.811+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:46.228 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:45 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:45.811+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:46.374 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:46.124+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:46.380 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:46.124+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:46.389 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:46.124+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:46.442 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:46 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:46.139+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:46.469 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:46 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:46.139+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:46.478 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:46 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:46.139+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:46.786 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:46.435+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:46.829 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:46.315+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:46.897 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:46.315+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:46.905 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:46 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:46.315+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:46.914 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:46.382+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:46.923 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:46.382+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.010716+0000 front 2024-04-23T18:49:13.010879+0000 (oldest deadline 2024-04-23T18:49:36.510743+0000) 2024-04-23T18:49:46.961 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:46.382+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.512101+0000 front 2024-04-23T18:49:20.512115+0000 (oldest deadline 2024-04-23T18:49:45.212267+0000) 2024-04-23T18:49:47.082 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:46 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:46.772+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:47.089 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:46 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:46.772+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:47.328 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:47 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:47.080+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:47.335 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:47 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:47.080+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:47.448 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:47 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:47.081+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:47.458 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:46.842+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:47.467 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:46 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:46.842+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:47.477 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:47 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:47.090+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:47.539 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:47 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:47.090+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:47.547 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:47 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:47.090+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:47.785 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:47 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:47.433+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:47.839 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:47 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:47.336+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:47.873 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:47 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:47.336+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:47.882 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:47 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:47.336+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:47.892 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:47 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:47.385+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:06.009521+0000 front 2024-04-23T18:49:06.009642+0000 (oldest deadline 2024-04-23T18:49:28.309690+0000) 2024-04-23T18:49:47.898 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:47 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:47.385+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.010716+0000 front 2024-04-23T18:49:13.010879+0000 (oldest deadline 2024-04-23T18:49:36.510743+0000) 2024-04-23T18:49:47.907 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:47 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5[44555]: 2024-04-23T18:49:47.385+0000 7fdcf1a4c700 -1 osd.5 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.512101+0000 front 2024-04-23T18:49:20.512115+0000 (oldest deadline 2024-04-23T18:49:45.212267+0000) 2024-04-23T18:49:48.060 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:47 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:47.726+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:48.067 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:47 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:47.726+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:48.078 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:47 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:47.819+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:48.088 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:47 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:47.819+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:48.443 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:48.069+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:48.456 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:48.069+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:48.461 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:48.069+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:48.470 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:48 smithi188 podman[76740]: 2024-04-23 18:49:48.223130866 +0000 UTC m=+5.685255344 container died debc47de70033a42cfbebf37b412cddfe650b67d192efb2e77cb1ffd5bb7e332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-04-23T18:49:48.531 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:48 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:48.125+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:48.540 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:48 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:48.125+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:48.549 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:48 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:48.125+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:48.687 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:48 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:48.415+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:48.719 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:48 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:48.415+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:49:48.728 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:48 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:48.335+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:48.735 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:48 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:48.335+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:48.742 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:48 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:48.335+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:48.798 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:48 smithi188 podman[76740]: 2024-04-23 18:49:48.556378457 +0000 UTC m=+6.018502937 container cleanup debc47de70033a42cfbebf37b412cddfe650b67d192efb2e77cb1ffd5bb7e332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-23T18:49:48.805 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:48 smithi188 bash[76740]: ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5 2024-04-23T18:49:48.942 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:48.679+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:48.951 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:48.679+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:48.958 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:48.679+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.712074+0000 front 2024-04-23T18:49:22.711969+0000 (oldest deadline 2024-04-23T18:49:48.611998+0000) 2024-04-23T18:49:49.077 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:48 smithi188 podman[76757]: 2024-04-23 18:49:48.807883964 +0000 UTC m=+0.585617590 container remove debc47de70033a42cfbebf37b412cddfe650b67d192efb2e77cb1ffd5bb7e332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-23T18:49:49.094 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:48 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:48.860+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:49.101 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:48 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:48.860+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:49.394 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:49 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:49.101+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:49.412 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:49 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:49.101+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:49.419 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:49 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:49.101+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:49.426 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:49 smithi188 podman[76846]: 2024-04-23 18:49:49.21712731 +0000 UTC m=+0.112867839 container create 5b2893f65122a796d2076fe629cf2af8dd5f05c0bb417ccee46c779e54fcfd11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-23T18:49:49.452 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:49 smithi188 podman[76846]: 2024-04-23 18:49:49.122346052 +0000 UTC m=+0.018086588 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:49:49.483 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:49 smithi188 podman[76846]: 2024-04-23 18:49:49.40269762 +0000 UTC m=+0.298438150 container init 5b2893f65122a796d2076fe629cf2af8dd5f05c0bb417ccee46c779e54fcfd11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-23T18:49:49.490 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:49 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:49.070+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:49.498 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:49 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:49.070+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:49.505 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:49 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:49.070+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:49.695 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:49 smithi188 podman[76846]: 2024-04-23 18:49:49.406058304 +0000 UTC m=+0.301798831 container start 5b2893f65122a796d2076fe629cf2af8dd5f05c0bb417ccee46c779e54fcfd11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=) 2024-04-23T18:49:49.753 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:49 smithi188 podman[76846]: 2024-04-23 18:49:49.494313557 +0000 UTC m=+0.390054108 container attach 5b2893f65122a796d2076fe629cf2af8dd5f05c0bb417ccee46c779e54fcfd11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-04-23T18:49:49.761 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:49.465+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:49.771 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:49.465+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:49:49.779 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:49.384+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:49.825 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:49.385+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:49.836 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:49 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:49.385+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:49.942 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:49 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:49.631+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:49.950 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:49 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:49.631+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:50.014 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:49 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:49.631+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.712074+0000 front 2024-04-23T18:49:22.711969+0000 (oldest deadline 2024-04-23T18:49:48.611998+0000) 2024-04-23T18:49:50.024 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:49 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:49.834+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:50.033 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:49 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:49.834+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:50.069 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 23 18:49:49 smithi188 podman[76846]: 2024-04-23 18:49:49.704441036 +0000 UTC m=+0.600181563 container died 5b2893f65122a796d2076fe629cf2af8dd5f05c0bb417ccee46c779e54fcfd11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:49:50.313 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:50 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:50.139+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:50.343 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:50 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:50.139+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:50.439 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:50 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:50.139+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:50.447 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:50.091+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:50.455 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:50.091+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:50.464 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:50.091+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:50.473 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.5.service' 2024-04-23T18:49:50.519 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:49:50.526 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-23T18:49:50.598 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-23T18:49:50.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.6 2024-04-23T18:49:50.786 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:50 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:50.441+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:50.793 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:50 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:50.441+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:49:50.803 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:50 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:50.364+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:50.843 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:50 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:50.364+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:50.864 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:50 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:50.364+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:50.928 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:50.638+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:50.969 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:50.638+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:50.978 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:50.638+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.712074+0000 front 2024-04-23T18:49:22.711969+0000 (oldest deadline 2024-04-23T18:49:48.611998+0000) 2024-04-23T18:49:51.055 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:50 smithi050 systemd[1]: Stopping Ceph osd.6 for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:49:51.064 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:50.928+0000 7f2b2fd2e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:49:51.073 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:50.928+0000 7f2b2fd2e700 -1 osd.6 147 *** Got signal Terminated *** 2024-04-23T18:49:51.083 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:50 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:50.928+0000 7f2b2fd2e700 -1 osd.6 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:49:51.192 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:51 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:51.088+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:51.202 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:51 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:51.088+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:51.210 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:51 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:51.088+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:51.328 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:50 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:50.874+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:51.363 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:50 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:50.874+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:51.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:51 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:51.117+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:51.380 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:51 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:51.117+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:51.463 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:51 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:51.117+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:51.786 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:51 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:51.398+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:51.792 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:51 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:51.398+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:49:51.860 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:51 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:51.374+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:51.908 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:51 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:51.374+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:51.918 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:51 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:51.374+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:51.981 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:51 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:51.647+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:51.988 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:51 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:51.647+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:51.997 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:51 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:51.647+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.712074+0000 front 2024-04-23T18:49:22.711969+0000 (oldest deadline 2024-04-23T18:49:48.611998+0000) 2024-04-23T18:49:52.327 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:51 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:51.874+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:52.336 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:51 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:51.874+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:52.343 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:52 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:52.123+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:52.350 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:52 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:52.123+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:52.431 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:52 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:52.123+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:52.442 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:52.114+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:52.450 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:52.114+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:52.457 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:52.114+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:52.786 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:52 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:52.393+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:52.793 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:52 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:52.393+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:49:52.802 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:52 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:52.365+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:52.844 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:52 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:52.365+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:52.880 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:52 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:52.365+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:52.943 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:52.654+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:52.989 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:52.654+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:52.997 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:52.654+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.712074+0000 front 2024-04-23T18:49:22.711969+0000 (oldest deadline 2024-04-23T18:49:48.611998+0000) 2024-04-23T18:49:53.077 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:52 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:52.827+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:53.108 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:52 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:52.827+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:53.115 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:52 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:52.827+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:49:53.443 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:53.162+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:53.453 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:53.162+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:53.508 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:53.162+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:53.518 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:53.162+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:49:53.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:53 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:53.117+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:53.587 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:53 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:53.117+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:53.667 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:53 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:53.117+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:53.785 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:53 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:53.344+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:53.795 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:53 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:53.344+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:49:53.803 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:53 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:53.344+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:27.443234+0000 front 2024-04-23T18:49:27.443368+0000 (oldest deadline 2024-04-23T18:49:53.343429+0000) 2024-04-23T18:49:53.836 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:53 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:53.386+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:53.869 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:53 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:53.386+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:53.930 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:53 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:53.386+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:53.943 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:53.641+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:53.952 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:53.641+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:53.960 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:53 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:53.641+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.712074+0000 front 2024-04-23T18:49:22.711969+0000 (oldest deadline 2024-04-23T18:49:48.611998+0000) 2024-04-23T18:49:54.086 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:53 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:53.838+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:54.093 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:53 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:53.838+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:54.105 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:53 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:53.838+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:49:54.443 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:54 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:54.151+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:54.450 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:54 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:54.151+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:54.459 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:54 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:54.151+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:54.469 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:54 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:54.151+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:49:54.577 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:54 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:54.095+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:54.587 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:54 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:54.095+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:54.596 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:54 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:54.095+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:54.786 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:54 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:54.295+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:54.817 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:54 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:54.295+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:49:54.826 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:54 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:54.295+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:27.443234+0000 front 2024-04-23T18:49:27.443368+0000 (oldest deadline 2024-04-23T18:49:53.343429+0000) 2024-04-23T18:49:54.896 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:54 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:54.347+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:54.905 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:54 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:54.347+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:54.914 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:54 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:54.347+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:54.942 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:54 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:54.613+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:54.952 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:54 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:54.613+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:54.960 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:54 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:54.613+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.712074+0000 front 2024-04-23T18:49:22.711969+0000 (oldest deadline 2024-04-23T18:49:48.611998+0000) 2024-04-23T18:49:54.969 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:54 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:54.613+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.612407+0000 front 2024-04-23T18:49:28.612447+0000 (oldest deadline 2024-04-23T18:49:53.912604+0000) 2024-04-23T18:49:55.136 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:54 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:54.880+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:55.160 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:54 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:54.880+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:55.167 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:54 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:54.880+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:49:55.443 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:55 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:55.192+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:55.455 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:55 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:55.192+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:55.464 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:55 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:55.192+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:55.473 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:55 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:55.192+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:49:55.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:55 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:55.144+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:55.584 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:55 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:55.144+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:55.593 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:55 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:55.144+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:55.786 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:55 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:55.317+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:55.796 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:55 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:55.317+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:49:55.805 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:55 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:55.317+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:27.443234+0000 front 2024-04-23T18:49:27.443368+0000 (oldest deadline 2024-04-23T18:49:53.343429+0000) 2024-04-23T18:49:55.814 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:55 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:55.363+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:55.821 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:55 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:55.363+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:55.829 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:55 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:55.363+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:55.836 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:55 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:55.363+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:49:55.943 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:55 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:55.578+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.808115+0000 front 2024-04-23T18:49:04.808087+0000 (oldest deadline 2024-04-23T18:49:28.308209+0000) 2024-04-23T18:49:55.949 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:55 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:55.578+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:09.409568+0000 front 2024-04-23T18:49:09.409379+0000 (oldest deadline 2024-04-23T18:49:35.309344+0000) 2024-04-23T18:49:55.956 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:55 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:55.578+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.712074+0000 front 2024-04-23T18:49:22.711969+0000 (oldest deadline 2024-04-23T18:49:48.611998+0000) 2024-04-23T18:49:55.964 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:55 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6[55500]: 2024-04-23T18:49:55.578+0000 7f2b2c322700 -1 osd.6 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.612407+0000 front 2024-04-23T18:49:28.612447+0000 (oldest deadline 2024-04-23T18:49:53.912604+0000) 2024-04-23T18:49:56.178 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:55 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:55.899+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:56.184 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:55 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:55.899+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:56.191 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:55 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:55.899+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:49:56.436 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:56 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:56.175+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:56.458 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:56 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:56.175+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:56.466 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:56 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:56.175+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:56.530 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:56 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:56.175+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:49:56.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:56 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:56.186+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:56.585 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:56 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:56.186+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:56.593 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:56 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:56.186+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:56.693 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:56 smithi050 podman[100926]: 2024-04-23 18:49:56.444977191 +0000 UTC m=+5.625569404 container died e126837afdae2fda8a829890b1b36dd1da3eb49e52a309900a2364bccee9a03f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-04-23T18:49:56.786 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:56 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:56.332+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:56.792 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:56 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:56.332+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:49:56.853 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:56 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:56.332+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:27.443234+0000 front 2024-04-23T18:49:27.443368+0000 (oldest deadline 2024-04-23T18:49:53.343429+0000) 2024-04-23T18:49:56.863 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:56 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:56.399+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:56.872 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:56 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:56.399+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:56.881 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:56 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:56.399+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:56.937 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:56 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:56.399+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:49:57.062 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:56 smithi050 podman[100926]: 2024-04-23 18:49:56.795015666 +0000 UTC m=+5.975607871 container cleanup e126837afdae2fda8a829890b1b36dd1da3eb49e52a309900a2364bccee9a03f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-23T18:49:57.069 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:56 smithi050 bash[100926]: ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6 2024-04-23T18:49:57.206 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:56 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:56.932+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:57.238 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:56 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:56.932+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:57.247 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:56 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:56.932+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:49:57.438 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:57 smithi050 podman[100942]: 2024-04-23 18:49:57.071312961 +0000 UTC m=+0.624561169 container remove e126837afdae2fda8a829890b1b36dd1da3eb49e52a309900a2364bccee9a03f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True) 2024-04-23T18:49:57.446 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:57 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:57.141+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:57.454 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:57 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:57.141+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:57.463 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:57 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:57.141+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:57.472 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:57 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:57.141+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:49:57.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:57 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:57.214+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:57.585 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:57 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:57.214+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:57.672 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:57 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:57.214+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:57.693 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:57 smithi050 podman[101047]: 2024-04-23 18:49:57.447011791 +0000 UTC m=+0.102835998 container create f92396cce54a55e6b6c59583b78e482705f5e3a6292b9489349bb1aeb4813635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:49:57.725 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:57 smithi050 podman[101047]: 2024-04-23 18:49:57.367221087 +0000 UTC m=+0.023045316 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:49:57.734 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:57 smithi050 podman[101047]: 2024-04-23 18:49:57.64954182 +0000 UTC m=+0.305366037 container init f92396cce54a55e6b6c59583b78e482705f5e3a6292b9489349bb1aeb4813635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-04-23T18:49:57.804 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:57 smithi050 podman[101047]: 2024-04-23 18:49:57.652760581 +0000 UTC m=+0.308584797 container start f92396cce54a55e6b6c59583b78e482705f5e3a6292b9489349bb1aeb4813635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:49:57.813 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:57 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:57.348+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:57.822 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:57 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:57.348+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:49:57.867 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:57 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:57.348+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:27.443234+0000 front 2024-04-23T18:49:27.443368+0000 (oldest deadline 2024-04-23T18:49:53.343429+0000) 2024-04-23T18:49:57.888 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:57 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:57.354+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:57.897 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:57 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:57.354+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:57.966 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:57 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:57.354+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:57.975 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:57 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:57.354+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:49:58.165 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:57 smithi050 podman[101047]: 2024-04-23 18:49:57.732578043 +0000 UTC m=+0.388402266 container attach f92396cce54a55e6b6c59583b78e482705f5e3a6292b9489349bb1aeb4813635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-23T18:49:58.176 INFO:journalctl@ceph.osd.6.smithi050.stdout:Apr 23 18:49:57 smithi050 podman[101047]: 2024-04-23 18:49:57.955470318 +0000 UTC m=+0.611294516 container died f92396cce54a55e6b6c59583b78e482705f5e3a6292b9489349bb1aeb4813635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-04-23T18:49:58.328 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:58 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:58.178+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:58.334 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:58 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:58.178+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:58.341 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:58 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:58.178+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:58.399 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:57 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:57.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:58.406 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:57 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:57.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:58.413 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:57 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:57.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:49:58.443 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:58 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:58.173+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:58.503 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:58 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:58.173+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:58.511 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:58 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:58.173+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:58.521 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:58 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:58.173+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:49:58.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.6.service' 2024-04-23T18:49:58.623 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:49:58.636 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-23T18:49:58.699 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-23T18:49:58.733 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.7 2024-04-23T18:49:58.772 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:58 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:58.301+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:58.778 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:58 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:58.301+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:49:58.785 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:58 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:58.301+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:27.443234+0000 front 2024-04-23T18:49:27.443368+0000 (oldest deadline 2024-04-23T18:49:53.343429+0000) 2024-04-23T18:49:58.793 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:58 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:58.396+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:58.801 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:58 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:58.396+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:58.848 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:58 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:58.396+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:58.857 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:58 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:58.396+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:49:59.036 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:58 smithi177 systemd[1]: Stopping Ceph osd.7 for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:49:59.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:58 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:58.903+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:49:59.232 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:58 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:58.903+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:49:59.240 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:58 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:58.903+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:49:59.310 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:59 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:59.055+0000 7fe555c33700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:49:59.334 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:59 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:59.055+0000 7fe555c33700 -1 osd.7 147 *** Got signal Terminated *** 2024-04-23T18:49:59.363 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:59 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:59.055+0000 7fe555c33700 -1 osd.7 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:49:59.373 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:59 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:59.319+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:49:59.433 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:59 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:59.319+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:49:59.440 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:49:59 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:49:59.319+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:27.443234+0000 front 2024-04-23T18:49:27.443368+0000 (oldest deadline 2024-04-23T18:49:53.343429+0000) 2024-04-23T18:49:59.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:59 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:59.223+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:49:59.586 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:59 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:59.223+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:49:59.593 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:49:59 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:49:59.223+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:49:59.693 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:59 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:59.205+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:49:59.700 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:59 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:59.205+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:49:59.751 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:59 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:59.205+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:49:59.762 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:49:59 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:49:59.205+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:49:59.786 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:59 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:59.428+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:49:59.795 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:59 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:59.428+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:49:59.863 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:59 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:59.428+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:49:59.873 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:49:59 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:49:59.428+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:00.174 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:59 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:59.925+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:00.199 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:59 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:59.925+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:00.232 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:49:59 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:49:59.925+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:00.536 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:00 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:50:00.293+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:50:00.545 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:00 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:50:00.293+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:50:00.555 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:00 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:50:00.293+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:27.443234+0000 front 2024-04-23T18:49:27.443368+0000 (oldest deadline 2024-04-23T18:49:53.343429+0000) 2024-04-23T18:50:00.563 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:00 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:00.452+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:00.572 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:00 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:00.452+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:00.579 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:00 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:00.452+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:00.587 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:00 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:00.452+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:00.595 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:00 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:00.182+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:00.665 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:00 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:00.182+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:00.674 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:00 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:00.182+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:00.693 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:00 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:00.225+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:00.701 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:00 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:00.225+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:00.727 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:00 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:00.225+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:00.737 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:00 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:00.225+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:01.243 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:01 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:50:01.251+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:50:01.258 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:01 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:50:01.251+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:50:01.305 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:01 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:50:01.251+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:27.443234+0000 front 2024-04-23T18:49:27.443368+0000 (oldest deadline 2024-04-23T18:49:53.343429+0000) 2024-04-23T18:50:01.328 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:01 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:01.192+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:01.442 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:01 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:01.192+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:01.453 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:01 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:01.192+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:01.462 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:01 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:01.192+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:01.470 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:00 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:00.958+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:01.477 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:00 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:00.958+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:01.485 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:00 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:00.958+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:01.491 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:00 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:00.958+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:38.843021+0000 front 2024-04-23T18:49:38.842812+0000 (oldest deadline 2024-04-23T18:50:00.542836+0000) 2024-04-23T18:50:01.553 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:01 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:01.418+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:01.569 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:01 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:01.418+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:01.577 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:01 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:01.418+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:01.584 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:01 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:01.418+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:01.664 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:01 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:01.418+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:01.672 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:01.184+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:01.679 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:01.184+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:01.729 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:01.184+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:01.757 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:01 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:01.184+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:02.221 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:01 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:01.960+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:02.228 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:01 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:01.960+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:02.237 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:01 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:01.960+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:02.246 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:01 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:01.960+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:38.843021+0000 front 2024-04-23T18:49:38.842812+0000 (oldest deadline 2024-04-23T18:50:00.542836+0000) 2024-04-23T18:50:02.443 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:02 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:02.147+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:02.456 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:02 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:02.147+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:02.465 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:02 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:02.147+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:02.474 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:02 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:02.147+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:02.537 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:02 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:50:02.251+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:50:02.546 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:02 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:50:02.251+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:50:02.555 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:02 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:50:02.251+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:27.443234+0000 front 2024-04-23T18:49:27.443368+0000 (oldest deadline 2024-04-23T18:49:53.343429+0000) 2024-04-23T18:50:02.563 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:02 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:02.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:02.623 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:02 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:02.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:02.633 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:02 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:02.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:02.642 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:02 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:02.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:02.699 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:02 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:02.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:02.737 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:02 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:02.229+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:02.747 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:02 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:02.229+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:02.828 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:02 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:02.229+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:02.838 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:02 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:02.229+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:03.190 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:02 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:02.937+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:03.218 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:02 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:02.937+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:03.231 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:02 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:02.937+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:03.240 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:02 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:02.937+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:38.843021+0000 front 2024-04-23T18:49:38.842812+0000 (oldest deadline 2024-04-23T18:50:00.542836+0000) 2024-04-23T18:50:03.443 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:03 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:03.132+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:03.451 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:03 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:03.132+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:03.459 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:03 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:03.132+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:03.468 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:03 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:03.132+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:03.479 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:03 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:03.132+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:03.546 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:03 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:50:03.276+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:13.039904+0000 front 2024-04-23T18:49:13.039925+0000 (oldest deadline 2024-04-23T18:49:35.940055+0000) 2024-04-23T18:50:03.556 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:03 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:50:03.276+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:22.742859+0000 front 2024-04-23T18:49:22.742650+0000 (oldest deadline 2024-04-23T18:49:47.442673+0000) 2024-04-23T18:50:03.565 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:03 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-23T18:50:03.276+0000 7fe54e21f700 -1 osd.7 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:27.443234+0000 front 2024-04-23T18:49:27.443368+0000 (oldest deadline 2024-04-23T18:49:53.343429+0000) 2024-04-23T18:50:03.598 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:03 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:03.406+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:03.628 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:03 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:03.406+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:03.637 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:03 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:03.406+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:03.711 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:03 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:03.406+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:03.717 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:03 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:03.406+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:03.727 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:03 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:03.199+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:03.736 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:03 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:03.199+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:03.747 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:03 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:03.199+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:03.755 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:03 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:03.199+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:04.216 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:03 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:03.916+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:04.234 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:03 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:03.916+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:04.241 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:03 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:03.916+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:04.248 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:03 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:03.916+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:38.843021+0000 front 2024-04-23T18:49:38.842812+0000 (oldest deadline 2024-04-23T18:50:00.542836+0000) 2024-04-23T18:50:04.443 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:04 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:04.123+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:04.452 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:04 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:04.123+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:04.460 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:04 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:04.123+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:04.469 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:04 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:04.123+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:04.495 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:04 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:04.123+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:04.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:04 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:04.224+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:04.584 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:04 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:04.224+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:04.635 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:04 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:04.224+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:04.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:04 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:04.224+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:04.786 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:04 smithi177 podman[77245]: 2024-04-23 18:50:04.55117198 +0000 UTC m=+5.602443964 container died b7b3ad0105d137b372d49a5474e35369f2d4f719dfec0063125f8dc27d39b1c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-23T18:50:04.793 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:04.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:04.801 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:04.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:04.808 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:04.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:04.821 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:04.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:04.866 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:04 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:04.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:05.088 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:04 smithi177 podman[77245]: 2024-04-23 18:50:04.880017066 +0000 UTC m=+5.931289037 container cleanup b7b3ad0105d137b372d49a5474e35369f2d4f719dfec0063125f8dc27d39b1c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:50:05.095 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:04 smithi177 bash[77245]: ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7 2024-04-23T18:50:05.180 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:04 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:04.899+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:05.190 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:04 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:04.899+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:05.200 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:04 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:04.899+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:05.210 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:04 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:04.899+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:38.843021+0000 front 2024-04-23T18:49:38.842812+0000 (oldest deadline 2024-04-23T18:50:00.542836+0000) 2024-04-23T18:50:05.426 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:05 smithi177 podman[77264]: 2024-04-23 18:50:05.0976592 +0000 UTC m=+0.543790791 container remove b7b3ad0105d137b372d49a5474e35369f2d4f719dfec0063125f8dc27d39b1c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-23T18:50:05.454 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:05 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:05.140+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:05.464 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:05 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:05.140+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:05.524 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:05 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:05.140+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:05.530 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:05 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:05.140+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:05.541 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:05 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:05.140+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:05.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:05 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:05.188+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:05.584 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:05 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:05.188+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:05.593 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:05 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:05.188+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:05.602 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:05 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:05.188+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:05.733 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:05 smithi177 podman[77355]: 2024-04-23 18:50:05.49852141 +0000 UTC m=+0.114702502 container create 0649ce431122793f311006048a88f3306597edc7a6e78236203182acd90cefd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-23T18:50:05.744 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:05 smithi177 podman[77355]: 2024-04-23 18:50:05.402067033 +0000 UTC m=+0.018248134 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:50:05.753 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:05 smithi177 podman[77355]: 2024-04-23 18:50:05.742650874 +0000 UTC m=+0.358831966 container init 0649ce431122793f311006048a88f3306597edc7a6e78236203182acd90cefd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2) 2024-04-23T18:50:05.762 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:05 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:05.434+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:05.809 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:05 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:05.435+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:05.818 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:05 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:05.435+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:05.827 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:05 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:05.435+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:05.856 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:05 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:05.435+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:06.028 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:05 smithi177 podman[77355]: 2024-04-23 18:50:05.745923839 +0000 UTC m=+0.362104932 container start 0649ce431122793f311006048a88f3306597edc7a6e78236203182acd90cefd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-23T18:50:06.035 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:05 smithi177 podman[77355]: 2024-04-23 18:50:05.834439851 +0000 UTC m=+0.450620952 container attach 0649ce431122793f311006048a88f3306597edc7a6e78236203182acd90cefd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-23T18:50:06.286 INFO:journalctl@ceph.osd.7.smithi177.stdout:Apr 23 18:50:06 smithi177 podman[77355]: 2024-04-23 18:50:06.037356205 +0000 UTC m=+0.653537289 container died 0649ce431122793f311006048a88f3306597edc7a6e78236203182acd90cefd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD) 2024-04-23T18:50:06.328 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:05 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:05.940+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:06.335 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:05 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:05.940+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:06.344 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:05 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:05.940+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:06.355 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:05 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:05.940+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:38.843021+0000 front 2024-04-23T18:49:38.842812+0000 (oldest deadline 2024-04-23T18:50:00.542836+0000) 2024-04-23T18:50:06.364 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:06 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:06.164+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:06.376 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:06 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:06.164+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:06.419 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:06 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:06.164+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:06.433 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:06 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:06.164+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:06.452 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:06 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:06.171+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:06.462 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:06 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:06.171+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:06.529 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:06 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:06.171+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:06.538 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:06 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:06.171+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:06.547 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:06 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:06.171+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:06.654 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:06 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:06.456+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:06.687 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:06 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:06.456+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:06.695 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:06 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:06.456+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:06.747 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:06 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:06.456+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:06.754 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:06 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:06.456+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:06.763 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.7.service' 2024-04-23T18:50:06.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:50:06.848 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-23T18:50:06.858 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-23T18:50:06.867 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.8 2024-04-23T18:50:07.178 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:06 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:06.929+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:07.192 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:06 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:06.929+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:07.200 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:06 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:06.929+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:07.209 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:06 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:06.929+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:38.843021+0000 front 2024-04-23T18:49:38.842812+0000 (oldest deadline 2024-04-23T18:50:00.542836+0000) 2024-04-23T18:50:07.296 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:06 smithi188 systemd[1]: Stopping Ceph osd.8 for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:50:07.443 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:07 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:07.209+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:07.450 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:07 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:07.209+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:07.481 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:07 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:07.209+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:07.531 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:07 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:07.209+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:07.540 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:07 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:07.209+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:07.621 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:07 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:07.257+0000 7f45d6529700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:50:07.628 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:07 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:07.257+0000 7f45d6529700 -1 osd.8 147 *** Got signal Terminated *** 2024-04-23T18:50:07.635 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:07 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:07.257+0000 7f45d6529700 -1 osd.8 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:50:07.642 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:07 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:07.472+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:07.650 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:07 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:07.472+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:07.657 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:07 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:07.472+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:07.664 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:07 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:07.472+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:07.737 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:07 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:07.472+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:07.744 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:07 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:07.187+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:07.752 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:07 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:07.187+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:07.759 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:07 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:07.187+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:07.817 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:07 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:07.187+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:08.328 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:07 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:07.935+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:08.335 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:07 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:07.935+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:08.371 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:07 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:07.935+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:08.404 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:07 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:07.935+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:38.843021+0000 front 2024-04-23T18:49:38.842812+0000 (oldest deadline 2024-04-23T18:50:00.542836+0000) 2024-04-23T18:50:08.414 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:08 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:08.161+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:08.458 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:08 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:08.161+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:08.467 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:08 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:08.161+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:08.477 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:08 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:08.161+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:08.693 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:08 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:08.204+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:08.701 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:08 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:08.204+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:08.732 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:08 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:08.204+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:08.741 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:08 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:08.204+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:08.786 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:08 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:08.204+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:08.797 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:08 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:08.452+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:08.806 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:08 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:08.452+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:08.814 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:08 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:08.452+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:08.859 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:08 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:08.452+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:08.868 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:08 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:08.452+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:09.328 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:09 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:09.153+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:09.341 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:09 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:09.153+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:09.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:09 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:09.153+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:09.381 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:09 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:09.153+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:09.452 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:08 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:08.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:09.459 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:08 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:08.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:09.468 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:08 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:08.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:09.475 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:08 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:08.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:38.843021+0000 front 2024-04-23T18:49:38.842812+0000 (oldest deadline 2024-04-23T18:50:00.542836+0000) 2024-04-23T18:50:09.482 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:08 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:08.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:42.744117+0000 front 2024-04-23T18:49:42.744148+0000 (oldest deadline 2024-04-23T18:50:08.544605+0000) 2024-04-23T18:50:09.693 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:09 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:09.253+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:09.701 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:09 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:09.253+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:09.741 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:09 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:09.253+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:09.769 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:09 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:09.253+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:09.776 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:09 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:09.253+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:09.786 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:09 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:09.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:09.846 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:09 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:09.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:09.853 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:09 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:09.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:09.861 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:09 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:09.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:09.904 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:09 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:09.407+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:10.328 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:09 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:09.954+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:10.335 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:09 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:09.954+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:10.425 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:09 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:09.954+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:10.430 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:09 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:09.954+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:38.843021+0000 front 2024-04-23T18:49:38.842812+0000 (oldest deadline 2024-04-23T18:50:00.542836+0000) 2024-04-23T18:50:10.438 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:09 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:09.954+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:42.744117+0000 front 2024-04-23T18:49:42.744148+0000 (oldest deadline 2024-04-23T18:50:08.544605+0000) 2024-04-23T18:50:10.445 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:10 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:10.184+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:10.452 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:10 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:10.184+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:10.459 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:10 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:10.184+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:10.467 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:10 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:10.184+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:10.513 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:10 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:10.184+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:10.693 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:10 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:10.292+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:10.699 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:10 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:10.292+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:10.706 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:10 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:10.292+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:10.758 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:10 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:10.292+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:10.783 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:10 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:10.292+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:10.790 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:10 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:10.292+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.521985+0000 front 2024-04-23T18:49:46.522025+0000 (oldest deadline 2024-04-23T18:50:10.021943+0000) 2024-04-23T18:50:10.813 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:10 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:10.447+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:10.838 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:10 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:10.447+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:10.847 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:10 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:10.447+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:10.909 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:10 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:10.447+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:10.918 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:10 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:10.447+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:10.927 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:10 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:10.447+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:11.209 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:10 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:10.938+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:11.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:10 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:10.938+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:11.224 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:10 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:10.938+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:11.233 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:10 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:10.938+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:38.843021+0000 front 2024-04-23T18:49:38.842812+0000 (oldest deadline 2024-04-23T18:50:00.542836+0000) 2024-04-23T18:50:11.296 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:10 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:10.938+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:42.744117+0000 front 2024-04-23T18:49:42.744148+0000 (oldest deadline 2024-04-23T18:50:08.544605+0000) 2024-04-23T18:50:11.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:11 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:11.217+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:11.585 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:11 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:11.217+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:11.593 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:11 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:11.217+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:11.655 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:11 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:11.217+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:11.664 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:11 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:11.217+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:11.693 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:11 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:11.309+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:11.751 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:11 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:11.309+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:11.786 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:11 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:11.309+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:11.796 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:11 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:11.309+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:11.873 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:11 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:11.309+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:11.882 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:11 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:11.309+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.521985+0000 front 2024-04-23T18:49:46.522025+0000 (oldest deadline 2024-04-23T18:50:10.021943+0000) 2024-04-23T18:50:11.893 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:11 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:11.469+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:11.902 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:11 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:11.469+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:11.911 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:11 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:11.469+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:11.920 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:11 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:11.469+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:11.929 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:11 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:11.469+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:12.004 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:11 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:11.469+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:12.221 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:11 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:11.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.438581+0000 front 2024-04-23T18:49:14.438562+0000 (oldest deadline 2024-04-23T18:49:37.338657+0000) 2024-04-23T18:50:12.230 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:11 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:11.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.839431+0000 front 2024-04-23T18:49:20.839483+0000 (oldest deadline 2024-04-23T18:49:46.040279+0000) 2024-04-23T18:50:12.239 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:11 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:11.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:30.141548+0000 front 2024-04-23T18:49:30.141597+0000 (oldest deadline 2024-04-23T18:49:52.441454+0000) 2024-04-23T18:50:12.299 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:11 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:11.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:38.843021+0000 front 2024-04-23T18:49:38.842812+0000 (oldest deadline 2024-04-23T18:50:00.542836+0000) 2024-04-23T18:50:12.305 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:11 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8[51568]: 2024-04-23T18:50:11.952+0000 7f45d2b1d700 -1 osd.8 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:42.744117+0000 front 2024-04-23T18:49:42.744148+0000 (oldest deadline 2024-04-23T18:50:08.544605+0000) 2024-04-23T18:50:12.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:12 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:12.229+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:12.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:12 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:12.229+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:12.654 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:12 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:12.229+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:12.663 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:12 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:12.229+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:12.760 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:12 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:12.229+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:12.768 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:12.293+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:12.777 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:12.293+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:12.801 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:12.293+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:12.811 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:12.293+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:12.819 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:12.293+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:12.827 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:12 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:12.293+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.521985+0000 front 2024-04-23T18:49:46.522025+0000 (oldest deadline 2024-04-23T18:50:10.021943+0000) 2024-04-23T18:50:12.894 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:12 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:12.446+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:12.902 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:12 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:12.446+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:12.912 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:12 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:12.446+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:12.918 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:12 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:12.446+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:12.970 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:12 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:12.446+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:12.977 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:12 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:12.446+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:13.078 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:12 smithi188 podman[76965]: 2024-04-23 18:50:12.764867381 +0000 UTC m=+5.629631191 container died 24e4f3519493267e1c558c3e725944fdd713dc9c02a4e4ba214e1c356ba7e868 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-23T18:50:13.483 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:13 smithi188 podman[76965]: 2024-04-23 18:50:13.295662453 +0000 UTC m=+6.160426249 container cleanup 24e4f3519493267e1c558c3e725944fdd713dc9c02a4e4ba214e1c356ba7e868 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:50:13.496 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:13 smithi188 bash[76965]: ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8 2024-04-23T18:50:13.503 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:13 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:13.191+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:13.583 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:13 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:13.191+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:13.595 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:13 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:13.191+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:13.603 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:13 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:13.191+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:13.611 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:13 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:13.191+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:13.693 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:13 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:13.299+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:13.700 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:13 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:13.299+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:13.709 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:13 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:13.299+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:13.748 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:13 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:13.299+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:13.774 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:13 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:13.299+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:13.783 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:13 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:13.299+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.521985+0000 front 2024-04-23T18:49:46.522025+0000 (oldest deadline 2024-04-23T18:50:10.021943+0000) 2024-04-23T18:50:13.792 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:13 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:13.490+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:13.852 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:13 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:13.490+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:13.862 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:13 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:13.490+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:13.872 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:13 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:13.490+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:13.904 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:13 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:13.490+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:13.939 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:13 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:13.490+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:13.949 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:13 smithi188 podman[76982]: 2024-04-23 18:50:13.491901929 +0000 UTC m=+0.730396618 container remove 24e4f3519493267e1c558c3e725944fdd713dc9c02a4e4ba214e1c356ba7e868 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-23T18:50:14.186 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:13 smithi188 podman[77070]: 2024-04-23 18:50:13.805557324 +0000 UTC m=+0.017862841 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:50:14.216 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:13 smithi188 podman[77070]: 2024-04-23 18:50:13.90937956 +0000 UTC m=+0.121685067 container create 864e20d92d7a8f59a148eeef70a9b777ab9dc16122759ea9a962da964198d22e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-23T18:50:14.224 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:14 smithi188 podman[77070]: 2024-04-23 18:50:14.111867633 +0000 UTC m=+0.324173151 container init 864e20d92d7a8f59a148eeef70a9b777ab9dc16122759ea9a962da964198d22e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-23T18:50:14.233 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:14 smithi188 podman[77070]: 2024-04-23 18:50:14.115242125 +0000 UTC m=+0.327547633 container start 864e20d92d7a8f59a148eeef70a9b777ab9dc16122759ea9a962da964198d22e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:50:14.244 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:14 smithi188 podman[77070]: 2024-04-23 18:50:14.194932017 +0000 UTC m=+0.407237551 container attach 864e20d92d7a8f59a148eeef70a9b777ab9dc16122759ea9a962da964198d22e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:50:14.578 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:14 smithi188 podman[77070]: 2024-04-23 18:50:14.404593874 +0000 UTC m=+0.616899383 container died 864e20d92d7a8f59a148eeef70a9b777ab9dc16122759ea9a962da964198d22e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-23T18:50:14.645 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:14 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:14.195+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:14.704 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:14 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:14.195+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:14.714 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:14 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:14.195+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:14.722 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:14 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:14.195+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:14.730 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:14 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:14.195+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:14.739 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:14 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:14.321+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:14.748 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:14 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:14.321+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:14.756 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:14 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:14.321+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:14.778 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:14 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:14.321+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:14.819 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:14 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:14.321+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:14.826 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:14 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:14.321+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.521985+0000 front 2024-04-23T18:49:46.522025+0000 (oldest deadline 2024-04-23T18:50:10.021943+0000) 2024-04-23T18:50:14.924 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:14 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:14.538+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:14.931 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:14 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:14.538+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:14.939 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:14 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:14.538+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:14.946 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:14 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:14.538+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:14.954 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:14 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:14.538+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:14.963 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:14 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:14.538+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:14.972 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 23 18:50:14 smithi188 podman[77096]: 2024-04-23 18:50:14.931642098 +0000 UTC m=+0.521361885 container remove 864e20d92d7a8f59a148eeef70a9b777ab9dc16122759ea9a962da964198d22e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:50:15.029 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.8.service' 2024-04-23T18:50:15.077 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:50:15.153 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-23T18:50:15.162 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-23T18:50:15.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.9 2024-04-23T18:50:15.458 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:15 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:15.209+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:15.481 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:15 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:15.209+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:15.530 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:15 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:15.209+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:15.564 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:15 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:15.209+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:15.571 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:15 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:15.209+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:15.579 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:15 smithi050 systemd[1]: Stopping Ceph osd.9 for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:50:15.588 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:15 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:15.278+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:15.595 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:15 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:15.278+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:15.678 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:15 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:15.278+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:15.684 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:15 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:15.278+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:15.692 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:15 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:15.278+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:15.754 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:15 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:15.278+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.521985+0000 front 2024-04-23T18:49:46.522025+0000 (oldest deadline 2024-04-23T18:50:10.021943+0000) 2024-04-23T18:50:15.943 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:15 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:15.489+0000 7fd5e4fc4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:50:15.954 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:15 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:15.489+0000 7fd5e4fc4700 -1 osd.9 147 *** Got signal Terminated *** 2024-04-23T18:50:15.962 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:15 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:15.489+0000 7fd5e4fc4700 -1 osd.9 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:50:16.036 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:15 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:15.581+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:16.043 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:15 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:15.581+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:16.104 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:15 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:15.581+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:16.111 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:15 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:15.581+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:16.136 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:15 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:15.581+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:16.185 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:15 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:15.581+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:16.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:16 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:16.166+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:16.606 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:16 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:16.166+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:16.616 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:16 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:16.166+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:16.624 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:16 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:16.166+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:16.633 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:16 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:16.166+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:16.693 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:16 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:16.263+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:16.701 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:16 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:16.263+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:16.759 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:16 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:16.263+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:16.831 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:16 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:16.263+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:16.840 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:16 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:16.263+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:16.848 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:16 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:16.263+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.521985+0000 front 2024-04-23T18:49:46.522025+0000 (oldest deadline 2024-04-23T18:50:10.021943+0000) 2024-04-23T18:50:17.036 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:16 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:16.554+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:17.049 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:16 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:16.554+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:17.057 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:16 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:16.554+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:17.066 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:16 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:16.554+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:17.115 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:16 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:16.554+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:17.148 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:16 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:16.554+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:17.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:17 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:17.155+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:17.607 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:17 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:17.155+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:17.716 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:17 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:17.155+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:17.729 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:17 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:17.155+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:17.738 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:17 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:17.155+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:17.747 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:17 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:17.235+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:17.758 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:17 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:17.235+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:17.767 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:17 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:17.235+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:17.816 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:17 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:17.235+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:17.902 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:17 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:17.235+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:17.935 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:17 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:17.235+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.521985+0000 front 2024-04-23T18:49:46.522025+0000 (oldest deadline 2024-04-23T18:50:10.021943+0000) 2024-04-23T18:50:17.945 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:17 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:17.556+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:17.954 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:17 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:17.556+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:17.965 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:17 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:17.556+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:17.974 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:17 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:17.556+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:18.118 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:17 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:17.556+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:18.170 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:17 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:17.556+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:18.479 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:18 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:18.195+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:18.487 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:18 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:18.195+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:18.494 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:18 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:18.195+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:18.501 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:18 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:18.195+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:18.534 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:18 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:18.195+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:18.693 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:18 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:18.204+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:18.713 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:18 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:18.204+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:18.770 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:18 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:18.204+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:18.800 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:18 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:18.204+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:18.807 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:18 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:18.204+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:18.815 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:18 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:18.204+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.521985+0000 front 2024-04-23T18:49:46.522025+0000 (oldest deadline 2024-04-23T18:50:10.021943+0000) 2024-04-23T18:50:18.822 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:18 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:18.204+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.223360+0000 front 2024-04-23T18:49:55.223340+0000 (oldest deadline 2024-04-23T18:50:18.024041+0000) 2024-04-23T18:50:18.956 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:18 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:18.547+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:18.963 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:18 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:18.547+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:18.969 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:18 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:18.547+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:19.003 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:18 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:18.547+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:19.033 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:18 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:18.547+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:19.107 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:18 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:18.547+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:19.116 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:18 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:18.547+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:53.759038+0000 front 2024-04-23T18:49:53.759176+0000 (oldest deadline 2024-04-23T18:50:17.858899+0000) 2024-04-23T18:50:19.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:19 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:19.191+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:19.607 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:19 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:19.191+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:19.616 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:19 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:19.191+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:19.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:19 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:19.191+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:19.632 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:19 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:19.191+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:19.642 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:19 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:19.191+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:19.700 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:19 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:19.206+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:19.708 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:19 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:19.206+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:19.719 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:19 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:19.206+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:19.728 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:19 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:19.206+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:19.737 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:19 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:19.206+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:19.750 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:19 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:19.206+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.521985+0000 front 2024-04-23T18:49:46.522025+0000 (oldest deadline 2024-04-23T18:50:10.021943+0000) 2024-04-23T18:50:19.764 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:19 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:19.206+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.223360+0000 front 2024-04-23T18:49:55.223340+0000 (oldest deadline 2024-04-23T18:50:18.024041+0000) 2024-04-23T18:50:19.871 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:19 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:19.570+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:19.918 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:19 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:19.570+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:19.929 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:19 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:19.570+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:20.005 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:19 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:19.570+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:20.016 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:19 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:19.570+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:20.025 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:19 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:19.570+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:20.035 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:19 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:19.570+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:53.759038+0000 front 2024-04-23T18:49:53.759176+0000 (oldest deadline 2024-04-23T18:50:17.858899+0000) 2024-04-23T18:50:20.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:20 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:20.160+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:20.643 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:20 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:20.160+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:20.705 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:20 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:20.160+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:20.714 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:20 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:20.160+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:20.722 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:20 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:20.160+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:20.731 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:20 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:20.160+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:20.755 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:20 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:20.222+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:04.312896+0000 front 2024-04-23T18:49:04.312869+0000 (oldest deadline 2024-04-23T18:49:28.413005+0000) 2024-04-23T18:50:20.765 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:20 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:20.222+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.214601+0000 front 2024-04-23T18:49:11.214894+0000 (oldest deadline 2024-04-23T18:49:36.514618+0000) 2024-04-23T18:50:20.776 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:20 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:20.222+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.516522+0000 front 2024-04-23T18:49:20.516350+0000 (oldest deadline 2024-04-23T18:49:45.816532+0000) 2024-04-23T18:50:20.784 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:20 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:20.222+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.717567+0000 front 2024-04-23T18:49:28.717613+0000 (oldest deadline 2024-04-23T18:49:52.118308+0000) 2024-04-23T18:50:20.819 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:20 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:20.222+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.620502+0000 front 2024-04-23T18:49:36.620426+0000 (oldest deadline 2024-04-23T18:50:02.520313+0000) 2024-04-23T18:50:20.856 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:20 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:20.222+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.521985+0000 front 2024-04-23T18:49:46.522025+0000 (oldest deadline 2024-04-23T18:50:10.021943+0000) 2024-04-23T18:50:20.899 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:20 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9[62368]: 2024-04-23T18:50:20.222+0000 7fd5dd5b0700 -1 osd.9 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.223360+0000 front 2024-04-23T18:49:55.223340+0000 (oldest deadline 2024-04-23T18:50:18.024041+0000) 2024-04-23T18:50:21.036 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:20 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:20.548+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:21.051 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:20 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:20.548+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:21.062 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:20 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:20.548+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:21.142 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:20 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:20.548+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:21.201 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:20 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:20.548+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:21.215 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:20 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:20.548+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:21.224 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:20 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:20.548+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:53.759038+0000 front 2024-04-23T18:49:53.759176+0000 (oldest deadline 2024-04-23T18:50:17.858899+0000) 2024-04-23T18:50:21.279 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:20 smithi050 podman[101156]: 2024-04-23 18:50:20.893409572 +0000 UTC m=+5.501318603 container died 2d03b62f9fc1c11738d3089cff6b892c86d92798509de095ffacf0d963f52724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:50:21.443 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:21 smithi050 podman[101156]: 2024-04-23 18:50:21.355296836 +0000 UTC m=+5.963205901 container cleanup 2d03b62f9fc1c11738d3089cff6b892c86d92798509de095ffacf0d963f52724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-23T18:50:21.478 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:21 smithi050 bash[101156]: ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9 2024-04-23T18:50:21.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:21 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:21.140+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:21.585 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:21 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:21.140+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:21.593 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:21 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:21.140+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:21.600 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:21 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:21.140+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:21.651 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:21 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:21.140+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:21.692 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:21 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:21.140+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:21.786 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:21 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:21.504+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:21.793 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:21 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:21.504+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:21.801 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:21 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:21.504+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:21.809 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:21 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:21.504+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:21.818 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:21 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:21.504+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:21.826 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:21 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:21.504+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:21.834 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:21 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:21.504+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:53.759038+0000 front 2024-04-23T18:49:53.759176+0000 (oldest deadline 2024-04-23T18:50:17.858899+0000) 2024-04-23T18:50:21.943 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:21 smithi050 podman[101173]: 2024-04-23 18:50:21.648604065 +0000 UTC m=+0.748188611 container remove 2d03b62f9fc1c11738d3089cff6b892c86d92798509de095ffacf0d963f52724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD) 2024-04-23T18:50:22.251 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:21 smithi050 podman[101266]: 2024-04-23 18:50:21.88071909 +0000 UTC m=+0.018609255 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:50:22.300 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:21 smithi050 podman[101266]: 2024-04-23 18:50:21.999371137 +0000 UTC m=+0.137261289 container create 77d4d93574bdf8d7ea8099c3c0bb9b7bca649c3e5de7b387183e78a4fbec7420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-23T18:50:22.313 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:22 smithi050 podman[101266]: 2024-04-23 18:50:22.193539571 +0000 UTC m=+0.331429724 container init 77d4d93574bdf8d7ea8099c3c0bb9b7bca649c3e5de7b387183e78a4fbec7420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-23T18:50:22.327 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:22 smithi050 podman[101266]: 2024-04-23 18:50:22.197095752 +0000 UTC m=+0.334985905 container start 77d4d93574bdf8d7ea8099c3c0bb9b7bca649c3e5de7b387183e78a4fbec7420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-23T18:50:22.337 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:22 smithi050 podman[101266]: 2024-04-23 18:50:22.259929158 +0000 UTC m=+0.397819319 container attach 77d4d93574bdf8d7ea8099c3c0bb9b7bca649c3e5de7b387183e78a4fbec7420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-23T18:50:22.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:22 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:22.144+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:22.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:22 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:22.144+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:22.651 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:22 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:22.144+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:22.681 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:22 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:22.144+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:22.689 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:22 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:22.144+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:22.699 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:22 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:22.144+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:22.709 INFO:journalctl@ceph.osd.9.smithi050.stdout:Apr 23 18:50:22 smithi050 podman[101266]: 2024-04-23 18:50:22.496808183 +0000 UTC m=+0.634698335 container died 77d4d93574bdf8d7ea8099c3c0bb9b7bca649c3e5de7b387183e78a4fbec7420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-23T18:50:22.786 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:22 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:22.478+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:22.795 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:22 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:22.478+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:22.872 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:22 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:22.478+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:22.878 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:22 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:22.478+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:22.936 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:22 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:22.478+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:22.945 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:22 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:22.478+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:22.955 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:22 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:22.478+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:53.759038+0000 front 2024-04-23T18:49:53.759176+0000 (oldest deadline 2024-04-23T18:50:17.858899+0000) 2024-04-23T18:50:23.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.9.service' 2024-04-23T18:50:23.108 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:50:23.115 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-23T18:50:23.122 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-23T18:50:23.131 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.10 2024-04-23T18:50:23.472 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:23 smithi177 systemd[1]: Stopping Ceph osd.10 for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:50:23.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:23.180+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:23.602 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:23.180+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:23.611 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:23.180+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:23.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:23.180+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:23.632 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:23.180+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:23.677 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:23.180+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:23.786 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:23.480+0000 7f32a9ec3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:50:23.827 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:23.480+0000 7f32a9ec3700 -1 osd.10 147 *** Got signal Terminated *** 2024-04-23T18:50:23.891 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:23.480+0000 7f32a9ec3700 -1 osd.10 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:50:23.903 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:23.485+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:23.910 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:23.485+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:23.920 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:23.485+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:23.930 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:23.485+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:23.940 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:23.485+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:23.950 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:23.485+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:23.962 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:23 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:23.485+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:53.759038+0000 front 2024-04-23T18:49:53.759176+0000 (oldest deadline 2024-04-23T18:50:17.858899+0000) 2024-04-23T18:50:24.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:24 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:24.206+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:24.602 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:24 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:24.206+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:24.616 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:24 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:24.206+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:24.627 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:24 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:24.206+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:24.650 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:24 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:24.206+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:24.659 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:24 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:24.206+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:24.786 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:24 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:24.451+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:24.799 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:24 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:24.451+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:24.809 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:24 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:24.451+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:24.838 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:24 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:24.451+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:24.851 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:24 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:24.451+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:24.863 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:24 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:24.451+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:24.876 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:24 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:24.451+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:53.759038+0000 front 2024-04-23T18:49:53.759176+0000 (oldest deadline 2024-04-23T18:50:17.858899+0000) 2024-04-23T18:50:25.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:25 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:25.248+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:25.605 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:25 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:25.248+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:25.613 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:25 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:25.248+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:25.621 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:25 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:25.248+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:25.675 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:25 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:25.248+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:25.746 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:25 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:25.248+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:25.755 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:25 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:25.248+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6820 osd.7 since back 2024-04-23T18:49:59.186170+0000 front 2024-04-23T18:49:59.186225+0000 (oldest deadline 2024-04-23T18:50:25.086139+0000) 2024-04-23T18:50:25.787 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:25 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:25.487+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:25.795 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:25 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:25.487+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:25.803 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:25 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:25.487+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:25.810 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:25 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:25.487+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:25.817 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:25 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:25.487+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:25.825 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:25 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:25.487+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:25.876 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:25 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:25.487+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:53.759038+0000 front 2024-04-23T18:49:53.759176+0000 (oldest deadline 2024-04-23T18:50:17.858899+0000) 2024-04-23T18:50:26.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:26 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:26.289+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:26.596 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:26 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:26.289+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:26.605 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:26 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:26.289+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:26.618 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:26 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:26.289+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:26.632 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:26 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:26.289+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:26.647 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:26 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:26.289+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:26.660 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:26 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:26.289+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6820 osd.7 since back 2024-04-23T18:49:59.186170+0000 front 2024-04-23T18:49:59.186225+0000 (oldest deadline 2024-04-23T18:50:25.086139+0000) 2024-04-23T18:50:26.786 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:26 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:26.502+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:26.820 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:26 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:26.502+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:26.828 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:26 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:26.502+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:26.837 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:26 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:26.502+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:26.844 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:26 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:26.502+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:26.852 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:26 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:26.502+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:26.861 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:26 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:26.502+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:53.759038+0000 front 2024-04-23T18:49:53.759176+0000 (oldest deadline 2024-04-23T18:50:17.858899+0000) 2024-04-23T18:50:27.828 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:27 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:27.339+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:27.848 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:27 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:27.339+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:27.882 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:27 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:27.339+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:27.950 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:27 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:27.339+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:27.996 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:27 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:27.339+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:28.005 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:27 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:27.339+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:28.015 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:27 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:27.339+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6820 osd.7 since back 2024-04-23T18:49:59.186170+0000 front 2024-04-23T18:49:59.186225+0000 (oldest deadline 2024-04-23T18:50:25.086139+0000) 2024-04-23T18:50:28.036 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:27 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:27.546+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:05.951771+0000 front 2024-04-23T18:49:05.951788+0000 (oldest deadline 2024-04-23T18:49:28.851857+0000) 2024-04-23T18:50:28.043 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:27 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:27.546+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:11.152857+0000 front 2024-04-23T18:49:11.152944+0000 (oldest deadline 2024-04-23T18:49:41.153522+0000) 2024-04-23T18:50:28.053 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:27 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:27.546+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:21.154127+0000 front 2024-04-23T18:49:21.154117+0000 (oldest deadline 2024-04-23T18:49:44.054100+0000) 2024-04-23T18:50:28.109 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:27 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:27.546+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6814 osd.3 since back 2024-04-23T18:49:28.755122+0000 front 2024-04-23T18:49:28.755201+0000 (oldest deadline 2024-04-23T18:49:55.155607+0000) 2024-04-23T18:50:28.144 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:27 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:27.546+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:36.856539+0000 front 2024-04-23T18:49:36.856553+0000 (oldest deadline 2024-04-23T18:50:00.956566+0000) 2024-04-23T18:50:28.194 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:27 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:27.546+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:46.857395+0000 front 2024-04-23T18:49:46.857446+0000 (oldest deadline 2024-04-23T18:50:10.357416+0000) 2024-04-23T18:50:28.205 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:27 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10[57317]: 2024-04-23T18:50:27.546+0000 7f32a24af700 -1 osd.10 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:53.759038+0000 front 2024-04-23T18:49:53.759176+0000 (oldest deadline 2024-04-23T18:50:17.858899+0000) 2024-04-23T18:50:28.828 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:28.351+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:28.904 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:28.351+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:28.918 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:28.351+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:28.927 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:28.351+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:28.937 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:28.351+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:29.006 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:28.351+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:29.051 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:28 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:28.351+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6820 osd.7 since back 2024-04-23T18:49:59.186170+0000 front 2024-04-23T18:49:59.186225+0000 (oldest deadline 2024-04-23T18:50:25.086139+0000) 2024-04-23T18:50:29.181 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:28 smithi177 podman[77483]: 2024-04-23 18:50:28.915381058 +0000 UTC m=+5.541075142 container died 1391ad4173dea460afbdd414453f3c8c05956885da28c2d64a2db84adb930a27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:50:29.536 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:29 smithi177 podman[77483]: 2024-04-23 18:50:29.27187463 +0000 UTC m=+5.897568715 container cleanup 1391ad4173dea460afbdd414453f3c8c05956885da28c2d64a2db84adb930a27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-23T18:50:29.566 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:29 smithi177 bash[77483]: ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10 2024-04-23T18:50:29.828 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:29.376+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:29.894 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:29.376+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:29.907 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:29.376+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:29.946 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:29.376+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:29.975 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:29.376+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:29.984 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:29.376+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:29.999 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:29 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:29.376+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6820 osd.7 since back 2024-04-23T18:49:59.186170+0000 front 2024-04-23T18:49:59.186225+0000 (oldest deadline 2024-04-23T18:50:25.086139+0000) 2024-04-23T18:50:30.015 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:29 smithi177 podman[77501]: 2024-04-23 18:50:29.547977447 +0000 UTC m=+0.635798368 container remove 1391ad4173dea460afbdd414453f3c8c05956885da28c2d64a2db84adb930a27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-23T18:50:30.159 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:29 smithi177 podman[77593]: 2024-04-23 18:50:29.748134939 +0000 UTC m=+0.018017288 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:50:30.181 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:29 smithi177 podman[77593]: 2024-04-23 18:50:29.848756092 +0000 UTC m=+0.118638443 container create 4dea0134f20aca9aaf3106a51feca356ac5edbab208ec94cf83036377ccdf31c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-23T18:50:30.190 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:30 smithi177 podman[77593]: 2024-04-23 18:50:30.084449296 +0000 UTC m=+0.354331646 container init 4dea0134f20aca9aaf3106a51feca356ac5edbab208ec94cf83036377ccdf31c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-23T18:50:30.200 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:30 smithi177 podman[77593]: 2024-04-23 18:50:30.087715357 +0000 UTC m=+0.357597708 container start 4dea0134f20aca9aaf3106a51feca356ac5edbab208ec94cf83036377ccdf31c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:50:30.236 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:30 smithi177 podman[77593]: 2024-04-23 18:50:30.167676686 +0000 UTC m=+0.437559045 container attach 4dea0134f20aca9aaf3106a51feca356ac5edbab208ec94cf83036377ccdf31c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-23T18:50:30.536 INFO:journalctl@ceph.osd.10.smithi177.stdout:Apr 23 18:50:30 smithi177 podman[77593]: 2024-04-23 18:50:30.381483855 +0000 UTC m=+0.651366205 container died 4dea0134f20aca9aaf3106a51feca356ac5edbab208ec94cf83036377ccdf31c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:50:30.828 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:30 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:30.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:30.858 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:30 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:30.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:30.885 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:30 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:30.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:30.897 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:30 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:30.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:30.906 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:30 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:30.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:30.915 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:30 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:30.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:30.954 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:30 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:30.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6820 osd.7 since back 2024-04-23T18:49:59.186170+0000 front 2024-04-23T18:49:59.186225+0000 (oldest deadline 2024-04-23T18:50:25.086139+0000) 2024-04-23T18:50:30.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.10.service' 2024-04-23T18:50:31.042 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:50:31.078 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-23T18:50:31.088 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-23T18:50:31.099 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.11 2024-04-23T18:50:31.397 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:31 smithi188 systemd[1]: Stopping Ceph osd.11 for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:50:31.828 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:31 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:31.404+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:31.851 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:31 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:31.404+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:31.908 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:31 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:31.404+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:31.954 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:31 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:31.404+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:31.990 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:31 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:31.404+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:31.998 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:31 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:31.404+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:32.006 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:31 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:31.404+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6820 osd.7 since back 2024-04-23T18:49:59.186170+0000 front 2024-04-23T18:49:59.186225+0000 (oldest deadline 2024-04-23T18:50:25.086139+0000) 2024-04-23T18:50:32.016 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:31 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:31.406+0000 7fe2e00ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:50:32.026 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:31 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:31.406+0000 7fe2e00ca700 -1 osd.11 147 *** Got signal Terminated *** 2024-04-23T18:50:32.041 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:31 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:31.406+0000 7fe2e00ca700 -1 osd.11 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T18:50:32.829 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:32 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:32.363+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:32.849 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:32 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:32.363+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:32.854 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:32 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:32.363+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:32.863 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:32 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:32.363+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:32.871 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:32 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:32.363+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:32.881 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:32 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:32.363+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:32.888 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:32 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:32.363+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6820 osd.7 since back 2024-04-23T18:49:59.186170+0000 front 2024-04-23T18:49:59.186225+0000 (oldest deadline 2024-04-23T18:50:25.086139+0000) 2024-04-23T18:50:33.828 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:33 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:33.343+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:33.852 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:33 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:33.343+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:33.905 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:33 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:33.343+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:33.970 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:33 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:33.343+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:33.979 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:33 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:33.343+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:33.988 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:33 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:33.343+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:33.997 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:33 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:33.343+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6820 osd.7 since back 2024-04-23T18:49:59.186170+0000 front 2024-04-23T18:49:59.186225+0000 (oldest deadline 2024-04-23T18:50:25.086139+0000) 2024-04-23T18:50:34.829 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:34 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:34.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:34.852 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:34 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:34.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:34.861 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:34 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:34.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:34.941 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:34 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:34.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:34.984 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:34 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:34.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:34.996 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:34 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:34.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:35.004 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:34 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:34.384+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6820 osd.7 since back 2024-04-23T18:49:59.186170+0000 front 2024-04-23T18:49:59.186225+0000 (oldest deadline 2024-04-23T18:50:25.086139+0000) 2024-04-23T18:50:35.828 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:35 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:35.353+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:35.845 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:35 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:35.353+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:35.854 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:35 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:35.353+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:35.887 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:35 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:35.353+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:35.896 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:35 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:35.353+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:35.928 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:35 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:35.353+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:35.987 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:35 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:35.353+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6820 osd.7 since back 2024-04-23T18:49:59.186170+0000 front 2024-04-23T18:49:59.186225+0000 (oldest deadline 2024-04-23T18:50:25.086139+0000) 2024-04-23T18:50:35.995 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:35 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:35.353+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2024-04-23T18:50:09.787117+0000 front 2024-04-23T18:50:09.787093+0000 (oldest deadline 2024-04-23T18:50:35.087080+0000) 2024-04-23T18:50:36.579 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:36 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:36.324+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-23T18:49:02.175922+0000 front 2024-04-23T18:49:02.176035+0000 (oldest deadline 2024-04-23T18:49:27.376629+0000) 2024-04-23T18:50:36.614 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:36 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:36.324+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6804 osd.1 since back 2024-04-23T18:49:14.378507+0000 front 2024-04-23T18:49:14.378354+0000 (oldest deadline 2024-04-23T18:49:36.078263+0000) 2024-04-23T18:50:36.621 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:36 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:36.324+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-23T18:49:20.079618+0000 front 2024-04-23T18:49:20.079594+0000 (oldest deadline 2024-04-23T18:49:43.579843+0000) 2024-04-23T18:50:36.628 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:36 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:36.324+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-04-23T18:49:35.282110+0000 front 2024-04-23T18:49:35.282008+0000 (oldest deadline 2024-04-23T18:50:00.582081+0000) 2024-04-23T18:50:36.635 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:36 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:36.324+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-23T18:49:43.983814+0000 front 2024-04-23T18:49:43.983676+0000 (oldest deadline 2024-04-23T18:50:09.883850+0000) 2024-04-23T18:50:36.688 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:36 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:36.324+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.50:6822 osd.6 since back 2024-04-23T18:49:55.184981+0000 front 2024-04-23T18:49:55.184879+0000 (oldest deadline 2024-04-23T18:50:19.185455+0000) 2024-04-23T18:50:36.748 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:36 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:36.324+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.177:6820 osd.7 since back 2024-04-23T18:49:59.186170+0000 front 2024-04-23T18:49:59.186225+0000 (oldest deadline 2024-04-23T18:50:25.086139+0000) 2024-04-23T18:50:36.776 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:36 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11[58436]: 2024-04-23T18:50:36.324+0000 7fe2dc6be700 -1 osd.11 147 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2024-04-23T18:50:09.787117+0000 front 2024-04-23T18:50:09.787093+0000 (oldest deadline 2024-04-23T18:50:35.087080+0000) 2024-04-23T18:50:37.266 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:36 smithi188 podman[77188]: 2024-04-23 18:50:36.993858429 +0000 UTC m=+5.681299863 container died a1e0911f2bb0ddea7a0a21d3f8496d568bddc1d3e03030f55e12b2c362e7abd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:50:37.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:37 smithi188 podman[77188]: 2024-04-23 18:50:37.373689635 +0000 UTC m=+6.061131070 container cleanup a1e0911f2bb0ddea7a0a21d3f8496d568bddc1d3e03030f55e12b2c362e7abd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:50:37.596 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:37 smithi188 bash[77188]: ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11 2024-04-23T18:50:37.923 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:37 smithi188 podman[77205]: 2024-04-23 18:50:37.633116357 +0000 UTC m=+0.636410458 container remove a1e0911f2bb0ddea7a0a21d3f8496d568bddc1d3e03030f55e12b2c362e7abd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:50:38.236 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:37 smithi188 podman[77299]: 2024-04-23 18:50:37.834091497 +0000 UTC m=+0.017873153 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T18:50:38.251 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:37 smithi188 podman[77299]: 2024-04-23 18:50:37.950679454 +0000 UTC m=+0.134461152 container create 27b46ab3f59e4b95c3e57f6b022f9543895b7a69c694a6e99fdc74cbf0af2bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T18:50:38.306 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:38 smithi188 podman[77299]: 2024-04-23 18:50:38.16987946 +0000 UTC m=+0.353661117 container init 27b46ab3f59e4b95c3e57f6b022f9543895b7a69c694a6e99fdc74cbf0af2bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:50:38.430 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:38 smithi188 podman[77299]: 2024-04-23 18:50:38.173389242 +0000 UTC m=+0.357170900 container start 27b46ab3f59e4b95c3e57f6b022f9543895b7a69c694a6e99fdc74cbf0af2bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-23T18:50:38.452 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:38 smithi188 podman[77299]: 2024-04-23 18:50:38.244811621 +0000 UTC m=+0.428593290 container attach 27b46ab3f59e4b95c3e57f6b022f9543895b7a69c694a6e99fdc74cbf0af2bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-23T18:50:38.578 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:38 smithi188 podman[77299]: 2024-04-23 18:50:38.461745519 +0000 UTC m=+0.645527170 container died 27b46ab3f59e4b95c3e57f6b022f9543895b7a69c694a6e99fdc74cbf0af2bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=) 2024-04-23T18:50:39.010 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 23 18:50:38 smithi188 podman[77325]: 2024-04-23 18:50:38.998258331 +0000 UTC m=+0.531701034 container remove 27b46ab3f59e4b95c3e57f6b022f9543895b7a69c694a6e99fdc74cbf0af2bf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-23T18:50:39.027 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@osd.11.service' 2024-04-23T18:50:39.073 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:50:39.081 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-23T18:50:39.218 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-23T18:50:39.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.a 2024-04-23T18:50:39.582 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:50:39 smithi050 systemd[1]: Stopping Ceph mds.a for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:50:39.943 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:50:39 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-a[68755]: 2024-04-23T18:50:39.589+0000 7f50a6162700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:50:39.969 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:50:39 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-a[68755]: 2024-04-23T18:50:39.589+0000 7f50a6162700 -1 mds.a *** got signal Terminated *** 2024-04-23T18:50:42.341 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:50:42 smithi050 podman[101387]: 2024-04-23 18:50:42.075829008 +0000 UTC m=+2.597933269 container died 48c5cf119a73dc3c8f3638e74ecdee1e39f1fa9ab781aad568100372ca0501ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD) 2024-04-23T18:50:42.693 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:50:42 smithi050 podman[101387]: 2024-04-23 18:50:42.448140427 +0000 UTC m=+2.970244694 container cleanup 48c5cf119a73dc3c8f3638e74ecdee1e39f1fa9ab781aad568100372ca0501ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-23T18:50:42.755 INFO:journalctl@ceph.mds.a.smithi050.stdout:Apr 23 18:50:42 smithi050 bash[101387]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-a 2024-04-23T18:50:42.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.a.service' 2024-04-23T18:50:42.928 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:50:42.968 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-23T18:50:42.976 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-23T18:50:42.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.d 2024-04-23T18:50:43.311 INFO:journalctl@ceph.mds.d.smithi050.stdout:Apr 23 18:50:43 smithi050 systemd[1]: Stopping Ceph mds.d for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:50:43.693 INFO:journalctl@ceph.mds.d.smithi050.stdout:Apr 23 18:50:43 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-d[67798]: 2024-04-23T18:50:43.318+0000 7f4d2ab60700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:50:43.712 INFO:journalctl@ceph.mds.d.smithi050.stdout:Apr 23 18:50:43 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-d[67798]: 2024-04-23T18:50:43.318+0000 7f4d2ab60700 -1 mds.d *** got signal Terminated *** 2024-04-23T18:50:47.288 INFO:journalctl@ceph.mds.d.smithi050.stdout:Apr 23 18:50:47 smithi050 podman[101506]: 2024-04-23 18:50:47.016316038 +0000 UTC m=+3.832367038 container died 637b26b1f203ffe36608b7127dc923c7f2810add95d66423d8555a61e1758752 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-23T18:50:47.694 INFO:journalctl@ceph.mds.d.smithi050.stdout:Apr 23 18:50:47 smithi050 podman[101506]: 2024-04-23 18:50:47.495879774 +0000 UTC m=+4.311930763 container cleanup 637b26b1f203ffe36608b7127dc923c7f2810add95d66423d8555a61e1758752 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-23T18:50:47.718 INFO:journalctl@ceph.mds.d.smithi050.stdout:Apr 23 18:50:47 smithi050 bash[101506]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-d 2024-04-23T18:50:47.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.d.service' 2024-04-23T18:50:47.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:50:47.950 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-23T18:50:47.959 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-23T18:50:47.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.g 2024-04-23T18:50:48.444 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:50:48 smithi050 systemd[1]: Stopping Ceph mds.g for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:50:48.460 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:50:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-g[68403]: 2024-04-23T18:50:48.257+0000 7f85070f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:50:48.470 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:50:48 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-g[68403]: 2024-04-23T18:50:48.257+0000 7f85070f8700 -1 mds.g *** got signal Terminated *** 2024-04-23T18:50:51.591 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:50:51 smithi050 podman[101628]: 2024-04-23 18:50:51.338852642 +0000 UTC m=+3.166328594 container died ba154573e0276b92162134c9b40df7dc9658ddf4625d57df92f42b7eb2418226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-23T18:50:51.943 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:50:51 smithi050 podman[101628]: 2024-04-23 18:50:51.699276987 +0000 UTC m=+3.526752936 container cleanup ba154573e0276b92162134c9b40df7dc9658ddf4625d57df92f42b7eb2418226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-23T18:50:51.959 INFO:journalctl@ceph.mds.g.smithi050.stdout:Apr 23 18:50:51 smithi050 bash[101628]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-g 2024-04-23T18:50:52.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.g.service' 2024-04-23T18:50:52.209 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:50:52.218 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-23T18:50:52.225 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-23T18:50:52.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.j 2024-04-23T18:50:52.630 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:50:52 smithi050 systemd[1]: Stopping Ceph mds.j for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:50:52.943 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:50:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-j[67415]: 2024-04-23T18:50:52.637+0000 7fc81f51b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:50:53.006 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:50:52 smithi050 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-j[67415]: 2024-04-23T18:50:52.637+0000 7fc81f51b700 -1 mds.j *** got signal Terminated *** 2024-04-23T18:50:56.330 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:50:56 smithi050 podman[101747]: 2024-04-23 18:50:56.068499189 +0000 UTC m=+3.623566744 container died cea48831c1140f23d1e57b2595929366498d7464b4f04a5837f35c5f544fdb96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:50:56.680 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:50:56 smithi050 podman[101747]: 2024-04-23 18:50:56.420779881 +0000 UTC m=+3.975847438 container cleanup cea48831c1140f23d1e57b2595929366498d7464b4f04a5837f35c5f544fdb96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:50:56.730 INFO:journalctl@ceph.mds.j.smithi050.stdout:Apr 23 18:50:56 smithi050 bash[101747]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-j 2024-04-23T18:50:56.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.j.service' 2024-04-23T18:50:56.900 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:50:56.922 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-23T18:50:56.931 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-23T18:50:56.940 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.b 2024-04-23T18:50:57.246 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:50:56 smithi177 systemd[1]: Stopping Ceph mds.b for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:50:57.501 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:50:57 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b[63024]: 2024-04-23T18:50:57.252+0000 7f6844333700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:50:57.516 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:50:57 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b[63024]: 2024-04-23T18:50:57.252+0000 7f6844333700 -1 mds.b *** got signal Terminated *** 2024-04-23T18:50:58.646 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:50:58 smithi177 podman[77700]: 2024-04-23 18:50:58.394107634 +0000 UTC m=+1.252143265 container died 87e740c0e35fe04487285f419c10974e84f25e1bb199cc30c738da247ba93f63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-23T18:50:58.904 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:50:58 smithi177 podman[77700]: 2024-04-23 18:50:58.728151609 +0000 UTC m=+1.586187230 container cleanup 87e740c0e35fe04487285f419c10974e84f25e1bb199cc30c738da247ba93f63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-23T18:50:58.911 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:50:58 smithi177 bash[77700]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b 2024-04-23T18:50:59.141 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.b.service' 2024-04-23T18:50:59.175 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:50:58 smithi177 podman[77725]: 2024-04-23 18:50:58.912657094 +0000 UTC m=+0.521106413 container remove 87e740c0e35fe04487285f419c10974e84f25e1bb199cc30c738da247ba93f63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-04-23T18:50:59.182 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:50:59 smithi177 systemd[1]: ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.b.service: Deactivated successfully. 2024-04-23T18:50:59.187 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:50:59 smithi177 systemd[1]: Stopped Ceph mds.b for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:50:59.196 INFO:journalctl@ceph.mds.b.smithi177.stdout:Apr 23 18:50:59 smithi177 systemd[1]: ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.b.service: Consumed 11min 36.668s CPU time. 2024-04-23T18:50:59.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:50:59.215 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-23T18:50:59.275 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-23T18:50:59.294 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.e 2024-04-23T18:50:59.537 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:50:59 smithi177 systemd[1]: Stopping Ceph mds.e for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:51:00.037 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:50:59 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e[62392]: 2024-04-23T18:50:59.665+0000 7fdcac055700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:51:00.044 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:50:59 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e[62392]: 2024-04-23T18:50:59.665+0000 7fdcac055700 -1 mds.e *** got signal Terminated *** 2024-04-23T18:51:02.699 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:51:02 smithi177 podman[77831]: 2024-04-23 18:51:02.394351476 +0000 UTC m=+2.835590793 container died 18dff365af235e26491c68160cfa6301c51821148ea5f32083e997f955815ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-23T18:51:03.016 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:51:02 smithi177 podman[77831]: 2024-04-23 18:51:02.831841243 +0000 UTC m=+3.273080577 container cleanup 18dff365af235e26491c68160cfa6301c51821148ea5f32083e997f955815ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-23T18:51:03.056 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:51:02 smithi177 bash[77831]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e 2024-04-23T18:51:03.249 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.e.service' 2024-04-23T18:51:03.276 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:51:03 smithi177 podman[77844]: 2024-04-23 18:51:03.024442482 +0000 UTC m=+0.631457117 container remove 18dff365af235e26491c68160cfa6301c51821148ea5f32083e997f955815ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-23T18:51:03.336 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:51:03 smithi177 systemd[1]: ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.e.service: Deactivated successfully. 2024-04-23T18:51:03.364 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:51:03 smithi177 systemd[1]: Stopped Ceph mds.e for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:51:03.371 INFO:journalctl@ceph.mds.e.smithi177.stdout:Apr 23 18:51:03 smithi177 systemd[1]: ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.e.service: Consumed 31.920s CPU time. 2024-04-23T18:51:03.381 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:51:03.389 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-23T18:51:03.395 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-23T18:51:03.402 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.h 2024-04-23T18:51:03.537 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:51:03 smithi177 systemd[1]: Stopping Ceph mds.h for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:51:04.037 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:51:03 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: 2024-04-23T18:51:03.718+0000 7f312e490700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:51:04.092 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:51:03 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h[66741]: 2024-04-23T18:51:03.718+0000 7f312e490700 -1 mds.h *** got signal Terminated *** 2024-04-23T18:51:06.669 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:51:06 smithi177 podman[77952]: 2024-04-23 18:51:06.392303268 +0000 UTC m=+2.786859627 container died e561ba7acda0a088a906bee368ba6b777c2306fce39be4c43cfcefc9c56c9b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-23T18:51:06.928 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:51:06 smithi177 podman[77952]: 2024-04-23 18:51:06.768776605 +0000 UTC m=+3.163332972 container cleanup e561ba7acda0a088a906bee368ba6b777c2306fce39be4c43cfcefc9c56c9b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-23T18:51:06.937 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:51:06 smithi177 bash[77952]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h 2024-04-23T18:51:07.174 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.h.service' 2024-04-23T18:51:07.253 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:51:06 smithi177 podman[77966]: 2024-04-23 18:51:06.936489851 +0000 UTC m=+0.504633223 container remove e561ba7acda0a088a906bee368ba6b777c2306fce39be4c43cfcefc9c56c9b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:51:07.278 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:51:07 smithi177 systemd[1]: ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.h.service: Deactivated successfully. 2024-04-23T18:51:07.287 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:51:07 smithi177 systemd[1]: Stopped Ceph mds.h for 5d677974-019c-11ef-bc93-c7b262605968. 2024-04-23T18:51:07.295 INFO:journalctl@ceph.mds.h.smithi177.stdout:Apr 23 18:51:07 smithi177 systemd[1]: ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.h.service: Consumed 33.960s CPU time. 2024-04-23T18:51:07.305 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:51:07.314 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-23T18:51:07.322 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-23T18:51:07.329 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.k 2024-04-23T18:51:07.537 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:51:07 smithi177 systemd[1]: Stopping Ceph mds.k for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:51:07.936 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:51:07 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-k[62069]: 2024-04-23T18:51:07.635+0000 7f7d03706700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:51:07.946 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:51:07 smithi177 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-k[62069]: 2024-04-23T18:51:07.635+0000 7f7d03706700 -1 mds.k *** got signal Terminated *** 2024-04-23T18:51:09.614 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:51:09 smithi177 podman[78071]: 2024-04-23 18:51:09.3034222 +0000 UTC m=+1.769205542 container died 6e3b8564481eb3abc27dd45376f912963cb174ca3e0294181f3c69f51bcd1c69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T18:51:09.663 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:51:09 smithi177 podman[78071]: 2024-04-23 18:51:09.622526709 +0000 UTC m=+2.088310042 container cleanup 6e3b8564481eb3abc27dd45376f912963cb174ca3e0294181f3c69f51bcd1c69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-k, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T18:51:10.028 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:51:09 smithi177 bash[78071]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-k 2024-04-23T18:51:10.063 INFO:journalctl@ceph.mds.k.smithi177.stdout:Apr 23 18:51:09 smithi177 podman[78085]: 2024-04-23 18:51:09.831667731 +0000 UTC m=+0.525606874 container remove 6e3b8564481eb3abc27dd45376f912963cb174ca3e0294181f3c69f51bcd1c69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-23T18:51:10.070 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.k.service' 2024-04-23T18:51:10.109 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:51:10.141 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-23T18:51:10.149 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-23T18:51:10.159 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.c 2024-04-23T18:51:10.488 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:51:10 smithi188 systemd[1]: Stopping Ceph mds.c for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:51:10.754 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:51:10 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-c[63700]: 2024-04-23T18:51:10.495+0000 7f69b0d7c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:51:10.763 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:51:10 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-c[63700]: 2024-04-23T18:51:10.495+0000 7f69b0d7c700 -1 mds.c *** got signal Terminated *** 2024-04-23T18:51:14.806 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:51:14 smithi188 podman[77404]: 2024-04-23 18:51:14.42167329 +0000 UTC m=+4.051942911 container died 9ced8839238f399099b97e16c52bbedcc533082d605f316e3eb5d87749009e93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T18:51:15.079 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:51:14 smithi188 podman[77404]: 2024-04-23 18:51:14.88700644 +0000 UTC m=+4.517276058 container cleanup 9ced8839238f399099b97e16c52bbedcc533082d605f316e3eb5d87749009e93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-04-23T18:51:15.099 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 23 18:51:14 smithi188 bash[77404]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-c 2024-04-23T18:51:15.312 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.c.service' 2024-04-23T18:51:15.372 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:51:15.387 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-23T18:51:15.395 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-23T18:51:15.404 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.f 2024-04-23T18:51:15.708 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:51:15 smithi188 systemd[1]: Stopping Ceph mds.f for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:51:16.079 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:51:15 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f[63391]: 2024-04-23T18:51:15.715+0000 7f8a49cb0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:51:16.086 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:51:15 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f[63391]: 2024-04-23T18:51:15.715+0000 7f8a49cb0700 -1 mds.f *** got signal Terminated *** 2024-04-23T18:51:18.766 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:51:18 smithi188 podman[77538]: 2024-04-23 18:51:18.420250664 +0000 UTC m=+2.809485323 container died 3323d250daaa620e5f6439f47bf6aab87e42acdd3e44cf29c6fae511ecc82a84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-23T18:51:19.079 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:51:18 smithi188 podman[77538]: 2024-04-23 18:51:18.998863382 +0000 UTC m=+3.388098039 container cleanup 3323d250daaa620e5f6439f47bf6aab87e42acdd3e44cf29c6fae511ecc82a84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:51:19.095 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 23 18:51:18 smithi188 bash[77538]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-f 2024-04-23T18:51:19.401 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.f.service' 2024-04-23T18:51:19.504 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:51:19.517 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-23T18:51:19.525 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-23T18:51:19.533 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.i 2024-04-23T18:51:19.829 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:51:19 smithi188 systemd[1]: Stopping Ceph mds.i for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:51:20.329 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:51:19 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-i[64020]: 2024-04-23T18:51:19.843+0000 7f942d2ef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:51:20.345 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:51:19 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-i[64020]: 2024-04-23T18:51:19.843+0000 7f942d2ef700 -1 mds.i *** got signal Terminated *** 2024-04-23T18:51:22.662 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:51:22 smithi188 podman[77657]: 2024-04-23 18:51:22.413866331 +0000 UTC m=+2.673745975 container died d24dc11dcb0a5663065f53b9a9bbeead0e14e63ce4d3ac39bab1909ea3fca65a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=) 2024-04-23T18:51:23.021 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:51:22 smithi188 podman[77657]: 2024-04-23 18:51:22.769494708 +0000 UTC m=+3.029374354 container cleanup d24dc11dcb0a5663065f53b9a9bbeead0e14e63ce4d3ac39bab1909ea3fca65a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True) 2024-04-23T18:51:23.036 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 23 18:51:22 smithi188 bash[77657]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-i 2024-04-23T18:51:23.180 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.i.service' 2024-04-23T18:51:23.226 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:51:23.235 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-23T18:51:23.244 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-23T18:51:23.253 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.l 2024-04-23T18:51:23.549 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:51:23 smithi188 systemd[1]: Stopping Ceph mds.l for 5d677974-019c-11ef-bc93-c7b262605968... 2024-04-23T18:51:23.829 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:51:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-l[63089]: 2024-04-23T18:51:23.555+0000 7f133cbc8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T18:51:23.849 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:51:23 smithi188 ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-l[63089]: 2024-04-23T18:51:23.555+0000 7f133cbc8700 -1 mds.l *** got signal Terminated *** 2024-04-23T18:51:26.766 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:51:26 smithi188 podman[77777]: 2024-04-23 18:51:26.481382692 +0000 UTC m=+3.033693810 container died 80cd05b82f1cf7aec72a0017579b1865e537811e78fc55567ecccf63b5f3a768 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T18:51:27.079 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:51:26 smithi188 podman[77777]: 2024-04-23 18:51:26.855970942 +0000 UTC m=+3.408282073 container cleanup 80cd05b82f1cf7aec72a0017579b1865e537811e78fc55567ecccf63b5f3a768 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-l, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-23T18:51:27.107 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 23 18:51:26 smithi188 bash[77777]: ceph-5d677974-019c-11ef-bc93-c7b262605968-mds-l 2024-04-23T18:51:27.231 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5d677974-019c-11ef-bc93-c7b262605968@mds.l.service' 2024-04-23T18:51:27.278 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T18:51:27.324 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-23T18:51:27.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5d677974-019c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T18:51:27.639 INFO:teuthology.orchestra.run.smithi050.stdout:Deleting cluster with fsid: 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:51:30.152 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5d677974-019c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T18:51:30.383 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:51:32.885 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5d677974-019c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T18:51:33.178 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T18:51:35.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:51:35.651 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:51:35.691 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T18:51:35.724 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-23T18:51:35.754 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101/remote/smithi050/crash 2024-04-23T18:51:35.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/crash -- . 2024-04-23T18:51:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:tar: /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T18:51:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:tar: Error is not recoverable: exiting now 2024-04-23T18:51:35.870 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101/remote/smithi177/crash 2024-04-23T18:51:35.877 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/crash -- . 2024-04-23T18:51:35.922 INFO:teuthology.orchestra.run.smithi177.stderr:tar: /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T18:51:35.936 INFO:teuthology.orchestra.run.smithi177.stderr:tar: Error is not recoverable: exiting now 2024-04-23T18:51:35.943 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101/remote/smithi188/crash 2024-04-23T18:51:35.950 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/crash -- . 2024-04-23T18:51:35.994 INFO:teuthology.orchestra.run.smithi188.stderr:tar: /var/lib/ceph/5d677974-019c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T18:51:36.036 INFO:teuthology.orchestra.run.smithi188.stderr:tar: Error is not recoverable: exiting now 2024-04-23T18:51:36.121 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-23T18:51:36.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-23T18:51:36.207 INFO:tasks.cephadm:Compressing logs... 2024-04-23T18:51:36.215 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T18:51:36.251 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T18:51:36.265 DEBUG:teuthology.orchestra.run.smithi188:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T18:51:36.336 INFO:teuthology.orchestra.run.smithi177.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-23T18:51:36.382 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.log 2024-04-23T18:51:36.392 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T18:51:36.400 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-23T18:51:36.409 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-23T18:51:36.436 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-23T18:51:36.459 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-23T18:51:36.468 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-23T18:51:36.477 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-23T18:51:36.486 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-23T18:51:36.495 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T18:51:36.556 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-23T18:51:36.612 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T18:51:36.680 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-23T18:51:36.693 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-23T18:51:36.700 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T18:51:36.712 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 84.5% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T18:51:36.719 INFO:teuthology.orchestra.run.smithi177.stderr: 87.7% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T18:51:36.726 INFO:teuthology.orchestra.run.smithi177.stderr: 90.6% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T18:51:36.733 INFO:teuthology.orchestra.run.smithi177.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T18:51:36.742 INFO:teuthology.orchestra.run.smithi177.stderr: 94.5% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-23T18:51:36.750 INFO:teuthology.orchestra.run.smithi177.stderr: 91.1% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T18:51:36.757 INFO:teuthology.orchestra.run.smithi177.stderr: 92.8% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-23T18:51:36.809 INFO:teuthology.orchestra.run.smithi050.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.log 2024-04-23T18:51:36.837 INFO:teuthology.orchestra.run.smithi050.stderr:: No such file or directory 2024-04-23T18:51:36.847 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T18:51:36.856 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-23T18:51:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.log: gzip/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.audit.log: -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-23T18:51:36.960 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-23T18:51:36.970 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-23T18:51:37.054 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.j.log: /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-23T18:51:37.104 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-23T18:51:37.184 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-23T18:51:37.192 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mon.a.log: /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mgr.x.log: 90.5% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T18:51:37.201 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T18:51:37.209 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T18:51:37.217 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-23T18:51:37.226 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.cephadm.log: 84.6% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T18:51:37.234 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T18:51:37.240 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-23T18:51:37.293 INFO:teuthology.orchestra.run.smithi050.stderr: 94.6% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.j.log.gzgzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-23T18:51:37.301 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-04-23T18:51:37.310 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-23T18:51:37.322 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.7% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-23T18:51:37.331 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T18:51:37.383 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.6% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-23T18:51:37.512 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cephadm.log: 91.1% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T18:51:37.627 INFO:teuthology.orchestra.run.smithi050.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T18:51:37.643 INFO:teuthology.orchestra.run.smithi050.stderr: 90.3% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-23T18:51:37.655 INFO:teuthology.orchestra.run.smithi188.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-23T18:51:37.663 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.log 2024-04-23T18:51:37.672 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-23T18:51:37.680 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-23T18:51:37.689 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T18:51:37.699 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-23T18:51:37.708 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-23T18:51:37.717 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-23T18:51:37.764 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-23T18:51:37.804 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-23T18:51:37.894 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T18:51:37.903 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-23T18:51:37.911 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-23T18:51:37.920 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T18:51:37.928 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-23T18:51:37.971 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T18:51:37.998 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 92.8% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-23T18:51:38.007 INFO:teuthology.orchestra.run.smithi188.stderr: 94.5% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-23T18:51:38.016 INFO:teuthology.orchestra.run.smithi188.stderr: 84.4% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T18:51:38.027 INFO:teuthology.orchestra.run.smithi188.stderr: 90.6% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T18:51:38.036 INFO:teuthology.orchestra.run.smithi188.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T18:51:38.044 INFO:teuthology.orchestra.run.smithi188.stderr: 87.7% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T18:51:38.053 INFO:teuthology.orchestra.run.smithi188.stderr: 91.1% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T18:51:38.062 INFO:teuthology.orchestra.run.smithi050.stderr: 90.2% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-23T18:51:48.911 INFO:teuthology.orchestra.run.smithi050.stderr: 94.0% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-23T18:51:50.851 INFO:teuthology.orchestra.run.smithi050.stderr: 93.7% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-23T18:51:51.699 INFO:teuthology.orchestra.run.smithi050.stderr: 93.9% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-23T18:51:52.919 INFO:teuthology.orchestra.run.smithi188.stderr: 92.0% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-23T18:51:53.126 INFO:teuthology.orchestra.run.smithi050.stderr: 93.8% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-23T18:51:53.128 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-04-23T18:51:53.128 INFO:teuthology.orchestra.run.smithi050.stderr:real 0m16.863s 2024-04-23T18:51:53.129 INFO:teuthology.orchestra.run.smithi050.stderr:user 0m59.587s 2024-04-23T18:51:53.129 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m2.014s 2024-04-23T18:52:25.393 INFO:teuthology.orchestra.run.smithi177.stderr: 93.4% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-23T18:52:59.140 INFO:teuthology.orchestra.run.smithi177.stderr: 94.0% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-23T18:53:09.521 INFO:teuthology.orchestra.run.smithi177.stderr: 93.9% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-23T18:53:11.652 INFO:teuthology.orchestra.run.smithi177.stderr: 94.8% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-23T18:53:27.952 INFO:teuthology.orchestra.run.smithi188.stderr: 94.0% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-23T18:53:39.481 INFO:teuthology.orchestra.run.smithi177.stderr: 93.8% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-23T18:53:40.449 INFO:teuthology.orchestra.run.smithi177.stderr: 95.1% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-23T18:53:43.666 INFO:teuthology.orchestra.run.smithi188.stderr: 93.8% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-23T18:54:11.400 INFO:teuthology.orchestra.run.smithi188.stderr: 94.0% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-23T18:54:17.396 INFO:teuthology.orchestra.run.smithi188.stderr: 94.9% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-23T18:54:20.168 INFO:teuthology.orchestra.run.smithi177.stderr: 94.0% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-23T18:55:24.387 INFO:teuthology.orchestra.run.smithi188.stderr: 93.9% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-23T18:57:22.717 INFO:teuthology.orchestra.run.smithi177.stderr: 93.3% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-23T18:57:22.719 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-23T18:57:22.720 INFO:teuthology.orchestra.run.smithi177.stderr:real 5m46.435s 2024-04-23T18:57:22.720 INFO:teuthology.orchestra.run.smithi177.stderr:user 4m20.581s 2024-04-23T18:57:22.720 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m9.962s 2024-04-23T19:09:09.658 INFO:teuthology.orchestra.run.smithi188.stderr: 93.0% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-23T19:09:36.270 INFO:teuthology.orchestra.run.smithi188.stderr: 93.2% -- replaced with /var/log/ceph/5d677974-019c-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-23T19:09:36.272 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-23T19:09:36.272 INFO:teuthology.orchestra.run.smithi188.stderr:real 17m59.923s 2024-04-23T19:09:36.272 INFO:teuthology.orchestra.run.smithi188.stderr:user 6m47.751s 2024-04-23T19:09:36.272 INFO:teuthology.orchestra.run.smithi188.stderr:sys 0m18.783s 2024-04-23T19:09:36.273 INFO:tasks.cephadm:Archiving logs... 2024-04-23T19:09:36.274 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101/remote/smithi050/log 2024-04-23T19:09:36.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T19:09:44.919 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101/remote/smithi177/log 2024-04-23T19:09:44.920 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T19:10:27.302 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101/remote/smithi188/log 2024-04-23T19:10:27.304 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T19:11:31.459 INFO:tasks.cephadm:Removing cluster... 2024-04-23T19:11:31.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5d677974-019c-11ef-bc93-c7b262605968 --force 2024-04-23T19:11:31.662 INFO:teuthology.orchestra.run.smithi050.stdout:Deleting cluster with fsid: 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T19:11:32.015 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5d677974-019c-11ef-bc93-c7b262605968 --force 2024-04-23T19:11:32.218 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T19:11:32.795 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5d677974-019c-11ef-bc93-c7b262605968 --force 2024-04-23T19:11:33.306 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: 5d677974-019c-11ef-bc93-c7b262605968 2024-04-23T19:11:33.938 INFO:tasks.cephadm:Removing cephadm ... 2024-04-23T19:11:33.939 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T19:11:33.958 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T19:11:33.983 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T19:11:33.998 INFO:tasks.cephadm:Teardown complete 2024-04-23T19:11:33.999 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-23T19:11:34.010 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-23T19:11:34.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T19:11:34.013 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T19:11:34.026 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T19:11:34.151 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-23T19:11:34.162 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-23T19:11:34.162 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T19:11:34.165 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T19:11:34.171 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T19:11:34.180 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-04-23T19:11:34.182 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-04-23T19:11:34.184 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-04-23T19:11:34.185 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T19:11:34.185 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-04-23T19:11:34.185 INFO:teuthology.orchestra.run.smithi050.stdout:^- hv01.front.sepia.ceph.com 2 6 377 34 +179us[ +181us] +/- 593ms 2024-04-23T19:11:34.185 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -302us[ -300us] +/- 56ms 2024-04-23T19:11:34.185 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +462us[ +463us] +/- 84ms 2024-04-23T19:11:34.185 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T19:11:34.201 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T19:11:34.201 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-04-23T19:11:34.202 INFO:teuthology.orchestra.run.smithi177.stdout:^- hv01.front.sepia.ceph.com 2 6 377 33 +164us[ +167us] +/- 593ms 2024-04-23T19:11:34.202 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -307us[ -304us] +/- 56ms 2024-04-23T19:11:34.202 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +453us[ +456us] +/- 84ms 2024-04-23T19:11:34.202 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T19:11:34.224 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T19:11:34.224 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-23T19:11:34.225 INFO:teuthology.orchestra.run.smithi188.stdout:^- hv01.front.sepia.ceph.com 2 6 377 29 +177us[ +177us] +/- 593ms 2024-04-23T19:11:34.225 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv02.front.sepia.ceph.com 3 7 377 93 -306us[ -308us] +/- 55ms 2024-04-23T19:11:34.225 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +453us[ +453us] +/- 84ms 2024-04-23T19:11:34.225 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T19:11:34.226 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-23T19:11:34.237 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-23T19:11:34.238 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-23T19:11:34.249 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T19:11:34.333 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T19:11:34.424 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T19:11:34.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T19:11:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T19:11:34.545 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T19:11:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T19:11:34.574 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T19:11:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T19:11:34.605 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-23T19:11:34.622 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-23T19:11:34.634 INFO:teuthology.task.internal:Duration was 4227.761564 seconds 2024-04-23T19:11:34.635 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-23T19:11:34.645 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-23T19:11:34.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T19:11:34.648 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T19:11:34.651 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T19:11:34.688 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T19:11:34.720 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T19:11:34.724 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T19:11:35.132 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-23T19:11:35.132 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2024-04-23T19:11:35.132 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T19:11:35.158 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-04-23T19:11:35.158 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T19:11:35.187 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-04-23T19:11:35.188 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T19:11:35.231 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-23T19:11:35.232 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T19:11:35.235 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T19:11:35.237 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T19:11:35.299 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-23T19:11:35.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T19:11:35.306 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T19:11:35.309 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T19:11:35.714 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-23T19:11:35.725 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-23T19:11:35.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T19:11:35.751 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T19:11:35.776 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T19:11:35.825 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-23T19:11:35.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T19:11:35.841 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T19:11:35.843 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T19:11:35.861 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2024-04-23T19:11:35.880 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-04-23T19:11:35.897 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-04-23T19:11:35.916 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T19:11:35.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T19:11:35.931 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T19:11:35.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T19:11:35.967 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T19:11:35.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T19:11:35.982 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-23T19:11:35.994 INFO:teuthology.task.internal:Transferring archived files... 2024-04-23T19:11:35.995 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101/remote/smithi050 2024-04-23T19:11:35.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T19:11:36.039 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101/remote/smithi177 2024-04-23T19:11:36.039 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T19:11:36.074 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101/remote/smithi188 2024-04-23T19:11:36.074 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T19:11:36.109 INFO:teuthology.task.internal:Removing archive directory... 2024-04-23T19:11:36.110 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T19:11:36.112 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T19:11:36.115 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T19:11:36.158 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-23T19:11:36.170 INFO:teuthology.task.internal:Not uploading archives. 2024-04-23T19:11:36.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-23T19:11:36.182 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-23T19:11:36.182 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T19:11:36.185 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T19:11:36.187 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T19:11:36.197 INFO:teuthology.orchestra.run.smithi050.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 19:11 /home/ubuntu/cephtest 2024-04-23T19:11:36.198 INFO:teuthology.orchestra.run.smithi177.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 19:11 /home/ubuntu/cephtest 2024-04-23T19:11:36.212 INFO:teuthology.orchestra.run.smithi188.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 19:11 /home/ubuntu/cephtest 2024-04-23T19:11:36.214 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-23T19:11:36.227 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-23T19:11:36.252 INFO:teuthology.nuke:Checking targets against current locks 2024-04-23T19:11:36.279 DEBUG:teuthology.nuke:shortname: smithi050 2024-04-23T19:11:36.280 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T19:11:36.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:55:22.059170', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGaaK12rhtBlQ2ZraE3HzZOX8bInYJtCLxd2m4a+aIEOq0tEQ/DmRtHB2ZsZKFwF1h2qtohYm3R7ducpnFjcA8='} 2024-04-23T19:11:36.309 DEBUG:teuthology.nuke:shortname: smithi177 2024-04-23T19:11:36.310 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T19:11:36.335 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:55:22.057853', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC9QRn+TObUz6O8/tWtEBg6n7xqlLin8R3q7SRJd0CArp42c1jR8FMV4CCgahXIxX7VFx1po6Vf6OAtGjwZHYmg='} 2024-04-23T19:11:36.337 DEBUG:teuthology.nuke:shortname: smithi188 2024-04-23T19:11:36.337 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T19:11:36.367 INFO:teuthology.orchestra.console.smithi050:Power off 2024-04-23T19:11:36.368 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T19:11:36.377 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_17:44:29-fs-wip-lusov-quiescer-distro-default-smithi/7670101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 17:55:22.056185', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL+gYn/+XUNQld8rFQ0rlBXLVFItgMBa/qRJwUNTE+WQ4StgyyRIyyLBbTFawnvfN7pwqXw8GLNCut9Nzodlks4='} 2024-04-23T19:11:36.393 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2024-04-23T19:11:36.393 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T19:11:36.414 INFO:teuthology.orchestra.console.smithi177:Power off 2024-04-23T19:11:36.415 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T19:11:36.430 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-04-23T19:11:36.441 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-04-23T19:11:36.441 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T19:11:36.462 INFO:teuthology.orchestra.console.smithi188:Power off 2024-04-23T19:11:36.463 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T19:11:36.473 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-23T19:11:36.487 DEBUG:teuthology.orchestra.console.smithi188:power off output: Chassis Power Control: Down/Off 2024-04-23T19:11:36.487 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T19:11:36.516 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-23T19:11:40.431 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T19:11:40.475 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T19:11:40.517 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T19:11:40.542 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-04-23T19:11:40.586 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-23T19:11:40.628 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-23T19:11:44.547 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T19:11:44.587 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T19:11:44.629 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T19:11:44.659 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2024-04-23T19:11:44.698 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-23T19:11:44.742 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-23T19:11:44.760 INFO:teuthology.orchestra.console.smithi050:Power off completed 2024-04-23T19:11:48.702 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T19:11:48.743 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T19:11:48.815 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-04-23T19:11:48.853 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2024-04-23T19:11:48.916 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-04-23T19:11:48.954 INFO:teuthology.orchestra.console.smithi188:Power off completed 2024-04-23T19:11:49.061 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 4227.761564254761 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi050 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=57e0caf76fdb7ab8b1358588f292d08519163844 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1ff78855679e49a5a20e6e4df305d879 status: fail success: false 2024-04-23T19:11:49.062 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T19:11:49.160 INFO:teuthology.run:FAIL